Omitted technical measures: establishing evidence for the necessity of problem formulation and validation in system alternative selection.

Abstract

Design and acquisition of large-scale complex engineered systems can use technical measures to compare system alternatives through setting constraints on those measures and/or providing objectives using the measures. Selecting a technical measure set can be uncertain, with little selection guidance available, and difficult to validate, potentially leading to omitting technical measures. This research examines the impact of omitting technical measures on system alternative selection using a case study of real-world technical measures and system alternatives. A requirements-based constraint framework and an optimization-based objective function framework are developed using a set of real-world technical measures. The research models how omissions of technical measures lead to choosing a different system alternative. The impacts are demonstrated through an application of the NASA Human Landing System (HLS) using 13 system alternatives, including the systems proposed to NASA by Blue Origin, SpaceX, and Dynetics. The research finds that omissions in the constraint framework open the design space, potentially changing the system alternative chosen. Omissions in the objective function framework alter the indicated ordinal preference for the system alternatives, changing the system alternative chosen. An omitted technical measure on the side of the acquirer may change the system alternative selected, directing millions of dollars towards a specific organization and system. This research highlights the practical impacts of omissions of technical measures for awarding contracts during system acquisition. The likelihood of omissions, the difficulty to validate outcomes, and the impacts of omissions demonstrated in this research form evidence that the connection amongst problem formulation, problem solving, and validation must be emphasized if used for system alternative selection.