When Do Concurrent Design Facilities Need to Be Flexible? Understanding the Limits of Fit With Novel Problem Types

As modern engineered systems become increasingly complex, effective design requires diverse expertise and knowledge to be deployed in an expedient way throughout the design process. Concurrent design facilities (e.g., NASA’s Team X and EADS Astrium Satellite Design Office) hold the promise of shorter design cycles with efficient cross-disciplinary integration. This improvement in efficiency is enabled by a priori decomposition of the design problem. The problem is broken down into tasks that can be worked by different experts in parallel. The tasks are loosely coupled, and the needed interactions among designers are embedded in formal or informal routines (such as team check-ins) or in computational tools (such as shared databases). When this standardized structure fits the problem well, it can enable many simultaneous interactions and drastically speed up design cycles, through hiding irrelevant information, avoiding unnecessary coordination, and freeing designers’ time to focus on the most challenging parts.

However, there is a hidden danger in this approach. Each concurrent design facility (CDF) customizes its organizational decomposition to support the types of problems they typically solve (e.g., earth orbiting spacecraft). However, occasionally the CDF might encounter a design problem that is substantially different from the typical problem (e.g., Martian rover)—with, for example, strong interdependency among subsystems whose designers typically have little need to coordinate. In such a case, an important trade might be discovered late in the design process, or the design might be slow to converge. As this example illustrates, when the organizational decomposition (designers and tasks) does not fit the technical decomposition of the product (the design problem), designers might miss important trades and make poor design decisions. As these concurrent design tools become more popular, it is important to understand these “hidden costs” of the a priori organizational decomposition that enables concurrent design’s efficiency improvements, so that these pitfalls can be avoided.

This study provides insight into the scope of applicability of concurrent design approaches—or, more generally, of assuming an a priori organizational architecture for a novel design problem. Therefore, we aim to understand how the technical architecture of a typical problem (in our case a spacecraft design problem) might deviate from the standard organizational architecture and assess the potential impact on the design process by evaluating the extent to which each of these differences generates problematic “fit” issues that could lead to inefficient or ineffective design processes.

To accomplish this, we work with one concurrent design facility, NASA Goddard’s Mission Design Lab (MDL). First, we describe the assumed “typical” technical architecture for which the organizational architecture is designed. Second, we identify the types of “perturbations” that a novel design problem might present—i.e., the ways in which the design problem’s technical architecture might differ from that which the organizational architecture is designed to accommodate. We do this based on qualitative coding of observational notes from six MDL concurrent design studies, focused on what made each study different from or harder than the norm. Third, we explore how each of these perturbations would impact the overall fit between the assumed organizational architecture and the technical architecture of the novel (perturbed) design problem. To do so, we generate a perturbed technical architecture and assess its fit with the fixed organizational architecture (see Ref. [1]).

The results suggest boundaries to the scope of novel design problems that a concurrent design facility can or should accommodate, and suggest which kinds of novelty can be tolerated and which should be treated more carefully. The perturbations we identify also constitute the start of a language or ontology for discussing the novelty and difficulty of design problems for concurrent design.

This section briefly reviews the literature on representing technical and organizational architecture, theory on fit between product and organizational architecture, and how these concepts relate to concurrent design facilities.

The most common way to represent product and organizational architecture is through a design structure matrix (DSM) [2,3]. DSMs are N × N matrices that show how each element of an architecture is related to the others. DSMs have been used to represent the architecture of many systems, including products, tasks, and organizations [2]. They present a convenient format for analysis [2,4] and have been used to analyze the fit between product and organizational architectures (discussed below) [1,5–7].

Coordination is key to organizational problem solving (including design); however, coordination has a high cost in terms of efficiency. Therefore, multiple scholars have proposed decomposition as a strategy for reducing coordination complexity [8,9]. Decomposition breaks the problem into a set of largely decoupled modules [10–12] which can be worked on separately and in parallel. Decomposition also allows information to be “hidden” within modules (subsystems or business units), which limits the scope of the problem being solved within each module [13] and also constrains the extent of coordination needed among modules.

Importantly, these efficiencies are only achieved when there is a good fit between the product and organizational architecture. In this literature, “fit” is defined in terms of the “mirroring hypothesis,” which holds that the structure of the organization will match the architecture of the product or system being developed [12,14] (organization structure includes, for example, communication links, co-location of teams, etc.). The hypothesis has been investigated and verified empirically [5]. An important corollary of this finding is that a lack of fit can cause problems in the design process or organizational operations because the information hiding can also limit the extent of the solution space explored. In addition, while the information hiding associated with decomposition is necessary to ensure tractability of the design process, it can also limit the effectiveness of the design process. Since, once decomposed, work happens primarily within modules, tradeoffs between design parameters that are shared across module boundaries are less likely to occur, even when they are needed [15,16]. This is because architecture defines how interdependent tasks are coordinated [8,13,17].

Within the mirroring literature, multiple studies have sought to measure fit across industries ranging from computing to aerospace to bicycles. In their comprehensive review, Colfer and Baldwin [5] identify a diversity of approaches to measuring fit, yet across these studies, there is a consistent finding of the importance of mirroring. In prior work [1], we argued for the need to translate mirroring theory into tools that could be adopted to diagnose fit problems in design organizations. To that end, we compared existing fit measures compatible with a DSM measurement framework and demonstrated that two measures [6,7] are most able to see fit problems related to design. Therefore, in this work, we adopt those measures to study the implications of lack of fit (described further in Sec. 3.4).

Concurrent design facilities (CDFs) embody a design approach based on the insight that design of complex systems is more efficient when all relevant (multi-disciplinary) domain experts work in parallel, and typically co-located, to support quick resolution of design conflicts between shared parameters as they arise [18]. To implement concurrent engineering in these settings requires that a typical design process be “baked” into the CDF infrastructure, such as a computational infrastructure that automatically pushes design parameter updates to relevant disciplines. CDFs have gained popularity, particularly within aerospace. Examples include the NASA Jet Propulsion Laboratory’s Team X [19], the European Space Agency’s Concurrent Design Facility [18], and the NASA Goddard Space Flight Center’s Integrated Mission Design Center [20], of which the MDL is a part. These teams were formed in response to the need to create spacecraft conceptual designs rapidly. They consist of a set of subject matter experts in the main subsystems required for spacecraft design, and a set of tools and facilities to support their work, co-located for a short intense period of study.

The past two decades have seen significant interest in concurrent design facilities among both practicing design organizations and academic researchers [18–26]. Much work has focused on the difficult challenge of developing data models, data sharing software, and more recently model-based systems engineering tools for supporting the exchange of technical information during concurrent design [22–24], while other studies have focused on optimizing the design process [25–27]. However, while most studies acknowledge the importance of understanding how humans and organizations interact with the software or process, there is little formal work on the topic beyond testing the above-mentioned process or data models.

As this research has shown, the CDF model works by imposing a particular (organizational) decomposition on complex (technical) design problems, making it possible to quickly design each subsystem and exchange information to deal with complex interdependencies among subsystems. This process has improved design speed by facilitating mundane data sharing tasks through a shared IT backbone. The co-location of the engineering disciplines makes experts accessible and frees them up to focus on the difficult parts of the design [20], but it has also fostered a more inflexible organizational decomposition. As a result, there is a higher likelihood of fit problems when novel design problems are solved by this fixed organizational architecture. This paper aims to explore when—for what types of novelty in design problems—such fit problems are likely to arise.

Identifying which types of design problems are best suited to a given organizational structure requires representing the product, organization, and representative perturbations to either. Here, we build on a long history of representing different aspects of a product, process, or organization in a DSM format (see Sec. 2), customized to suit the specific research setting. To describe the perturbations, which have not previously been characterized, we draw on qualitative inductive techniques. Each aspect is elaborated below.

This work draws on data from NASA Goddard’s MDL, which has a fundamentally similar approach to those used in (at least) two other spacecraft CDFs, particularly in the aspects we study here: the overall interconnectedness of the design problems and the organization. We discuss generalizability further in Sec. 5.

Consider Fig. 1, which summarizes the central concepts of our research approach. The value of the CDF structure lies in its ability to efficiently focus interdisciplinary experts on the most important interactions and otherwise free them to work on their discipline-specific tasks. Therefore, a CDF’s organization is designed to match specific types of problems. For example, the MDL efficiently solves a typical spacecraft design problem. Let the technical architecture of this typical problem be $Pα$ and that of the CDF’s organization, which is designed to match its structure, be $Oα$⁠. However, since most CDFs are tasked to solve a wide range of design problems, the specific design problem encountered in a given study will have the architecture $Pβ$⁠, which may differ in certain ways from $Pα$⁠. When $Pβ$ is solved by $Oα$⁠, if these technical and organizational architectures are a poor fit, problems may result during the design process. For example, key interactions may not be identified until later in the process if there is no organizational structure defined to facilitate information sharing. The extent of this mismatch can be viewed as a difference between $Oβ$⁠, the ideal organization that would have been created to solve this specific design problem $Pβ$⁠, and the more generic $Oα$ that is used instead. This paper investigates what kinds of novelty in the design problem—what kinds of $Pβ$’s—lead to varying degrees of poor fit and thus to likely problems in the design process.

To investigate this question, we leverage qualitative insights from in-depth interviews and observations of six MDL studies to inform a simulation study of the impact of design problem perturbations on organizational fit. Figure 2 provides an overview of the research process. We begin by representing MDL’s $Oα$ based on expert input. Then we draw on observational field notes and interviews to identify archetypal design problem “perturbations”—the ways that particular studies deviated from the assumed “typical” spacecraft architecture. Next, we designed a computer experiment which systematically varied the extent of perturbation embodied in a sequence of $Oβ$’s. Finally, we measured the fit between each generated $Oβ$ and the reference $Oα$⁠, which enables us to compare the impact of different types of perturbations on fit, while controlling for the extent of the change. The subsections below describe each step in detail.

The first step is to represent the standard organizational architecture, labeled $Oα$ in Fig. 1. Following standard approaches, we represent the architecture of an organization as a DSM [2,28,29]. The rows and columns of the DSM represent the organizational units, and the entry in each cell represents the extent of organizational ties, such as meetings or shared databases, between the two organizational units. In the concurrent design context, each row or column represents a single designer, typically responsible for one spacecraft subsystem or for an integrative role or discipline such as systems engineering.

A concurrent design team does not have the same kinds of organizational ties, such as formal liaising roles, that would occur in a larger design organization. Instead, these ties take the form of a routinized process. For example, these routines create the expectation that the power designer will receive an estimate on spacecraft power requirements on the first afternoon, or that the communication and operations designers will coordinate on the operations approach on the second morning. This routinized process is efficient because it saves time and attention: since power knows she will receive the estimated power requirements on the first afternoon, she does not waste time asking for it earlier and plans her design work accordingly.

To capture the organizational architecture of the MDL, we rely on a previously undertaken study of the activity within the MDL by a senior systems engineer. He produced a series of notional activity timelines for each design engineer, which captured precisely the routinized process that constitutes the MDL organization. The timelines showed when each required design variable (DV) is “passed” between different designers over the course of a study. Each DV represents a core element on any spacecraft design (such as antenna type or propellant tank size). Many of them align to well-defined subsystems (such as communications or propulsion), but others describe a necessary functional component of the design (such as a pointing budget or a mass estimate). The “passing” of a design variable may represent different kinds of information; for example, the systems engineer and customer roles typically provide initial requirements on a DV, and later, designers may pass initial values, updates, or estimates to one another. This represents the exchange of necessary information to design each subsystem of the spacecraft. The timelines thus capture the expected exchange of key information among designers, i.e., the routinized process that constitutes the organizational architecture.

To represent this organizational architecture in a DSM as described above, it was necessary to map the information contained in the timelines, which shows what DVs are passed among designers at a particular time in the study, to a DSM structure, which shows the extent of organizational ties among the designers (for clarity, we henceforth refer to these designers as “roles”). To do so, we first construct a matrix that we label a DV-R matrix, because it maps DVs to the roles (R) that deal with those design variables. This can be thought of as the mapping from $Pα$ to $Oα$⁠. The DV-R matrix contains a row for each DV and a column for each role (R). The entry in each cell indicates the number of times that each role “passed” each DV—meaning that the DV was either input or output by that role. This information was extracted directly from the timelines described above.

Figure 3 provides an example. The DV-R matrix (top) includes two design variables and three roles. For the first design variable, systems is included in four handoffs, mechanical in two, and propulsion in three. The second design variable is handed off twice between systems and mechanical.

The next step is to convert the DV-R matrix into an R–R matrix which describes the organizational architecture as a DSM; this corresponds to $Oα$⁠. Each cell in this R–R matrix counts the number of DV exchanges among each pair of roles. To compute this from the DV-R matrix requires a series of steps. Figure 3 provides an example. We proceed through the rows of the DV-R matrix one by one. For each DV, we first create a separate R–R matrix. The DV’s row in the DV-R matrix contains some number of nonzero entries; let $R$ denote the set of roles with nonzero values in their columns for this DV, and let R₁…R_N denote the roles belonging to $R$ in order from the lowest value to the highest value. Let the value in R_n’s column of the DV row be h_n, representing the number of handoffs of this design variable by role R_n. We begin with R₁, the role with the lowest number of handoffs of this design variable (in Fig. 3’s example, for the DV1 row, R₁ is mechanical). In the R–R matrix, we enter h₁ in the cells corresponding to R₁’s interactions with all other roles in $R$⁠. This assumes that each handoff that involves R₁ also involves all the other roles concerned with this design variable. Next, we consider R₂. In the R–R matrix, we enter h₂ in the cells corresponding to R₂’s interactions with all other roles in $R$ except R₁. This assumes that each handoff, beyond those accounted for with R₁, also involves all the remaining roles concerned with this design variable. The pattern continues through R_n−1, for whom we enter h_n−1 in all the cells corresponding to R_n−1’s interactions with all other designers in $R$ except R₁…R_n−2, i.e., with R_n only. At this point, the process stops, and we proceed to the next DV.

The final step is to sum all the individual R–R matrices for each DV into a single R–R matrix, which represents the final organization as a DSM. The entries in the DSM indicate the number of handoffs of any DV between each pair of roles. The number of handoffs of DVs among roles indicates the extent of organizational ties between those roles—the routinized process of concurrent design. Thus, this DSM represents the organizational architecture, or $Oα$⁠. More specifically, this DSM approximates the organizational architecture, because it does not represent some details such as the order or directionality of handoffs. While these aspects are important drivers of design outcomes, they are less important in understanding fit, which depends on coordination between roles and is inherently bidirectional. Therefore, this simplification is useful for our purpose because it eases interpretation of the comparison between the $Oα$ and $Oβ$ architectures.

The second step is to observe how particular studies differ from the assumed “typical” spacecraft architecture and identify a set of archetypal perturbations. Spacecraft design problems may differ from $Pα$ in that they have unique subsystems, capabilities, or operating modes that generate interdependencies that deviate considerably from the standard architecture. For example, a low orbit that limits ground station visibility, combined with a requirement for rapid and high-volume data transfer, might make for stronger-than-expected interdependencies between communications and orbit design variables. Alternatively, a new instrument type with strange or unwieldy geometry might make the mechanical designer’s job much more difficult than usual. To understand the common ways in which novel spacecraft designs may differ from the standard architecture, we used qualitative interviews and observations [30] of six Goddard MDL studies. We looked for evidence of what was unusual or unusually difficult about the design problem (as distinct from problems with the design process). Details are described in the following paragraphs.

The data came from a series of concurrent design studies by NASA Goddard’s MDL. We chose to study a series of related and comparable conceptual design studies for a single type of science mission (details are not provided due to confidentiality requirements). The missions all had the same cost cap, technical risk acceptance level, and were primarily using mature technologies (outside of their instruments). Because of the category of high-risk, high-cost mission, there was comparably high variation in the driving requirements across studies. This was important because it ensured variation in the fit between $Pβ$ and $Oα$⁠. Six studies were chosen to balance depth and breadth of observation [31]. The studies were conducted in the MDL. Each study ran approximately 30 h, over the course of a week, ending with a set of presentations by each subsystem design engineer. Between one and three, researchers (including at least one of the authors) attended each study.

Before each study, the facilitators and lead systems engineers were interviewed to review mission-specific study details and to estimate the technical challenges, referred to as “long poles,” that might arise. At the end of each day, the researchers again interviewed the facilitators to ask whether any new challenges had arisen, how “long poles” were changing, which challenges were resolved, and whether there were any other “unusual” qualities of the study. During the study, the researchers observed all communications among different roles, as well as key discussions including the twice-daily tag-up meetings. They noted key challenges and resolutions from the key discussions and tag-ups.

The researchers took detailed notes either during or immediately after the interviews and observations. These field notes were then transformed into a structured narrative for each study [30,32]. The narrative first described the spacecraft design problem including the science goals and key features of the design, then listed the main challenges or “long poles” that constrained the design, and finally described any other challenges that arose—such as late discovery of an error or unresolved problems. These latter were not included in the present analysis, because they relate to problems in the design process rather than features of the design problem itself.

These narratives were then examined to identify the common types of unusual features, or perturbations from $Pα$⁠, in these design problems. We distinguished these from other issues that arose for reasons unrelated to the nature of the design problem, such as inadequate definition of requirements or absent designers; the latter were not included in the study. We sought to identify archetypes or categories of perturbations, rather than to specify which perturbations characterized each Goddard study. To do so, we used standard open coding techniques [30,33,34]. The relevant descriptions from the narratives were “coded” to identify each type of unusual feature, and similarly coded features were then examined to refine the codes and their applications to the data. This resulted in a small set of common perturbations observed across multiple studies. For example, in one study, the attitude control system (ACS) design was particularly difficult due to an odd geometry, high mass, and fast slew rate. In a second study, onboard processing requirements for multiple data streams made the avionics design challenging. Both of these were coded as representing one archetypal perturbation: difficult role, or a particularly challenging design for one designer. In contrast, other features were coded as a difficult design variable(s) archetype, wherein the multiple roles’ subsystems constrain one another and they must work together to resolve a problem with one or several design variables—for example, in one study, a low orbit resulted in limited windows of ground station visibility, constraining the total volume of data that could be downloaded.

This process resulted in a small set of perturbation archetypes that describe the common types of unusual design problem features in the six studies we observed. These results are summarized in the four left-most columns of Table 1. Each type of perturbation is defined, then evidence and examples are given from the Goddard studies. Across all six studies, ten examples of unusual challenges were observed, and these were classified among five types of perturbations: difficult role, difficult design variable(s), trade, novel scope, and novel interdependencies. These results are unlikely to be surprising: experienced designers are well acquainted with many of these challenges. However, it is nevertheless useful to separate them into these categories to assist practitioners in determining which type they might be facing and what to do about it. Since our goal with this analysis is to identify archetypal perturbations, it is less important that some perturbations occur with more frequency than others. In the results we will first focus on differences among archetypes and then revisit the differences in frequency in providing practical guidance to organizations adopting CDF design models.

Each type of perturbation is operationalized in the $Oβ$ versus $Oα$ framework by modifying the DV-R matrix. These operationalizations are explained and justified in the two right-most columns of Table 1, and examples that illustrate how the DV-R matrix is perturbed are given in Fig. 4 (recall that the DV-R matrix describes how often each DV is passed between roles; it therefore represents a mapping between $Pβ$ and $Oβ$⁠). For example, the difficult role perturbation is operationalized by modifying the DV-R matrix as follows: The difficult role’s column is multiplied by a difficulty factor to represent its additional coordination burden, such as updates to the values for its design variables. As a second example, the difficult design variable(s) perturbation is also operationalized by multiplying part of the DV-R matrix by a difficulty factor, but here we multiply the rows for each of the affected design variables. Determining these operationalizations for each archetype was necessarily a subjective process, but it was informed by our interviews and observations of the design process.

The third step is to generate a set of systematically perturbed $Oβ$’s to understand how different types of perturbations, or archetypal mismatches, affect fit. To do this, we design a set of scenarios—particular DV-R matrices—with which to compare the effects on fit of the various types of perturbations. We designed three scenarios for each type of perturbation which vary in the “connectedness” of the affected roles and design variables. For each scenario, we also tested two or three different difficulty factors (f) to vary the extent of the perturbation. Most of the scenarios were loosely based on the qualitative observations. The scenarios are given in Table 2.

Each scenario represents a different perturbation of a design problem, with a different corresponding ideal organizational architecture $Oβ$⁠. Once the perturbed DV-R matrix is created, a new R–R matrix can be produced by following the steps described in Sec. 3.1. This new R–R matrix represents the ideal organization $Oβ$ for solving the perturbed problem $Pβ$⁠. In all, 36 systematically varied $Oβ$’s were created (for two types of perturbations with three scenarios each and three difficulty factors, plus three types of perturbations with three scenarios each and two difficulty factors).

The final step is to determine how each of these perturbations could impact the fit between the organization’s standard architecture $Oα$ and the ideal architecture for a particular design problem $Oβ$⁠. To do so, we use a measure to evaluate the fit between $Oα$ and $Oβ$⁠. This fit measure quantifies the extent of “overfit” and “underfit”—in other words, the extent to which the standard organization $Oα$ has too much or too little capacity to handle the needed communication among designers. It can also identify where, i.e., for which designers, the over- and underfit conditions manifest. With these results, we can identify the types of problems that might be expected in the design process with each type of perturbation.

Finally, an overall coordination deficit D can be computed by summing all the role deficits and dividing by two, to avoid counting the same deficits twice.

A surplus indicates the extent to which the organization has unneeded coordination pathways. In our study, there are no surpluses, so we do not describe the method for computing them in detail.

Intuitively, the coordination deficits D and D_i represent the percent of needed coordination that is missing, out of all the coordination that is needed in the entire network. The local coordination deficit L_i, on the other hand, represents the percent of needed coordination that is missing, out of all the coordination needed by role i. Even where D_i is relatively low, the local L_i may be higher, indicating a large change for role i’s coordination requirements.

These measures were applied to all the $Oβ$’s generated from the scenarios described in Table 2 to measure the fit between the standard organizational architecture $Oα$ and the perturbed architectures $Oβ$⁠. The results are given in the next section.

The analysis of the perturbation scenarios (see Table 2) shows how they impact fit between the standard organizational architecture $Oα$ and the ideal organizational architecture for the perturbed design problem, $Oβ$⁠. Fit was measured as an overall coordination deficit for the study, D, and as a deficit for each role, D_i (complete results for D_i are given in Fig. 8 in the  Appendix). Larger deficits indicate that the standard organization is missing more required coordination pathways—indicating a higher likelihood of problems with the design process.

To gain insight into how the deficit changes as the design problem gets more and more novel, we plot coordination deficit against the total magnitude of changes in the DV-R matrix, which indicates roughly the extent of the perturbation. (As an example of computing this x-axis value, recall the difficult DV example from Fig. 4, in which the DV2 row is multiplied by the difficulty factor. Assume the difficulty factor f is 3. There are two roles that work on this DV, systems and mechanical, each with two handoffs in the original DV-R matrix. When we multiply the entire row by f = 3, the perturbed DV-R matrix shows each of these roles with 2 × 3 = 6 handoffs each, for an increased magnitude of 6 − 2 = 4 handoffs per role, or 8 additional handoffs total in the perturbed DV-R matrix.)

Consider two views of the overall results (Fig. 5). Each plot shows the coordination deficit as a function of the total magnitude of changes to the DV-R matrix (explained above). Figure 5(a) shows results for only the scenarios with the lowest difficulty factors (either 3 or 1, as shown in Table 2). By focusing only on the lowest difficulty factors, we see only the perturbations that are likely to arise on a regular basis in real studies, without the “extreme” scenarios. Figure 5(b) includes all scenarios with all difficulty factors. Least squares lines were plotted for each perturbation type to show the approximate slope for each type of perturbation.

Consider first the slope for each type of perturbation, which indicates how much the coordination deficit rises as larger changes are made in the DV-R matrix. Each change to the DV-R matrix potentially, but not necessarily, generates an unmet need for coordination. The slope is therefore important because it shows which types of perturbations—novel elements in the design problem, represented as changes to the DV-R matrix—generate small coordination deficits and which generate large coordination deficits, which may lead to problems in the design process. For example, in Fig. 5(a), both the difficult DV perturbation and the novel interdependencies perturbation have scenarios with an x-axis value near 20, but they generate very different coordination deficits for this same value of the x-axis. The former might be easily accommodated by the standard organization whereas the latter might generate problems with the design process. For this reason, it is useful to examine the slope for each perturbation to distinguish which types of perturbations are likely to lead to large coordination deficits even for minor perturbations in the design problem.

Unsurprisingly, all types of perturbations have upward slopes, indicating that they generate increased coordination deficits as more changes are made to the DV-R matrix. However, some types of perturbations generate much higher coordination deficits for each change to the DV-R matrix.

The perturbation with the lowest slope is a difficult role: the overall coordination deficit rises very little even for high x-values. This is because the increased difficulty in one role propagates very little to other roles. This can be clearly seen in Fig. 6 (or Fig. 8(d) in the  Appendix), which shows the coordination deficit for each role: The affected role has the largest coordination deficit, the integrator, Systems, has a somewhat smaller one, but there is little or no coordination deficit for other roles. Indeed, it does not matter how extreme the difficulty rises for any one role—in our scenarios, the coordination deficit is the same whether the difficulty factor is 3 or 9. This is because, in our operationalization, it is only handoffs between roles that generate a need for coordination, and increasing the difficulty for one role alone increases its own handoffs but may not affect those of other roles. (For example, the role might issue 9 times the usual number of updates to its DVs, but the need for coordination among roles is determined by the number of handoffs others expect, not the number of updates issued). This is a reasonable model of practice, in which difficulty in one role only “spills over” to others when they are involved in solving the problems. Of course, the design process may still go poorly due to the difficulty of the design problem, but our results suggest these problems are not due specifically to a lack of fit between the organizational and technical architectures.

The same principle explains why the trades perturbation generates the second-lowest slope and the difficult DVs perturbation generates a higher slope. Difficult DV perturbs an entire row whereas trades perturb only a few cells in each row. Therefore, in the former case, the need for coordination goes up across all the affected roles, but in the latter, the coordination deficit is lower and less far-reaching. (These localized effects are clearly visible in Figs. 8(a) and 8(b) in the  Appendix.) While the magnitude of the highest coordination deficit in each scenario is roughly comparable, the difficult DV scenarios generate coordination deficits for a much wider range of roles. Trades thus have more localized effects than difficulty in an entire design variable.

A novel scope—novel design variables—have a slightly greater potential to generate large coordination deficits than difficult DVs. They add entirely new rows (DVs); therefore, they have similar effects as difficult DVs but may also generate completely new interdependencies, leading to greater coordination deficits. On the same principle, novel interdependencies have the highest slope. For a given change in the DV-R matrix, it leads to the highest coordination deficit because it generates entirely new interdependencies, guaranteeing that a coordination deficit will be created where no coordination was required before.

It is also worth considering the actual magnitude (rather than the slope) of the coordination deficits generated by each scenario. This provides some sense of how much coordination might be missing and how big the resulting problems might become. For this analysis, we focus on Fig. 5(a), which shows only those scenarios with the lowest difficulty factor—those most likely to arise in practice. We also examine the local coordination deficits for each role, L_i, which show how much more communication is needed than expected for a particular role (see Fig. 9 in the  Appendix for complete results).

Difficult roles lead to very small coordination deficits overall, less than 3% of the total needed communication, suggesting limited problems would result. The most challenging scenarios in Fig. 5(a) generate deficits between 10% and 20% of the total needed communication, suggesting a potential for some difficulties, depending on how those deficits are spread throughout the organization. Examining the local coordination deficits (see Fig. 9 in the  Appendix) reflects that spread and, more importantly, shows how much additional communication might be required of a given role than expected by the standard organizational routines. The results show the potential for quite serious problems in the design process in some scenarios. For example, Fig. 7 (or Fig. 9(b) in the  Appendix) shows that for scenario T.1, a realistic trade between ACS, power, and mechanical roles involving three design variables, the three affected roles may need to communicate 28–38% more than they would under the routine process. This is a significant change for each of these roles, even though the overall coordination deficit is only about 10%. Similar issues can be seen for other scenarios: for difficult DV, local deficits up to 50% and 11% for overall deficits of 20% and 6%, respectively; for trades, local deficits up to 38% and 29% for overall deficits of 10% and 6%, respectively; and for novel interdependencies, local deficits up to 33% and 17% for overall deficits of 13% and 7%, respectively. In the case of a trades perturbation, it is relatively easy to predict where the need for increased coordination will arise—among those roles involved in the trade—but in the case of difficult DVs and novel interdependencies, this is not as easy to predict and therefore more likely to go unnoticed until problems arise late in the design process.

Design organizations are increasingly adopting rapid design capabilities using variations on concurrent design organizations. These structures offer the potential for drastic efficiency gains because they facilitate only the most important design interactions among disciplines. However, there is a hidden risk to pre-decomposed structures of this kind. The features that enable efficiency do so by hiding unnecessary information from other disciplines. This works extremely well when there is a strong match between the pre-decomposed organization and the product being designed [12], but in cases where the product does not match the organization, key interactions can be missed until late in the design process. This can lead to much higher costs due to late design iterations (see “rule of 10” [35]).

So far, guidance on how much mismatch is problematic has been limited to categorical assessments which suggest that design outcomes are poor when there is a lack of fit [5–7,36,37], with little or no discussion of which types of mis-fit will be most problematic. With this study, we provide new insight into this question. Specifically, we inductively characterize typical perturbation types and assess their impact on coordination deficit, a measure of the extent of fit. As expected, the greater the extent of mismatch, the more coordination deficit was observed. Much more interestingly, different perturbation archetypes lead to systematically different levels of coordination deficits, even when the magnitude of the perturbation is the same. For example, a difficult role leads to very limited coordination deficits regardless of how large the perturbation becomes, while novel interdependencies generate very large coordination deficits for even small perturbations to the design problem. This means that if designers only focus on the extent of the design problem perturbation (which is easily observable), they may drastically underestimate the potential impact (difficult to observe or predict), depending on the perturbation type. Now that we have connected type of perturbation to impact, we provide a practical sense of scale to the expected impact.

These results are practically important because they suggest different actions for organizations seeking to adopt CDF-like structures. Perturbations localized to one or a few roles, such as trades or difficult roles, can be handled within the existing structure. This is because they affect a particular role (or roles) but have limited spillover effects and do not lead to unpredictable coordination issues. For example, trades tend to generate smaller coordination deficits for a given magnitude of perturbation to the design problem, and the local coordination deficits arise in predictable locations—among those roles involved in the trade. These types of perturbations can be accommodated by, for example, adding a junior Mechanical Designer to a particular study, or planning extra (focused) interactions associated with tricky trades through planned sidebar discussions. Therefore, even large perturbations of this type do not necessitate a novel structure.

In contrast, diffuse perturbations, like those associated with difficult design variables, novel scope, and novel interdependencies, present a higher potential for coordination deficits and their effects are more difficult to predict. In this study, we only measured percentage coordination deficit and not the potential impact. However, as coordination deficit increases, so does the potential to miss critical trades. This is the case where the information hiding that intends to support efficiency actually leads to serious design problems later in the study. We were told anecdotally that the success of one of NASA’s CDFs had prompted increasingly “different” studies to be proposed. In one case, an organization that was originally designed to support Earth-observing satellite missions was tasked to study a planetary rover. In that case, the key design interactions were between disciplines who do not typically interact (novel interdependencies and novel scope) which led to less-than-satisfactory study results. Fortunately, thanks to talented and experienced study facilitators, the issues were caught, but this ad hoc adaptation also limited the efficiency gains of the typical structure. A related phenomenon was also reported in the literature through Henderson’s seminal study of the photo lithography industry, where institutional routines (that hid information) caused the firm to miss the implications of adopting next generation technology on their processes [38].

This suggests that while it may be possible to accommodate fit problems through ad hoc adaption, this may limit the value of adopting a concurrent design approach in the first place. Particularly in the case of diffuse fit problems, an organization may be better off using a different structure for studies that deviate from the reference in this way. The implications are twofold. First, raising awareness of the potential challenges resulting from particularly novel design problems can help CDF facilitators and designers to catch problems early and setup missing coordination pathways, thus enabling ad hoc adaptation as needed. Our work suggests when—for what types of design problems—CDF personnel should be particularly alert to such problems. Second, our results suggest there may be a need for alternative and/or more flexible CDF organizational architectures that can handle a wider range of design problems. Future work could build on our framework to design such architectures.

It is worth reflecting on the generalizability of our findings, given the relatively simple simulation framework and the fact that our evidence is drawn from one context. While there is certainly room for analysis of other spacecraft and non-spacecraft CDF processes, much more detailed system representations, and a wider range of perturbation scenarios, we believe that the basic insights will hold for the following reasons (respectively). First, the propagation of perturbation-driven problems within the organization depends on the interconnectedness and size of the CDF organization (Goddard’s MDL in this case). Our scenarios investigated perturbations to elements with different levels of connectedness, with results following the observed trends. Furthermore, any set of design problems that warrants a specialized approach such as concurrent design is likely to exhibit complex interconnections and a non-trivial size, so we expect the results to be relevant to a wide range of CDF organizations. Care should nevertheless be taken in applying the insights to problems that differ significantly from spacecraft design. Second, while more complex system representations could be employed, here we leverage empirical insights in an abstract experimental setup to gain both relevance and more general applicability [39]. In doing so, we represented generic perturbations to any system that can be represented as a DSM [2]. Thus, while the specific perturbations are derived from our observations of spacecraft design, their operationalization as weighting rows and columns and introducing new dependencies cover the logical space of potential changes, while also having “face validity” based on discussion with expert designers. Therefore, these insights should apply to any context that can be adequately represented by these abstracted perturbations. The area where there is most potential for future work is in the design of perturbation scenarios. Here we covered a range of scale and scope, but there are many more types of scenarios that could have been considered. The clear differences in the regression lines give us confidence that the observed differences are meaningful, but for this guidance to be more directly actionable, there is likely benefit to more focused study in the range of “perturbations” that a particular design organization is most likely to experience.

Overall, our results suggest that standardized organizational structures and routinized design processes, such as those embodied in concurrent design approaches, are appropriate for many types of design problems. Indeed, the most common perturbations we observed in practice are those least likely to lead to high coordination deficits—difficulty confined to one role or to a single trade. However, our work also highlights the potential for large problems when the rarer cases of difficulty in entire design variables, novel scope, and novel interdependencies arise. To that end, this work provides important insight into the differences among perturbation types in impacting design outcomes, and thereby suggests boundaries to the scope of design problems that a CDF can or should accommodate. We also create a language or ontology for classifying and discussing the types of perturbations that might be encountered in novel design problems, and a framework to provide more specific guidance about how to manage mismatches that arise. This provides an important foundation for future work to directly link these mismatches to design and process outcomes.

This material is based upon work supported by the National Science Foundation under Grant No. 1563408. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We also drew on earlier work on related parts of this project by Connor Forsythe and Victoria Nilsen. Finally, we are very grateful to the leaders and designers of Goddard’s MDL and JPL’s Team X for their time and insight.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.

The appendix includes two figures, Figs. 8 and 9, with more complete results, as described in the paper.