The performance of a team with the right characteristics can exceed the mere sum of the constituent members' individual efforts. However, a team having the wrong characteristics may perform more poorly than the sum of its individuals. Therefore, it is vital that teams are assembled and managed properly in order to maximize performance. This work examines how the properties of configuration design problems can be leveraged to select the best values for team characteristics (specifically team size and interaction frequency). A computational model of design teams which has been shown to effectively emulate human team behavior is employed to pinpoint optimized team characteristics for solving a variety of configuration design problems. These configuration design problems are characterized with respect to the local and global structure of the design space, the alignment between objectives, and the resources allotted for solving the problem. Regression analysis is then used to create equations for predicting optimized values for team characteristics based on problem properties. These equations achieve moderate to high accuracy, making it possible to design teams based on those problem properties. Further analysis reveals hypotheses about how the problem properties can influence a team's search for solutions. This work also conducts a cognitive study on a different problem to test the predictive equations. For a configuration problem of moderate size, the model predicts that zero interaction between team members should lead to the best outcome. A cognitive study of human teams verifies this surprising prediction, offering partial validation of the predictive theory.
It is crucial that teams be managed appropriately in order to maximize performance. This often involves balancing between extremes. Teams that interact too frequently can be plagued by premature convergence and subsequent fixation on a poor solution. Likewise, teams that interact too little may never benefit from the progressive narrowing of focus onto the most promising solutions. This work investigates how the properties of a design problem can be used to inform the selection of the best team characteristics for solving it. Developing this relationship requires team performance to be assessed on a variety of different design problems and with respect to a variety of different values for a set of team characteristics. This quickly compounds the number of conditions that would need to be evaluated, resulting in a research study requiring an unmanageable number of participant-hours. For that reason, this work simulates the performance of engineering design teams using a computational model that can be parameterized to reflect team characteristics. We specifically investigate the development of such relationships for configuration-style design problems and make recommendations for small teams that are without any specialized hierarchical structure.
A variety of definitions for the word team have been supplied in the literature [1–4], but two concepts are pervasive across these definitions: multi-agency (the composition of a team as two or more individuals) and communication (the ability of those individuals to exchange information). These two concepts are central to the nature of teamwork, and investigating their relationship to team effectiveness should provide fundamental insights into the team-based search for solutions. This work operationalizes the concepts of multi-agency and communication by specifically investigating the impacts of team size and frequency of interaction. These two characteristics are of specific interest for engineering design because they help to highlight the tension that exists between breadth and depth of search in the design space. Larger teams can enhance the breadth of search for solutions, but extremely divergent search may be unnecessary and wasteful for some problems . More frequent interaction leads to deeper and more focused search, but it may also lead to design fixation . The appropriate selection of values for these characteristics ensures that teams are able to diverge and converge in a way that is appropriate for the task at hand. The results of this research provide a means for selecting appropriate values for these characteristics before work begins on a design problem.
Team size plays a role in the search for solutions, but prior findings are mixed. Studies that report negative results for larger teams typically find that larger teams are plagued by low efficiency and coordination issues [7–9]. In contrast, other work has shown that larger teams may benefit from concurrent team work and a greater breadth of experience and opinions [10–12]. A meta-analysis of team characteristics showed that there is a small positive relationship between team size and performance . However, a more detailed analysis indicated that the relationship between team size and performance depends on the type of team—teams brought together to accomplish a finite project benefit from larger sizes, but teams that work together continuously do not . It was also discovered that management teams benefit from larger sizes . This aligns with other work indicating that optimal team size may be task- or at least domain-dependent [14,15].
In practice, many design tasks are limited by constraints on human power, (e.g., a fixed number of billable hours available for a project). This raises a perennial question: should resources be concentrated within a small team, spread among many individuals in a larger team, or something in between? With respect to software development, research has shown that the answer to this question depends in part on how easily the project can be partitioned into subtasks, and whether or not significant communication overhead is necessary after partitioning [16,17]. Analysis of completed projects has shown that while increasingly complex projects demand larger teams, larger teams also tend to be less efficient . This implies that there exists an optimal team size that depends on project properties. This possibility leads to the creation of a theoretical model relating the optimal size of software development teams to the predicted size of the project .
The impact of team size has also been explored in other domains. Computational work in social choice theory has shown that smaller teams are capable of making decisions that more fairly represent the preferences of the team . In addition, work on team-inspired agent-based optimization algorithms has shown that the optimal agent team size depends on the objective function .
The frequency of interaction is a common measure of communication within a team [22–25]. It has been shown that the relationship between frequency of interaction and task performance is approximately quadratic in cross-functional teams . High and low interaction frequencies result in lower performance, with a well-defined optimal interaction frequency . A computational model was developed to investigate this phenomenon further, and it indicated that higher interaction frequency tended to decrease the quality of communication . Further computational work demonstrated a relationship between optimal interaction frequency and project complexity .
With respect to design, the benefit derived from interaction generally arises from the ability of individuals in a team to explore a variety of options, but then to collaboratively focus their attention on a shrinking set of the most promising alternatives . However, interaction is not always beneficial when it is allowed. Computational simulations indicate that excessively frequent interaction can be detrimental to teams, resulting in the implicit prioritization of consensus over the search for good solutions . This shift in priorities within a team is similar to a psychological phenomenon known as groupthink which can be harmful to decision-making teams . In contrast, less frequent interaction may lead to the formation of weak ties between members of a team . Weak ties can be beneficial because they facilitate the transmission of diverse perspectives between individuals . Other research has examined how team interaction can be structured to make teams more resilient to change by weakening confirmation bias .
In a study of the connection between problem formulation and creative ideation outcomes, participants were presented individually with a conceptual design problem . Characteristics of their problem formulation process were tracked using the P-map framework , and the outcomes of their work were quantified with respect to the ideation effectiveness metrics developed by Shah et al. . Regression analysis was then used to relate the P-map variables to those ideation metrics, providing a predictive relationship .
The current work also uses regression to provide predictive equations, but focuses on predicting the team characteristics that lead to the best solutions to configuration design problems. Further, the current work does not involve human studies to determine the relationship, but instead utilizes the cognitively inspired simulated annealing teams (CISAT) modeling framework  (discussed in more detail in Sec. 2). This framework provides simulation capabilities for predicting the performance of human design teams. CISAT allows the user to accurately control the activity of simulated engineering design teams and also makes it possible to rapidly and efficiently evaluate the performance of large numbers of teams with different characteristics.
The primary objective of this work is to establish and demonstrate the usefulness of a method for predicting the best team characteristics for solving a given configuration design problem based on the properties of that problem. Two cases are specifically addressed: Case A addresses the scenario in which a design team with a fixed size must determine how frequently to interact while solving the problem at hand. This is a more likely scenario in smaller firms that have less staffing flexibility. Case B examines the scenario in which the team will interact at a fixed frequency, and the best team size must be chosen. This case might apply if the members of a team are not co-located and meetings are scheduled intermittently, resulting in a fixed interaction pattern. Several stages of work are necessary to produce predictive relationships for these two cases:
A set of design problems is first defined (see Sec. 3).
The properties of these design problems are then computed (see Sec. 4).
The CISAT framework is used to find the best team characteristics for solving each problem (see Sec. 5).
The result of this process is a tool that can be used to inform the selection of team size and interaction frequency for solving a given configuration design problem. Although team size and interaction frequency are the focus of this paper, the method can be used to examine and design other team characteristics that can also be tested within the CISAT framework. Further, the relationships identified through this method apply generally and are not solely applicable to the problem domains used to train the model.
For most design problems, common practice dictates that a team should be formed to collaboratively produce a solution. As will be shown, however, the predictive equations developed through this work indicate that regular interaction is an inferior approach for the design of an internet-connected cooling system. The method instead predicts that the optimal design of the team would have each member working individually and in isolation, with the best solution chosen as the final design. A cognitive study of human teams (Sec. 8) demonstrates that this surprising prediction is actually correct, thus providing a partial validation of the predictive theory. This result also demonstrates that the predictive models developed earlier in the paper apply across a range of configuration problems and not only to the problem domains on which the models were trained.
Overview of the CISAT Modeling Framework
The cognitively inspired simulated annealing teams (CISAT) modeling framework is used here in the place of human studies to evaluate the performance of engineering design teams because of the large number of conditions involved in this work. CISAT is an agent-based platform that is designed to emulate patterns of behavior and performance exhibited by small human teams while solving configuration design problems . CISAT simulations have been compared to the results of a human study, and CISAT effectively reproduced human trends in solution quality, level of divergence within the team, and the selection of topology versus parameter modifications .
In CISAT, every agent represents a human team member. These agents utilize simulated annealing  constructs in their search for solutions that optimize given objective functions. Simulated annealing algorithms progressively transition from stochastic search to more deterministic search during solving. A similar pattern is evident in individual human problem-solvers, which allows simulated annealing to effectively model individual behavior and performance for both puzzle problems  and design problems . CISAT employs simulated annealing within a multi-agent framework, enabling the effective modeling of design teams for configuration problems.
Within this multi-agent framework, CISAT agents are allowed to adaptively employ different search strategies , learn the best order in which to apply rule-based modifications to the current solution , and change their breadth of search according to their proximity to target design values . This allows each agent to develop its own solution concept independently from other agents in the team. For multi-objective problems, CISAT agents choose a randomized weighting of objective functions, further facilitating the development of unique activity and solution preferences.
When interaction occurs, agents have an opportunity to share their current solution concepts. The way in which CISAT agents interact and collaborate is structured in order to reflect behavior that has been observed in human design teams. At the beginning of every iteration, each agent probabilistically decides whether or not to interact with the other agents in its team. If an agent chooses not to interact, it continues to work on its current solution. If it does choose to interact, then it evaluates the solutions currently being pursued within the team and probabilistically selects one to adopt as its own. The weights used in this selection process are initially computed according to solution quality, with higher weight being placed on higher quality solutions. Weights are then modified so that agents do not greedily pursue the solutions with highest apparent quality . The weights are further modified so that an agent has a greater preference for their own solution [43,44].
The CISAT framework has been used in previous work to effectively replicate the results of a cognitive study [5,30] and to investigate the effects of operation sequencing in engineering design . These prior studies used CISAT to simulate the design of truss structures with well-defined rules dictating how structures could be modified. The current work uses the CISAT framework in order to quickly and efficiently evaluate the performance of large numbers of different teams on several different design problems, each of which has a rule-based description similar to the prior CISAT studies.
The CISAT framework is used in this work to simulate the performance of more than 100,000 teams across 1120 different conditions that vary with respect to team characteristics, design problem, and time (number of solution evaluations) allowed for solving. Evaluating a similar number of teams without CISAT through traditional human studies would be excessively time consuming and prohibitively expensive.
Design Problem Definitions
The relationship between design problem properties and optimal team characteristics is studied using both fluid network and structural configuration design problems. These problem classes are used because their solutions are dictated by dissimilar physical phenomena, guaranteeing a broad range of problem characteristics. The problems also lend themselves well to computational design since the quality of potential solutions can be readily quantified. Within each problem class, four design problems are defined, with the intent of providing a variety of different problems within the class.
Example solutions for the structural design problems are shown in Fig. 1. The problem types, all the trusses, include two tower-style problems with both vertical and lateral loads (Figs. 1(a) and 1(b)), a single-span bridge problem (Fig. 1(c)), and a double-span bridge problem (Fig. 1(d)). Pin supports (which resist both vertical and horizontal translation) are denoted by a solid triangle, while roller supports (which only resist vertical translation) are denoted by a black triangle on top of two circles. Loads are denoted by arrows.
These structural design problems charge CISAT-simulated teams with maximizing the factor-of-safety of their solutions while minimizing the mass. Support location and type are specified for each of the problems and cannot be modified by CISAT agents. The location, magnitude, and direction of loads are similarly specified and immutable. CISAT agents are permitted to act upon solutions by adding and removing joints, adding and removing structural members, changing the size of members, and changing the location of joints (provided that the joints are neither supports nor loaded). A detailed account of these permitted modifications is given in previous work .
Example solutions to the fluid network design problems are shown in Fig. 2. The arrangement of inlets and outlets is specified as either concentric or eccentric. A concentric layout indicates that the high-pressure inlet is placed near the geometric center of the low-pressure outlets (Figs. 2(a) and 2(c)). An eccentric layout denotes that the high-pressure inlet is placed far away from the center of the low-pressure outlets (Figs. 2(b) and 2(d)). The working fluid for the network is also specified as either water (Figs. 2(a) and 2(b)) or oil (Figs. 2(c) and 2(d)). These fluids differ greatly in viscosity, impacting the structure of the design space. Laminar flow is assumed in order to afford quick closed-form evaluations of solution quality.
These fluid design problems require CISAT-simulated teams to maximize the flow rate at each of the outlets while minimizing the total length of pipe used for the solution. The location and pressure of inlets and outlets are specified as part of the problem statement and cannot be changed by CISAT agents. Agents are permitted to modify solutions by adding or removing junctions, adding or removing lengths of pipe, and changing the diameter of pipes. The rules for performing these modifications are defined similarly to the rules for the structural design problems.
Characterization of Design Problems
The nature of each design problem is quantified with respect to three properties, each of which provides information relevant to the selection of team characteristics. These properties are:
the alignment between the objective functions of the problem (), which indicates the importance of search breadth (particularly if objective functions disagree)
the local structure of the design space (), which measures local roughness of the design space and can therefore limit how efficiently an individual is able to search for local minima
the global structure of the design space (), which provides an indication of how multimodal the design space is (i.e., prevalence of local minima) and thus bears on the extent to which a team has to coordinate their search of the space
These three properties can be computed by taking a random walk of finite length through valid solutions in the design space. Random walks are produced in this work by applying a finite number of rule-based modifications to a starting solution, thus traversing a set of solutions within the design space. If we consider solutions to the design problem as nodes in a graph and arcs as the rule-based modifications between solutions, then these random walks have finite geodesic path length. The solutions traversed during the walk are evaluated with respect to each of the objective functions associated with the current design problem, with the results stored in separate vectors. This result is a set of vectors, , where is the total number of objective functions. The vector contains values along the ith objective function for the solutions traversed during the random walk. The vectors will be used below to offer mathematical definitions of the three properties.
Objective Function Alignment.
Engineering design often necessitates the consideration of multiple objectives for a given design problem . One can imagine a scenario in which a machine part is being designed with the objectives of minimizing mass while simultaneously minimizing cost. If the total cost of a part is driven by the cost of bulk material, then these objectives are aligned—they may even be related by a constant value (the per-mass cost of material). To illustrate the other extreme, imagine a scenario in which the mass of the part may only be decreased by machining away material. In this case, the cost of the part would likely be dominated by machining costs, so the objectives of minimum mass and minimum cost would be opposed. When objective functions disagree in this way, a team may need to search more divergently in order to discover a region of the design space in which objective functions are more aligned or a region in which all the objective functions reach acceptable values. Less frequent interaction enables team members to pursue their own solutions in detail, potentially allowing them to search the space divergently.
where denotes the Spearman rank correlation between objective functions i and j for the sampled solutions. Spearman's rank correlation coefficient is a nonparametric measure of the correlation between two samples . Rank correlation is used in lieu of linear correlation because an ordinal relationship is sufficient to indicate alignment between objective functions.
A value of indicates that the objective functions show some level of misalignment or opposition. As noted above, this would necessitate a higher level of divergence, which could be facilitated by infrequent interaction. A value of indicates that the objective functions show a meaningful degree of alignment. In this case, a team could benefit from frequent interaction which could enable a quicker, convergent search for the solution.
In other words, is the fractal dimension of the roughest objective function. A large value of indicates a design problem with at least one locally rough objective function, while a lower value of indicates that all the objective functions are locally smooth (and perhaps traversable with gradient methods). When a design space is locally rough, the local minima are not easy to find since gradient methods cannot be used. Therefore, infrequent interaction within a team could be beneficial, allowing individuals to intensively search different neighborhoods in the design space.
In other words, is the Hurst exponent of the most multimodal objective function. A value of near zero indicates a design problem with at least one objective function that is highly multimodal. A value of near one indicates that all the objective functions have few local optima. As a function becomes more multimodal, the team has to search broadly to find and evaluate local minima. A low level of interaction between team members could enhance breadth of search. This would implicitly encourage independent search of the design space, which in turn would delay convergence to a common solution .
Figure 3 shows an example of a random walk taken through a design space with two objective functions. Based on this random walk, the first objective function yields a Hurst exponent of and a fractal dimension of , while the second function yields a Hurst exponent of and a fractal dimension of .
Based on the above values, the global structure property can be computed as and the local structure property can be computed as . Further, the alignment property can be determined by computing the Spearman correlation coefficients between the two objective functions, .
The properties of each design problem in this work are determined by computing the mean values each of , , and obtained from 100 separate random walks. The repetition of the random walks ensures that the properties are estimated with high accuracy, thus reducing a possible source of error in the subsequent regression analysis.
Finding Optimal Team Characteristics
This section details how the optimal team characteristics are found for both case A (in which the team size is fixed, and interaction frequency must be chosen) and case B (in which the interaction frequency is fixed, and team size must be chosen). First, team performance is assessed with the CISAT modeling framework for every combination of design problem (eight problems defined in Sec. 3), team size (from 2 to 6), interaction frequency (values of 0, 1/32, 1/16, 1/8, 1/4, 1/2, and 1, indicating the fraction during which teams interact), and total number of solution evaluations allotted to the agent team (values of 500, 1000, 1500, and 2000). The variable is analogous to the number of billable hours available for a design project and provides a critical limitation on the resolution with which the space can be searched. will be referred to as resource availability in the remainder of this paper.
For every combination of the above variables, the CISAT modeling framework is used to simulate 100 design teams. A postprocessing step is used to determine the fraction of teams that were able to achieve at least one solution that met the target values for all the objective functions. This fraction is the criterion for selecting the best team characteristics. Further postprocessing (outlined in the next two paragraphs) is used to extract sets of data for the regression analyses.
Case A only requires an interaction frequency to be chosen—the team size is fixed. This allows team size to be used as a predictor variable since its values are given a priori. Multiple values of optimal interaction frequency are chosen for every combination of design problem and resource availability, one for each value of team size that was simulated (see Fig. 4(a)). The optimal interaction frequency is denoted by (optimal interaction frequency given team size). When applied across all the simulations, this procedure results in a data set of 160 samples (8 design problems × 4 values of × 5 team sizes). Every observation consists of a single-dependent variable () and five independent variables (, , , , and the given team size, ). This data set forms the basis of the regression analysis for case A.
Serving as the converse of case A, case B only requires a team size to be chosen—the interaction frequency is fixed. This allows interaction frequency to be used as a predictor variable since its values are given a priori. Case B uses the same information as case A, but multiple values of optimal team size are chosen for every combination of design problem and resource availability, one for each value of interaction frequency that was simulated (see Fig. 4(b)). The optimal team size is denoted by (optimal team size given interaction frequency). When applied across all the simulations, this procedure results in a data set of 224 samples (8 design problems × 4 values of × 7 interaction frequencies). Every observation consists of a single-dependent variable () and five independent variables (, , , , and the given interaction frequency, ). This data set forms the basis of the regression analysis for case B.
Regression analysis can be used to create equations that relate the properties of the design problems (computed in Sec. 4 for the design problems defined in Sec. 3) to the best team characteristics for solving those problems (elucidated in Sec. 5). The resulting regression equations are tools that can be used to organize a team to most efficiently solve a design problem.
Case A: Selecting Interaction Frequency With Team Size Fixed.
where is the given size of the team, which is known a priori. An equation to predict optimal interaction frequency can be found using least-squares regression. First, only main effects are included in the model. This regression model explains over 70% of the observed variance (, , and ). The contribution of a term to the accuracy of a model can be assessed by defining a new model that omits the term. Comparing the accuracy of the new model to that of the complete model indicates the contribution or added value of the omitted term. Figure 5 shows the relative contribution from each term in the model, computed in this fashion. For this model, the largest and most significant contributions come from the three problem properties that describe local structure, global structure, and objective alignment.
Next, the main effects model is elaborated by adding interaction terms to account for the interaction between variables. Adding these terms to the model increases accuracy by about 10% so that the model explains 83% of the observed variance in (, , and ). Although the accuracy increases substantially with the inclusion of interaction terms, the number of terms in the model also increases, representing an increase in complexity. The contribution from each term in this extended model is provided in Fig. 6. While many of the interaction terms explain substantial portions of the variance, the largest and most significant contributors are the interaction of local structure with objective alignment (), and global structure with objective alignment ().
Case B: Selecting Team Size With Interaction Frequency Fixed.
where is the given interaction frequency, which is now known a priori. As in case A, we first include only main effects in the model. Main effects are capable of explaining approximately 26% of the observed variance (, , and ). The contributions from each of the main effects are shown in Fig. 7. Much of the predictive power of this model is derived from knowledge of the available resources and the global structure of the design space.
Next, the main effects model is extended by adding interaction terms. This extended regression model explains approximately 55% of the observed variance in optimal team size (, , and ). Although the accuracy more than doubles with the inclusion of interaction terms, the number of terms (indicative of model complexity) also increases substantially. The contribution from each term in this extended model is shown in Fig. 8.
Adding interaction terms more than doubles the accuracy of the model, increasing the percentage of variance explained from 26% to 55%. Many of the interaction terms contribute to this boost in accuracy with the largest contributions resulting from the interaction of local structure with objective alignment () and global structure with objective alignment (). These interaction terms also contribute substantially to the interaction effects model for interaction frequency in case A (see Fig. 6). Main effects for objective alignment (), local structure (), and global structure () also contribute substantially to model accuracy.
Discussion of Regression Analysis Results
See Table 1 for a summary of the regression analyses performed in Sec. 6. Case A addresses the prediction of optimal interaction frequency for a given team size, and case B addresses the prediction of optimal team size for a given interaction frequency. For both of these cases, a model was first defined that used main effects only; next, the model was extended through the addition of interaction terms.
For case A (predicting optimal interaction frequency), both the initial model (main effects only) and the extended model (main effects plus interaction effects) achieve high accuracy ( for both models). Adding interaction effects to the model adds ten additional terms, more than doubling the total number of terms. This large increase in complexity only results in a modest 10% increase in accuracy (the main effects model is shown in Fig. 5, and the extended version with interaction effects is shown in Fig. 6). In addition, adding more terms to the model makes it more challenging to interpret and greatly increases the likelihood of overfitting. Therefore, the initial main effects model is preferred by the authors for its combination of parsimony and high accuracy.
Case B involved the development of a model for predicting optimal team size. The inclusion of interaction effects nearly doubles model accuracy (see Figs. 7 and 8), but it also greatly increases model complexity, adding ten more terms. The increase in model complexity is justified, in this case, because of the twofold increase in model accuracy. Therefore, the extended version of the model (main effects plus interaction effects) is preferred by the authors because it achieves much higher accuracy.
Predicting Optimal Team Size.
The preferred model for predicting optimal team size contains interaction effects in addition to main effects. This model explains more than half of the observed variance, indicating moderate accuracy. Figure 8 shows that the terms corresponding to the alignment of objectives (), local structure (), and the interaction between them () contribute substantially to the team size model regardless of the presence of interaction effects.
The objective alignment property is negatively related to team size, indicating that optimal team size decreases as objective functions become more aligned. For problems that contain unaligned objective functions, the larger team size may allow for search patterns that are more divergent in nature. This, in turn, could enable the team to more readily search for a portion of the design space in which objectives are aligned or in which all the objectives reach acceptable values. The local structure property is positively related to team size, which indicates that optimal team size increases as the design space becomes more locally rough. Rougher design spaces are more difficult to search deterministically, so increasing the size of a team might allow for greater search breadth. The interaction term between alignment and local structure is positively correlated with team size, meaning that small changes in these design space properties have the largest impact when the design space is aligned and smooth (that is, when is high and is low).
The availability of resources () contributes substantially to the accuracy of the model, either as a main effect or through interacting with other variables. As a main effect, availability of resources is positively related to optimal team size. In other words, when more resources are available it is beneficial to increase team size, spreading resources among a greater number of individuals to increase the extent to which work can be completed concurrently. This corresponds to the conventional approaches used in smaller software development teams [16,17,19].
This work shows that it may be possible to predict optimal team size using design problem properties, but the best models found here only explain slightly more than half of the variance in the data. One possible cause for this large residual is that the relationship between the design problem properties and optimal team size cannot be adequately captured with linear models, necessitating higher order methods. Another possibility is that one or more design problem properties that were not included in this work play a role in determining optimal team size.
Predicting Optimal Interaction Frequency.
When predicting optimal interaction frequency, the inclusion of interaction terms did not appreciably boost model accuracy. Therefore, this discussion will focus on the trends observed in the main effects. The design space properties (, , and ) are the most impactful terms, while the resource availability variable (, the number of solution evaluations allotted to the team) and (the team size) provide little predictive value.
The objective alignment property is positively related to interaction frequency. In other words, less frequent interaction is preferred for design problems in which objective functions are not aligned. If objective functions are not aligned, then infrequent interaction may allow members of the team to divergently search the design space for regions in which all the objective functions reach suitable values. The local structure property is negatively related to optimal interaction frequency, which indicates that more frequent interaction is only beneficial if the design space is locally smooth. If the design space is locally rough, infrequent interaction enables individuals to perform diligent local search before considering trade-offs between solutions, staving off premature convergence. Finally, the global structure property is positively correlated with optimal interaction frequency, indicating that multimodal design spaces require less frequent interaction. The reasoning for this relationship is much the same as that for the objective function alignment and local structure properties: infrequent interaction allows individuals to act independently as they individually find different local minima.
Considering the extreme values of the design space properties provides two illustrative examples: one in which a low interaction frequency would be predicted as optimal, and the other in which a high interaction frequency would be predicted. On the one hand, infrequent interaction would be predicted as optimal for design problems in which objective functions are unaligned, exhibit rough local structure, and are highly multimodal. This could enable individuals to spend time independently refining solutions (essentially finding a set of local minima within the team) before the team interacts to consider trade-offs between the solutions in the set. In this scenario, frequent interaction could lead to premature convergence on a poor local minimum. On the other hand, frequent interaction would be predicted as optimal for a design space in which objectives are aligned, there are few local minima, and the objective functions are locally smooth. This could enable a team of individuals to rapidly converge on a solution without spending undue time on divergent search.
Other work has studied the relationship between project complexity and task performance . In that work, project complexity was measured through ambiguity (comparable to the inverse of this work's alignment variable, ) and multiplicity (similar to the inverse of this work's global structure variable, ). That work identified nontrivial interaction between multiplicity and ambiguity, a result that is echoed in the interaction between the global structure and objective alignment variables () in this work (see Fig. 6).
There is also existent evidence for the positive correlation between objective alignment and optimal interaction frequency. Brokers are used on some creative projects to integrate diverse ideas and perspectives . These brokers may exclude certain individuals from participating in order to avoid premature convergence, in effect maintaining diverse viewpoints by decreasing the frequency of interaction . Although this example falls outside the class of design problems used in this work, it does illustrate a potential benefit from infrequent interaction when objectives are misaligned.
Generalization and Limitations.
This work investigates interaction frequency using the CISAT modeling framework, warranting a discussion of the similarities and differences of the interaction between agents in CISAT and the interaction between humans in a real team. The model for interaction between agents in CISAT entails only the direct exchange of solutions, whereas real interaction in teams is often a complex and multifaceted construct that involves more than the simple exchange of solutions. For instance, humans may coordinate their search in such a way that every member of the team investigates solutions with distinct characteristics, resulting in a pattern of divergent search. This type of coordination between agents is not enabled in CISAT, and divergent search is instead driven by the early stochastic nature of the underlying simulated annealing algorithm. Be that as it may, CISAT has been shown in previous work  to accurately reproduce the behavior of human teams solving configuration design problems even when human interaction consisted of verbal discussion in addition to the direct transmission of solutions. For special cases of interaction, it is possible that there may be a departure between the behavior exhibited by the CISAT framework and that of human teams. This includes instances when interaction is extremely frequent (such as pair programing ) or extremely rich in information (such as teams that engage in peer teaching ).
Care was taken in this work to ensure that each of the design problem properties used as independent variables could be computed before solving begins using a random walk procedure. This procedure makes two important assumptions about how the design problem is formulated. These two assumptions must be true in order to compute the design problem properties defined in this paper, and these properties are in turn prerequisite for computing optimal team characteristics. These assumptions are usually true for configuration design problems like those used in this work, but may also be true for other problem types.
The first assumption is that the quality of a solution can be quantified with one or more well-defined objective functions. It may not always be feasible to define numerical objective functions for a design problem, but it is possible to employ user surveys to robustly quantify subjective criteria like elegance or sportiness . Such rating-based data could be used to compute properties of the design problem, and from those properties, the optimal team characteristics could be estimated.
The second assumption is that existing solutions can be modified efficiently, making it possible to produce a random walk through the solution space and thus compute values for the problem space properties. This assumption was addressed here by ensuring that well-defined rules for modifying solutions were associated with each design problem, allowing the random walk procedure to be automated. It is straightforward to produce a random walk for any design problem for which such rules exist (such as problems defined by design grammars). Applying the random walk procedure to design problems that entail continuous parameters requires special attention to detail. The magnitude by which the continuous parameters are changed during the random walk procedure can have a substantial effect on the estimation of the fractal dimension and Hurst exponent. For that reason, intuitive discrete mappings should be used when possible (e.g., using discrete bolt sizes instead of addressing continuous bolt diameter).
It may be possible in some problems to estimate problem properties. For instance, it may be possible to infer an approximate value for the alignment of objectives () based on known relationships between objectives. In the example used previously, it can be readily recognized that there is a high positive alignment between objectives to minimize mass and minimize material costs. Similarly, it is usually the case that the cost of a part or product is inversely related to the strength of the product, indicating a negative value for . Such qualitative insights could be used to inform the selection of good team characteristics in the absence of a more comprehensive evaluation of the design problem. It may be possible to develop quantitative guidelines for estimating other properties as well.
Human Study: How Often Should Teams Collaborate
Practice and research generally assume that interacting teams are more effective than individuals across a variety of tasks. The superior performance of teams has been demonstrated in several studies, including computer-facilitated idea generation [58,59] and concept evaluation and selection . Some practitioners even propose that teams are always more effective than individuals when properly instructed . The theoretically optimal interaction frequency for a team can be selected using the computationally derived predictive equations presented in Secs. 6 and 7. This approach was applied to a configuration problem of modest size requiring single-domain knowledge. Surprisingly, the model predicted that interacting teams are an inferior approach; instead, the predicted optimal approach is to choose the best solution produced by a group of individuals working independently. A cognitive study of human teams was conducted in order to test this prediction. The study demonstrated that noninteracting teams performed better than frequently interacting teams for this task, a result that agrees with the prediction.
The design problem tasked participants with the design of a system of internet-connected products to maintain the temperature within a house consisting of 13 rooms. This task can be typified as a configuration problem and is thus similar in formulation to the structural and fluid configuration problems used in developing the predictive equations. However, this design problem is dictated by different physical phenomena and is derived from a different domain, thus providing a test of the generalizability of the predictive equations.
Three product types could be used to create solutions: sensors, processors, and coolers. Sensors measured the temperature of rooms in which they were placed. Coolers took external air and delivered it to the internal environment at a lower-than-ambient temperature. Processors provided a means for connecting sensors and coolers, taking temperature information from sensors, and deciding whether or not to activate coolers. Processors were only capable of receiving information from or acting on products to which they were explicitly connected. Using processors, participants were able to create numerous independent subsystems as part of the same solution. In searching for an adequate solution, participants were allowed to add, delete, and move products. They were also allowed to tune the power and flow rate of coolers.
To evaluate a solution, the distribution of temperatures within the home was simulated for an average day with external temperature varying between 20 °C and 30 °C. The mean temperature within each room of the home was solved using principles of heat and mass transfer. Two metrics (peak temperature and total cost) were computed based on the log of temperatures and product activation for the simulation. Peak temperature was defined as the highest temperature obtained in any room in the house during the simulated time period. Total cost was computed as the sum of the cost of the products making up the system and the projected 10-yr operating cost.
Characterization and Prediction.
One hundred random walks were taken for the cooling system design problem, and values for , , and were computed for each random walk. The average values for the properties were , , and . The error term in these measurements represents the standard error of the mean. Further, a team size of 3 () was selected and it was determined that 50 design actions should be allowed per individual, resulting in a total of 150 actions per team (). These values were chosen based on the results of human pilot studies with the objective of enabling all the participants to complete the study within 1 h. Substituting the above values into the main effects model for case B yields a predicted optimal interaction frequency of . Although the value is negative, it is not significantly different from 0. Therefore, cognitive study results should show a strong preference toward interaction frequencies that are near or at zero. Essentially, individuals should not be working in teams to solve this problem, but rather working independently!
This study was conducted with senior undergraduates and graduate students in mechanical engineering with ages 21–31 and a median age of 22. There were 40 male students and 14 female students and 37 senior undergraduate students and 17 graduate students. The 54 participants were partitioned into three conditions: 12 to condition 1 (interaction frequency 0.0), 21 to condition 2 (interaction frequency 0.1), and 21 to condition 3 (interaction frequency 0.2). Conditions 2 and 3 required participants to work collaboratively in teams (with minimal or greater reaction, respectively), but condition 1 required participants to work independently. Since the activities of individual participants in condition 1 were statistically independent of one another, later analysis involving the performance of condition 1 is based on the set of all the possible team combinations that could be assembled from those individuals. Considering team combinations in this way provides a better estimate of condition characteristics than simply randomly assigning individuals to teams .
To facilitate the design process, each participant was given access to a computer on which was loaded a design interface. This interface served a number of critical functions. First, the design interface allowed participants to construct and evaluate solutions and provided immediate feedback on design quality after every design modification. It also tracked every operation performed by the participants, enabling the reconstruction of each team's search for solutions following the conclusion of the study. Finally, the design interface indicated how many actions each participant had left during the study and prompted participants to interact with their team at the correct times by displaying their teammates' solutions to them. At this point, participants were permitted to discuss the solutions verbally and share them directly through the interface.
Participants first completed a guided tutorial program that introduced them to the functionality of the design interface (10 min). Following the tutorial, each participant was provided with a design statement that instructed them to design their system in order to minimize the peak temperature in the home (preferably below 24 °C) and minimize the total cost of the system (preferably below $20,000), and given 3 min to read it. Participants were then allowed to open the design interface and solve the task (30 min). During this time, every participant completed 50 total design actions. Participants were prompted to interact at regular intervals according to their condition (after every ten actions for condition 2, and after every five actions for condition 3). The experiment concluded with a postdesign survey (5 min).
A log transformation was applied to the total cost and peak temperature of each team's best solution in order to normalize the distribution. Figure 9 summarizes the mean of the log-transformed total cost of the best solutions achieved by teams in each of the conditions. Both human teams and teams simulated in CISAT are shown as well as a vertical line at the predicted optimal interaction frequency for this specific design problem. For the human data, both condition 1 (no interaction) and condition 2 (0.1 interaction frequency) achieved solutions with significantly lower cost than those of condition 3 (0.2 interaction frequency). No significant between-condition comparisons or trends were discovered with respect to the log-transformed peak temperature. This may indicate that participants were more comfortable minimizing for cost and preferred to treat peak temperature more like a constraint.
Figure 9 indicates that both human teams and CISAT teams tended to produce better solutions at lower interaction frequencies. This result is consistent with the prediction that noninteracting teams (condition 1) would achieve the best solutions to the cooling system design problem and thus offers partial validation for the computationally derived predictive equations. This is a surprising result, given the common assumption that teams are better than individuals.
Social facilitation is the tendency of individuals to perform better when in the presence of others, especially for tasks that have been practiced  and has been cited as one reason to use teams over individuals . All the participants in this study practiced the actions involved in solving the problem during the tutorial, and participants in conditions 2 and 3 accomplished the design task in the presence of their team and interacted with them both verbally and through the direct sharing of solutions. A poststudy survey also revealed that individuals who interacted more frequently were significantly more satisfied with their performance, indicating that they were affected by interaction with their team. Therefore, social facilitation should have been present in conditions 2 and 3 (boosting their performance) but not for condition 1 (in which participants worked individually). Despite this cognitive effect, condition 1 still displayed mean performance that was on par with the lowest interacting team and significantly better than the higher interacting team conditions. This underscores the influence that the properties of a design problem can exert on the effectiveness of the problem-solving approaches used by designers.
Conclusions and Future Work
This work defined and partially validated a relationship between the properties of configuration design problems and the team characteristics that lead to the best solutions to those problems. The selection of the optimal number of individuals in a team is a complicated relationship, depending greatly on the design space properties as well as the interactions between them. In addition, the availability of resources plays a large role in the selection of an optimal team size. The selection of an optimal interaction frequency can be predicted with high accuracy based on the main effects of design space properties without the need to consider interaction effects. If a design problem has unaligned objectives (e.g., satisfying one objective makes it more difficult to satisfy others), rough local structure (e.g., modifications to a solution may have volatile effects on quality), and a large number of local minima (e.g., a large number of possible “good” solutions exist), the resulting prediction will indicate that less frequent interaction is optimal. In contrast, if a design problem has aligned objectives, smooth local structure, and few local minima, then the optimal prediction will indicate more frequent interaction.
This paper focused on the selection of optimal team size and optimal interaction frequency, but the approach used here of exploring team characteristics using computational simulations of human teams could be applied to a variety of additional team characteristics as long as they are manipulable through the CISAT framework. A similar approach could also be used to develop guidelines for the selection of optimally directed parameters for computational design algorithms based on measurable problem properties.
A cognitive study was also conducted to validate the predictive equations by investigating the performance of interacting versus noninteracting teams. It was predicted that noninteracting teams would perform better than interacting teams for a cooling system configuration problem, and this prediction was born out by the results of the human study. Here, the problem was solved by a set of individuals who had similar skills, abilities, and knowledge. However, problems that require significant knowledge from different disciplines may benefit extensively from interaction between team members. The same is true for problems that can be decomposed into distinct and well-defined subtasks. The investigation of these cases is left for future study. Systematic investigation should also be undertaken to assess the range of design problem types (conceptual design, topology design, and detailed design) to which the predictive methodology and the CISAT modeling framework apply.
This work considered the selection of team characteristics that remain unchanged during solving. However, a team may uncover a portion of the design space during work that has properties that differ drastically from those measured in an initial assessment of the space. This may require some degree of adaptation on behalf of the team and may even require a different set of values for team characteristics to ensure optimal performance. Future work should consider how to monitor the values of design problem properties during solving, thus making it possible to efficiently update optimal values for team characteristics in real time.
In conclusion, this paper demonstrated that computational models could be used to create equations that enable the prediction of optimal characteristics of human teams for solving configuration design problems. Further, a cognitive study with human teams showed that these equations are an efficient and effective means of selecting optimal design team characteristics. Extensions of this work have the potential to develop a deeper and richer understanding of the search process.
This material is based upon the work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE125252 and the United States Air Force Office of Scientific Research through Grant No. FA9550-16-1-0049. An earlier version of part of this work was included in the Proceedings of the Design Theory and Methodology Conference at ASME IDETC 2016 .