Design reviews are typically used for three types of design activities: (1) identifying errors, (2) assessing the impact of the errors, and (3) suggesting solutions for the errors. This experimental study focuses on understanding the second issue as it relates to the number of errors considered, the existence of controls, and the level of domain familiarity of the assessor. A set of design failures and associated controls developed for a completed industry sponsored project is used as the experimental design problem. Nondomain generalists (students from an undergraduate psychology class), domain generalists (first year engineering students), and domain specialists (graduate mechanical engineering students) are provided a set of failure modes and asked to provide their own opinion or confidence on whether the system would still successfully achieve the stated objectives. The confidence level for all domain populations decreased significantly as the number of design errors increased (largest p-value = 0.0793), and this decrease in confidence is more significant as the number of design errors increases. The impact on confidence is lower when solutions (controls) are provided to prevent the errors (largest p-value = 0.0334) as the confidence decreased faster for domain general engineers as compared to domain specialists (p = < 0.0001). The domain specialists showed higher confidence in making decisions than domain generalists and nondomain generalists as the design errors increase.

References

References
1.
Schmidt
,
R.
,
2010
, “
Risk Assessment in Research Projects
,”
Annual Reliability and Maintainability Symposium (RAMS)
, IEEE, San Jose, CA, pp.
1
6
.
2.
Clarkson
,
P. J.
,
Simons
,
C.
, and
Eckert
,
C.
,
2004
, “
Predicting Change Propagation in Complex Design
,”
ASME J. Mech. Des.
,
126
(
5
), pp.
788
797
.
3.
Shankar
,
P.
,
Morkos
,
B. W.
, and
Summers
,
J. D.
,
2012
, “
Reasons for Change Propagation: A Case Study in an Automotive OEM
,”
Res. Eng. Des.
,
23
(
4
), pp.
291
303
.
4.
Shankar
,
P.
,
Summers
,
J.
, and
Phelan
,
K.
,
2014
, “
A Verification and Validation Planning Method to Address Propagation Effects in Engineering Design
,”
International Symposium on Tools and Methods of Competitive Engineering
, Budapest, Hungary, pp. 635–648.
5.
Pfeiffer
,
T.
,
2002
,
Quality Management: Strategies, Methods, Techniques
,
Hanser Fachbuchverlag
,
Munich, Germany
.
6.
Hawkins
,
P. G.
, and
Woollons
,
D. J.
,
1998
, “
Failure Modes and Effects Analysis of Complex Engineering Systems Using Functional Models
,”
Artif. Intell. Eng.
,
12
(
4
), pp.
375
397
.
7.
Cotnareanu
,
T.
,
1999
, “
Old Tools–New Uses: Equipment FMEA—A Tool for Preventive Maintenance
,”
Qual. Prog.
,
32
(
12
), pp.
48
52
.
8.
Haik
,
Y.
, and
Shahin
,
T.
,
2010
,
Engineering Design Process
,
Cengage Learning
, Stamford, CT.
9.
Dieter
,
G. E.
, and
Schmidt
,
L. C.
,
2009
,
Engineering Design
,
McGraw-Hill Higher Education
, New York.
10.
Shimizu
,
H.
,
Imagawa
,
T.
, and
Noguchi
,
H.
,
2003
, “
Reliability Problem Prevention Method for Automotive Components—Development of GD3 Activity and DRBFM (Design Review Based on Failure Mode)
,” SAE Technical Paper No. 2003-01-2877.
11.
Shimizu
,
H.
,
Otsuka
,
Y.
, and
Noguchi
,
H.
,
2010
, “
Design Review Based on Failure Mode to Visualise Reliability Problems in the Development Stage of Mechanical Products
,”
Int. J. Veh. Des.
,
53
(
3
), pp.
149
165
.
12.
Schmidt
,
R.
,
Riedel
,
G. J.
, and
Kangas
,
K.
,
2011
, “
Risk Assessment Using Design Review Based on Failure Mode
,”
Annual Reliability and Maintainability Symposium
(
RAMS
), IEEE, New York, Jan. 24–27.
13.
Hannah
,
R.
,
Joshi
,
S.
, and
Summers
,
J. D.
,
2011
, “
A User Study of Interpretability of Engineering Design Representations
,”
J. Eng. Des.
,
23
(
6
), pp.
443
468
.
14.
Sulistyawati
,
K.
,
Wickens
,
C. D.
, and
Chui
,
Y. P.
,
2011
, “
Prediction in Situation Awareness: Confidence Bias and Underlying Cognitive Abilities
,”
Int. J. Aviat. Psychol.
,
21
(
2
), pp.
153
174
.
15.
Lin
,
S. W.
, and
Bier
, V
. M.
,
2008
, “
A Study of Expert Overconfidence
,”
Reliab. Eng. Syst. Saf.
,
93
(
2008
), pp.
711
721
.
16.
Bier
,
V.
,
2004
, “
Implications of the Research on Expert Overconfidence and Dependence
,”
Reliab. Eng. Syst. Saf.
,
85
, pp.
321
329
.
17.
Pahl
,
G.
,
Beitz
,
W.
,
Wallace
,
K.
, and
Blessing
,
L.
,
2007
,
Engineering Design: A Systematic Approach
,
Springer-Verlag
,
London
.
18.
Ostergaard
,
K. J.
,
Wetmore
,
W. R.
, III
,
Divekar
,
A.
,
Vitali
,
H.
, and
Summers
,
J. D.
,
2005
, “
An Experimental Methodology for Investigating Communication in Collaborative Design Review Meetings
,”
CoDesign
,
1
(
3
), pp.
169
185
.
19.
Sater-Black
,
K.
, and
Iverson
,
N.
,
1994
, “
How to Conduct a Design Review
,”
Mech. Eng.
,
116
(
3
), pp.
89
93
.
20.
Wetmore
,
W. R.
, and
Summers
,
J. D.
,
2003
, “
Group Decision Making: Friend or Foe?
,” Engineering Management Conference,
IEMC’03
,
Managing Technologically Driven Organizations: The Human Side of Innovation and Change
, Nov. 2–4, pp.
405
409
.
21.
Wetmore
,
W. R.
, III
,
Summers
,
J. D.
, and
Greenstein
,
J. S.
,
2010
, “
Experimental Study of Influence of Group Familiarity and Information Sharing on Design Review Effectiveness
,”
J. Eng. Des.
,
21
(
1
), pp.
111
126
.
22.
Collins
,
S. T.
,
Yassine
,
A. A.
, and
Borgatti
,
S. P.
,
2009
, “
Evaluating Product Development Systems Using Network Analysis
,”
Syst. Eng.
,
12
(
1
), pp.
55
68
.
23.
Hammond
,
J.
,
Lafayette
,
W.
,
Koubek
,
R. J.
, and
Harvey
,
C. M.
,
2001
, “
Distributed Collaboration for Engineering Design: A Review and Reappraisal
,”
Hum. Factors Ergon. Manuf.
,
11
(
1
), pp.
35
52
.
24.
Pugh
,
S.
,
1991
,
Total Design: Integrated Methods for Successful Product Engineering
,
Addison-Wesley Publishing Company
,
Wokingham, UK
.
25.
Hilts
,
S.
,
Johnson
,
K.
, and
Turoff
,
M.
,
1986
, “
Experiments in Group Decision-Making: Communication Process and Outcome in Face-to-Face Versus Computerized Conferences
,”
Hum. Commun. Res.
,
13
(
2
), pp.
225
252
.
26.
DoD
,
1980
, “
Procedures for Performing a Failure Mode, Effects and Criticality Analysis
,” U.S. Department of Defense, Washington, DC, Standard No. MIL-STD-1629A.
27.
Teoh
,
P. C.
, and
Case
,
K.
,
2005
, “
An Evaluation of Failure Modes and Effects Analysis Generation Method for Conceptual Design
,”
Int. J. Comput. Integr. Manuf.
,
18
(
4
), pp.
279
293
.
28.
Xu
,
K.
,
Tang
,
L. C.
,
Xie
,
M.
,
Ho
,
S. L.
, and
Zhu
,
M. L.
,
2002
, “
Fuzzy Assessment of FMEA for Engine Systems
,”
Reliab. Eng. Syst. Saf.
,
75
(
1
), pp.
17
29
.
29.
Dinsmore
,
D. L.
, and
Parkinson
,
M. M.
,
2012
, “
What are Confidence Judgments Made of? Students' Explanations for Their Confidence Ratings and What That Means for Calibration
,”
Learn. Instr.
,
24
, pp.
1
11
.
30.
Thorndike
,
E. L.
, and
Gates
,
A. I.
,
1929
,
Elementary Principles of Education
,
Macmillan Company
,
New York
.
31.
Pedhazur
,
E. J.
,
1997
,
Multiple Regression in Behavioral Research
,
Harcourt Brace
,
Orlando, FL
.
32.
Finney
,
S. J.
, and
DiStefano
,
C.
,
2006
,
Non-Normal and Categorical Data in Structural Equation Modeling
,
Information Age Publishing
,
Greenwich, CT
.
33.
Schraw
,
G.
,
Potenza
,
M. T.
, and
Nebelsick-Gullet
,
L.
,
1993
, “
Constraints on the Calibration of Performance
,”
Contemp. Educ. Psychol.
,
18
(
4
), pp.
455
463
.
34.
Bernstein
,
E. M.
, and
Putnam
,
F. W.
,
1986
, “
Development, Reliability, and Validity of a Dissociation Scale
,”
J. Nerv. Ment. Dis.
,
174
(
12
), pp.
727
735
.
35.
Aitken
,
R. C.
,
1969
, “
Measurement of Feelings Using Visual Analogue Scales
,”
Proc. R. Soc. Med.
,
62
(
10
), pp.
989
–993.
36.
Miller
,
G. A.
,
1956
, “
The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information
,”
Psychol. Rev.
,
63
(
2
), pp.
81
–97.
37.
Cowan
,
N.
,
2001
, “
The Magical Number 4 in Short-Term Memory: A Reconsideration of Mental Storage Capacity
,”
Behav. Brain Sci.
,
24
(
1
), pp.
87
114
.
38.
Farrington
,
J.
,
2011
, “
Seven Plus or Minus Two
,”
Perform. Improv. Q.
,
23
(
4
), pp.
113
116
.
39.
Thimmaiah
,
S.
,
2013
, “
An Experimental Study on the Influence That Failure Number, Specialization, and Domain Have on Confidence in Predicting System Failures
,”
M.S. thesis
, Clemson University, Clemson, SC.
You do not currently have access to this content.