This paper proposes an approach to investigate associations between design concepts and design outcomes. In the proposed approach, three deliverables (concept sketches, proof-of-concept (POC) prototypes, and final products) are evaluated using metrics called creative product semantic scale (CPSS). CPSS scores are analyzed using two methods: correlation analysis and classification tree analysis. Correlation analysis is used to investigate associations between the concept sketches or the POC prototypes and the final products. Classification tree analysis (together with leave-one-out cross-validation (LOOCV) and permutation test) is used to investigate relationships between CPSS scores and the retention of design concepts (i.e., concepts that are embodied in the final products). The proposed approach is illustrated using deliverables in a project-based design class.

References

References
1.
Leary
,
W. E.
,
2002
, “
The Inquiring Minds Behind 200 Years of Inventions
,” The New York Times, Oct. 22, 2002.
2.
Brown
,
T.
,
2008
, “
Design Thinking
,”
Harv. Bus. Rev.
,
86
(
6
), pp.
84
92
.
3.
Åstebro
,
T.
,
1998
, “
Basic Statistics on the Success Rate and Profits for Independent Inventors
,”
Entrepreneurship Theory Pract.
,
23
(
2
), pp.
41
48
.
4.
Ward
,
A. C.
,
Liker
,
J. K.
,
Cristiano
,
J. J.
, and
Sobek
, II,
D. K.
,
1995
, “
The Second Toyota Paradox: How Delaying Decisions Can Make Better Cars Faster
,”
Sloan Manage. Rev.
,
36
(
3
), pp.
43
61
.
5.
Fredman
,
G.
,
2002
,
The IDEO Difference
,
Hemispheres
, Pace Communications, Greensboro, NC, pp.
52
57
.
6.
Zuber
,
C. D.
,
Alterescu
,
V.
, and
Chow
,
M. P.
,
2005
, “
Fail Often to Succeed Sooner: Adventures in Innovation
,”
Perm. J.
,
9
(
4
), pp.
44
49
.
7.
National Academy of Engineering
,
2004
,
The Engineer of 2020: Visions of Engineering in the New Century
,
National Academies
,
Washington, DC
.
8.
National Academy of Engineering
,
2005
,
Educating the Engineer of 2020: Adapting Engineering Education to the New Century
,
National Academies
,
Washington, DC
.
9.
Dym
,
C. L.
,
Agogino
,
A. M.
,
Frey
,
D. D.
,
Eris
,
O.
, and
Leifer
,
L. J.
,
2005
, “
Engineering Design Thinking, Teaching, and Learning
,”
J. Eng. Educ.
,
94
(
1
), pp.
103
120
.10.1002/j.2168-9830.2005.tb00832.x
10.
Baillie
,
C.
,
2002
, “
Enhancing Creativity in Engineering Students
,”
Eng. Sci. Educ. J.
,
11
(
5
), pp.
185
192
.10.1049/esej:20020503
11.
Liu
,
Z.
, and
Schönwetter
,
D. J.
,
2004
, “
Teaching Creativity in Engineering
,”
Int. J. Eng. Educ.
,
20
(
5
), pp.
801
808
.
12.
Amabile
,
T. M.
,
1982
, “
Social Psychology of Creativity: A Consensual Assessment Technique
,”
J. Pers. Soc. Psychol.
,
43
(
5
), pp.
997
1013
.10.1037/0022-3514.43.5.997
13.
Campbell
,
D. T.
,
1960
, “
Blind Variation and Selective Retention in Creative Though as in Other Knowledge Process
,”
Psychol. Rev.
,
67
(
6
), pp.
380
400
.10.1037/h0040373
14.
Mednick
,
S. A.
,
1962
, “
The Association Basis of the Creative Process
,”
Psychol. Rev.
,
69
(
3
), pp.
220
232
.10.1037/h0048850
15.
Rothenberg
,
A.
,
1979
,
The Emerging Goddess: The Creative Process in Art, Science, and Other Fields
,
University of Chicago
,
Chicago, IL
.
16.
Finke
,
R. A.
,
Ward
,
T. B.
, and
Smith
,
S. M.
,
1992
,
Creative Cognition: Theory, Research, and Applications
,
MIT
,
Cambridge, MA
.
17.
Finke
,
R. A.
,
1995
, “
Creative Realism
,”
Creative Cognition Approach
,
S. M.
Smith
,
T. B.
Ward
, and
R. A.
Finke
, eds.,
MIT
,
Cambridge, MA
.
18.
Ward
,
T. B.
,
Smith
,
S. M.
, and
Vaid
,
J.
,
1997
,
Creative Thought: An Investigation of Conceptual Structures and Processes
,
American Psychological Association
,
Washington, DC
.10.1037/10227-000
19.
Ward
,
T. B.
,
Smith
,
S. M.
, and
Vaid
,
J.
,
1997
, “
Conceptual Structures and Processes in Creative Thought
,”
Creative Thought: An Investigation of Conceptual Structures and Processes
,
T. B.
Ward
,
S. M.
Smith
, and
J.
Vaid
, eds.,
American Psychological Association
,
Washington, DC
.
20.
Domino
,
G.
,
1970
, “
Identification of Potentially Creative Persons From the Adjective Check List
,”
J. Consult. Clin. Psychol.
,
35
(
1
), pp.
48
51
.10.1037/h0029624
21.
Gough
,
H. A.
,
1979
, “
A Creative Personality Scale for the Adjective Check List
,”
J. Pers. Soc. Psychol.
,
37
(
8
), pp.
1398
1407
.10.1037/0022-3514.37.8.1398
22.
Strack
,
S.
, and
Lorr
,
M.
,
1990
, “
Item Factor Structure of the Personality Adjective Check List
,”
J. Pers. Assess.
,
55
(
1–2
), pp.
86
94
.10.1207/s15327752jpa5501&2_9
23.
Kaduson
,
H. G.
,
1991
, “
Concurrent Validity of the Creative Personality Scale of the Adjective Check List
,”
Psychol. Rep.
,
69
(
2
), pp.
601
602
.10.2466/pr0.1991.69.2.601
24.
Torrance
,
E. P.
,
1962
,
Guiding Creative Talent
,
Prentice-Hall
,
Englewood Cliffs, NJ
.10.1037/13134-000
25.
Jansson
,
D. G.
, and
Smith
,
S. M.
,
1991
, “
Design Fixation
,”
Des. Stud.
,
12
(
1
), pp.
3
11
.10.1016/0142-694X(91)90003-F
26.
Purcell
,
A. T.
, and
Gero
,
J. S.
,
1996
, “
Design and Other Types of Fixation
,”
Des. Stud.
,
17
(
4
), pp.
363
383
.10.1016/S0142-694X(96)00023-3
27.
Kokotovich
,
V.
, and
Purcell
,
T.
,
2000
, “
Mental Synthesis and Creativity in Design: An Experimental Examination
,”
Des. Stud.
,
21
(
5
), pp.
437
449
.10.1016/S0142-694X(00)00017-X
28.
Hey
,
J.
,
Linsey
,
J.
,
Agogino
,
A.
, and
Wood
,
K.
,
2008
, “
Analogies and Metaphors in Creative Design
,”
Int. J. Eng. Educ.
,
24
(
2
), pp.
283
294
.
29.
Linsey
,
J.
,
Wood
,
K.
, and
Markman
,
A.
,
2008
, “
Modality and Representation in Analogy
,”
Artif. Intell. Eng. Des. Anal. Manuf.
,
22
(
2
), pp.
85
100
.10.1017/S0890060408000061
30.
Tseng
,
I.
,
Moss
,
J.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2008
, “
The Role of Timing and Analogical Similarity in the Stimulation of Idea Generation in Design
,”
Des. Stud.
,
29
(
3
), pp.
203
221
.10.1016/j.destud.2008.01.003
31.
Ball
,
L. J.
, and
Christensen
,
B. T.
,
2009
, “
Analogical Reasoning and Mental Simulation in Design: Two Strategies Linked to Uncertainty Resolution
,”
Des. Stud.
,
30
(
2
), pp.
169
186
.10.1016/j.destud.2008.12.005
32.
Nagai
,
Y.
,
Taura
,
T.
, and
Mukai
,
F.
,
2009
, “
Concept Blending and Dissimilarity: Factors for Creative Concept Generation Process
,”
Des. Stud.
,
30
(
6
), pp.
648
675
.10.1016/j.destud.2009.05.004
33.
Linsey
,
J.
,
Tseng
,
I.
,
Fu
,
K.
,
Cagan
,
J.
,
Wood
,
K.
, and
Schunn
,
C.
,
2010
, “
A Study of Design Fixation, its Mitigation and Perception in Engineering Design Faculty
,”
ASME J. Mech. Des.
,
132
(
4
), p.
041003
.10.1115/1.4001110
34.
Linsey
,
J. S.
,
Clauss
,
E. F.
,
Kurtoglu
,
T.
,
Murphy
,
J. T.
,
Wood
,
K. L.
, and
Markman
,
A. B.
,
2011
, “
An Experimental Study of Group Idea Generation Techniques: Understanding the Roles of Idea Representation and Viewing Methods
,”
ASME J. Mech. Des.
,
133
(
3
), p.
031008
.10.1115/1.4003498
35.
Linsey
,
J.
,
Markman
,
A.
, and
Wood
,
K.
,
2012
, “
Design by Analogy: A Study of the WordTree Method for Problem Re-Representation
,”
ASME J. Mech. Des.
,
134
(
4
), p.
041009
.10.1115/1.4006145
36.
Schütze
,
M.
,
Sachse
,
P.
, and
Römer
,
A.
,
2003
, “
Support Value of Sketching in the Design Process
,”
Res. Eng. Des.
,
14
(
2
), pp.
89
97
.
37.
Dong
,
A.
,
2004
, “
A Document Analysis Method for Characterizing Design Team Performance
,”
ASME J. Mech. Des.
,
126
(
3
), pp.
378
385
.10.1115/1.1711818
38.
Ruder
,
J. A.
, and
Sobek
, II,
D. K.
,
2007
, “
Experiment on a System-Level Design Tool
,”
J. Eng. Des.
,
18
(
4
), pp.
327
342
.10.1080/09544820601008928
39.
Song
,
S.
, and
Agogino
,
A. M.
,
2004
, “
Insights on Designers' Sketching Activities in New Product Design Teams
,”
ASME
Paper No. DETC2004-57474.10.1115/DETC2004-57474
40.
Yang
,
M. C.
,
2009
, “
Observations on Concept Generation and Sketching in Engineering Design
,”
Res. Eng. Des.
,
20
(
1
), pp.
1
11
.10.1007/s00163-008-0055-0
41.
Wilde
,
D. J.
,
2009
,
Teamology: The Construction and Organization of Effective Teams
,
Springer
,
NY
.
42.
Shah
,
J. J.
,
Kulkarni
,
S. V.
, and
Vargas-Hernandez
,
N.
,
2000
, “
Evaluation of Idea Generation Methods for Conceptual Design: Effectiveness Metrics and Design of Experiments
,”
ASME J. Mech. Des.
,
122
(
4
), pp.
377
384
.10.1115/1.1315592
43.
Shah
,
J. J.
,
Vargas-Hernandez
,
N.
, and
Smith
,
S. M.
,
2003
, “
Metrics for Measuring Ideation Effectiveness
,”
Des. Stud.
,
24
(
2
), pp.
111
134
.10.1016/S0142-694X(02)00034-0
44.
Chusilp
,
P.
, and
Jin
,
Y.
,
2006
, “
Impact of Mental Iteration on Concept Generation
,”
ASME J. Mech. Des.
,
128
(
1
), pp.
14
25
.10.1115/1.2118707
45.
O'Quin
,
K.
, and
Besemer
,
S. P.
,
1989
, “
The Development, Reliability, and Validity of the Revised Creative Product Semantic Scale
,”
Creativity Res. J.
,
2
(
4
), pp.
267
278
.10.1080/10400418909534323
46.
Besemer
,
S. P.
, and
Treffinger
,
D. J.
,
1981
, “
Analysis of Creative Products: Review and Synthesis
,”
J. Creative Behav.
,
15
(
3
), pp.
158
178
.10.1002/j.2162-6057.1981.tb00287.x
47.
Besemer
,
S. P.
, and
O'Quin
,
K.
,
1986
, “
Analyzing Creative Products: Refinement and Test of a Judging Instrument
,”
J. Creative Behav.
,
20
(
2
), pp.
115
126
.10.1002/j.2162-6057.1986.tb00426.x
48.
Besemer
,
S. P.
, and
O'Quin
,
K.
,
1987
, “
Creative Product Analysis: Testing a Model by Developing a Judging Instrument
,”
Frontiers of Creativity Research
,
S. G.
Isaksen
, ed.,
Bearly Limited
,
Buffalo, NY
, pp.
341
357
.
49.
Dahl
,
D. W.
,
Chattopadhyay
,
A.
, and
Gorn
,
G. J.
,
2001
, “
The Importance of Visualisation in Concept Design
,”
Des. Stud.
,
22
(
1
), pp.
5
26
.10.1016/S0142-694X(99)00028-9
50.
Kim
,
M. H.
,
Kim
,
Y. S.
,
Lee
,
H. S.
, and
Park
,
J. A.
,
2007
, “
An Underlying Cognitive Aspect of Design Creativity: Limited Commitment Mode Control Strategy
,”
Des. Stud.
,
28
(
6
), pp.
585
604
.10.1016/j.destud.2007.04.006
51.
Takai
,
S.
,
2011
, “
A Pre-Post Assessment of Creativity Methods in an Experimental Course
,”
Int. J. Eng. Educ.
,
27
(
2
), pp.
380
388
.
52.
White
,
A.
, and
Smith
,
B. L.
,
2001
, “
Assessing Advertising Creativity Using the Creative Product Semantic Scale
,”
J. Advertising Res.
,
41
(
6
), pp.
27
34
.
53.
White
,
A.
,
Shen
,
F.
, and
Smith
,
B. L.
,
2002
, “
Judging Advertising Creativity Using the Creative Product Semantic Scale
,”
J. Creative Behav.
,
36
(
4
), pp.
241
253
.10.1002/j.2162-6057.2002.tb01067.x
54.
Smith
,
B. L.
,
1993
, “
Interpersonal Behaviors that Damage the Productivity of Creative Problem Solving Groups
,”
J. Creative Behav.
,
27
(
3
), pp.
171
187
.10.1002/j.2162-6057.1993.tb00705.x
55.
Bytheway
,
C. W.
,
2007
,
FAST Creativity & Innovation: Rapidly Improving Processes, Product Development and Solving Complex Problems
,
J. Ross Pub.
,
Fort Lauderdale, FL
.
56.
Olivero
,
M.
,
1990
, “
Get Crazy! How to Have a Breakthrough Idea
,”
Working Woman
,
213
(
1
), pp.
145
147
, 222.
57.
Osborn
,
A. F.
,
1963
,
Applied Imagination: Principles and Procedures of Creative Problem-Solving
,
Scribner
,
NY
.
58.
Pahl
,
G.
, and
Beitz
,
W.
,
1996
,
Engineering Design—A Systematic Approach
,
Springer Limited
,
London, UK
.
59.
Ojala
,
M.
, and
Garriga
,
G. C.
,
2010
, “
Permutation Tests for Studying Classifier Performance
,”
J. Mach. Learn. Res.
,
11
(
6
), pp.
1833
1863
.
60.
Shrout
,
P. E.
, and
Fleiss
,
J. L.
,
1979
, “
Intraclass Correlations: Uses in Assessing Rater Reliability
,”
Psychol. Bull.
,
86
(
2
), pp.
420
428
.10.1037/0033-2909.86.2.420
61.
Tseng
,
I.
,
Cagan
,
J.
,
Kotovsky
,
K.
, and
Wood
,
W.
,
2012
, “
Form Function Fidelity
,”
ASME J. Mech. Des.
,
135
(
1
), p.
011006
.10.1115/1.4007933
62.
Hallgren
,
K. A.
,
2012
, “
Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial
,”
Tutorials Quant. Methods Psychol.
,
8
(
1
), pp.
23
34
.
63.
Cicchetti
,
D. V.
,
1994
, “
Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology
,”
Psychol. Assess.
,
6
(
4
), pp.
284
290
.10.1037/1040-3590.6.4.284
64.
Multon
,
K.
,
2010
, “
Interrater Reliability
,”
Encyclopedia of Research Design
,
N.
Salkind
, ed.,
SAGE Publications, Inc.
,
Thousand Oaks, CA
, pp.
627
629
.
65.
Takai
,
S.
, and
Ishii
,
K.
,
2010
, “
A Use of Subjective Clustering to Support Affinity Diagram Results in Customer Needs Analysis
,”
Concurrent Eng.: Res. Appl.
,
18
(
2
), pp.
101
109
.10.1177/1063293X10372792
You do not currently have access to this content.