Abstract

Design researchers have long sought to understand the mechanisms that support creative idea development. However, one of the key challenges faced by the design community is how to effectively measure the nebulous construct of creativity. The social science and engineering communities have adopted two vastly different approaches to solving this problem, both of which have been deployed throughout engineering design research. The goal of this paper was to compare and contrast these two approaches using design ratings of nearly 1000 engineering design ideas. The results of this study identify that while these two methods provide similar ratings of idea quality, there was a statistically significant negative relationship between these methods for ratings of idea novelty. In addition, the results show discrepancies in the reliability and consistency of global ratings of creativity. The results of this study guide the deployment of idea ratings in engineering design research and evidence.

References

1.
Liikkanen
,
L. A.
,
Hämäläinen
,
M. M.
,
Häggman
,
A.
,
Björklund
,
T.
, and
Koskinen
,
M. P.
, “
Quantitative Evaluation of the Effectiveness of Idea Generation in the Wild
,”
Proceedings of International Conference on Human Centered Design
,
Orlando, FL
,
July 9–14
, Springer, pp.
120
129
.
2.
Amabile
,
T.
,
1996
,
Creativity in Context
,
Westview Press
,
Boulder, CO
.
3.
Shah
,
J. J.
,
Smith
,
S. M.
, and
Vargas-Hernandez
,
N.
,
2003
, “
Metrics for Measuring Ideation Effectiveness
,”
Des. Studies
,
24
(
2
), pp.
111
124
. 10.1016/S0142-694X(02)00034-0
4.
Christiaans
,
H. H.
,
2002
, “
Creativity as a Design Criterion
,”
Commun. Res. J.
,
14
(
1
), pp.
41
54
.
5.
Eshun
,
E. F.
, and
de Graft-Johnson
,
K.
,
2012
, “
Learner Perceptions of Assessment of Creative Products in Communication Design
,”
Art. Des. Commun. Higher Education
,
10
(
1
), pp.
89
102
. 10.1386/adch.10.1.89_1
6.
Borgianni
,
Y.
,
Cascini
,
G.
, and
Rotini
,
F.
,
2013
, “
Assessing Creativity of Design Projects: Criteria for the Service Engineering Field
,”
Int. J. Des. Creativity Innovation
,
1
(
3
), pp.
131
159
. 10.1080/21650349.2013.806029
7.
Johnson
,
T. A.
,
Caldwell
,
B. W.
,
Cheeley
,
A.
, and
Green
,
M. G.
,
2016
, “
Comparison and Extension of Novelty Metrics for Problem-Solving Tasks
,”
Proceedings of ASME 2016 International Design Engineering Technical Conferences & Computers and Information Engineering Conference
,
Charlotte, NC
,
Aug. 21–24
, ASME, pp.
1
12
.
8.
Hernandez
,
N.
,
Okudan Kremer
,
G.
, and
Schmidt
,
L. C.
,
2012
, “
Effectiveness Metrics for Ideation: Merging Genealogy Trees and Improving Novelty Metric
,”
International Design Engineering Technical Conferences
,
Chicago, IL
,
Aug. 12–15
, pp.
85
93
.
9.
Nelson
,
B. A.
,
Wilson
,
J. O.
,
Rosen
,
D.
, and
Yen
,
J.
,
2009
, “
Refined Metrics for Measuring Ideation Effectiveness
,”
Des. Studies
,
30
(
6
), pp.
737
743
. 10.1016/j.destud.2009.07.002
10.
Peeters
,
J.
,
Verhaegen
,
P.-A.
,
Vandevenne
,
D.
, and
Duflou
,
J.
,
2010
, “
Refined Metrics for Measuring Novelty in Ideation
,”
Proceedings of IDMME Virtual Concept 2010
,
Bordeaux, France
,
Oct. 20–22
, pp 1–4.
11.
Baer
,
J.
,
2012
, “
Domain Specificity and the Limits of Creativity Theory
,”
J. Creative Behav.
,
46
(
1
), pp.
16
29
. 10.1002/jocb.002
12.
Casakin
,
H.
, and
Kreitler
,
S.
,
2005
, “
The Nature of Creativity in Design
,”
Studying Des.
,
5
, pp.
87
100
.
13.
Williams
,
A. P.
,
Ostwald
,
M. J.
, and
Askland
,
H. H.
,
2011
, “
The Relationship Between Creativity and Design and Its Implication for Design Education
,”
Des. Principles Practice: An Int. J.
,
5
(
1
), pp.
57
72
.
14.
Gosnell
,
C. A.
, and
Miller
,
S. R.
,
2014
, “
A Novel Method for Assessing Design Concept Creativity Using Single-Word Adjectives and Semantic Similarity
,”
ASME Design Engineering Technical Conferences
,
Buffalo, NY
.
15.
D'Souza
,
N.
, and
Dastmalchi
,
M. R.
,
2016
, “
Creativity on the Move: Exploring Little-c (p) and Big-C (p) Creative Events Within a Multidisciplinary Design Team Process
,”
Des. Studies
,
46
, pp.
6
37
. 10.1016/j.destud.2016.07.003
16.
Nikander
,
J. B.
,
Liikkanen
,
L. A.
, and
Laakso
,
M.
,
2014
, “
The Preference Effect in Design Concept Evaluation
,”
Des. Studies
,
35
(
5
), pp.
473
499
. 10.1016/j.destud.2014.02.006
17.
Baer
,
J.
, and
Kaufman
,
J. C.
,
2019
, “Assessing Creativity with the Consensual Assessment Technique,”
The Palgrave Handbook of Social Creativity Research
,
L.
Izabela
,
and G.
Vlad Petre
, ed.,
Springer
,
New York
, pp.
27
37
.
18.
Cseh
,
G. M.
, and
Jeffries
,
K. K.
,
2019
, “
A Scattered CAT: A Critical Evaluation of the Consensual Assessment Technique for Creativity Research
,”
Psychol. Aesthetics, Creativity, Arts
,
13
(
2
), p.
159
. 10.1037/aca0000220
19.
Stefanic
,
N.
, and
Randles
,
C.
,
2015
, “
Examining the Reliability of Scores From the Consensual Assessment Technique in the Measurement of Individual and Small Group Creativity
,”
Music Education Res.
,
17
(
3
), pp.
278
295
. 10.1080/14613808.2014.909398
20.
Alipour
,
L.
,
Faizi
,
M.
,
Moradi
,
A. M.
, and
Akrami
,
G.
,
2017
, “
The Impact of Designers’ Goals on Design-by-Analogy
,”
Des. Studies
,
51
, pp.
1
24
. 10.1016/j.destud.2017.04.001
21.
Cheng
,
P.
,
Mugge
,
R.
, and
Schoormans
,
J. P.
,
2014
, “
A New Strategy to Reduce Design Fixation: Presenting Partial Photographs to Designers
,”
Des. Studies
,
35
(
4
), pp.
374
391
. 10.1016/j.destud.2014.02.004
22.
Chan
,
J.
,
Dow
,
S. P.
, and
Schunn
,
C. D.
,
2015
, “
Do the Best Design Ideas (Really) Come From Conceptually Distant Sources of Inspiration?
,”
Des. Studies
,
36
, pp.
31
58
. 10.1016/j.destud.2014.08.001
23.
Baer
,
J.
,
2015
, “
The Importance of Domain-Specific Expertise in Creativity
,”
Roeper Rev.
,
37
(
3
), pp.
165
178
. 10.1080/02783193.2015.1047480
24.
Galati
,
F.
,
2015
, “
Complexity of Judgment: What Makes Possible the Convergence of Expert and Nonexpert Ratings in Assessing Creativity
,”
Creativity Res. J.
,
27
(
1
), pp.
24
30
. 10.1080/10400419.2015.992667
25.
Vandevenne
,
D.
,
Pieters
,
T.
, and
Duflou
,
J.
,
2016
, “
Enhancing Novelty With Knowledge-Based Support for Biologically-Inspired Design
,”
Des. Studies
,
46
, pp.
152
173
. 10.1016/j.destud.2016.05.003
26.
Atilola
,
O.
,
Tomko
,
M.
, and
Linsey
,
J. S.
,
2016
, “
The Effects of Representation on Idea Generation and Design Fixation: A Study Comparing Sketches and Function Trees
,”
Des. Studies
,
42
, pp.
110
136
. 10.1016/j.destud.2015.10.005
27.
Toh
,
C. A.
, and
Miller
,
S. R.
,
2015
, “
How Engineering Teams Select Design Concepts: A View Through the Lens of Creativity
,”
Des. Studies
,
38
, pp.
111
138
. 10.1016/j.destud.2015.03.001
28.
Tsenn
,
J.
,
Atilola
,
O.
,
McAdams
,
D. A.
, and
Linsey
,
J. S.
,
2014
, “
The Effects of Time and Incubation on Design Concept Generation
,”
Des. Studies
,
35
(
5
), pp.
500
526
. 10.1016/j.destud.2014.02.003
29.
Doboli
,
A.
, and
Umbarkar
,
A.
,
2014
, “
The Role of Precedents in Increasing Creativity During Iterative Design of Electronic Embedded Systems
,”
Des. Studies
,
35
(
3
), pp.
298
326
. 10.1016/j.destud.2014.01.001
30.
Sluis-Thiescheffer
,
W.
,
Bekker
,
T.
,
Eggen
,
B.
,
Vermeeren
,
A.
, and
De Ridder
,
H.
,
2016
, “
Measuring and Comparing Novelty for Design Solutions Generated by Young Children Through Different Design Methods
,”
Des. Studies
,
43
, pp.
48
73
. 10.1016/j.destud.2016.01.001
31.
Doboli
,
A.
,
Umbarkar
,
A.
,
Subramanian
,
V.
, and
Doboli
,
S.
,
2014
, “
Two Experimental Studies on Creative Concept Combinations in Modular Design of Electronic Embedded Systems
,”
Des. Studies
,
35
(
1
), pp.
80
109
. 10.1016/j.destud.2013.10.002
32.
Starkey
,
E.
,
Toh
,
C. A.
, and
Miller
,
S. R.
,
2016
, “
Abandoning Creativity: The Evolution of Creative Ideas in Engineering Design Course Projects
,”
Design Studies
,
47
, pp.
47
72
. 10.1016/j.destud.2016.08.003
33.
Moreno
,
D. P.
,
Hernandez
,
A. A.
,
Yang
,
M. C.
,
Otto
,
K. N.
,
Hölttä-Otto
,
K.
,
Linsey
,
J. S.
,
Wood
,
K. L.
, and
Linden
,
A.
,
2014
, “
Fundamental Studies in Design-by-Analogy: A Focus on Domain-Knowledge Experts and Applications to Transactional Design Problems
,”
Des. Studies
,
35
(
3
), pp.
232
272
. 10.1016/j.destud.2013.11.002
34.
Liu
,
W.
,
Tan
,
R.
,
Cao
,
G.
,
Zhang
,
Z.
,
Huang
,
S.
, and
Liu
,
L.
,
2019
, “
A Proposed Radicality Evaluation Method for Design Ideas at Conceptual Design Stage
,”
Comput. Ind. Eng.
,
132
, pp.
141
152
. 10.1016/j.cie.2019.04.027
35.
Christensen
,
B. T.
, and
Ball
,
L. J.
,
2016
, “
Dimensions of Creative Evaluation: Distinct Design and Reasoning Strategies for Aesthetic, Functional and Originality Judgments
,”
Des. Studies
,
45
, pp.
116
136
. 10.1016/j.destud.2015.12.005
36.
Fischer
,
S.
,
Oget
,
D.
, and
Cavallucci
,
D.
,
2016
, “
The Evaluation of Creativity From the Perspective of Subject Matter and Training in Higher Education: Issues, Constraints and Limitations
,”
Thinking Skills Creativity
,
19
, pp.
123
135
. 10.1016/j.tsc.2015.10.002
37.
Kershaw
,
T. C.
,
Bhowmick
,
S.
,
Seepersad
,
C. C.
, and
Hölttä-Otto
,
K.
,
2019
, “
A Decision Tree Based Methodology for Evaluating Creativity in Engineering Design
,”
Front. Psychol.
,
10
, p.
32
. 10.3389/fpsyg.2019.00032
38.
Goucher-Lambert
,
K.
,
Gyory
,
J. T.
,
Kotovsky
,
K.
, and
Cagan
,
J.
,
2020
, “
Adaptive Inspirational Design Stimuli: Using Design Output to Computationally Search for Stimuli That Impact Concept Generation
,”
ASME J. Mech. Des.
,
142
(
9
), p.
091401
. 10.1115/1.4046077
39.
Amabile
,
T. M.
,
1982
, “
Social Psychology of Creativity: A Consensual Assessment Technique
,”
J. Personality Social Psychol.
,
43
(
5
), pp.
997
1013
. 10.1037/0022-3514.43.5.997
40.
Amabile
,
T.
,
1983
, “
Brilliant but Cruel: Perceptions of Negative Evaluators
,”
J. Experimental Psychol.
,
19
(
2
), pp.
146
156
.
41.
Gosnell
,
C. A.
, and
Miller
,
S. R.
,
2016
, “
But Is It Creative? Delineating the Impact of Expertise and Concept Ratings on Creative Concept Selection
,”
ASME J. Mech. Des.
,
138
(
2
), p.
021101
. 10.1115/1.4031904
42.
Sarkar
,
P.
, and
Chakrabarti
,
A.
,
2011
, “
Assessing Design Creativity
,”
Des. Studies
,
32
(
4
), pp.
348
383
. 10.1016/j.destud.2011.01.002
43.
Baer
,
J.
,
Kaufman
,
J. C.
, and
Gentile
,
C. A.
,
2004
, “
Extension of the Consensual Assessment Technique to Nonparallel Creative Products
,”
Creativity Res. J.
,
16
(
1
), pp.
113
117
. 10.1207/s15326934crj1601_11
44.
Kaufman
,
J. C.
,
Baer
,
J.
,
Agars
,
M. D.
, and
Loomis
,
D.
,
2010
, “
Creativity Stereotypes and the Consensual Assessment Technique
,”
Creativity Res. J.
,
22
(
2
), pp.
200
205
. 10.1080/10400419.2010.481529
45.
Hennessey
,
B. A.
,
Amabile
,
T. M.
, and
Mueller
,
J. S.
,
1999
, “
Consensual Assessment
,”
Encyclopedia Creativity
,
1
, pp.
346
359
.
46.
Cropley
,
D. H.
,
Kaufman
,
J. C.
, and
Cropley
,
A. J.
,
2011
, “
Measuring Creativity for Innovation Management
,”
J. Technol. Manage. Innovation
,
6
(
3
), pp.
13
30
. 10.4067/S0718-27242011000300002
47.
Horn
,
D.
, and
Salvendy
,
G.
,
2009
, “
Measuring Consumer Perception of Product Creativity: Impact on Satisfaction and Purchasability
,”
Human Factors Ergonomics Manuf. Service Ind.
,
19
(
3
), pp.
223
240
. 10.1002/hfm.20150
48.
Jeffries
,
K. K.
,
2017
, “
A CAT With Caveats: Is the Consensual Assessment Technique a Reliable Measure of Graphic Design Creativity?
,”
Int. J. Des. Creativity Innovation
,
5
(
1–2
), pp.
16
28
. 10.1080/21650349.2015.1084893
49.
Bessemer
,
S. P.
,
1998
, “
Creative Product Analysis Matrix: Testing the Model Structure and a Comparison Among Products-Three Novel Chairs
,”
Creativity Res. J.
,
11
(
4
), pp.
333
346
. 10.1207/s15326934crj1104_7
50.
Kaufman
,
J. C.
, and
Baer
,
J.
,
2012
, “
Beyond New and Appropriate: Who Decides What Is Creative?
,”
Creativity Res. J.
,
24
(
1
), pp.
83
91
. 10.1080/10400419.2012.649237
51.
Kaufman
,
J. C.
,
Baer
,
J.
,
Cole
,
J. C.
, and
Sexton*
,
J. D.
,
2008
, “
A Comparison of Expert and Nonexpert Raters Using the Consensual Assessment Technique
,”
Creativity Res. J.
,
20
(
2
), pp.
171
178
. 10.1080/10400410802059929
52.
Hennessey
,
B. A.
,
1994
, “
The Consensual Assessment Technique: An Examination of the Relationship Between Ratings of Product and Process Creativity
,”
Creativity Res. J.
,
7
(
2
), pp.
193
208
. 10.1080/10400419409534524
53.
Kaufman
,
J. C.
,
Baer
,
J.
,
Cropley
,
D. H.
,
Reiter-Palmon
,
R.
, and
Sinnett
,
S.
,
2013
, “
Furious Activity vs. Understanding: How Much Expertise is Needed to Evaluate Creative Work?
,”
Psychol. Aesthetics, Creativity, Arts
,
7
(
4
), p.
332
. 10.1037/a0034809
54.
Redmond
,
M. R.
,
Mumford
,
M. D.
, and
Teach
,
R.
,
1993
, “
Putting Creativity to Work: Effects of Leader Behavior on Subordinate Creativity
,”
Org. Behav. Human Decision Processes
,
55
(
1
), pp.
120
151
. 10.1006/obhd.1993.1027
55.
Hunter
,
S. T.
,
Bedell-Avers
,
K. E.
,
Hunsicker
,
C. M.
,
Mumford
,
M. D.
, and
Ligon
,
G. S.
,
2008
, “
Applying Multiple Knowledge Structures in Creative Thought: Effects on Idea Generation and Problem-Solving
,”
Creativity Res. J.
,
20
(
2
), pp.
137
154
. 10.1080/10400410802088779
56.
Lovelace
,
J. B.
, and
Hunter
,
S. T.
,
2013
, “
Charismatic, Ideological, and Pragmatic Leaders’ Influence on Subordinate Creative Performance Across the Creative Process
,”
Creativity Res. J.
,
25
(
1
), pp.
59
74
. 10.1080/10400419.2013.752228
57.
Daly
,
S. R.
,
Seifert
,
C. M.
,
Yilmaz
,
S.
, and
Gonzalez
,
R.
,
2016
, “
Comparing Ideation Techniques for Beginning Designers
,”
ASME J. Mech. Des.
,
138
(
10
), p.
101108
. 10.1115/1.4034087
58.
Ter Harr
,
S.
,
Clausling
,
D.
, and
Eppinger
,
S.
,
1993
,
Integration of Quality Function Deployment in the Design Structure Matrix
,
Laboratory for Manufacturing and Productivity, Massachusetts Institute of Technology
,
Cambridge, MA
, Working Paper No. LMP-93-004.
59.
Pugh
,
S.
,
1991
,
Total Design
,
Addison-Wesley
,
Reading, MA
.
60.
Linsey
,
J. S.
,
Clauss
,
E. F.
,
Kurtoglu
,
T.
,
Murphy
,
J. T.
,
Wood
,
K. L.
, and
Markman
,
A. B.
,
2011
, “
An Experimental Study of Group Idea Generation Techniques: Understanding the Roles of Idea Representation and Viewing Methods
,”
ASME J. Mech. Des.
,
133
(
3
), p.
031008
. 10.1115/1.4003498
61.
Kurtoglu
,
T.
,
Campbell
,
M. I.
, and
Linsey
,
J. S.
,
2009
, “
An Experimental Study on the Effects of a Computational Design Tool on Concept Generation
,”
Des. Studies
,
30
(
6
), pp.
676
703
. 10.1016/j.destud.2009.06.005
62.
Oman
,
S. K.
,
Tumer
,
I. Y.
,
Wood
,
K.
, and
Seepersad
,
C.
,
2013
, “
A Comparison of Creativity and Innovation Metrics and Sample Validation Through In-Class Design Projects
,”
Res. Eng. Des.
,
24
(
1
), pp.
65
92
. 10.1007/s00163-012-0138-9
63.
Starkey
,
E. M.
,
Hunter
,
S. T.
, and
Miller
,
S. R.
,
2019
, “
Are Creativity and Self-Efficacy at Odds? An Exploration in Variations of Product Dissection in Engineering Education
,”
ASME J. Mech. Des.
,
141
(
1
), p.
012001
. 10.1115/1.4041173
64.
Besemer
,
S. P.
, and
O'Quin
,
K.
,
1999
, “
Confirming the Three-Factor Creative Product Analysis Matrix Model in an American Sample
,”
Creativity Res. J.
,
12
(
4
), pp.
287
296
. 10.1207/s15326934crj1204_6
65.
Toh
,
C.
, and
Miller
,
S.
,
2014
, “
The Role of Individual Risk Attitudes on the Selection of Creative Concepts in Engineering Design
,”
ASME Design Engineering Technical Conferences
,
Buffalo, NY
,
Aug. 17–20
, p. V007T07A027.
66.
Toh
,
C. A.
, and
Miller
,
S. R.
,
2014
, “
The Impact of Example Modality and Physical Interactions on Design Creativity
,”
ASME J. Mech. Des.
,
136
(
9
), p.
091004
. 10.1115/1.4027639
67.
Toh
,
C. A.
, and
Miller
,
S. R.
,
2016
, “
Choosing Creativity: The Role of Individual Risk and Ambiguity Aversion on Creative Concept Selection in Engineering Design
,”
Res. Eng. Des.
,
27
(
3
), pp.
195
219
. 10.1007/s00163-015-0212-1
68.
Cohen
,
J.
,
1988
,
The t Test for Means. Statistical Power Analysis for the Behavioural Sciences
,
Lawrence Erlbaum Associates
,
Hillsdale, NJ
.
69.
LeBreton
,
J. M.
, and
Senter
,
J. L.
,
2008
, “
Answers to 20 Questions About Interrater Reliability and Interrater Agreement
,”
Org. Res. Meth.
,
11
(
4
), pp.
815
852
. 10.1177/1094428106296642
70.
Lance
,
C. E.
,
Butts
,
M. M.
, and
Michels
,
L. C.
,
2006
, “
The Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
,”
Org. Res. Meth.
,
9
(
2
), pp.
202
220
. 10.1177/1094428105284919
71.
James
,
L. R.
,
Demaree
,
R. G.
, and
Wolf
,
G.
,
1984
, “
Estimating Within-Group Interrater Reliability With and Without Response Bias
,”
J. Appl. Psychol.
,
69
(
1
), p.
85
. 10.1037/0021-9010.69.1.85
72.
Besemer
,
S.
, and
O’Quin
,
K.
,
1986
, “
Analyzing Creative Products: Refinement and Test of a Judging Instrument
,”
J. Creative Behav.
,
20
(
2
), pp.
115
126
. 10.1002/j.2162-6057.1986.tb00426.x
73.
Case
,
B.
,
1994
, “
Walking Around the Elephant: A Critical-Thinking Strategy for Decision Making
,”
J. Continuing Education Nursing
,
25
(
3
), pp.
101
109
.
74.
Cronbach
,
L. J.
, and
Meehl
,
P. E.
,
1955
, “
Construct Validity in Psychological Tests
,”
Psychol. Bull.
,
52
(
4
), p.
281
. 10.1037/h0040957
75.
Ma
,
H.-H.
,
2009
, “
The Effect Size of Variables Associated With Creativity: A Meta-analysis
,”
Creativity Res. J.
,
21
(
1
), pp.
30
42
. 10.1080/10400410802633400
76.
Toh
,
C.
, and
Miller
,
S. R.
,
2019
, “
Does the Preferences for Creativity Scale Predict Engineering Students’ Ability to Generate and Select Creative Design Alternatives?
,”
ASME J. Mech. Des.
,
141
(
6
), p.
062001
. 10.1115/1.4042154
77.
Oldham
,
G. R.
, and
Cummings
,
A.
,
1996
, “
Employee Creativity: Personal and Contextual Factors at Work
,”
Acad. Manage. J.
,
39
(
3
), pp.
607
634
. 10.5465/256657
You do not currently have access to this content.