Assessing similarity between design ideas is an inherent part of many design evaluations to measure novelty. In such evaluation tasks, humans excel at making mental connections among diverse knowledge sets to score ideas on their uniqueness. However, their decisions about novelty are often subjective and difficult to explain. In this paper, we demonstrate a way to uncover human judgment of design idea similarity using two-dimensional (2D) idea maps. We derive these maps by asking participants for simple similarity comparisons of the form “Is idea A more similar to idea B or to idea C?” We show that these maps give insight into the relationships between ideas and help understand the design domain. We also propose that novel ideas can be identified by finding outliers on these idea maps. To demonstrate our method, we conduct experimental evaluations on two datasets—colored polygons (known answer) and milk frother sketches (unknown answer). We show that idea maps shed light on factors considered by participants in judging idea similarity and the maps are robust to noisy ratings. We also compare physical maps made by participants on a white-board to their computationally generated idea maps to compare how people think about spatial arrangement of design items. This method provides a new direction of research into deriving ground truth novelty metrics by combining human judgments and computational methods.

References

References
1.
Starkey
,
E.
,
Toh
,
C. A.
, and
Miller
,
S. R.
,
2016
, “
Abandoning Creativity: The Evolution of Creative Ideas in Engineering Design Course Projects
,”
Des. Stud.
,
47
, pp.
47
72
.
2.
Hammedi
,
W.
,
van Riel
,
A. C.
, and
Sasovova
,
Z.
,
2011
, “
Antecedents and Consequences of Reflexivity in New Product Idea Screening
,”
J. Prod. Innovation Manage.
,
28
(
5
), pp.
662
679
.
3.
Lopez-Mesa
,
B.
, and
Vidal
,
R.
,
2006
, “
Novelty Metrics in Engineering Design Experiments
,”
Ninth International Design Conference (DESIGN)
, Dubrovnik, Croatia, May 15–18, pp. 557–564.
4.
Sarkar
,
P.
, and
Chakrabarti
,
A.
,
2011
, “
Assessing Design Creativity
,”
Des. Stud.
,
32
(
4
), pp.
348
383
.
5.
Johnson
,
T. A.
,
Cheeley
,
A.
,
Caldwell
,
B. W.
, and
Green
,
M. G.
,
2016
, “
Comparison and Extension of Novelty Metrics for Problem-Solving Tasks
,”
ASME
Paper No. DETC2016-60319.
6.
Simonton
,
D. K.
,
2012
, “
Taking the U.S. Patent Office Criteria Seriously: A Quantitative Three-Criterion Creativity Definition and Its Implications
,”
Creativity Res. J.
,
24
(
2–3
), pp.
97
106
.
7.
Maher
,
M. L.
, and
Fisher
,
D. H.
,
2012
, “
Using AI to Evaluate Creative Designs
,”
Second International Conference on Design Creativity
, Glasgow, UK, Sept. 18–20, pp. 45–54.https://www.designsociety.org/download-publication/32461/Using+AI+to+Evaluate+Creative+Designs
8.
Verhaegen
,
P.-A.
,
Vandevenne
,
D.
, and
Duflou
,
J.
,
2012
, “
Originality and Novelty: A Different Universe
,”
12th International Design Conference
, Dubrovnik, Croatia, May 21–24, pp. 1961–1966.
9.
Seymore
,
S. B.
,
2011
, “
Rethinking Novelty in Patent Law
,”
Duke Law J.
,
60
, pp.
919
976
.https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1558617
10.
Chen
,
L.
,
Xu
,
P.
, and
Liu
,
D.
,
2015
, “
Experts Versus the Crowd: A Comparison of Selection Mechanisms in Crowdsourcing Contests
,” SSRN Electronic J. (epub).
11.
Chen
,
L.
, and
Liu
,
D.
,
2012
, “
Comparing Strategies Winning Expert-Rated Crowd-Rated Crowdsourcing Contests: First Findings
,” 18th Americas Conference on Information Systems (AMCIS), Seattle, WA, Aug. 9–12, pp.
97
107
.
12.
Green
,
M.
,
Seepersad
,
C. C.
, and
Hölttä-Otto
,
K.
,
2014
, “
Crowd-Sourcing the Evaluation of Creativity in Conceptual Design: A Pilot Study
,”
ASME
Paper No. .
13.
Surowiecki
,
J.
,
2004
,
The wisdom of Crowds: Why the Many are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations
, Vol.
296
, American Psychological Association, Washington, DC.
14.
Yu
,
B.
,
Willis
,
M.
,
Sun
,
P.
, and
Wang
,
J.
,
2013
, “
Crowdsourcing Participatory Evaluation of Medical Pictograms Using Amazon Mechanical Turk
,”
J. Med. Internet Res.
,
15
(
6
), p. e108.https://www.jmir.org/2013/6/e108/
15.
Wu
,
H.
,
Corney
,
J.
, and
Grant
,
M.
,
2015
, “
An Evaluation Methodology for Crowdsourced Design
,”
Adv. Eng. Inf.
,
29
(
4
), pp.
775
786
.
16.
Görzen
,
T.
, and
Kundisch
,
D.
,
2016
, “
Can the Crowd Substitute Experts in Evaluating Creative Jobs? the Case of Business Models
,” European Conference on Information Systems, Rome, Italy, Sept. 4–9.
17.
Hennessey
,
B. A.
, and
Amabile
,
T. M.
,
1999
, “
Consensual Assessment
,”
Encycl. Creativity
,
1
, pp.
347
359
.
18.
Licuanan
,
B. F.
,
Dailey
,
L. R.
, and
Mumford
,
M. D.
,
2007
, “
Idea Evaluation: Error in Evaluating Highly Original Ideas
,”
J. Creative Behav.
,
41
(
1
), pp.
1
27
.
19.
Shah
,
J. J.
,
Kulkarni
,
S. V.
, and
Vargas-Hernandez
,
N.
,
2000
, “
Evaluation of Idea Generation Methods for Conceptual Design: Effectiveness Metrics and Design of Experiments
,”
ASME J. Mech. Des.
,
122
(
4
), pp.
377
384
.
20.
Verhaegen
,
P.-A.
,
Vandevenne
,
D.
,
Peeters
,
J.
, and
Duflou
,
J. R.
,
2013
, “
Refinements to the Variety Metric for Idea Evaluation
,”
Des. Stud.
,
34
(
2
), pp.
243
263
.
21.
Shah
,
J. J.
,
Smith
,
S. M.
, and
Vargas-Hernandez
,
N.
,
2003
, “
Metrics for Measuring Ideation Effectiveness
,”
Des. Stud.
,
24
(
2
), pp.
111
134
.
22.
Oman
,
S. K.
,
Tumer
,
I. Y.
,
Wood
,
K.
, and
Seepersad
,
C.
,
2013
, “
A Comparison of Creativity and Innovation Metrics and Sample Validation Through In-Class Design Projects
,”
Res. Eng. Des.
,
24
(
1
), pp.
65
92
.
23.
Brown
,
D. C.
,
2014
, “
Problems With the Calculation of Novelty Metrics
,”
Sixth International Conference on Design Computing and Cognition
(
DCC'14
), London, June 23–25.https://pdfs.semanticscholar.org/449d/1bbdadd385aa6055b3f1ef1c10804169b53a.pdf
24.
Baer
,
J.
,
2012
, “
Domain Specificity and the Limits of Creativity Theory
,”
J. Creative Behav.
,
46
(
1
), pp.
16
29
.
25.
Casakin
,
H.
, and
Kreitler
,
S.
,
2005
, “
The Nature of Creativity in Design
,”
Studying Designers
,
5
, pp.
87
100
.
26.
Richardson
,
T.
,
Nekolny
,
B.
,
Holub
,
J.
, and
Winer
,
E. H.
,
2014
, “
Visualizing Design Spaces Using Two-Dimensional Contextual Self-Organizing Maps
,”
AIAA J.
,
52
(
4
), pp.
725
738
.
27.
Tang
,
J.
,
Liu
,
J.
,
Zhang
,
M.
, and
Mei
,
Q.
,
2016
, “
Visualizing Large-Scale and High-Dimensional Data
,”
25th International Conference on World Wide Web, International World Wide Web Conferences Steering Committee
, Montreal, QC, Canada, Apr. 11–15, pp.
287
297
.
28.
Maaten
,
L. V. D.
, and
Hinton
,
G.
,
2008
, “
Visualizing Data Using t-Sne
,”
J. Mach. Learn. Res.
,
9
, pp.
2579
2605
.http://www.jmlr.org/papers/v9/vandermaaten08a.html
29.
Chen
,
W.
,
Fuge
,
M.
, and
Chazan
,
N.
,
2017
, “
Design Manifolds Capture the Intrinsic Complexity and Dimension of Design Spaces
,”
ASME J. Mech. Des.
,
139
(
5
), p.
051102
.
30.
Li
,
L.
,
Malave
,
V.
,
Song
,
A.
, and
Yu
,
A. J.
,
2016
, “
Extracting Human Face Similarity Judgments: Pairs or Triplets?
,”
J. Vision
,
16
(
12
), pp.
719
719
.
31.
Torgerson
,
W. S.
,
1958
,
Theory and Methods of Scaling
, Wiley, Hoboken, NJ.
32.
van der Maaten
,
L.
, and
Weinberger
,
K.
,
2012
, “
Stochastic Triplet Embedding
,”
IEEE
International Workshop on Machine Learning for Signal Processing
, Santander, Spain, Sept. 23–26, pp.
1
6
.
33.
Stewart
,
N.
,
Brown
,
G. D.
, and
Chater
,
N.
,
2005
, “
Absolute Identification by Relative Judgment
,”
Psychol. Rev.
,
112
(
4
), p.
881
.
34.
Agarwal
,
S.
,
Wills
,
J.
,
Cayton
,
L.
,
Lanckriet
,
G.
,
Kriegman
,
D.
, and
Belongie
,
S.
,
2007
, “
Generalized Non-Metric Multidimensional Scaling
,” PMLR, 2, pp.
11
18
.
35.
Tamuz
,
O.
,
Liu
,
C.
,
Belongie
,
S.
,
Shamir
,
O.
, and
Kalai
,
A. T.
,
2011
, “
Adaptively Learning the Crowd Kernel
,”
28th International Conference on Machine Learning
(
ICML'11
), June 28–July 2, Bellevue, WA, pp.
673
680
.https://people.csail.mit.edu/celiu/pdfs/2011_ICML_TamLiuBelShamKal.pdf
36.
Sankaranarayanan
,
S.
,
Alavi
,
A.
, and
Chellappa
,
R.
,
2016
, “
Triplet Similarity Embedding for Face Verification
,” Preprint
arXiv: 1602.03418.
https://arxiv.org/abs/1602.03418
37.
Nhat
,
V. D. M.
,
Vo
,
D.
,
Challa
,
S.
, and
Lee
,
S.
,
2008
, “
Nonmetric MDS for Sensor Localization
,”
Third International Symposium on Wireless Pervasive Computing
(
ISWPC
), Santorini, Greece, May 7–9, pp.
396
400
.
38.
Haghiri
,
S.
,
Ghoshdastidar
,
D.
, and
von Luxburg
,
U.
,
2017
, “
Comparison-Based Nearest Neighbor Search
,”
PMLR
,
54
, pp. 851–859.http://proceedings.mlr.press/v54/haghiri17a/haghiri17a.pdf
39.
Ukkonen
,
A.
,
Derakhshan
,
B.
, and
Heikinheimo
,
H.
,
2015
. “
Crowdsourced Nonparametric Density Estimation Using Relative Distances
,” Third AAAI Conference on Human Computation and Crowdsourcing (
HCOMP
), San Diego, CA, Nov. 8–11, pp. 188–197.https://www.aaai.org/ocs/index.php/HCOMP/HCOMP15/paper/viewFile/11586/11441
40.
Demiralp
,
Ç.
,
Bernstein
,
M. S.
, and
Heer
,
J.
,
2014
, “
Learning Perceptual Kernels for Visualization Design
,”
IEEE Trans. Visualization Comput. Graphics
,
20
(
12
), pp.
1933
1942
.
41.
Siangliulue
,
P.
,
Arnold
,
K. C.
,
Gajos
,
K. Z.
, and
Dow
,
S. P.
,
2015
, “
Toward Collaborative Ideation at Scale: Leveraging Ideas From Others to Generate More Creative and Diverse Ideas
,”
18th ACM Conference on Computer Supported Cooperative Work & Social Computing
(
CSCW
), Vancouver, BC, Canada, Mar. 14–18, pp.
937
945
.https://dl.acm.org/citation.cfm?id=2675239
42.
Kawakita
,
J.
,
1991
, “
The Original KJ Method
,”
Tokyo: Kawakita Res. Inst.
, pp. 233–237.
43.
Lin
,
H.
, and
Bilmes
,
J.
,
2011
, “
A Class of Submodular Functions for Document Summarization
,”
49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies
, Portland, OR, June 19–24, pp.
510
520
.
44.
Gower
,
J. C.
,
1975
, “
Generalized Procrustes Analysis
,”
Psychometrika
,
40
(
1
), pp.
33
51
.
45.
Toh
,
C. A.
, and
Miller
,
S. R.
,
2016
, “
Choosing Creativity: The Role of Individual Risk and Ambiguity Aversion on Creative Concept Selection in Engineering Design
,”
Res. Eng. Des.
,
27
(
3
), pp.
195
219
.
46.
Amid
,
E.
,
Vlassis
,
N.
, and
Warmuth
,
M. K.
,
2016
, “
Low-Dimensional Data Embedding Via Robust Ranking
,” Preprint
arXiv: 1611.09957
.https://arxiv.org/abs/1611.09957
47.
Hoffmann
,
H.
,
2007
, “
Kernel PCA for Novelty Detection
,”
Pattern Recognit.
,
40
(
3
), pp.
863
874
.
48.
Ahmed
,
F.
, and
Fuge
,
M.
,
2018
, “
Ranking Ideas for Diversity and Quality
,”
ASME J. Mech. Des.
,
140
(
1
), p.
011101
.
49.
Ahmed
,
F.
,
Fuge
,
M.
, and
Gorbunov
,
L. D.
,
2016
, “
Discovering Diverse, High Quality Design Ideas From a Large Corpus
,”
ASME
Paper No. DETC2016-59926.
50.
Wilber
,
M.
,
Kwak
,
I. S.
,
Kriegman
,
D.
, and
Belongie
,
S.
,
2015
, “
Learning Concept Embeddings With Combined Human-Machine Expertise
,”
International Conference on Computer Vision
(
ICCV
), Santiago, Chile, Dec. 7–13, pp. 981–989.https://vision.cornell.edu/se3/wp-content/uploads/2015/09/main.pdf
51.
Elsbach
,
K. D.
, and
Kramer
,
R. M.
,
2003
, “
Assessing Creativity in Hollywood Pitch Meetings: Evidence for a Dual-Process Model of Creativity Judgments
,”
Acad. Manage. J.
,
46
(
3
), pp.
283
301
.
52.
Amid
,
E.
, and
Ukkonen
,
A.
,
2015
, “
Multiview Triplet Embedding: Learning Attributes in Multiple Maps
,”
International Conference on Machine Learning
, Lille, France, July 6–11, pp.
1472
1480
.https://pdfs.semanticscholar.org/0124/4108e4b6a190aa4c5085518e3c3412fa0087.pdf
You do not currently have access to this content.