When selecting ideas or trying to find inspiration, designers often must sift through hundreds or thousands of ideas. This paper provides an algorithm to rank design ideas such that the ranked list simultaneously maximizes the quality and diversity of recommended designs. To do so, we first define and compare two diversity measures using determinantal point processes (DPP) and additive submodular functions. We show that DPPs are more suitable for items expressed as text and that a greedy algorithm diversifies rankings with both theoretical guarantees and empirical performance on what is otherwise an NP-Hard problem. To produce such rankings, this paper contributes a novel way to extend quality and diversity metrics from sets to permutations of ranked lists. These rank metrics open up the use of multi-objective optimization to describe trade-offs between diversity and quality in ranked lists. We use such trade-off fronts to help designers select rankings using indifference curves. However, we also show that rankings on trade-off front share a number of top-ranked items; this means reviewing items (for a given depth like the top ten) from across the entire diversity-to-quality front incurs only a marginal increase in the number of designs considered. While the proposed techniques are general purpose enough to be used across domains, we demonstrate concrete performance on selecting items in an online design community (OpenIDEO), where our approach reduces the time required to review diverse, high-quality ideas from around 25 h to 90 min. This makes evaluation of crowd-generated ideas tractable for a single designer. Our code is publicly accessible for further research.

References

References
1.
Pauling
,
L.
,
2001
,
Linus Pauling: Selected Scientific Papers
, Vol.
2
,
World Scientific
, Singapore.
2.
Ahmed
,
F.
,
Fuge
,
M.
, and
Gorbunov
,
L. D.
,
2016
, “
Discovering Diverse, High Quality Design Ideas From a Large Corpus
,”
ASME
Paper No. DETC2016-59926.
3.
Shah
,
J. J.
,
Kulkarni
,
S. V.
, and
Vargas-Hernandez
,
N.
,
2000
, “
Evaluation of Idea Generation Methods for Conceptual Design: Effectiveness Metrics and Design of Experiments
,”
ASME J. Mech. Des.
,
122
(
4
), pp.
377
384
.
4.
Verhaegen
,
P.-A.
,
Vandevenne
,
D.
,
Peeters
,
J.
, and
Duflou
,
J. R.
,
2013
, “
Refinements to the Variety Metric for Idea Evaluation
,”
Des. Stud.
,
34
(
2
), pp.
243
263
.
5.
Hennessey
,
B. A.
, and
Amabile
,
T. M.
,
1999
, “
Consensual Assessment
,”
Encycl. Creativity
,
1
, pp.
347
359
.
6.
Fuge
,
M.
,
Stroud
,
J.
, and
Agogino
,
A.
,
2013
, “
Automatically Inferring Metrics for Design Creativity
,”
ASME
Paper No. DETC2013-12620.
7.
Kudrowitz
,
B. M.
, and
Wallace
,
D.
,
2013
, “
Assessing the Quality of Ideas From Prolific, Early-Stage Product Ideation
,”
J. Eng. Des.
,
24
(
2
), pp.
120
139
.
8.
Green
,
M.
,
Seepersad
,
C. C.
, and
Hölttä-Otto
,
K.
,
2014
, “
Crowd-Sourcing the Evaluation of Creativity in Conceptual Design: A Pilot Study
,”
ASME
Paper No. DETC2014-34434.
9.
Von Hippel
,
E.
,
2005
, “
Democratizing Innovation: The Evolving Phenomenon of User Innovation
,”
J. Für Betriebswirtschaft
,
55
(
1
), pp.
63
78
.
10.
Chiu
,
I.
, and
Shu
,
L.
,
2012
, “
Investigating Effects of Oppositely Related Semantic Stimuli on Design Concept Creativity
,”
J. Eng. Des.
,
23
(
4
), pp.
271
296
.
11.
Ali
,
K.
, and
Van Stam
,
W.
,
2004
, “
Tivo: Making Show Recommendations Using a Distributed Collaborative Filtering Architecture
,”
Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
(
KDD
), Seattle, WA, Aug. 22–25, pp.
394
401.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.444.9998&rep=rep1&type=pdf
12.
Coombs
,
C. H.
, and
Avrunin
,
G. S.
,
1977
, “
Single-Peaked Functions and the Theory of Preference
,”
Psychol. Rev.
,
84
(
2
), p.
216
.
13.
Ziegler
,
C.-N.
,
McNee
,
S. M.
,
Konstan
,
J. A.
, and
Lausen
,
G.
,
2005
, “
Improving Recommendation Lists Through Topic Diversification
,”
14th International Conference on World Wide Web
(
WWW
), Chiba, Japan, May 10–14, pp.
22
32
.http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.62.9683&rep=rep1&type=pdf
14.
Puthiya Parambath
,
S. A.
,
Usunier
,
N.
, and
Grandvalet
,
Y.
,
2016
, “
A Coverage-Based Approach to Recommendation Diversity on Similarity Graph
,”
Tenth ACM Conference on Recommender Systems
(
RecSys
), Boston, MA, Sept. 15–19, pp.
15
22
.
15.
Santos
,
R. L.
,
Macdonald
,
C.
, and
Ounis
,
I.
,
2010
, “
Exploiting Query Reformulations for Web Search Result Diversification
,”
19th International Conference on World Wide Web
(
WWW
), Raleigh, NC, Apr. 26–30, pp.
881
890
.http://wwwconference.org/proceedings/www2010/www/p881.pdf
16.
Zhang
,
B.
,
Li
,
H.
,
Liu
,
Y.
,
Ji
,
L.
,
Xi
,
W.
,
Fan
,
W.
,
Chen
,
Z.
, and
Ma
,
W.-Y.
,
2005
, “
Improving Web Search Results Using Affinity Graph
,”
28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
(
SIGIR
), Salvador, Brazil, Aug. 15–19, pp.
504
511
.
17.
He
,
J.
,
Tong
,
H.
,
Mei
,
Q.
, and
Szymanski
,
B.
,
2012
, “
Gender: A Generic Diversified Ranking Algorithm
,”
Advances in Neural Information Processing Systems
(
NIPS
), Stateline, NV, Dec. 3–8, pp.
1151
1159
.https://papers.nips.cc/paper/4647-gender-a-generic-diversified-ranking-algorithm.pdf
18.
Vargas
,
S.
, and
Castells
,
P.
,
2011
, “
Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems
,”
Fifth ACM Conference on Recommender Systems
(
RecSys
), Chicago, IL, Oct. 23–27, pp.
109
116
.
19.
Castells
,
P.
,
Hurley
,
N. J.
, and
Vargas
,
S.
,
2015
, “
Novelty and Diversity in Recommender Systems
,”
Recommender Systems Handbook
,
Springer
, New York, pp.
881
918
.
20.
Zhang
,
Y. C.
,
Séaghdha
,
D. Ó.
,
Quercia
,
D.
, and
Jambor
,
T.
,
2012
, “
Auralist: Introducing Serendipity Into Music Recommendation
,”
Fifth ACM International Conference on Web Search and Data Mining
(
WSDM
), Seattle, WA, Feb. 8–12, pp.
13
22.
21.
Fisher
,
D.
,
Jain
,
A.
,
Keikha
,
M.
,
Croft
,
W.
, and
Lipka
,
N.
,
2015
, “
Evaluating Ranking Diversity and Summarization in Microblogs Using Hashtags
,” University of Massachusetts, Boston, MA,
Technical Report
.https://pdfs.semanticscholar.org/9f0c/53afcc5e33b8b22722add0812bf14ccf875b.pdf
22.
Patil
,
G.
, and
Taillie
,
C.
,
1982
, “
Diversity as a Concept and Its Measurement
,”
J. Am. Stat. Assoc.
,
77
(
379
), pp.
548
561
.
23.
Zhu
,
X.
,
Goldberg
,
A. B.
,
Van Gael
,
J.
, and
Andrzejewski
,
D.
,
2007
, “
Improving Diversity in Ranking Using Absorbing Random Walks
,”
Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics
(
NAACL-HLT
), Rochester, NY, Apr. 22–27, pp.
97
104
.http://pages.cs.wisc.edu/~jerryzhu/pub/grasshopper.pdf
24.
Zhao
,
P.
, and
Lee
,
D. L.
,
2016
, “
How Much Novelty Is Relevant? It Depends on Your Curiosity
,”
39th International ACM SIGIR Conference on Research and Development in Information Retrieval
(
SIGIR
), Pisa, Italy, July 17–21, pp. 315–324.
25.
Wang
,
X.
,
Dou
,
Z.
,
Sakai
,
T.
, and
Wen
,
J.-R.
,
2016
, “
Evaluating Search Result Diversity Using Intent Hierarchies
,”
39th International ACM SIGIR Conference on Research and Development in Information Retrieval
(
SIGIR
), Pisa, Italy, July 17–22, pp.
415
424
.
26.
Chapelle
,
O.
,
Ji
,
S.
,
Liao
,
C.
,
Velipasaoglu
,
E.
,
Lai
,
L.
, and
Wu
,
S.-L.
,
2011
, “
Intent-Based Diversification of Web Search Results: Metrics and Algorithms
,”
Inf. Retr.
,
14
(
6
), pp.
572
592
.
27.
Clarke
,
C. L.
,
Kolla
,
M.
,
Cormack
,
G. V.
,
Vechtomova
,
O.
,
Ashkan
,
A.
,
Büttcher
,
S.
, and
MacKinnon
,
I.
,
2008
, “
Novelty and Diversity in Information Retrieval Evaluation
,”
31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
(
SIGIR
), Singapore, July 20–24, pp.
659
666
.
28.
Carterette
,
B.
,
2009
, “
An Analysis of Np-Completeness in Novelty and Diversity Ranking
,”
International Conference on Theory of Information Retrieval: Advances in Information Retrieval Theory
(
ICTIR
), Cambridge, UK, Sept. 10–12, pp.
200
211
.
29.
Chen
,
W.
,
Chazan
,
J.
, and
Fuge
,
M.
,
2016
, “
How Designs Differ: Non-Linear Embeddings Illuminate Intrinsic Design Complexity
,”
ASME
Paper No. DETC2016-60112.
30.
Yumer
,
M. E.
,
Asente
,
P.
,
Mech
,
R.
, and
Kara
,
L. B.
,
2015
, “
Procedural Modeling Using Autoencoder Networks
,”
28th Annual ACM Symposium on User Interface Software & Technology
(
UIST
), Charlotte, NC, Nov. 11–15, pp.
109
118
.
31.
Burnap
,
A.
,
Pan
,
Y.
,
Liu
,
Y.
,
Ren
,
Y.
,
Lee
,
H.
,
Gonzalez
,
R.
, and
Papalambros
,
P. Y.
,
2016
, “
Improving Design Preference Prediction Accuracy Using Feature Learning
,”
ASME J. Mech. Des.
,
138
(
7
), p.
071404
.
32.
Yu
,
Q.
,
Yang
,
Y.
,
Song
,
Y.-Z.
,
Xiang
,
T.
, and
Hospedales
,
T.
,
2015
, “
Sketch-A-Net That Beats Humans
,”
Int. J. Com. Vision
,
122
(3), pp. 411–425http://www.eecs.qmul.ac.uk/~tmh/papers/yu2015sketchanet.pdf.
33.
Dong
,
A.
,
2005
, “
The Latent Semantic Approach to Studying Design Team Communication
,”
Des. Stud.
,
26
(
5
), pp.
445
461
.
34.
Pu
,
Y.
,
Gan
,
Z.
,
Henao
,
R.
,
Yuan
,
X.
,
Li
,
C.
,
Stevens
,
A.
, and
Carin
,
L.
,
2016
, “
Variational Autoencoder for Deep Learning of Images, Labels and Captions
,” Advances in Neural Information Processing Systems (
NIPS
), Barcelona, Spain, Dec. 5–10, pp.
2352
2360
.https://zhegan27.github.io/Papers/vae_nips2016_poster.pdf
35.
Tamuz
,
O.
,
Liu
,
C.
,
Belongie
,
S.
,
Shamir
,
O.
, and
Kalai
,
A. T.
,
2011
, “
Adaptively Learning the Crowd Kernel
,” International Conference on Machine Learning (
ICML
), Bellevue, WA, June 28–July 2, pp. 673–680https://dl.acm.org/citation.cfm?id=3104567.
36.
Qian
,
L.
, and
Gero
,
J. S.
,
1996
, “
Function–Behavior–Structure Paths and Their Role in Analogy-Based Design
,”
Artificial Intell. Eng., Des., Anal. Manuf.
,
10
(
4
), pp.
289
312
.
37.
Kirschman
,
C.
,
Fadel
,
G.
, and
Jara-Almonte
,
C.
,
1998
, “
Classifying Functions for Mechanical Design
,”
ASME J. Mech. Des.
,
120
(
3
), pp.
475
482
.
38.
Stone
,
R. B.
, and
Wood
,
K. L.
,
2000
, “
Development of a Functional Basis for Design
,”
ASME J. Mech. Des.
,
122
(
4
), pp.
359
370
.
39.
Vishwanathan
,
S. V. N.
,
Schraudolph
,
N. N.
,
Kondor
,
R.
, and
Borgwardt
,
K. M.
,
2010
, “
Graph Kernels
,”
J. Mach. Learn. Res.
,
11
, pp.
1201
1242
.http://www.jmlr.org/papers/v11/vishwanathan10a.html
40.
Lin
,
H.
, and
Bilmes
,
J.
,
2011
, “
A Class of Submodular Functions for Document Summarization
,”
49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies
(
HLT
), Portland, OR, June 19–24, pp.
510
520
.https://dl.acm.org/citation.cfm?id=2002537
41.
Lin
,
H.
, and
Bilmes
,
J. A.
,
2012
, “
Learning Mixtures of Submodular Shells With Application to Document Summarization
,”
Twenty-Eighth Conference on Uncertainty in Artificial Intelligence
(
UAI
), Catalina Island, CA, Aug. 14–18, pp. 479–490https://dl.acm.org/citation.cfm?id=3020652.3020704.
42.
Kulesza
,
A.
, and
Taskar
,
B.
,
2012
,
Determinantal Point Processes for Machine Learning
, Now Publishers Inc., Hanover, MA.
43.
Boim
,
R.
,
Milo
,
T.
, and
Novgorodov
,
S.
,
2011
, “
Diversification and Refinement in Collaborative Filtering Recommender
,”
20th ACM International Conference on Information and Knowledge Management
(
CIKM
), Glasgow, Scotland, Oct. 24–28, pp.
739
744
.
44.
Feige
,
U.
,
Mirrokni
,
V. S.
, and
Vondrak
,
J.
,
2011
, “
Maximizing Non-Monotone Submodular Functions
,”
SIAM J. Comput.
,
40
(
4
), pp.
1133
1153
.
45.
Manning
,
C. D.
, and
Schütze
,
H.
,
1999
,
Foundations of Statistical Natural Language Processing
, Vol.
999
,
MIT Press
, Cambridge, MA.
46.
Ng
,
A. Y.
,
Jordan
,
M. I.
, and
Weiss
,
Y.
,
2002
, “
On Spectral Clustering: Analysis and An Algorithm
,”
Advances in Neural Information Processing Systems
(
NIPS
), Vancouver, BC, Canada, Dec. 3–8, pp.
849
856
.https://dl.acm.org/citation.cfm?id=2980649
47.
Kulesza
,
A.
, and
Taskar
,
B.
,
2011
, “
Learning Determinantal Point Processes
,”
27th Conference on Uncertainty in Artificial Intelligence
(
UAI
), Barcelona, Spain, July 14–17, pp. 1–9.https://homes.cs.washington.edu/~taskar/pubs/ldpps_uai11.pdf
48.
Kulesza
,
A.
, and
Taskar
,
B.
,
2011
, “
k-Dpps: Fixed-Size Determinantal Point Processes
,”
28th International Conference on Machine Learning
(
ICML
), Bellevue, WA, June 28–July 2, pp.
1193
1200
.https://homes.cs.washington.edu/~taskar/pubs/kdpps_icml11.pdf
49.
Borodin
,
A.
,
2009
, “
Determinantal Point Processes
,” preprint
arXiv:0911.1153
.https://arxiv.org/abs/0911.1153
50.
Toubia
,
O.
, and
Florès
,
L.
,
2007
, “
Adaptive Idea Screening Using Consumers
,”
Mark. Sci.
,
26
(
3
), pp.
342
360
.
51.
Mollick
,
E.
, and
Nanda
,
R.
,
2015
, “
Wisdom or Madness? Comparing Crowds With Expert Evaluation in Funding the Arts
,”
Manage. Sci.
,
62
(
6
), pp.
1533
1553
.
52.
Ahmed
,
F.
, and
Fuge
,
M.
,
2017
, “
Capturing Winning Ideas in Online Design Communities
,”
20th ACM Conference on Computer-Supported Cooperative Work & Social Computing
(
CSCW
), Portland, OR, Feb. 25–Mar. 1, pp. 1675–1687.
53.
Järvelin
,
K.
, and
Kekäläinen
,
J.
,
2002
, “
Cumulated Gain-Based Evaluation of IR Techniques
,”
ACM Trans. Inf. Syst.
,
20
(
4
), pp.
422
446
.
54.
Carbonell
,
J.
, and
Goldstein
,
J.
,
1998
, “
The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries
,”
21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
(
SIGIR
), Melbourne, Australia, Aug. 24–28, pp.
335
336
.
55.
Deb
,
K.
,
Pratap
,
A.
,
Agarwal
,
S.
, and
Meyarivan
,
T.
,
2002
, “
A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II
,”
IEEE Trans. Evol. Comput.
,
6
(
2
), pp.
182
197
.
56.
Fuge
,
M.
,
Tee
,
K.
,
Agogino
,
A.
, and
Maton
,
N.
,
2014
, “
Analysis of Collaborative Design Networks: A Case Study of OpenIDEO
,”
ASME J. Comput. Inf. Sci. Eng.
,
14
(
2
), p.
021009
.
57.
Chiu
,
P.-W.
, and
Bloebaum
,
C.
,
2008
, “
Hyper-Radial Visualization (HRV) With Weighted Preferences for Multi-Objective Decision Making
,”
AIAA
Paper No. 2008-5986.
58.
Hofmann
,
K.
,
Whiteson
,
S.
, and
Rijke
,
M. D.
,
2013
, “
Fidelity, Soundness, and Efficiency of Interleaved Comparison Methods
,”
ACM Trans. Inf. Syst. (TOIS)
,
31
(
4
), p.
17
.
59.
Deb
,
K.
, and
Gupta
,
S.
,
2011
, “
Understanding Knee Points in Bicriteria Problems and Their Implications as Preferred Solution Principles
,”
Eng. Optim.
,
43
(
11
), pp.
1175
1204
.
60.
Jain
,
L.
,
Jamieson
,
K. G.
, and
Nowak
,
R.
,
2016
, “
Finite Sample Prediction and Recovery Bounds for Ordinal Embedding
,”
Advances in Neural Information Processing Systems
(
NIPS
), Barcelona, Spain, Dec. 5–10, pp.
2703
2711
.https://papers.nips.cc/paper/6554-finite-sample-prediction-and-recovery-bounds-for-ordinal-embedding
61.
Chakrabarti
,
A.
,
Shea
,
K.
,
Stone
,
R.
,
Cagan
,
J.
,
Campbell
,
M.
,
Hernandez
,
N. V.
, and
Wood
,
K. L.
,
2011
, “
Computer-Based Design Synthesis Research: An Overview
,”
ASME J. Comput. Inf. Sci. Eng.
,
11
(
2
), p.
021003
.
You do not currently have access to this content.