Abstract

Design can be viewed as a sequential and iterative search process. Fundamental understanding and computational modeling of human sequential design decisions are essential for developing new methods in design automation and human–AI collaboration. This paper presents an approach for predicting designers’ future search behaviors in a sequential design process under an unknown objective function by combining sequence learning with game theory. While the majority of existing studies focus on analyzing sequential design decisions from the descriptive and prescriptive point of view, this study is motivated to develop a predictive framework. We use data containing designers’ actual sequential search decisions under competition collected from a black-box function optimization game developed previously. We integrate the long short-term memory networks with the Delta method to predict the next sampling point with a distribution, and combine this model with a non-cooperative game to predict whether a designer will stop searching the design space or not based on their belief of the opponent’s best design. In the function optimization game, the proposed model accurately predicts 82% of the next design variable values and 92% of the next function values in the test data with an upper and lower bound, suggesting that a long short-term memory network can effectively predict the next design decisions based on their past decisions. Further, the game-theoretic model predicts that 60.8% of the participants stop searching for designs sooner than they actually do while accurately predicting when the remaining 39.2% of the participants stop. These results suggest that a majority of the designers show a strong tendency to overestimate their opponents’ performance, leading them to spend more on searching for better designs than they would have, had they known their opponents’ actual performance.

References

1.
Walden
,
D. D.
,
Roedler
,
G. J.
,
Forsberg
,
K. J.
,
Hamelin
,
R. D.
, and
Shortell
,
T. M.
,
2015
,
INCOSE Systems Engineering Handbook: A Guide for System Life Cycle Processes and Activities
, 4 ed.,
Wiley
,
Hoboken, NJ
.
2.
VDI
,
1993
, “
2221: Systematic Approach to the Design of Technical Systems and Products
,” Standard DINGCD22,
Deutsches Institut für Normung
,
Düsseldorf, Germany
.
3.
Shiau
,
C.-S. N.
, and
Michalek
,
J. J.
,
2009
, “
Should Designers Worry About Market Systems?
,”
ASME J. Mech. Des.
,
131
(
1
), p.
011011
. 10.1115/1.3013848
4.
Tversky
,
A.
, and
Kahneman
,
D.
,
1974
, “
Judgment Under Uncertainty: Heuristics and Biases
,”
Science
,
185
(
4157
), pp.
1124
1131
. 10.1126/science.185.4157.1124
5.
Jacowitz
,
K. E.
, and
Kahneman
,
D.
,
1995
, “
Measures of Anchoring in Estimation Tasks
,”
Personality Soc. Psychol. Bull.
,
21
(
11
), pp.
1161
1166
. 10.1177/01461672952111004
6.
Tversky
,
A.
, and
Kahneman
,
D.
,
1973
, “
Availability: A Heuristic for Judging Frequency and Probability
,”
Cognit. Psychol.
,
5
(
2
), pp.
207
232
. 10.1016/0010-0285(73)90033-9
7.
Yassine
,
A.
,
2004
, “
An Introduction to Modeling and Analyzing Complex Product Development Processes Using the Design Structure Matrix (DSM) Method
,”
Quaderni di Manage. (Italian Manage. Rev.)
,
9
, pp.
71
88
.
8.
Yassine
,
A. A.
,
Whitney
,
D. E.
, and
Zambito
,
T.
,
2001
, “
Assessment of Rework Probabilities for Simulating Product Development Processes Using the Design Structure Matrix (DSM)
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Pittsburgh, PA
,
Sept. 9–12
, American Society of Mechanical Engineers, New York.
9.
Browning
,
T. R.
,
2001
, “
Applying the Design Structure Matrix to System Decomposition and Integration Problems: A Review and New Directions
,”
IEEE Trans. Eng. Manage.
,
48
(
3
), pp.
292
306
. 10.1109/17.946528
10.
Unal
,
M.
,
Miller
,
S. W.
,
Chhabra
,
J. P.
,
Warn
,
G. P.
,
Yukish
,
M. A.
, and
Simpson
,
T. W.
,
2017
, “
A Sequential Decision Process for the System-Level Design of Structural Frames
,”
Struct. Multidiscip. Optim.
,
56
(
5
), pp.
991
1011
. 10.1007/s00158-017-1697-1
11.
Miller
,
S. W.
,
Yukish
,
M. A.
, and
Simpson
,
T. W.
,
2018
, “
Design as a Sequential Decision Process: A Method for Reducing Design Set Space Using Models to Bound Objectives
,”
Struct. Multidiscip. Optim.
,
57
(
1
), pp.
305
324
. 10.1007/s00158-017-1756-7
12.
Griffin
,
S.
,
Welton
,
N. J.
, and
Claxton
,
K.
,
2010
, “
Exploring the Research Decision Space: The Expected Value of Information for Sequential Research Designs
,”
Med. Decis. Making
,
30
(
2
), pp.
155
162
. 10.1177/0272989X09344746
13.
Shahriari
,
B.
,
Swersky
,
K.
,
Wang
,
Z.
,
Adams
,
R. P.
, and
De Freitas
,
N.
,
2016
, “
Taking the Human Out of the Loop: A Review of Bayesian Optimization
,”
Proc. IEEE
,
104
(
1
), pp.
148
175
. 10.1109/JPROC.2015.2494218
14.
Meier
,
C.
,
Yassine
,
A. A.
, and
Browning
,
T. R.
,
2007
, “
Design Process Sequencing With Competent Genetic Algorithms
,”
ASME J. Mech. Des.
,
129
(
6
), pp.
566
585
. 10.1115/1.2717224
15.
Duff
,
M. O.
, and
Barto
,
A.
,
2002
, “
Optimal Learning: Computational Procedures for Bayes-Adaptive Markov Decision Processes
,” Ph.D. Thesis,
University of Massachusetts
,
Amherst
.
16.
Chaudhari
,
A. M.
,
Bilionis
,
I.
, and
Panchal
,
J. H.
,
2020
, “
Descriptive Models of Sequential Decisions in Engineering Design: An Experimental Study
,”
ASME J. Mech. Des.
,
142
(
8
), p.
081704
. 10.1115/1.4045605
17.
McComb
,
C.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2017
, “
Capturing Human Sequence-Learning Abilities in Configuration Design Tasks Through Markov Chains
,”
ASME J. Mech. Des.
,
139
(
9
), p.
091101
. 10.1115/1.4037185
18.
McComb
,
C.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2017
, “
Mining Process Heuristics From Designer Action Data Via Hidden Markov Models
,”
ASME J. Mech. Des.
,
139
(
11
), p.
111412
. 10.1115/1.4037308
19.
Gero
,
J. S.
,
Kan
,
J. W.
, and
Pourmohamadi
,
M.
,
2011
, “Analysing Design Protocols: Development of Methods and Tools,”
Research Into Design – Supporting Sustainable Product Development
,
Chakrabarti
,
A.
, ed.,
Research Publishing
,
Bangalore, India
, pp.
3
10
.
20.
Rahman
,
M. H.
,
Schimpf
,
C.
,
Xie
,
C.
, and
Sha
,
Z.
,
2019
, “
A Computer-Aided Design Based Research Platform for Design Thinking Studies
,”
ASME J. Mech. Des.
,
141
(
12
), p.
121102
. 10.1115/1.4044395
21.
McComb
,
C.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2015
, “
Lifting the Veil: Drawing Insights About Design Teams From a Cognitively-Inspired Computational Model
,”
Des. Stud.
,
40
(
C
), pp.
119
142
. 10.1016/j.destud.2015.06.005
22.
Chaudhari
,
A. M.
, and
Panchal
,
J. H.
,
2019
, “An Experimental Study of Human Decisions in Sequential Information Acquisition in Design: Impact of Cost and Task Complexity,”
Research Into Design for a Connected World
, Vol.
134
,
Springer
,
Berlin
, pp.
321
332
.
23.
Sexton
,
T.
, and
Ren
,
M. Y.
,
2017
, “
Learning An Optimization Algorithm Through Human Design Iterations
,”
ASME J. Mech. Des.
,
139
(
10
), p.
101404
. 10.1115/1.4037344
24.
Panchal
,
J. H.
,
Sha
,
Z.
, and
Kannan
,
K. N.
,
2017
, “
Understanding Design Decisions Under Competition Using Games With Information Acquisition and a Behavioral Experiment
,”
ASME J. Mech. Des.
,
139
(
9
), p.
091402
. 10.1115/1.4037253
25.
Raina
,
A.
,
McComb
,
C.
, and
Cagan
,
J.
,
2019
, “
Learning to Design From Humans: Imitating Human Designers Through Deep Learning
,”
ASME J. Mech. Des.
,
141
(
11
), p.
111102
.
26.
Rahman
,
M. H.
,
Xie
,
C.
, and
Sha
,
Z.
,
2019
, “
A Deep Learning Based Approach to Predict Sequential Design Decisions
,”
ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Anaheim, CA
,
Aug. 18–21
, Vol.
59179
, American Society of Mechanical Engineers, New York, p.
V001T02A029
.
27.
Sha
,
Z.
,
Kannan
,
K. N.
, and
Panchal
,
J. H.
,
2015
, “
Behavioral Experimentation and Game Theory in Engineering Systems Design
,”
ASME J. Mech. Des.
,
137
(
5
), p.
051405
. 10.1115/1.4029767
28.
Shiau
,
C.-S. N.
, and
Michalek
,
J. J.
,
2009
, “
Optimal Product Design Under Price Competition
,”
ASME J. Mech. Des.
,
131
(
7
), p.
071003
. 10.1115/1.3125886
29.
Frischknecht
,
B. D.
,
Whitefoot
,
K.
, and
Papalambros
,
P. Y.
,
2010
, “
On the Suitability of Econometric Demand Models in Design for Market Systems
,”
ASME J. Mech. Des.
,
132
(
12
), p.
121007
. 10.1115/1.4002941
30.
Gurnani
,
A.
, and
Lewis
,
K.
,
2008
, “
Collaborative, Decentralized Engineering Design at the Edge of Rationality
,”
ASME J. Mech. Des.
,
130
(
12
), p.
121101
. 10.1115/1.2988479
31.
Skaperdas
,
S.
,
1996
, “
Contest Success Functions
,”
Econ. Theory
,
7
(
2
), pp.
283
290
. 10.1007/BF01213906
32.
Hopfield
,
J. J.
,
1982
, “
Neural Networks and Physical Systems With Emergent Collective Computational Abilities
,”
Proc. Natl. Acad. Sci. U.S.A.
,
79
(
8
), pp.
2554
2558
. 10.1073/pnas.79.8.2554
33.
Rumelhart
,
D. E.
,
Hinton
,
G. E.
, and
Williams
,
R. J.
,
1986
, “
Learning Representations by Back-Propagating Errors
,”
Nature
,
323
(
6088
), pp.
533
536
. 10.1038/323533a0
34.
Hochreiter
,
S.
, and
Schmidhuber
,
J.
,
1997
, “
Long Short-term Memory
,”
Neural Comput.
,
9
(
8
), pp.
1735
1780
. 10.1162/neco.1997.9.8.1735
35.
Greff
,
K.
,
Srivastava
,
R. K.
,
Koutník
,
J.
,
Steunebrink
,
B. R.
, and
Schmidhuber
,
J.
,
2016
, “
LSTM: A Search Space Odyssey
,”
IEEE Trans. Neural Netw. Learning Syst.
,
28
(
10
), pp.
2222
2232
. 10.1109/TNNLS.2016.2582924
36.
Olah
,
C.
,
2015
, “
Understanding LSTM Networks
”. http://colah.github.io/posts/2015-08-Understanding-LSTMs/
37.
De Veaux
,
R. D.
,
Schumi
,
J.
,
Schweinsberg
,
J.
, and
Ungar
,
L. H.
,
1998
, “
Prediction Intervals for Neural Networks Via Nonlinear Regression
,”
Technometrics
,
40
(
4
), pp.
273
282
. 10.2307/1270528
38.
Baldwin
,
C. Y.
, and
Clark
,
K. B.
,
2000
,
The Power of Modularity, Design Rules
, Vol.
1
,
The MIT Press
,
Cambridge, MA
.
39.
Papalambros
,
P. Y.
, and
Wilde
,
D. J.
,
2017
,
Principles of Optimal Design: Modeling and Computation
, 3rd ed.,
Cambridge University Press
,
Cambridge, UK
.
40.
Khosravi
,
A.
,
Nahavandi
,
S.
,
Creighton
,
D.
, and
Atiya
,
A. F.
,
2011
, “
Comprehensive Review of Neural Network-Based Prediction Intervals and New Advances
,”
IEEE Trans. Neural Networks
,
22
(
9
), pp.
1341
1356
. 10.1109/TNN.2011.2162110
41.
Chan
,
E.
, and
Ybarra
,
O.
,
2002
, “
Interaction Goals and Social Information Processing: Underestimating One’s Partners But Overestimating One’s Opponents
,”
Social Cognit.
,
20
(
5
), pp.
409
439
. 10.1521/soco.20.5.409.21126
You do not currently have access to this content.