Abstract

Generative design problems often encompass complex action spaces that may be divergent over time, contain state-dependent constraints, or involve hybrid (discrete and continuous) domains. To address those challenges, this work introduces Design Strategy network (DSN), a data-driven deep hierarchical framework that can learn strategies over these arbitrary complex action spaces. The hierarchical architecture decomposes every action decision into first predicting a preferred spatial region in the design space and then outputting a probability distribution over a set of possible actions from that region. This framework comprises a convolutional encoder to work with image-based design state representations, a multi-layer perceptron to predict a spatial region, and a weight-sharing network to generate a probability distribution over unordered set-based inputs of feasible actions. Applied to a truss design study, the framework learns to predict the actions of human designers in the study, capturing their truss generation strategies in the process. Results show that DSNs significantly outperform nonhierarchical methods of policy representation, demonstrating their superiority in complex action space problems.

References

1.
Yukish
,
M. A.
,
Miller
,
S. W.
, and
Simpson
,
T. W.
,
2015
, “
A Preliminary Model of Design as a Sequential Decision Process
,”
Procedia Comput. Sci.
,
44
(
C
), pp.
174
183
.
2.
Hazelrigg
,
G. A.
,
1998
, “
A Framework for Decision-Based Engineering Design
,”
ASME J. Mech. Des.
,
120
(
4
), pp.
653
658
.
3.
Gigerenzer
,
G.
, and
Gaissmaier
,
W.
,
2011
, “
Heuristic Decision Making
,”
Annu. Rev. Psychol
,
62
(
1
), pp.
451
482
.
4.
Smith
,
R. P.
, and
Eppinger
,
S. D.
,
1997
, “
Identifying Controlling Features of Engineering Design Iteration
,”
Manage. Sci
,
43
(
3
), pp.
276
293
.
5.
Wielinga
,
B.
, and
Schreiber
,
G.
,
1997
, “
Configuration-Design Problem Solving
,”
IEEE Expert.
,
12
(
2
), pp.
49
56
.
6.
Chen
,
W.
,
Hoyle
,
C.
, and
Wassenaar
,
H. J.
,
2013
,
Decision Based Design
, 1st ed., Vol.
1
,
Springer
,
London, UK
, pp.
1
358
.
7.
Allis
,
L. V.
, and
Lavé
,
T.
,
1994
,
Searching for Solutions in Games and Artificial Intelligence
,
PhD Thesis
, 1st ed., Vol.
1
,
Rijksuniversiteit Limburg
,
Maastricht, Netherlands
, pp.
1
223
.
8.
Sutton
,
R. S.
, and
Barto
,
A. G.
,
2018
,
Reinforcement Learning: An Introduction
,
A Bradford Book
,
Cambridge, MA
.
9.
Abbeel
,
P.
, and
Ng
,
A. Y.
,
2004
, “
Apprenticeship Learning via Inverse Reinforcement Learning
,”
Proceedings, Twenty-First International Conference on Machine Learning, ICML 2004
,
Banff, Alberta, Canada
,
July
, ACM, pp.
1
8
.
10.
Mnih
,
V.
,
Kavukcuoglu
,
K.
,
Silver
,
D.
,
Rusu
,
A. A.
,
Veness
,
J.
,
Bellemare
,
M. G.
,
Graves
,
A.
, et al
,
2015
, “
Human-Level Control Through Deep Reinforcement Learning
,”
Nature
,
518
(
7540
), pp.
529
533
.
11.
Silver
,
D.
,
Schrittwieser
,
J.
,
Simonyan
,
K.
,
Antonoglou
,
I.
,
Huang
,
A.
,
Guez
,
A.
,
Hubert
,
T.
, et al
,
2017
, “
Mastering the Game of Go Without Human Knowledge
,”
Nature
,
550
(
7676
), pp.
354
359
.
12.
Schrittwieser
,
J.
,
Antonoglou
,
I.
,
Hubert
,
T.
,
Simonyan
,
K.
,
Sifre
,
L.
,
Schmitt
,
S.
,
Guez
,
A.
, et al
,
2020
, “
Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model
,”
Nature
,
588
(
7839
), pp.
604
609
.
13.
Vinyals
,
O.
,
Babuschkin
,
I.
,
Czarnecki
,
W. M.
,
Mathieu
,
M.
,
Dudzik
,
A.
,
Chung
,
J.
,
Choi
,
D. H.
, et al
,
2019
, “
Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning
,”
Nature
,
575
(
7782
), pp.
350
354
.
14.
Bojarski
,
M.
,
Del Testa
,
D.
,
Dworakowski
,
D.
,
Firner
,
B.
,
Flepp
,
B.
,
Goyal
,
P.
,
Jackel
,
L. D.
, et al
,
2016
, “
End to End Learning for Self-Driving Cars
,”
ArXiv
.
15.
McComb
,
C.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2017
, “
Capturing Human Sequence-Learning Abilities in Configuration Design Tasks Through Markov Chains
,”
ASME J. Mech. Des.
,
139
(
9
), p.
091101
.
16.
McComb
,
C.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2017
, “
Mining Process Heuristics From Designer Action Data Via Hidden Markov Models
,”
ASME J. Mech. Des.
,
139
(
11
), p.
111412
.
17.
Rahman
,
M. H.
,
Yuan
,
S.
,
Xie
,
C.
, and
Sha
,
Z.
,
2020
, “
Predicting Human Design Decisions With Deep Recurrent Neural Network Combining Static and Dynamic Data
,”
Des. Sci.e
,
6
(
15
), pp.
1
26
.
18.
Shergadwala
,
M.
,
Bilionis
,
I.
,
Kannan
,
K. N.
, and
Panchal
,
J. H.
,
2018
, “
Quantifying the Impact of Domain Knowledge and Problem Framing on Sequential Decisions in Engineering Design
,”
ASME J. Mech. Des.
,
140
(
10
), p.
101402
.
19.
Bayrak
,
A. E.
, and
Sha
,
Z.
,
2021
, “
Integrating Sequence Learning and Game Theory to Predict Design Decisions Under Competition
,”
ASME J. Mech. Des.
,
143
(
5
), p.
051401
.
20.
Puentes
,
L.
,
Cagan
,
J.
, and
McComb
,
C.
,
2021
, “
Data-Driven Heuristic Induction From Human Design Behavior
,”
ASME J. Comput. Inf. Sci. Eng.
,
21
(
2
), p.
024501
.
21.
McComb
,
C.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2015
, “
Rolling with the Punches: An Examination of Team Performance in a Design Task Subject to Drastic Changes
,”
Des. Stud.
,
36
(
C
), pp.
99
121
.
22.
Michell
,
A. G. M.
,
1904
, “
LVIII. The Limits of Economy of Material in Frame-Structures
,”
Lond. Edinb. Dublin Philos. Mag. J. Sci.
,
8
(
47
), pp.
589
597
.
23.
Raina
,
A.
,
Puentes
,
L.
,
Cagan
,
J.
, and
McComb
,
C.
,
2021
, “
Goal-Directed Design Agents: Integrating Visual Imitation With One-Step Lookahead Optimization for Generative Design
,”
ASME J. Mech. Des.
,
143
(
12
), p.
124501
.
24.
Wood
,
W. H.
, and
Agogino
,
A. M.
,
2005
, “
Decision-Based Conceptual Design: Modeling and Navigating Heterogeneous Design Spaces
,”
ASME J. Mech. Des.
,
127
(
1
), pp.
2
11
.
25.
Lewis
,
K. E.
,
Chen
,
W.
, and
Schmidt
,
L. C.
,
2010
,
Decision Making in Engineering Design
,
ASME Press
,
New York, NY
.
26.
Yu
,
R.
, and
Gero
,
J.
,
2015
, “
An Empirical Foundation for Design Patterns in Parametric Design
,”
CAADRIA 2015—20th International Conference on Computer-Aided Architectural Design Research in Asia: Emerging Experiences in the Past, Present and Future of Digital Architecture
,
Daegu, South Korea
,
May
, pp.
551
560
.
27.
Kan
,
J.
, and
Gero
,
J.
,
2009
, “Using the FBS Ontology to Capture Semantic Design Information in Design Protocol Studies,”
About Designing Analysing Design Meetings
,
J.
McDonnell
, and
P.
Lloyd
, eds.,
CRC Press
,
Oxford, UK
, pp.
213
229
.
28.
Raina
,
A.
,
Cagan
,
J.
, and
McComb
,
C.
,
2019
, “
Transferring Design Strategies From Human to Computer and Across Design Problems
,”
ASME J. Mech. Des.
,
141
(
11
), p.
114501
.
29.
Sexton
,
T.
, and
Ren
,
M. Y.
,
2017
, “
Learning an Optimization Algorithm Through Human Design Iterations
,”
ASME J. Mech. Des.
,
139
(
10
), p.
101404
.
30.
Panchal
,
J. H.
,
Sha
,
Z.
, and
Kannan
,
K. N.
,
2017
, “
Understanding Design Decisions Under Competition Using Games With Information Acquisition and a Behavioral Experiment
,”
ASME J. Mech. Des.
,
139
(
9
), p.
091401
.
31.
Chaudhari
,
A. M.
,
Bilionis
,
I.
, and
Panchal
,
J. H.
,
2020
, “
Descriptive Models of Sequential Decisions in Engineering Design: An Experimental Study
,”
ASME J. Mech. Des.
,
142
(
8
), p.
081704
.
32.
Raina
,
A.
,
McComb
,
C.
, and
Cagan
,
J.
,
2019
, “
Learning to Design From Humans: Imitating Human Designers Through Deep Learning
,”
ASME. J. Mech. Des.
,
141
(
11
), p.
111102
.
33.
Levy
,
A.
,
Konidaris
,
G.
,
Platt
,
R.
, and
Saenko
,
K.
,
2019
, “
Learning Multi Level Hierarchies with Hindsight
,”
International Conference on Learning Representation
,
New Orleans, LA
,
May 2019
.
34.
Masson
,
W.
,
Ranchod
,
P.
, and
Konidaris
,
G.
,
2016
, “
Reinforcement Learning With Parameterized Actions
,”
30th AAAI Conference on Artificial Intelligence AAAI 2016
,
Phoenix, AZ
,
February
, pp.
1934
1940
.
35.
Hausknecht
,
M.
, and
Stone
,
P.
,
2016
, “
Deep Reinforcement Learning in Parameterized Action Space
,”
4th International Conference on Learning Representations, ICLR 2016—Conference Track Proceedings.
,
San Juan, Puerto Rico
,
May
.
36.
Delalleau
,
O.
,
Peter
,
M.
,
Alonso
,
E.
, and
Logut
,
A.
,
2020
, “
Discrete and Continuous Action Representation for Practical RL in Video Games
,”
Arxiv
, pp.
1
10
.
37.
Silver
,
D.
,
Schrittwieser
,
J.
,
Simonyan
,
K.
,
Antonoglou
,
I.
,
Huang
,
A.
,
Guez
,
A.
,
Hubert
,
T.
, et al
,
2017
, “
Mastering the Game of Go Without Human Knowledge
,”
Nature
,
550
(
7676
), pp.
354
359
.
38.
Anthony
,
T.
,
Tian
,
Z.
, and
Barber
,
D.
,
2017
, “
Thinking Fast and Slow with Deep Learning and Tree Search
,”
NIPS
,
Long Beach, CA
,
December
, pp.
5361
5371
.
39.
Berner
,
C.
,
Brockman
,
G.
,
Chan
,
B.
,
Cheung
,
V.
,
Dębiak
,
P. P.
,
Dennison
,
C.
,
Farhi
,
D.
, et al
,
2019
, “
Dota 2 with Large Scale Deep Reinforcement Learning
,”
ArXiv
.
40.
Huang
,
Y.
,
2020
, “Deep Q-Networks,”
Deep Reinforcement Learning: Fundamentals, Research, and Applications
,
H.
Dong
,
Z.
Ding
, and
Z.
Shanghang
, eds.,
Springer Nature
,
London, UK
, pp.
135
160
.
41.
Lillicrap
,
T. P.
,
Hunt
,
J. J.
,
Pritzel
,
A.
,
Heess
,
N.
,
Erez
,
T.
,
Tassa
,
Y.
,
Silver
,
D.
, and
Wierstra
,
D.
,
2016
, “
Continuous Control With Deep Reinforcement Learning
,”
4th International Conference on Learning Representations, ICLR 2016—Conference Track Proceedings.
,
San Juan, Puerto Rico
,
May
.
42.
Xiong
,
J.
,
Wang
,
Q.
,
Yang
,
Z.
,
Sun
,
P.
,
Han
,
L.
,
Zheng
,
Y.
,
Fu
,
H.
,
Zhang
,
T.
,
Liu
,
J.
, and
Liu
,
H.
,
2018
, “
Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space
,”
ArXiv
.
43.
Mnih
,
V.
,
Heess
,
N.
,
Graves
,
A.
, and
Kavukcuoglu
,
K.
,
2014
, “
Recurrent Models of Visual Attention
,”
Proceedings of the 27th International Conference on Neural Information Processing Systems—Volume 2
,
Montreal, Canada
,
Dec 2014
.
44.
Ba
,
J.
,
Mnih
,
V.
, and
Kavukcuoglu
,
K.
,
2015
, “
Multi Object Recognition with Visual Attention
,”
ArXiv
,
1
(
1
), pp.
1
10
.
45.
Vaswani
,
A.
,
Shazeer
,
N.
,
Parmar
,
N.
,
Uszkoreit
,
J.
,
Jones
,
L.
,
Gomez
,
A. N.
,
Kaiser
,
Ł
, and
Polosukhin
,
I.
,
2017
, “
Attention is All you Need
,”
Advances in Neural Information Processing Systems
,
Long Beach, CA
,
December
, pp.
5999
6009
.
46.
Liu
,
C.
,
Chen
,
Y.
,
Tai
,
L.
,
Ye
,
H.
,
Liu
,
M.
, and
Shi
,
B. E.
,
2019
, “
A Gaze Model Improves Autonomous Driving
,”
Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications
,
Denver, CO
,
June 2019
.
47.
Deng
,
T.
,
Yan
,
H.
,
Qin
,
L.
,
Ngo
,
T.
, and
Manjunath
,
B. S.
,
2020
, “
How Do Drivers Allocate Their Potential Attention? Driving Fixation Prediction via Convolutional Neural Networks
,”
IEEE Trans. Intell. Transp. Syst
,
21
(
5
), pp.
2146
2154
.
48.
Johnson
,
L.
,
Sullivan
,
B.
,
Hayhoe
,
M.
, and
Ballard
,
D.
,
2014
, “
Predicting Human Visuomotor Behaviour in a Driving Task
,”
Philos. Trans. R. Soc. B Biol. Sci.
,
369
(
1636
), p.
20130044
.
49.
Li
,
Y.
,
Liu
,
M.
, and
Rehg
,
J.
,
2021
, “
In the Eye of the Beholder: Gaze and Actions in First Person Video
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
NA
(
early access
), pp.
1
16
.
50.
Chen
,
Y.
,
Liu
,
C.
,
Tai
,
L.
,
Liu
,
M.
, and
Shi
,
B. E.
,
2019
, “
Gaze Training by Modulated Dropout Improves Imitation Learning
,”
2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Macau, China
,
November
.
51.
Liao
,
H.
,
Dong
,
Q.
,
Dong
,
X.
,
Zhang
,
W.
,
Zhang
,
W.
,
Qi
,
W.
,
Fallon
,
E.
, and
Kara
,
L. B.
,
2020
, “
Attention Routing: Track-Assignment Detailed Routing Using Attention-Based Reinforcement Learning
,”
Proceedings of the ASME Design Engineering Technical Conferences
,
Virtual Conference
,
August
.
52.
Mott
,
A.
,
Zoran
,
D.
,
Chrzanowski
,
M.
,
Wierstra
,
D.
, and
Rezende
,
D. J.
,
2019
, “
Towards Interpretable Reinforcement Learning Using Attention Augmented Agents
,”
Neural Information Processing Systems
,
Vancouver, Canada
,
December
.
53.
Simonyan
,
K.
, and
Zisserman
,
A.
,
2015
, “
Very Deep Convolutional Networks for Large-Scale Image Recognition
,”
3rd International Conference on Learning Representations, ICLR 2015—Conference Track Proceedings
,
San Diego, CA
,
May 2015
, pp.
1
14
.
54.
He
,
K.
,
Zhang
,
X.
,
Ren
,
S.
, and
Sun
,
J.
,
2016
, “
Deep Residual Learning for Image Recognition
,”
IEEE Conference on Computer Vision and Pattern Recognition
,
Las Vegas, NV
,
June
, pp.
770
778
.
55.
Parmar
,
N.
,
Vaswani
,
A.
,
Uszkoreit
,
J.
,
Kaiser
,
Ł
,
Shazeer
,
N.
,
Ku
,
A.
, and
Tran
,
D.
,
2018
, “Image Transformer,” arXiv, abs/1802.0.
56.
Dosovitskiy
,
A.
,
Beyer
,
L.
,
Kolesnikov
,
A.
,
Weissenborn
,
D.
,
Zhai
,
X.
,
Unterthiner
,
T.
,
Dehghani
,
M.
, et al
,
2020
, “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” ArXiv, abs/2010.1.
57.
Qi
,
C. R.
,
Su
,
H.
,
Mo
,
K.
, and
Guibas
,
L. J.
,
2017
, “
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
,”
Proceedings of the 30th IEEE Conference on Computer Vision Pattern Recognition, CVPR 2017
,
Honolulu, HI
,
January
.
58.
Qi
,
C. R.
,
Yi
,
L.
,
Su
,
H.
, and
Guibas
,
L. J.
,
2017
, “
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
,”
Proceedings of the 31st International Conference on Neural Information Processing Systems
,
Long Beach, CA
,
December
.
59.
Qi
,
C. R.
,
Liu
,
W.
,
Wu
,
C.
,
Su
,
H.
, and
Guibas
,
L. J.
,
2017
, “
Frustum PointNets for 3D Object Detection from RGB-D Data
,”
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
,
Salt Lake City, UT
,
June
, pp.
77
85
.
60.
Zaheer
,
M.
,
Kottur
,
S.
,
Ravanbhakhsh
,
S.
,
Póczos
,
B.
,
Salakhutdinov
,
R.
, and
Smola
,
A. J.
,
2017
, “
Deep Sets
,”
Neural Information Processing Systems
,
Long Beach, CA
,
December
.
61.
Zhang
,
W.
,
Yang
,
Z.
,
Jiang
,
H.
,
Nigam
,
S.
,
Yamakawa
,
S.
,
Furuhata
,
T.
,
Shimada
,
K.
, and
Kara
,
L. B.
,
2019
, “
3D Shape Synthesis for Conceptual Design and Optimization Using Variational Autoencoders
,”
Proceedings of the ASME Design Engineering Technical Conferences
,
Anaheim, CA
,
August
.
62.
Puentes
,
L.
,
Raina
,
A.
,
Cagan
,
J.
, and
McComb
,
C.
,
2020
, “
Modeling a Strategic Human Engineering Design Process: Human-Inspired Heuristic Guidance Through Learned Visual Design Agents
,”
Proceedings of the Design Society: Design Conference
,
Cavtat, Croatia
,
May
.
63.
Hayhoe
,
M.
, and
Ballard
,
D.
,
2005
, “
Eye Movements in Natural Behavior
,”
Trends Cognit. Sci
,
9
(
4
), pp.
188
194
.
64.
McComb
,
C.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2018
, “
Data on the Design of Truss Structures by Teams of Engineering Students
,”
Data Br.
,
18
(
1
), pp.
160
163
.
65.
McComb
,
C.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2017
, “Utilizing Markov Chains to Understand Operation Sequencing in Design Tasks,”
Design Computing and Cognition ’16
, 1, Vol.
1
,
J.
Gero
, ed.,
Springer
,
London, UK
, pp.
401
418
.
66.
Springenberg
,
J. T.
,
Dosovitskiy
,
A.
,
Brox
,
T.
, and
Riedmiller
,
M. A.
,
2015
, “
Striving for Simplicity: The All Convolutional Net
,”
CoRR
.
67.
Hastie
,
T.
,
Tibshirani
,
R.
, and
Friedman
,
J.
,
2009
,
The Elements of Statistical Learning—Data Mining, Inference, and Prediction
,
Springer
,
London, UK
.
68.
Daumé
,
H.
,
Langford
,
J.
, and
Marcu
,
D.
,
2009
, “
Search-Based Structured Prediction
,”
Mach. Learn.
,
75
(
3
), pp.
297
325
.
69.
Ross
,
S.
, and
Bagnell
,
D.
,
2010
, “
Efficient Reductions for Imitation Learning
,”
Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics
,
Chia Laguna Resort, Sardinia, Italy
,
May
, PMLR, pp.
661
668
.
70.
J
,
Ho
, and
Ermon
,
S.
,
2016
, “
Generative Adversarial Imitation Learning
,”
Neural Information Processing Systems
,
Barcelona, Spain
,
December
, pp.
4572
4580
.
You do not currently have access to this content.