Abstract

As artificial intelligence (AI) assistance tools become more ubiquitous in engineering design, it becomes increasingly necessary to understand the influence of AI assistance on the design process and design effectiveness. Previous work has shown the advantages of incorporating AI design agents to assist human designers. However, the influence of AI assistance on the behavior of designers during the design process is still unknown. This study examines the differences in participants’ design process and effectiveness with and without AI assistance during a complex drone design task using the HyForm design research platform. Data collected from this study are analyzed to assess the design process and effectiveness using quantitative methods, such as hidden Markov models and network analysis. The results indicate that AI assistance is most beneficial when addressing moderately complex objectives but exhibits a reduced advantage in addressing highly complex objectives. During the design process, the individual designers working with AI assistance employ a relatively explorative search strategy, while the individual designers working without AI assistance devote more effort to parameter design.

References

1.
Song
,
B.
,
Soria Zurita
,
N. F.
,
Zhang
,
G.
,
Stump
,
G.
,
Balon
,
C.
,
Miller
,
S. W.
,
Yukish
,
M.
,
Cagan
,
J.
, and
McComb
,
C.
,
2020
, “
Toward Hybrid Teams: A Platform to Understand Human–Computer Collaboration During the Design of Complex Engineered Systems
,”
Proceedings of the Design Society: Design Conference
,
Cavtat, Croatia
,
Oct. 26–29
, Vol. 1, pp.
1551
1560
.
2.
Soria Zurita
,
N. F.
, and
Tumer
,
I. Y.
,
2017
, “
A Survey: Towards Understanding Emergent Behavior in Complex Engineered Systems
,”
Proceedings of the ASME Design Engineering Technical Conference
,
Cleveland, OH
,
Aug. 6–9
, p. V007T06A015.
3.
Koch
,
J.
, and
Paris-Saclay
,
I.
,
2017
, “
Design Implications for Designing With a Collaborative AI
,”
AAAI Spring Symposium Series
,
Palo Alto, CA
,
Mar. 27–29
, pp.
415
418
.
4.
Camburn
,
B.
,
Arlitt
,
R.
,
Anderson
,
D.
,
Sanaei
,
R.
,
Raviselam
,
S.
,
Jensen
,
D.
, and
Wood
,
K. L.
,
2020
, “
Computer-Aided Mind Map Generation Via Crowdsourcing and Machine Learning
,”
Res. Eng. Des.
,
31
(
4
), pp.
383
409
.
5.
Camburn
,
B.
,
He
,
Y.
,
Raviselvam
,
S.
,
Luo
,
J.
, and
Wood
,
K.
,
2020
, “
Machine Learning-Based Design Concept Evaluation
,”
ASME J. Mech. Des.
,
142
(
3
), p.
031113
.
6.
Dering
,
M. L.
,
Tucker
,
C. S.
, and
Kumara
,
S.
,
2018
, “
An Unsupervised Machine Learning Approach to Assessing Designer Performance During Physical Prototyping
,”
ASME J. Comput. Inf. Sci. Eng.
,
18
(
1
), p.
011002
.
7.
Williams
,
G.
,
Meisel
,
N. A.
,
Simpson
,
T. W.
, and
McComb
,
C.
,
2019
, “
Design Repository Effectiveness for 3D Convolutional Neural Networks: Application to Additive Manufacturing
,”
ASME J. Mech. Des.
,
141
(
11
), p.
111701
.
8.
Daugherty
,
P. R.
, and
Wilson
,
H. J.
,
2018
,
Human + Machine: Reimagining Work in the Age of AI
,
Harvard Business Press
,
Boston, MA
.
9.
Wilson
,
H. J.
, and
Daugherty
,
P. R.
,
2018
, “
Collaborative Intelligence: Humans and AI Are Joining Forces
,”
Harvard Bus. Rev.
,
96
(
4
), pp.
114
123
.
10.
Zhang
,
G.
,
Raina
,
A.
,
Cagan
,
J.
, and
McComb
,
C.
,
2021
, “
A Cautionary Tale About the Impact of AI on Human Design Teams
,”
Des. Stud.
,
72
, p.
100990
.
11.
Rao
,
S. S.
,
Nahm
,
A.
,
Shi
,
Z.
,
Deng
,
X.
, and
Syamil
,
A.
,
1999
, “
Artificial Intelligence and Expert Systems Applications in New Product Development—A Survey
,”
J. Intell. Manuf.
,
10
(
3
), pp.
231
244
.
12.
Raina
,
A.
,
Cagan
,
J.
, and
McComb
,
C.
,
2019
, “
Transferring Design Strategies From Human to Computer and Across Design Problems
,”
ASME J. Mech. Des.
,
141
(
11
), p.
114501
.
13.
Amabile
,
T. M.
,
1998
,
How to Kill Creativity
,
Harvard Business School Publishing
,
Boston, MA
.
14.
Sekiguchi
,
K.
, and
Hori
,
K.
,
2020
, “
Organic and Dynamic Tool for Use With Knowledge Base of AI Ethics for Promoting Engineers’ Practice of Ethical AI Design
,”
AI Soc.
,
35
(
1
), pp.
51
71
.
15.
Selin
,
J.
, and
Rossi
,
M.
,
2018
, “
The Functional Design Method for Buildings (FDM) With Gamification of Information Models and AI Help to Design Safer Buildings
,”
2018 Federated Conference on Computer Science and Information Systems (FedCSIS)
,
Poznan, Poland
,
Sept. 9–12
, pp.
907
911
.
16.
Li
,
B. H.
,
Hou
,
B. C.
,
Yu
,
W. T.
,
Lu
,
X. B.
, and
Yang
,
C. W.
,
2017
, “
Applications of Artificial Intelligence in Intelligent Manufacturing: A Review
,”
Front. Inf. Technol. Electron. Eng.
,
18
(
1
), pp.
86
96
.
17.
Fischer
,
G.
, and
Nakakoji
,
K.
,
1992
, “
Beyond the Macho Approach of Artificial Intelligence: Empower Human Designers—Do Not Replace Them
,”
Knowl. Based Syst.
,
5
(
1
), pp.
15
30
.
18.
Boden
,
M. A.
,
1998
, “
Creativity and Artificial Intelligence
,”
Artif. Intell.
,
103
(
1–2
), pp.
347
356
.
19.
Chan
,
F. T. S.
,
Jiang
,
B.
, and
Tang
,
N. K. H.
,
2000
, “
Development of Intelligent Decision Support Tools to Aid the Design of Flexible Manufacturing Systems
,”
Int. J. Prod. Econ.
,
65
(
1
), pp.
73
84
.
20.
Karan
,
E.
, and
Asadi
,
S.
,
2019
, “
Intelligent Designer: A Computational Approach to Automating Design of Windows in Buildings
,”
Autom. Constr.
,
102
, pp.
160
169
.
21.
Guzdial
,
M.
,
Liao
,
N.
,
Chen
,
J.
,
Chen
,
S.-Y.
,
Shah
,
S.
,
Shah
,
V.
,
Reno
,
J.
,
Smith
,
G.
, and
Riedl
,
M. O.
,
2019
, “
Friend, Collaborator, Student, Manager: How Design of an Ai-Driven Game Level Editor Affects Creators
,”
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
,
Glasgow, UK
,
May 4–9
.
22.
Feldman
,
S.
, “
Co-Creation: Human and AI Collaboration in Creative Expression
,”
Electronic Visualisation and the Arts
,
London, UK
,
July 11–13
, pp.
422
429
.
23.
Chattaraman
,
V.
,
Kwon
,
W. S.
,
Gilbert
,
J. E.
, and
Ross
,
K.
,
2019
, “
Should AI-Based, Conversational Digital Assistants Employ Social- or Task-Oriented Interaction Style? A Task-Competency and Reciprocity Perspective for Older Adults
,”
Comput. Hum. Behav.
,
90
, pp.
315
330
.
24.
Morley
,
J. E.
,
Morris
,
J. C.
,
Berg-Weger
,
M.
,
Borson
,
S.
,
Carpenter
,
B. D.
,
del Campo
,
N.
,
Dubois
,
B.
,
Fargo
,
K.
,
Fitten
,
L. J.
,
Flaherty
,
J. H.
,
Ganguli
,
M.
,
Grossberg
,
G. T.
,
Malmstrom
,
T. K.
,
Petersen
,
R. D.
,
Rodriguez
,
C.
,
Saykin
,
A. J.
,
Scheltens
,
P.
,
Tangalos
,
E. G.
,
Verghese
,
J.
,
Wilcock
,
G.
,
Winblad
,
B.
,
Woo
,
J.
, and
Vellas
,
B.
,
2015
, “
Brain Health: The Importance of Recognizing Cognitive Impairment: An IAGG Consensus Conference
,”
J. Am. Med. Dir. Assoc.
,
16
(
9
), pp.
731
739
.
25.
Sweller
,
J.
,
2016
,
Evolutionary Perspectives on Child Development and Education
,
D. C.
Geary
, and
D. B.
Berch
, eds.,
Springer International Publishing
,
Cham, Switzerland
, pp.
291
306
.
26.
Zhou
,
J.
,
Yu
,
K.
,
Chen
,
F.
,
Wang
,
Y.
, and
Arshad
,
S. Z.
,
2018
,
The Handbook of Multimodal-Multisensor Interfaces: Foundations, User Modeling, and Common Modality Combinations—Volume 2
,
S.
Oviatt
,
S.
Bjorn
,
P. R.
Cohen
,
D.
Sonntag
,
G.
Potamianos
, and
A.
Kruger
, eds.,
Association for Computing Machinery and Morgan & Claypool
,
New York City
, pp.
287
329
.
27.
Dykstra
,
J.
, and
Paul
,
C. L.
,
2018
, “
Cyber Operations Stress Survey (COSS): Studying Fatigue, Frustration, and Cognitive Workload in Cybersecurity Operations
,”
11th USENIX Workshop on Cyber Security Experimentation and Test (CSET)
,
Baltimore MD
,
August 13
.
28.
Fallahi
,
M.
,
Motamedzade
,
M.
,
Heidarimoghadam
,
R.
,
Soltanian
,
A. R.
, and
Miyake
,
S.
,
2016
, “
Effects of Mental Workload on Physiological and Subjective Responses During Traffic Density Monitoring: A Field Study
,”
Appl. Ergon.
,
52
, pp.
95
103
.
29.
Nolte
,
H.
, and
McComb
,
C.
,
2020
, “
Identifying Stress Signatures Across the Engineering Design Process: Perceived Stress During Concept Generation, Concept Selection, and Prototyping
,”
Proceedings of the Design Society: Design Conference
,
Cavtat, Croatia
,
October 26-29
.
30.
Nolte
,
H.
, and
McComb
,
C.
,
2021
, “
The Cognitive Experience of Engineering Design: An Examination of First-Year Student Stress Across Principal Activities of the Engineering Design Process
,”
Des. Sci.
,
7
, p.
e3
.
31.
Lake
,
B. M.
,
Ullman
,
T. D.
,
Tenenbaum
,
J. B.
, and
Gershman
,
S. J.
,
2017
, “
Building Machines That Learn and Think Like People
,”
Behav. Brain Sci.
,
40
, p.
E253
.
32.
Dellermann
,
D.
,
Ebel
,
P.
,
Söllner
,
M.
, and
Leimeister
,
J. M.
,
2019
, “
Hybrid Intelligence
,”
Bus. Inf. Syst. Eng.
,
61
(
5
), pp.
637
643
.
33.
VanLehn
,
K.
,
Burkhardt
,
H.
,
Cheema
,
S.
,
Kang
,
S.
,
Pead
,
D.
,
Schoenfeld
,
A.
, and
Wetzel
,
J.
,
2019
, “
Can an Orchestration System Increase Collaborative, Productive Struggle in Teaching-by-Eliciting Classrooms?
Interact. Learn. Environ.
34.
Liew
,
C.
,
2018
, “
The Future of Radiology Augmented With Artificial Intelligence: A Strategy for Success
,”
Eur. J. Radiol.
,
102
, pp.
152
156
.
35.
Shah
,
J. J.
,
Vargas-Hernandez
,
N.
, and
Smith
,
S. M.
,
2003
, “
Metrics for Measuring Ideation Effectiveness
,”
Des. Stud.
,
24
(
2
), pp.
111
134
.
36.
Nguyen
,
L.
, and
Shanks
,
G.
,
2009
, “
A Framework for Understanding Creativity in Requirements Engineering
,”
Inf. Softw. Technol.
,
51
(
3
), pp.
655
662
.
37.
Bashir
,
H. A.
, and
Thomson
,
V.
,
1999
, “
Metrics for Design Projects: A Review
,”
Des. Stud.
,
20
(
3
), pp.
263
277
.
38.
Bashir
,
H. A.
, and
Thomson
,
V.
,
2001
, “
Models for Estimating Design Effort and Time
,”
Des. Stud.
,
22
(
2
), pp.
141
155
.
39.
Song
,
B.
,
Gyory
,
J. T.
,
Soria Zurita
,
N. F.
,
Stump
,
G. M.
,
Martin
,
J. D.
,
Miller
,
S. W.
,
Balon
,
C. M.
,
Yukish
,
M. A.
,
McComb
,
C.
, and
Cagan
,
J.
,
2021
, “
Decoding the Agility of Human-Artificial Intelligence Hybrid Teams in Complex Problem Solving
,”
Des. Stud.
40.
Stump
,
G. M.
,
Miller
,
S. W.
,
Yukish
,
M. A.
,
Simpson
,
T. W.
, and
Tucker
,
C.
,
2019
, “
Spatial Grammar-Based Recurrent Neural Network for Design Form and Behavior Optimization
,”
ASME J. Mech. Des.
,
141
(
12
), p.
124501
.
41.
Hart
,
S. G.
, and
Staveland
,
L. E.
,
1988
, “
Development of NASA-TLX (Task Load Index): Results of Emperical and Theoretical Research
,”
Adv. Psychol.
,
52
(
C
), pp.
139
183
.
42.
Hart
,
S. G.
,
2006
, “
Nasa-Task Load Index (NASA-TLX); 20 Years Later
,”
Proceedings of the Human Factors and Ergonomics Society Annual Meeting
,
Los Angeles, CA
,
October
.
43.
Zhang
,
G.
,
Soria Zurita
,
N. F.
,
Stump
,
G.
,
Song
,
B.
,
Cagan
,
J.
, and
McComb
,
C.
,
2021
, “
Data on the Design and Operation of Drones by Both Individuals and Teams
,”
Data Br.
,
36
, p.
107008
.
44.
McComb
,
C.
,
Cagan
,
J.
, and
Kotovsky
,
K.
,
2017
, “
Mining Process Heuristics From Designer Action Data Via Hidden Markov Models
,”
ASME J. Mech. Des.
,
139
(
11
), p.
111412
.
45.
Baum
,
L. E.
,
Petrie
,
T.
,
Soules
,
G.
, and
Weiss
,
N.
,
1970
, “
A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains
,”
Ann. Math. Stat.
,
41
(
1
), pp.
164
171
.
46.
Durbin
,
R.
,
Eddy
,
S. R.
,
Krogh
,
A.
, and
Mitchison
,
G.
,
1998
,
Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids
,
Cambridge University Press
,
Cambridge, UK
.
47.
Mehta
,
P.
,
Malviya
,
M.
,
McComb
,
C.
,
Manogharan
,
G.
, and
Berdanier
,
C. G. P.
,
2020
, “
Mining Design Heuristics for Additive Manufacturing Via Eye-Tracking Methods and Hidden Markov Modeling
,”
ASME J. Mech. Des.
,
142
(
12
), p.
124502
.
48.
Maier
,
T.
,
Zurita
,
N. F. S.
,
Starkey
,
E.
,
Spillane
,
D.
,
Menold
,
J.
, and
McComb
,
C.
,
2020
, “
Analyzing the Characteristics of Cognitive-Assistant-Facilitated Ideation Groups
,”
Proceedings of the ASME Design Engineering Technical Conference
,
Online, virtual
,
Aug. 17–19
, p.
V008T08A046
.
49.
Goucher-Lambert
,
K.
, and
McComb
,
C.
,
2019
, “
Using Hidden Markov Models to Uncover Underlying States in Neuroimaging Data for a Design Ideation Task
,”
Proceedings of the International Conference on Engineering Design, ICED
,
Delft, The Netherlands
,
August 5-8
.
50.
Mahan
,
T.
,
Meisel
,
N.
,
McComb
,
C.
, and
Menold
,
J.
,
2019
, “
Pulling at the Digital Thread: Exploring the Tolerance Stack Up Between Automatic Procedures and Expert Strategies in Scan to Print Processes
,”
ASME J. Mech. Des.
,
141
(
2
), p.
021701
.
51.
Di Luca
,
M.
,
Mintchev
,
S.
,
Su
,
Y.
,
Shaw
,
E.
, and
Breuer
,
K.
,
2020
, “
A Bioinspired Separated Flow Wing Provides Turbulence Resilience and Aerodynamic Efficiency for Miniature Drones
,”
Sci. Robot.
,
5
(
38
), p.
8533
.
52.
Brachten
,
F.
,
Brünker
,
F.
,
Frick
,
N. R. J.
,
Ross
,
B.
, and
Stieglitz
,
S.
,
2020
, “
On the Ability of Virtual Agents to Decrease Cognitive Load: An Experimental Study
,”
Inf. Syst. e-Bus. Manage.
,
18
(
2
), pp.
187
207
.
53.
de Melo
,
C. M.
,
Kim
,
K.
,
Norouzi
,
N.
,
Bruder
,
G.
, and
Welch
,
G.
,
2020
, “
Reducing Cognitive Load and Improving Warfighter Problem Solving With Intelligent Virtual Assistants
,”
Front. Psychol.
,
11
(
554706
), pp.
1
12
.
54.
Maier
,
T.
,
Donghia
,
V.
,
Chen
,
C.
,
Menold
,
J.
, and
McComb
,
C.
,
2019
, “
Assessing the Impact of Cognitive Assistants on Mental Workload in Simple Tasks
,”
Proceedings of the ASME Design Engineering Technical Conference
,
Anaheim, CA
,
Aug. 18–21
, p.
V007T06A021
.
55.
Maier
,
T.
,
Abdullah
,
S.
,
McComb
,
C.
, and
Menold
,
J.
,
2021
, “
A Query Conundrum: The Mental Challenges of Using a Cognitive Assistant
,”
SN Comput. Sci.
,
2
(
3
), p.
194
.
You do not currently have access to this content.