Abstract

Large Language Models (LLMs) have emerged as pivotal technology in the evolving world. Their significance in design lies in their transformative potential to support engineers and collaborate with design teams throughout the design process. However, it is not known whether LLMs can emulate the cognitive and social attributes which are known to be important during design, such as cognitive style. This research evaluates the efficacy of LLMs to emulate aspects of Kirton’s Adaption–Innovation theory, which characterizes individual preferences in problem-solving. Specifically, we use LLMs to generate solutions for three design problems using two different cognitive style prompts (adaptively framed and innovatively framed). Solutions are evaluated with respect to feasibility and paradigm relatedness, which are known to have discriminative value in other studies of cognitive style. We found that solutions generated using the adaptive prompt tend to display higher feasibility and are paradigm-preserving, while solutions generated using the innovative prompts were more paradigm-modifying. This aligns with prior work and expectations for design behavior based on Kirton's Adaption–Innovation theory. Ultimately, these results demonstrate that LLMs can be prompted to accurately emulate cognitive style.

References

1.
Radwan
,
N.
,
Menold
,
J.
,
McTernan
,
J.
,
Jablokow
,
K.
, and
McComb
,
C.
,
2022
, “
Towards Characterizing Cognitive Style Coping Behavior in Engineering Design
,”
Proceedings of the Volume 6: 34th International Conference on Design Theory and Methodology (DTM)
,
St. Louis, MO
,
Aug. 14–17
,
American Society of Mechanical Engineers
, p.
V006T06A035
.
2.
Badke-Schaub
,
P.
,
Goldschmidt
,
G.
, and
Meijer
,
M.
,
2010
, “
How Does Cognitive Conflict in Design Teams Support the Development of Creative Ideas?
,”
Creat. Innov. Manag.
,
19
(
2
), pp.
119
133
.
3.
Buffinton
,
K. W.
, and
Jablokow
,
K. W.
,
2002
, “
Characterization of Project Team Dynamics for Engineering and Management Students Based on Cognitive Style
,”
ASEE Annual Conference Proceedings
,
Montreal, Canada
,
June 16–19
, pp.
10031
10048
.
4.
Goldsmith
,
R. E.
, and
Kerr
,
J. R.
,
1991
, “
Entrepreneurship and Adaption-Innovation Theory
,”
Technovation
,
11
(
6
), pp.
373
382
.
5.
Kirton
,
M. J.
,
2003
,
Adaption-Innovation: In the Context of Diversity and Change
,
Routledge
,
London, New York
.
6.
Erbe
,
N. D.
,
2014
,
Approaches to Managing Organizational Diversity and Innovation
,
Business Science Reference, an Imprint of IGI Global
,
Hershey, PA
.
7.
Kirton
,
M. J.
,
1977
,
Manual of the Kirton Adaption-Innovation Inventory
,
National Foundation for Educational Research
,
London
.
8.
Hammerschmidt
,
P.
,
1996
, “
The Kirton Adaption Innovation Inventory Find Group Problem Solving Success Rates
,”
J. Creat. Behav.
,
30
(
1
), pp.
61
74
.
9.
Heininger
,
K.
,
Chen
,
H.-E.
,
Jablokow
,
K.
, and
Miller
,
S. R.
,
2018
, “
How Engineering Design Students' Creative Preferences and Cognitive Styles Impact Their Concept Generation and Screening
,” Proceedings of the
Volume 7: 30th International Conference on Design Theory and Methodology
,
Quebec City, Quebec, Canada
,
Aug. 2018
,
American Society of Mechanical Engineers
.
10.
Purwar
,
A.
, and
Chakraborty
,
N.
,
2023
, “
Deep Learning-Driven Design of Robot Mechanisms
,”
ASME J. Comput. Inf. Sci. Eng.
,
23
(
6
), p.
060811
.
11.
Jablokow
,
K.
,
Teerlink
,
W.
,
Yilmaz
,
S.
,
Daly
,
S.
, and
Silk
,
E.
,
2015
, “
The Impact of Teaming and Cognitive Style on Student Perceptions of Design Ideation Outcomes
,”
Proceedings of the 2015 ASEE Annual Conference and Exposition Proceedings
,
Seattle
,
WA, June 14–17
, p.
26.1548.1
26.1548.22
.
12.
Bai
,
L.
,
Liu
,
X.
, and
Su
,
J.
,
2023
, “
ChatGPT: The Cognitive Effects on Learning and Memory
,”
Brain-X
,
1
(
3
), p.
e30
.
13.
Linsey
,
J. S.
, and
Viswanathan
,
V. K.
,
2014
, “Overcoming Cognitive Challenges in Bioinspired Design and Analogy,”
Biologically Inspired Design: Computational Methods and Tools
,
A. K.
Goel
,
D. A.
McAdams
, and
R. B.
Stone
, eds.,
Springer
,
London
, pp.
221
244
.
14.
Mudd
,
S.
,
1995
, “
Kirton Adaption-Innovation Theory: Organizational Implications
,”
Technovation
,
15
(
3
), pp.
165
175
.
15.
Stoyanov
,
S.
,
Jablokow
,
K.
,
Rosas
,
S. R.
,
Wopereis
,
I. G. J. H.
, and
Kirschner
,
P. A.
,
2017
, “
Concept Mapping – An Effective Method for Identifying Diversity and Congruity in Cognitive Style
,”
Eval. Program Plann.
,
60
, pp.
238
244
.
16.
Kirton
,
M. J.
,
1980
, “
Adaptors and Innovators in Organizations
,”
Hum. Relat.
,
33
(
4
), pp.
213
224
.
17.
Chilton
,
M. A.
, and
Bloodgood
,
J. M.
,
2010
, “
Adaption-Innovation Theory and Knowledge Use in Organizations
,”
Manage. Decis.
,
48
(
8
), pp.
1159
1180
.
18.
Kirton
,
M. J.
,
1976
, “
Adaptors and Innovators: A Description and Measure
,”
J. Appl. Psychol.
,
61
(
5
), pp.
622
629
.
19.
Markowitz
,
D. M.
,
2024
, “
Can Generative AI Infer Thinking Style From Language? Evaluating the Utility of AI as a Psychological Text Analysis Tool
,”
Behav. Res.
,
56
(
4
), pp.
3548
3559
.
20.
Aritzeta
,
A.
,
Senior
,
B.
, and
Swailes
,
S.
,
2005
, “
Team Role Preference and Cognitive Styles A Convergent Validity Study
,”
Small Group Res.
,
36
(
4
), pp.
404
436
.
21.
Chowdhary
,
K. R.
,
2020
, “
Natural Language Processing
,”
Fundamentals of Artificial Intelligence
, pp.
603
649
.
22.
Zhang
,
C.
,
Lai
,
Y.
,
Feng
,
Y.
, and
Zhao
,
D.
,
2021
, “
A Review of Deep Learning in Question Answering Over Knowledge Bases
,”
AI Open
,
2
, pp.
205
215
.
23.
Radwan
,
N.
, and
McComb
,
C.
,
2022
, “
A Computational Study of Design Team Robustness Through the Lens of Cognitive Style
,”
ASME J. Mech. Des.
,
144
(
10
), p.
104502
.
24.
Tan
,
Z.
,
Wang
,
S.
,
Yang
,
Z.
,
Chen
,
G.
,
Huang
,
X.
,
Sun
,
M.
, and
Liu
,
Y.
,
2020
, “
Neural machine translation: A review of methods, resources, and tools
,”
AI Open
,
1
, p.
5
.
25.
Chang
,
Y.
,
Wang
,
X.
,
Wang
,
J.
,
Wu
,
Y.
,
Yang
,
L.
,
Zhu
,
K.
,
Chen
,
H.
, et al
,
2024
, “
A Survey on Evaluation of Large Language Models
,”
ACM Trans. Intell. Syst. Technol.
,
15
(
3
), pp.
1
45
.
26.
Shaki
,
J.
,
Kraus
,
S.
, and
Wooldridge
,
M.
,
2023
, Cognitive Effects in Large Language Models. 10.3233/FAIA230505.
27.
Bosley
,
M.
,
Jacobs-Harukawa
,
M.
,
Licht
,
H.
, and
Hoyle
,
A.
,
2023
, “Do We Still Need BERT in the Age of GPT? Comparing the Benefits of Domain-Adaptation and In-Context-Learning Approaches to Using LLMs for Political Science Research.”
28.
Zhu
,
Q.
, and
Luo
,
J.
,
2023
, “
Generative Transformers for Design Concept Generation
,”
ASME J. Comput. Inf. Sci. Eng.
,
23
(
4
), p.
041003
.
29.
Ma
,
K.
,
Grandi
,
D.
,
McComb
,
C.
, and
Goucher-Lambert
,
K.
,
2023
, “
Conceptual Design Generation Using Large Language Models
,”
Proceedings of the Volume 6: 35th International Conference on Design Theory and Methodology (DTM)
,
Boston, MA, USA
,
Aug. 2023
,
American Society of Mechanical Engineers
.
30.
Zhu
,
Q.
, and
Luo
,
J.
,
2022
, “
Generative Pre-Trained Transformer for Design Concept Generation: An Exploration
,”
Proc. Des. Soc.
,
2
, pp.
1825
1834
.
31.
Qiu
,
Y.
, and
Jin
,
Y.
,
2022
, “
Engineering Document Summarization: A Bidirectional Language Model-Based Approach
,”
ASME J. Comput. Inf. Sci. Eng.
,
22
(
6
), p.
061004
.
32.
Makatura
,
L.
,
Foshey
,
M.
,
Wang
,
B.
,
Hähnlein
,
F.
,
Ma
,
P.
,
Deng
,
B.
,
Tjandrasuwita
,
M.
, et al.
,
2024
, “
How Can Large Language Models Help Humans in Design And Manufacturing? Part 2: Synthesizing an End-To-End LLM-Enabled Design and Manufacturing Workflow
,”
Harvard Data Science Review
(Special Issue
5
).
33.
Rastogi
,
C.
,
Tulio Ribeiro
,
M.
,
King
,
N.
,
Nori
,
H.
, and
Amershi
,
S.
,
2023
, “
Supporting Human-AI Collaboration in Auditing LLMs With LLMs
,”
Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, in AIES ‘23
,
Montreal, Canada
,
Aug. 2023
, ACM, pp.
913
926
.
34.
Göpfert
,
J.
,
Weinand
,
J. M.
,
Kuckertz
,
P.
, and
Stolten
,
D.
,
2024
, “
Opportunities for Large Language Models and Discourse in Engineering Design
,”
Energy AI
,
17
, p.
100383
.
35.
Dean
,
D. L.
,
Hender
,
J.
,
Rodgers
,
T.
, and
Santanen
,
E.
,
2006
, “
Identifying Good Ideas: Constructs and Scales for Idea Evaluation
,”
J. Assoc. Inf. Syst.
,
7
(
10
), pp.
646
699
.
36.
Menold
,
J.
, and
Jablokow
,
K.
,
2019
, “
Exploring the Effects of Cognitive Style Diversity and Self-Efficacy Beliefs on Final Design Attributes in Student Design Teams
,”
Des. Stud.
,
60
, pp.
10
102
.
37.
Chiarello
,
F.
,
Barandoni
,
S.
,
Majda Škec
,
M.
, and
Fantoni
,
G.
,
2024
, “
Generative Large Language Models in Engineering Design: Opportunities and Challenges
,”
Proc. Des. Soc.
,
4
, pp.
1959
1968
.
38.
Lapp
,
S.
,
Jablokow
,
K.
, and
McComb
,
C.
,
2019
, “
KABOOM: an Agent-Based Model for Simulating Cognitive Style in Team Problem Solving
,”
Des. Sci.
,
5
, p.
e13
.
39.
Loudon
,
G.
,
Hewett
,
B.
,
Dale
,
J.
,
Barham
,
G.
,
Gill
,
S.
, and
Johnson
,
P.
,
2010
, “
The Traditional Design Process Versus a New Design Methodology
,”
IFIP International Federation for Information Processing
.
40.
Tian
,
Y.
,
Liu
,
A.
,
Dai
,
Y.
,
Nagato
,
K.
, and
Nakao
,
M.
,
2024
, “
Systematic Synthesis of Design Prompts for Large Language Models in Conceptual Design
,”
CIRP Ann.
,
73
(
1
), pp.
85
88
.
41.
Wasi
,
A. T.
,
2024
, “CogErgLLM: Exploring Large Language Model Systems Design Perspective Using Cognitive Ergonomics.” https://arxiv.org/abs/2407.02885
42.
Jadhav
,
Y.
, and
Farimani
,
A. B.
,
2024
, “Large Language Model Agent as a Mechanical Designer.” https://arxiv.org/abs/2404.17525
43.
Goucher-Lambert
,
K.
, and
Cagan
,
J.
,
2019
, “
Crowdsourcing Inspiration: Using Crowd Generated Inspirational Stimuli to Support Designer Ideation
,”
Des. Stud.
,
61
, pp.
1
29
.
44.
Brown
,
T. B.
,
Mann
,
B.
,
Ryder
,
N.
,
Subbiah
,
M.
,
Kaplan
,
J.
,
Dhariwal
,
P.
,
Neelakantan
,
A.
, et al
,
2020
, “Language Models Are Few-Shot Learners.”
45.
Silk
,
E.
,
Daly
,
S.
,
Jablokow
,
K.
,
Yilmaz
,
S.
,
Rechkemmer
,
A.
, and
Wegner
,
J.
,
2016
, “
Using Paradigm-Relatedness to Measure Design Ideation Shifts
,”
2016 ASEE Annual Conference & Exposition Proceedings
,
New Orleans, LA
,
June 26–29
,
ASEE Conferences
.
46.
Henderson
,
D.
,
Booth
,
T.
,
Jablokow
,
K.
, and
Sonalkar
,
N.
,
2020
, “
Best Fits and Dark Horses: Can Design Teams Tell the Difference?
Proceedings of the Volume 3: 17th International Conference on Design Education (DEC)
,
New Orleans, LA
,
Aug. 17–19
.
47.
Cole
,
C.
,
Marhefka
,
J.
,
Jablokow
,
K.
,
Mohammed
,
S.
,
Ritter
,
S.
, and
Miller
,
S.
,
2023
, “
An Exploration of the Relationships Between Cognitive Style, Psychological Safety, and the Paradigm-Relatedness of Design Solutions in Engineering Design Teams in Education
,”
ASME J. Mech. Des.
,
145
(
7
), p.
072301
.
48.
Silk
,
E. M.
,
Daly
,
S. R.
,
Jablokow
,
K. W.
, and
McKilligan
,
S.
,
2019
, “
Incremental to Radical Ideas: Paradigm-Relatedness Metrics for Investigating Ideation Creativity and Diversity
,”
Int. J. Des. Creat. Innov.
,
7
(
1–2
), pp.
30
49
.
49.
Garfield
,
M. J.
,
Taylor
,
N. J.
,
Dennis
,
A. R.
, and
Satzinger
,
J. W.
,
2001
, “
Research Report: Modifying Paradigms – Individual Differences, Creativity Techniques, and Exposure to Ideas in Group Idea Generation
,”
Inf. Syst. Res.
,
12
(
3
), pp.
322
333
.
50.
McKight
,
P. E.
, and
Najab
,
J.
,
2010
, “
Kruskal‐Wallis Test
,”
The Corsini Encyclopedia of Psychology
, p.
1
.
You do not currently have access to this content.