Using electroencephalography (EEG) to predict design outcomes could be used in many applications as it facilitates the correlation of engagement and cognitive workload with ideation effectiveness. It also establishes a basis for the connection between EEG measurements and common constructs in engineering design research. In this paper, we propose a support vector machine (SVM)-based prediction model for design outcomes using EEG metrics and some demographic factors as predictors. We trained and validated the model with more than 100 concepts, and then evaluated the relationship between EEG data and concept-level measures of novelty, quality, and elaboration. The results characterize the combination of engagement and workload that is correlated with good design outcomes. Findings also suggest that EEG technologies can be used to partially replace or augment traditional ideation metrics and to improve the efficacy of ideation research.

References

References
1.
Dinar
,
M.
,
Shah
,
J. J.
,
Cagan
,
J.
,
Leifer
,
L.
,
Linsey
,
J.
,
Smith
,
S. M.
, and
Hernandez
,
N. V.
,
2015
, “
Empirical Studies of Designer Thinking: Past, Present, and Future
,”
ASME J. Mech. Des.
,
137
(
2
), p.
021101
.
2.
Proctor
,
R. W.
, and
Van Zandt
,
T.
,
2008
,
Human Factors in Simple and Complex Systems
,
2nd ed.
,
CRC Press
, Boca Raton, FL.
3.
Runco
,
M. A.
, and
Chand
,
I.
,
1995
, “
Cognition and Creativity
,”
Educ. Psychol. Rev.
,
7
(
3
), pp.
243
267
.
4.
Carroll
,
E. A.
,
2013
, “
Quantifying the Personal Creative Experience: Evaluation of Digital Creativity Support Tools Using Self-Report and Physiological Responses
,”
Ph.D. thesis
,
University of North Carolina at Charlotte, Charlotte, NC
.
5.
Amabile
,
T. M.
,
Conti
,
R.
,
Coon
,
H.
,
Lazenby
,
J.
, and
Herron
,
M.
,
1996
, “
Assessing the Work Environment for Creativity
,”
Acad. Manage. J.
,
39
(
5
), pp.
1154
1184
.
6.
Besemer
,
S. P.
,
1998
, “
Creative Product Analysis Matrix: Testing the Model Structure and a Comparison Among Products: Three Novel Chairs
,”
Creativity Res. J.
,
11
(
4
), pp.
333
346
.
7.
Gero
,
J. S.
, and
Mc Neill
,
T.
,
1998
, “
An Approach to the Analysis of Design Protocols
,”
Des. Stud.
,
19
(
1
), pp.
21
61
.
8.
Cai
,
H.
,
Do
,
E. Y. L.
, and
Zimring
,
C. M.
,
2010
, “
Extended Linkography and Distance Graph in Design Evaluation: An Empirical Study of the Dual Effects of Inspiration Sources in Creative Design
,”
Des. Stud.
,
31
(
2
), pp.
146
168
.
9.
Rubio
,
S.
,
Díaz
,
E.
,
Martín
,
J.
, and
Puente
,
J. M.
,
2004
, “
Evaluation of Subjective Mental Workload: A Comparison of SWAT, NASA-TLX, and Workload Profile Methods
,”
Appl. Psychol.
,
53
(
1
), pp.
61
86
.
10.
Dorta
,
T.
,
Pérez
,
E.
, and
Lesage
,
A.
,
2008
, “
The Ideation Gap: Hybrid Tools, Design Flow and Practice
,”
Des. Stud.
,
29
(
2
), pp.
121
141
.
11.
Shah
,
J. J.
,
Vargas-Hernandez
,
N.
, and
Smith
,
S. M.
,
2003
, “
Metrics for Measuring Ideation Effectiveness
,”
Des. Stud.
,
24
(
2
), pp.
111
134
.
12.
Guilford
,
J. P.
,
1967
,
The Nature of Human Intelligence
,
McGraw-Hill
,
New York
.
13.
Torrance
,
E. P.
,
1966
,
Torrance Tests of Creative Thinking: Norms-Technical Manual
,
Personnel Press
, Princeton, NJ.
14.
Dean
,
D. L.
,
Hender
,
J. M.
,
Rodgers
,
T. L.
, and
Santanen
,
E.
,
2006
, “
Identifying Good Ideas: Constructs and Scales for Idea Evaluation
,”
J. Assoc. Inf. Syst.
,
7
(
10
), pp.
646
699
.
15.
Nelson
,
B. A.
,
Wilson
,
J. O.
,
Rosen
,
D.
, and
Yen
,
J.
,
2009
, “
Refined Metrics for Measuring Ideation Effectiveness
,”
Des. Stud.
,
30
(
6
), pp.
737
743
.
16.
Dippo
,
C.
, and
Kudrowitz
,
B.
,
2015
, “
The Effects of Elaboration in Creativity Tests as it Pertains to Overall Scores and How it Might Prevent a Person From Thinking of Creative Ideas During the Early Stages of Brainstorming and Idea Generation
,”
ASME
Paper No. DETC2015-46789.
17.
O’Quin
,
K.
, and
Besemer
,
S. P.
,
2006
, “
Using the Creative Product Semantic Scale as a Metric for Results-Oriented Business
,”
Creativity Innovation Manage.
,
15
(
1
), pp.
34
44
.
18.
Amabile
,
T. M.
,
1982
, “
A Consensual Assessment Technique
,”
J. Pers. Soc. Psychol.
,
43
(
5
), pp.
997
1013
.
19.
Reis
,
S. M.
, and
Renzulli
,
J. S.
,
1991
, “
The Assessment of Creative Products in Programs for Gifted and Talented Students
,”
Gifted Child Q.
,
35
(
3
), pp.
128
134
.
20.
Oman
,
S.
, and
Tumer
,
I. Y.
,
2009
, “
The Potential of Creativity Metrics for Mechanical Engineering Concept Design
,”
The 17th International Conference on Engineering Design
(
ICED 09
), Stanford, CA, Aug. 24–27, Vol.
2
, pp.
145
156
.
21.
Oman
,
S. K.
,
Tumer
,
I. Y.
,
Wood
,
K.
, and
Seepersad
,
C.
,
2013
, “
A Comparison of Creativity and Innovation Metrics and Sample Validation Through In-Class Design Projects
,”
Res. Eng. Des.
,
24
(
1
), pp.
65
92
.
22.
Srivathsavai
,
R.
,
Genco
,
N.
,
Hölttä-Otto
,
K.
, and
Seepersad
,
C. C.
,
2010
, “
Study of Existing Metrics Used in Measurement of Ideation Effectiveness
,”
ASME
Paper No. DETC2010-28802.
23.
Reinig
,
B.
,
Briggs
,
R.
, and
Nunamaker
,
J.
,
2007
, “
On the Measurement of Ideation Quality
,”
J. Manage. Inf. Syst.
,
23
(
4
), pp.
143
161
.
24.
Fuge
,
M.
,
Stroud
,
J.
, and
Agogino
,
A.
,
2013
, “
Automatically Inferring Metrics for Design Creativity
,”
ASME
Paper No. DETC2013-12620.
25.
Kudrowitz
,
B. M.
, and
Wallace
,
D.
,
2013
, “
Assessing the Quality of Ideas From Prolific, Early-Stage Product Ideation
,”
J. Eng. Des.
,
24
(
2
), pp.
120
139
.
26.
Nisbett
,
R. E.
, and
Wilson
,
T. D.
,
1977
, “
Telling More Than We Can Know: Verbal Reports on Mental Processes
,”
Psychol. Rev.
,
84
(
3
), pp.
231
259
.
27.
Kounios
,
J.
,
Fleck
,
J. I.
,
Green
,
D. L.
,
Payne
,
L.
,
Stevenson
,
J. L.
,
Bowden
,
E. M.
, and
Jung-Beeman
,
M.
,
2008
, “
The Origins of Insight in Resting-State Brain Activity
,”
Neuropsychologia
,
46
(
1
), pp.
281
291
.
28.
Fish
,
J.
, and
Scrivener
,
S.
,
1990
, “
Amplifying the Mind’s Eye: Sketching and Visual Cognition
,”
Leonardo
,
23
(
1
), pp.
117
126
.
29.
Goldschmidt
,
G.
,
1991
, “
The Dialectics of Sketching
,”
Creativity Res. J.
,
4
(
2
), pp.
123
143
.
30.
Diamond
,
A.
,
2013
, “
Executive Functions
,”
Annu. Rev. Psychol.
,
64
(
1
), pp.
135
168
.
31.
Malenka
,
R. C.
,
Nestler
,
E. J.
, and
Hyman
,
S. E.
,
2009
, “
Higher Cognitive Function and Behavioral Control
,”
Molecular Neuropharmacology: A Foundation for Clinical Neuroscience
,
A.
Sydor
and
R. Y.
Brown
, eds.,
2nd ed.
,
McGraw-Hill Medical
,
New York
, p.
318
.
32.
Baddeley
,
A. D.
, and
Hitch
,
G.
,
1974
, “
Working Memory
,”
Psychol. Learn. Motiv.
,
8
, pp.
47
89
.
33.
Martindale
,
C.
,
1999
, “
Biological Bases of Creativity
,”
Handbook of Creativity
,
Cambridge University Press
,
Cambridge, UK
, pp.
137
152
.
34.
Simonton
,
D. K.
,
2010
, “
Creative Thought as Blind-Variation and Selective-Retention: Combinatorial Models of Exceptional Creativity
,”
Phys. Life Rev.
,
7
(
2
), pp.
156
179
.
35.
Amabile
,
T. M.
,
2012
,
Componential Theory of Creativity
,
Harvard Business School
,
Boston, MA
.
36.
Alexiou
,
K.
,
Zamenopoulos
,
T.
,
Johnson
,
J. H.
, and
Gilbert
,
S. J.
,
2009
, “
Exploring the Neurological Basis of Design Cognition Using Brain Imaging: Some Preliminary Results
,”
Des. Stud.
,
30
(
6
), pp.
623
647
.
37.
Sylcott
,
B.
,
Cagan
,
J.
, and
Tabibnia
,
G.
,
2013
, “
Understanding Consumer Tradeoffs Between Form and Function Through Metaconjoint and Cognitive Neuroscience Analyses
,”
ASME J. Mech. Des.
,
135
(
10
), p.
101002
.
38.
Schaufeli
,
W.
, and
Bakker
,
A. B.
,
2010
, “
Defining and Measuring Work Engagement: Bringing Clarity to the Concept
,”
Work Engagement: A Handbook of Essential Theory and Research
, US: Psychology Press, New York, pp.
10
24
.
39.
Nguyen
,
T. A.
, and
Zeng
,
Y.
,
2010
, “
Analysis of Design Activities Using EEG Signals
,”
ASME
Paper No. DETC2010-28477.
40.
Nguyen
,
T. A.
, and
Zeng
,
Y.
,
2012
, “
Clustering Designers’ Mental Activities Based on EEG Power
,”
The Ninth International Symposium on Tools and Methods of Competitive Engineering
(
TMCE 2012
), Karlsruhe, Germany, May 7–11.
41.
Nguyen
,
T. A.
, and
Zeng
,
Y.
,
2014
, “
A Physiological Study of Relationship Between Designer’s Mental Effort and Mental Stress During Conceptual Design
,”
Comput. Aided Des.
,
54
, pp.
3
18
.
42.
Carroll
,
E. A.
,
2011
, “
Convergence of Self-Report and Physiological Responses for Evaluating Creativity Support Tools
,”
8th ACM Conference on Creativity and Cognition
(
C&C 11
), Atlanta, Georgia, Nov. 3–6, pp.
455
456
.
43.
Carroll
,
E. A.
, and
Latulipe
,
C.
,
2012
, “
Triangulating the Personal Creative Experience: Self-Report, External Judgments, and Physiology
,” Proceedings of Graphics Interface (
GI '12
), Toronto, ON, May 28–30, pp.
53
60
.
44.
Toh
,
C.
,
Miller
,
S.
, and
Simpson
,
T.
,
2015
, “
The Impact of Virtual Product Dissection Environments on Student Design Learning and Self-Efficacy
,”
J. Eng. Des.
,
26
(
1–3
), pp.
48
73
.
45.
Hart
,
S. G.
, and
Staveland
,
L. E.
,
1988
, “
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
,”
Adv. Psychol.
,
52
, pp.
139
183
.
46.
Hu
,
W.-l.
,
Booth
,
J.
, and
Reid
,
T.
,
2015
, “
Reducing Sketch Inhibition During Concept Generation: Psychophysiological Evidence of the Effect of Interventions
,”
ASME
Paper No. DETC2015-47669.
47.
Booth
,
J. W.
,
Taborda
,
E. A.
,
Ramani
,
K.
, and
Reid
,
T.
,
2016
, “
Interventions for Teaching Sketching Skills and Reducing Inhibition for Novice Engineering Designers
,”
Des. Stud.
,
43
, pp.
1
23
.
48.
Durand
,
F.
,
Helms
,
M. E.
,
Tsenn
,
J.
,
McAdams
,
D. A.
, and
Linsey
,
J. S.
,
2015
, “
In Search of Effective Design Problems for Design Research
,”
ASME
Paper No. DETC2015-47701.
49.
David Hairston
,
W.
,
Whitaker
,
K. W.
,
Ries
,
A. J.
,
Vettel
,
J. M.
,
Cortney Bradford
,
J.
,
Kerick
,
S. E.
, and
McDowell
,
K.
,
2014
, “
Usability of Four Commercially-Oriented EEG Systems
,”
J. Neural Eng.
,
11
(
4
), p.
046018
.
50.
Hocevar
,
D.
,
1979
, “
The Development of the Creative Behavior Inventory (CBI)
,”
Annual Meeting of the Rocky Mountain Psychological Association
, Paper No.
ED170350
.
51.
Berka
,
C.
,
Levendowski
,
D. J.
,
Cvetinovic
,
M. M.
,
Petrovic
,
M. M.
,
Davis
,
G.
,
Lumicao
,
M. N.
,
Zivkovic
,
V. T.
,
Popovic
,
M. V.
, and
Olmstead
,
R.
,
2004
, “
Real-Time Analysis of EEG Indexes of Alertness, Cognition, and Memory Acquired With a Wireless EEG Headset
,”
Int. J. Hum. Comput. Interact.
,
17
(
2
), pp.
151
170
.
52.
Berka
,
C.
,
Levendowski
,
D. J.
,
Lumicao
,
M. N.
,
Yau
,
A.
,
Davis
,
G.
,
Zivkovic
,
V. T.
,
Olmstead
,
R. E.
,
Tremoulet
,
P. D.
, and
Craven
,
P. L.
,
2007
, “
EEG Correlates of Task Engagement and Mental Workload in Vigilance, Learning, and Memory Tasks
,”
Aviat., Space, Environ. Med.
,
78
(
5
), pp.
B231
B244
.
53.
de Guinea
,
A. O.
,
Titah
,
R.
, and
Léger
,
P.-M.
,
2013
, “
Measure for Measure: A Two Study Multi-Trait Multi-Method Investigation of Construct Validity in IS Research
,”
Comput. Hum. Behav.
,
29
(
3
), pp.
833
844
.
54.
Johnson
,
R. R.
,
Popovic
,
D. P.
,
Olmstead
,
R. E.
,
Stikic
,
M.
,
Levendowski
,
D. J.
, and
Berka
,
C.
,
2011
, “
Drowsiness/Alertness Algorithm Development and Validation Using Synchronized EEG and Cognitive Performance to Individualize a Generalized Model
,”
Biol. Psychol.
,
87
(
2
), pp.
241
250
.
55.
Ries
,
A. J.
,
Touryan
,
J.
,
Vettel
,
J.
,
McDowell
,
K.
, and
Hairston
,
W. D.
,
2014
, “
A Comparison of Electroencephalography Signals Acquired From Conventional and Mobile Systems
,”
J. Neurosci. Neuroeng.
,
3
(
1
), pp.
10
20
.
56.
Oberauer
,
K.
,
Süß
,
H. M.
,
Schulze
,
R.
,
Wilhelm
,
O.
, and
Wittmann
,
W. W.
,
2000
, “
Working Memory Capacity—Facets of a Cognitive Ability Construct
,”
Pers. Individ. Differ.
,
29
(
6
), pp.
1017
1045
.
57.
Sun
,
X.
,
Zhang
,
X.
,
Chen
,
X.
,
Zhang
,
P.
,
Bao
,
M.
,
Zhang
,
D.
,
Chen
,
J.
,
He
,
S.
, and
Hu
,
X.
,
2005
, “
Age-Dependent Brain Activation During Forward and Backward Digit Recall Revealed by fMRI
,”
Neuroimage
,
26
(
1
), pp.
36
47
.
58.
Stemler
,
S. E.
,
2004
, “
A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability
,”
Pract. Assess., Res. Eval.
,
9
(
4
).
59.
Neuendorf
,
K. A.
,
2002
,
The Content Analysis Guidebook
,
SAGE
, Thousand Oaks, CA.
60.
Kroeker
,
M.
,
Lacy
,
M.
,
Mark
,
A.
, and
Van De Loo
,
J.
,
1996
, “
Interrater Reliability in the Assessment of Sitting Balance in Hemiparetic Patients
,”
Masters thesis
,
University of Osteopathic Medicine and Health Sciences
, Des Moines, IA.
61.
LeBreton
,
J. M.
, and
Senter
,
J. L.
,
2008
, “
Answers to 20 Questions About Interrater Reliability and Interrater Agreement
,”
Organ. Res. Methods
,
11
(
4
), pp.
815
852
.
62.
Kershaw
,
T. C.
,
Peterson
,
R. L.
,
McCarthy
,
M. A.
,
Young
,
A. P.
,
Seepersad
,
C. C.
,
Williams
,
P. T.
,
Hölttä-Otto
,
K.
, and
Bhowmick
,
S.
,
2015
, “
A Cross-Sectional and Longitudinal Examination of the Development of Innovation Capability in Undergraduate Engineering Students
,”
ASME
Paper No. DETC2015-47650.
63.
Charyton
,
C.
,
Jagacinski
,
R. J.
, and
Merrill
,
J. A.
,
2008
, “
CEDA: A Research Instrument for Creative Engineering Design Assessment
,”
Psychol. Aesthetics, Creativity, Arts
,
2
(
3
), pp.
147
154
.
64.
Maletic
,
J. I.
, and
Marcus
,
A.
,
2000
, “
Data Cleansing: Beyond Integrity Analysis
,”
2000 Conference on Information Quality (IQ-2000)
, pp.
200
209
.
65.
Xiong
,
H.
, and
Pandey
,
G.
,
2006
, “
Enhancing Data Analysis With Noise Removal
,”
IEEE Trans. Knowl. Data Eng.
,
18
(
3
), pp.
304
319
.
66.
Lotte
,
F.
,
Congedo
,
M.
,
Lécuyer
,
A.
,
Lamarche
,
F.
, and
Arnaldi
,
B.
,
2007
, “
A Review of Classification Algorithms for EEG-Based Brain-Computer Interfaces
,”
J. Neural Eng.
,
4
(
2
), pp.
R1
R13
.
67.
Sohaib
,
A. T.
,
Qureshi
,
S.
,
Hagelbäck
,
J.
,
Hilborn
,
O.
, and
Jerčić
,
P.
,
2013
, “
Evaluating Classifiers for Emotion Recognition Using EEG
,”
Foundations of Augmented Cognition: 7th International Conference, Part of HCI International 2013
(
AC 2013
), Las Vegas, NV, July 21–26,
2013
,
D. D.
Schmorrow
and
C. M.
Fidopiastis
, eds., Springer, Berlin, pp.
492
501
.
68.
Singla
,
R.
,
Chambayil
,
B.
,
Khosla
,
A.
, and
Santosh
,
J.
,
2011
, “
Comparison of SVM and ANN for Classification of eye Events in EEG
,”
J. Biomed. Sci. Eng.
,
4
(
01
), pp.
62
69
.
69.
Beal
,
C. R.
, and
Cirett Galán
,
F.
,
2012
, “
EEG Estimates of Cognitive Workload and Engagement Predict Math Problem Solving Outcomes
,”
International Conference on User Modeling, Adaptation, and Personalization, Society for Research on Educational Effectiveness
(
UMAP 2012
), Montreal, QC, July 16–20, pp.
51
62
.
70.
Ghali
,
R.
, and
Frasson
,
C.
,
2015
, “
Classification and Regression of Learner’s Scores in Logic Environment
,”
J. Educ. Training Stud.
,
3
(
5
), pp.
242
253
.
71.
Myrden
,
A.
, and
Chau
,
T.
,
2015
, “
Effects of User Mental State on EEG-BCI Performance
,”
Front. Hum. Neurosci.
,
9
, p.
308
.
72.
Borghini
,
G.
,
Astolfi
,
L.
,
Vecchiato
,
G.
,
Mattia
,
D.
, and
Babiloni
,
F.
,
2014
, “
Measuring Neurophysiological Signals in Aircraft Pilots and Car Drivers for the Assessment of Mental Workload, Fatigue and Drowsiness
,”
Neurosci. Biobehav. Rev.
,
44
, pp.
58
75
.
73.
Kohavi
,
R.
,
1995
, “
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
,”
International Joint Conference on Artificial Intelligence
(
IJCAI'95
), Montreal, QC, Aug. 20–25, Vol.
14
, pp.
1137
1143
.
74.
Wu
,
C. H.
,
Cheng
,
Y.
,
Ip
,
H. M.
, and
Mcbride-Chang
,
C.
,
2005
, “
Age Differences in Creativity: Task Structure and Knowledge Base
,”
Creativity Res. J.
,
17
(
4
), pp.
321
326
.
75.
Agogué
,
M.
,
Poirel
,
N.
,
Pineau
,
A.
,
Houdé
,
O.
, and
Cassotti
,
M.
,
2014
, “
The Impact of Age and Training on Creativity: A Design-Theory Approach to Study Fixation Effects
,”
Thinking Skills Creativity
,
11
, pp.
33
41
.
76.
Purcell
,
A. T.
, and
Gero
,
J. S.
,
1996
, “
Design and Other Types of Fixation
,”
Des. Stud.
,
17
(
4
), pp.
363
383
.
77.
Sobek
,
D. K.
, and
Jain
,
V. K.
,
2007
, “
Relating Design Process to Quality: A Virtual Design of Experiments Approach
,”
ASME J. Mech. Des.
,
129
(
5
), pp.
483
490
.
78.
Chusilp
,
P.
, and
Jin
,
Y.
,
2004
, “
Cognitive Modeling of Iteration in Conceptual Design
,”
ASME
Paper No. DETC2004-57521.
79.
Chusilp
,
P.
, and
Jin
,
Y.
,
2006
, “
Impact of Mental Iteration on Concept Generation
,”
ASME J. Mech. Des.
,
128
(
1
), pp.
14
25
.
80.
Liu
,
Z.
,
Yang
,
D.-S.
,
Wen
,
D.
,
Zhang
,
W.-M.
, and
Mao
,
W.
,
2011
, “
Cyber-Physical-Social Systems for Command and Control
,”
IEEE Intell. Syst.
,
26
(
4
), pp.
92
96
.
81.
Borgman
,
C. L.
,
Abelson
,
H.
,
Dirks
,
L.
,
Johnson
,
R.
,
Koedinger
,
K. R.
,
Linn
,
M. C.
,
Lynch
,
C. A.
,
Oblinger
,
D. G.
,
Pea
,
R. D.
,
Salen
,
K.
,
Smith
M. S.
, and
Szalay
,
A.
,
2008
, “
Fostering Learning in the Networked World: The Cyberlearning Opportunity and Challenge. A 21st Century Agenda for the National Science Foundation
,” Report of the NSF Task Force on Cyberlearning, Report No.
nsf08204
.
82.
Linsey
,
J. S.
,
Tseng
,
I.
,
Fu
,
K.
,
Cagan
,
J.
,
Wood
,
K. L.
, and
Schunn
,
C.
,
2010
, “
A Study of Design Fixation, Its Mitigation and Perception in Engineering Design Faculty
,”
ASME J. Mech. Des.
,
132
(
4
), p.
041003
.
83.
Viswanathan
,
V.
,
Atilola
,
O.
,
Esposito
,
N.
, and
Linsey
,
J.
,
2014
, “
A Study on the Role of Physical Models in the Mitigation of Design Fixation
,”
J. Eng. Des.
,
25
(
1–3
), pp.
25
43
.
You do not currently have access to this content.