Usability of in-vehicle systems has become increasingly important for ease of operations and safety of driving. The user interface (UI) of in-vehicle systems is a critical focus of usability study. This paper studies how to use advanced computational, physiology- and behavior-based tools and methodologies to determine affective/emotional states and behavior of an individual in real time and in turn how to adapt the human-vehicle interaction to meet users' cognitive needs based on the real-time assessment. Specifically, we set up a set of physiological sensors that are capable of collecting EEG, facial EMG, skin conductance response, and respiration data and a set of motion sensing and tracking equipment that is capable of capturing eye ball movement and objects which the user is interacting with. All hardware components and software are integrated into an augmented sensor platform that can perform as “one coherent system” to enable multimodal data processing and information inference for context-aware analysis of emotional states and cognitive behavior based on the rough set inference engine. Meanwhile subjective data are also recorded for comparison. A usability study of in-vehicle system UI is shown to demonstrate the potential of the proposed methodology.

References

References
1.
Kim
,
S.
,
Dey
,
A. K.
,
Lee
,
J.
, and
Forlizzi
,
J.
,
2011
, “
Usability of Car Dashboard Displays for Elder Drivers
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
, ACM, Vancouver, BC, Canada, pp.
493
502
.
2.
Harvey
,
C.
,
Stanton
,
N. A.
,
Pickering
,
C. A.
,
McDonald
,
M.
, and
Zheng
,
P.
,
2011
, “
A Usability Evaluation Toolkit for In-Vehicle Information Systems (IVISs)
,”
Appl. Ergon.
,
42
(
4
), pp.
563
574
.10.1016/j.apergo.2010.09.013
3.
Pauzié
,
A.
,
2008
, “
Evaluating Driver Mental Workload Using the Driving Activity Load Index (DALI)
,”
European Conference on Human Centred Design for Intelligent Transport Systems
, Lyon, France.
4.
Young
,
K.
, and
Regan
,
M.
,
2007
, “
Driver distraction: A review of the literature
,”
Distracted Driving
,
I. J.
Faulks
,
M.
Regan
,
M.
Stevenson
,
J.
Brown
,
A.
Porter
, and
J. D.
Irwin
, eds.,
Australasian College of Road Safety
,
Sydney, NSW
, pp.
379
405
.
5.
Hix
,
D.
,
Swan
,
E. J.
,
Gabbard
,
J. L.
,
McGee
,
M.
,
Durbin
,
J.
, and
King
,
T.
, “
User-Centered Design and Evaluation of a Real-Time Battlefield Visualization Virtual Environment
,”
Proc. IEEE VR'99 Conference, IEEE Computer Society Press
, pp.
96
103
.
6.
Gabbard
,
J. L.
,
Swan
,
J. E.
,
Hix
,
D.
,
Lanzagorta
,
M.
,
Livingston
,
M.
,
Brown
,
D.
, and
Julier
,
S.
,
2002
, “
Usability Engineering: Domain Analysis Activities for Augmented Reality Systems
,”
9th Annual Engineering Reality of Virtual Reality Conference
, San Jose, CA.
7.
Rosson
,
M. B.
, and
Carroll
,
J. M.
,
2002
,
Usability Engineering: Scenario-Based Development of Human-Computer Interaction
,
Academic Press
,
San Fancisco
.
8.
Hix
,
D.
, and
Hartson
,
H. R.
,
1993
,
Developing User Interfaces: Ensuring Usability through Product & Process
,
John Wiley and Sons, Inc.
,
New York
.
9.
Gabbard
,
J. L.
,
Hix
,
D.
,
Swan
,
J. E.
,
Livingston
,
M. A.
,
Höllerer
,
T. H.
,
Julier
,
S. J.
,
Brown
,
D.
, and
Baillot
,
Y.
,
2003
, “
Usability Engineering for Complex Interactive Systems Development
,”
Human Systems Integration Symposium
, Vienna, VA.
10.
Zhou
,
F.
,
Xu
,
Q.
, and
Jiao
,
R.
,
2011
, “
Fundamentals of Product Ecosystem Design for User Experience
,”
Res. Eng. Des.
,
22
(
1
), pp.
43
61
.10.1007/s00163-010-0096-z
11.
Zhou
,
F.
,
Ji
,
Y. J.
, and
Jiao
,
R. J.
,
2012
, “
Affective and Cognitive Design for Mass Personalization: Status and Prospect
,”
J. Intell. Manuf.
,
24
(
5
), pp.
1047
1069
10.1007/s10845-012-0673-2.
12.
Grandjean
,
D.
,
Sander
,
D.
, and
Scherer
,
K. R.
,
2008
, “
Conscious Emotional Experience Emerges as a Function of Multilevel, Appraisal-Driven Response Synchronization
,”
Conscious Cogn.
,
17
(
2
), pp.
484
495
.10.1016/j.concog.2008.03.019
13.
Stone
,
A. A.
, and
Shiffman
,
S.
,
1994
, “
Ecological Momentary Assessment (EMA) in Behavioral Medicine
,”
Ann. Behav. Med.
,
16
(
3
), pp.
199
202
.
14.
Paas
,
F.
,
Tuovinen
,
J. E.
,
Tabbers
,
H. K.
, and
Van Gerven
,
P. W. M.
,
2003
, “
Cognitive Load Measurement as a Means to Advance Cognitive Load Theory
,”
Educ. Psychol.
,
38
(
1
), pp.
63
71
.10.1207/S15326985EP3801_8
15.
Haapalainen
,
E.
,
Kim
,
S.
,
Forlizzi
,
J. F.
, and
Dey
,
A. K.
,
2010
, “
Psycho-Physiological Measures for Assessing Cognitive Load
,”
Proceedings of the 12th ACM international conference on Ubiquitous computing
, ACM, Copenhagen, Denmark, pp.
301
310
.
16.
Ericsson
,
K.
, and
Simon
,
H.
,
1993
,
Protocol Analysis: Verbal Reports as Data
,
MIT Press
,
Boston, MA
.
17.
Detenber
,
B. H.
,
2001
, “
Measuring Emotional Responses in Human Factors Research: Some Theoretical and Practical Considerations
,”
Proceedings of the International Conference on Affective Human Factors Design
,
M.
Helander
,
H.
Khalid
, and
M.
Tham
, eds.,
Asean Academic Press
,
London
, pp.
124
130
.
18.
Annett
,
J.
,
2002
, “
Subjective Rating Scales: Science or Art?
,”
Ergonomics
,
45
(
14
), pp.
966
987
.10.1080/00140130210166951
19.
Reid
,
G. B.
,
Shingledecker
,
C. A.
,
Nygren
,
T. E.
, and
Eggemeier
,
F. T.
,
1981
, “
Development of Multidimensional Subjective Measures of Workload
,”
IEEE International Conference on Cybernetics & Society
, Atlanta, GA.
20.
Hart
,
S. G.
, and
Staveland
,
L. E.
,
1988
, “
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
,”
Human Mental Workload
,
P. A.
Hancock
, and
N.
Meshkati
, eds.,
North-Holland Press
,
Amsterdam
.
21.
Nielsen
,
J.
,
1994
,
Usability Engineering
,
Morgan Kaufmann
,
San Francisco, CA
.
22.
Mital
,
A.
, and
Govindaraju
,
M.
,
1999
, “
Is It Possible to Have a Single Measure for All Work?
,”
Int. J. Indus. Eng. Theory
,
6
(
3
), pp.
190
195
.
23.
Young
,
K. L.
,
Regan
,
M. A.
, and
Lee
,
J. D.
,
2009
, “
Measuring the Effects of Driver Distraction: Direct Driving Performance Methods and Measures
,”
Driver Distraction: Theory, Effects and Mitigation
,
M. A.
Regan
,
J. D.
Lee
:
K. L.
Young
, eds.,
CRC Press
,
Boca Raton, FL
, pp.
85
105
.
24.
Wittmann
,
M.
,
Kiss
,
M.
,
Gugg
,
P.
,
Steffen
,
A.
,
Fink
,
M.
,
Pöppel
,
E.
, and
Kaniya
,
H.
,
2006
, “
Effect of Display Position of a Visual In-Vehicle Task on Simulated Driving
,”
Appl. Ergon.
,
37
(
2
), pp.
187
199
.10.1016/j.apergo.2005.06.002
25.
Chiang
,
D. P.
,
Brooks
,
A. M.
, and
Weir
,
D. H.
,
2004
, “
On the Highway Measures of Driver Glance Behavior With An Example Automobile Navigation System
,”
Appl. Ergon.
,
35
(
3
), pp.
215
223
.10.1016/j.apergo.2004.01.005
26.
Liu
,
C.-C.
,
Doong
,
J.-L.
,
Hsu
,
C.-C.
,
Huang
,
W.-S.
, and
Jeng
,
M.-C.
,
2009
, “
Evidence for the Selective Attention Mechanism and Dual-Task Interference
,”
Appl. Ergon.
,
40
(
3
), pp.
341
347
.10.1016/j.apergo.2008.11.014
27.
Zhou
,
F.
,
Qu
,
X.
,
Helander
,
M. G.
, and
Jiao
,
J.
,
2011
, “
Affect Prediction From Physiological Measures Via Visual Stimuli
,”
Int. J. Hum.-Comput. Stud.
,
69
(
12
), pp.
801
819
.10.1016/j.ijhcs.2011.07.005
28.
Schiano
,
D. J.
,
Ehrlich
,
S. M.
, and
Sheridan
,
K.
,
2004
, “
Categorical Imperative NOT: Facial Affect is Perceived Continuously
,”
SIGCHI Conference on Human Factors in Computing Systems
, Vienna, Austria.
29.
Bailenson
,
J. N.
,
Pontikakis
,
E. D.
,
Mauss
, I
. B.
,
Gross
,
J. J.
,
Jabon
,
M. E.
,
Hutcherson
,
C. A. C.
,
Nass
,
C.
, and
Oliver
,
J.
,
2008
, “
Real-Time Classification of Evoked Emotions Using Facial Feature Tracking and Physiological Responses
,”
Int. J. Hum.-Comput. Stud.
,
66
(
5
), pp.
303
317
.10.1016/j.ijhcs.2007.10.011
30.
Fairclough
,
S. H.
,
2009
, “
Fundamentals of Physiological Computing
,”
Interact. Comput.
,
21
(
1-2
), pp.
133
145
.10.1016/j.intcom.2008.10.011
31.
Mandryk
,
R.
, and
Atkins
,
M.
,
2007
, “
A Fuzzy Physiological Approach for Continuously Modeling Emotion During Interaction With Play Technologies
,”
Int. J. Hum.-Comput. Stud.
,
65
(
4
), pp.
329
347
.10.1016/j.ijhcs.2006.11.011
32.
Niedermeyer
,
E.
, and
Lopes Da Silva
,
F.
,
2004
,
Electroencephalography: Basic Principles, Clinical Applications, and Related Fields
,
Lippincott Williams & Wilkins
,
Philadelphia, PA
.
33.
Bos
,
D. O.
,
2008
, “
EEG-Based Emotion Recognition: The Influence of Visual and Auditory Stimuli
.” Available from: http://hmi.ewi.utwente.nl/verslagen/capita-selecta/CS-Oude_Bos-Danny.pdf
34.
Lang
,
P. J.
,
Greenwald
,
M. K.
,
Bradley
,
M. M.
, and
Hamm
,
A. O.
,
1993
, “
Looking at Pictures: Affective, Facial, Visceral, and Behavioral Reactions
,”
Psychophysiology
,
30
(
3
), pp.
261
273
.10.1111/j.1469-8986.1993.tb03352.x
35.
Ablassmeier
,
M.
,
Poitschke
,
T.
,
Wallhoff
,
F.
,
Bengler
,
K.
, and
Rigoll
,
G.
,
2007
, “
Eye Gaze Studies Comparing Head-Up and Head-Down Displays in Vehicles
,”
IEEE International Conference on Multimedia and Expo
, Beijing, China.
36.
Pawlak
,
Z.
,
1991
,
Rough Sets: Theoretical Aspects of Reasoning About Data
,
Kluwer
,
Dordrecht, The Netherlands
.
37.
Duntsch
,
I.
, and
Gediga
,
G.
,
1998
, “
Uncertainty Measures of Rough Set Prediction
,”
AI Commun.
,
106
(
1
), pp.
109
137
.
38.
Picard
,
R. W.
,
Vyzas
,
E.
, and
Healey
,
J.
,
2001
, “
Toward Machine Emotional Intelligence: Analysis of Affective Physiological State
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
23
(
10
), pp.
1175
1191
.10.1109/34.954607
39.
Welch
,
P. D.
,
1967
, “
The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging Over Short, Modified Periodograms
,”
IEEE Trans. Audio Electroacoust.
,
AU-15
, pp.
70
73
.10.1109/TAU.1967.1161901
40.
Fodor
,
I. K.
,
2002
, “
A Survey of Dimension Reduction Techniques
,” Lawrence Livermore National Laboratory, Livermore, CA, Technical Report No. UCRL-ID-148494.
41.
Martinez
,
A. M.
, and
Kak
,
A. C.
,
2001
, “
PCA Versus LDA
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
23
(
2
), pp.
228
233
.10.1109/34.908974
42.
Fisher
,
R. A.
,
1938
, “
The Statistical Utilization of Multiple Measurements
,”
Ann. Eugen.
,
8
(
4
), pp.
376
386
.10.1111/j.1469-1809.1938.tb02189.x
43.
Bazan
,
J.
, and
Szczuka
,
M.
,
2000
,
RSES and RSESlib—A Collection of Tools for Rough Set Computations
, Lecture Notes in Artificial Intelligence,
Springer-Verlag
,
Berlin, Heidelberg
, pp.
592
601
.
44.
Gora
,
G.
, and
Wojna
,
A.
,
2002
, “
Riona: A New Classification System Combining Rule Induction and Instance-Based Learning
,”
Fund. Inform.
,
51
(
4
), pp.
369
390
.
45.
Cacioppo
,
J. T.
, and
Tassinary
,
L. G.
,
1990
, “
Inferring Psychological Significance From Physiological Signals
,”
Am. Psychol.
,
45
(
1
), pp.
16
28
.10.1037/0003-066X.45.1.16
46.
Gu
,
Y.
,
Tan
,
S. L.
,
Wong
,
K. J.
,
Ho
,
M.-H. R.
, and
Qu
,
L.
,
2010
, “
A GMM Based 2-Stage Architecture for Multi-Subject Emotion Recognition Using Physiological Responses
,”
The 1st Augmented Human International Conference
, Megève, France.
47.
Zhou
,
F.
, and
Jiao
,
R. J.
,
2013
, “
A Hierarchical Bayesian Model for Affective Parameter Shaping in User Experience Prospect Evaluation
,”
IEEE Trans. Affect. Comput.
(submitted).
48.
Victor
,
T. W.
,
Harbluk
,
J. L.
, and
Engstro
,
J. A.
,
2005
, “
Sensitivity of Eye-Movement Measures to In-Vehicle Task Difficulty
,”
Transp. Res. Part F
,
8
(
2
), pp.
167
190
.10.1016/j.trf.2005.04.014
49.
Dingus
,
T. A.
,
Klauer
,
S. G.
,
Neale
,
V. L.
,
Petersen
,
A.
,
Lee
,
S. E.
, and
Sudweeks
,
J.
,
2006
, “
The 100-Car Naturalistic Driving Study, Phase II—Results of the 100-Car Field Experiment
,” National Highway Traffic Safety Administration, Washington, DC, Paper No. DOT HS 810-593.
50.
Lee
,
J. D.
,
2009
, “
Can Technology Get Your Eyes Back on the Road?
,”
Science
,
324
, pp.
344
346
.10.1126/science.1168085
51.
Roman
,
B.
,
Pavel
,
S.
,
Miroslav
,
P.
,
Petr
,
V.
, and
Lubomír
,
P.
,
2001
, “
Fatigue Indicators of Drowsy Drivers Based on Analysis of Physiological Signals
,”
Medical Data Analysis
,
J.
Crespo
,
V.
Maojo
, and
F.
Martin
, eds.,
Springer
,
Berlin/Heidelberg
, pp.
62
68
.
52.
Grace
,
R.
,
Byrne
,
V. E.
,
Bierman
,
D. M.
,
Legrand
,
J.-M.
,
Gricourt
,
D.
,
Davis
,
B. K.
,
Staszewski
,
J. J.
, and
Carnahan
.,
B.
,
1998
, “
A Drowsy Driver Detection System for Heavy Vehicles
,”
17th Digital Avionics Systems Conference
, Seattle, WA.
You do not currently have access to this content.