Abstract

One of the expectations for the next generation of industrial robots is to work collaboratively with humans as robotic co-workers. Robotic co-workers must be able to communicate with human collaborators intelligently and seamlessly. However, industrial robots in prevalence are not good at understanding human intentions and decisions. We demonstrate a steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) which can directly deliver human cognition to robots through a headset. The BCI is applied to a part-picking robot. The BCI sends decisions to the robot while operators visually inspecting the quality of parts. The BCI is verified through a human subject study. In the study, a camera by the side of the conveyor takes photos of each industrial part and presents it to the operator automatically. When the operator looks at the photo, the electroencephalography (EEG) is collected through the BCI. The inspection decision is extracted through SSVEPs in EEG. When a defective part is identified by the operator, the signal is communicated to the robot, which locates the defective part by a second camera and removes it from the conveyor. The robot can grasp various part with our random grasp planning algorithm (2FRG). We have developed a CNN-CCA model for SSVEP extraction. The model is trained on a dataset collected in our offline experiment. Our approach outperforms the existing CCA, CCA-SVM, and PSD-SVM models. The CNN-CCA model is further validated in an online experiment and achieved 93% accuracy in identifying and removing defective parts.

References

1.
Henry
,
J. C.
,
2006
, “
Electroencephalography: Basic Principles, Clinical Applications, and Related Fields
,”
Neurology
,
67
(
11
), pp.
2092
2092
.
2.
Bozinovski
,
S.
,
Sestakov
,
M.
, and
Bozinovska
,
L.
,
1988
, “
Using Eeg Alpha Rhythm to Control a Mobile Robot
,”
Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Pts 1–4
,
Seattle, WA
,
Nov. 9–12
, IEEE, pp.
1515
1516
.
3.
Hirzinger
,
G.
,
Sporer
,
N.
,
Albu-Schaffer
,
A.
,
Hahnle
,
M.
,
Krenn
,
R.
,
Pascucci
,
A.
, and
Schedl
,
M.
,
2002
, “
Dlr’s Torque-Controlled Light Weight Robot III-Are We Reaching the Technological Limits Now?
Proceedings 2002 IEEE International Conference on Robotics and Automation
,
Nice, France
,
Sept. 22–26
, Vol. 2. IEEE, pp.
1710
1716
.
4.
Zinn
,
M.
,
Roth
,
B.
,
Khatib
,
O.
, and
Salisbury
,
J. K.
,
2004
, “
A New Actuation Approach for Human Friendly Robot Design
,”
Int. J. Rob. Res.
,
23
(
4–5
), pp.
379
398
.
5.
Shin
,
D.
,
Tanaka
,
A.
,
Kim
,
N.
, and
Khatib
,
O.
,
2016
, “
A Centrifugal Force-based Configuration-Independent High-Torque-Density Passive Brake for Human-Friendly Robots
,”
IEEE/ASME Trans. Mechatron.
,
21
(
6
), pp.
2827
2835
.
6.
Haddadin
,
S.
,
Albu-Schaffer
,
A.
,
De Luca
,
A.
, and
Hirzinger
,
G.
,
2008
, “
Collision Detection and Reaction: A Contribution to Safe Physical Human-Robot Interaction
,”
2008 IEEE/RSJ International Conference on Intelligent Robots and Systems
,
Nice, France
,
Sept. 22–26
, IEEE, pp.
3356
3363
.
7.
Geravand
,
M.
,
Flacco
,
F.
, and
De Luca
,
A.
,
2013
, “
Human-Robot Physical Interaction and Collaboration Using An Industrial Robot with a Closed Control Architecture
,”
2013 IEEE International Conference on Robotics and Automation
,
Karlsruhe, Germany
,
May 6–10
, IEEE, pp.
4000
4007
.
8.
Zanchettin
,
A. M.
,
Ceriani
,
N. M.
,
Rocco
,
P.
,
Ding
,
H.
, and
Matthias
,
B.
,
2015
, “
Safety in Human-Robot Collaborative Manufacturing Environments: Metrics and Control
,”
IEEE Trans. Autom. Sci. Eng.
,
13
(
2
), pp.
882
893
.
9.
Wilcox
,
R.
,
Nikolaidis
,
S.
, and
Shah
,
J.
,
2013
, “
Optimization of Temporal Dynamics for Adaptive Human-Robot Interaction in Assembly Manufacturing
,”
Robotics
,
8
, p.
441
.
10.
Haddadin
,
S.
,
Albu-Schäffer
,
A.
, and
Hirzinger
,
G.
,
2009
, “
Requirements for Safe Robots: Measurements, Analysis and New Insights
,”
Int. J. Rob. Res.
,
28
(
11–12
), pp.
1507
1527
.
11.
Cordero
,
C. A.
,
Carbone
,
G.
,
Ceccarelli
,
M.
,
Echávarri
,
J.
, and
Muñoz
,
J. L.
,
2014
, “
Experimental Tests in Human–Robot Collision Evaluation and Characterization of a New Safety Index for Robot Operation
,”
Mech. Mach. Theory
,
80
, pp.
184
199
.
12.
Haddadin
,
S.
,
Albu-Schaffer
,
A.
, and
Hirzinger
,
G.
,
2008
, “
The Role of the Robot Mass and Velocity in Physical Human-Robot Interaction-Part I: Non-Constrained Blunt Impacts
,”
2008 IEEE International Conference on Robotics and Automation
,
Karlsruhe, Germany
,
May 6–10
, IEEE, pp.
1331
1338
.
13.
Gemignani
,
G.
,
Veloso
,
M.
, and
Nardi
,
D.
, “
Language-Based Sensing Descriptors for Robot Object Grounding
,”
19th Annual RoboCup International Symposium
,
Hefei, China
,
July 23
, pp.
3
15
.
14.
Sheikholeslami
,
S.
,
Moon
,
A.
, and
Croft
,
E. A.
,
2015
, “
Exploring the Effect of Robot Hand Configurations in Directional Gestures for Human-Robot Interaction
,”
2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Hamburg, Germany
,
Sept. 28–Oct. 2
, IEEE, pp.
3594
3599
.
15.
Gleeson
,
B.
,
MacLean
,
K.
,
Haddadi
,
A.
,
Croft
,
E.
, and
Alcazar
,
J.
,
2013
, “
Gestures for Industry Intuitive Human-Robot Communication From Human Observation
,”
2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI)
,
Tokyo, Japan
,
Mar. 3–6
, IEEE, pp.
349
356
.
16.
Beetz
,
M.
,
Bartels
,
G.
,
Albu-Schäffer
,
A.
,
Bálint-Benczédi
,
F.
,
Belder
,
R.
,
Beßler
,
D.
,
Haddadin
,
S.
,
Maldonado
,
A.
,
Mansfeld
,
N.
,
Wiedemeyer
,
T.
, and
Weitschat
,
R.
,
2015
, “
Robotic Agents Capable of Natural and Safe Physical Interaction with Human Co-Workers
,”
2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Hamburg, Germany
,
Sept. 28–Oct. 2
, IEEE, pp.
6528
6535
.
17.
Oshin
,
O.
,
Bernal
,
E. A.
,
Nair
,
B. M.
,
Ding
,
J.
,
Varma
,
R.
,
Osborne
,
R. W.
,
Tunstel
,
E.
, and
Stramandinoli
,
F.
,
2019
, “
Coupling Deep Discriminative and Generative Models for Reactive Robot Planning in Human-Robot Collaboration
,”
2019 IEEE International Conference on Systems, Man and Cybernetics (SMC)
,
Bari, Italy
,
Oct. 6–9
, IEEE, pp.
1869
1874
.
18.
Vogel
,
J.
,
Haddadin
,
S.
,
Simeral
,
J. D.
,
Stavisky
,
S. D.
,
Bacher
,
D.
,
Hochberg
,
L. R.
,
Donoghue
,
J. P.
, and
Van Der Smagt
,
P.
,
2014
, “Continuous Control of the DLR Light-Weight Robot III by a Human with Tetraplegia Using the Braingate2 Neural Interface System,”
Experimental Robotics
,
Springer
,
Berlin, Heidelberg
, pp.
125
136
.
19.
Edlinger
,
G.
,
Holzner
,
C.
, and
Guger
,
C.
,
2011
, “
A Hybrid Brain-Computer Interface for Smart Home Control
,”
International Conference on Ergonomics and Health Aspects of Work with Computers (EHAWC)/14th International Conference on Human-Computer Interaction (HCI)
,
Orlando, FL
,
July 9–14
, Springer, pp.
417
426
.
20.
Riaz
,
A.
,
Akhtar
,
S.
,
Iftikhar
,
S.
,
Khan
,
A. A.
, and
Salman
,
A.
,
2014
, “
Inter Comparison of Classification Techniques for Vowel Speech Imagery Using Eeg Sensors
,”
The 2014 2nd International Conference on Systems and Informatics (ICSAI 2014)
,
Shanghai, China
,
Nov. 15–17
, IEEE, pp.
712
717
.
21.
Yin
,
E.
,
Zhou
,
Z.
,
Jiang
,
J.
,
Chen
,
F.
,
Liu
,
Y.
, and
Hu
,
D.
,
2013
, “
A Novel Hybrid Bci Speller Based on the Incorporation of Ssvep Into the P300 Paradigm
,”
J. Neural. Eng.
,
10
(
2
), p.
026012
.
22.
Ying
,
R.
,
Weisz
,
J.
, and
Allen
,
P. K.
,
2018
, “
Grasping with Your Brain: A Brain-Computer Interface for Fast Grasp Selection
,”
12th International Symposium on Robotics Research (ISRR)
,
Sestri Levante, Italy
,
Sept. 12–15, 2015
, Springer, pp.
325
340
.
23.
Hortal
,
E.
,
Planelles
,
D.
,
Costa
,
A.
,
Iánez
,
E.
,
Úbeda
,
A.
,
Azorín
,
J. M.
, and
Fernández
,
E.
,
2015
, “
SVM-based Brain–machine Interface for Controlling a Robot Arm Through Four Mental Tasks
,”
Neurocomputing
,
151
, pp.
116
121
.
24.
Gandhi
,
V.
,
Prasad
,
G.
,
Coyle
,
D.
,
Behera
,
L.
, and
McGinnity
,
T. M.
,
2014
, “
Eeg-based Mobile Robot Control Through An Adaptive Brain–robot Interface
,”
IEEE. Trans. Syst. Man. Cybernet.: Syst.
,
44
(
9
), pp.
1278
1285
.
25.
LaFleur
,
K.
,
Cassady
,
K.
,
Doud
,
A.
,
Shades
,
K.
,
Rogin
,
E.
, and
He
,
B.
,
2013
, “
Quadcopter Control in Three-dimensional Space Using a Noninvasive Motor Imagery-based Brain–computer Interface
,”
J. Neural. Eng.
,
10
(
4
), p.
046003
.
26.
Li
,
Y.
, and
Kesavadas
,
T.
,
2018
, “
Brain Computer Interface Robotic Co-Workers: Defective Part Picking System
,”
In ASME 2018 13th International Manufacturing Science and Engineering Conference, American Society of Mechanical Engineers Digital Collection
,
College Station, TX
,
June 18–22
.
27.
Zhang
,
Y.
,
Zhou
,
G.
,
Jin
,
J.
,
Wang
,
X.
, and
Cichocki
,
A.
,
2014
, “
Frequency Recognition in SSVEP-based BCI Using Multiset Canonical Correlation Analysis
,”
Int. J. Neural Syst.
,
24
(
04
), p.
1450013
.
28.
Lin
,
Z.
,
Zhang
,
C.
,
Wu
,
W.
, and
Gao
,
X.
,
2006
, “
Frequency Recognition Based on Canonical Correlation Analysis for SSVEP-based BCIS
,”
IEEE Trans. Biomed. Eng.
,
53
(
12
), pp.
2610
2614
.
29.
Resalat
,
S. N.
, and
Setarehdan
,
S. K.
,
2013
, “
An Improved Ssvep Based BCI System Using Frequency Domain Feature Classification
,”
Am. J. Biomed. Eng.
,
3
(
1
), pp.
1
8
.
30.
Aznan
,
N. K. N.
,
Bonner
,
S.
,
Connolly
,
J.
,
Al Moubayed
,
N.
, and
Breckon
,
T.
,
2018
, “
On the Classification of SSVEP-based Dry-Eeg Signals Via Convolutional Neural Networks
,”
2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
,
Miyazaki, Japan
,
Oct. 7–10
, IEEE, pp.
3726
3731
.
31.
Li
,
Y.
,
Xiang
,
J.
, and
Kesavadas
,
T.
,
2020
, “
Convolutional Correlation Analysis for Enhancing the Performance of Ssvep-based Brain-computer Interface
,”
IEEE. Trans. Neural. Syst. Rehabil. Eng.
,
28
(
12
), pp.
2681
2690
.
32.
Wang
,
Y.
,
Chen
,
X.
,
Gao
,
X.
, and
Gao
,
S.
,
2016
, “
A Benchmark Dataset for Ssvep-based Brain–computer Interfaces
,”
IEEE. Trans. Neural. Syst. Rehabil. Eng.
,
25
(
10
), pp.
1746
1752
.
You do not currently have access to this content.