Abstract

In human–robot collaborative assembly, leveraging multimodal commands for intuitive robot control remains a challenge from command translation to efficient collaborative operations. This article investigates multimodal data-driven robot control for human–robot collaborative assembly. Leveraging function blocks, a programming-free human–robot interface is designed to fuse multimodal human commands that accurately trigger defined robot control modalities. Deep learning is explored to develop a command classification system for low-latency and high-accuracy robot control, in which a spatial-temporal graph convolutional network is developed for a reliable and accurate translation of brainwave command phrases into robot commands. Then, multimodal data-driven high-level robot control during assembly is facilitated by the use of event-driven function blocks. The high-level commands serve as triggering events to algorithms execution of fine robot manipulation and assembly feature-based collaborative assembly. Finally, a partial car engine assembly deployed to a robot team is chosen as a case study to demonstrate the effectiveness of the developed system.

Issue Section:
Research Papers
Keywords:
robot, assembly, multimodal data, human–robot collaboration, brain robotics
Topics:
Electroencephalography, Manufacturing, Robots, Signals

References

1.
Wang
,
L.
,
Gao
,
R.
,
Váncza
,
J.
,
Krüger
,
J.
,
Wang
,
X. V.
,
Makris
,
S.
, and
Chryssolouris
,
G.
,
2019
, “
Symbiotic Human-Robot Collaborative Assembly
,”
CIRP. Ann.
,
68
(
2
), pp.
701
–
726
.
2.
Matheson
,
E.
,
Minto
,
R.
,
Zampieri
,
E. G.
,
Faccio
,
M.
, and
Rosati
,
G.
,
2019
, “
Human–Robot Collaboration in Manufacturing Applications: A Review
,”
Robotics
,
8
(
4
), p.
100
.
3.
Wang
,
L.
,
Wang
,
X. V.
,
Váncza
,
J.
, and
Kemény
,
Z.
,
2021
,
Advanced Human–Robot Collaboration in Manufacturing
,
Springer Nature
,
Switzerland AG
.
4.
Liu
,
S.
,
Wang
,
L.
, and
Wang
,
X. V.
,
2021
, “
Function Block-Based Multimodal Control for Symbiotic Human–Robot Collaborative Assembly
,”
ASME J. Manuf. Sci. Eng.
,
143
(
9
), p.
091001
.
5.
Papanastasiou
,
S.
,
Kousi
,
N.
,
Karagiannis
,
P.
,
Gkournelos
,
C.
,
Papavasileiou
,
A.
,
Dimoulas
,
K.
,
Baris
,
K.
,
Koukas
,
S.
,
Michalos
,
G.
, and
Makris
,
S.
,
2019
, “
Towards Seamless Human Robot Collaboration: Integrating Multimodal Interaction
,”
Int. J. Adv. Manuf. Technol.
,
105
(
9
), pp.
3881
–
3897
.
6.
Liu
,
S.
,
Wang
,
L.
, and
Wang
,
X. V.
,
2020
, “
Symbiotic Human-Robot Collaboration: Multimodal Control Using Function Blocks
,”
Procedia CIRP
,
93
, pp.
1188
–
1193
.
7.
Chen
,
F.
,
Sekiyama
,
K.
,
Cannella
,
F.
, and
Fukuda
,
T.
,
2013
, “
Optimal Subtask Allocation for Human and Robot Collaboration Within Hybrid Assembly System
,”
IEEE Trans. Autom. Sci. Eng.
,
11
(
4
), pp.
1065
–
1075
.
8.
Villani
,
V.
,
Pini
,
F.
,
Leali
,
F.
, and
Secchi
,
C.
,
2018
, “
Survey on Human-Rrobot Collaboration in Industrial Settings: Safety, Intuitive Interfaces and Applications
,”
Mechatronics
,
55
, pp.
248
–
266
.
9.
Adamson
,
G.
,
Wang
,
L.
, and
Moore
,
P.
,
2017
, “
Feature-Based Control and Information Framework for Adaptive and Distributed Manufacturing in Cyber Physical Systems
,”
J. Manuf. Syst.
,
43
, pp.
305
–
315
.
10.
Liu
,
S.
,
Wang
,
L.
,
Wang
,
X. V.
,
Cooper
,
C.
, and
Gao
,
R. X.
,
2021
, “
Leveraging Multimodal Data for Intuitive Robot Control Towards Human-Robot Collaborative Assembly
,”
Procedia CIRP
,
104
, pp.
206
–
211
.
11.
Xue
,
T.
,
Wang
,
W.
,
Ma
,
J.
,
Liu
,
W.
,
Pan
,
Z.
, and
Han
,
M.
,
2020
, “
Progress and Prospects of Multimodal Fusion Methods in Physical Human–Robot Interaction: A Review
,”
IEEE Sensors J.
,
20
(
18
), pp.
10355
–
10370
.
12.
Du
,
G.
,
Chen
,
M.
,
Liu
,
C.
,
Zhang
,
B.
, and
Zhang
,
P.
,
2018
, “
Online Robot Teaching With Natural Human–Robot Interaction
,”
IEEE. Trans. Ind. Electron.
,
65
(
12
), pp.
9571
–
9581
.
13.
Wang
,
L.
,
Liu
,
S.
,
Cooper
,
C.
,
Wang
,
X. V.
, and
Gao
,
R. X.
,
2021
, “
Function Block-Based Human-Robot Collaborative Assembly Driven by Brainwaves
,”
CIRP. Ann.
,
70
(
1
), pp.
5
–
8
.
14.
Zhang
,
J.
,
Liu
,
H.
,
Chang
,
Q.
,
Wang
,
L.
, and
Gao
,
R. X.
,
2020
, “
Recurrent Neural Network for Motion Trajectory Prediction in Human-Robot Collaborative Assembly
,”
CIRP. Ann.
,
69
(
1
), pp.
9
–
12
.
15.
Arinez
,
J. F.
,
Chang
,
Q.
,
Gao
,
R. X.
,
Xu
,
C.
, and
Zhang
,
J.
,
2020
, “
Artificial Intelligence in Advanced Manufacturing: Current Status and Future Outlook
,”
ASME J. Manuf. Sci. Eng.
,
142
(
11
), p.
110804
.
16.
Bryan
,
A.
,
Hu
,
S. J.
, and
Koren
,
Y.
,
2013
, “
Assembly System Reconfiguration Planning
,”
ASME J. Manuf. Sci. Eng.
,
135
(
4
), p.
041005
.
17.
Pérez
,
L.
,
Rodríguez-Jiménez
,
S.
,
Rodríguez
,
N.
,
Usamentiaga
,
R.
,
García
,
D. F.
, and
Wang
,
L.
,
2020
, “
Symbiotic Human–Robot Collaborative Approach for Increased Productivity and Enhanced Safety in the Aerospace Manufacturing Industry
,”
Int. J. Adv. Manuf. Technol.
,
106
(
3
), pp.
851
–
863
.
18.
Wang
,
Y.
,
Jiao
,
Y.
,
Xiong
,
R.
,
Yu
,
H.
,
Zhang
,
J.
, and
Liu
,
Y.
,
2018
, “
Masd: A Multimodal Assembly Skill Decoding System for Robot Programming by Demonstration
,”
IEEE Trans. Autom. Sci. Eng.
,
15
(
4
), pp.
1722
–
1734
.
19.
Tian
,
R.
,
Sun
,
L.
,
Bajcsy
,
A.
,
Tomizuka
,
M.
, and
Dragan
,
A. D.
,
2021
, “Safety Assurances for Human-Robot Interaction via Confidence-Aware Game-Theoretic Human Models,” arXiv preprint arXiv:2109.14700.
20.
Wang
,
Z.
,
Mülling
,
K.
,
Deisenroth
,
M. P.
,
Ben Amor
,
H.
,
Vogt
,
D.
,
Schölkopf
,
B.
, and
Peters
,
J.
,
2013
, “
Probabilistic Movement Modeling for Intention Inference in Human–Robot Interaction
,”
Int. J. Rob. Res.
,
32
(
7
), pp.
841
–
858
.
21.
Tian
,
R.
,
Sun
,
L.
,
Tomizuka
,
M.
, and
Isele
,
D.
,
2021
, “
Anytime Game-Theoretic Planning With Active Reasoning About Humans’ Latent States for Human-Centered Robots
,”
2021 IEEE International Conference on Robotics and Automation
,
Xi'an, China
, IEEE, pp.
4509
–
4515
.
22.
Roveda
,
L.
,
Magni
,
M.
,
Cantoni
,
M.
,
Piga
,
D.
, and
Bucca
,
G.
,
2021
, “
Human–Robot Collaboration in Sensorless Assembly Task Learning Enhanced by Uncertainties Adaptation Via Bayesian Optimization
,”
Rob. Auton. Syst.
,
136
, p.
103711
.
23.
Liu
,
S.
,
Wang
,
L.
, and
Wang
,
X. V.
,
2021
, “
Sensorless Haptic Control for Human-Robot Collaborative Assembly
,”
CIRP. J. Manuf. Sci. Technol.
,
32
, pp.
132
–
144
.
24.
Lin
,
K.
,
Li
,
Y.
,
Sun
,
J.
,
Zhou
,
D.
, and
Zhang
,
Q.
,
2020
, “
Multi-Sensor Fusion for Body Sensor Network in Medical Human–Robot Interaction Scenario
,”
Inf. Fusion
,
57
, pp.
15
–
26
.
25.
Wang
,
P.
,
Fan
,
Z.
,
Kazmer
,
D. O.
, and
Gao
,
R. X.
,
2017
, “
Orthogonal Analysis of Multisensor Data Fusion for Improved Quality Control
,”
ASME J. Manuf. Sci. Eng.
,
139
(
10
), p.
101008
.
26.
Pohlt
,
C.
,
Schlegl
,
T.
, and
Wachsmuth
,
S.
,
2020
, “
Weakly-Supervised Learning for Multimodal Human Activity Recognition in Human-Robot Collaboration Scenarios
,”
2020 IEEE/RSJ International Conference on Intelligent Robots and Systems
,
Las Vegas, NV
, IEEE, pp.
8381
–
8386
.
27.
Berg
,
J.
, and
Lu
,
S.
,
2020
, “
Review of Interfaces for Industrial Human-Robot Interaction
,”
Curr. Rob. Rep.
,
1
(
2
), pp.
27
–
34
.
28.
Thramboulidis
,
K.
,
2016
, “
Comments on ‘Bridging Service-Oriented Architecture and Iec 61499 for Flexibility and Interoperability’
,”
IEEE Trans. Ind. Inform.
,
13
(
4
), pp.
1494
–
1496
.
29.
Tapoglou
,
N.
,
Mehnen
,
J.
,
Vlachou
,
A.
,
Doukas
,
M.
,
Milas
,
N.
, and
Mourtzis
,
D.
,
2015
, “
Cloud-Based Platform for Optimal Machining Parameter Selection Based on Function Blocks and Real-Time Monitoring
,”
ASME J. Manuf. Sci. Eng.
,
137
(
4
), p.
040909
.
30.
Ande
,
S. K.
,
Kuchibotla
,
M. R.
, and
Adavi
,
B. K.
,
2021
, “
Robot Acquisition, Control and Interfacing Using Multimodal Feedback
,”
J. Ambient Intell. Humanized Comput.
,
12
(
3
), pp.
3909
–
3919
.
31.
de Gea Fernández
,
J.
,
Mronga
,
D.
,
Günther
,
M.
,
Knobloch
,
T.
,
Wirkus
,
M.
,
Schröer
,
M.
,
Trampler
,
M.
,
Stiene
,
S.
,
Kirchner
,
E.
,
Bargsten
,
V.
,
Bänziger
,
T.
,
2017
, “
Multimodal Sensor-Based Whole-Body Control for Human–Robot Collaboration in Industrial Settings
,”
Rob. Auton. Syst.
,
94
, pp.
102
–
119
.
32.
Schumacher
,
M.
,
Wojtusch
,
J.
,
Beckerle
,
P.
, and
von Stryk
,
O.
,
2019
, “
An Introductory Review of Active Compliant Control
,”
Rob. Auton. Syst.
,
119
, pp.
185
–
200
.
33.
Abu-Dakka
,
F. J.
, and
Saveriano
,
M.
,
2020
, “
Variable Impedance Control and Learning—A Review
,”
Front. Rob. AI
,
7
, p.
590681
.
34.
Lv
,
Q.
,
Zhang
,
R.
,
Sun
,
X.
,
Lu
,
Y.
, and
Bao
,
J.
,
2021
, “
A Digital Twin-Driven Human-Robot Collaborative Assembly Approach in the Wake of Covid-19
,”
J. Manuf. Syst.
,
60
, pp.
837
–
851
.
35.
Billard
,
A.
, and
Kragic
,
D.
,
2019
, “
Trends and Challenges in Robot Manipulation
,”
Science
,
364
(
6446
), pp.
1
–
10
.
36.
Wang
,
L.
,
Schmidt
,
B.
,
Givehchi
,
M.
, and
Adamson
,
G.
,
2015
, “
Robotic Assembly Planning and Control With Enhanced Adaptability Through Function Blocks
,”
Int. J. Adv. Manuf. Technol.
,
77
(
1–4
), pp.
705
–
715
.
37.
Kardos
,
C.
,
Kovács
,
A.
, and
Váncza
,
J.
,
2020
, “
A Constraint Model for Assembly Planning
,”
J. Manuf. Syst.
,
54
, pp.
196
–
203
.
38.
Liu
,
S.
,
Wang
,
L.
, and
Wang
,
X. V.
,
2021
, “
Sensorless Force Estimation for Industrial Robots Using Disturbance Observer and Neural Learning of Friction Approximation
,”
Rob. Comput. Integrat. Manuf.
,
71
, p.
102168
.
39.
Boonyakitanont
,
P.
,
Lek-Uthai
,
A.
,
Chomtho
,
K.
, and
Songsiri
,
J.
,
2020
, “
A Review of Feature Extraction and Performance Evaluation in Epileptic Seizure Detection Using EEG
,”
Biomed. Signal Process. Control
,
57
, p.
101702
.
40.
Craik
,
A.
,
He
,
Y.
, and
Contreras-Vidal
,
J. L.
,
2019
, “
Deep Learning for Electroencephalogram (eeg) Classification Tasks: A Review
,”
J. Neural. Eng.
,
16
(
3
), p.
031001
.
41.
Wang
,
P.
,
Jiang
,
A.
,
Liu
,
X.
,
Shang
,
J.
, and
Zhang
,
L.
,
2018
, “
Lstm-Based EEG Classification in Motor Imagery Tasks
,”
IEEE. Trans. Neural. Syst. Rehabil. Eng.
,
26
(
11
), pp.
2086
–
2095
.
42.
Zhang
,
S.
,
Tong
,
H.
,
Xu
,
J.
, and
Maciejewski
,
R.
,
2019
, “
Graph Convolutional Networks: A Comprehensive Review
,”
Comput. Soc. Netw.
,
6
(
1
), pp.
1
–
23
.
43.
Wu
,
Z.
,
Pan
,
S.
,
Chen
,
F.
,
Long
,
G.
,
Zhang
,
C.
, and
Philip
,
S. Y.
,
2020
, “
A Comprehensive Survey on Graph Neural Networks
,”
IEEE Trans. Neural Netw. Learn. Syst.
,
32
(
1
), pp.
4
–
24
.
44.
Zhong
,
P.
,
Wang
,
D.
, and
Miao
,
C.
,
2020
, “
EEG-Based Emotion Recognition Using Regularized Graph Neural Networks
,” IEEE Transactions on Affective Computing
45.
Yu
,
B.
,
Yin
,
H.
, and
Zhu
,
Z.
,
2017
, “
Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting
,” arXiv preprint arXiv:1709.04875.
46.
Sun
,
B.
,
Zhang
,
H.
,
Wu
,
Z.
,
Zhang
,
Y.
, and
Li
,
T.
,
2021
, “
Adaptive Spatiotemporal Graph Convolutional Networks for Motor Imagery Classification
,”
IEEE Signal Process. Lett.
,
28
, pp.
219
–
223
.
47.
Jia
,
Z.
,
Lin
,
Y.
,
Wang
,
J.
,
Zhou
,
R.
,
Ning
,
X.
,
He
,
Y.
, and
Zhao
,
Y.
,
2020
, “
Graphsleepnet: Adaptive Spatial-Temporal Graph Convolutional Networks for Sleep Stage Classification
,”
29th International Joint Conference on Artificial Intelligence
,
Yokohama, Japan
, IJCAI, pp.
1324
–
1330
.
48.
Thekumparampil
,
K. K.
,
Wang
,
C.
,
Oh
,
S.
, and
Li
,
L. -J.
,
2018
, “
Attention-Based Graph Neural Network for Semi-Supervised Learning
,” arXiv preprint arXiv:1803.03735.
49.
Wang
,
H.
,
Xu
,
L.
,
Bezerianos
,
A.
,
Chen
,
C.
, and
Zhang
,
Z.
,
2020
, “
Linking Attention-Based Multiscale Cnn With Dynamical Gcn for Driving Fatigue Detection
,”
IEEE Trans. Instrum. Meas.
,
70
, pp.
1
–
11
.
50.
Gao
,
R. X.
, and
Yan
,
R.
,
2011
,
Wavelets: Theory and Applications for Manufacturing
,
Springer
,
New York, Dordrecht, Heidelberg, London
.
51.
Barua
,
S.
,
Ahmed
,
M. U.
,
Ahlström
,
C.
, and
Begum
,
S.
,
2019
, “
Automatic Driver Sleepiness Detection Using EEG, EOG and Contextual Information
,”
Expert. Syst. Appl.
,
115
, pp.
121
–
135
.
52.
Zhang
,
R.
,
Zong
,
Q.
,
Dou
,
L.
, and
Zhao
,
X.
,
2019
, “
A Novel Hybrid Deep Learning Scheme for Four-Class Motor Imagery Classification
,”
J. Neural. Eng.
,
16
(
6
), p.
066004
.
53.
Bashivan
,
P.
,
Rish
,
I.
,
Yeasin
,
M.
, and
Codella
,
N.
,
2015
, “
Learning Representations From EEG With Deep Recurrent-Convolutional Neural Networks
,” arXiv preprint arXiv:1511.06448.
54.
Lawhern
,
V. J.
,
Solon
,
A. J.
,
Waytowich
,
N. R.
,
Gordon
,
S. M.
,
Hung
,
C. P.
, and
Lance
,
B. J.
,
2018
, “
Eegnet: A Compact Convolutional Neural Network for EEG-Based Brain–Computer Interfaces
,”
J. Neural. Eng.
,
15
(
5
), p.
056013
.
55.
Hou
,
Y.
,
Jia
,
S.
,
Zhang
,
S.
,
Lun
,
X.
,
Shi
,
Y.
,
Li
,
Y.
,
Yang
,
H.
,
Zeng
,
R.
, and
Lv
,
J.
,
2020
, “
Deep Feature Mining via Attention-Based Bilstm-GCN for Human Motor Imagery Recognition
,” arXiv preprint arXiv:2005.00777.
56.
Abadi
,
M.
,
Barham
,
P.
,
Chen
,
J.
,
Chen
,
Z.
,
Davis
,
A.
,
Dean
,
J.
,
Devin
,
M.
,
Ghemawat
,
S.
,
Irving
,
G.
,
Isard
,
M.
, and
Kudlur
,
M.
,
2016
, “
“Tensorflow: A System for Large-Scale Machine Learning
,”
12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)
,
Savannah, GA
, pp.
265
–
283
.
57.
Lin
,
G.
,
Kang
,
X.
,
Liao
,
K.
,
Zhao
,
F.
, and
Chen
,
Y.
,
2021
, “
Deep Graph Learning for Semi-Supervised Classification
,”
Pattern Recognit.
,
118
, p.
108039
.
Copyright © 2022 by ASME
You do not currently have access to this content.