Abstract

In human–robot collaboration, robots and humans must work together in shared, overlapping, workspaces to accomplish tasks. If human and robot motion can be coordinated, then collisions between robot and human can seamlessly be avoided without requiring either of them to stop work. A key part of this coordination is anticipating humans’ future motion so robot motion can be adapted proactively. In this work, a generative neural network predicts a multi-step sequence of human poses for tabletop reaching motions. The multi-step sequence is mapped to a time-series based on a human speed versus motion distance model. The input to the network is the human’s reaching target relative to current pelvis location combined with current human pose. A dataset was generated of human motions to reach various positions on or above the table in front of the human starting from a wide variety of initial human poses. After training the network, experiments showed that the predicted sequences generated by this method matched the actual recordings of human motion within an L2 joint error of 7.6 cm and L2 link roll–pitch–yaw error of 0.301 rad on average. This method predicts motion for an entire reach motion without suffering from the exponential propagation of prediction error that limits the horizon of prior works.

Graphical Abstract Figure
Graphical Abstract Figure
Close modal

References

1.
Zhang
,
J.
,
Liu
,
H.
,
Chang
,
Q.
,
Wang
,
L.
, and
Gao
,
R. X.
,
2020
, “
Recurrent Neural Network for Motion Trajectory Prediction in Human-Robot Collaborative Assembly
,”
CIRP Ann. Manuf. Technol.
,
69
(
1
), pp.
9
12
.
2.
Liu
,
H.
, and
Wang
,
L.
,
2017
, “
Human Motion Prediction for Human–Robot Collaboration
,”
ASME J. Manuf. Syst.
,
44
(
2
), pp.
287
294
.
3.
Maeda
,
G.
,
Ewerton
,
M.
,
Lioutikov
,
R.
,
Ben Amor
,
H.
,
Peters
,
J.
, and
Neumann
,
G.
,
2014
, “
Learning Interaction for Collaborative Tasks With Probabilistic Movement Primitives
,”
IEEE/RAS International Conference on Humanoid Robots
,
Madrid, Spain
,
Nov. 18–20
, pp.
527
534
.
4.
Maeda
,
G.
,
Neumann
,
G.
,
Ewerton
,
M.
,
Lioutikov
,
R.
,
Kroemer
,
O.
, and
Peters
,
J.
,
2017
, “
Probabilistic Movement Primitives for Coordination of Multiple Human-Robot Collaborative Tasks
,”
Auton. Rob.
,
41
(
3
), pp.
593
612
.
5.
Mainprice
,
J.
, and
Berenson
,
D.
,
2013
, “
Human-Robot Collaborative Manipulation Planning Using Early Prediction of Human Motion
,”
IEEE/RSJ International Conference on Intelligent Robots and Systems
,
Tokyo, Japan
,
Nov. 3–8
, pp.
299
306
.
6.
Wang
,
Y.
,
Sheng
,
Y.
,
Wang
,
J.
, and
Zhang
,
W.
,
2018
, “
Optimal Collision-Free Robot Trajectory Generation Based on Time Series Prediction of Human Motion
,”
IEEE Rob. Autom. Lett.
,
3
(
1
), pp.
226
233
.
7.
Kanazawa
,
A.
,
Kinugawa
,
J.
, and
Kosuge
,
K.
,
2019
, “
Adaptive Motion Planning for a Collaborative Robot Based on Prediction Uncertainty to Enhance Human Safety and Work Efficiency
,”
IEEE Trans. Rob.
,
35
(
4
), pp.
817
832
.
8.
Liu
,
R.
, and
Liu
,
C.
,
2021
, “
Human Motion Prediction Using Adaptable Recurrent Neural Networks and Inverse Kinematics
,”
IEEE Contr. Syst. Lett.
,
5
(
5
), pp.
1651
1656
.
9.
Li
,
Q.
,
Zhang
,
Z.
,
You
,
Y.
,
Mu
,
Y.
, and
Feng
,
C.
,
2020
, “
Data Driven Models for Human Motion Prediction in Human-Robot Collaboration
,”
IEEE Access
,
8
, pp.
227690
227702
.
10.
Callens
,
T.
,
der Have
,
T. V.
,
Rossom
,
S. V.
,
Schutter
,
J. D.
, and
Aertbeliën
,
E.
,
2020
, “
A Framework for Recognition and Prediction of Human Motions in Human-Robot Collaboration Using Probabilistic Motion Models
,”
IEEE Rob. Autom. Lett.
,
5
(
4
), pp.
5151
5158
.
11.
Martinez
,
J.
,
Black
,
M. J.
, and
Romero
,
J.
,
2017
, “
On Human Motion Prediction Using Recurrent Neural Networks
,” IEEE CVPR, pp.
4674
4683
.
12.
Mao
,
W.
,
Liu
,
M.
,
Salzmann
,
M.
, and
Li
,
H.
,
2019
, “
Learning Trajectory Dependencies for Human Motion Prediction
,” IEEE/CVF International Conference on Computer Vision (ICCV), pp.
9488
9496
.
13.
Li
,
C.
,
Zhang
,
Z.
,
Lee
,
W. S.
, and
Lee
,
G. H.
,
2018
, “
Convolutional Sequence to Sequence Model for Human Dynamics
,”
IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
,
Salt Lake City, UT
,
June 18–23
.
14.
Goodfellow
,
I. J.
,
Pouget-Abadie
,
J.
,
Mirza
,
M.
,
Xu
,
B.
,
Warde-Farley
,
D.
,
Ozair
,
S.
,
Courville
,
A. C.
, and
Bengio
,
Y.
,
2014
, “
Generative Adversarial Nets
,”
Conference on Neural Information Processing Systems
,
Montreal, Canada
,
Dec. 8–13
.
15.
Radford
,
A.
,
Metz
,
L.
, and
Chintala
,
S.
,
2016
, “
Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks
,”
International Conference on Learning Representations
,
San Juan, Puerto Rico
,
May 2–4
.
16.
Papadaki
,
A.
, and
Pateraki
,
M.
,
2023
, “
6D Object Localization in Car-Assembly Industrial Environment
,”
J. Imag.
,
9
(
3
), pp.
72
94
.
17.
Flowers
,
J. T.
, and
Wiens
,
G. J.
,
2022
, “
Comparison of Human Skeleton Trackers Paired With a Novel Skeleton Fusion Algorithm
,”
Manufacturing Science and Engineering Conference
,
West Lafayette, IN
,
June 27–July 1
.
18.
Pellegrinelli
,
S.
,
Moro
,
F. L.
,
Pedrocchi
,
N.
,
Molinari Tosatti
,
L.
, and
Tolio
,
T.
,
2016
, “
A Probabilistic Approach to Workspace Sharing for Human–Robot Cooperation in Assembly Tasks
,”
CIRP Ann.
,
65
(
1
), pp.
57
60
.
19.
Salvador
,
S.
, and
Chan
,
P.
,
2007
, “
FastDTW: Toward Accurate Dynamic Time Warping in Linear Time and Space
,”
Intell. Data Anal.
,
11
(
5
), pp.
561
580
.
20.
Dumoulin
,
V.
, and
Visin
,
F.
,
2018
, “
A Guide to Convolution Arithmetic for Deep Learning
,”
arXiv
. https://arxiv.org/abs/1603.07285
21.
Ioffe
,
S.
, and
Szegedy
,
C.
,
2015
, “
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
,”
International Conference on Machine Learning
,
Lille, France
,
July 7–9
, pp.
448
456
.
22.
Klambauer
,
G.
,
Unterthiner
,
T.
,
Mayr
,
A.
, and
Hochreiter
,
S.
,
2017
, “
Self-Normalizing Neural Networks
,”
Conference on Neural Information Processing Systems
,
Long Beach, CA
,
Dec. 4–9
.
23.
Kingma
,
D. P.
, and
Ba
,
J.
,
2015
, “
Adam: A Method for Stochastic Optimization
,” International Conference on Learning Representations, San Diego, CA.
24.
Flowers
,
J.
, and
Wiens
,
G.
,
2023
, “
A Spatio-Temporal Prediction and Planning Framework for Proactive Human–Robot Collaboration
,”
ASME J. Manuf. Sci. Eng.
,
145
(
12
), p.
121011
.
You do not currently have access to this content.