Abstract

This study aims to use an improved rotational region convolutional neural network (R2CNN) algorithm to detect the grasping bounding box for the robotic arm that reaches supermarket goods. This algorithm can calculate the final predicted grasping bounding box without any additional architecture, which significantly improves the speed of grasp inferences. In this study, we added the force-closure condition so that the final grasping bounding box could achieve grasping stability in a physical sense. We experimentally demonstrated that deep model-treated object detection and grasping detection are the same tasks. We used transfer learning to improve the prediction accuracy of the grasping bounding box. In particular, the ResNet-101 network weights, which were originally used in object detection, were used to continue training with the Cornell dataset. In terms of grasping detection, we used the trained model weights that were originally used in object detection as the features of the to-be-grasped objects and fed them to the network for continuous training. For 2828 test images, this method achieved nearly 98% accuracy and a speed of 14–17 frames per second.

Graphical Abstract Figure
Graphical Abstract Figure
Close modal

References

1.
Kumbla
,
N. B.
,
Thakar
,
S.
,
Kaipa
,
K. N.
,
Marvel
,
J.
, and
Gupta
,
S. K.
,
2018
, “
Handling Perception Uncertainty in Simulation-Based Singulation Planning for Robotic Bin Picking
,”
ASME J. Comput. Inf. Sci. Eng.
,
18
(
2
), p.
021004
.
2.
Tribaldos
,
J. A.
, and
Sen
,
C.
,
2023
, “
An Evolutionary Approach of Grasp Synthesis for Sheet Metal Parts With Multitype Grippers
,”
ASME J. Comput. Inf. Sci. Eng.
,
23
(
5
), p.
051009
.
3.
Wei
,
W.
,
Li
,
D.
,
Wang
,
P.
,
Li
,
Y.
,
Li
,
W.
,
Luo
,
Y.
, and
Zhong
,
J.
,
2022
, “
DVGG: Deep Variational Grasp Generation for Dextrous Manipulation
,”
IEEE Rob. Autom. Lett.
,
7
(
2
), pp.
1659
1666
.
4.
Correll
,
N.
,
Bekris
,
K. E.
,
Berenson
,
D.
,
Brock
,
O.
,
Causo
,
A.
,
Hauser
,
K.
,
Okada
,
K.
,
Rodriguez
,
A.
,
Romano
,
J. M.
, and
Wurman
,
P. R.
,
2016
, “
Analysis and Observations From the First Amazon Picking Challenge
,”
IEEE Trans. Autom. Sci. Eng.
,
15
(
1
), pp.
172
188
.
5.
Du
,
G.
,
Wang
,
K.
,
Lian
,
S.
, and
Zhao
,
K.
,
2021
, “
Vision-Based Robotic Grasping From Object Localization, Object Pose Estimation to Grasp Estimation for Parallel Grippers: A Review
,”
Artif. Intell. Rev.
,
54
(
3
), pp.
1677
1734
.
6.
Liu
,
Y.
,
Jiang
,
D.
,
Tao
,
B.
,
Qi
,
J.
,
Jiang
,
G.
,
Yun
,
J.
,
Huang
,
L.
,
Tong
,
X.
,
Chen
,
B.
, and
Li
,
G.
,
2022
, “
Grasping Posture of Humanoid Manipulator Based on Target Shape Analysis and Force Closure
,”
Alexandria Eng. J.
,
61
(
5
), pp.
3959
3969
.
7.
Jiang
,
Y.
,
Zhu
,
X.
,
Wang
,
X.
,
Yang
,
S.
,
Li
,
W.
,
Wang
,
H.
,
Fu
,
P.
, and
Luo
,
Z.
,
2017
, “R2CNN: Rotational Region CNN for Orientation Robust Scene Text Detection,” preprint arXiv:1706.09579.
8.
Markenscoff
,
X.
,
Ni
,
L.
, and
Papadimitriou
,
C. H.
,
1990
, “
The Geometry of Grasping
,”
Int. J. Rob. Res.
,
9
(
1
), pp.
61
74
.
9.
Won
,
J.-S.
, and
Robson
,
N.
,
2019
, “
Control-Oriented Finger Kinematic Model: Geometry-Based Approach
,”
J. Mech. Rob.
,
11
(
6
), p.
061007
.
10.
Li
,
J.-W.
,
Liu
,
H.
, and
Cai
,
H.-G.
,
2003
, “
On Computing Three-Finger Force-Closure Grasps of 2-D and 3-D Objects
,”
IEEE Trans. Rob. Autom.
,
19
(
1
), pp.
155
161
.
11.
Ponce
,
J.
,
Stam
,
D.
, and
Faverjon
,
B.
,
1993
, “
On Computing Two-Finger Force-Closure Grasps of Curved 2D Objects
,”
Int. J. Rob. Res.
,
12
(
3
), pp.
263
273
.
12.
Davidson
,
C.
, and
Blake
,
A.
,
1998
, “
Caging Planar Objects With a Three-Finger One-Parameter Gripper
,”
Proceedings of IEEE International Conference on Robotics and Automation
,
Leuven, Belgium
,
May 20
,
Vol. 98, pp. 2722–2727
.
13.
Rimon
,
E.
, and
Blake
,
A.
,
1996
, “
Caging 2D Bodies by 1-Parameter Two-Fingered Gripping Systems
,”
Proceedings of IEEE International Conference on Robotics and Automation
,
Trondheim, Norway
,
Apr. 22–28
,
Vol. 2, pp. 1458–1464
.
14.
Rimon
,
E.
, and
Blake
,
A.
,
1999
, “
Caging Planar Bodies by One-Parameter Two-Fingered Gripping Systems
,”
Int. J. Rob. Res.
,
18
(
3
), pp.
299
318
.
15.
Dong
,
S.
,
Yuan
,
W.
, and
Adelson
,
E. H.
,
2017
, “
Improved Gelsight Tactile Sensor for Measuring Geometry and Slip
,”
Proceedings of IEEE/RSJ International Conference Intellegent Robots and System (IROS)
,
Vancouver, BC, Canada
,
Sept. 24–28
, pp.
137
144
.
16.
Laskey
,
M.
,
Mahler
,
J.
,
McCarthy
,
Z.
,
Pokorny
,
F. T.
,
Patil
,
S.
,
Van Den Berg
,
J.
,
Kragic
,
D.
,
Abbeel
,
P.
, and
Goldberg
,
K.
,
2015
, “
Multi-Armed Bandit Models for 2D Grasp Planning With Uncertainty
,”
Proceedings of IEEE International Conference on Automation Science and Engineering (CASE)
,
Gothenburg, Sweden
,
Aug. 24–28
, pp.
572
579
.
17.
Zapata-Impata
,
B. S.
,
Gil
,
P.
,
Pomares
,
J.
, and
Torres
,
F.
,
2019
, “
Fast Geometry-Based Computation of Grasping Points on Three-Dimensional Point Clouds
,”
Int. J. Adv. Rob. Syst.
,
16
(
1
).
18.
Lei
,
Q.
, and
Wisse
,
M.
,
2016
, “
Object Grasping by Combining Caging and Force Closure
,”
Proceedings of 14th International Conference on Control, Automation, Robotics and Vision (ICARCV)
,
Phuket, Thailand
,
Nov. 13–15
, pp.
1
8
.
19.
Azizi
,
V.
,
Kimmel
,
A.
,
Bekris
,
K.
, and
Kapadia
,
M.
,
2017
, “
Geometric Reachability Analysis for Grasp Planning in Cluttered Scenes for Varying End-Effectors
,”
Proceedings of 13th IEEE Conference on Automation Science and Engineering (CASE)
,
Xi'an, China
,
Aug. 20–23
, pp.
764
769
.
20.
You
,
F.
,
Mende
,
M.
,
Štogl
,
D.
,
Hein
,
B.
, and
Kröger
,
T.
,
2018
, “
Model-Free Grasp Planning for Configurable Vacuum Grippers
,”
Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Madrid, Spain
,
Oct. 1–5
, pp.
4554
4561
.
21.
Kundu
,
O.
, and
Kumar
,
S.
,
2019
, “
A Novel Geometry-Based Algorithm for Robust Grasping in Extreme Clutter Environment
,”
Proceedings of IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)
,
New Delhi, India
,
Oct. 14–18
, pp.
1
8
.
22.
Redmon
,
J.
,
Divvala
,
S.
,
Girshick
,
R.
, and
Farhadi
,
A.
,
2016
, “
You Only Look Once: Unified, Real-Time Object Detection
,”
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
,
Las Vegas, NV
,
June 27–30
, pp.
779
788
.
23.
Redmon
,
J.
, and
Farhadi
,
A.
,
2017
, “
YOLO9000: Better, Faster, Stronger
,”
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
,
Honolulu, HI
,
July 21–26
, pp.
7263
7271
.
24.
Redmon
,
J.
, and
Farhadi
,
A.
,
2018
, “Yolov3: An Incremental Improvement,” preprint arXiv:1804.02767.
25.
Yan
,
B.
,
Fan
,
P.
,
Lei
,
X.
,
Liu
,
Z.
, and
Yang
,
F.
,
2021
, “
A Real-Time Apple Targets Detection Method for Picking Robot Based on Improved YOLOv5
,”
Remote Sens.
,
13
(
9
), p.
1619
.
26.
Liu
,
W.
,
Anguelov
,
D.
,
Erhan
,
D.
,
Szegedy
,
C.
,
Reed
,
S.
,
Fu
,
C.-Y.
, and
Berg
,
A. C.
,
2016
, “
SSD: Single Shot Multibox Detector
,” Computer Vision—ECCV 2016: 14th European Conference, Proceedings, Part I 14, Amsterdam, Oct. 11–14, pp.
21
37
.
27.
Girshick
,
R.
,
Donahue
,
J.
,
Darrell
,
T.
, and
Malik
,
J.
,
2014
, “
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
,”
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
,
Columbus, OH
,
June 23–28
, pp.
580
587
.
28.
Girshick
,
R.
,
2015
, “
Fast R-CNN
,”
Proceedings of IEEE International Conference on Computer Vision (ICCV)
,
Santiago, Chile
,
Dec. 7–13
, pp.
1440
1448
.
29.
Ren
,
S.
,
He
,
K.
,
Girshick
,
R.
, and
Sun
,
J.
,
2015
, “
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
,”
Proceedings of Advances in Neural Information Processing Systems 28 (NIPS)
,
Montreal, Canada
,
Dec. 7–12
.
30.
Redmon
,
J.
, and
Angelova
,
A.
,
2015
, “
Real-Time Grasp Detection Using Convolutional Neural Networks
,”
Proceedings of IEEE International Conference on Robotics and Automation (ICRA)
,
Seattle, WA
,
May 26–30
, pp.
1316
1322
.
31.
Shukla
,
P.
, and
Nandi
,
G.
,
2019
, “
Robotized Grasp: Grasp Manipulation Using Evolutionary Computing
,”
Proceedings of International Conference on Electrical, Electronics and Computer Engineering (UPCON)
,
Aligarh, India
,
Nov. 8–10
, pp.
1
6
.
32.
Zhang
,
S.
,
Guo
,
Z.
,
Huang
,
J.
,
Ren
,
W.
, and
Xia
,
L.
,
2020
, “
Robotic Grasping Position of Irregular Object Based Yolo Algorithm
,”
Proceedings of International Conference on Automation, Control and Robotics Engineering (CACRE)
,
Dalian, China
,
Sept. 19–20
, pp.
1
6
.
33.
Zhao
,
X.
,
Cao
,
Z.
,
Jia
,
Q.
,
Pang
,
L.
,
Yu
,
Y.
, and
Tan
,
M.
,
2018
, “
A Vision-Based Robotic Grasping Approach Under the Disturbance of Obstacles
,”
Proceedings of IEEE International Conference on Mechatronics and Automation (ICMA)
,
Changchun, China
,
Aug. 5–8
, pp.
2175
2179
.
34.
Guo
,
D.
,
Sun
,
F.
,
Kong
,
T.
, and
Liu
,
H.
,
2016
, “
Deep Vision Networks for Real-Time Robotic Grasp Detection
,”
Int. J. Adv. Rob. Syst.
,
14
(
1
), p.
1729881416682706
.
35.
Zeiler
,
M. D.
, and
Fergus
,
R.
,
2014
, “
Visualizing and Understanding Convolutional Networks
,”
Proceedings of Computer Vision–ECCV 2014: 13th European Conference
,
Zurich, Switzerland
,
Sept. 6–12
, pp.
818
833
.
36.
Lin
,
Q.
, and
Chen
,
D.
,
2018
, “
Target Recognition and Optimal Grasping Based on Deep Learning
,”
Proceedings of WRC Symposium on Advanced Robotics and Automation (WRC SARA)
,
Beijing, China
,
Aug. 16
, pp.
1
6
.
37.
He
,
K.
,
Gkioxari
,
G.
,
Dollár
,
P.
, and
Girshick
,
R.
,
2017
, “
Mask R-CNN
,”
Proceedings of IEEE International Conference on Computer Vision (ICCV)
,
Venice, Italy
,
Oct. 22–29
, pp.
2961
2969
.
38.
Karaoguz
,
H.
, and
Jensfelt
,
P.
,
2019
, “
Object Detection Approach for Robot Grasp Detection
,”
Proceedings of International Conference on Robotics and Automation (ICRA)
,
Palais des congres de Montreal, Montreal, Canada
,
May 20–24
, pp.
4953
4959
.
39.
Shah
,
B. K.
,
Kedia
,
V.
,
Raut
,
R.
,
Ansari
,
S.
, and
Shroff
,
A.
,
2020
, “
Evaluation and Comparative Study of Edge Detection Techniques
,”
IOSR J. Comput. Eng.
,
22
(
5
), pp.
6
15
.
40.
Vijai Kumar
,
S.
, and
Vuik
,
C.
,
2021
, “
A Simple and Fast Hole Detection Algorithm for Triangulated Surfaces
,”
ASME J. Comput. Inf. Sci. Eng.
,
21
(
4
), p.
044502
.
41.
Thilagamani
,
S.
, and
Shanthi
,
N.
,
2014
, “
Gaussian and Gabor Filter Approach for Object Segmentation
,”
ASME J. Comput. Inf. Sci. Eng.
,
14
(
2
), p.
021006
.
42.
Simonyan
,
K.
, and
Zisserman
,
A.
,
2014
, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” preprint arXiv:1409.1556.
43.
He
,
K.
,
Zhang
,
X.
,
Ren
,
S.
, and
Sun
,
J.
,
2016
, “
Deep Residual Learning for Image Recognition
,”
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
,
Las Vegas, NV
,
June 27–30
, pp.
770
778
.
44.
Jiang
,
Y.
,
Moseson
,
S.
, and
Saxena
,
A.
,
2011
, “
Efficient Grasping From RGBD Images: Learning Using a New Rectangle Representation
,”
Proceedings of IEEE International Conference on Robotics and Automation
,
Shanghai, China
,
May 9–13
, pp.
3304
3311
.
45.
Lenz
,
I.
,
Lee
,
H.
, and
Saxena
,
A.
,
2015
, “
Deep Learning for Detecting Robotic Grasps
,”
Int. J. Rob. Res.
,
34
(
4–5
), pp.
705
724
.
46.
Watson
,
J.
,
Hughes
,
J.
, and
Iida
,
F.
,
2017
, “
Real-World, Real-Time Robotic Grasping With Convolutional Neural Networks
,” Towards Autonomous Robotic Systems: 18th Annual Conference, TAROS 2017, Proceedings 18, Guildford, UK, July 19–21, pp.
617
626
.
47.
Asif
,
U.
,
Bennamoun
,
M.
, and
Sohel
,
F. A.
,
2017
, “
RGB-D Object Recognition and Grasp Detection Using Hierarchical Cascaded Forests
,”
IEEE Trans. Rob.
,
33
(
3
), pp.
547
564
.
48.
Wang
,
Z.
,
Li
,
Z.
,
Wang
,
B.
, and
Liu
,
H.
,
2016
, “
Robot Grasp Detection Using Multimodal Deep Convolutional Neural Networks
,”
Adv. Mech. Eng.
,
8
(
9
), p.
1687814016668077
.
49.
Song
,
Y.
,
Gao
,
L.
,
Li
,
X.
, and
Shen
,
W.
,
2020
, “
A Novel Robotic Grasp Detection Method Based on Region Proposal Networks
,”
Rob. Comput. Integr. Manuf.
,
65
, p.
101963
.
50.
Zhang
,
H.
,
Lan
,
X.
,
Bai
,
S.
,
Zhou
,
X.
,
Tian
,
Z.
, and
Zheng
,
N.
,
2019
, “
ROI-Based Robotic Grasp Detection for Object Overlapping Scenes
,”
Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Macau, China
,
Nov. 3–8
, pp.
4768
4775
.
51.
Luo
,
Z.
,
Tang
,
B.
,
Jiang
,
S.
,
Pang
,
M.
, and
Xiang
,
K.
,
2020
, “
Grasp Detection Based on Faster Region CNN
,”
Proceedings of International Conference on Advanced Robotics and Mechatronics (ICARM)
,
Shenzhen, China
,
Dec. 18–21
, pp.
1
6
.
You do not currently have access to this content.