Abstract

Current manual practices of replacing bolts on structures are time-consuming and costly, especially because of numerous bolts. Thus, an automated method that can visually detect and localize bolt positions would be highly beneficial. We demonstrate the use of deep neural networks using domain randomization for detecting and localizing bolts on a workpiece. In contrast to previous approaches that require training on real images, the use of domain randomization enables all training in simulation. The key idea is to create a wide variety of computer-generated synthetic images by varying the texture, color, camera position and orientation, distractor objects, and noise, and train the neural network on these images such that the neural network is robust to scene variability and hence provides accurate results when deployed on real images. Using domain randomization, we train two neural networks, a faster regional convolutional neural network for detecting the bolt and placing a bounding box, and a regression convolutional neural network for estimating the x- and y-position of the bolts relative to the coordinates fixed to the workpiece. Our results indicate that in the best case, we can detect bolts with 85% accuracy and can predict 75% of bolts within 1.27 cm accuracy. The novelty of this work is in using domain randomization to detect and localize: (1) multiples of a single object and (2) small-sized objects (0.6 cm × 2.5 cm).

References

References
1.
Li
,
C.
,
Wei
,
Z.
, and
Xing
,
J.
,
2016
, “
Online Inspection System for the Automatic Detection of Bolt Defects on a Freight Train
,”
Proc. Inst. Mech. Eng., Part F: J. Rail Rapid Transit
,
230
(
4
), pp.
1213
1226
. 10.1177/0954409715588119
2.
Reynolds
,
W. D.
,
Doyle
,
D.
, and
Arritt
,
B.
,
2010
, “
Active Loose Bolt Detection in a Complex Satellite Structure
,”
SPIE Optics and Photonics
,
San Diego, CA, Aug. 2–4
.
3.
Henderson
,
S. J.
, and
Feiner
,
S.
,
2009
, “
Evaluating the Benefits of Augmented Reality for Task Localization in Maintenance of an Armored Personnel Carrier Turret
,”
8th IEEE International Symposium on Mixed and Augmented Reality
,
Orlando, FL
, Oct. 19–22, IEEE, pp.
135
144
.
4.
Baek
,
G. R.
,
Mo
,
Y. H.
,
Jeong
,
J. S.
,
Park
,
J. M.
, and
Lim
,
M. T.
,
2011
, “
Cognition System of Bolt Hole Using Template Matching
,”
28th International Symposium on Automation and Robotics in Construction, ISARC 2011
,
Seoul, South Korea, June 29–July 2
.
5.
Nagarajan
,
P.
,
Perumaal
,
S. S.
, and
Yogameena
,
B.
,
2016
, “
Vision Based Pose Estimation of Multiple Peg-in-Hole for Robotic Assembly
,”
International Conference on Computer Vision, Graphics, and Image Processing
,
Guwahati, India
, Dec. 19, Springer, pp.
50
62
.
6.
Choe
,
Y.
,
Lee
,
H.-C.
,
Kim
,
Y.-J.
,
Hong
,
D.-H.
,
Park
,
S.-S.
, and
Lim
,
M.-T.
,
2009
, “
Vision-Based Estimation of Bolt-Hole Location Using Circular Hough Transform
,”
ICCAS-SICE, 2009
,
Fukuoka, Japan
, Aug. 18–21, IEEE, pp.
4821
4826
.
7.
Knepper
,
R. A.
,
Layton
,
T.
,
Romanishin
,
J.
, and
Rus
,
D.
,
2013
, “
Ikeabot: An Autonomous Multi-Robot Coordinated Furniture Assembly System
,”
2013 IEEE International Conference on Robotics and Automation
,
Karlsruhe, Germany
, May 6–10, IEEE, pp.
855
862
.
8.
Miller
,
J. M.
, and
Hoffman
,
R. L.
,
1989
, “
Automatic Assembly Planning with Fasteners
,”
1989 IEEE International Conference on Robotics and Automation
,
Scottsdale, AZ
, May 14–19, IEEE, pp.
69
74
.
9.
Rusu
,
R. B.
,
Bradski
,
G.
,
Thibaux
,
R.
, and
Hsu
,
J.
,
2010
, “
Fast 3d Recognition and Pose Using the Viewpoint Feature Histogram
,”
2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Taipei, Taiwan
, Oct. 18–22, IEEE, pp.
2155
2162
.
10.
Zhang
,
Z.
,
2012
, “
Microsoft Kinect Sensor and Its Effect
,”
IEEE Multimedia
,
19
(
2
), pp.
4
10
. 10.1109/MMUL.2012.24
11.
Collet
,
A.
, and
Srinivasa
,
S. S.
,
2010
, “
Efficient Multi-View Object Recognition and Full Pose Estimation
,”
2010 IEEE International Conference on Robotics and Automation (ICRA)
,
Anchorage, AK
, May 3–8, IEEE, pp.
2050
2055
.
12.
Gopalan
,
R.
,
Li
,
R.
, and
Chellappa
,
R.
,
2011
, “
Domain Adaptation for Object Recognition: An Unsupervised Approach
,”
2011 IEEE International Conference on Computer Vision (ICCV)
,
Barcelona, Spain
, Nov. 6–13, IEEE, pp.
999
1006
.
13.
Matasci
,
G.
,
Tuia
,
D.
, and
Kanevski
,
M.
,
2012
, “
Svm-Based Boosting of Active Learning Strategies for Efficient Domain Adaptation
,”
IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens.
,
5
(
5
), pp.
1335
1343
. 10.1109/JSTARS.2012.2202881
14.
Tobin
,
J.
,
Fong
,
R.
,
Ray
,
A.
,
Schneider
,
J.
,
Zaremba
,
W.
, and
Abbeel
,
P.
,
2017
, “
Domain Randomization for Transferring Deep Neural Networks From Simulation to the Real World
,”
2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Vancouver, Canada
, Sept. 24–28, IEEE, pp.
23
30
.
15.
Bousmalis
,
K.
,
Irpan
,
A.
,
Wohlhart
,
P.
,
Bai
,
Y.
,
Kelcey
,
M.
,
Kalakrishnan
,
M.
,
Downs
,
L.
,
Ibarz
,
J.
,
Pastor
,
P.
,
Konolige
,
K.
,
Levine
,
S.
, and
Vanhoucke
,
V.
,
2018
, “
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping
,”
2018 IEEE International Conference on Robotics and Automation (ICRA)
,
Paris, France
, May 21–25, IEEE, pp.
4243
4250
.
16.
Tremblay
,
J.
,
Prakash
,
A.
,
Acuna
,
D.
,
Brophy
,
M.
,
Jampani
,
V.
,
Anil
,
C.
,
To
,
T.
,
Cameracci
,
E.
,
Boochoon
,
S.
, and
Birchfield
,
S.
,
2018
, “
Training Deep Networks With Synthetic Data: Bridging the Reality Gap by Domain Randomization
,”
2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops
,
Salt Lake City, UT, June 18–22
, pp.
969
977
.
17.
Tobin
,
J.
,
Zaremba
,
W.
, and
Abbeel
,
P.
,
2017
,
Domain Randomization and Generative Models for Robotic Grasping
, .
18.
Borrego
,
J.
,
Dehban
,
A.
,
Figueiredo
,
R.
,
Moreno
,
P.
,
Bernardino
,
A.
, and
Santos-Victor
,
J.
,
2018
,
Applying Domain Randomization to Synthetic Data for Object Category Detection
, .
19.
Peng
,
X. B.
,
Andrychowicz
,
M.
,
Zaremba
,
W.
, and
Abbeel
,
P.
,
2018
, “
Sim-to-Real Transfer of Robotic Control With Dynamics Randomization
,”
2018 IEEE International Conference on Robotics and Automation (ICRA)
,
Brisbane, Australia
, May 21–25, IEEE, pp.
1
8
.
20.
Tan
,
J.
,
Zhang
,
T.
,
Coumans
,
E.
,
Iscen
,
A.
,
Bai
,
Y.
,
Hafner
,
D.
,
Bohez
,
S.
, and
Vanhoucke
,
V.
,
2018
,
>Sim-to-Real: Learning Agile Locomotion for Quadruped Robots
.
21.
Mordatch
,
I.
,
Lowrey
,
K.
, and
Todorov
,
E.
,
2015
, “
Ensemble-CIO: Full-Body Dynamic Motion Planning That Transfers to Physical Humanoids
,”
2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
,
Hamburg, Germany
, Sept. 28–Oct. 2, IEEE, pp.
5307
5314
.
22.
Chen
,
F.-C.
, and
Jahanshahi
,
M. R.
,
2017
, “
NB-CNN: Deep Learning-Based Crack Detection Using Convolutional Neural Network and Naïve Bayes Data Fusion
,”
IEEE Trans. Ind. Electron.
,
65
(
5
), pp.
4392
4400
. 10.1109/TIE.2017.2764844
23.
Lin
,
H.
,
Li
,
B.
,
Wang
,
X.
,
Shu
,
Y.
, and
Niu
,
S.
,
2019
, “
Automated Defect Inspection of Led Chip Using Deep Convolutional Neural Network
,”
J. Intell. Manuf.
,
30
(
6
), pp.
2525
2534
. 10.1007/s10845-018-1415-x
24.
Ince
,
T.
,
Kiranyaz
,
S.
,
Eren
,
L.
,
Askar
,
M.
, and
Gabbouj
,
M.
,
2016
, “
Real-Time Motor Fault Detection by 1-d Convolutional Neural Networks
,”
IEEE Trans. Ind. Electron.
,
63
(
11
), pp.
7067
7075
. 10.1109/TIE.2016.2582729
25.
Tsai
,
I.-S.
, and
Hu
,
M.-C.
,
1996
, “
Automatic Inspection of Fabric Defects Using an Artificial Neural Network Technique
,”
Text. Res. J.
,
66
(
7
), pp.
474
482
. 10.1177/004051759606600710
26.
Ko
,
K. W.
, and
Cho
,
H. S.
,
2000
, “
Solder Joints Inspection Using a Neural Network and Fuzzy Rule-Based Classification Method
,”
IEEE Trans. Electron. Packaging Manuf.
,
23
(
2
), pp.
93
103
. 10.1109/6104.846932
27.
Zheng
,
S.-j.
,
Li
,
Z.-q.
, and
Wang
,
H.-t.
,
2011
, “
A Genetic Fuzzy Radial Basis Function Neural Network for Structural Health Monitoring of Composite Laminated Beams
,”
Expert Syst. Appl.
,
38
(
9
), pp.
11837
11842
. 10.1016/j.eswa.2011.03.072
28.
Ameperosa
,
E.
, and
Bhounsule
,
P. A.
,
2019
, “
Domain Randomization for Detection and Position Estimation of Multiples of a Single Object With Applications to Localizing Bolts on Structures
,”
ASME 2019 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Anaheim, CA, Aug. 18–21
.
29.
Australian Government
,
2018
,
Airbus a380-842 Horizontal Stabiliser Structure—Bushing Migrated
.
30.
Ren
,
S.
,
He
,
K.
,
Girshick
,
R.
, and
Sun
,
J.
,
2015
, “
Faster R-CNN: Towards Real-Time Object Detection With Region Proposal Networks
,”
Advances in Neural Information Processing System
,
Montreal, Canada
.
31.
Russakovsky
,
O.
,
Deng
,
J.
,
Su
,
H.
,
Krause
,
J.
,
Satheesh
,
S.
,
Ma
,
S.
,
Huang
,
Z.
,
Karpathy
,
A.
,
Khosla
,
A.
,
Bernstein
,
M.
,
Berg
,
A. C.
, and
Fei-Fei
,
L.
,
2015
, “
Imagenet Large Scale Visual Recognition Challenge
,”
Int. J. Comput. Vis.
,
115
(
3
), pp.
211
252
. 10.1007/s11263-015-0816-y
You do not currently have access to this content.