Abstract

We present a new data generation method to facilitate an automatic machine interpretation of 2D engineering part drawings. While such drawings are a common medium for clients to encode design and manufacturing requirements, a lack of computer support to automatically interpret these drawings necessitates part manufacturers to resort to laborious manual approaches for interpretation which, in turn, severely limits processing capacity. Although recent advances in trainable computer vision methods may enable automatic machine interpretation, it remains challenging to apply such methods to engineering drawings due to a lack of labeled training data. As one step toward this challenge, we propose a constrained data synthesis method to generate an arbitrarily large set of synthetic training drawings using only a handful of labeled examples. Our method is based on the randomization of the dimension sets subject to two major constraints to ensure the validity of the synthetic drawings. The effectiveness of our method is demonstrated in the context of a binary component segmentation task with a proposed list of descriptors. An evaluation of several image segmentation methods trained on our synthetic dataset shows that our approach to new data generation can boost the segmentation accuracy and the generalizability of the machine learning models to unseen drawings.

References

1.
Fonseca
,
M. J.
,
Ferreira
,
A.
, and
Jorge
,
J. A.
,
2005
, “
Content-Based Retrieval of Technical Drawings
,”
Int. J. Comput. Appl. Technol.
,
23
(
2–4
), pp.
86
100
.
2.
Kasimov
,
Denis R.
,
Kuchuganov
,
Aleksandr V.
, and
Kuchuganov
,
Valeriy N.
,
2015
, “
Individual Strategies in the Tasks of Graphical Retrieval of Technical Drawings
,”
J. Vis. Lang. Comput.
,
28
, pp.
134
146
.
3.
Sajadfar
,
N.
, and
Ma
,
Y.
,
2015
, “
A Hybrid Cost Estimation Framework Based on Feature-Oriented Data Mining Approach
,”
Adv. Eng. Inform.
,
29
(
3
), pp.
633
647
.
4.
Kulkarni
,
P.
,
Marsan
,
A.
, and
Dutta
,
D.
,
2000
, “
A Review of Process Planning Techniques in Layered Manufacturing
,”
Rapid Prototyp. J.
,
6
, pp.
18
35
.
5.
Mitsubishi UFJ Research & Consulting Co., L.
,
2019
, “
A Survey on Projects and Issues in Japan’s Manufacturing Industry
,” https://www.meti.go.jp/meti˙lib/report/2020FY/000066.pdf
6.
Redmon
,
J.
,
Divvala
,
S.
,
Girshick
,
R.
, and
Farhadi
,
A.
,
2016
, “
You Only Look Once: Unified, Real-Time Object Detection
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Las Vegas, NV
,
June 26–July 1
, pp.
779
788
.
7.
He
,
K.
,
Gkioxari
,
G.
,
Dollár
,
P.
, and
Girshick
,
R.
,
2017
, “
Mask R-CNN
,”
Proceedings of the IEEE International Conference on Computer Vision
,
Venice
,
Oct. 22–29
, pp.
2961
2969
.
8.
Wang
,
J.
,
Sun
,
K.
,
Cheng
,
T.
,
Jiang
,
B.
,
Deng
,
C.
,
Zhao
,
Y.
,
Liu
,
D.
, et al.,
2020
, “
Deep High-Resolution Representation Learning for Visual Recognition
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
43
(
10
), pp.
3349
3364
.
9.
Long
,
J.
,
Shelhamer
,
E.
, and
Darrell
,
T.
,
2015
, “
Fully Convolutional Networks for Semantic Segmentation
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Boston, MA
,
June 8–10
, pp.
3431
3440
.
10.
Ronneberger
,
O.
,
Fischer
,
P.
, and
Brox
,
T.
,
2015
, “
U-net: Convolutional Networks for Biomedical Image Segmentation
,”
International Conference on Medical Image Computing and Computer-Assisted Intervention
,
Munich, Germany
,
Oct. 5–9
, Springer, pp.
234
241
.
11.
Chen
,
L.-C.
,
Zhu
,
Y.
,
Papandreou
,
G.
,
Schroff
,
F.
, and
Adam
,
H.
,
2018
, “
Encoder–Decoder With Atrous Separable Convolution for Semantic Image Segmentation
,”
Proceedings of the European Conference on Computer Vision (ECCV)
,
Munich, Germany
,
Sept. 8–14
, pp.
801
818
.
12.
Li
,
L. H.
,
Yatskar
,
M.
,
Yin
,
D.
,
Hsieh
,
C.-J.
, and
Chang
,
K.-W.
,
2019
, “
VisualBERT: A Simple and Performant Baseline for Vision and Language
,”
arXiv preprint
. https://arxiv.org/abs/1908.03557
13.
Jiang
,
H.
,
Misra
,
I.
,
Rohrbach
,
M.
,
Learned-Miller
,
E.
, and
Chen
,
X.
,
2020
, “
In Defense of Grid Features for Visual Question Answering
,”
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
,
Virtual
,
June 14–19
, pp.
10267
10276
.
14.
Wang
,
W.
,
Bao
,
H.
,
Dong
,
L.
, and
Wei
,
F.
,
2022
, “
Vlmo: Unified Vision-Language Pre-Training With Mixture-of-Modality-Experts
,”
2022 Conference on Neural Information Processing Systems
,
New Orleans, LA
,
Nov. 28–Dec. 9
.
15.
Kang
,
G.
,
Dong
,
X.
,
Zheng
,
L.
, and
Yang
,
Y.
,
2017
, “
Patchshuffle Regularization
,” preprint arXiv:1707.07103.
16.
Zhong
,
Z.
,
Zheng
,
L.
,
Kang
,
G.
,
Li
,
S.
, and
Yang
,
Y.
,
2020
, “
Random Erasing Data Augmentation
,”
Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34
,
New York
,
Feb. 7–12
, pp.
13001
13008
.
17.
Chatfield
,
K.
,
Simonyan
,
K.
,
Vedaldi
,
A.
, and
Zisserman
,
A.
,
2014
, “
Return of the Devil in the Details: Delving Deep Into Convolutional Nets
,” preprint arXiv:1405.3531.
18.
Inoue
,
H.
,
2018
, “
Data Augmentation by Pairing Samples for Images Classification
,” preprint arXiv:1801.02929.
19.
Dosovitskiy
,
A.
,
Ros
,
G.
,
Codevilla
,
F.
,
Lopez
,
A.
, and
Koltun
,
V.
,
2017
, “
Carla: An Open Urban Driving Simulator
,”
Conference on Robot Learning
,
Mountain View, CA
,
Nov. 13–15
, pp.
1
16
.
20.
Smolyakov
,
M.
,
Frolov
,
A.
,
Volkov
,
V.
, and
Stelmashchuk
,
I.
,
2018
, “
Self-Driving Car Steering Angle Prediction Based on Deep Neural Network an Example of Carnd Udacity Simulator
,”
2018 IEEE 12th International Conference on Application of Information and Communication Technologies (AICT)
,
Almaty, Kazakhstan
,
Oct. 17–19
, pp.
1
5
.
21.
Lukač
,
D.
,
2018
, “
Simulation of a Pick-and-Place Cube Robot by Means of the Simulation Software Kuka Sim Pro
,”
2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)
,
Opatija, Croatia
,
May 21–25
, pp.
0846
0849
.
22.
Ummenhofer
,
B.
,
Prantl
,
L.
,
Thuerey
,
N.
, and
Koltun
,
V.
,
2020
, “
Lagrangian Fluid Simulation With Continuous Convolutions
,”
International Conference on Learning Representations
,
Addis Ababa, Ethiopia
,
Apr. 26–30
.
23.
Kashefi
,
A.
,
Rempe
,
D.
, and
Guibas
,
L. J.
,
2021
, “
A Point-Cloud Deep Learning Framework for Prediction of Fluid Flow Fields on Irregular Geometries
,”
Phys. Fluids
,
33
(
2
), p.
027104
.
24.
Rasp
,
S.
,
Dueben
,
P. D.
,
Scher
,
S.
,
Weyn
,
J. A.
,
Mouatadid
,
S.
, and
Thuerey
,
N.
,
2020
, “
Weatherbench: A Benchmark Data Set for Data-Driven Weather Forecasting
,”
J. Adv. Model. Earth Syst.
,
12
(
11
), p.
e2020MS002203
.
25.
Kohler
,
R.
,
1981
, “
A Segmentation System Based on Thresholding
,”
Comput. Graph. Image Process.
,
15
(
4
), pp.
319
338
.
26.
Wang
,
X.-Y.
,
Zhang
,
X.-J.
,
Yang
,
H.-Y.
, and
Bu
,
J.
,
2012
, “
A Pixel-Based Color Image Segmentation Using Support Vector Machine and Fuzzy C-Means
,”
Neural Netw.
,
33
, pp.
148
159
.
27.
Yang
,
Y.
,
Hallman
,
S.
,
Ramanan
,
D.
, and
Fowlkes
,
C. C.
,
2011
, “
Layered Object Models for Image Segmentation
,”
IEEE Trans. Pattern Anal. Mach. Intell.
,
34
(
9
), pp.
1731
1743
.
28.
Chen
,
C. W.
,
Luo
,
J.
, and
Parker
,
K. J.
,
1998
, “
Image Segmentation Via Adaptive K-Mean Clustering and Knowledge-Based Morphological Operations With Biomedical Applications
,”
IEEE Trans. Image Process.
,
7
(
12
), pp.
1673
1683
.
29.
Dhanachandra
,
N.
,
Manglem
,
K.
, and
Chanu
,
Y. J.
,
2015
, “
Image Segmentation Using K-Means Clustering Algorithm and Subtractive Clustering Algorithm
,”
Procedia Comput. Sci.
,
54
, pp.
764
771
.
30.
Shotton
,
J.
,
Winn
,
J.
,
Rother
,
C.
, and
Criminisi
,
A.
,
2009
, “
Textonboost for Image Understanding: Multi-Class Object Recognition and Segmentation by Jointly Modeling Texture, Layout, and Context
,”
Int. J. Comput. Vision
,
81
(
1
), pp.
2
23
.
31.
Krähenbühl
,
P.
, and
Koltun
,
V.
,
2011
, “
Efficient Inference in Fully Connected CRFs With Gaussian Edge Potentials
,”
Adv. Neural Inf. Process. Syst.
,
24
, pp.
109
117
.
32.
Chen
,
L.-C.
,
Papandreou
,
G.
,
Kokkinos
,
I.
,
Murphy
,
K.
, and
Yuille
,
A. L.
,
2014
, “
Semantic Image Segmentation With Deep Convolutional Nets and Fully Connected CRFS
,”
IEEE Trans. Patt. Anal. Mach. Intell.
,
40
(
4
), pp.
834
848
.
33.
Zheng
,
S.
,
Jayasumana
,
S.
,
Romera-Paredes
,
B.
,
Vineet
,
V.
,
Su
,
Z.
,
Du
,
D.
,
Huang
,
C.
, and
Torr
,
P. H.
,
2015
, “
Conditional Random Fields as Recurrent Neural Networks
,”
Proceedings of the IEEE International Conference on Computer Vision
,
Santiago, Chile
,
Dec. 13–16
, pp.
1529
1537
.
34.
Chandra
,
S.
, and
Kokkinos
,
I.
,
2016
, “
Fast, Exact and Multi-Scale Inference for Semantic Image Segmentation With Deep Gaussian CRFs
,”
European Conference on Computer Vision
,
Amsterdam, The Netherlands
,
Oct. 8–16
, Springer, pp.
402
418
.
35.
Lakshman Naika
,
R.
,
Dinesh
,
R.
, and
Prabhanjan
,
S.
,
2019
, “
Handwritten Electric Circuit Diagram Recognition: An Approach Based on Finite State Machine
,”
Int. J. Mach. Learn. Comput.
,
9
(
3
), pp.
374
380
.
36.
Feng
,
G.
,
Viard-Gaudin
,
C.
, and
Sun
,
Z.
,
2009
, “
On-Line Hand-Drawn Electric Circuit Diagram Recognition Using 2d Dynamic Programming
,”
Pattern Recognit.
,
42
(
12
), pp.
3215
3223
.
37.
Schäfer
,
B.
,
Keuper
,
M.
, and
Stuckenschmidt
,
H.
,
2021
, “
Arrow R-CNN for Handwritten Diagram Recognition
,”
Int. J. Doc. Anal. Recognit.
,
24
(
1
), pp.
3
17
.
38.
Delalandre
,
M.
,
Valveny
,
E.
,
Pridmore
,
T.
, and
Karatzas
,
D.
,
2010
, “
Generation of Synthetic Documents for Performance Evaluation of Symbol Recognition & Spotting Systems
,”
Int. J. Doc. Anal. Recognit.
,
13
(
3
), pp.
187
207
.
39.
Kara
,
L. B.
,
Gennari
,
L.
, and
Stahovich
,
T. F.
,
2008
, “
A Sketch-Based Tool for Analyzing Vibratory Mechanical Systems
,”
ASME J. Mech. Des.
,
130
(
10
), p.
101101
.
40.
Lu
,
T.
,
Yang
,
H.
,
Yang
,
R.
, and
Cai
,
S.
,
2007
, “
Automatic Analysis and Integration of Architectural Drawings
,”
Int. J. Doc. Anal. Recognit.
,
9
, pp.
31
47
.
41.
Kang
,
S.-O.
,
Lee
,
E.-B.
, and
Baek
,
H.-K.
,
2019
, “
A Digitization and Conversion Tool for Imaged Drawings to Intelligent Piping and Instrumentation Diagrams (P&ID)
,”
Energies
,
12
(
13
), p.
2593
.
42.
Ouyang
,
T. Y.
, and
Davis
,
R.
,
2011
, “
Chemink: A Natural Real-Time Recognition System for Chemical Drawings
,”
Proceedings of the 16th International Conference on Intelligent User Interfaces
,
Palo Alto, CA
,
Feb. 13–16
, pp.
267
276
.
43.
Weber
,
M.
,
Liwicki
,
M.
, and
Dengel
,
A.
,
2010
, “
A. Scatch-a Sketch-Based Retrieval for Architectural Floor Plans
,”
2010 12th International Conference on Frontiers in Handwriting Recognition
,
Kolkata, India
,
Nov. 16–18
, IEEE, pp.
289
294
.
44.
Sharma
,
D.
,
Gupta
,
N.
,
Chattopadhyay
,
C.
, and
Mehta
,
S.
,
2019
, “
A Novel Feature Transform Framework Using Deep Neural Network for Multimodal Floor Plan Retrieval
,”
Int. J. Doc. Anal. Recognit.
,
22
(
4
), pp.
417
429
.
45.
Ahmed
,
S.
,
Liwicki
,
M.
,
Weber
,
M.
, and
Dengel
,
A.
,
2012
, “
Automatic Room Detection and Room Labeling From Architectural Floor Plans
,”
2012 10th IAPR International Workshop on Document Analysis Systems
,
Gold Coast, Australia
,
Mar. 27–29
, IEEE, pp.
339
343
.
46.
de las Heras
,
L.-P.
,
2015
,
Relational Models for Visual Understanding of Graphical Documents. Application to Architectural Drawings
,
Universitat Autònoma de Barcelona
,
Cerdanyola del Vallès
.
47.
Yun
,
X.-L.
,
Zhang
,
Y.-M.
,
Ye
,
J.-Y.
, and
Liu
,
C.-L.
,
2019
, “
Online Handwritten Diagram Recognition With Graph Attention Networks
,”
International Conference on Image and Graphics
,
Beijing
,
Aug. 23–25
, Springer, pp.
232
244
.
48.
Ye
,
J.-Y.
,
Zhang
,
Y.-M.
, and
Liu
,
C.-L.
,
2016
, “
Joint Training of Conditional Random Fields and Neural Networks for Stroke Classification in Online Handwritten Documents
,”
2016 23rd International Conference on Pattern Recognition (ICPR)
,
Cancún, Mexico
,
Dec. 4–8
, IEEE, pp.
3264
3269
.
49.
Van Phan
,
T.
, and
Nakagawa
,
M.
,
2016
, “
Combination of Global and Local Contexts for Text/Non-text Classification in Heterogeneous Online Handwritten Documents
,”
Pattern Recognit.
,
51
, pp.
112
124
.
50.
Zhang
,
W.
,
Joseph
,
J.
,
Yin
,
Y.
,
Xie
,
L.
,
Furuhata
,
T.
,
Yamakawa
,
S.
,
Shimada
,
K.
, and
Kara
,
L. B.
,
2022
, “
Component Segmentation of Engineering Drawings Using Graph Convolutional Networks
,”
Comput. Ind.
,
147
, p.
103885
.
51.
Ramer
,
U.
,
1972
, “
An Iterative Procedure for the Polygonal Approximation of Plane Curves
,”
Comput. Graph. Image Process.
,
1
(
3
), pp.
244
256
.
52.
Douglas
,
D. H.
, and
Peucker
,
T. K.
,
1973
, “
Algorithms for the Reduction of the Number of Points Required to Represent a Digitized Line or Its Caricature
,”
Cartogr. Int. J. Geogr. Inf. Geovis.
,
10
(
2
), pp.
112
122
.
53.
Kingma
,
D. P.
, and
Ba
,
J.
,
2014
, “
Adam: A Method for Stochastic Optimization
,”
3rd International Conference on Learning Representations, ICLR 2015
,
San Diego, CA
,
May 7–9
.
54.
Haar
,
C.
,
Kim
,
H.
, and
Koberg
,
L.
,
2022
, “
AI-Based Engineering and Production Drawing Information Extraction
,”
Flexible Automation and Intelligent Manufacturing: The Human-Data-Technology Nexus: Proceedings of FAIM 2022
,
June 19–23
,
Detroit, MI
, Springer, pp.
374
382
.
55.
Scheibel
,
B.
,
Mangler
,
J.
, and
Rinderle-Ma
,
S.
,
2021
, “
Extraction of Dimension Requirements From Engineering Drawings for Supporting Quality Control in Production Processes
,”
Comput. Ind.
,
129
, p.
103442
.
56.
Ahmed
,
S.
,
Weber
,
M.
,
Liwicki
,
M.
,
Langenhan
,
C.
,
Dengel
,
A.
, and
Petzold
,
F.
,
2014
, “
Automatic Analysis and Sketch-Based Retrieval of Architectural Floor Plans
,”
Pattern Recognit. Lett.
,
35
, pp.
91
100
.
You do not currently have access to this content.