Sketches are a ubiquitous form of communication in engineering design due to their simplicity and efficiency. However, because of a lack of suitable machine-interpretation techniques, they are virtually unusable with current computer-aided design and engineering tools. The informal nature of sketches and their inherent ambiguity present a number of challenges to the development of such techniques. Here we address one particular challenge, the task of reliably locating and recognizing the intended visual objects from a continuous stream of pen strokes. We present an integrated sketch parsing and recognition approach, based on a novel mark-group-recognize paradigm, which is tailored to the domain of mechanical systems. In the first step of processing, the sketch is examined to identify certain delimiting symbols called “markers.” The remaining pen strokes are then partitioned into distinct clusters, each representing a single symbol. Finally, a trainable symbol recognizer is used to find the best interpretation of each cluster. We have used these techniques to build a sketch-based tool for designing and analyzing vibratory mechanical systems. This tool enables designers to analyze and animate vibratory systems by simply sketching them on a tablet computer. User studies indicate that even first-time users find our tool to be effective.

1.
Shpitalni
,
M.
, and
Lipson
,
H.
, 1995, “
Classification of Sketch Strokes and Corner Detection Using Conic Sections and Adaptive Clustering
,”
ASME J. Mech. Des.
0161-8458,
119
, pp.
131
135
.
2.
Ullman
,
D. G.
,
Wood
,
S.
, and
Craig
,
D.
, 1990, “
The Importance of Drawing in the Mechanical Design Process
,”
Comput. Graph.
0097-8930,
14
(
2
), pp.
263
274
.
3.
Schutze
,
M.
,
Sachse
,
P.
, and
Romer
,
A.
, 2003, “
Support Value of Sketching in the Design Process
,”
Res. Eng. Des.
,
14
, pp.
89
97
. 0097-8930
4.
Yang
,
M. C.
, 2003, “
Concept Generation and Sketching: Correlations With Design Outcome
,”
ASME Design Engineering Technical Conferences and Design Theory and Methodology Conference
, Chicago, IL, September 2–6.
5.
Song
,
S.
, and
Agogino
,
A. M.
, 2004, “
Insights on Designers’ Sketching Activities in New Product Design Teams
,”
ASME Design Engineering Technical Conferences and Computers and Information in Engineering Conference
, Salt Lake City, UT, September 28–October 2.
6.
Yang
,
M. C.
, and
Cham
,
J. G.
, 2007, “
An Analysis of Sketching Skill and its Role in Early Stage Engineering Design
,”
ASME J. Mech. Des.
0161-8458,
129
(
5
), pp.
476
482
.
7.
Chusilp
,
P.
, and
Jin
,
Y.
, 2006, “
Impact of Mental Iteration on Concept Generation
,”
ASME J. Mech. Des.
0161-8458,
128
(
1
), pp.
14
25
.
8.
de Silva
,
R.
,
Bischel
,
D. T.
,
Lee
,
W.
,
Peterson
,
E. J.
,
Calfee
,
R. C.
, and
Stahovich
,
T. F.
, 2007, “
Kirchhoff’s Pen: A Pen-Based Circuit Analysis Tutor
,”
Eurographics Workshop on Sketch-Based Interfaces and Modeling
, June 11–13, Annecy, France.
9.
Fonseca
,
M. J.
,
Pimentel
,
C.
, and
Jorge
,
J. A.
, 2002, “
Cali-an Online Scribble Recognizer for Calligraphic Interfaces
,”
AAAI Spring Symposium on Sketch Understanding
, March 25–27, Palo Alto, CA, pp.
51
58
.
10.
Narayanaswamy
,
S.
, 1996, “
Pen and Speech Recognition in the User Interface for Mobile Multimedia Terminals
,” Ph.D. thesis, University of California at Berkeley, Berkeley, CA.
11.
Landay
,
J. A.
, and
Myers
,
B. A.
, 2001, “
Sketching Interfaces: Toward More Human Interface Design
,”
IEEE Computer
,
34
(
3
), pp.
56
64
.
12.
Gennari
,
L.
,
Kara
,
L. B.
, and
Stahovich
,
T. F.
, 2004, “
Combining Geometry and Domain Knowledge to Interpret Hand-Drawn Diagrams
,”
AAAI Fall Symposium Series 2004: Making Pen-Based Interaction Intelligent and Natural
, October 21–24, Washington, D.C.
13.
Kara
,
L. B.
, and
Stahovich
,
T. F.
, 2004, “
Hierarchical Parsing and Recognition of Hand-Sketched Diagrams
,”
User Interface Software Technology (UIST)
, October 24–27, Santa Fe, NM.
14.
Saund
,
E.
,
Mahoney
,
J.
,
Fleet
,
D.
,
Larner
,
D.
, and
Lank
,
E.
, 2002, “
Perceptual Organisation as a Foundation for Intelligent Sketch Editing
,”
AAAI Spring Symposium on Sketch Understanding
, March 25–27, Palo Alto, CA, pp.
118
125
.
15.
Jacobs
,
D. W.
, 1988, “
The Use of Grouping in Visual Object Recognition
,”
MIT AI Lab
, Technical Report No. 1023.
16.
LaViola
J.
, and
Zeleznik
,
R.
, 2004, “
Mathpad2: A System for the Creation and Exploration of Mathematical Sketches
,”
ACM Transactions on Graphics (Proceedings of SIGGRAPH 2004)
, August 8–12, Los Angeles, CA, Vol.
23
, pp.
432
440
.
17.
Notowidigdo
,
M.
, and
Miller
,
R. C.
, 2004, “
Off-Line Sketch Interpretation
,”
AAAI Fall Symposium Series 2004: Making Pen-Based Interaction Intelligent and Natural
, October 21–24, Washington, D.C.
18.
Costagliola
,
G.
, and
Deufemia
,
V.
, 2003, “
Visual Language Editors Based on lr Parsing Techniques
,”
Proceedings of the Eighth International Workshop on Parsing Technologies (IWPT’03)
, Nancy, France.
19.
Sezgin
,
T. M.
, and
Davis
,
R.
, 2005,
Hmm-Based Efficient Sketch Recognition, International Conference on Intelligent User Interfaces (IUI’05)
, New York, January 9–12.
20.
Yasuda
,
H.
,
Takahashi
,
K.
, and
Matsumoto
,
T.
, 2000, “
A Discrete Hmm for Online Handwriting Recognition
,”
Int. J. Pattern Recognit. Artif. Intell.
,
14
(
5
), pp.
675
688
. 0218-0014
21.
Grimson
W. E. L.
, 1991, “
The Combinatorics of Heuristic Search Termination for Object Recognition in Cluttered Environments
,”
IEEE Trans. Pattern Anal. Mach. Intell.
0162-8828,
13
(
9
), pp.
920
935
.
22.
Shilman
,
M.
,
Pasula
,
H.
,
Russell
,
S.
, and
Newton
,
R.
2002, “
Statistical Visual Language Models for Ink Parsing
,”
AAAI Spring Symposium on Sketch Understanding
, March 25–27, Palo Alto, CA, pp.
126
132
.
23.
Rubine
,
D.
, 1991, “
Specifying Gestures by Example
,”
Comput. Graph.
,
25
, pp.
329
337
. 0097-8930
24.
Shilman
,
M.
, and
Viola
,
P.
, 2004, “
Spatial Recognition and Grouping of Text and Graphics
,”
EU-ROGRAPHICS Workshop on Sketch-Based Interfaces and Modeling
, August 30–31, Grenoble, France.
25.
Alvarado
,
C.
, and
Davis
,
R.
, 2005, “
Dynamically Constructed Bayes Nets for Multi-Domain Sketch Understanding
,”
International Joint Conference on Artificial Intelligence
, July 30–August 5, Edinburgh, Scotland, UK.
26.
Kimura
,
T. D.
,
Apte
,
A.
, and
Sengupta
,
S.
, 1994, “
A Graphic Diagram Editor for Pen Computers
,”
Software Concepts and Tools
, pp.
82
95
.
27.
Ozer
,
O. F.
,
Ozun
,
O.
,
Tuzel
,
C. O.
,
Atalay
,
V.
, and
Cetin
,
A. E.
, 2001, “
Vision-Based Single-Stroke Character Recognition for Wearable Computing
,”
IEEE Intell. Syst.
1094-7167,
16
(
3
), pp.
33
37
.
28.
Apte
,
A.
,
Vo
,
V.
, and
Kimura
,
T. D.
, 1993, “
Recognizing Multistroke Geometric Shapes: An Experimental Evaluation
,”
Proceedings of the UIST 93
, November 3–5, Atlanta, GA, pp.
121
128
.
29.
Fonseca
,
M. J.
, and
Jorge
,
J. A.
, 2000, “
Using Fuzzy Logic to Recognize Geometric Shapes Interactively
,”
Proceedings of the Ninth International Conference on Fuzzy Systems (FUZZ-IEEE 2000)
, San Antonio, TX, May 2000.
30.
Matsakis
,
N. E.
, 1999, “
Recognition of Handwritten Mathematical Expressions
,” MS thesis, MIT, Cambridge.
31.
Hse
,
H.
, and
Newton
,
A. R.
, 2004, “
Sketched Symbol Recognition Using Zernike Moments
,”
17th International Conference on Pattern Recognition
, Cambridge, UK, Vol. 1, pp. 367–370.
32.
Kara
,
L. B.
, and
Stahovich
,
T. F.
, 2004, “
An Image-Based Trainable Symbol Recognizer for Sketch-Based Interfaces
,”
AAAI Fall Symposium Series 2004: Making Pen-Based Interaction Intelligent and Natural
, October 21–24, Washington, D.C.
33.
Erman
,
L. D.
,
Hayes-Roth
,
F.
,
Lesser
,
V. R.
, and
Raj Reddy
,
D.
, 1980, “
The Hearsay-II Speech Understanding System: Integrating Knowldge to Resolve Uncertainty
,”
ACM Comput. Surv.
0360-0300,
12
(
2
), pp.
213
253
.
34.
Duda
,
R. O.
,
Hart
,
P. E.
, and
Stork
,
D. G.
, 2001,
Pattern Classification
,
2nd ed.
,
Wiley
,
New York
.
35.
Kara
,
L. B.
, 2004, “
Automatic Parsing and Recognition of Hand-Drawn Sketches for Pen-Based Computer Interfaces
,” Ph.D. thesis, Carnegie Mellon University, Pittsburgh, PA.
36.
Stahovich
,
T. F.
, 2004, “
Segmentation of Pen Strokes Using Pen Speed
,”
AAAI Fall Symposium Series 2004: Making Pen-Based Interaction Intelligent and Natural
, October 21–24, Washington, D.C.
37.
Domingos
,
P.
, and
Pazzani
,
M. J.
, 1997, “
Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier
,”
Mach. Learn.
,
29
, pp.
103
130
. 0885-6125
You do not currently have access to this content.