Abstract

Drawing curves is a fundamental task in mid-air interactive applications such as 3D sketching, geometric modeling, hand-writing recognition, and authentication. Existing research in mid-air drawing is solely focused on determining what the user drew assuming that the intended curve is segmented from the continuous user-generated trajectory. In this work, our aim is to address the complementary problem: to determine when the user actually intended to draw without the use of any prescribed gestures or hand-held controllers (e.g., Wii remote, HTC Vive). In our previously published work, we demonstrated that in mid-air drawing tasks, not only it is possible to statistically learn drawing intent from hand motion, but it is also perceived to be more natural by users. Our idea was to simply classify each instance of hand trajectories as either a stroke or a hover. Our current work investigates new representations of the users’ motion beyond a single point (such as a tracked palm) to richer multi-point trajectories obtained with other skeletal joints such as wrist and elbow. We trained several binary classifiers on five such trajectory representations obtained from 3D drawing data from 25 users using a hand tracking device. We compare these representations and the corresponding classifiers for predicting user intent for mid-air drawing. Our extended approach resulted in improved prediction accuracy (mean: 80.17%, min: 79.92%, max: 91.30%) with respect to our earlier work (mean: 76.75%, min: 74.23%, max: 84.01%).

References

1.
Chen
,
Y.
,
Liu
,
J.
, and
Tang
,
X.
,
2008
, “
Sketching in the Air: A Vision-Based System for 3D Object Design
,”
IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008
,
Anchorage, AK
,
June 23–28
, IEEE, pp.
1
6
.
2.
Arora
,
R.
,
Kazi
,
R. H.
,
Anderson
,
F.
,
Grossman
,
T.
,
Singh
,
K.
, and
Fitzmaurice
,
G. W.
,
2017
, “
Experimental Evaluation of Sketching on Surfaces in VR
,”
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17)
,
Denver, CO
,
May 6–11
, CHI, pp.
5643
5654
.
3.
Taele
,
P.
,
2014
, “
Intelligent Sketching Interfaces for Richer Mid-Air Drawing Interactions
,”
CHI’14 Extended Abstracts on Human Factors in Computing Systems
,
Toronto, ON, Canada
,
Apr. 26–May 1
, ACM, pp.
339
342
.
4.
Aslan
,
I.
,
Uhl
,
A.
,
Meschtscherjakov
,
A.
, and
Tscheligi
,
M.
,
2014
, “
Mid-Air Authentication Gestures: An Exploration of Authentication Based on Palm and Finger Motions
,”
Proceedings of the 16th International Conference on Multimodal Interaction
,
Istanbul, Turkey
,
Nov. 12–16
, ACM, pp.
311
318
.
5.
Schick
,
A.
,
Morlock
,
D.
,
Amma
,
C.
,
Schultz
,
T.
, and
Stiefelhagen
,
R.
,
2012
, “
Vision-Based Handwriting Recognition for Unrestricted Text Input in Mid-Air
,”
Proceedings of the 14th ACM International Conference on Multimodal Interaction
,
Santa Monica, CA
,
Oct. 22–26
, ACM, pp.
217
220
.
6.
Vikram
,
S.
,
Li
,
L.
, and
Russell
,
S.
,
2013
, “
Writing and Sketching in the Air, Recognizing and Controlling on the Fly
,”
CHI’13 Extended Abstracts on Human Factors in Computing Systems
,
Paris, France
,
Apr. 27–May 2
, ACM, pp.
1179
1184
.
7.
Agarwal
,
C.
,
Dogra
,
D. P.
,
Saini
,
R.
, and
Roy
,
P. P.
,
2015
, “
Segmentation and Recognition of Text Written in 3d Using Leap Motion Interface
,”
2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR)
,
Kuala Lumpur, Malaysia
,
Nov. 3–6
, IEEE, pp.
539
543
.
8.
Taranta II
,
E. M.
,
Samiei
,
A.
,
Maghoumi
,
M.
,
Khaloo
,
P.
,
Pittman
,
C. R.
, and
La Viola
,
J. J.
, Jr.
,
2017
, “
Jackknife: A Reliable Recognizer With Few Samples and Many Modalities
,”
Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17
,
Denver, CO
,
May 6–11
, ACM, pp.
5850
5861
.
9.
Bohari
,
Umema
,
Chen
,
Ting-Ju
, and
Vinayak
,
2018
, “
To Draw Or Not to Draw: Recognizing Stroke-Hover Intent in Non-Instrumented Gesture-Free Mid-Air Sketching
,”
23rd International Conference on Intelligent User Interfaces
,
Tokyo, Japan
,
Mar. 7–11
, ACM, pp.
177
188
.
10.
Noris
,
G.
,
Sỳkora
,
D.
,
Shamir
,
A.
,
Coros
,
S.
,
Whited
,
B.
,
Simmons
,
M.
,
Hornung
,
A.
,
Gross
,
M.
, and
Sumner
,
R.
,
2012
, “
Smart Scribbles for Sketch Segmentation
,”
Comput. Graphics Forum
,
31
(
8
), pp.
2516
2527
. 10.1111/j.1467-8659.2012.03224.x
11.
Paulson
,
B.
, and
Hammond
,
T.
,
2008
, “
Paleosketch: Accurate Primitive Sketch Recognition and Beautification
,”
Proceedings of the 13th International Conference on Intelligent User Interfaces, IUI ’08
,
Maspalomas, Gran Canaria, Spain
,
Jan. 13–16
, ACM, pp.
1
10
.
12.
Field
,
M.
,
Gordon
,
S.
,
Peterson
,
E.
,
Robinson
,
R.
,
Stahovich
,
T.
, and
Alvarado
,
C.
,
2010
, “
The Effect of Task on Classification Accuracy: Using Gesture Recognition Techniques in Free-Sketch Recognition
,”
Comput. Graph.
,
34
(
5
), pp.
499
512
. 10.1016/j.cag.2010.07.001
13.
Bae
,
S.-H.
,
Balakrishnan
,
R.
, and
Singh
,
K.
,
2008
, “
Ilovesketch: As-Natural-as-Possible Sketching System for Creating 3d Curve Models
,”
Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, UIST ’08
,
Monterey, CA
,
Oct. 19–22
, ACM, pp.
151
160
.
14.
Anthony
,
L.
, and
Wobbrock
,
J. O.
,
2010
, “
A Lightweight Multistroke Recognizer for User Interface Prototypes
,”
Proceedings of Graphics Interface 2010, GI ’10
,
Ottawa, Ontario, Canada
,
May 31–June 2
, pp.
245
252
.
15.
Anthony
,
L.
, and
Wobbrock
,
J. O.
,
2012
, “
$ N-Protractor: A Fast and Accurate Multistroke Recognizer
,”
Proceedings of Graphics Interface 2012, GI ’12
,
Toronto, Ontario, Canada
,
May 28–30
, pp.
117
120
.
16.
Wobbrock
,
J. O.
,
Wilson
,
A. D.
, and
Li
,
Y.
,
2007
, “
Gestures Without Libraries, Toolkits Or Training: A $1 Recognizer for User Interface Prototypes
,”
Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, UIST ’07
,
ACM
, pp.
159
168
.
17.
Vatavu
,
R.-D.
,
Anthony
,
L.
, and
Wobbrock
,
J. O.
,
2012
, “
Gestures As Point Clouds: A $p Recognizer for User Interface Prototypes
,”
Proceedings of the 14th ACM International Conference on Multimodal Interaction, ICMI ’12
,
ACM
, pp.
273
280
.
18.
Suryanarayan
,
P.
,
Subramanian
,
A.
, and
Mandalapu
,
D.
,
2010
, “
Dynamic Hand Pose Recognition Using Depth Data
,”
Proceedings of the 2010 20th International Conference on Pattern Recognition, ICPR ’10
,
IEEE Computer Society
, pp.
3105
3108
.
19.
Arandjelović
,
R.
, and
Sezgin
,
T. M.
,
2011
, “
Sketch Recognition by Fusion of Temporal and Image-Based Features
,”
Pattern Recognit.
,
44
(
6
), pp.
1225
1234
. 10.1016/j.patcog.2010.11.006
20.
Willems
,
D.
,
Niels
,
R.
,
van Gerven
,
M.
, and
Vuurpijl
,
L.
,
2009
, “
Iconic and Multi-Stroke Gesture Recognition
,”
Pattern Recognit.
,
42
(
12
), pp.
3303
3312
.
New Frontiers in Handwriting Recognition
. 10.1016/j.patcog.2009.01.030
21.
Regazzoni
,
D.
,
Rizzi
,
C.
, and
Vitali
,
A.
,
2018
, “
Virtual Reality Applications: Guidelines to Design Natural User Interface
,”
ASME 2018 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
,
Quebec City, Canada
,
Aug. 26–29
, American Society of Mechanical Engineers Digital Collection.
22.
Melax
,
S.
,
Keselman
,
L.
, and
Orsten
,
S.
,
2013
, “
Dynamics Based 3D Skeletal Hand Tracking
,”
Proceedings of Graphics Interface 2013, GI ’13
,
Regina, Saskatchewan, Canada
,
May 29–31
, Canadian Information Processing Society, pp.
63
70
.
23.
Sharp
,
T.
,
Keskin
,
C.
,
Robertson
,
D.
,
Taylor
,
J.
,
Shotton
,
J.
,
Kim
,
D.
,
Rhemann
,
C.
,
Leichter
,
I.
,
Vinnikov
,
A.
,
Wei
,
Y.
,
Freedman
,
D.
,
Kohli
,
P.
,
Krupka
,
E.
,
Fitzgibbon
,
A.
, and
Izadi
,
S.
,
2015
, “
Accurate, Robust, and Flexible Real-Time Hand Tracking
,”
Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15
,
Seoul, South Korea
,
Apr. 18–23
, ACM, pp.
3633
3642
.
24.
Tagliasacchi
,
A.
,
Schröder
,
M.
,
Tkach
,
A.
,
Bouaziz
,
S.
,
Botsch
,
M.
, and
Pauly
,
M.
,
2015
, “
Robust Articulated-ICP for Real-Time Hand Tracking
,”
Computer Graphics Forum
,
34
(
5
), pp.
101
114
. https://doi.org/10.1111/cgf.12700
25.
Ren
,
Z.
,
Meng
,
J.
, and
Yuan
,
J.
,
2011
, “
Depth Camera Based Hand Gesture Recognition and Its Applications in Human-Computer-Interaction
,”
2011 8th International Conference on Information, Communications Signal Processing
,
Singapore
,
Dec. 13–16
, IEEE, pp.
1
5
.
26.
Ni
,
T.
,
Schmidt
,
G. S.
,
Staadt
,
O. G.
,
Livingston
,
M. A.
,
Ball
,
R.
, and
May
,
R.
,
2006
, “
A Survey of Large High-Resolution Display Technologies, Techniques, and Applications
,”
Proceedings of the IEEE Conference on Virtual Reality, VR ’06
,
Alexandria, VA
,
Mar. 25–29
, IEEE Computer Society, pp.
223
236
.
27.
Czerwinski
,
M.
,
Robertson
,
G.
,
Meyers
,
B.
,
Smith
,
G.
,
Robbins
,
D.
, and
Tan
,
D.
,
2006
, “
Large Display Research Overview
,”
CHI ’06 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’06
,
Montreal, Quebec, Canada
,
Apr. 22–27
, ACM, pp.
69
74
.
28.
Lischke
,
L.
,
Grüninger
,
J.
,
Klouche
,
K.
,
Schmidt
,
A.
,
Slusallek
,
P.
, and
Jacucci
,
G.
,
2015
, “
Interaction Techniques for Wall-Sized Screens
,”
Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces, ITS ’15
,
Funchal, Portugal
,
Nov. 15–18
, ACM, pp.
501
504
.
29.
Grossman
,
T.
,
Balakrishnan
,
R.
,
Kurtenbach
,
G.
,
Fitzmaurice
,
G.
,
Khan
,
A.
, and
Buxton
,
B.
,
2002
, “
Creating Principal 3d Curves With Digital Tape Drawing
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’02
,
Minneapolis, MN
,
Apr. 20–25
, ACM, pp.
121
128
.
30.
Grossman
,
T.
,
Balakrishnan
,
R.
,
Kurtenbach
,
G.
,
Fitzmaurice
,
G.
,
Khan
,
A.
, and
Buxton
,
B.
,
2001
, “
Interaction Techniques for 3d Modeling on Large Displays
,”
Proceedings of the 2001 Symposium on Interactive 3D Graphics, I3D ’01
,
Research Triangle Park, NC
,
Mar. 19–21
, ACM, pp.
17
23
.
31.
Laundry
,
B.
,
Masoodian
,
M.
, and
Rogers
,
B.
,
2010
, “
Interaction With 3D Models on Large Displays Using 3d Input Techniques
,”
Proceedings of the 11th International Conference of the NZ Chapter of the ACM Special Interest Group on Human-Computer Interaction, CHINZ ’10
,
Auckland, New Zealand
,
July 8–9
, ACM, pp.
49
56
.
32.
Yang
,
C.-K.
,
Chen
,
Y.-H.
,
Chuang
,
T.-J.
,
Shankhwar
,
K.
, and
Smith
,
S.
,
2019
, “
An Augmented Reality-Based Training System With a Natural User Interface for Manual Milling Operations
,”
Virtual Reality
, pp.
1
13
. 10.1007/s10055-019-00415-8
33.
Dominio
,
F.
,
Donadeo
,
M.
, and
Zanuttigh
,
P.
,
2014
, “
Combining Multiple Depth-Based Descriptors for Hand Gesture Recognition
,”
Pattern Recogn. Lett.
,
50
(
C
), pp.
101
111
. 10.1016/j.patrec.2013.10.010
34.
Krishnan
,
N. C.
, and
Cook
,
D. J.
,
2014
, “
Activity Recognition on Streaming Sensor Data
,”
Pervasive Mob. Comput.
,
10
(
Part B
), pp.
138
154
. 10.1016/j.pmcj.2012.07.003
35.
Holz
,
C.
, and
Wilson
,
A.
,
2011
, “
Data Miming: Inferring Spatial Object Descriptions From Human Gesture
,”
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’11
,
Vancouver, BC, Canada
,
May 7–12
, ACM, pp.
811
820
.
36.
Vinayak
, and
Ramani
,
K.
,
2015
, “
A Gesture-Free Geometric Approach for Mid-Air Expression of Design Intent in 3d Virtual Pottery
,”
Comput.-Aided Des.
,
69
, pp.
11
24
. 10.1016/j.cad.2015.06.006
37.
Weichert
,
F.
,
Bachmann
,
D.
,
Rudak
,
B.
, and
Fisseler
,
D.
,
2013
, “
Analysis of the Accuracy and Robustness of the Leap Motion Controller
,”
Sensors
,
13
(
5
), pp.
6380
6393
. 10.3390/s130506380
38.
Okazaki
,
S.
,
Muraoka
,
Y.
, and
Suzuki
,
R.
,
2017
, “
Validity and Reliability of Leap Motion Controller for Assessing Grasping and Releasing Finger Movements
,”
J. Ergon. Technol.
,
17
(
1
), pp.
32
42
.
39.
Niechwiej-Szwedo
,
E.
,
Gonzalez
,
D.
,
Nouredanesh
,
M.
, and
Tung
,
J.
,
2018
, “
Evaluation of the Leap Motion Controller During the Performance of Visually-Guided Upper Limb Movements
,”
PLoS One
,
13
(
3
), p.
e0193639
. 10.1371/journal.pone.0193639
40.
Khademi
,
M.
,
Mousavi Hondori
,
H.
,
McKenzie
,
A.
,
Dodakian
,
L.
,
Lopes
,
C. V.
, and
Cramer
,
S. C.
,
2014
, “
Free-Hand Interaction With Leap Motion Controller for Stroke Rehabilitation
,”
CHI ‘14 Extended Abstracts on Human Factors in Computing Systems, CHI EA ‘14
,
Toronto, ON, Canada
,
Apr. 26–May 1
, Association for Computing Machinery, pp.
1663
1668
.
41.
Hantrakul
,
L.
, and
Kaczmarek
,
K.
,
2014
, “
Implementations of the Leap Motion Device in Sound Synthesis and Interactive Live Performance
,”
Proceedings of the 2014 International Workshop on Movement and Computing
,
Paris, France
,
June 16–17
, pp.
142
145
42.
Sutton
,
J.
,
2013
, “
Air Painting With Corel Painter Freestyle and the Leap Motion Controller: a Revolutionary New Way to Paint!
,”
SIGGRAPH '13: ACM SIGGRAPH 2013 Studio Talks
,
Anaheim, CA
,
July 21–25
, p.
1
.
You do not currently have access to this content.