We introduce an intuitive gesture-based interaction technique for creating and manipulating simple three-dimensional (3D) shapes. Specifically, the developed interface utilizes low-cost depth camera to capture user's hand gesture as the input, maps different gestures to system commands and generates 3D models from midair 3D sketches (as opposed to traditional two-dimensional (2D) sketches). Our primary contribution is in the development of an intuitive gesture-based interface that enables novice users to rapidly construct conceptual 3D models. Our development extends current works by proposing both design and technical solutions to the challenges of the gestural modeling interface for conceptual 3D shapes. The preliminary user study results suggest that the developed framework is intuitive to use and able to create a variety of 3D conceptual models.

References

References
1.
Robertson
,
B. F.
, and
Radcliffe
,
D. F.
,
2009
, “
Impact of Cad Tools on Creative Problem Solving in Engineering Design
,”
Comput.-Aided Des.
,
41
(
3
), pp.
136
146
.
2.
Sharma
,
A.
,
Madhvanath
,
S.
,
Shekhawat
,
A.
, and
Billinghurst
,
M.
,
2011
, “
Mozart: A Multimodal Interface for Conceptual 3D Modeling
,”
13th International Conference on Multimodal Interfaces
(
ICMI
), Alicante, Spain, Nov. 14–18, pp.
307
310
.https://www.researchgate.net/publication/221052364_MozArt_A_Multimodal_Interface_for_Conceptual_3D_Modeling
3.
Rubine
,
D.
,
1991
, “
Specifying Gestures by Example
,”
18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '91)
, Providence, RI, Apr. 27–30, pp.
329
337
.
4.
Takahashi
,
T.
, and
Kishino
,
F.
,
1992
, “
A Hand Gesture Recognition Method and Its Application
,”
Syst. Comput. Jpn.
,
23
(
3
), pp.
38
48
.
5.
Radhakrishnan
,
S.
,
Lin
,
Y.
,
Zeid
,
I.
, and
Kamarthi
,
S.
,
2013
, “
Finger-Based Multitouch Interface for Performing 3D CAD Operations
,”
Int. J. Human-Comput. Stud.
,
71
(
3
), pp.
261
275
.
6.
Seth
,
A.
,
Su
,
H.-J.
, and
Vance
,
J. M.
,
2006
, “
Sharp: A System for Haptic Assembly and Realistic Prototyping
,”
ASME
Paper No. DETC2006-99476
.
7.
Edit
,
V.
,
2008
, “
Using Hand Motions in Conceptual Shape Design
,”
Product Engineering
, Springer, Berlin, pp. 367–382.
8.
Segen
,
J.
, and
Kumar
,
S.
,
1998
, “
Gesture Vr: Vision-Based 3D Hand Interace for Spatial Interaction
,”
Sixth ACM International Conference on Multimedia
, Bristol, UK, Sept. 13–16, pp.
455
464
.
9.
Wang
,
R.
,
Paris
,
S.
, and
Popović
,
J.
,
2011
, “
6D Hands: Markerless Hand-Tracking for Computer Aided Design
,”
24th Annual ACM Symposium on User Interface Software and Technology
, Santa Barbara, CA, Oct. 16–19, pp.
549
558
.https://homes.cs.washington.edu/~jovan/papers/wang-2011-6dh.pdf
10.
Bai
,
H.
,
Gao
,
L.
,
El-Sana
,
J.
, and
Billinghurst
,
M.
, “
Free-Hand Interaction for Handheld Augmented Reality Using an RGB-Depth Camera
,”
SIGGRAPH Asia 2013 Symposium on Mobile Graphics and Interactive Applications
, Hong Kong, Nov. 19–22, Paper No. 22.
11.
Stannus
,
S.
,
Fu
,
W.-T.
, and
Lucieer
,
A.
,
2014
, “
Natural 7DOF Input for 3D Navigation
,”
26th Australian Computer-Human Interaction Conference on Designing Futures: The Future of Design
, Sydney, Australia, Dec. 2–5, pp.
216
219
.
12.
Song
,
J.
,
Cho
,
S.
,
Baek
,
S.-Y.
,
Lee
,
K.
, and
Bang
,
H.
,
2014
, “
Gafinc: Gaze and Finger Control Interface for 3D Model Manipulation in Cad Application
,”
Comput.-Aided Des.
,
46
, pp.
239
245
.
13.
Vinayak
,
Murugappan
,
S.
,
Piya
,
C.
, and
Ramani
,
K.
,
2013
, “
Handy-Potter: Rapid Exploration of Rotationally Symmetric Shapes Through Natural Hand Motions
,”
ASME J. Comput. Inf. Sci. Eng.
,
13
(
2
), p.
021008
.
14.
Han
,
Y. C.
, and
Han
,
B-J.
,
2014
, “
Virtual Pottery: A Virtual 3D Audiovisual Interface Using Natural Hand Motions
,”
Multimedia Tools Appl.
,
73
(
2
), pp.
917
933
.
15.
Vinayak
,
K.
, and
Ramani
,
2015
, “
A Gesture-Free Geometric Approach for Mid-Air Expression of Design Intent in 3D Virtual Pottery
,”
Comput.-Aided Des.
,
69
, pp.
11
24
.
16.
Colombo
,
G.
,
Facoetti
,
G.
,
Rizzi
,
C.
, and
Vitali
,
A.
,
2016
, “
Mixed Reality to Design Lower Limb Prosthesis
,”
Comput.-Aided Des. Appl.
,
13
(
6
), pp.
799
807
.
17.
Fuge
,
M.
,
Yumer
,
M. E.
,
Orbay
,
G.
, and
Kara
,
L. B.
,
2012
, “
Conceptual Design and Modification of Freeform Surfaces Using Dual Shape Representations in Augmented Reality Environments
,”
Comput.-Aided Des.
,
44
(
10
), pp.
1020
1032
.
18.
Jinmiao
,
H.
,
Prakhar
,
J.
, and
Rahul
,
R.
,
2018
, “
Gesture-Based System for Next Generation Natural and Intuitive Interfaces
,”
Ai Edam
,
15
, p.
1
.
19.
Olsen
,
L.
,
Samavati
,
F. F.
,
Sousa
,
M. C.
, and
Jorge
,
J. A.
,
2009
, “
Sketch-Based Modeling: A Survey
,”
Comput. Graph.
,
33
(
1
), pp.
85
103
.
20.
Takeo
,
I.
,
Satoshi
,
M.
, and
Hidehiko
,
T.
,
2007
, “
Teddy: A Sketching Interface for 3D Freeform Design
,”
ACM SIGGRAPH 2007 Courses
, ACM, New York, p.
21
21.
Nealen
,
A.
,
Igarashi
,
T.
,
Sorkine
,
O.
, and
Alexa
,
M.
,
2007
, “
Fibermesh: Designing Freeform Surfaces With 3D Curves
,”
ACM Trans. Graph. (TOG)
,
26
(
3
), p.
41
.
22.
Karpenko
,
O.
,
Hughes
,
J. F.
, and
Raskar
,
R.
,
2002
, “
Free-Form Sketching With Variational Implicit Surfaces
,”
Comput. Graph. Forum
,
21
(
3
), pp.
585
594
.
23.
Tai
,
C.-L.
,
Zhang
,
H.
, and
Fong
,
J. C.-K.
,
2004
, “
Prototype Modeling From Sketched Silhouettes Based on Convolution Surfaces
,”
Comput. Graph. Forum
,
23
(
1
), pp.
71
83
.
24.
Alexe
,
A.
,
Gaildrat
,
V.
, and
Barthe
,
L.
,
2004
, “
Interactive Modelling From Sketches Using Spherical Implicit Functions
,”
Third International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa
, Stellenbosch, South Africa, Nov. 3–5, pp.
25
34
.
25.
Schmidt
,
R.
,
Wyvill
,
B.
,
Sousa
,
M. C.
, and
Jorge
,
J. A.
,
2006
, “
Shapeshop: Sketch-Based Solid Modeling With Blobtrees
,” ACM SIGGRAPH 2006 Courses, ACM, New York, p.
14
26.
Gharib
,
I.
, and
Qin
,
S.
,
2013
, “
Integration of Sketch-Based Conceptual Design and Commercial Cad Systems for Manufacturing
,”
Int. J. Adv. Manuf. Technol.
,
68
(
9–12
), pp.
2669
2681
.
27.
Zeleznik
,
R. C.
,
Herndon
,
K. P.
, and
Hughes
,
J. F.
,
2007
, “
Sketch: An Interface for Sketching 3D Scenes
,” ACM SIGGRAPH 2007 Courses, ACM, New York, p.
19
.
28.
Jatupoj
,
P.
, 2005, “
Sketchboard: The Simple 3D Modeling From Architectural Sketch Recognition
,”
Proceedings of the 10th International Conference on Computer Aided Architectural Design Research in Asia
, CAADRIA, New Delhi, India, pp. 3–22.
29.
Qin
,
S. F.
,
Wright
,
D. K.
, and
Jordanov
,
I. N.
,
2000
, “
From On-Line Sketching to 2D and 3D Geometry: A System Based on Fuzzy Knowledge
,”
Comput.-Aided Des.
,
32
(
14
), pp.
851
866
.
30.
Chaudhuri
,
S.
,
Kalogerakis
,
E.
,
Guibas
,
L.
, and
Koltun
,
V.
,
2011
, “
Probabilistic Reasoning for Assembly-Based 3D Modeling
,”
ACM Trans. Graph. (TOG)
,
30
(
4
), p.
35
.
31.
Kalogerakis
,
E.
,
Chaudhuri
,
S.
,
Koller
,
D.
, and
Koltun
,
V.
,
2012
, “
A Probabilistic Model for Component-Based Shape Synthesis
,”
ACM Trans. Graph.
,
31
(
4
), p. 55.
32.
Chaudhuri
,
S.
,
Kalogerakis
,
E.
,
Giguere
,
S.
, and
Funkhouser
,
T.
,
2013
, “
Attribit: Content Creation With Semantic Attributes
,”
26th Annual ACM Symposium on User Interface Software and Technology
, St. Andrews, UK, Oct. 8–11, pp.
193
202
.
33.
Jaiswal
,
P.
,
Huang
,
J.
, and
Rai
,
R.
,
2016
, “
Assembly-Based Conceptual 3D Modeling With Unlabeled Components Using Probabilistic Factor Graph
,”
Comput.-Aided Des.
,
74
, pp.
45
54
.
34.
Sung
,
M.
,
Su
,
H.
,
Kim
,
V. G.
,
Chaudhuri
,
S.
, and
Guibas
,
L.
,
2017
, “
Complement Me: Weakly-Supervised Component Suggestions for 3D Modeling
,”
ACM Trans. Graph. (TOG)
,
36
(
6
), p.
226
.
35.
Krupka
,
E.
,
Karmon
,
K.
,
Bloom
,
N.
,
Freedman
,
D.
,
Gurvich
,
I.
,
Hurvitz
,
A.
,
Leichter
,
I.
,
Smolin
,
Y.
,
Tzairi
,
Y.
,
Vinnikov
,
A.
, and
Bar-Hillel, A.
,
2017
, “
Toward Realistic Hands Gesture Interface: Keeping It Simple for Developers and Machines
,”
CHI Conference on Human Factors in Computing Systems
, Denver, CO, May 6–11, pp.
1887
1898
.
36.
Vitali
,
A.
, and
Rizzi
,
C.
,
2017
, “
A Virtual Environment to Emulate Tailor's Work
,”
Comput.-Aided Des. Appl.
,
14
(
5
), pp.
671
679
.
37.
Microsoft
,
2018
, “
Gestures Mixed Reality
,” Microsoft, Redmond, WA, accessed Aug. 8, 2018, https://docs.microsoft.com/en-us/windows/mixed-reality/gestures
38.
The CGAL Project,
2018
,
CGAL User and Reference Manual
,
4.12 ed.
,
CGAL Editorial Board
.
39.
Chew
,
L. P.
,
1989
, “
Constrained Delaunay Triangulations
,”
Algorithmica
,
4
(
1–4
), pp.
97
108
.
40.
Mapari
,
R. B.
, and
Kharat
,
G.
,
2016
, “
American Static Signs Recognition Using Leap Motion Sensor
,”
Second International Conference on Information and Communication Technology for Competitive Strategies
, Udaipur, India, Mar. 4–5, p.
67
.
41.
Marin
,
G.
,
Dominio
,
F.
, and
Zanuttigh
,
P.
,
2014
, “
Hand Gesture Recognition With Leap Motion and Kinect Devices
,”
IEEE International Conference on Image Processing
(
ICIP
), Paris, France, Oct. 27–30, pp.
1565
1569
.
42.
Zeiler
,
M. D.
, and
Fergus
,
R.
,
2014
, “
Visualizing and Understanding Convolutional Networks
,”
European Conference on Computer Vision
, Zurich, Switzerland, Sept. 6–12, pp.
818
833
.https://cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf
43.
Simonyan
,
K.
, and
Zisserman
,
A.
,
2014
, “
Very Deep Convolutional Networks for Large-Scale Image Recognition
,” arXiv Preprint:
1409:1556
.https://arxiv.org/abs/1409.1556
44.
Russakovsky
,
O.
,
Deng
,
J.
,
Su
,
H.
,
Krause
,
J.
,
Satheesh
,
S.
,
Ma
,
S.
,
Huang
,
Z.
,
Karpathy
,
A.
,
Khosla
,
A.
,
Bernstein
,
M.
,
Berg
,
A. C.
, and
Fei-Fei
,
L.
,
2015
, “
ImageNet Large Scale Visual Recognition Challenge
,”
Int. J. Comput. Vision (IJCV)
,
115
(
3
), pp.
211
252
.
45.
Chollet
,
F.
,
2015
, “
Keras
,” Keras, accessed Aug. 7, 2018, https://keras.io
46.
Dalal
,
N.
, and
Triggs
,
B.
,
2005
, “
Histograms of Oriented Gradients for Human Detection
,”
IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(
CVPR
), San Diego, CA, June 20–25, pp.
886
893
.
47.
Hammond
,
T.
, and
Davis
,
R.
,
2005
, “
Ladder, a Sketching Language for User Interface Developers
,”
Comput. Graph.
,
29
(
4
), pp.
518
532
.
48.
Kara
,
L. B.
, and
Stahovich
,
T. F.
,
2005
, “
An Image-Based, Trainable Symbol Recognizer for Hand-Drawn Sketches
,”
Comput. Graph.
,
29
(
4
), pp.
501
517
.
49.
Rabiner
,
L. R.
,
1990
, “
Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition
,”
Readings in Speech Recognition
,
Morgan Kaufmann Publishers Inc
.,
San Francisco, CA
, pp.
267
296
50.
Viterbi
,
A.
,
1967
, “
Error Bounds for Convolutional Codes and an Asymptotically Optimum Decoding Algorithm
,”
IEEE Transactions Inf. Theory
,
13
(
2
), pp.
260
269
.
51.
Tsang
,
S.
,
Balakrishnan
,
R.
,
Singh
,
K.
, and
Ranjan
,
A.
,
2004
, “
A Suggestive Interface for Image Guided 3D Sketching
,”
SIGCHI Conference on Human Factors in Computing Systems
, Vienna, Austria, Apr. 24–29, pp.
591
598
.
52.
Wigdor
,
D.
, and
Wixon
,
D.
,
2011
,
Brave Nui World: Designing Natural User Interfaces for Touch and Gesture
,
Elsevier
, Burlington, MA.
53.
Nielsen
,
M.
,
Störring
,
M.
,
Moeslund
,
T. B.
, and
Granum
,
E.
,
2003
, “
A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI
,” International Gesture Workshop, Genoa, Italy, Apr. 15–17.
54.
Thakur
,
A.
, and
Rai
,
R.
,
2015
, “
User Study of Hand Gestures for Gesture Based 3D CAD Modeling
,”
ASME
Paper No. DETC2015-46086
.
55.
Mansutti
,
A.
,
Rodriguez
,
M. C.
,
Bordegoni
,
M.
, and
Cugini
,
U.
,
2017
,
Tactile Display for Virtual 3D Shape Rendering
,
Springer
, Cham, Switzerland.
You do not currently have access to this content.