We explore tangible 3D interactions that allow for geometric and perceptual correspondence between a midair modality and the 3D elements it controls. To demonstrate our approach, we use a concrete application scenario through Proto-TAI++, a multimodal system using pen-based drawing of planar shapes and their subsequent midair assembly via a hand-held planar proxy. The planarity of the proxy is a key element that physically embodies virtual planar shapes during 3D manipulation. We also investigate advanced operations such as patterning and blending using asymmetric bimanual interactions that augment midair motion with multitouch gestures. We describe a three-stage user evaluation with our system wherein our goal is to (a) study the effectiveness of the planar proxy as a tangible 3D modality, (b) contrast key features of our approach with a GUI-based planar shape assembly system, and (c) evaluate user experience and performance in creative tasks using Proto-TAI++.

References

References
1.
Fitzmaurice
,
G. W.
,
Ishii
,
H.
, and
Buxton
,
W. A.
,
1995
, “
Bricks: Laying the Foundations for Graspable User Interfaces
,”
SIGCHI
Conference on Human Factors in Computing Systems
, Denver, CO, May 7–11, pp.
442
449
.
2.
Ishii
,
H.
, and
Ullmer
,
B.
,
1997
, “
Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms
,”
SIGCHI
Conference on Human Factors in Computing Systems
, Atlanta, GA, Mar. 22–27, ACM, New York, pp.
234
241
.
3.
Hinckley
,
K.
,
Pausch
,
R.
,
Goble
,
J. C.
, and
Kassell
,
N. F.
,
1994
, “
Passive Real-World Interface Props for Neurosurgical Visualization
,”
SIGCHI
Conference on Human Factors in Computing Systems
, Boston, MA, Apr. 24–28, ACM, New York, pp.
452
458
.
4.
Arisandi
,
R.
,
Otsuki
,
M.
,
Kimura
,
A.
,
Shibata
,
F.
, and
Tamura
,
H.
,
2014
, “
Virtual Handcrafting: Building Virtual Wood Models Using ToolDevice
,”
Proc. IEEE
,
102
(
2
), pp.
185
195
.
5.
Oh
,
Y.
,
Johnson
,
G.
,
Gross
,
M.
, and
Do
,
E. Y.-L.
,
2006
, “
The Designosaur and the Furniture Factory
,”
Design Computing and Cognition ‘06
, Springer, Dordrecht, The Netherlands, pp.
123
140
.
6.
Johnson
,
G.
,
Gross
,
M.
,
Do
,
E. Y.-L.
, and
Hong
,
J.
,
2012
, “
Sketch It, Make It: Sketching Precise Drawings for Laser Cutting
,”
SIGCHI
Extended Abstracts on Human Factors in Computing Systems
, Austin, TX, May 5–10, ACM, New York, pp.
1079
1082
.
7.
Piya
,
C.
, and
Ramani
,
K.
,
2014
, “
Proto-TAI: Quick Design Prototyping Using Tangible Assisted Interfaces
,”
ASME
Paper No. DETC2014-35442.
8.
McCrae
,
J.
,
Umetani
,
N.
, and
Singh
,
K.
,
2014
, “
FlatFitFab: Interactive Modeling With Planar Sections
,” 27th Annual
ACM
Symposium on User Interface Software and Technology
, Honolulu, HI, Oct. 5–8, ACM, New York, pp.
13
22
.
9.
Rivers
,
A.
,
Durand
,
F.
, and
Igarashi
,
T.
,
2010
, “
3D Modeling With Silhouettes
,”
SIGGRAPH ‘10
Special Interest Group on Computer Graphics and Interactive Techniques Conference
, Los Angeles, CA, July 26–30, ACM, New York, Paper No. 109.
10.
McCrae
,
J.
,
Singh
,
K.
, and
Mitra
,
N. J.
,
2011
, “
Slices: A Shape-Proxy Based on Planar Sections
,”
ACM Trans. Graphics
,
30
(
6
), p.
168
.
11.
Hildebrand
,
K.
,
Bickel
,
B.
, and
Alexa
,
M.
,
2012
, “
crdbrd: Shape Fabrication by Sliding Planar Slices
,”
Comput. Graphics Forum
,
31
(
2 Pt. 3
), pp.
583
592
.
12.
Mueller
,
S.
,
Lopes
,
P.
, and
Baudisch
,
P.
,
2012
, “
Interactive Construction: Interactive Fabrication of Functional Mechanical Devices
,” 25th Annual
ACM
Symposium on User Interface Software and Technology
, Cambridge, MA, Oct. 7–10, ACM, New York, pp.
599
606
.
13.
Saul
,
G.
,
Lau
,
M.
,
Mitani
,
J.
, and
Igarashi
,
T.
,
2011
, “
SketchChair: An All-in-One Chair Design System for End Users
,”
Fifth International Conference on Tangible, Embedded, and Embodied Interaction
, Funchal, Portugal, Jan. 22–26, ACM, New York, pp.
73
80
.
14.
Marquardt
,
N.
,
Jota
,
R.
,
Greenberg
,
S.
, and
Jorge
,
J. A.
,
2011
, “
The Continuous Interaction Space: Interaction Techniques Unifying Touch and Gesture on and Above a Digital Surface
,”
Human-Computer Interaction—
INTERACT 2011
, Springer, Lisbon, Portugal, pp.
461
476
.
15.
Huppmann
,
D.
,
Luderschmidt
,
J.
,
Haubner
,
N.
,
Lehmann
,
S.
,
Dörner
,
R.
, and
Schwanecke
,
U.
,
2012
, “
Exploring and Evaluating the Combined Multi-Touch and In-the-Air Tabletop Interaction Space
,”
Workshop Virtuelle und Erweiterte Realität der GI-Fachgruppe VR/AR
, C. Geiger, J. Herder, and T. Vierjahn, ed., Shaker Verlag, Aachen, Germany, Vol. 9, pp. 37–48.
16.
De Araújo
,
B. R.
,
Casiez
,
G.
,
Jorge
,
J. A.
, and
Hachet
,
M.
,
2013
, “
Mockup Builder: 3D Modeling on and Above the Surface
,”
Comput. Graphics
,
37
(
3
), pp.
165
178
.
17.
Guiard
,
Y.
,
1987
, “
Asymmetric Division of Labor in Human Skilled Bimanual Action: The Kinematic Chain as a Model
,”
J. Mot. Behave.
,
19
(
4
), pp.
486
517
.
18.
Wang
,
R.
,
Paris
,
S.
, and
Popović
,
J.
,
2011
, “
6D Hands: Markerless Hand-Tracking for Computer Aided Design
,” 24th Annual
ACM
Symposium on User Interface Software and Technology
, Santa Barbara, CA, Oct. 16–19, ACM, New York, pp.
549
558
.
19.
Vinayak
,
Murugappan
,
S.
,
Liu
,
H.
, and
Ramani
,
K.
,
2013
, “
Shape-It-Up: Hand Gesture Based Creative Expression of 3D Shapes Using Intelligent Generalized Cylinders
,”
Comput.-Aided Des.
,
45
(
2
), pp.
277
287
.
20.
Oikonomidis
,
I.
,
Kyriazis
,
N.
, and
Argyros
,
A. A.
,
2011
, “
Efficient Model-Based 3D Tracking of Hand Articulations Using Kinect
,”
22nd British Machine Vision Conference
,
BMVC 2011
, Dundee, Scotland, UK, Aug. 29–Sept. 2, Vol.
1
, pp.
101.1
101.11
.
21.
Song
,
P.
,
Goh
,
W.
,
Hutama
,
W.
,
Fu
,
C.
, and
Liu
,
X.
,
2012
, “
A Handle Bar Metaphor for Virtual Object Manipulation With Mid-Air Interaction
,”
CHI
Conference on Human Factors in Computing Systems
, Austin, TX, May 5–10, ACM, New York, pp.
1297
1306
.
22.
Jerald
,
J.
,
Mlyniec
,
P.
,
Yoganandan
,
A.
,
Rubin
,
A.
,
Paullus
,
D.
, and
Solotko
,
S.
,
2013
, “
MakeVR: A 3D World-Building Interface
,”
IEEE Symposium on 3D User Interfaces
(
3DUI
), Orlando, FL, Mar. 16–17, pp.
197
198
.
23.
Schkolne
,
S.
,
Pruett
,
M.
, and
Schröder
,
P.
,
2001
, “
Surface Drawing: Creating Organic 3D Shapes With the Hand and Tangible Tools
,”
Human Factors in Computing Systems
, Seattle, WA, Mar. 31–Apr. 5, ACM, New York, pp.
261
268
.
24.
Mine
,
M.
,
Yoganandan
,
A.
, and
Coffey
,
D.
,
2014
, “
Making VR Work: Building a Real-World Immersive Modeling Application in the Virtual World
,”
Symposium on Spatial User Interaction
, Honolulu, HI, Oct. 4–5, ACM, New York, pp.
80
89
.
25.
Sheng
,
J.
,
Balakrishnan
,
R.
, and
Singh
,
K.
,
2006
, “
An Interface for Virtual 3D Sculpting Via Physical Proxy
,”
International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia
,
GRAPHITE ‘06
, Kuala Lumpur, Malaysia, Nov. 28–Dec. 2, Vol.
6
, pp.
213
220
.
26.
Kato
,
H.
,
Billinghurst
,
M.
,
Poupyrev
,
I.
,
Imamoto
,
K.
, and
Tachibana
,
K.
,
2000
, “
Virtual Object Manipulation on a Table-Top AR Environment
,”
IEEE and ACM International Symposium on Augmented Reality
(
ISAR
), Munich, Germany, Oct. 5–6, pp.
111
119
.
27.
Bowman
,
D.
,
Kruijff
,
E.
,
LaViola
,
J. J.
, Jr.
, and
Poupyrev
,
I.
,
2004
,
3D User Interfaces: Theory and Practice, CourseSmart eTextbook
,
Addison-Wesley
, Boston, MA.
28.
Zhai
,
S.
,
Milgram
,
P.
, and
Buxton
,
W.
,
1996
, “
The Influence of Muscle Groups on Performance of Multiple Degree-of-Freedom Input
,”
SIGCHI
Conference on Human Factors in Computing Systems
, Vancouver, BC, Canada, Apr. 13–18, ACM, New York, pp.
308
315
.
29.
Buchta
,
C.
,
Kober
,
M.
,
Feinerer
,
I.
, and
Hornik
,
K.
,
2012
, “
Spherical k-Means Clustering
,”
J. Stat. Software
,
50
(
10
), pp.
1
22
.
30.
LaViola
,
J. J.
,
2003
, “
Double Exponential Smoothing: An Alternative to Kalman Filter-Based Predictive Tracking
,”
Workshop on Virtual Environments
, Zurich, Switzerland, May 22–23, ACM, New York, pp.
199
206
.
31.
Schwarz
,
J.
,
Marais
,
C. C.
,
Leyvand
,
T.
,
Hudson
,
S. E.
, and
Mankoff
,
J.
,
2014
, “
Combining Body Pose, Gaze, and Gesture to Determine Intention to Interact in Vision-Based Interfaces
,”
CHI
Conference on Human Factors in Computing Systems
, Toronto, ON, Canada, Apr. 26–May 1, ACM, New York, pp.
3443
3452
.
32.
Thiel
,
Y.
,
Singh
,
K.
, and
Balakrishnan
,
R.
,
2011
, “
Elasticurves: Exploiting Stroke Dynamics and Inertia for the Real-Time Neatening of Sketched 2D Curves
,” 24th Annual
ACM
Symposium on User Interface Software and Technology
, Santa Barbara, CA, Oct. 16–19, ACM, New York, pp.
383
392
.
33.
Baudel
,
T.
,
1994
, “
A Mark-Based Interaction Paradigm for Free-Hand Drawing
,” 7th
ACM
Symposium on User Interface and Software Technology
, Marina del Rey, CA, Nov. 2–4, ACM, New York, pp.
185
192
.
34.
Olafsdottir
,
H.
, and
Appert
,
C.
,
2014
, “
Multi-Touch Gestures for Discrete and Continuous Control
,”
International Working Conference on Advanced Visual Interfaces
, Como, Italy, May 27–29, ACM, New York, pp.
177
184
.
35.
Eick
,
S. G.
, and
Wills
,
G. J.
,
1995
, “
High Interaction Graphics
,”
Eur. J. Oper. Res.
,
81
(
3
), pp.
445
459
.
36.
Gower
,
J. C.
,
1975
, “
Generalized Procrustes Analysis
,”
Psychometrika
,
40
(
1
), pp.
33
51
.
You do not currently have access to this content.