Laparoscopic robots have been widely adopted in modern medical practice. However, explicitly interacting with these robots may increase the physical and cognitive load on the surgeon. An attention-aware robotic laparoscope system has been developed to free the surgeon from the technical limitations of visualization through the laparoscope. This system can implicitly recognize the surgeon's visual attention by interpreting the surgeon's natural eye movements using fuzzy logic and then automatically steer the laparoscope to focus on that viewing target. Experimental results show that this system can make the surgeon–robot interaction more effective, intuitive, and has the potential to make the execution of the surgery smoother and faster.
Issue Section:
Research Papers
References
1.
Wikipedia
, 2014
, “Laparoscopic Surgery
,” accessed Sept. 29, 2014, http://en.wikipedia.org/wiki/Laparoscopic_surgery2.
Martins Rus
, J. R.
, Jatene
, F. B.
, de Campos
, J. R.
, Monteiro
, R.
, Tedde
, M. L.
, Samano
, M. N.
, Bernardo
, W. M.
, and Das-Neves-Perira
, J. C.
, 2009
, “Robotic Versus Human Camera Holding in Video-Assisted Thoracic Sympathectomy: A Single Blind Randomized Trial of Efficacy and Safety
,” Interact. Cardiovasc. Thorac. Surg.
, 8
(2
), pp. 195
–199
.3.
Stolzenburg
, J. U.
, Franz
, T.
, Kallidonis
, P.
, Minh
, D.
, Dietel
, A.
, Hicks
, J.
, Nicolaus
, M.
, Al-Aown
, A.
, and Liatsikos
, E.
, 2010
, “Comparison of the FreeHand® Robotic Camera Holder With Human Assistants During Endoscopic Extraperitoneal Radical Prostatectomy
,” BJU Int.
, 107
(6
), pp. 970
–974
.4.
Taniguchi
, K.
,Nishikawa
, A.
, Sekimoto
, M.
,Kobayashi
, T.
, Kazuhara
, K.
, Ichihara
, T.
, Kurashita
, N.
, Takiguchi
, S.
, Doki
, Y.
, Mori
, M.
, and Miyazaki
, F.
, 2010
, “Classification, Design and Evaluation of Endoscope Robots
,” Robot Surgery
, Seung Kyuk Baik, ed., InTech
, Rijeka, Croatia, pp. 1
–24
.5.
Sackier
, J. M.
, and Wang
, Y.
, 1994
, “Robotically Assisted Laparoscopic Surgery: From Concept to Development
,” Surg. Endoscopy
, 8
(1
), pp. 63
–66
.6.
Aiono
, S.
, Gilbert
, J. M.
, Soin
, B.
, Finlay
, P. A.
, and Gordan
, A.
, 2002
, “Controlled Trial of the Introduction of a Robotic Camera Assistant for Laparoscopic Cholecystectomy
,” Surg. Endoscopy
, 16
(9
), pp. 1267
–1270
.7.
Prosurgics
, 2014
, “FreeHand
,” Prosurgics Inc., San Jose, CA, accessed Nov. 13, 2014, http://www.freehandsurgeon.com8.
Polet
, R.
, and Donnez
, J.
, 2004
, “Gynecologic Laparoscopic Surgery With a Palm-Controlled Laparoscope Holder
,” J. Am. Assoc. Gynecol. Laparoscopists
, 11
(1
), pp. 73
–78
.9.
Tanoue
, K.
, Yasunaga
, T.
, Kobayashi
, E.
, Miyamoto
, S.
, Sakuma
, I.
, Dohi
, T.
, Konishi
, K.
, Yamaguchi
, S.
, Kinjo
, N.
, Takenaka
, K.
, Maehara
, Y.
, and Hashizume
, M.
, 2006
, “Laparoscopic Cholecystectomy Using a Newly Developed Laparoscope Manipulator for 10 Patients With Cholelithiasis
,” Surg. Endoscopy
, 20
(5
), pp. 753
–756
.10.
Guthart
, G. S.
, and Salisbury
, J. K.
, Jr., 2000
, “The Intuitive™ Telesurgery System: Overview and Application
,” IEEE International Conference on Robotics and Automation
(ICRA '00
), San Francisco, CA, Apr. 24–28, pp. 618
–621
.11.
Kraft
, B. M.
, Jager
, C.
, Kraft
, K.
, Leibl
, B. J.
, and Bittner
, R.
, 2004
, “The AESOP Robot System in Laparoscopic Surgery; Increased Risk or Advantage for Surgeon and Patient?
” Surg. Laparoscope
, 18
(8
), pp. 1216
–1223
.12.
Gilbert
, J. M.
, 2009
, “The EndoAssist Robotic Camera Holder as an Aid to the Introduction of Laparoscopic Colorectal Surgery
,” Ann. R. Coll. Surg. Engl.
, 91
(5
), pp. 389
–393
.13.
Wei
, G.
, Arbter
, K.
, and Hirzinger
, G.
, 1997
, “Real-Time Visual Servoing for Laparoscopic Surgery. Controlling Robot Motion With Color Image Segmentation
,” IEEE Eng. Med. Biol. Mag.
, 16
(1
), pp. 40
–45
.14.
Wang
, Y. F.
, Uecker
, D. R.
, and Wang
, Y.
, 1998
, “A New Framework for Vision-Enabled and Robotically Assisted Minimally Invasive Surgery
,” Comput. Med. Imaging Graphics
, 22
(6
), pp. 429
–437
.15.
Nishikawa
, A.
, Asano
, S.
, Fujita
, R.
, Yohda
, T.
, Miyazaki
, F.
, Sekimoto
, M.
, Yasui
, M.
, Takiguchi
, S.
, and Monden
, M.
, 2003
, “Robust Visual Tracking of Multiple Surgical Instruments for Laparoscopic Surgery
,” Computer Assisted Radiology and Surgery 17th International Congress and Exhibition
(CARS 2003
), London, June 25–28, p. 1372
.16.
Ko
, S. Y.
, and Kwon
, D. S.
, 2004
, “A Surgical Knowledge Based Interaction Method for a Laparoscopic Assistant Robot
,” 13th IEEE International Workshop on Robot and Human Interactive Communication
(ROMAN 2004
), Kurashiki, Japan, Sept. 20–22, pp. 313
–318
.17.
Lee
, C.
, Wang
, Y.-F.
, Uecker
, D. R.
, and Wang
, Y.
, 1994
, “Image Analysis for Automated Tracking in Robot-Assisted Endoscopic Surgery
,” 12th IAPR International Conference on Pattern Recognition
(ICPR
), Conference A: Computer Vision and Image Processing, Jerusalem, Oct. 9–13, pp. 88
–92
.18.
Kranzfelder
, M.
, Schneider
, A.
, Blahusch
, G.
, Schaaf
, H.
, and Feussner
, H.
, 2009
, “Feasibility of Opto-Electronic Surgical Instrument Identification
,” Minimally Invasive Ther. Allied Technol.
, 18
(5
), pp. 253
–258
.19.
Nishikawa
, A.
, Asano
, S.
, fujita
, R.
, Yamaguchi
, S.
, Yohda
, T.
, Miyazaki
, F.
, Sekimoto
, M.
, Yasui
, M.
, Miyake
, Y.
, Takiguchi
, S.
, and Monden
, M.
, 2003
, “Selective Use of Face Gesture Interface and Instrument Tracking System for Control of a Robotic laproscope Positioner
,” 6th International Conference on Medical Image Computing and Computer-Assisted Intervention
(MICCAI 2003
), Montréal, Canada, Nov. 15–18, pp. 973
–974
.20.
Merchant
, J.
, Morrissette
, R.
, and Porterfield
, J.
, 1974
, “Remote Measurement of Eye Direction Allowing Subject Motion Over One Cubic Foot of Space
,” IEEE Trans. Biomed. Eng.
, 21
(4
), pp. 309
–317
.21.
Morimoto
, C. H.
, and Mimica
, M. R. M.
, 2000
, “Eye Gaze Tracking Techniques for Interactive Applications
,” Comput. Vision Image Understanding
, 18
(4
), pp. 331
–335
.22.
Stampe Dave
, M.
, 1993
, “Heuristic Filtering and Reliable Calibration Method for Video-Based Pupil-Tracking Systems
,” Behav. Res. Methods Instrum. Comput.
, 25
(2
), pp. 137
–142
.23.
Ji
, Q.
, and Zhu
, Z.
, 2002
, “Eye and Gaze Tracking for Interactive Graphic Display
,” Machine Vision and Application
, Springer
, New York
, pp. 139
–148
.24.
Barea
, R.
, Boquete
, L.
, Ortega
, S.
, Lopez
, E.
, and Rodriguez-Ascariz
, J. M.
, 2012
, “EOG-Based Eye Movements Codification for Human Computer Interaction
,” Expert Syst. Appl.
, 39
(3
), pp. 2677
–2683
.25.
Krolak
, A.
, and Strumillo
, P.
, 2012
, “Eye-Blink Detection System for Human-Computer Interaction
,” Univers. Access Inf. Soc.
, 11
(4
), pp. 409
–419
.26.
Liu
, C.
, Ishi
, C. T.
, Ishiguro
, H.
, and Hagita
, N.
, 2012
, “Generation of Nodding, Head Tilting and Eye Gazing for Human-Robot Dialogue Interaction
,” 7th ACM/IEEE International Conference on Human-Robot Interaction
(HRI
), Boston, Mar. 5–8, pp. 285
–292
.27.
Ma
, J.
, Zhang
, Y.
, Cichocki
, A.
, and Matsuno
, F.
, 2015
, “A Novel EOG/EEG Hybrid Human-Machine Interface Adopting Eye Movements and ERPs: Application to Robot Control
,” IEEE Trans. Biomed. Eng.
, 62
(3
), pp. 876
–889
.28.
Hansen
, D. W.
, and Ji
, Q.
, 2010
, “In the Eye of the Beholder: A Survey of Models for Eye and Gaze
,” IEEE Trans. Pattern Anal. Mach. Intell.
, 32
(3
), pp. 478
–500
.29.
30.
COGAIN Communication by Gaze Interaction
, 2013
, “Gazetalk
,” COGAIN Association, Frederiksberg, Denmark, accessed Oct. 3, 2014, http://wiki.cogain.org/index.php/Gazetalk31.
Tobii
, 2014
, “Computer Access Through Eye Gaze
,” Tobii AB, Danderyd, Sweden, accessed July 3, 2014, http://www.tobii.com/PCEye201132.
Purwanto
, D.
, Mardiyanto
, R.
, and Arai
, K.
, 2009
, “Electric Wheelchair Control With Gaze Direction and Eye Blinking
,” Artif. Life Rob.
, 14
(3
), pp. 397
–400
.33.
Atienza
, R.
, and Zelinsky
, A.
, 2005
, “Intuitive Interface Through Active 3D Gaze Tracking
,” International Conference on Active Media Technology
(AMT 2005
), Kagawa, Japan, May 19–21, pp. 16
–21
.34.
Noonan
, D. P.
, Mylonas
, G. P.
, Shang
, J.
, Payne
, C. J.
, Darzi
, A.
, and Yang
, G.-Z.
, 2010
, “Gaze Contingent Control for an Articulated Mechatronic Laparoscope
,” 3rd IEEE, RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics
(BioRob
), Tokyo, Sept. 26–29.35.
Staub
, C.
, Can
, S.
, Jensen
, B.
, Knoll
, A.
, and Kohlbecher
, S.
, 2012
, “Human-Computer Interfaces for Interaction With Surgical Tools in Robotic Surgery
,” 4th IEEE, RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics
(BioRob
), Rome, June 24–27, pp. 81
–86
.36.
Zhang
, X.
, Li
, S.
, Zhang
, J.
, and Williams
, H.
, 2013
, “Gaze Contingent Control for a Robotic Laparoscope Holder
,” ASME J. Med. Devices
, 7
(2
), p. 020915
.37.
Li
, S.
, Zhang
, J.
, Xue
, L.
, Fernando
, K.
, and Zhang
, X.
, 2013
, “Attention-Aware Robotic Laparoscope for Human-Robot Cooperative Surgery
,” IEEE International Conference Robotics and Biomimetics
(ROBIO
), Shenzhen, China, Dec. 12–14, pp. 792
–797
.38.
Jacob
, R. J. K.
, and Karn
, K. S.
, 2003
, “Eye Tracking in Human-Computer Interaction and Usability Research: Ready to Deliver the Promises
,” Mind
, 2
(3
), pp. 577
–605
.39.
Glenstrup
, A. J.
, and Engell-Nielsen
, T.
, 1995
, “Eye Controlled Media: Present and Future
,” Bachelor's thesis, Information Psychology, Laboratory of Psychology, University of Copenhagen, Copenhagen, Denmark.40.
Møllenbach
, E.
, Hansen
, J. P.
, and Lillholm
, M.
, 2013
, “Eye Movements in Gaze Interaction
,” J. Eye Mov. Res.
, 6
(2
), pp. 1
–15
.41.
Zhang
, X.
, and Nelson
, C. A.
, 2008
, “Kinematic Analysis and Optimization of a Novel Robot for Surgical Tool Manipulation
,” ASME J. Med. Devices
, 2
(2
), p. 021003
.42.
Nelson
, C. A.
, Zhang
, X.
, Shah
, B. C.
, Goede
, R. M.
, and Oleynikov
, D.
, 2009
, “Multipurpose Surgical Robot as a Laparoscope Assistant
,” Surg. Endoscopy
, 24
(7
), pp. 1528
–1532
.43.
Nelson
, C. A.
, Zhang
, X.
, and Oleynikov
, D.
, 2009
, “Tool Guidance Using a Compact Robotic Assistant
,” J. Rob. Surg.
, 3
(3
), pp. 171
–173
.44.
Kim
, J.
, Abdel-Malek
, K.
, Mi
, Z.
, and Nebel
, K.
, 2004
, “Layout Design Using an Optimization-Based Human Energy Consumption Formulation
,” SAE
Technical Paper 2004-01-2175.45.
46.
Wikipedia
, 2014
, “Fuzzy Control System
,” accessed Jan. 20, 2015, http://en.wikipedia.org/wiki/Fuzzy_control_system47.
Dadios
, E. P.
, Biliran
, J. J. C.
, Garcia
, R.-R. G.
, Johnson
, D.
, and Valencia
, A. R. B.
, 2012
, “Humanoid Robot: Design and Fuzzy Logic Control Technique for Its Intelligent Behaviors
,” Fuzzy Logic: Controls, Concepts, Theories and Applications
, InTech
, Rijeka, Croatia
, pp. 3
–20
.48.
Hong
, T. S.
, Nakhaeinia
, D.
, and Karasfi
, B.
, 2012
, “Application of Fuzzy Logic in Mobile Robot Navigation
,” Fuzzy Logic: Controls, Concepts, Theories and Applications
, InTech
, Rijeka, Croatia
, pp. 21
–36
.49.
Ahmad
, S.
, Siddique
, N. H.
, and Tokhi
, M. O.
, 2012
, “Modular Fuzzy Logic Controller for Motion Control of Two-Wheeled Wheelchair
,” Fuzzy Logic: Controls, Concepts, Theories and Applications
, InTech
, Rijeka, Croatia
, pp. 37
–58
.50.
Brooke
, J.
, 1996
, “SUS: A Quick and Dirty Usability Scale
,” Usability Evaluation in Industry
, Taylor & Francis
, London
, p. 189
.Copyright © 2015 by ASME
You do not currently have access to this content.
