Abstract

Endoscopic endonasal surgery is a commonly practiced minimally invasive neurosurgical operation for the treatment of a wide range of skull base pathologies including pituitary tumors. A common shortcoming of this surgery is the necessity of a third hand when the endoscope has to be handled to allow active use of both hands of the main surgeon. The robot surgery assistant NeuRoboScope system has been developed to take over the endoscope from the main surgeon's hand while providing the surgeon with the necessary means of controlling the location and direction of the endoscope. One of the main novelties of the NeuRoboScope system is its human–robot interface designs which regulate and facilitate the interaction between the surgeon and the robot assistant. The human–robot interaction design of the NeuRoboScope system is investigated in two domains: direct physical interaction (DPI) and master–slave teleoperation (MST). The user study indicating the learning curve and ease of use of the MST is given and this paper is concluded via providing the reader with an outlook of possible new human–robot interfaces for the robot assisted surgery systems.

Introduction

This paper presents the novel human–robot interfaces developed for a robot-assisted surgery (RAS) system, the NeuRoboScope system. This system is specifically developed for the minimally invasive endoscopic pituitary tumor surgery. In the next part, the specifics of the pituitary tumors, minimally invasive surgical procedure for pituitary tumor surgery, and the need for the RAS system for this surgery are explained. These explanations constitute the grounds on why the human–robot interfaces of this system are developed. After the necessity of the RAS system for the pituitary tumor resection surgery is explained, background on the existing systems is presented to clarify the differences and novelty of the NeuRoboScope system.

Endoscopic Pituitary Tumor Surgery.

The pituitary gland is a special secretory organ located at the base of the skull between both optic nerves. The main task of the pituitary gland is to maintain the continuity of life by regulating the metabolism with the hormones it contains. Although it is a small pea-sized gland, it is the main endocrine organ governing other secretory glands in the body such as adrenal, thyroid, reproductive organs. The regulation of many functions such as growth and development, water and salt balance, sugar balance, blood pressure, pregnancy and birth, milk production, development of sexual organs and reproduction is maintained by the hormones secreted from this gland into the body. Thus, pathologies of this gland such as tumors often reveal many signs and symptoms such as loss of vision, excessive weight gain, hypertension, diabetes, uncontrolled growth in face and extremities, decrease in sexual desire and infertility.

Tumors that originate from endocrine cells in the anterior part of the pituitary gland are called pituitary adenomas. The prevalence of pituitary adenomas in the population was found to be as high as 22.5% in radiological studies and 14.4% in autopsy studies [1]. Adenomas are classified as micro (<1 cm) or macroadenoma (>1 cm) according to their size. It is estimated that the prevalence of macroadenomas in the population is one in 600. It constitutes the third most common tumor group among intracranial tumors that show clinical symptoms and require surgical treatment, with a rate of 10–25% after glial and meningeal tumors [2]. Despite the high prevalence in the population, clinical findings occur in only a portion of the cases and treatment is required for those patients. But, potentially, pituitary adenomas grow over the years and cause clinical problems, so early diagnosis and effective treatment are important.

Pituitary adenomas can be resected either through nose (via endoscopic or microscopic transsphenoidal approach) or opening in the skull. Recently, endoscopic approach through the nasal cavity has gained popularity over microscopic techniques due its certain advantages. In microscopic transsphenoidal pituitary surgery, microsurgery is performed by looking through a rigid speculum placed in the nose under microscope magnifying the image with 4–10×. However, since it has to be viewed through the speculum and from a stationary distance (7–15 cm), visualization is very limited particularly for the suprasellar area and cavernous sinus walls laterally. Compared to microscopic technique, endoscopic pituitary surgery has the following advantages:

  • Its better illumination and fiberoptic-lens system used in different directions provides a wide and angled field of view near the lesion.

  • The anatomical structures in the nasal passage are better visualized and protected.

  • When reaching the base of the skull, no retractors are used, and a variety of specialized surgical instruments can be comfortably used.

  • Residual tumors can be detected and removed with angled endoscopes.

  • The surgical procedure is less traumatic, and there is no need for postoperative nasal packing, so patients are more comfortable in the postoperative period and can be discharged on the first day after surgery if there are no complications [3,4].

In a recently published meta-analysis study comparing both methods, endoscopic surgery has shown higher tumor resection and clinical cure rates and shorter hospital stay, better patient comfort [5,6]. It has also been demonstrated that the endocrinological and radiological cure rates provided by endoscopic pituitary surgery are higher, and that the endoscopic approach is more cost-effective (low cost, high effect) than microscopic surgery [7]. Thus, endoscopic surgery has become the standard operative procedure in the treatment of pituitary tumors as well as a wide range of other skull base pathologies.

In endoscopic surgery, the optical system called the endoscope is inserted by the surgeon through the patient's nostril, first into the nasal cavity, and then into the sphenoid sinus (air cavity) in front of the saddle-shaped pit called sella turcica at the base of skull, where the pituitary gland is located and the image is obtained. The tumor in the sellar region is resected with special surgical instruments, which are also entered by the surgeon through the nostrils. The depiction of surgical approach (a) and representative coronal (b) and sagittal (c) radiological images is presented in Fig. 1.

Among the limitations of the endoscope are as follows:

  • Due to the lack of stereoscopic vision provided by the microscope, the image can be obtained in two dimensions and the sense of depth is not perceived,

  • the endoscope lens is frequently obscured by blood and debris,

  • the surgeon (or the assistant) must use one hand to hold the endoscope; therefore, there is difficulty of manipulation/limitation of maneuver in the surgical area that is already limited (this problem can be partially overcome by working through the both of the nostrils).

Problem Statement.

One of the most important problems encountered during this surgery is that the surgeon has to solely spare one hand for the control of the endoscope. This situation causes fatigue for the surgeon during the operation, which takes approximately 2–4 h, and reduces maneuverability and efficiency due to the use of one hand for the rest of the surgical tools. As a solution to this problem, control of the endoscope by the assistant surgeon, fixed endoscope holders, voice-controlled robotic endoscope routers have been used [8]. Figure 2 shows the situation where the surgeon holds the endoscope in his left hand, uses a surgical instrument in his right hand, and the surgical assistant assists the surgeon with another surgical instrument using at least one hand. This way of working is often required during surgery. However, the clinical use of these methods is limited by the fact that there are significant disadvantages of surgeries performed with the assistance of an assistant and those performed with existing endoscope guiding/holding systems. The disadvantages are usually related to ease of use, speed of operation, and safety.

To propose a solution to the abovementioned problems, a RAS system called the NeuRoboScope has been developed to instantly direct the imaging tool (endoscope) according to the demands of the main surgeon. The system allows the primary surgeon to use three instruments, one of which being endoscope itself indirectly controlled via the human–robot interface of NeuRoboScope, simultaneously. In this way, it is expected that the performance and efficiency of the surgical operation will be increased and the operation time will be shortened. In the NeuRoboScope system, thanks to the master system, which can be worn by the surgeon or can be easily mounted to any surgical instrument during surgery, the surgeon's demands are detected instantly, processed appropriately, and acquired motion information is transmitted to the robot arm handling the endoscope. In this way, the main surgeon is able to direct the endoscope at any time during the surgery while using two different surgical instruments with two hands at the same time. Robot handling the endoscope has a macromini manipulator design, which integrates a passive serial arm with no actuation system and an active (actuated) parallel manipulator with a remote-center-of-motion (RCM). In this way, the macromovements are achieved by the surgeon via physical interaction with the passive arm and small/mini motions are performed by the active arm. In this way, only precise small motions are handled by the master–slave teleoperation system increasing the safety of the system. Thus, in this scenario, master system is the wearable ring and the slave system is the active arm. To the best of our knowledge, this setting of the master–slave system, which has the wearable ring based master subsystem in a RAS setting, has not been reported in the literature of minimally invasive surgery systems previously. The investigations of such systems that handle the endoscope are given in the next part.

Endoscope Holder Systems in Robot-Assisted Surgery.

The use of robotic devices in surgery is increasing day by day [9]. While some of these devices carry out the whole surgical operation, some are developed as assistive devices for the surgeon. Among the RAS systems, endoscope holding devices have been developed for surgeries using endoscopes as imaging devices. Taniguchi et al. [10] noted 27 endoscope robots developed between 1994 and 2009. Eight of these were used on humans while the others were tested on animals or presented as models only. In most of these systems, the endoscope device is positioned by a robot arm while the surgeon usually controls this arm with a remote control. As an alternative to the control systems in which the surgeon controls the endoscope handling robot via his/her hands or feet, there are systems that are controlled using speech recognition [11] or an image processing system that tracks the surgeon's head movement [12].

Taniguchi et al. [10] state that yaw, pitch, and translation motions along the endoscope's telescope axis of the robot are sufficient for directing and positioning the endoscope. While an active system is required to achieve these movements, the position of the endoscope with respect to the patient is usually provided by a passive arm or a mobile table. Passive positioning arms are widely available in the market. On the other hand, motor-driven positioning arms that the surgeon can control have also been developed [11,13,14].

In this part, only the prominent endoscope holder and guiding mechanisms are investigated. Kobayashi et al. [15] developed an assistive robot called Naviot to be used in surgeries performed with oral entry. The Naviot system includes a control device that can be placed on any device. The surgeon controls the two buttons on the control system with his thumb. The rotational motions of the endoscope are controlled by a 5 bar mechanism and the arm holding the endoscope. In addition, the image size can be enlarged up to six times with the active optical focusing system. Taniguchi et al. [16] utilized a two-degree-of-freedom (DoF) serial arm for controlling the orientation of the endoscope and they used zooming rather than a translational motion along the endoscope's telescope axis. Nishikawa et al. [17] developed a system that can direct any laparoscope with the hydraulically driven 6DoF Stewart–Gough platform called P-arm.

There are also endoscope-holding devices developed especially for surgeries performed by entering through the nose and sinus cavity. Wurm et al. [18] developed a series of 6DoF robotic arm systems including an endoscope as the end-effector for use in paranasal sinus surgery. Zimmerman et al. [19] developed a surgical robot on the mobile platform called evolution 1, which includes a 6DoF Stewart–Gough platform connected to a serial 5DoF arm. The parallel mechanism with 6DoF is reported to be controlled with 10 μm positioning accuracy in forward–backward motion along the telescope axis and rotation about three axes. Nimsky et al. [20] studied the use of this robot in transsphenoidal surgery.

When the references are examined, it is seen that either kinematically redundant robot systems are used for the control of the endoscope, or a robotic arm with 2DoF angular orientation capability and an optical focusing system has been developed instead of forward and backward movement along the telescope axis. In minimally invasive pituitary tumor surgery, the telescope is directed about a pivot point with small-range rotations in two axes as reported in our earlier study [21]. Although the endoscope has the capability to adjust the magnification by use of the buttons on it, in practice, surgeons adjust the magnification once in the beginning of the surgery and keep it as it is throughout the surgery most of the time. Therefore, even inside the surgical field, a small-range translational motion is required. Another critical issue is the use of angled view telescopes to visualize all portions of the surgical field for any residual tumor. For this action, surgeons need an extra rotation about the telescope axis.

Passive arms have been used in the previous RAS examples and a passive arm is also required for the RAS system for the pituitary tumor operation (Fig. 3). A passive arm is to be used for locating an active arm to the surgical field by the surgeon. In this way, macromotions of the endoscope are not achieved by a controlled robot arm. Hence, the small-workspace active arm can have high precision motions that are required for this system.

It is known that parallel mechanisms are more advantageous than serial ones for high precision positioning in a small workspace [22]. Therefore, the active arm system is designed as a 3DoF parallel manipulator to move about a pivot point. Among these three DoFs, two of them are rotations about axes perpendicular to the telescope axis and one of them is the translation along the telescope axis. The kinematics and dynamics of this active arm are already published prior to this paper [21,23,24]. The rotation about the telescope axis, which is seldom required for scanning the surgery area, is left for the surgeon to manually control by means of the design of a special quick-release mechanism for the endoscope. This quick-release mechanism is introduced for the first time in an academic paper. In addition, in the developed system, it is possible for the surgeon to direct the endoscope robot when there are surgical instruments in both hands and the instruments are in the operation area. This study differs from previous RAS system architectures with the kinematic architecture of the parallel manipulator, teleoperation system, and various human–robot interfaces.

To explain the human–robot interface of the NeuRoboScope system, all of its subsystems should be introduced and investigated. The human–robot interaction of the NeuRoboScope system takes place at two distinct domains. The first human–robot interaction domain is the physical interaction of the endoscope with the patient's soft tissues inside the nostrils. This is a natural physical-human–robot interaction case for an interventional surgical robot system. The second domain of human–robot interaction is the interaction of the surgeon with the NeuRoboScope system. The second domain of interaction happens in two ways: (a) direct physical interaction (DPI) and (b) master–slave teleoperation (MST).

Briefly, the works we have carried out to address a solution to the problem stated previously are as follows:

  • Design of a novel active endoscope holder (AEH) (published before in Refs. [23] and [24]).

  • Design of a passive arm to locate the active endoscope holder

  • Design of a quick-release system to release the endoscope from the system whenever necessary and enable the rotation of the endoscope about its telescope axis

  • Design of the master system including a wearable ring with Bluetooth communication and an inertial measurement unit (IMU)

  • Design of the teleoperation system to enhance the user ergonomics

In the System Architecture of the NeuRoboScope System section, system architecture of the NeuRoboScope system is introduced by presenting its subsystems and the human–robot interfaces that are specifically designed and developed for DPI and MST domains.

System Architecture of the NeuRoboScope System

The NeuRoboScope system is a teleoperated RAS system in which the surgeon is working alongside the robotic system. The surgeon wears a specifically designed ring that has an IMU as the wearable master system in the teleoperation setting. The slave system consists of the AEH and passive balanced arm (PBA); however, only the active arm can be controlled via the master system in the teleoperation setting. There is Bluetooth communication between the wearable ring and the AEH. The master system includes a pedal to activate the voluntary demands of the surgeon during the surgery. PBA is not directed via teleoperation but it is moved by the surgeon manually. A block diagram indicating the information exchange between the NeuRoboScope's master and slave systems and their components is presented in Fig. 4.

At the beginning of the surgery, the surgeon sets the required height of the PBA via a separate remote control. Then, the surgeon presses the PBA's brakes release (PABR) button, which is indicated as MKB in Fig. 4, and manually locates the tip of the telescope to the tip of the patient's nose and triggers the surgery zone assignment (SZA) button, which is indicated as ASB button in Fig. 4. This button is placed near the surgeon along with the emergency stop button and PABR button. At this time, the endoscope is ready to be placed inside the surgery zone and therefore, the surgeon places the endoscope inside the surgery one by manually moving the endoscope while the PABR is still pressed. When the endoscope is inside the surgery zone, PABR is released and the brakes of the PBA are activated. After this instant, the PBA acts as an inertial frame for the AEH.

After the abovementioned steps, teleoperation of the AEH is initiated whenever the surgeon sees it necessary to change the view of the image reflected by the endoscope. The activation of the teleoperation is commanded by the surgeon when the foot pedal is pressed and the teleoperation continues as long as the foot pedal is maintained in the pressed state.

The endoscope is connected to the AEH via an easy plug-in/out (EPIO) system. The surgeon can quickly release the endoscope from the NeuRoboScope system via the EPIO system by using one hand and carry out the surgery manually if he/she wishes to do so. The EPIO system also enables the surgeon to rotate the endoscope about its telescope axis when an angled telescope is used to monitor unreachable locations of the surgery zone.

Active endoscope holder includes a force sensor located at the connection port of the endoscope quick release system. The excessive forces/torques measured by this sensor during the operation is displayed to the surgeon via a light-emitting-diode panel (indicated as “user interface” in Fig. 4) located on the monitor that displays the endoscope images. Therefore, there is no direct haptic feedback in this teleoperation setting but the haptic information is displayed to the surgeon in an indirect way via visualization.

The subsystems of the NeuRoboScope system are shown in Fig. 5. The subsystems are introduced in the Passive Balanced Arm, Active Endoscope Holder, and Teleoperation System sections by giving details about the human–robot interfaces used in the development of these subsystems.

Passive Balanced Arm.

Passive balanced arm is a statically balanced manipulator with only one active joint and 6DoFs. The main purpose of the PBA is to provide a fixed frame to the AEH and to enable the manual positioning of the AEH around the surgery table.

As seen in Fig. 6, PBA has a portable structure that can be connected to the operation table. The first joint, which moves vertically with respect to the operating table, is an active prismatic joint which enables the surgeon to adjust the height of the AEH, which is attached to the end point of the PBA, before the operation starts. The linear motor used as the drive element operates completely independently thanks to its own power supply and remote control unit.

Apart from the active prismatic joint, there are five passive revolute joints in PBA. Third passive revolute joint structure is modified to a parallelogram mechanism and in this way; a spring-based static balancing system could be used. Another spring-based gravity balancing is designed for the fourth joint. According to the safety requirements, the gravity balancing system is designed so that when the PBA is free to move, it moves the AEH upward, i.e., away from the patient's head.

In each of the revolute joints, an electromagnetic brake and an angular position sensor are placed. The angular position sensors in these joints are absolute encoders with 12-bit resolution. The measurements taken from the angular position sensors are used in forward kinematics to calculate the position and orientation of the AEH. This information is used in the control algorithms of the general system to determine whether the endoscope is in the surgical area.

Other than the first prismatic joint, all other joints of the PBA are designed to be manually backdrivable when their brakes are released. Hence, this type of a physical human–robot interaction constitutes the first human–robot interface design in the DPI domain of NeuRoboScope system.

Active Endoscope Holder.

Active endoscope holder is the teleoperated slave system of the NeuRoboScope system that is designed to direct and position the endoscope according to the commands received from the master system. As shown in Fig. 7 [23], the 3DoF parallel mechanism mobile platform has 2R1T (two rotations and one translation) motion with RCM at the pivot point D. The kinematic architecture of the mechanism is such that three planes comprising A0A1A2A3, B0B1B2B3, and C0C1C2C3 legs intersect along the line A3D, which represents the endoscope axis. The side legs A0A1A2A3 and C0C1C2C3 connecting the base platform to the end effector are serial chains with five revolute joints, where the first and the last axes are within the side plane and the three joint axes in between are perpendicular to the plane. For the side legs, the first joint axes along A0D and C0D are actuated and these joints are responsible for orienting the end-effector line A3D. The actuated joints are located within the red rectangles in Fig. 7. The middle leg B0B1B2B3 is a serial chain with five revolute joints, where the first axis is within the middle plane and the remaining three joint axes are perpendicular to the plane. Points B2, B3, and D are on the end effector. As opposed to the other legs, not the fixed joint axis, but the second joint axis (passing through B0, perpendicular to the X-axis) is actuated to provide translational motion of the endoscope through the pivot point D. The details of position and velocity level kinematics of the mechanism is presented in Ref. [24].

Taniguchi et al. [10] stated that for directing an endoscope in the surgical zone, two rotations about the axes perpendicular to the telescope axis and translational motion along the telescope axis around RCM are necessary and sufficient. Using these three DoFs, various robot mechanisms have been proposed and tested [26].

The workspace of the AEH is constrained as the motion within the surgical field, which is defined as the sellar/parasellar area. The workspace requirements of the AEH are determined via combining two methods. In the first method, a specifically developed IMU-based sensor is attached to the endoscope while the surgeons imitate the actual surgery on a cadaver. While the surgeons moved the endoscope inside the surgery zone, the angular motion ranges are measured and acquired. In the second method, the surgeons measured the morphometric limits of surgical zone using CT images of the patients. Surgeons use the tip of the nose as their RCM during the surgery. From these radiological imaging data, the position of the RCM with respect to the surgical zone is also defined. A statistical evaluation of the CT results provided us with the mean and standard deviations of the measurements. The IMU-based measurements received from the first method are extrapolated to the 99% confidence interval of the measurements from the CT images. The workspace of the endoscope motion is determined as ±20 deg rotational motion range about the X-axis, ±15 deg rotational motion range about the Y-axis in Fig. 7 and up to 100 mm insertion of the endoscope tip through pivot point D. The link lengths of the AEH are designed accordingly based on optimized mechanical advantage within the desired workspace [21,24]. The designed link lengths are summarized in Table 1.

Each actuation system has a capstan drive with 1:5 speed reduction ratio and a 12-bit encoder attached to the driver's shaft. The middle leg's rotated arm has a total motion of 153.74 deg which result in 100 mm motion of the endoscope's telescope along the pivot point. Hence, the nominal resolution of this motion is 87 counts per mm of linear motion. To move within the boundaries of the workspace, the side legs rotate ±25.19 deg and the actuation system configuration is the same. Consequently, the nominal resolution received during the motion about the X-axis is 71 counts per degree and the Y-axis is 94 counts per degree.

The novelty of the RCM mechanism designed for the NeuRoboScope system is about the legs of the parallel mechanism distributed around the mobile platform carrying the endoscope. The legs are located at the backside of the endoscope (toward the head of operating table) with 45 deg apart from each other. This leaves enough space for the surgeon to reach the endoscope and change the image parameters via the buttons on top of the endoscope (Fig. 8). Another key issue in the constructional design is to have a minimized footprint of the design so that the legs of the RCM mechanism do not block the view of the surgeon while he/she looks at the monitor displaying the images captured by the endoscope. The details about the kinematics and constructional design of the RCM mechanisms are available in Ref. [24].

Although there is already a human–machine interface on the endoscope, the design of the RCM mechanism facilitated the access to these buttons on the endoscope. This easy access of the endoscope is used to develop another DPI-type human–robot interface which we called the EPIO mechanism for the endoscope (Fig. 8). The EPIO mechanism is developed so that the surgeon can continue the surgery by separating the endoscope from the robot with one hand in an emergency situation or whenever the surgeon sees the need to do so. The components of the EPIO design are shown in Fig. 9 presenting its prototype.

In the EPIO mechanism, the locking process occurs as a result of the linear movement. In ideal conditions, the mechanism works in two positions. In order to constrain the motion of the push button within these positions, there are two stopper pins at the top and bottom in the working area of the push button. A cam mechanism is used for converting the rotational motion to linear locking motion. The pressure flange is kept in constant contact with the cam and the compression springs. Since the moving bowl is combined with the pressure flange via connecting pins, every movement of the pressure flange are observed in the moving bowl. When the push button is pressed, the pressure flange moves downward due to the shape of the cam together with the moving bowl. As a result of this movement, the mobile bowl slides downward opening the motion path of the pushing pin, and the pushing pin moves forward with the effect of the compression spring behind it, allowing the endoscope to plug out easily and prevent the moving bowl from going upward. Consequently, in this position, the endoscope is released from the NeuRoboScope system.

When the endoscope is desired to be reinstalled, the surgeon places the endoscope on top of the moving bowl and pushes back the barrier pin and as a result, the moving bowl makes a linear movement upward with the effect of the tension spring that is installed on the push button the and locks the endoscope to the NeuRoboScope system. This locking action is a result of the normal force generated by the tension spring, which causes the friction between the endoscope's mounting piece and the EPIO.

In addition to the features described above, the surgeon can manually rotate the endoscope around its telescope axis without removing it from the EPIO mechanism. When this rotation action is necessary, the push button is pressed to an intermediate position and the friction force between the endoscope mounting piece and the EPIO is minimized so that the connection becomes a shape-closed revolute joint. The intermediate position of the push button is sensed by an encoder and the surgeon is notified that he/she is in the intermediate position by the small vibration on the EPIO induced by a vibration motor.

Teleoperation System.

The information exchange of the NeuRoboScope's unilateral teleoperation system is shown in Fig. 10. The master system of the teleoperation scheme consists of two subcomponents: a wearable ring remote control device worn by the surgeon during the entire surgery to detect the surgeon's hand movements, and a foot pedal that initiates/terminates the teleoperation. The foot pedal is used also to determine the motion mapping scenario/operation state of the teleoperation.

Inside the ring, an IoT Sensor Development Kit developed by the SmartBond company is placed. This IoT Sensor Development Kit has a low-power Bluetooth Smart SoC (Fig. 11), IMU with 6DoF (3-axis accelerometer and 3-axis gyroscope), 3DoF geomagnetic field sensor, and integrated peripheral unit (for pressure, temperature and humidity measurement) [28]. The part indicated as breakable connection header in Fig. 11 is used only in the development process to embed the algorithm into the microprocessor. After the program is embedded inside the microprocessor, this part can be broken so that the overall hardware size is small enough to be placed inside the ring shown in Fig. 11.

The surgeon wears the ring preferably to his hand that he/she used to handle the endoscope during surgery. This system transmits the surgeon's hand movements as angular velocity and acceleration information at a frequency of 100 Hz.

The slave system consists of three components; (1) a microcontroller that receives the teleoperation data and runs the teleoperation control algorithm that generates the speed input for the control of the AEH, (2) endoscope robot controller, which is a microcontroller that processes the speed input generated for the endoscope and sends it to the motor drivers, and (3) AEH, which the endoscope robot with joint actuator and sensors.

In this unilateral teleoperation setting, there are two communication channels; (1) wireless channel: angular velocity and acceleration vectors measured by the ring system are transmitted to the microcontroller, and (2) wired channel: the status of the pedal is transmitted to the microcontroller.

Via the teleoperation control algorithm, hand movements are processed by integrating the angular velocity data transmitted by the ring. When the foot pedal is pressed, it takes the angular position difference and converts these angles into angular velocity input for the endoscope robot according to the selected operation state of the system. This operation style of the teleoperation system is generally defined as the mapping of position to velocity [29].

The system has three operation states: OFF, ROTATION, and DOLLY. In the OFF state, the pedal is released (not pressed), the teleoperation scheme is disabled, and the angle integrator is reset. In the case of ROTATION and DOLLY, pedal is pressed and teleoperation scheme is active. Active system's operation state is determined by the way the surgeon presses the foot pedal. A visual example of the conversion of the angle change in the pitch movement of the ring into the velocity input for the endoscope's pitching motion input is shown in Fig. 12.

Evaluation of the Master–Slave Teleoperation Domain Human–Robot Interface

Since only rotational motion is measured via the wearable ring, one of the rotational motions should be mapped to the linear motion, which is denoted as dolly motion in Fig. 12. Three different operation scenarios are examined in an earlier study [27]. The testing of the operation scenarios is the key evaluation step to assess the ease of use of the NeuRoboScope system's MST type human–robot interface. In all operation scenarios, rotational motions and dolly (translational) motion are separated. A single press on the foot pedal initiates the rotational motion mapping and a single click and then a press on the foot pedal (called also the double-click) initiates the dolly motion mapping. The scenarios are as follows:

  • Scenario 1: All three axes of the measured rotational motion are mapped to the two rotational and one translational motion of the endoscope, separately. In this case, all fingers are coupled to the surgical tool that the surgeon handles with this hand wearing the ring. Hence, this scenario is called coupled 3D.

  • Scenario 2: Only two axes of the measured rotational motion are used in mapping. Since dolly motion is issued separately (with the double click of the foot pedal), one of the two measured rotations is used for this mapping. In this case, all fingers are coupled to the surgical tool that the surgeon handles with this hand wearing the ring. Hence, this scenario is called coupled 2D.

  • Scenario 3: Surgeon uses the index finger to wear the ring and this finger is unattached to the surgical tool when he/she teleoperates the endoscope. The rest of the mapping is the same as scenario 1. Hence, this scenario is called uncoupled 2D.

Training software is prepared in which different virtual reality scenarios can be integrated. Figure 13 shows the training software with the virtual reality scenario including the endoscopic image of the surgery field. The test setup in Fig. 13 is composed of a laptop displaying the virtual reality scenario, a skull mockup, surgical tools, and the master system of the teleoperation consisting of the wearable ring device and the foot pedal.

To test the ease of use of the teleoperation scenarios, this test setup's virtual reality screen is changed to a screen with objectives that can be easily followed as shown in Fig. 14. Since the surgeon handles a tool with the hand that he/she wears the ring, the location of the tool is displayed on the screen. The aim is to move the tool on the screen to the current goal location and adjust its diameter with respect to the diameter of the current goal.

Ten subjects, who are not surgeons, are used in this study. In each test run, the locations and diameters of the goals are changed. The results indicated that after ten trials the average task completion time (Fig. 15) and the amount of foot pedal pressings (Fig. 16) decreased to the similar values for all three operation scenarios [27].

The task completion duration trend indicates the learning curve. It can be clearly observed by focusing on the initial trial results from Figs. 13 and 14 that the uncoupled mode is more intuitive with respect the other modes. However, these results indicate that after a certain amount of training, any of the operation modes can be used depending on the choice of the surgeon.

Conclusions

The goal of this article is to present the novel DPI and MST type human–robot interfaces of a RAS system specifically developed for the minimally invasive endoscopic pituitary tumor surgery. Nevertheless, the human–robot interfaces presented in this paper can be adapted to other endoscopic/laparoscopic/arthroscopic surgery systems.

NeuRoboScope system includes two DPI and one MST type human–robot interfaces that reduce the workload of the surgeon during the surgery, increase efficiency, and shorten the duration of the surgery. The DPI type human–robot interfaces are tested and evaluated by the surgeons in a subjective way and they are tuned according to their personal requirements. The parameters that are tuned for DPI domain human–robot interfaces are the EPIO mechanism's springs' stiffness, location of the push button of the EPIO, mounting location of the PBA on the surgery table.

The MST type human–robot interface's operation scenarios are evaluated via a specific test setup designed for this purpose. The results indicated that any of the operation modes can be used after an appropriate amount of training. In fact, via a preliminary test procedure to be designed, the most efficient operation can be selected and tuned for each surgeon.

Another outlook of this study is including another DPI type human–robot interface by providing the surgeon with the capability to backdrive the AEH during the surgery. In this way, whenever the surgeon deems necessary, he/she can bypass the MST type human–robot interface and use this new DPI type human–robot interface to fine-tune the direction of the endoscope.

Acknowledgment

We would like to acknowledge all the graduate and undergraduate-level research assistants who took part in the development process of the NeuRoboScope system. These students are: Gizem Ateş, Sercan Erat, Bengisu Uzun, Omar Maaroof, Eren Vardarli, Merve Özkahya, Cevahir Karagöz, Abdullah Yaşir, Oğulcan Işitman, Orhan Ayit, Emir Mobedi, Ronny Majani, Görkem Yavaş, Tural Hidayatov, Yiğit Güngör.

Funding Data

  • Türkiye Bilimsel ve Teknolojik Araştirma Kurumu (Grant Nos. 115E725 and 115E726; Funder ID: 10.13039/501100004410).

References

References
1.
Ezzat
,
S.
,
Asa
,
S. L.
,
Couldwell
,
W. T.
,
Barr
,
C. E.
,
Dodge
,
W. E.
,
Vance
,
M. L.
, and
McCutcheon
,
I. E.
,
2004
, “
The Prevalence of Pituitary Adenomas
,”
Cancer
,
101
(
3
), pp.
613
619
.10.1002/cncr.20412
2.
Jane
,
J. A.
, Jr.
,
Sulton
,
L. D.
, and
Laws
,
E. R.
, Jr.
,
2005
, “
Surgery for Primary Brain Tumors at United States Academic Training Centers: Results From the Residency Review Committee for Neurological Surgery
,”
J. Neurosurg.
,
103
(
5
), pp.
789
–7
93
.10.3171/jns.2005.103.5.0789
3.
Cappabianca
,
P.
,
Cavallo
,
L. M.
, and
de Divitiis
,
E.
,
2004
, “
Endoscopic Endonasal Transsphenoidal Surgery
,”
Neurosurgery
,
55
(
4
), pp.
933
941
.10.1227/01.NEU.0000137330.02549.0D
4.
Berker
,
M.
,
Hazer
,
D. B.
,
Yücel
,
T.
,
Gürlek
,
A.
,
Cila
,
A.
,
Aldur
,
M.
, and
Onerci
,
M.
,
2012
, “
Complications of Endoscopic Surgery of the Pituitary Adenomas: Analysis of 570 Patients and Review of the Literature
,”
Pituitary
,
15
(
3
), pp.
288
300
.10.1007/s11102-011-0368-2
5.
Rotenberg
,
B.
,
Tam
,
S.
,
Ryu
,
W. H.
, and
Duggal
,
N.
,
2010
, “
Microscopic Versus Endoscopic Pituitary Surgery: A Systematic Review
,”
Laryngoscope
,
120
(
7
), pp.
1292
1297
.10.1002/lary.20949
6.
Gao
,
Y.
,
Zhong
,
C.
,
Wang
,
Y.
,
Xu
,
S.
,
Guo
,
Y.
,
Dai
,
C.
,
Zheng
,
Y.
,
Wang
,
Y.
,
Luo
,
Q.
, and
Jiang
,
J.
,
2017
, “
Endoscopic Versus Microscopic Transsphenoidal Pituitary Adenoma Surgery: A Meta-Analysis
,”
World J. Surg. Oncol.
,
101
, pp.
236
246
.10.1016/j.wneu.2017.01.022
7.
Rudmik
,
L.
,
Starreveld
,
Y. P.
,
Vandergrift
,
W. A.
,
Banglawala
,
S. M.
, and
Soler
,
Z. M.
,
2015
, “
Cost-Effectiveness of the Endoscopic Versus Microscopic Approach for Pituitary Adenoma Resection
,”
Laryngoscope
,
125
(
1
), pp.
16
24
.10.1002/lary.24780
8.
Nathan
,
C. A. O.
,
Chakradeo
,
V.
,
Malhotra
,
K.
,
D'Agostino
,
H.
, and
Patwardhan
,
R.
,
2006
, “
The Voice-Controlled Robotic Assist Scope Holder AESOP for the Endoscopic Approach to the Sella
,”
Skull Base
,
16
(
3
), pp.
123
131
.10.1055/s-2006-939679
9.
Kim
,
K. C.
,
2014
,
Robotics in General Surgery
,
Springer
,
New York
.
10.
Taniguchi
,
K.
,
Nishikawa
,
A.
,
Sekimoto
,
M.
,
Kobayashi
,
T.
,
Kazuhara
,
K.
,
Ichihara
,
T.
,
Kurashita
,
N.
,
Takiguchi
,
S.
,
Doki
,
Y.
,
Mori
,
M.
, and
Miyazaki
,
F.
,
2010
,
Robot Surgery
(Classification, Design and Evaluation of Endoscope Robots),
Intech Open Access Book
,
London, UK
.
11.
Wang
,
Y.
,
Laby
,
K. P.
,
Uecker
,
D. R.
,
Mangaser
,
A. A.
, and
Ghodoussi
,
M.
,
1999
, “
Automated Endoscope System for Optimal Positioning
,” Patent No. U.S.5878193.
12.
Nishikawa
,
A.
,
Hosoi
,
T.
,
Koara
,
K.
,
Daiji
,
N.
,
Hikita
,
A.
,
Asano
,
S.
,
Miyazaki
,
F.
,
Sekimoto
,
M.
,
Miyake
,
Y.
,
Yasui
,
M.
, and
Monden
,
M.
,
2001
, “
Real-Time Visual Tracking of the Surgeon's Face for Laparoscopic Surgery
,” Medical Image Computing and Computer-Assisted Intervention—MICCAI 2001,
Springer
,
Berlin
.
13.
Aiono
,
S.
,
Gilbert
,
J. M.
,
Soin
,
B.
,
Finlay
,
P. A.
, and
Gordan
,
A.
,
2002
, “
Controlled Trial of the Introduction of a Robotic Camera Assistant (EndoAssist) for Laparoscopic Cholecystectomy
,”
Surg. Endoscopy
,
16
(
9
), pp.
1267
1270
.10.1007/s00464-001-9174-7
14.
Hourley
,
P.
,
2006
, “
How to Maintain the Quality of Laparoscopic Surgery in the Era of Lack of Hands?
,”
Acta Chir. Belg.
,
106
, pp.
22
26
.10.1080/00015458.2006.11679826
15.
Kobayashi
,
E.
,
Masamune
,
K.
,
Sakuma
,
I.
,
Dohi
,
T.
, and
Hashimoto
,
D.
,
1999
, “
A New Safe Laparoscopic Manipulator System With a Five-Bar Linkage Mechanism and an Optimal Zoom
,”
Comput. Aided Surg.
,
4
(
4
), pp.
182
192
.10.3109/10929089909148172
16.
Taniguchi
,
K.
,
Nishikawa
,
A.
,
Yohda
,
T.
, Ito, K.,
Sekimoto
,
M.
,
Yasui
,
M.
,
Takiguchi
,
S.
,
Monden
,
M.
, and
Miyazaki
,
F.
,
2007
, “
Development of a Compact Oblique-viewing Endoscope Robot for Laparoscopic Surgery
,”
Trans. Jpn. Soc. Med. Biol. Eng.
, 45(
1
), pp.
36
47
.10.11239/jsmbe.45.36
17.
Nishikawa
,
A.
,
Sekimoto
,
M.
,
Taniguchi
,
K.
,
Yamada
,
Y.
,
Miyoshi
,
N.
,
Takiguchi
,
S.
,
Doki
,
Y.
,
Mori
,
M.
, and
Miyazaki
,
F.
,
2010
,
Advanced Strategies for Robot Manipulators
(Design and Control of a Compact Endoscope Manipulator: A Biologically Inspired Approach),
InTech Open Access Book
,
London, UK
.
18.
Wurm
,
J.
,
Steinhart
,
H.
,
Bumm
,
K.
,
Vogele
,
M.
,
Nimsky
,
C.
, and
Iro
,
H.
,
2003
, “
A Novel Robot System for Fully Automated Paranasal Sinus Surgery
,”
Int. Congr. Ser.
,
1256
, pp.
633
638
.10.1016/S0531-5131(03)00284-X
19.
Zimmermann
,
M.
,
Krishnan
,
R.
,
Raabe
,
A.
, and
Seifert
,
V.
,
2002
, “
Robot-Assisted Navigated Neuroendoscopy
,”
Neurosurgery
,
51
(
6
), pp.
1446
1452
.10.1097/00006123-200212000-00015
20.
Nimsky
,
C.
,
Rachinger
,
J.
,
Iro
,
H.
, and
Fahlbusch
,
R.
,
2004
, “
Adaptation of a Hexapod-Based Robotic System for Extended Endoscope-Assisted Transsphenoidal Skull Base Surgery
,”
Minimally Invasive Neurosurg.
,
47
(
1
), pp.
41
46
.10.1055/s-2003-812465
21.
iDede
,
M.
,
Kiper
,
C.
,
Ayav
,
G.
,
Tatlıcıoğlu
,
T.
,
Özdemirel
,
E. B.
,
Maaroof
,
O.
,
Ateş
,
W.
,
Berker
,
G.
,
Işıkay
,
M. İ.
, and
Hanalioğlu
,
Ş.
,
2018
, “
Cerrahın Anlık Yönlendirilebildiği Robot Yardımlı Endoskop Kontrol Sistemi Mimarisi—NeuRoboScope
,” Türkiye Robotbilim Konferansı, İstanbul, Turkey, Apr. 12– 14, pp.
1
6
.
22.
Merlet
,
J. P.
,
2006
,
Parallel Robots
,
Springer
,
Dordrecht, The Netherlands
.
23.
Yaşır
,
A.
, and
Kiper
,
G.
,
2017
, “
Structural Synthesis of 2R1T Type Mechanisms for Minimally Invasive Surgery Applications
,”
Proceedings of the Fourth MeTrApp Conference
, Trabzon, Turkey, July 3–5, pp.
31
38
.10.1007/978-3-319-60702-3_4
24.
Yaşır
,
A.
,
Kiper
,
G.
, and
Dede
,
M. İ. C.
,
2020
, “
Kinematic Design of a Non-Parasitic 2R1T Parallel Mechanism With Remote Center of Motion to Be Used in Minimally Invasive Surgery Applications
,”
Mech. Mach. Theory
,
153
, p. 104013.10.1016/j.mechmachtheory.2020.104013
25.
Ateş
,
G.
,
2018
, “
Teleoperation System Design of a Robot Assisted Endoscopic Pituitary Surgery
,”
M.Sc. thesis
,
İzmir Institute of Technology
,
İzmir, Turkey
.http://hdl.handle.net/11147/6958
26.
Gogu
,
G.
,
2012
,
Structural Synthesis of Parallel Robots - Part 4: Other Topologies With Two and Three Degrees of Freedom
,
Springer
,
Amsterdam, The Netherlands
.
27.
Ateş
,
G.
,
Majani
,
R.
, and
Dede
,
M. İ. C.
,
2018
, “
Design of a Teleoperation Scheme With a Wearable Master for Minimally Invasive Surgery
,”
Proceedings of the Sixth International Workshop on Medical and Service Robots
, Cassino, Italy, July 4–6, pp.
45
53
.10.1007/978-3-030-00329-6_6
28.
Dialog Semiconductor,
2020
, “
DAI14583 IoT Sensor Development Kit
,” Dialog Semiconductor, Reading, UK, accessed Sept. 13, 2020, https://www.dialog-semiconductor.com/iotsensor
29.
Dede
,
M. İ. C.
,
Maaroof
,
O.
,
Ateş
,
W.
,
Berker
,
G.
,
Işıkay
,
M.
, and
Hanalioğlu
,
İ. Ş.
,
2016
, “
Unilateral Teleoperation Design for a Robotic Endoscopic Pituitary Surgery System
,”
Proceedings of the Fifth International Workshop on Medical and Service Robots
, Graz, Austria, July 4–8, pp.
101
115
.10.1007/978-3-319-59972-4_8