This article discusses various challenges and puts forth the concepts of intelligent surgical robots and complementary situational awareness (CSA) as a means for achieving new surgical systems with unprecedented capabilities in terms of safety, ease of operation, and exact execution of pre-operative surgical plans. Situational awareness is defined in accordance with by the three stages of sensory acquisition, sensory comprehension, and projection. A robotic system with CSA assists the user not only in manipulation, but also in forming the situational awareness regarding the task at hand by using perception resources beyond the capabilities of the user. In the future, it is expected that CSA systems will increasingly be embedded within a larger framework of Computer-Integrated Interventional Medicine, in which patient-specific information such as images, lab results, and genomics are combined with general knowledge to model and diagnose the patients’ condition and to develop an optimized treatment plan.
During the past thirty years, surgeons gradually converted open surgical procedures to minimally invasive laparoscopy and then to robot-assisted multi-port minimally invasive surgery (MIS). This conversion from open to MIS surgical technique has been driven by the aim of decreasing patient trauma, wound site infection, risk of incisional hernia, and post-operative recovery time and scarring.
Surgeons use multiple incisions (typically three to six incisions) to access the anatomy during multi-port MIS. By using insufflation of the surgical site (e.g. the abdomen) the operational field is enlarged to facilitate visualization and operation of multiple tools. Typically, three tools are used (right and left arms for manipulation/ablation and a third arm for visualization). Other access ports may be used for organ retraction and auxiliary tasks of suction or delivery of tools such as blood vessel clips. For instance, the da-Vinci Si system (Figure 1) uses a quadruple-armed tele-manipulator allowing the operation of additional tools and collaboration among two surgeons. The dexterous tools of the da-Vinci slave robot are matched via a telemanipulation interface to the mechanical architecture of the wrists of the master user interface. A high level telemanipulation computer relays commands from the master user interface to the slave arms thus allowing motion replication of surgeon's hand movements with tremor filtering and motion scaling.
Robotic systems such as in Figure 1 have successfully enabled many surgical procedures using the multiport MIS approach. However, with the aim of further reducing trauma to the patient, the last decade has seen significant growth in works investigating MIS in confined spaces, single port access (SPA) and natural orifice trans-luminal endoscopic surgery (NOTES). These new surgical paradigms present surgeons with unprecedented challenges that require a new approach to robotic assistance. This paper discusses these challenges and puts forth the concepts of intelligent surgical robots and complementary situational awareness (CSA) as means for achieving new surgical systems with unprecedented capabilities in terms of safety, ease of operation, and exact execution of pre-operative surgical plans. Within the context of this paper, intelligent surgical robots are robots capable of sensing and regulating their interaction with the environment in order to assist the surgeon in achieving safe surgical intervention and to facilitate CSA. Situational awareness is defined in accordance with  by the three stages of sensory acquisition, sensory comprehension, and projection (projecting the interpretation of sensed data to decide on a future action). A robotic system with CSA assists the user not only in manipulation, but also in forming the situational awareness regarding the task at hand by using perception resources beyond the capabilities of the user.
In the following sections, we show that the emerging surgical paradigms such as NOTES require new robot designs and human-robot interaction framework that go beyond the use of robots and computer assistance to allow manipulation augmentation. We will show that, while the two approaches of haptics and sensory immersion through virtual reality help surgeons overcome the sensory acquisition step, they do not help surgeons with obtaining full situational awareness. We will put forth the concept of CSA as a natural progression beyond these two approaches thereby allowing robots to help surgeons In Interpreting the surgical scene and in projecting the perceived intraoperative sensory data to allow exact safe operation and the execution of pre-operative surgical plans.
From Manual to Robot and Computer-Assisted MIS
The early drivers for robot surgical assistance stemmed from the desire to improve patient outcomes by achieving two goals: 1) offering patients the benefits while sparing surgeons the difficulties of laparoscopic MIS, 2) improving the accuracy of surgical execution of pre-operative surgical plans to ensure optimal outcomes. These two goals have driven most of the medical robotics research in the past three decades and resulted in several robotic systems reviewed in  and more recently in .
The concept of robotics for manipulation augmentation was introduced to overcome the challenges of manual laparoscopy. This radical move decreased the learning curve of surgeons who no longer had to contend with the reverse manipulation mapping typical of manual laparoscopy. Compared to laparoscopy robots provided increased dexterity, allowed the manipulation of multiple arms, improved precision and steadiness and lowered the physiological performance requirements of surgeons.
To improve surgical plan execution accuracy, computer assisted surgery was introduced in order to provide perception augmentation through intraoperative imaging and guidance. By using image registration a pre-operative surgical plan is matched to the intraoperative surgical reality. While image registration proved feasible for rigid anatomy (e.g. Orthopedics), it has been elusive in general surgery due to organ flexibility, deformation and gravitational shift and/or swelling.
The advantages of telemanipulation robot assistance came at the cost of removing the surgical tool from the surgeon's hand, thus, resulting in limited sensory perception and situational awareness. Surgeons cannot feel the tool Interaction with the anatomy and current commercial systems still do not provide force feedback. Surgeons are also challenged with interpreting and relating the surgical scene with pre-operative imaging.
Haptic feedback, augmented reality and assistive manipulation have been proposed to alleviate the loss of sensory presence and situational awareness. Haptic feedback aims at restoring sensory presence through force feedback or by sensory substitution (e.g. substituting force with sound feedback). Augmented reality partially restores situational awareness via image overlay allowing the surgeon to superimpose an intraoperative image or visual cues to anatomical structures on the image display of the telemanipulation master console.Assistive manipulation uses control laws to help surgeons achieve critical surgical tasks. These control laws include active constraints and virtual fixtures. Active constraints enforce safety barriers preventing the surgical tool from venturing into sensitive anatomy. Virtual fixtures generalize this concept to facilitate tracing of a target geometry such as an ablation path or an anatomical surface (for an up-to-date review see ). Assistive manipulation can be applied during cooperative manipulation or telemanipulation. During cooperative manipulation, the robot and surgeon hold the tool and admittance control allows the surgeon to move the tool while benefiting from tremor filtering and enforcement of assistive manipulation laws. Examples of cooperative manipulation robots are the REMS robot and the Mako Rio® shown in Figure 2.
By and large, the frameworks of assistive manipulation, augmented reality, and haptics have historically evolved with multiport MIS in mind as an application domain. Intraoperative information seamlessly gathered by the surgeon's hand (e.g. stiffness/constraint cues) during open surgery is not used. Also, existing frameworks for assistive manipulation typically assume single and known contact between the end effector and the environment. The newly emerging surgical
New Surgical Paradigms and Challenges
Multi-port MIS requires several small incisions that generally heal well, but can also be associated with pain, scarring and potential wound infection and/or hematoma. To ameliorate surgical outcomes, SPA and NOTES have been proposed to reduce or eliminate the number of surgical access incisions. During an SPA procedure a single access port is placed in the abdomen to provide surgical access to the necessary tools. Figure 3 shows an example of the insertable robot effectors platform (IREP)  developed to operate on the abdomen through a single Ø15 mm port. During NOTES procedures natural orifices are used to access internal anatomy. Examples of NOTES access routes include transurethral, trans-nasal, trans-oral, trans-esophgeal, trans-gastric, trans-anal, and trans-vaginal surgeries. Figure 4 shows an example of the highly articulated robotic platform (HARP) designed to provide deep access into the anatomy and recently used for trans-oral surgery . Much of the understanding we gained regarding the limitations of traditional manipulation paradigms has been through experience in using these two systems.
Figure 5 illustrates the encumbered difficulty of NOTES/SPA compared to multi-port MIS. In addition to the challenges of MIS, NOTES adds the complexity of operating in constrained workspace and traversing anatomical passageways. Unlike multi-port MIS where contact with the anatomy occurs only at the dexterous wrists on rigid shaft tools, in NOTES contact occurs along the length of the robot as it is inserted in anatomical passageways, Figure 6-(a). Also, in procedures such as trans-gastric abdominal surgery there is the significant challenge of obtaining wound closure within the gastric wall after completing the procedure. And compared to MIS where generally there is a correspondence between the motion range and shape of the wristed surgical tools and the surgeon's hand (e.g. Figure 1), in NOTES the robots must have many degrees of freedom and arms and this correspondence become significantly more complex to learn. Finally, while situational awareness is limited in MIS, the limitation is exacerbated in NOTES due to the further limited perception of robot shape and its interaction with the anatomy. Figure 6-(b) illustrates the risks of operating in confined space subject to the limited perception of the endoscope: collisions between the robot and the anatomy can occur outside the perception range of the surgeon.
Design, Control and Sensing for Enabling CSA
When designing systems for NOTES/SPA, the prerequisites of safe deployment into the anatomical passageways, distal dexterity and collaborative multiarm workspace must be satisfied. Depending on the mechanical embodiment of the robot, there are four ways of achieving deep access into the anatomy. The first approach tasks the surgeon with steering the front end of the robot using camera visualization and uses the robot controller to execute a follow-the-leader task. Such design requires a very large number of actuators to control the shape of the robot and can lead to slow deployment. The second approach implements passive compliance with a steerable end tip. Once the robot reaches the site the robot structure can be actively locked. The Transport® endoscopic access system by USGI Medical Inc. is an example of such an approach. The third approach is by steering the robot tip while allowing the back end of the robot to alternate motions of locking and relaxation of the robot backbone in order to match the anatomical passageway during insertion. The HARP robot in Figure 4 uses this approach. Finally, the fourth approach requires the robot to use its sensing capabilities to actively comply with the constraint forces of the anatomy while tasking the surgeon with advancing the robot. Figure 7-(a) shows an example of this approach; which has also been reported in  to facilitate rapid trans-nasal access into the upper airway.
The second design prerequisite is distal dexterity with a dual-arm dexterous workspace. To effectively achieve dual-arm tasks, the robot arms must be able to oppose each other as human arms can. This challenging task is called triangulation of tools. This calls for designs enabling multiple dexterous arms to operate with almost parallel shafts. For instance the IREP, shown in Figure 3, has been designed with the ability to control the distance between its two snake-like arms in order to facilitate dexterous dual-arm operations such as knot tying. Other example of NOTES systems have been reviewed in .
Even if the design prerequisites of NOTES are satisfied, the success and safe use of these systems hinges on implementing advanced sensory and control capabilities to overcome the challenges in Figure 5. As initially proposed in , smart surgical tools can facilitate manipulation augmentation. In our work, we propose that intelligent surgical slaves are a critical component for enabling CSA. For example, intelligent robots with sensing capabilities can act as both surgical intervention and diagnostic tools much in the same way the surgeon's hand manipulates anatomy and helps in identifying stiff nodules invisible to the naked eye. These robots can act as perception augmentation tools by deploying sensory modalities that extend the human perception (e.g. ultrasound, optical coherence tomography, or confocal microscopy). As an example, the continuum robot shown in Figure 7-(b) is able to estimate forces and moments acting on its tip and it has been shown in  to enable palpation and building a stiffness map of a model prostate. To discern invisible contacts (as in Figure 6)  proposed kinematics-based models for detecting and localizing the contact. Figure 7-(c) shows a continuum robot demonstrating contact detection.
Finally, one of the key benefits of intelligent surgical slaves is their ability to offload cognitively burdensome tasks. To achieve this, the low level controllers of these robots must support force and motion regulation. For example, hybrid force/motion control can be used to facilitate controlled ablation along a path while maintaining a fixed contact force between the ablation probe and the anatomy. Our team is also working on other advanced capabilities of these robots including exploration of an unknown flexible environment with the aim of localizing and mapping the environment shape and stiffness and using this information for registration to a pre-operative model.
Towards Complimentary Situational Awareness
Figure 8 shows our envisioned tele-robotic system with advanced features of computer-aided surgery, taking advantage of the fact that telesurgery systems essentially place a computer between the surgeon and the patient. In addition to basic telesurgery capabilities such as high quality stereo video, distal dexterity, motion scaling, and tremor filtering, these Complementary Situational Awareness (CSA) systems have the capability to combine sensing, imaging, and model information to provide the surgeon with significantly enhanced information and decision support, using augmented reality visualization, haptic feedback, and other means. These systems can also assist the surgeon in manipulating tissue through the use of virtual fixtures and other assistive methods. Further, haptic, imaging, and other intraoperative sensing during the procedure can update and refine the computers of the patient and surgical situation.
In the future, we expect that CSA systems will increasingly be embedded within a larger framework of Computer-Integrated Interventional Medicine (Figure 9), in which patient-specific information such as images, lab results, and genomics are combined with general knowledge to model and diagnose the patients’ condition and to develop an optimized treatment plan. All of this information will be available to a CSA system to help the physician carry out the treatment plan and assess the results. This closed-loop process (blue loops) will occur over multiple time scales, from an entire patient treatment cycle down to every second in the operating room or interventional suite. Further the CSA system will function as a flight data recorder enabling the creation of a much more complete record of what happened in the operating room. All this information can then be related to observed outcomes and statistical methods can be used to improve treatment processes for future patients (red loop).
Research reported in this paper has been supported in part by the National Robotics Initiative through NSF grants IIS-1327566, IIS-1426655 and IIS-1327657 and by university funds.