Virtual Reality Interface Design for Multi-Modal Teleoperation

A multi modal teleoperation interface is introduced featuring an integrated virtual reality (VR) based simulation augmented by sensors and image processing capabilities on-board the remotely operated vehicle. The proposed virtual reality interface fuses an existing VR model with live video feed and prediction states, thereby creating a multi modal control interface. Virtual reality addresses the typical limitations of video-based teleoperation caused by signal lag and limited field of view. The 3D environment in VR along with visual cues generated from real time sensor data allows the operator to navigate in a continuous fashion. The vehicle incorporates an on-board computer and a stereo vision system to facilitate obstacle detection. A vehicle adaptation system with a priori risk maps and real state tracking system enables temporary autonomous operation of the vehicle for local navigation around obstacles and automatic re-establishment of the vehicle’s teleoperated state. Finally, the system provides real time update of the virtual environment based on anomalies encountered by the vehicle. The VR interface architecture is discussed and implementation results are presented. The VR based multi modal teleoperation interface is expected to be more adaptable and intuitive when compared to other interfaces.