Evaluation and Optimization of Dual-Arm Robot Path Planning for Human–Robot Collaborative Tasks in Smart Manufacturing Contexts

Robotics technology plays a significant role in the manufacturing evolution as it can facilitate the manufacturing process to be more effective and intelligent than traditional labor-intensive manners [1–4]. Industrial robots have been extensively utilized in different kinds of manufacturing tasks such as welding, sorting, polishing, and 3D printing [5,6]. With the complexity increasing and configuration updating of products, however, the limitation of traditional robotics technology occurs gradually. For example, in the automotive final assembly phase, about 60% of tasks are manually accomplished by human workers because fenced industrial robots are inflexible and too large for automotive final assembly processes. Assembly is the most challenging subtask in vehicle production processes due to its nature of heavy manual labor, which affects the manufacturing quality and ergonomics distinctively. Such similar issues also exist in other application areas such as 3C industry, agriculture, and food industry [7,8]. In recent years, collaborative robotics technology provides effective solutions to these challenges. Collaborative robots are able to work with humans as partners to conduct shared tasks. Through human–robot collaboration, the tasks can be co-finished appropriately by taking advantages of the intelligence of humans and the accuracy of robots.

In human–robot collaborative tasks, handing over the parts or tools to each other is an indispensable and significant physical interaction [9]. Hand-over is known as a cooperative action/behavior to meet the needs of each other by delivering a desired object [10]. For example, in the automotive maintenance process, two human workers may hand over auto parts or tools to each other for the vehicle inspection or repairing. With the increasing applications in different kinds of manufacturing contexts, the collaborative robot is also expected to grasp and hand over target parts to its human partner seamlessly in human–robot collaboration just like human–human hand-over.

During the robot-to-human hand-over process, the performance of robot path planning has a direct impact on the hand-over execution, or even the collaborative task quality. Imaging that a collaborative robot grasps a target part and delivers to its human partner on a conveyor-type assembly line, if the robot plans its motion with too many waypoints or a long path length from the start-point to the destination, the human may feel uncomfortable or the human–robot team may miss the part assembly. In addition, for different sorts of tasks in diverse working settings, the robot may need different kinds of corresponding motion planning algorithms to achieve robust path planning and stable operation performance to keep the collaborative tasks well conducted. Motivated by these issues in human–robot collaboration, in this work, we develop an optimization strategy for the robot path planning by evaluating and discussing 12 kinds of path planning algorithms in multiple types of representative human–robot collaborative tasks. This study also opens a window for exploring more complicated and general robot motion planning applications to human–robot collaborative tasks in smart manufacturing contexts.

The major contributions of this work are (1) we develop an optimal path planning selection strategy for the dual-arm robot to grasp and deliver desired parts with maximum efficiency for its human partner in human–robot collaboration and (2) we conduct a comprehensive evaluation study on multiple robot path planning algorithms with different metrics and reveal their pros and cons in typical human–robot collaborative manufacturing contexts.

With the increasing employment of robotics technology in a variety of fields over the past decades, robot motion planning issues have been attracting extensive attention from both academia and industries. Such issues can be generally formulized as searching the task workspace of one or more complex geometric bodies for a collision-free trajectory, which connects the given start-point and target-point in obstacle-constrained or obstacle-free working environments [11,12]. The research of robot motion planning emerged in the mid-1960s and stepped into a prosperity period along with the contributions of Lozano-Perez on spatial planning [13]. Currently, the family of robot motion planner has dozens of members [14]. In this work, to investigate and optimize the robot motion planning strategy in different kinds of human–robot collaborative tasks, we study and discuss 12 kinds of typical robot motion planners. As presented in Table 1, these motion planners are categorized by different categories.

When searching paths in configuration spaces, some planners generally build one exploring tree to expand and generate a valid route from the start-point to the destination. They are also known as unidirectional motion planners, such as RRT, FMT, PRM, EST, and SPARS. In order to improve searching performance, several planners, including RRT-connect, SBL, and BKPIECE, explore paths via the bidirectional search manner by constructing two trees rooted at the start and goal configurations. These trees expand paths simultaneously, and an efficient route will be generated when they are connected.

The query types of these path planners contain single query and multiple query. In single query based planners, the heuristic algorithm generally generates the valid collision-free path by adding samples to the search space incrementally [22]. For multiple query based planners (e.g., PRM and SPARS), they usually construct roadmaps for the entire workspace. The valid configuration states, which are connected to neighboring states by edges, are sampled in the configuration space during the roadmap construction process.

In path searching processes, these planners can expand sample spaces by the random manner or the structured manner. To decrease the computation time of path selection, several planners (e.g., SBL and LazyPRM) employ the lazy collision checking strategy, by which the valid collision-free path is able to be decided from all sampled points [23–25]. As listed in Table 1, each of these path planners has its own advantages and disadvantages in robot motion planning because of their different features. Therefore, for different situations in human–robot collaboration, the best robot path planner should be selected to reduce robot motion time and improve robot motion stability.

In robot–human hand-over processes, human hand-over intentions (e.g., “I need something”) can be characterized by multiple methods such as vision systems [26] and natural language processing [27]. In this study, we employ a practical and easy-to-use approach by using a wearable sensory system to parameterize human gestures to have the robot understand human hand-over intentions. The sensory system we selected is Myo, which is configured by a 9-degree-of-freedom inertial measurement unit (IMU) and 8 electromyography (EMG) sensors and can be worn by the human at his/her forearm [28,29]. The human gesture information, including forearm muscle activities and rotation angles, are able to be tracked and detected by the EMG sensor and the IMU, respectively.

When the human needs something from the robot, human intentions, which maneuver the robot-to-human hand-over procedures, can be considered as observation states of the HMM. Each intention is described with M (M = 11 in this study) observation symbols, which include three forearm rotation angles and eight EMG signals collected by the Myo system. In this study, N is initialized as 30 based on the validation set approach [31]. The initial state distribution π is able to start with any hand-over intentions. The probability matrices A and B are decided by the training data of different human intentions. In the HMM training process, the goal is to adjust and evaluate the parameters in Eq. (1) to obtain the maximum probability of the human intention characterized by the observation state.

By taking advantage of the trained HMMs, we input the given human gesture information to each HMM, then we can obtain the output probabilities of this intention. Therefore, the human hand-over intention is able to be classified and matched by the corresponding HMM, which has the maximum output probability. After that, using the recognized human intention, the robot can be controlled to pick up objects and deliver them to its human partner. Since it is not the main contribution of this study, more details of human hand-over intention recognition process via HMMs can be found in our previous work [28].

In order to comprehensively evaluate robot path planners in Table 1 in different kinds of human–robot collaboration contexts, we employ three metrics, including path length, computation time, and execution time, for them when they plan the robot from the start-point to the destination.

In the robot arm and path planner selection process, the static constant W_A works as the weight of the attention that humans pay to the robot’s separate running result or short-term running performance, while the dynamic constant W_B acts as the weight of the attention that humans pay to the robot’s long-term operation performance. The weights W_A and W_B provide a lot of flexibility for tuning the motion planning preferences in different applications. If we alter the weights W_A and W_B in human–robot collaborative tasks, some certain features will become comparatively advantageous for different performance objectives to reflect the different preferences of the applications. For example, if the human needs to work with the robot for a same all-day collaboration task, he/she may adjust W_B to be greater than W_A. On the contrary, if the human wants to work with the robot for some separate tasks in a short term, he/she can set W_A with a bigger value than W_B. The min-max normalization method [32] is employed in this study.

Note that if a given path planner cannot generate a valid path solution for the robot in the allocated task, we will regard this path searching as failure, which will affect the value of cost function. When calculating L′, $TC′$⁠, $TE′$⁠, σ_L, $σTC$⁠, and $σTE$ in this study, we set L, T_C, and T_E as zero in case the given planner fails to search a valid path for the robot.

In this work, we implement and verify the proposed approach on a multi-modal collaborative assembly platform, which is structured by our lab for the study of human–robot collaboration in smart manufacturing contexts. The platform consists of a dual-arm robot YuMi and a set of multi-modal based human–robot interactive interfaces, such as wearable sensory systems, 3D vision systems, and physiological sensing systems. All the equipment of the platform are controlled by the robot operating system (ROS) [33]. To get an objective evaluation for path planners, we employ the same computer as a control center to run high level robot control algorithms and motion planning algorithms in the same working condition.

In this experimental study, the robot works with its human partner to assemble a high-fidelity model vehicle in the manufacturing context. The human wears the Myo sensory system to trigger the robot for hand-over tasks based on our human intention recognition model (see Sec. 3.1). The robot’s motion paths are generated by the planners listed in Table 1. The Open Motion Planning Library [34] is employed to operate path planners in the ROS environment.

As presented in Fig. 1, during the robot-to-human hand-over process, the robot moves its left arm or right arm from point A₁ or A₂ to point B₁, B_2, or B₃, then picks up the vehicle seat and delivers it to the human at point c. When the robot grasps the vehicle seat at point B₁, B_2, or B₃, as shown in Fig. 1(b), its grasping orientation is assigned to be downward. The initial positions of each arm are same for each picking up-and-delivering experiment. The path length, computation time, and execution time of each experiment include the task of moving the arm from the initial position, picking up the target object, and delivering to the human. In order to obtain reliable data of the length path, computation time, and execution time, each path planner run 20 times for the allocated motion task on our experimental platform. In robot-to-human hand-over tasks, three vehicle seats (front right seat (seat 1), rear seat (seat 2), and front left seat (seat 3)) on the workbench need to be picked up and delivered by the robot. The front right seat is located near the robot right arm, the rear seat is located between the right arm and left arm, and the front left seat is located near the robot left arm. In the hand-over process, we develop a real-time robot control system using MoveIt! package [35] based on the ROS environment to acquire the number of way points, the computation time, and the execution time of each path planner. After that, the metrics we defined in Sec. 3.2 can be calculated via these collected data.

To comprehensively test and evaluate the path planners in different working environments, we employ three typical kinds of hand-over scenarios in obstacle-constrained environments. As shown in Fig. 2, different types of obstacles are arranged for each vehicle seat. In these tasks, via the generated paths from path planners, the robot utilizes its right arm or left arm to pick up the front right seat, front left seat, and rear seat, then delivers them to the human, respectively. Through the collision-aware plug-in in our simulation system, the path planner can be aware of the geometrical information of obstacles. Then, the path planner attempts to search a valid path for the robot based on the obstacle-constrained configuration space. By taking advantage of the valid paths, the robot can employ its arms to pick up vehicle seats and work with its human partner in each scenario.

The path length calculation of each path planner for the vehicle seat picking up and delivery are shown in Figs. 3 and 4, respectively. When the robot operates the seats via its right arm, as presented in Fig. 3, different path planners generate diverse lengths of valid paths. The path lengths of seat 1 operation are generally shorter than those of seat 3 operation because seat 1 is closest to the right arm. In this set of evaluation, the PRM planner shows a shortest path for seat 1 operation, while the RRT-connect planner generates a shortest path for seat 3 operation. When operating seat 2, the RRT* planner presents the best performance in path length evaluation. Similarly, in Fig. 4, when the vehicle seats are operated by the robot left arm, all the planners generate shorter valid paths for seat 3 operation than those for seat 1 operation since the seat 3 is located near the robot left arm. Among these planners, the RRT-connect generate a shortest path for the farthest seat (seat 1), while it presents a longest path for the closest seat (seat 3).

Based on the collected path lengths of each path planner, as listed in Table 2, we calculate their standard deviations. When the vehicle seats are operated using the robot right arm, it can be seen that the T-RRT planner, the FMT planner, and the LazyPRM planner present lowest standard deviations for seat 1 operation, seat 2 operation, and seat 3 operation, separately. However, the performance of these planners is different when the robot left arm is employed. In this situation, the SPARS planner shows a lowest standard deviation for seat 1 operation, while the RRT-connect planner and the FMT planner are more stable in seat 2 operation and seat 3 operation, respectively.

The computation time of each path planner in vehicle seat operation are calculated and presented in Figs. 5 and 6. When the robot right arm is employed, as shown in Fig. 5, the RRT-connect planner is able to search a valid path for seat 1 operation with the least computation time. For seat 3 operation, the EST planner shows a best performance in computation time cost. When operating seat 2, most planners’ computation time is under 0.4 s. However, the BKPIECE planner, which costs about 1.1 s for searching a valid path, shows a worst performance in computation time evaluation.

The standard deviations of computation time of each path planner are shown in Table 3. When the robot right arm is used for the vehicle seat operation, the EST planner presents lowest standard deviation for seat 1 operation and seat 3 operation. For seat 2 operation, the RRT* planner shows a distinct stable when searching a valid path. When the vehicle seats are operated through the robot right arm, it can be observed that the PRM* planner, the PRM planner, and the RRT* planner have lowest standard deviations for seat 1 operation, seat 2 operation, and seat 3 operation, respectively.

Figures 7 and 8 depict the execution time evaluation of each path planner in vehicle seat operation. As presented in Fig. 7, when the robot right arm is utilized for seat 1 (the closest seat) operation, the execution time on most generated paths are similar to each other. However, for the farthest seat (seat 3), the diversity of the execution time is shown obviously. Similarly, in Fig. 8, it can be seen that the execution time of seat 3 operation are close to each other when the robot right arm is employed. When operating seat 2, the RRT planner shows a fastest path for the robot. However, the RRT-connect planner generated a most time-saved path for the robot when seat 1 need to be picked up and delivered.

The standard deviations of execution time of each path planner are illustrated in Table 4. When the robot right arm is employed to pick up and deliver vehicle seats, the FMT planner generates the most stable path than that of other planners in seat 1 operation and seat 2 operation. The LazyPRM planner presents a lowest standard deviation of execution time in seat 3 operation. When operating seat 1 and seat 2 using the robot left arm, the RRT planner shows the most robust performance. However, the PRM* is most time-saved option in seat 3 operation.

In this study, since we consider the proposed approach to be used in a long time human–robot collaborative work, the values of W_A and W_B in Eq. (7) are selected as W_A = 0.4 and W_B = 0.6, respectively. Based on the evaluation results in Sec. 5.1, we figure out the cost comparisons of the two robot arms in vehicle seat operation. As shown in Fig. 9, when seat 1 is operated, it can be seen that it is better to choose robot right arm than left arm. In addition, when the right arm is employed, it is the best to select the RRT planner for the robot path planning. In addition, Fig. 10 suggests that, when seat 2 is operated, the robot right arm with FMT planner is the best option. Moreover, from Fig. 11, it can be observed that the robot left arm with the T-RRT planner is the best way to operate seat 3.

The results of robot path planning optimization based on the cost function are tested and verified in realistic human–robot collaborative manufacturing contexts. Based on our developed approach in Ref. [21] and the use of standard human gesture expressions in the hand-over experiments of this study, all the human “Need” intentions could be always recognized.

Figure 12 presents one of the optimized strategies that the robot employs its right arm with the FMT planner to pick up seat 2 and deliver it to its human partner. The path length of the seat 2 operation is about 1.4 m, the computation time is about 0.19 s, the execution time is about 17 s, and the cost of is about 1.25. As shown in Fig. 12(a), the robot is ready for the collaborative task in the workspace. Based on the optimized strategy, the robot moves to the target seat in Figs. 12(b) and 12(c). After picking up the seat, as presented in Fig. 12(d), the robot hands it over to the human. Then it returns to its original position, as depicted in Figs. 12(e) and 12(f). The successful robot-to-human hand-over verification indicates that our proposed evaluation and optimization approaches are able to provide positive strategies for the robot path planner selection in human–robot collaborative tasks in smart manufacturing contexts.

In this work, we have proposed an evaluation study on multiple robot path planners with different metrics, such as path length, computation time, and execution time, for human–robot collaboration in smart manufacturing contexts. Based on the proposed metrics, we have defined a cost function for the robot to choose optimized path planning solutions to finish corresponding human–robot collaborative tasks. Additionally, by analyzing the empirical study data, we have successfully enabled the robot to select best ways to finish its tasks in different kinds of scenarios. Moreover, we have implemented the proposed evaluation and optimization approaches to multiple realistic human–robot collaborative manufacturing contexts. Experimental results and evaluations have suggested that our approaches can provide positive solutions for the robot path planner selection.

In the experiments of this study, we have got all three optimal planners (RRT, FMT, and T-RRT) are unidirectional, single query, and non-lazy algorithms. However, for other applications, the optimal planners may not be unidirectional, single query, or non-lazy algorithms. In addition, if we change the values of W_A and W_B of Eq. (7) in human–robot collaborative tasks, some certain features of different path planners would also be altered. Therefore, more complicated and general robot path planning applications will be investigated in future work to reveal deeper property comparisons of unidirectional and bidirectional, single query and multiple query, and non-lazy and lazy for these path planners in smart manufacturing contexts.

This work was supported by the National Science Foundation under Grant No.: IIS-1845779.