Current surgical instruments with fewer degrees-of-freedom (DOF) for minimally invasive surgery (MIS) have limited capability to perform complicated and precise procedures, such as suturing and knot-tying. To address such a problem, a modular dexterous hand-held surgical robot with an ergonomic handle and 4DOF interchangeable instruments was developed. The kinematic arrangement of the instrument and that of the handle were designed to be the same. A compact roll-yaw-roll transmission was proposed applying cable-driven mechanism. Performance experiments were carried out to evaluate the effectiveness of the overall system. The measured grip forces of the robot ranged from 8.63 N to 19.18 N. The suturing performance score of the robot was significantly higher than that of the conventional instrument (28.8 ± 5.02 versus 17.2 ± 7.43, p = 0.041). The trajectory tracking test and animal experiment verified the accuracy and feasibility of the robot. The proposed robot could improve the surgical performance of MIS, providing various end-effectors and having an intuitive interface in the meantime.