Flexible pipes are distinctive multi-layer structures that are designed to resist different loads when they are utilized in severe deep-water environments. However, they lack a special structural layer to withstand torsion. Tensile armors mainly resist torque although they are designed to bear only tension with the consideration of torque balance. Especially, when a flexible pipe is loaded out from the cargo vessel into the installation vessel, twist angle could be accumulated at high level so that some failure modes will occur due to the large torsion. However, the failure mechanism is very complicated owing to the interaction effect between the different layers. First, the interaction mechanism between the layers of flexible pipes is analyzed under large torsion, and a few potential failure modes are identified, such as the tensile armors strength failure and inner structural layers collapse failure. In addition, to offer a quantitative prediction of the maximum allowable twist angle for flexible pipes, a mechanical model is set up to analyze their torsion behavior. The theoretical descriptions of the involved failure behaviors are investigated, and the theoretical methodology of the failure criteria for predicting the torsion resistance capacity is proposed. Finally, a numerical model is established through experimental verification. The numerical results illustrate that the theoretical prediction methodology is conservative, which can be used to predict the torsion resistance capacity of flexible pipes and to guide their operation and installation in engineering.