A trochoid cam gear (TCG) is a kind of precise transmission device with high performance, such as non-backlash, high-precision, low noise, etc. However, the nonlinear dynamics has not been studied. In the present paper, the nonlinear dynamic differential equation of the TCG is constructed. First, the trochoid equation and the tooth profile equation are deduced to satisfy the conditions of the continuous transmission. Second, a pure torsional nonlinear dynamic model is established to further construct the nonlinear model of the TCG according to a Lagrange equation. Finally, the dynamic characteristics of the TCG are investigated. For the short amplitude coefficient K, break of quasiperiodic torus is the main reason for the chaotic vibration. By increasing K, the performance of TCG can be improved to maintain stable motion.