Gear–wheelset system is a crucial substructure in railway vehicles which affects the operation safety and system reliability, especially in the process of traction and braking conditions. Unlike the general gear transmission system, the gear–wheelset system of railway vehicles operates under an environment with several nonsmooth factors; therefore, it is necessary to analyze the nonsmooth dynamics of the gear–wheelset system for understanding dynamic characteristics better. Herein, a planar dynamic model of the gear–wheelset system of railway vehicles considering motor-driving torque, braking torque, wheel–rail nonlinear interaction forces, nonlinear meshing damping, and piecewise continuous time-varying meshing stiffness is proposed. Then, the proposed model is validated by a simpack model using wheelset's longitudinal velocity. Subsequently, two numerical simulations were performed to reveal the nonsmooth dynamic characteristics under traction and braking conditions. The simulation results indicate that the dynamic stationary point exists in nonsmooth dynamics under traction and braking conditions, which is a critical boundary for transiting any state to a dynamic equilibrium. Besides, the results exhibit the inseparable relationship between time-frequency dynamic characteristics, slip velocity, and wheel–rail nonlinear interaction forces. The effects of harmonic torque under traction conditions and compound braking behavior under braking conditions significantly affect these dynamic characteristics. Additionally, sufficient driving torque can increase the proportion of forward contact and improve the smoothness of the rotation, and the intermittent gear contact phenomenon occurs alternately and frequently in the traction condition. Conversely, only reverse contact occurs in the braking condition.