As the ultimate protection of a pressure system, pressure safety valves (PSV) can respond in an unstable manner in the form of flutter and chatter, which will affect service life, reliability, and performance. In order to study the dynamic instability caused by multisource forces including the flow force, the spring compression force, and the pressure wave forces, a high-fidelity computational fluid dynamics (CFD) model of the system is proposed. A complete CFD model, incorporating the PSV, connected pipes, and the pressure vessel, is developed, in which advanced techniques in Fluent using User Defined Function (UDF) and Dynamic Layering method are combined to allow the PSV to be coupled to the system dynamics. Based on this model, the valve's opening and reclosing process is monitored to examine the influence of design parameters on the dynamic instability of the PSV. Specifically, the propagation of pressure waves along the connecting pipes is successfully captured, helping to assess the instability mechanism and provide the ability to optimize the design and setup of pressure relief systems.