In turbomachinery, the rotor dynamic (RD) fluid force generated in a fluid element is one of the causes of shaft vibration. RD fluid force is caused by the interaction between shaft vibration and fluid force, and its precise prediction for various rotor’s orbit is difficult. This study performs a two-way coupled analysis of the fluid flow in the annular plain seal and shaft vibration using the shooting method. The frequency response is obtained and compared with that obtained from a direct numerical simulation of the coupled system, and the validity of the analysis is confirmed. The onset speed of instability is effectively and accurately obtained using the two-way coupled analysis with the shooting method, and the effects of the parameters on it are investigated.