This paper deals with a theoretical instability analysis of a rotating flexible disk subjected to swirling fluid flow, resulting in the generation of a strong coupling between the flexible disk vibration and swirling fluid flow around it, and a possible mechanism governing unstable vibration. In the instability analysis, the basic equations of swirling fluid flow around the rotating disk are based on Navier-Stokes equations integrated over the gap width between the rotating disk surface and the shroud wall. The structural vibration equation of the rotating flexible disk is based on the Kirchhoff-Love plate model. The equations of coupled fluid-structure motion take into account the moving boundary conditions with respect to both the rotating disk and the fluid flow. These equations are linearized for small deflection of the disk near the equilibrium state, and the solution of these equations is obtained using the multimodal expansion approximation and applying the Galerkin method. The theoretical results show a good agreement to the experiment with respect to the unstable vibration mode shapes and dependence of the instability frequency on the rotating speed, but only a qualitative agreement for the critical speed at which the disk’s unstable vibration occurs.