A labyrinth seal is commonly used to decrease the flow leakage loss between rotating and static components in aero engines. It is susceptible to aeroelastic instability because of its low stiffness. The aim of this study was to establish methods to predict and suppress it effectively. To achieve this, both numerical and experimental investigations are conducted using ansyscfx and ansys mechanical. These solvers are coupled to simulate the flutter precisely. Also, to assess the accuracy of the simulation qualitatively and quantitatively, a test rig is built. In the first part of this study, the accuracy of the numerical method is confirmed for several test cases with different seal clearance variations. Flutter inception is evaluated in detail for various pressure ratios and rotation speeds. The numerical results show good agreement with the experimental results. It is also confirmed that the aeroelastic instability is very sensitive to the seal clearance variations. These results show the same tendency as those in previous works. In the second part of this study, this paper tries to develop a flutter suppression method with higher leakage performance. This is achieved by changing the seal geometry. To detect the important geometric parameters, the contribution of each geometric component to aeroelastic instability is carefully analyzed. On the basis of this, the seal geometry is modified and its performance is evaluated. The optimized labyrinth seal shows good performance in terms of flow leakage and aeroelastic stability. Through this study, a new flutter suppression method is established.