In this paper, an unsteady tip leakage flow phenomenon is identified and investigated in a centrifugal compressor with a vaneless diffuser at near-stall conditions. This phenomenon is associated with the inception of a rotating instability in the compressor. The study is based on numerical simulations that are supported by experimental measurements. The study confirms that the unstable flow is governed by a Kelvin–Helmholtz type instability of the shear layer formed between the main-stream flow and the tip leakage flow. The shear layer instability induces large-scale vortex roll-up and forms vortex tubes, which propagate circumferentially, resulting in measured pressure fluctuations with short wavelength and high amplitude which rotate at about half of the blade speed. The 3D vortex tube is also found to interact with the main blade leading edge, causing the reduction of the blade loading identified in the experiment. The paper also reveals that the downstream volute imposes a once-per-rev circumferential nonuniform back pressure at the impeller exit, inducing circumferential loading variation at the impeller inducer, and causing circumferential variation in the unsteady tip leakage flow.