In this paper, an unsteady tip leakage flow phenomenon is identified and investigated in a centrifugal compressor with a vaneless diffuser at near-stall conditions. This phenomenon is associated with the inception of a rotating instability in the compressor. The study is based on numerical simulations which are supported by experimental measurements. The study confirms that the unstable flow is primarily driven by the Kelvin-Helmholtz instability inherent in the shear layer between the main-stream flow and the tip leakage flow. The shear layer instability induces large scale vortex shedding which propagates circumferentially, leading to pressure perturbations with short wavelength and high amplitude which rotates at about half of the blade speed. The shed vortex is also found to interact with the main blade leading edge, resulting in the reduction of the blade loading identified in the experiment. The paper also reveals that the downstream volute imposes a long wavelength circumferential non-uniform back pressure at the impeller exit, which is found to either excite or supress the vortex shedding within the blade passage depending on the relative circumferential position of the impeller and the volutes.