The separation efficiency of a vane-type separator is greatly affected by swirl instability. The separator consists of a swirling vane, a recovery vane and a main pipe. Driven by centrifugal force, the bubbly flow tends to develop into stratified flow with a continuous gas core floating in the central axis of the separator and facilitating the separation. Yet, the straight gas core can turn into a double helix under some circumstances, for example, if the pressure difference across the orifices of recovery vane falls below the critical value, and swirl instability occurs. In order to reveal the underlying mechanism, a device with adjustable operating pressure was introduced to reproduce the dynamic process of gas core transform between stable and unstable. With the increase of pressure difference, the gas core morphology near the recovery vane will turn from double-helix to straight-line within several seconds. The whole process was investigated further by using the tomographic particle image velocimetry (PIV). Results show that the development of vorticity structures in the swirl flow gives rise to the evolution of gas core morphology and keeps it stable. Furthermore, the direction of axial velocity, which becomes negative by low pressure differences, is found to be crucial in controlling the formation of inner forced vortex and hence leading to the occurrence of swirl instability. In addition, the magnitude of positive axial velocity is identified to be of great significance in vorticity enhancement.