In this paper, a novel fractional-order variable-gain super-twisting control (FVSTC) scheme is proposed and applied to improve the tracking performance of wafer stages in the photolithography systems. The FVSTC overcomes the drawbacks of the super-twisting control (STC) such as slow response speed and incomplete compensation to disturbances. First, to improve the dynamics of the states on the sliding surface, a fractional-order sliding surface is designed. Moreover, to improve the dynamics of the sliding mode variable, an equivalent-control-based method is utilized, and a switching controller based on a variable-gain super-twisting algorithm is deployed. Via such designed schemes, the proposed controller is robust against external disturbances and model uncertainties. Stability proof of the closed-loop system is provided. Numerical simulations to track a sinusoidal signal and experiments on a wafer stage testbed are conducted. The results show that the proposed FVSTC scheme can achieve much better tracking performance than conventional methods.