Secondary flow transfers the boundary layer from the endwall to suction side corner region, which deteriorates the performance of compressors. Vortex generators (VGs) can attenuate the secondary flow when implemented at the endwall of the blade passage. In order to make an efficient optimization design of the VGs layout, mathematical models are introduced to replace the actual VGs. However, the model is only effective at the small incidence. In this paper, first, a mathematical model is presented to replace the actual VGs facing a lager incidence, and validated by computational fluid dynamics results with actual VGs. Second, an efficient design method for optimizing VGs layout is presented based on VG model and genetic algorithm (GA)-support vector machine (SVM) method. The method is illustrated in a highly loaded compressor stator, and the VGs layout is optimized and applied in the blade passage. The results show that the VGs can prevent the secondary flow from the suction corner, and reduce the total pressure loss coefficient at the trailing edge of the stator by 15%. Finally, the flow mechanism of the VGs is investigated using numerical method. The results show that the VGs can induce a primary vortex at the tip and prevent the circumferential migration of boundary layers. Moreover, the secondary flow can produce a large incidence at the hub of the VGs, thus increasing the aerodynamic losses of the VGs secondary vortex.