In order to prevent mesh distortion problem arising in topology optimization of compliant mechanism with massive displacement, a meshless Galerkin method was proposed and studied in this paper. The element-free Galerkin method (EFG) is more accurate than the finite element method, and it does not need grids. However, it is difficult to impose complex boundaries. This paper presents a topology optimization method based on interpolation meshless method, which retains the advantages of the finite element method (FEM) that is easy to impose boundary conditions and high accuracy of the meshless method. At the same time, a method of gradually reducing step is proposed to solve the problem of non-linear convergence caused by low-density points in topology optimization. Numerical example shows that these techniques are valid in topology optimization of compliant mechanism considering the geometric nonlinearity, and simultaneously these techniques can also improve the convergence of nonlinearity.