According to the actual operating conditions of the reactor coolant pump (RCP), based on the fluid lubrication theory, considering the complexity of the micro-narrow gap flow and the effect of surface roughness of the sealing end face, a mechanical seal model with different base film thicknesses at inner radius for the fixed waviness end face was established. The k-ε turbulent model was applied to solve the Navier-Stokes equation. Under different pressures, the fluid film pressure, open force and leakage rate of the mechanical seal end face with different base film thicknesses were obtained. According to the waviness end face of the specified wave amplitude at outer radius, the mechanical seal was manufactured by the grinding method of extrusion deformation and experiment. Comparing the experimental results with the numerical calculation results, it shows that the experimental results are in good agreement with the numerical results. It illustrates that the designed mechanical seal meets the requirements in engineering applications. Under different pressures, the base film thickness of the actual operation is accurately predicted by comparing the experimental leakage rate with the calculated ones. The calculating results show that the leakage rate increases with the increasing of pressure and the base film thickness under the same pressure. The open force decreased with the increasing of the base film thickness under the same pressure. Compared with the ideal surface, the presence of surface roughness will significantly increase the leakage.