Applying an appropriate roughness model is one of the most effective ways to improve the accuracy of the numerical simulation for centrifugal compressors, and it is also the key to the study of surface deterioration such as compressor fouling and corrosion. The basis for using the roughness model is to determine the conversion relation between physical roughness and equivalent roughness. In this paper, numerical and experimental studies on the roughness conversion relation for the centrifugal compressors were performed by means of a unique self-developed “pseudo front stage (PFS)” device. An equivalent roughness conversion coefficient of 8.5 was determined on the basis of fitting performance of the U-bend and return channel model. The validity of the conversion coefficient in improving the performance prediction accuracy of the centrifugal compressors was verified by three centrifugal compressor models. The concept of weighted average roughness was proposed to quantify the roughness sensitivity and roughness level of the centrifugal compressor. Quantitative research on the roughness effect of centrifugal compressors indicated that the peak efficiency degradation due to the roughness effect in the compressors was linearly associated with the logarithm of the relative roughness when the impeller of the centrifugal compressor was in the hydraulically rough regime. Doubling or halving the roughness value of all components of a centrifugal compressor in the hydraulically rough flow regime will result in a change in the peak efficiency of the compressor by about 1.72%.