Self-piercing riveting (SPR) is a key joining technique for lightweight materials, and it has been widely used in the automobile manufacturing. However, complex process parameters and huge configurations of substrate materials can cause potential button cracks, which bring significant challenges for quality inspection. This paper presents a failure crack detection and evaluation method based on image processing. Firstly, the SPR rivet cracks image is preprocessed through gray-scale transformation and interested area selection; next, the binary crack image is utilized to identify the crack parameters; finally, a crack evaluation method is developed to evaluate the rivet crack quality with quantized scores. In addition, subject matter experts (SME)’ knowledge is incorporated to verify the crack detection and quality evaluation, and case study is conducted to demonstrate feasibility of the proposed method.