Automatically detecting surface defects from images is an essential capability in manufacturing applications. Traditional image processing techniques were useful in solving a specific class of problems. However, these techniques were unable to handle noise, variations in lighting conditions, and background with complex textures. Increasingly deep learning is being explored to automate defect detection. This survey paper presents three different ways of classifying various efforts. These are based on defect detection context, learning techniques, and defect localization and classification method. The existing literature is classified using this methodology. The paper also identifies future research directions based on the trends in the deep learning area.