Vibration-based damage detection has become one of the principal practices to prevent structural collapses in civil, mechanical, and other engineering disciplines. Meanwhile, with the advancement of computing technology, various machine learning (ML) approaches have been applied toward structural damage detection through the application of post-processing algorithms. To accurately predict damages with ML, large amounts of structural response data are collected from a series of sensors attached to the structure. Therefore, the damage diagnosis requires high computational efforts. To address such an issue, this paper presents a revolutionary approach utilizing an image-based pre-trained convolutional neural network (CNN) to detect bridge damage locations and severities. Our research adopted scalograms from wavelet transform to convert structure acceleration data into image data. Compared with the traditional frequency analysis derived from the Fourier transform, the new method maintains both spatial and temporal information from the original structural behaviors. To generate CNN learning features, six channels of acceleration data are gathered from six strategically selected points of a finite element (FE) bridge model. Two pre-trained CNN, AlexNet and Resnet, are selected to conduct transfer machine learning for higher training efficiency. The performances of the proposed method are assessed with various damage scenarios. The prediction accuracies of AlexNet and Resnet are 98% and 100%, respectively.