Railway provides more than 40% of the freight ton-miles moved in the U.S. each year, surpassing all other modes of transportation. In addition to moving more tonnage farther than other modes, trains have better fuel efficiency than trucks and airplanes due to the low friction between the wheels and the rails. With traffic accumulation, rails will degrade which may lead to different types of defects, including but not limited to spalling, separation, crack, and corrugation. Rail head fissures or surface crack is often associated with rolling fatigue and must be addressed through grinding or other maintenance activities to restore the smooth-running surface. This ensures the riding conforms to operational safety requirements. The growth pattern of rail surface cracks has not been thoroughly understood or well-quantified yet due to the difficulties of rail crack inspection and insufficient data. This paper presents a study that uses image analysis techniques to detect and quantify cracks in images of rail segments that were taken in the field. Various crack detection techniques were tested and compared with visual inspection, including thresholding, edge detection, and bottom-hat filtering. The crack length, direction, and curvature were also quantified with each approach. Cracks were found to grow not perpendicular to the rail head, but with a certain angle from the vertical direction and relatively evenly distributed along the rail. The bottom-hat filtering technique was found to be the best in terms of accuracy among the methods tested in this study. The results from the study fill the gap of the literature by quantitatively characterizing the rail crack growth pattern and helping to identify possible approaches for future autonomous crack detection.