This paper examines the feasibility and limitations of Forward Looking Infrared Radiometer (FLIR) Aerial Technology for detecting fouled ballast. The method is intended to provide an efficient and ready-to-use approach that can help the railroads detect fouled ballast in their early stages. Ballast fouling commonly occurs as a result of fine particles clogging off water passage through them. Subsequently, this results in trapped water that often results in poor foundation strength, rotting of the ties, and other ill effects. This study includes a novel approach to evaluate the railway ballast fouling by using thermal imaging techniques. In particular, the thermal characteristics of clean and fouled ballast are studied using FLIR cameras that can be used onboard rolling stock, Hyrail trucks, or drones. Laboratory tests are primarily performed to measure the surface temperature changing rate of clean and fouled ballast in response to ambient temperature changes. For the purpose of laboratory testing, the camera is set up in stationary and moving configurations. The test results indicate that clean and fouled ballast have different thermal characteristics. In particular, different thermal patterns are obtained during naturally-occurring daily temperature change. The test results also indicate that the FLIR cameras can be used on a moving platform for quick scanning of thermal images of the ballast that could be used for assessing the early stage of fouling.