One of the challenges facing the railway community is deciding when new inspection technologies are both accurate and reliable enough for regular use in safety assurance. New sensors and data processing techniques recently introduced in the industry have resulted in a spate of new inspection-related technologies. Track inspection technologies play an increasingly critical role in ensuring the safe and reliable operations. Therefore, railways must be sure these new inspection technologies are effective. Also, objective characterization of inspection technology effectiveness helps to establish confidence in new technology, thereby facilitating its adoption. Railways have typically relied on field confirmation of known defects or used repeatability tests to prove consistency when evaluating track inspection technologies. The Federal Railroad Administration (FRA) sponsored ENSCO to develop a formal procedure for the design and execution of performance evaluation tests. As part of this effort ENSCO explored how to apply model-assisted probability of detection (MAPOD) in the railroad industry. MAPOD combines the results from a small number of physical tests with more extensive results from computer simulation to increase confidence in the performance evaluation results. This paper presents the results of a feasibility analysis for applying MAPOD to various inspection technologies and describes a MAPOD implementation roadmap for a handheld ultrasonic rail flaw detection system.