In this paper, we describe a novel method for image-based rail defect detections for railroad maintenance. While we developed the framework to handle a broad range of defect types, in this paper we illustrate the approach on the specific example of detecting cracks located on fishplates connecting rails in images. Our algorithm pipeline consists of three major components: a preprocessing and localization module, a classification module, and an on-line retraining module. The pipeline first performs preprocessing tasks such as intensity normalization or snow pixel modification to better prepare the images, and then localizes various candidate regions of interest (ROIs) where the defects of interest may reside. The resulting candidate ROIs are then analyzed by trained classifier(s) to determine whether the defect is present. The classifiers are trained off-line using labeled training samples. While the system is being used in the real-world, more samples can be gathered. This gives us opportunity to refine and improve the initial models. Experimental results show the effectiveness of our algorithm pipeline for detecting fishplate cracks as well as several other defects of interest.