Consistent monitoring of a right-of-way (ROW) is an important task for protecting the integrity of pipeline infrastructure. Pipeline monitoring is typically conducted visually by ground based and airborne inspection crews. In this paper, we present a real-time full-fledged automated airborne monitoring system that can detect, recognize, and localize machinery threats such as construction equipment, occurring on a pipeline ROW. In our approach, a modular key frame (MKF) selection technique is developed to improve data processing speed, a pyramid Fourier histogram feature is used for feature extraction, and a cascaded classifier is introduced for object categorization. Experimental results using two real-world datasets indicate that the proposed system is able to detect and recognize objects in challenging environments such as low illumination, varying resolution and partial occlusion. The results also show that our system can reach real-time processing speeds with good accuracy which offers a new and useful tool for wide area pipeline surveillance.