Moving object segmentation plays an important role in a complex object tracking system. This system decides whether the current block belongs to the object region or not. In this article, a scheme using background modeling based on runtime-weighted features for robustly adaptive moving object segmentation in infrared (IR) image sequence is proposed. Proposed background modeling for an open hardware (H/W) architecture design decreases the size of the search area to construct a sparse block template of search area in infrared images. The authors also compensate for motion compensation when the image moves in previous and current frames of IR imaging system. The method of separation of background and objects applies to adaptive values through time analysis of pixel intensity. The proposed method uses more feature information such as intensity, deviation, block matching error, and velocity. The weighting values give a higher weight to feature information which has a large difference between the object and the background region. Based on experimental results, the proposed method showed real-time moving object segmentation through background modeling in the proposed embedded system.