In this paper, we propose a new fire monitoring system that automatically detect fire flames in night-time using a CCD camera. The proposed system consists of two cascading steps to reliably detect fire regions. First, ELASTIC-YOLOv3 is proposed to better detect a small fires. The main role of ELASTIC-YOLOv3 is to find fire candidate regions in images as the first step. The candidate fire regions are passed to the second verification step to detect more reliable fire region results. The second step takes into account the dynamic characteristic of the fire. To do this, we construct fire-tubes by connecting the fire candidate regions detected in several frames, and extract the histogram of optical flow (HOF) from the fire-tube. However, because the extracted HOF feature vector has a considerably large size, the feature vector is reduced by applying a predefined bag of feature (BOF) and then applied to the fast random forest classifier to verify the final fire regions instead of heavy recurrent neural network (RNN). The proposed method has been experimentally shown a faster processing time and higher fire detection accuracy with lower missing and false alarm.