To resist the adverse effect of background translation, rotation, scale variation and viewpoint change under moving camera and improve performance, a background modeling method that combines matching and tracking is proposed on the basis of regional dual-mode Gaussian model. Firstly, Oriented FAST and Rotated BRIEF (ORB) is modified as Modified ORB (MORB) for matching points to address the problem that ORB descriptors are excessively dense. To enhance the overall description and reduce the amount of calculation, random feature points Lucas–KanadeRandom (LKR) based on LucasKanade (LK) optical flow is proposed for obtaining tracking points. Confident precise feature points are calculated by united match and tracking points to improve inter-frame motion estimation accuracy. Then, regional model is applied for improving real-time performance. The regional background is modeled by online-candidate dual-mode Gaussian distribution. Experimental results show that the proposed method achieves real-time detection performance with average frame rate of 40.6 fps under the abovementioned background motion with stronger robustness and higher accuracy in a variety of datasets.