Background subtraction is a fundamental problem in computer vision. Despite having made significant progress over the past decade, accurate foreground extraction in complex scenarios is still challenging. Recently, sparse signal recovery has attracted a considerable attention due
to the fact that moving objects in videos are sparse. Considering the coherent of the foreground in spatial and temporal domain, many works use the structured sparsity or fused sparsity to regularize the foreground signals. However, existing methods ignore the group prior of foreground signals
on multi-channels (such as the RGB). In fact, a pixel should be considered as a multi-channel signal. If a pixel is equal to the adjacent ones that means all the three RGB coefficients should be equal. In this paper, we propose a Multi-Channel Fused Lasso regularizer to explore the smoothness
of multi-channels signals. The proposed method is validated on various challenging video sequences. Experiments demonstrate that our approach effectively works on a wide range of complex scenarios, and achieves a state-of-the-art performance.