Background subtraction is a fundamental problem in computer vision. Despite having made significant progress over the past decade, accurate foreground extraction in complex scenarios is still challenging. Recently, sparse signal recovery has attracted a considerable attention due to the fact that moving objects in videos are sparse. Considering the coherent of the foreground in spatial and temporal domain, many works use the structured sparsity or fused sparsity to regularize the foreground signals. However, existing methods ignore the group prior of foreground signals on multi-channels (such as the RGB). In fact, a pixel should be considered as a multi-channel signal. If a pixel is equal to the adjacent ones that means all the three RGB coefficients should be equal. In this paper, we propose a Multi-Channel Fused Lasso regularizer to explore the smoothness of multi-channels signals. The proposed method is validated on various challenging video sequences. Experiments demonstrate that our approach effectively works on a wide range of complex scenarios, and achieves a state-of-the-art performance.