Recently, a remarkably simple method was developed to solve the illumination and reflectance spectra separation problem (IRSS) based on the standard low-dimensionality assumption of reflectance. However, because this method assumes the scene is under one uniform illumination, it can not handle scene contains multiple illuminations or dominant shadows. In this paper, we address this problem by formulating the multiple illuminations and reflectance separation problem as a Conditional Random Field (CRF) optimization task over local separations. We then improve local illumination and reflectance separation by incorporating spatial information in each local patch.