Shadows and reflections cause one of the most confusing effects on detection and tracking of moving objects. While there are a few effective tracking methods that take advantage of shadows casted by objects in motion, largearea shadows casted on the objects by surrounding environments, e.g. buildings, are still disturbing. Therefore, factoring out interferences caused by illumination changes can provide stable appearances and features of the moving objects. In this paper we present a method to normalize the target image sequence with regards to illumination variation, i.e. eliminating shadows and brightness changes of moving objects, as a preprocessing stage of video surveillance and object tracking. The method is demonstrated over our urban scene image dataset which has drastic lighting variation.