In this contribution we present a new technique of highlight substitution. From a color image sequence, acquired with a hand-held camera, a so-called light field is generated. Additionally, a highlight mask is calculated for each image of the sequence. The highlight mask is then used as a confidence map for the light field. This results in color pixel interpolations at highlight pixels, taken from images in which these pixelswere not over-imposed by highlights, resulting in better images. We demonstrate the technique on medical endoscopic images and evaluate the results on both, natural and synthetic data.