Objects in real three-dimensional environments receive illumination from all directions, characterized in computer graphics by an environmental illumination map. The spectral content of this illumination can vary widely with direction [1], which means that the computational task of recovering surface color under environmental illumination cannot be reduced to correction for a single illuminant. We report the performance of human observers in selecting a target surface color from three distractors, one rendered under the same environmental illumination as the target, and two rendered under a different environmental illumination. Surface colors were selected such that, in the vast majority of trials, observers could identify the environment that contained non-identical surface colors, and color constancy performance was analyzed as the percentage of correct choices between the remaining two surfaces. The target and distractor objects were either matte or glossy and presented either with surrounding context or in a dark void. Mean performance ranged from 70% to 80%. There was a significant improvement in the presence of context, but no difference for matte and glossy stimuli, and no interaction between gloss and context. Analysis of trial-by-trial responses showed a dependence on the statistical properties of previously viewed images. Such analyses provide a means of investigating mechanisms that depend on environmental features, and not only on the properties of the instantaneous proximal image.