This paper is about human perception of color and brightness. It is well known that a light of a given spectral energy distribution can produce many alternative percepts depending on other lights nearby or viewed previously. Consider, for example, a patch of light that appears white when viewed against a dim achromatic background. The same patch appears charcoal gray when viewed against an intense background. Varying the background also affects color perception. A patch that appears orange against the dim background is perceived as brown on the intense one. Typically, the influence of background light on color or brightness is inferred from measurements of the change in appearance of one light (a patch) caused by introducing a second light (either a surrounding background or an ‘adapting field’ on which the patch is superimposed). This work has been fruitful but, as discussed below, has important limitations for understanding the color and brightness of visual stimuli composed of more than two lights.