Key challenges in multispectral imaging involve two notable“axes”—spectrum and space. Limits to achievable spectral and spatial resolutions in a multispectral measurement device are often considered independently. But the high dimensionality of the multispectral images (and the associated complexities) underscores the difficulty in maximizing the information throughput via this approach. To this effect, I will argue that we must move beyond the“bag-of-pixels” paradigm—that is, treating the spectral design axis independently of the spatial design axis—to begin coding the spatial and spectral information jointly.Indeed there is significant interplay between the reflectivity of a surface and its structure in space. In this talk I will describe how to leverage these spatio-spectral characteristics to re-evaluate the capacity of the multispectral imaging systems in order to improve upon design for acquisition, analysis, and display. Among benefits to this approach are an analysis framework revealing surprising inefficiencies in existing systems, a method for disambiguating between the reflectance and illuminant, and improvements in fidelity, computation, and signal-to-noise ratio.