It is well-known that the knowledge of the natural geometry of a problem is often crucial in finding solutions. Problems involving functions on a circle are, for example, often solved using the theory of Fourier series. This is the mathematical explanation of the enormous success of the DFT, FFT and DCT-based methods. Another example is the relation between scaling properties and wavelet theory.In this paper we show that spaces of spectral distributions, like color stimuli, have a natural cone-like structure. We use the framework of the Karhunen-Loéve transform in a Hilbert space context to describe this cone-like structure and demonstrate how to compute natural coordinate systems from empirical data, like multi-spectral measurements and images. We will illustrate the theoretical findings with databases consisting of collections of multispectral measurements of color chips from color systems like Munsell, NCS and Pantone, multi-channel images of natural scenes, satellite data and daylight spectra.We will also comment on the possible application of group theoretical methods in color science based on those findings.