The physical properties of color are usually described by their spectra, eigenvector expansions or low-dimensional descriptors such as RGB or CIE-Lab. In the first part of the paper we show that many of the traditional methods can be unified in a framework where color spectra are elements of an infinite-dimensional Hilbert space that are described by projections onto low-dimensional spaces. We derive some fundamental geometrical properties of the subset of the Hilbert space formed by all color spectra. We describe the projection operators that map the elements of the Hilbert space to elements in a finite dimensional vector space. This leads to a generalization of the concepts of spectral locus and purple line. It will be shown that for geometrical reasons the color space is topologically equivalent to a cone. In the second part of the paper we illustrate the theoretical concepts with four large databases of spectra from color systems and a series of multi-spectral images of natural scenes. We verify the conical property of color space for these databases and compute their, geometrically defined, spectral locus and chromaticity properties. In the last section we relate the natural co-ordinate system in the conical color space to the traditional polar co-ordinates in CIELab. We show that there is a good agreement between the geometrically defined hue-variable and the angular part of the polar co-ordinate system in CIE-Lab. There is also a clear correlation between the geometrical and the CIE-Lab saturation descriptors.