This article presents a framework for understanding, modeling, and computing color categories on the basis of knowledge from the color imaging science. One of the main assumptions advocated in this article is that the structure of color categories originates from the statistical structure of the perceived color environment that was observed throughout an individual's life. The perceived color environment can be modeled as color statistics of natural images in some perceptual and approximately uniform color space (e.g., the CIELUV color space). The process of color categorization can be modeled as the grouping of the color statistics by clustering algorithms (e.g., K-means). The proposed computational model enables one to predict the location, rank, and number of color categories. The model is examined on the basis of K-means clustering analysis of statistics of 630 natural images in the CIELUV color space. In general, the model predictions are consistent with data from psycholinguistic studies. The model might be applied in different areas of imaging science such as color quantization, image quality, and gamut mapping.