We investigate the efficiency of color similarity measures in the context of color classification tasks. Color similarity measures are categorized into three major types: Point-type CSMs for classifying colors by absolute value, diffuse reflection component based CSMs for classifying “body colors” of object surfaces made of dielectric materials, and dichromatic reflection component based CSMs for classifying colors that include both the diffuse and the specular reflection component. First, the results of an evaluation of existing color similarity measures with regard to their ability to separate color classes using more than hundred test cases of the background-frame-differencing type are reported. It was found that especially two of the existing color similarity measures (S₇ and S₈) have good properties with respect to brightness, saturation and hue changes, and that they have a compact scalar field structure in RGB space. In addition, they have the largest margin for class separation. Second, we propose several new color similarity measures of the diffuse reflection component based color similarity type and report on the results of an evaluation regarding their class separation ability using more than 50 test images (for facial skin color classification). It was found that two color similarity measures (S₁₃ and S₁₄) perform best for separating color classes when the colors are treated as body color reflections. The properties of these measures with respect to brightness, saturation and hue changes are not perfect, but nonetheless reasonably good, and they have a compact scalar field structure in RGB space. Third, we propose a new color similarity measure of the dichromatic reflection component based type and discuss the problems associated with it. Finally, we introduce a shifted reference color similarity measure which can be used to compute the similarity between colors that may be the result of a systematic shift due to some physical phenomenon.