Many content-based image retrieval systems are not taking into account high level semantic information. In this paper we describe our attempts to include color-based emotion-related properties of images in the search. We show that using color emotion metrics in content-based image retrieval leads to interesting methods for image retrieval and classification based on semantic concepts. The color emotion metric used is derived from psychophysical experiments and uses three scales: activity, weight and heat. It was originally designed for single-color combinations and later extended to include pairs of colors. We show that a modified approach for statistical analysis of color emotions in images, involving transformations of ordinary RGB-histograms, provides a useful tool for image classification and retrieval. The methods used are both very fast in feature extraction, and descriptor vectors are very short. This is essential in our application where we intend to use it for searching huge image databases containing millions or billions of images.