We present a statistical framework to automatically determine an associated color for a given arbitrary semantic expression. The expression can not only be a color name but any word or character string. In addition to the color value, we are also able to compute the result's significance, which determines how meaningful defining the color is for the expression. To demonstrate the framework's strength we apply it to two well known tasks: assessing memory colors and finding the color values for a given color name (color naming). We emphasize that we solve these tasks fully automatic without any psychophysical experiment or human intervention. Further, we outline the potential of our automatic framework and in particular the significance for the imaging community.