Many different fashion related computer vision applications have been developed over the past few years. However, as an important attribute of fashion garments, color in fashion is rarely studied subject, and a color name matching algorithm is highly desired by the online fashion community that maps a garment color from an image to a verbal description. As a continuation of our previous nearest-neighbor-based fashion color matching method, in this paper, we propose a psychophysical experiment to collect fashion color naming data and a new data-driven classification model using random forest for color name classification. Our reversed color naming experiment uses a simple and straightforward procedure to extract users’ color naming schema. The random-forest classifier utilizes a set of linear and non-linear features in CIELab color space. It achieves more than 80% accuracy, and shows great improvement over our nearest-neighbor-based model. Furthermore, this data-driven approach also has the ability of actively and dynamically learn and improve the algorithm; and it is also able to learn new users’ color vocabularies.