Establishing accurate hair diagnosis at roots is a significant challenge with strong impact on hair coloration, beauty personalization and clinical evaluation. The roots of hair - viz. the first centimeter away from the scalp - represent clean hair fibers that have not been subjected to color change due to hair dyeing or environmental conditions. Therefore, they are a measure of a person’s baseline hair characteristics, including natural hair tone. A device that acquires high resolution macro images of hair roots under a well-defined illumination geometry has been designed in order to assess natural hair tones. Image analysis in this scenario is not a trivial task since the acquired images present an overlap of scalp and hair, with other possible artifacts due to dandruff and hair transparency. In this paper, we propose to train a Convolutional Neural Network (CNN) on a data-set composed of images from subjects who had their hair tone evaluated by trained color experts. Our method is compared with other popular CNNs as well as conventional color image processing approaches developed for this task. We found that the proposed model not only offers higher precision but also provides faster computation times, due to its lighter architecture in contrast to popular CNNs. Thus, we achieve sufficiently accurate results in real time on the low-end chip embedded in our device.