Human color constancy has been defined as the ability to recognize the colors of objects independent of the characteristics of the light source [1]. Computational color constancy, hereafter referred to as “color constancy” for brevity, is a technique developed in digital imaging to emulate the process of human color constancy within the context of digital sensors. Its implementation in consumer devices is often referred to as “Automatic White Balance”, in contrast to “Manual White Balance” where the user is in charge of removing (or changing) the color cast from the acquired image.

Color constancy is typically addressed as a two-stage process, with illuminant estimation followed by illuminant correction. The first stage aims at characterizing the illuminant source, and it is the main focus of most algorithms for color constancy. The second stage employs a mechanism for chromatic adaptation to correct the image. The most common and simple solution for illuminant correction consists in applying to tristimulus data a diagonal transformation matrix, referred to as a “wrong von Kries transform” [2]. Color constancy is commonly evaluated in terms of angular errors [3–5]. The recovery angular error [4] treats the ground truth illuminant and the estimated illuminant as vectors in RGB space, and measures the angle between such vectors, thus comparing only the chromaticities and ignoring their absolute intensities. The reproduction angular error [5] simulates the effect of a possibly-wrong estimation on the correction of a neutral surface; the ground truth illuminant (representing the appearance of a neutral surface) is normalized by the estimation, applying a correction through a wrong von Kries transform. The resulting vector is then compared in terms of its angular distance with the vector of perfect white, defined by RGB = [1,1,1]. Despite the wide adoption of angular error metrics for the evaluation and comparison of color constancy algorithms, there exists no consensus about angular thresholds for acceptability and noticeability. After informal experiments first conducted in 2005, Finlayson et al. [6] considered a 1∘ perturbation of the image illuminant in any given color direction as not visibly noticeable, while a 3∘ perturbation was found to be noticeable but “generally acceptable”, as later supported by Fredembach et al. [7] and adopted as a threshold in the comparison of color constancy methods [8]. A 5∘ perturbation was defined as “generally acceptable, but unacceptable for some images”, thus manifesting the impossibility to assign a unique threshold for acceptability, even within the same experiment. Similarly, in 2006, Hordley [9] conducted another informal analysis, suggesting an angular error of 2∘ to be acceptable for color constancy. Rather than proposing an absolute threshold, Gijsenij et al. [10, 11] addressed the concept of noticeability in relative terms, determining that the difference in terms of angular error between two methods should be at least 0.06 times the maximum of the two errors, in order for it to be noticeable. This idea was later adopted for method comparison to determine the significance of angular error differences [12]. Gijsenij et al. [10, 11] also conducted a broader analysis of alternative metrics for error evaluation of color constancy, proposing a weighted Euclidean distance, and a corresponding coefficient for the threshold of relative noticeability. Other authors also considered alternative metrics and domains for the definition of color thresholds. Hordley [9] reported a CIELab error of 1 as a just noticeable threshold for two colors viewed side by side in isolation, and a CIELab error of 6 as an acceptable threshold for the assessment of complex scenes.

An intrinsic limitation of defining acceptability and noticeability in RGB space is that such representation is not perceptually uniform, whether it is assumed to be sRGB or a device-specific raw-RGB. A possible solution to the problem lies in measuring the error through perceptual difference metrics, such as the CIE76 ΔE ∗ ab [13], based on the Euclidean distance between two colors in CIELab color space. One advantage of this solution is the availability of de facto standard thresholds, such as the JND (Just Noticeable Difference) set to 2.3, or the acceptability in complex scenes defined by Hordley to be 6. This approach has, however, two main drawbacks: first, it ignores the reality that the majority of methods comparison in the domain of color constancy is, in fact, performed with angular error metrics, thus raising a retro-compatibility issue. Secondly, any single-value metric will, by definition, remove any information about the chromaticity of the estimation error, thus depriving of an important clue for the analysis of a method’s faults (this limit is shared with the traditional angular error). Furthermore, there exists a disagreement between acceptability thresholds defined in angular error and ΔE ∗ ab, as manifest in Figure 1: here, three example corrections from the fully convolutional color constancy with confidence-weighted pooling (FC4) method [14] are visualized, together with the corresponding recovery angular error (always under 2∘, considered acceptable), and the corresponding ΔE ∗ ab (always above 6, considered unacceptable).

For these reasons, we propose a solution for the visualization of errors in the domain of color constancy, based on embedding contour lines of perceptual color differences at predefined thresholds into a chromaticity diagram that encodes RGB angular distances as 2-dimensional Euclidean distances, called Angle-Retaining Chromaticity (ARC) [15, 16]. First, using a chromaticity diagram enables the visualization of errors’ chromaticities, thus providing an intrinsically richer representation of the errors. Secondly, with this configuration, the errors can be visualized both in terms of the commonly used angular error (represented by the Euclidean distance in 2D space) and in terms of the more informative CIE76 ΔE ∗ ab (with thresholds represented by contour lines).

Let G = {gR, gG, gB} be a ground truth illuminant in device-specific raw-RGB, and let E = {eR, eG, eB} be an estimated illuminant in the same color space. The angular distance between illuminants G and E is the recovery angular error [4]. Alternatively, the reproduction angular error [5] is obtained through the reproduction vector R: The angular distance between the reproduction vector R and a neutral surface W = {1,1,1} is, by definition, the reproduction angular error.

The ARC diagram [15, 16] is a general-purpose 2D representation of color information, which may be obtained from any given RGB vector {ρR, ρG, ρB} as follows:

Alternatively, the polar chromaticity coordinates {αA, αR} may be represented in Cartesian form {αX, αY } for visualization purposes. When converting a reproduction vector R into ARC, each pair of ground truth and estimated illuminants becomes a single two-dimensional point, whose Euclidean distance from the diagram center corresponds exactly to the reproduction error.

Our goal is to assign a perceptual error to every point in the ARC diagram and to eventually visualize this information with contour lines. Such error is always computed between the reproduction vector R and the white vector W, in line with the computation of the reproduction angular error. This strategy entails a reduction of dimensionality that enables data visualization since displaying perceptual errors for all possible illuminant pairs would not be possible. As each illuminant is represented by at least two chromaticity coordinates, in fact, this would lead to four independent variables and one dependent variable (the error value). By contrast, the advantage of always referencing a single point for comparison (W) is that the only independent variable is the two-dimensional ratio vector in chromaticity, thus allowing for error visualization.

Let the coordinate system of the reproduction vector R be called “reproduction-RGB”, which differs from raw-RGB as explained in the following. The procedure used to compute the perceptual error for R is delineated here, which starts by bringing the reproduction vector back into raw-RGB color space.

In this section, we apply the proposed visualization to images from the ColorChecker dataset [17]. This dataset is composed of 86 images acquired with a Canon EOS-1DS camera, and 482 images acquired with a Canon EOS 5D camera. Every image includes a Macbeth ColorChecker target within the scene. From this, the brightest of the six achromatic patches with no oversaturated channels is used to determine the ground truth illuminant for the color constancy task, following Hemrit et al. [18]. For both devices in the dataset, we reference the corresponding model-specific raw-to-XYZ transformation matrices from the DCRAW software (https://www.dechifro.org/dcraw/.), since no calibration data is available and all images include a color target for device-specific illuminant annotation, no absolute reference is provided, thus preventing the definition of a proper color profile.

We compare estimations from three methods for computational color constancy, selected as representatives of different categories; the second-order Grey Edge (GE2) in its implementation by van de Weijer et al. [19] as an instance of a statistics-based algorithm, Quasi-Unsupervised color constancy (QU) by Bianco et al. [20] as an instance of semi-supervised machine learning, and FC4 by Hu et al. [14] as an instance of fully-supervised machine learning. The learning-based methods QU and FC4 are designed for full white balancing, i.e. they are trained to identify neutral regions in the scene, and to consequently correct the image so that these regions are rendered as gray. GE, on the other hand, is purely an illuminant estimation method, with no assumptions on how to perform the final correction. Additionally, we note that this selection of methods is to be intended as a use case for the general application of our visualization methodology to any single-illuminant algorithm for the color constancy of white. The relationship between angular errors and ΔE ∗ ab, visually presented in Fig. 1, is synthesized in Table I. For all combinations of cameras and methods, we report aggregate error statistics as well as the correlation between angular error (recovery or reproduction) and ΔE ∗ ab. This incomplete correlation is later explored thanks to our visualization.

Our proposed visualization is presented in the chromaticity diagrams of Figure 2. Here, Euclidean distances from the center correspond to the reproduction angular error expressed in degrees and can be visually judged from the diagram’s own frame of reference. The rendered ΔE ∗ ab contour lines visually highlight how the sensitivity of the human visual system is not fully correlated to the absolute angular error, depending instead on the hue of the error itself. Specifically, errors distributed on the correlated color temperatures of a black body radiator, moving from blue to yellow/red, are less perceivable than errors distributed on the orthogonal axis, moving from green to magenta. Both cameras from the ColorChecker dataset lead to similar conclusions, although a difference in sensitivity on the green-magenta axis can be clearly observed. Every cross (×) in the diagram is a reproduction vector computed according to Eq. (1) using the single gray-patch-based ground truth for a given image, and the corresponding illuminant estimation obtained from a given method. The three analyzed methods produce errors distributed in different regions of the chromaticity diagram, with GE2 and QU exhibiting a stronger bias towards blue overcorrections, and FC4 towards yellow overcorrections. In general, they all display an error distribution that roughly follows the blue to yellow/red axis of a black body radiator. While this may be expected of data-driven methods such as QU and FC4, which tend to inherit the bias of their corresponding training datasets, it is a surprising result for the data-free GE2 method. A possible explanation lies in the inherent limitation of annotating a single illuminant in what is actually a multiple-illuminant scenario [21] due, for example, to mutual surface inter-reflections or the coexistence of sun and shadow areas. In this case, the chromaticity of the error is potentially revealing non-annotated illuminant information. This interpretation could be further tested by extending the evaluation to images acquired under non-blackbody illuminants, a feature that is known to be lacking from many of the existing datasets for color constancy [3].

We have presented a visualization technique for color constancy errors that visualizes, at the same time, the traditionally-used angular error, its chromaticity, and its perceivability. Our visualization has been applied to three color constancy methods with different levels of complexity and accuracy, on two cameras of the popular ColorChecker dataset, providing a broad overview of its applicability. For future development, we consider expanding this analysis to a wider range of methods, datasets, and perceptual metrics. An inherent difficulty in assigning a perceivability score to color constancy errors derives from the necessity to evaluate the whole image in context, as opposed to only comparing colors “in a void”, which poses an additional challenge in terms of data visualization.