The HMD displays were measured by placing a front-surface reflective mirror at 45∘ inside HoloLens at the eye position and measuring the reflected image from above, as in Fig. 1 (top right). Because the HoloLens is a complete computer rather than simply a display device, displayed content was controlled using the Unity game engine, showing a rendered scene with a diffuse patch of infinite size with defined color and illuminated with directional light at 45∘ in color (255, 255, 255). The patch was viewed at an angle normal to the Unity main camera. The HoloLens was set to its maximum luminance in all conditions, and measured in a dark room.

Primaries’ spectra were directly measured with PhotoResearch PR655 spectroradiometer at 12 different locations on both left and right displays. Fig. 1 (bottom right) shows the 24 sets of primary chromaticities in u′v′ space where the black line is the spectrum locus. The average chromaticities of R, G, B in u′v′ space are (0.5087, 0.5227), (0.0655, 0.5794), and (0.1850, 0.1046), respectively. Primary chromaticities are relatively stable between different locations, though luminance varies. Primary ramps were measured at one point near the center of the display. The average black level, aka flare, XY Z tristimulus values are (0.1565, 0.1209, 0.2231). Ramps were normalized and interpolated after subtracting flare as in Figure 2 (left). It is shown here that luminance for all channels reaches a peak before maximum digital intensity values.

The transmittance of the displays was also measured, with the HoloLens turned off. An incandescent light source in an integrating sphere was placed in front of the HoloLens and reflected by the mirror. Spectral radiance was measured with the PR655 at five different locations on both left and right lenses. The same setting was measured again without the HoloLens and transmittance was calculated as shown in Fig. 2 (right), where the left lens is in red and the right lens in green. The transmittance curve is slightly different for the left and right lenses. However, the difference was small, and because no hint of binocular rivalry was observed in any experiments, the average of all curves, drawn in blue, was used as the overall transmittance of HoloLens.

A camera was used as a colorimeter to measure the spatial variance of display primaries with the same mirror setup as above, because it cannot be measured accurately with a spectrophotometer. A Nikon D40 DSLR was set to manual mode with ISO 800, shutter time 1∕6 sec, aperture F32, and focal length 40 mm. These parameters were set so that the shutter time was long enough to cover RGB updating cycles, but the aperture was small enough to avoid overexposure. When the mirror angle was changed, the pattern moved for both left and right lenses. This indicates that the result can only represent one viewing position. Eight photos with the HoloLens showing R, G, B and W were shot through the mirror in a dark room for both left and right displays as shown in Fig. 1 (left). Photos were saved in raw image format for data processing (NEF for Nikon camera).

In order to estimate CIE XY Z for each pixel in the images, a camera characterization was performed. The same camera parameters as above were used to shoot a raw format photo of a Macbeth Colorchecker (MCC) target in a GretagMacbeth Spectralight III light booth under D65 illumination with 0∕45 geometry. CIE 1931 XY Z values were also measured with the same illumination and measurement geometry for all patches. The raw NEF image of the MCC was first converted into DNG format using Adobe DNG Converter with no compression, then read into MATLAB as color filter array (CFA). The original CFA image was then linearized with the compression curve stored in the TIFF file and demosaicked into camera RGB for all pixels with ‘BGGR’ pattern. The parameter ’AsShotNeutral’ stored in TIFF file indicates the signal intensity of each channel for neutrals. Camera RGB was then weighted by the reciprocal of ‘AsShotNeutral’ for each channel so that R = G = B is neutral [22]. For each patch on the MCC, the camera RGB value was taken as the average of the center 100 × 100 pixels. A camera RGB to CIE XY Z matrix M was calculated with pseudoinverse. The estimated XY Z of 24 patches were calculated with matrix M, and both measured and estimated XY Z were converted to Y u′v′ to validate the matrix. The conversion error is 3.0 ± 0.8ΔE.

Images of the HoloLens, left and right lenses displaying R, G, B, and W were processed the same way as above to obtain camera RGB values for each pixel. The 3008 × 2000 images were blurred with a 20 × 20 pixels median filter for each channel separately to avoid discrete camera signal for later sampling. Camera RGB values were then converted into CIE XY Z using the matrix calculated above. From these XY Z images, a 21 × 56 grid with size of 10 × 10 pixels in the display area was built for sampling. The calculated chromaticities of sampled points of HoloLens R, G, B, and W are shown in Figure 3 in comparison to the measured primary average in Y u′v′ space. It is clear from the R, G, B points that the display only varies spatially in luminance while the chromaticities are stable for each primary. We conclude that the chromaticity variation observed in W is due to the composition of the differently nonuniform R, G, B fields. In the next section, the chromaticity and spatial luminance maps were used to build the display model.

The AR display model converts one triad of RGB input values to an XY Z image: a set of XY Z tristimulus values with spatial location to represent the spatial variance of the virtual patch based on the measurement described above. The model includes two parts: first, data from the two separate displays were combined into one binocular display, and the relative luminance was mapped to absolute luminance. Second, input RGB values were converted to XY Z tristimulus values using primary ramps and applying appropriate chromatic adaptation. The model is explained here and applied to the color matching experiment in the next section.

The two sets of spatial luminance variance calculated above, of the left and right lenses, were combined into one by averaging and normalizing them to represent binocular vision. The left 10% area of the left display and right 10% of the right display were discarded before merging because they are assumed to be outside of binocular vision. To restore the absolute luminance, the maximum spectroradiometer-measured luminance of each channel characterization section was used as peak luminance to scale the camera-based, normalized luminance map to absolute luminance in cd/m2. The R, G, and B channel spatial luminance maps are shown in Figure 4, along with the spatial luminance of white as the summation of three channels. The G luminance reaches 315.7 cd/m2 while R and B peak at 230.2 cd/m2 and 17.2 cd/m2, respectively.

The model was applied as follows: for each spatial pixel, HoloLens RGB values were converted into normalized XY Z for each channel separately using normalized ramp data as a lookup table and scaled to actual XY Z with the spatial luminance map. Final XY Z values were calculated taking the summation of three channels and then adding the flare and background XY Z. Chromatic adaptation was applied on the tristimulus values of the patches using CIECAT02 model, using parameters selected according to the viewing condition [23]. For the experiment described in the next section, the background luminance surrounding the virtual patches (1.811 cd/m2) was used as the adapting luminance, the white point was (94.95, 100.0, 65.39), and the viewing condition was set to “average.”

A color matching experiment was performed between the HoloLens and a LCD display. The HoloLens was driven by Unity through Holographic Remoting, showing a diffuse patch of 0.3 × 0.3 m illuminated with white directional light at 45∘. In real-world room coordinates, this virtual patch was positioned at the observers’ eye level at a distance of 2.4 m, just in front of a black board background with luminance at 1.811 cd/m2. A LCD display (Apple Cinema Display Model A1083) was positioned at the same distance height, to the right of virtual patch. The LCD was covered by a black board with 0.3 × 0.3 m square cut out in which to show a reference patch. The virtual patch shown with HoloLens and reference patch shown with LCD were each about 7∘ of visual angle in size, separated by 1∘ –7∘ depending on the initial head position of the observer. Thus, the viewing angle from the left edge of virtual patch to the right edge of reference patch was from about 15∘ to 21∘ (Figure 5, left). Observers were free to move their head during the experiment, but thanks to the head position sensing in the HoloLens, the virtual patch remained fixed in space.

In the experiment, observers were asked to adjust the color of the virtual patch to match the reference, using a keypad to adjust dimensions of hue, saturation, and value (HSV). Selected HSV values were sent to MATLAB and then to Unity through TCP/IP connection to update the color of virtual patch (Fig. 5, right). A total of 27 reference patches (a 3 × 3 × 3 grid in RGB, visible in both Figures 6 and 7) were selected for matching. After the matching experiment, two questions were asked about the experiment: Did you notice the nonuniformity during matching? Is it difficult to match with the nonuniformity? 18 observers with normal color vision participated in the experiment, of whom 12 were male and 6 female. 16 out of 18 observers noticed the spatial nonuniformity of the virtual patch. The other two noticed side-to-side difference or edge strips but not spatial variance. No observers reported that the patch is impossible to match due to the color difference between two displays.

Tristimulus values of all of the LCD-displayed reference patches were measured under the experiment illumination condition from the observer’s eye position using a PR655 and combined with the measured transmittance of HoloLens HMD. CIECAT02 was applied with the reference white patch as the white point and background luminance as adapting luminance as described in section 3.2.

Virtual patches were reconstructed from observers’ matching results: the HSV values were converted to RGB and used as input to the AR display model described above, using the same chromatic adaptation parameters as the reference patches. For every set of input RGB values, the AR display model output a spatial image, essentially a set of colors with spatial coordinates. A General Observer for each patch was created by averaging all observers’ matching result in CIE XY Z space, shown along with the references in the Y u′v′ chromaticity diagrams in Fig. 6. The reconstructed General Observer colors in HoloLens are marked as dots while reference colors are marked as diamonds. Each reference patch diamond is inside or very near the cluster for most patches, and in most cases the reference patch is lower in luminance than the match. The spatial images of the reconstructed virtual patches are shown in Fig. 7, and below each reconstructed image the corresponding reference patch colors are shown as uniform color bars.

Given the visible nonuniformity in the AR OST-HMD, it is unknown what portion of the display observers were using when performing color matches. Our three plausible hypotheses are that observers may be matching the brightest area, most uniform area, or simply the initial location as presented in the side-by-side arrangement in the color matching experiment.

Color difference in CIE Delta-E2000 (ΔE) for each patch was calculated between all spatial sampling points on the reconstructed patch and reference patch. In all patches, there are certain areas whose color difference is much smaller than other areas, suggesting that observers may be using this in their match. To test the plausibility of different matching criteria, a window size of 8 × 8 (in spatial sampling units) was slid over the reconstructed display images to find the position of the minimum ΔE between the reference and the General Observer’s match. The window position was considered minimum only when the summation over all 27 patches was the smallest.

The ΔE values of the brightest area and the most uniform area were calculated in a similar manner by finding the overall brightest or least variance window position, respectively. The position of the initial location window was selected assuming observers kept their head position fixed with an initial forward gaze toward the space between the virtual and reference patches during the experiment, an assumption that seems plausible based on watching the observers’ behavior. The positions of the four computed window positions are drawn on the reconstructed patches in Fig. 7, and the ΔE values of the four windows for all patches are shown in Figure 8. The minimum ΔE in red is overall the smallest, as expected. The ΔE of the initial location window in cyan is generally smaller than the other two windows and close to the minimum ΔE window, meaning the two hypotheses of matching brightest and most uniform area are unlikely to be correct. It is likely that observers simply oriented their heads to look toward the middle in the experiment and held their heads relatively still while making matches. Looking at the four window locations drawn on the reconstructed patches in Fig. 7, it can be seen that the minimum ΔE window location greatly overlaps the initial location window, further supporting the initial location hypothesis.

Additionally, the observers’ variance in making matches was calculated in the minimum ΔE window with CI of the Poisson distribution [24]. Observer’s average ΔE is significantly larger than General Observer’s ΔE, which is reasonable since the matching result is a cluster around the reference point in 3D space, and the average of the cluster should be closer to the reference than the ΔE from any single matching point. Meanwhile, the observer’s ΔE CI seems stable among different reference colors within the range of ± 1.6ΔE.

To explore how different reference colors affect the matching ΔEs, the relationships between minimum difference window ΔE and chroma and lightness for all 27 patches are shown in Figure 9. Except two blue patches that are out of gamut (marked in triangle), there appears a trend that the more chromatic the patch is, the more accurate the color is matched. The computed linear correlation coefficient between matched chroma and minimum color difference is − 0.5933 with p value of 0.0018. This could be simply because in the AR OST-HMD more chromatic colors are produced using fewer color channels and are thus affected less by the channel spatial variance. Composite neutral colors like white and gray are affected the most, which fits our observations. Lightness of the patches on the other hand does not show a clear relation with correlation coefficient and p value of − 0.0104 and 0.9607, respectively.

Nonuniformity of near-eye displays is common in both augmented reality (AR) and virtual reality (VR) displays. Attempts have been made to create more uniform near-eye displays, mostly by focusing on better waveguides [25, 26]. In the current case, we are not sure if the nonuniformity originates in the LCoS or the optical waveguide, but the nonuniformity patterns exist only in each RGB channel’s luminance distribution, and they move rigidly with eye location. This suggests that nonuniformity compensation is possible by adjusting the input channel spatial intensity to achieve better uniformity; this may be a more practical solution than optimizing the optical elements for uniformity.

Based on the color matching experiment, though most observers noticed the nonuniformity, no observer reported the matching task was impossible, nor did they have difficulty in fusing two images from the left and right displays. The AR display model showed that the magnitude of spatial variance can be measured and applied to psychophysical results. It may be an interesting question to explore how much spatial variance would be necessary to cause binocular rivalry.

The display nonuniformity was first found when performing display measurement, meaning a large area of solid color in a darkened measurement lab. This was not noticed when viewing example holograms in a normal-use environment, so it is reasonable to assume that the noticeability and acceptability of nonuniformity are related to background and content complexity. Considering more common and more complex AR situations, it would be very interesting to quantify the noticeability and acceptability of nonuniformity as they depend on content complexity, including dimensionality, 3D model polygon count, texture maps, and motion, and also how they depend on background complexity in color, texture, and illumination.

The eye position relative to the display can affect the nonuniformity pattern. For HoloLens, the large eye relief and relatively large eye box offer room for flexible eye position, which also introduces more variance in color appearance. In this research, we only measured and sampled the spatial variance from one viewpoint, while in the psychophysical experiment, observers’ eye locations could have been different. Thus, the observed virtual patches may not completely agree with the modeled patches. An improved future experiment might try to constrain eye position more precisely, but this would be difficult with the commercial HMD. Another possible improvement would be to measure the display from additional eye positions inside the eye box and then model the display with an additional variable of eye position. If the relative eye positions to the HMD could be measured, the model could predict the virtual image accurately and position-dependent compensation strategies could be used.