To create the latent representation, StyleGAN2-ADA [20] was chosen as a deep generative model. It is a GAN consisting of two convolutional neural networks (CNNs). One is the generator that synthesizes images from random noise that belong to the distribution of the training set. The other is the discriminator, which distinguishes between the real images of the training set and the fake images generated by the generator. The generator’s task is to fool the discriminator so that it predicts that the images synthesized by the generator are real. In this way, both CNNs participate in a zero-sum minimax game against each other. Specifically, the generator has to maximize the adversarial loss, and the discriminator has to minimize it. During training, the loss oscillates until an equilibrium is reached. In this way, the model learns to capture the distribution of the training images, and the generator learns to synthesize an image from any point in the latent space.

StyleGAN2-ADA is an improvement of the StyleGAN2 [22] model, which itself is the successor of the StyleGAN model [21]. StyleGAN improves the GAN architecture by introducing a mapping network that maps the random noise vector Z to an intermediate latent representation W using an 8-layer fully connected network. This mapped W space better disentangles the different features of the synthesized image. StyleGAN2 rearranges the modules in the architecture to improve the quality of synthesis. StyleGAN2-ADA introduces the Adaptive Discriminator Augmentation (ADA) technique that allows it to be trained on smaller datasets. The ADA achieves this by augmenting the training images with different types of transformations to increase the size of the effective training set. At the same time, it prevents these augmentations from leaking into the latent representation of the model.

The W space of the StyleGAN2-ADA model is the latent space considered in this work. It is a 512D space that is inserted into each module of the generator. By manipulating this vector, the aspects of the resulting image can be changed. Each latent vector corresponds to an image in the latent space of the model. The generator will always generate the same image from a latent vector unless further steps of backpropagation are carried out.

Two StyleGAN2-ADA models were trained: one on the fixed lighting set (referred to as Model 1) with three shapes, and the other on the multiple light direction set (referred to as Model 2) with only spheres. The pretrained weights from the StyleGAN2-ADA model trained by Nimma and Gigilashvili [30] were used to initialize both models. Nimma and Gigilashvili [30] trained their model on a rendered set of images of spheres with varying optical parameters. Their images were also rendered using Mitsuba.

Model 1 was trained on the uniform lighting (i.e., fixed across the images) dataset for 1000 kimg, and Model 2 was trained on the multiple lighting dataset for 5000 kimg. One kimg corresponds to the discriminator seeing 1000 real images. The uniform lighting dataset was able to converge in 1000 kimg due to using the pretrained weights that were obtained by training on a similar dataset by Nimma and Gigilashvili [30]. It took 11 hours in total. However, the dataset with multiple lighting directions needed to be trained for 5000 kimg because it also had variation in the lighting conditions for which the pretrained weights were not optimized. This model took 48 hours to converge. Two gamma values were tested. A gamma value of 0.8, which is the default value, was found to be the best performing one. A gamma value of 10 was also tested, which is the recommended value by Karras et al. [20], but it did not perform well. The gamma hyperparameter controls the strength of regularization of the model. Higher values reduce the chance of overfitting. Furthermore, the augmentations used by Liao et al. [27] were tested and found to improve performance compared to the default augmentations. All other hyperparameters were kept as default.

To assess the generalization, Model 1 was used to synthesize 625 images for unseen parameter combinations. The resulting pairs of GAN-generated and Mitsuba-rendered images enabled a controlled evaluation of the model’s performance on unseen data.

We conducted three magnitude estimation experiments to scale perceived translucency, gloss, and lightness in different parts of the latent space. In total, 160 images were shown from both models. The observers were asked to rate the object inside the test image on a scale of 1 to 10, where 1 corresponds to opaque/matte/dark and 10 corresponds to transparent/highly glossy/light in the three experiments separately. Each trial included a reference image illustrating the different extremes (opaque and transparent; matte and glossy; dark and white objects). The observers underwent training before each experiment, where the scaled concepts were explained and examples were shown. The observers were explicitly instructed to evaluate only the specific attribute that the respective experiment was about and to ignore other appearance factors as much as they could. More details about the experimental interface can be found in Figure S.2.1. of the Supplementary Material.

This was a category judgment experiment, where 40 images generated by the model trained on the multiple light direction dataset were shown. Observers were given four options to choose the primary direction where the light was shining toward the object: Left Side of the Object, Behind the Object on the Left Side, Behind the Object on the Right Side, and Right Side of the Object. Similarly to previous experiments, the observers were instructed about lighting directions and how they are set up inside the renderings. They were also told that the directions of light vary on the X and Y axes and not on the Z axis. Here the Z axis refers to the vertical axis. The observers were also made aware of this. Two reference images for each extreme were shown both for glossy and matte objects.

The dataset was sampled to a smaller subset of images to be included in the experiments.

The number of images sampled from the latent space of Model 1 was 120 and that from the latent space of Model 2 was 40, giving a total of 160 images. Two kinds of sampling were performed; both assumed a uniform distribution over the input. One was random sampling, and the other was randomly sampling points that vary only along one axis—meaning points on straight lines parallel to both axes separately. For each shape, 20 points were randomly sampled from the whole cluster and 20 points were randomly sampled from the points inside the cluster whose locations lay on a straight line parallel to the principal axis of the 2D latent space. The reason for sampling points that lie along (parallel to) the axis was to measure how the various optical and perceived parameters of each varied as compared to their location in the latent space. These points helped to calculate the correlation among different parameters and rates of change to understand the surface of the latent space.

Figure 3 shows the sampled data points from the latent space of Model 1. Figure 3(a) shows the points that were randomly sampled for each shape cluster. This was performed to obtain points that represent each shape cluster and capture its variance. Figure 3(b) shows the sampled points that lie on a straight line along the principal axis. Each cluster was visually inspected and its midpoints were identified. The midpoint of the sphere cluster is located at (−3, 7), for the cube it is at (11, −1), and for the Stanford bunny it is (−6, −7). The points of each cluster were filtered so that only the points that had an X axis similar to or close to (rounded to one decimal point) the identified X axis and the points that had a Y axis similar to or close to the identified Y axis were included. Then a uniform random sampling was performed on this filtered list of points. For example, considering the sphere cluster, all points that had an X axis of −3 were put in a list and all points that had a Y axis of 7 were put into another list. These two lists were combined. Then, a uniform random sampling was performed to select 20 samples from this combined list. This procedure was performed on the other shape clusters as well as the cluster in the latent space of Model 2. This kind of sampling was performed to obtain points that capture the rate of change of various parameters of the data points. The parameters can include optical parameters and results of the perceived attributes of the psychophysical experiments and also the axis of the latent space itself. Sampled data from the latent space of Model 2 is illustrated in Figure S.3.1(d–f) in Supplementary Material.

The psychophysical experiments were conducted on a 24′′ Samsung T37F LED display calibrated for the sRGB color space with a gamma of 2.2, 5700 K reference white, and 80 cd/m2 of maximum luminance. The sRGB color space was chosen because the image part of the experiment is encoded in that color space. The vertical illuminance measured with a lux meter was 6.4 lux in front of the observer looking at the display at a distance of 1.6 m and 0.5 lux elsewhere in the room. Informed consent to voluntary participation was obtained from the observers. They were informed about the experiment and how the data would be processed. During the experiment, no personal data was collected except age.

To aggregate the magnitude estimates and obtain a single value for each image, the geometric mean (to mitigate the bias due to potential outliers) over all 20 observers’ estimates of translucency, gloss, and lightness was calculated for each experiment separately. The categorical labels were converted to numerical values: Right Side of the Object  = 1, Behind the Object on the Right Side  = 2, Behind the Object on the Left Side  = 3, and Left Side of the Object  = 4.

Figure 7(a) shows the visualization of the latent space of Model 1. The model has formed three distinct clusters for the three different shapes in the latent space, indicating that the model has captured the shape difference well. The albedo, IoR, and α values are separated within each shape cluster. In each cluster, the albedo seems to have higher values toward the decreasing Y axis. However, for α and IoR, each cluster has placed the extreme values at slightly different locations. For α, the higher values are placed generally toward the increasing Y axis, but inside the bunny-shape cluster, the high values are placed toward the negative X axis direction while in the cube cluster, they are placed toward the positive X axis direction. This is true for IoR as well. Inside the sphere cluster, the high values of IoR and α are placed toward the increasing Y axis direction. Finally, σT does not have a discernible pattern. It is possible that the variation of the values is more visible in 3D or higher dimensions.

Figure 7(b) shows the visualization of the latent space of Model 2. Since there is only one shape in this dataset, the latent space also has one large cluster. The albedo, α, and light direction have distinct extremes. The IoR has an oscillating pattern where the value varies from high to low in the diagonal direction from the positive Y axis and negative X axis to the negative Y axis and positive X axis (from top left to bottom right). Both models have learned to disentangle the albedo, α, and light direction parameters in 2D. Model 1 has learned to also disentangle the IoR parameter as well as the shape. The latent space of the model captures the variation in intrinsic and extrinsic properties of the dataset and can distinguish between them, indicating that the models have learned the statistical structure of the images inside the training dataset.

It is interesting to understand whether the latent space is linear or non-linear with respect to the optical parameters. For this, data points that lie on a straight line parallel to the principal axes were considered, and their coordinates in the latent space were plotted as a function of optical properties. Additionally, the Pearson correlation coefficient and the distance correlation [39] were calculated.

Figure 8(a) shows the relationship for sphere between the optical parameter values and the X axis values of the data points that lie on a straight line along the X axis inside the latent space of Model 1 (for other shapes, see Figure S7.1. in Supplementary Material). Observing the figure, σT has a non-linear correlation with the X axis in the sphere and cube clusters. The distance correlation is 0.44 for sphere and 0.40 for cube while the R coefficient is −0.14 for sphere and 0.15 for cube. It has a linear correlation in the bunny cluster. The albedo has a linear positive correlation in the cube cluster and a slight negative non-linear relation in the bunny cluster. Moving on, the IoR has a slight positive non-linear correlation in the bunny cluster and a linear one in the cube cluster. Figure 8(b) shows the relationship between the optical parameter values and the Y axis values. It is observed that σT has a strong positive linear correlation and albedo has a strong negative linear correlation inside the sphere cluster. The IoR and α have a negative non-linear relationship with the Y axis inside this cluster. The albedo has a strong linear relationship in the sphere and cube clusters. As in the case of the X axis, here the IoR also has a positive linear correlation in the cube cluster. However, the relationship in the bunny cluster is not indicative of correlation. Moreover, there is a negative non-linear correlation in the sphere. Finally, α is non-linearly related in the sphere and bunny groups. Overall, σT and α have a non-linear rate of change as compared to the X and Y axes. The IoR changes positively and linearly inside the cube cluster on both the X and Y axes of the latent space.

Figure S7.2. in Supplementary Material shows the results for Model 2. The albedo has a positive linear correlation with its X and Y axis locations. Similarly, α also has the same relation. The light direction is inversely correlated with the X axis of the latent space and positively correlated with the Y axis of the latent space. In general, the optical parameters demonstrate mostly linear correlation inside the latent space of Model 2.

Figure 7 illustrates that the latent space disentangles shape, lighting direction, and many optical properties reasonably well. However, as mentioned in Section 1, the correlation between optical properties and perception is not straightforward. We conducted psychophysical experiments to explore how perceived attributes change in the 2D latent space. Figure 9 shows the magnitude estimates of the perceived attributes of translucency, gloss, and lightness for Model 1. The observers have rated almost all of the data points as having lower than 4 translucency (on a 1–10 scale). However, there is a pattern to the location of the values. Higher values tend to be located toward the negative Y axis. This pattern is consistent with the distribution of the albedo. Similarly to translucency, observers have given an overall low rating to gloss. In the bunny cluster, the few high values are grouped at the rightmost edge of the cluster toward the positive X axis. In the sphere cluster, higher values are in the middle, whereas for the cube no data point was rated above 6. There is also a negative correlation between gloss and α. Finally, perceived lightness has a prominent pattern with high values generally toward the negative Y axis direction similar to the albedo, which is logical, since high albedo objects usually appear lighter due to subsurface scattering. The low scores for perceived translucency verify the findings that the model is incapable of synthesizing transparent and close-to-transparent appearance.

The results for Model 2 are given in Supplementary Material S8. As for perceived translucency, the trends are similar here with overall low ratings. Some high values are seen in the bottom left corner of the cluster. The perceived gloss has the same pattern but with more ratings that are higher than 6. The perceived lightness has higher values toward the middle of the cluster and in the positive X axis direction, correlating with the albedo. The perceived light direction has a high correlation with the light direction parameter, indicating that the model has captured the light direction extrinsic parameter well and also that the observers were able to mostly successfully predict the directions. Figures S.9.1. and S.9.2. of the Supplementary Material show similar plots including only the data points that lie on a straight line parallel to the principal axes, which makes the trends easier to spot.

To gain a deeper insight into perceptual navigability of the latent space, latent-space coordinates were plotted as a function of perceived attribute magnitudes and correlations were found among them. A similar study on the correlation between optical and perceptual properties can be found in Section S11 and Figures S.11.1.–S.11.4 of the Supplementary Material. Figure 10(a) shows the relationship between the perceived attribute values for a sphere and the X axis values of the data points that lie on a straight line along the X axis inside the latent space of Model 1 (the results for other shapes are illustrated in Supplementary Material S10). The perceived translucency inside the sphere and cube clusters is the only strong linear correlation while the perceived gloss inside the sphere and cube clusters shows a non-linear correlation. The perceived lightness scatter plots are chaotic and do not exhibit a relation. Figure 10(b) shows the relationship for the Y axis (Supplementary Material S10 for cube and bunny). The perceived lightness has a strong negative correlation with the Y axis of the latent space inside all shape clusters. Similarly to the observation in the X axis, the perceived gloss shows a slight non-linear correlation in the sphere and cube clusters. The perceived translucency shows a reduced correlation strength within the cube group compared to the same relationship with the X axis. In general, the perceived translucency exhibits linear correlations in the X axis. The perceived gloss has a non-linear relation and the perceived lightness has a linear correlation with only the Y axis.

The results for Model 2 are shown in Supplementary Figure S.10.1(g)–(h). The perceived gloss shows a strong negative linear correlation with both axes. The perceived translucency has a non-linear relationship with both axes. There is a positive linear correlation between the perceived lightness and the X axis of the latent space. Finally, the light direction is non-linearly related to the X axis and has a linear relation with the Y axis.

To assess the generalization ability of StyleGAN2-ADA beyond its training distribution, we evaluated 625 unseen parameter combinations by comparing GAN-generated outputs with Mitsuba-rendered ground truths. The results showed that the model was generally capable of producing images with high perceptual and structural similarity for parameters interpolated between seen data as well as for those extrapolated beyond the training range. Quantitative metrics support this: the SSIM achieved a median of 0.875, indicating strong alignment in geometry and fine image structures. The PSNR values were more variable, reflecting differences in brightness and pixel level fidelity, yet remained centered around 18.12 dB, which is within an acceptable range for high-quality synthesis. The LPIPS scores, which assess perceptual similarity using deep features, were consistently low for most images, affirming that the GAN captured perceptually important aspects such as gloss and translucency. A small subset of outliers showed poor alignment, often with low SSIM and high LPIPS, which correspond to extreme parameter combinations that were intentionally chosen to lie outside the training distribution. Importantly, these combinations resulted in physically implausible appearances (e.g., albedo > 1), which resulted in highly noisy Mitsuba renderings. The GAN’s failure to replicate them is consistent with the expectation that the model should not generalize to implausible material configurations. Thus, these misalignments serve not as failures but rather as further validation that the model has learned a grounded, perceptually coherent generative prior. The histogram in Figure 11 confirms this trend, with the majority of examples falling within a high-quality perceptual and structural range.

To evaluate whether StyleGAN2’s latent-space projection retains sufficient structure for interpretable parameter recovery, we attempted to reverse-map 3D PCA-reduced latent vectors back to the original optical parameters (σt, albedo, IoR, α) used in Mitsuba rendering. We expected that 3D better captures the data structure than 2D while still remaining user-friendly and human-intelligible. Initial analysis revealed that the first three PCA components together explained only 25.48% of the variance in the 512-dimensional latent space, indicating substantial information loss. Figure 12 illustrates the cumulative variance curve, showing that as many as 219 components are needed to capture 95% of the variance, highlighting the high-dimensional complexity of the latent space. Regression models trained on the 3D PCA vectors, ranging from linear methods to random forests, achieved limited success (Table II) in predicting optical material properties, with the best model (random forest) yielding an R2 of 0.77 but still suffering from inaccurate predictions. Even neural networks trained on 3D PCA inputs exhibited significant gaps between training and validation performance and consistently collapsed toward predicting midrange parameter values (Table III), confirming underfitting. These issues were partially resolved by increasing the dimensionality: using 220 PCA components dramatically improved performance, reducing mean squared error from 0.34 to 0.013 and mean absolute error from 0.31 to 0.06. However, despite improved metric scores (SSIM  = 0.90, LPIPS  = 0.084), color comparison (Table IV) revealed perceptual mismatches between GAN- and Mitsuba-generated images, particularly in hue and translucency. We hypothesize that the variation in hue in generated images is due to the interpolation of the pixel values. The training dataset consists of colors that are different shades of gray from very dark to very bright images. When images from the trained StyleGAN2-ADA latent space are generated, they can be sampled from a location that is in between a bright image and a dark image, generating an interpolated image in terms of pixel values, leading to the appearance of hues that were not initially present in the dataset. This morphing has also been demonstrated in [33]. The role of the light probe whose mirror reflections are visible in highly glossy images also cannot be ruled out. These findings confirm a fundamental limitation: although StyleGAN2-ADA’s latent space can encode meaningful visual information, heavily reducing its dimensionality, even to enable intuitive PCA navigation, results in the loss of excessive information to accurately reconstruct physical parameters, making such mappings unreliable without retaining a large portion of the original latent structure. Even though 2D PCA disentangles the attributes to an extent sufficient for qualitative exploration, it is too lossy for accurate mapping back to the optical properties. When many physical parameters vary, mapping to a low-dimensional space is non-injective, that is, a set of coordinates in the low-dimensional space may not map to the unique set of optical properties, making reversing impossible.