The term intrinsic images is first introduced in the literature by Barrow and Tenenbaum [13] to refer to mid-level components like reflectance and shading. A comprehensive survey of Intrinsic Image Decomposition methods is presented by Bonneel et al. [14] where different methods are categorized based on their assumptions and choice of priors. Here, we focus on methods that employ additional scene information beyond a single RGB image and discuss representative examples of such methods.

Chen et al. [15] use RGB-D to implement surface normal priors. Nonetheless, a depth image does not provide enough information about the scene to perform a full inverse rendering and complexities like cast shadows remain ambiguous. Similarly stereo matching-based methods [16] use disparity to infer some level of geometric information in order to introduce additional constraints and improve the results while maintaining consistency between the two views. Image sequences containing camera or scene motion could be used to further resolve ambiguities [17]. Multi-view stereo-based methods [6, 18] use a 360∘ view of the scene to extract full geometry and environment map.

Light field as a special case of multi-view stereo, provides some level of angular information which could improve decomposition in the presence of view-dependent components such as specularity [3]. In addition, disparity information can be extracted which allows the inclusion of geometry prior. Similar to the case of stereo and RGB-D, geometry cues might not be helpful in resolving complexities like cast shadows. However, multiple instances of the same data might improve the decomposition robustness, e.g., in the presence of noise.

To the best of our knowledge, there are two intrinsic image decomposition methods for light fields in the literature [2, 3]. They both require highly dense light field with small baselines. On the contrary, our method is capable of handling a sparse set of views and a large baseline. In terms of optimization priors, we show that by focusing on reliable and well-crafted priors, we are able to outperform existing methods which often use more complex priors [14]. Our method is capable of handling both dense and sparse light field.

Artusi et al. [19] provide an extensive survey on specularity removal in natural images. While a vast majority of intrinsic image decomposition methods ignore specularities [2, 16, 17, 20], some successful attempts have also been reported [3, 21, 22]. However, these methods focus on narrow-baseline, dense light fields, and are not directly applicable to sparse light fields. Also, while aiming at physically correct specularity extraction, many failure cases are reported for such methods [23] that might lead to visually disturbing artifacts, e.g, for large area highlights. In our application the extraction of specularity layer can lead to many interesting appearance editing effects. We resort to more approximate solutions that do not lead to obvious visual artifacts in our appearance editing at the expense of physical correctness. A detailed review of gloss perception is done in a recent study by Chadwick et al. [24]. We take inspiration from [8] and [10] to extract a psuedo specular/highlight layer from a given image.

Multi-scale edge-preserving image decompositions have been used to enhance image detail or to achieve other image appearance changes or transfers [25–29]. We also perform a multi-scale image decomposition, but we apply it to intrinsic image layers, which gives us a better control over the wide range of effects we can produce. Recently, deep/machine learning methods have been successfully used for overall image appearance changes and various stylizations [30], but spatially selective, continuous range, and large scope material editing that is intuitive to the user has not been demonstrated so far.

Light-field manipulations [31] focused mostly on retargeting [32], shape deformation [33], in-painting and recolorization [34], compositing [35], and morphing [36]. However, research on light-field appearance editing is relatively sparse [37, 38], and the key problem that is considered there is intuitive propagation of sparse user edits to all views [39, 40].

Material appearance editing based on a single image has widely been investigated [4, 5, 41–45]. In all these cases, an attempt of inverse rendering has been performed. Since the problem of reconstructing all needed data from a single image is strongly under-constrained [46], manual intervention might be required to reconstruct the missing scene lighting or geometry information [5, 44].

Gryaditskaya et al. [23] recover such information from light fields, which is then employed in a spatio-angular filter that enhances the roughness of glossy objects. To our knowledge, this is the only work where material appearance editing has been investigated for light fields, and while the full inverse rendering has not been performed, the filtering quality relies on the accuracy of normal vector reconstruction. The scope of roughness manipulation is limited and even simple roughness reduction has not been shown in this framework.

Many successful editing examples have been presented in the discussed work, which was greatly facilitated by the limited sensitivity of visual system for even substantial departures from physical correctness [5, 7, 8, 47]. However, the 3D scene data, as required by these techniques, is often reconstructed with low precision, which might lead to visible artifacts, or at best reduce the generality of proposed manipulations. In this work, we rely on a much simpler, purely 2D approach, and we take an inspiration from the work of Boyadzhiev et al. [12] who introduce the band-sifting operations.

The relationship between image statistics and reflectance perception is discussed in [9, 10]. In [8], the perception of glossiness is explored using visual cues. Band sifting is a simple and effective technique for image-based material editing based on above findings. The basic idea is to do a multi-scale decomposition of luminance channel into a set of subbands. And then selectively manipulate these subbands based on frequency, amplitude, and sign. The manipulation is pretty simple where the selected part of subband is either boosted (multiplied by a factor greater than 1) or reduced (multiplied by a factor between 0 and 1). The above relationship between 2D frequency bands and material perception is discussed in [11].

We are different from [12] in the way that we introduce a new invert operation in addition to the previous reduce and boost operations. By applying operations individually on intrinsic layers, we are able to achieve new kinds of material editing effects. Moreover, the range of these manipulations is also increased.

Intrinsic images/videos have been used for material editing [17, 48], but only recoloring and tone mapping curve manipulations have been demonstrated, respectively, for the reflectance and shading layers. In this work, we additionally consider the specular layer, and a wider scope of material manipulations.

Real-world scenes are often illuminated by non-white light sources and contain glossy surfaces with specular highlights, which is typically ignored by existing intrinsic decomposition methods [2, 16, 17, 20]. In this work, we introduce a pre-processing step to correct illumination color and extract specularity. This allows us to simplify intrinsic image decomposition to the generic problem, where it is assumed that Lambertian surfaces are lit by achromatic light. In Figure 2, we show our decomposition results with and without this pre-processing step for a better and fairer comparison to the above mentioned methods.

We estimate the illumination chroma using the general gray-world approach [50], which is based on the assumption that “the pth-Minkowski norm of a scene is achromatic after local smoothing.” We based our illumination estimation on Eq. (17) in [50], and we found that p = 3 performed well for our goals. To white balance each image, we divide each pixel color by the estimation of normalized illumination color (preserving its brightness).

As our approximate specular layer extraction is based on band-sifting operations, we provide all relevant details in Section 5.1 after band sifting is properly introduced.

Current light-field decomposition methods use epipolar images to derive the intrinsic layers [2, 3]. This requires narrow-baseline and dense light field with a known structure. Our method, on the other hand, is designed for sparse light fields and wide baselines where such techniques are not applicable. Therefore, we regard light field as a generalization of stereoscopic images, and we refer to methods that consider stereo pairs [16, 51].

Bonneel et al. [14] states that many complex intrinsic image methods are not better than the baseline for image editing, and some priors may be even harmful. Please note that the baseline correspond to simply assigning chroma to reflectance and brightness to shading. Having an overcomplex optimization with several weak (ill-posed) energy terms does not always provide a benefit. Instead, our intrinsic image optimization is tailored particularly to appearance editing needs.

Since white-balancing operation is performed in the pre-processing step (Section 4.1), we assume grayscale shading. The main goal in intrinsic image decomposition is decomposing pixel intensity into reflectance and shading. Many works in the literature [2, 16, 52] choose to solve for shading, and then compute reflectance using, r = id − s. We found that often existing light-field datasets contain some inconsistencies regarding image brightness due to e.g., flickering light. By definition, reflectance is invariant to illumination. Therefore, we formulate our optimization by solving for reflectance instead of shading. This further simplifies formulation of our optimization: where Er, Ed, and Ea are retinex, disparity, and absolute shading scale terms, respectively with their corresponding weights, and λs is a regularization parameter assuring reflectance sparsity.

We use fixed weights λr = 2, λd = 1, λa = 0.7, and λs = 0.1 for all the results presented in this article. In the supplementary materials, we provide an analysis of these parameter weights along with additional results. Below, we provide an explanation of each term.

In case of a dense light field, the first step is to do a sparse sampling. The intrinsic decomposition of the sparse samples is performed using the method described in Section 4.2. The sparse reflectance is then propagated among all the views to get a dense reflectance output. Let us consider that we have sparse reflectance for views at position a and c, given by Ra and Rc respectively.

The reflectance at intermediate position b, say Rb, is obtained by finding a minimizer for the energy functional in Eq. (10).

Please note that ζmn was used to avoid smoothing of reflectance at reflectance edges (refer Eqs (3) and (5)). In Eq. (10) we use 1 − ζmn to impose gradient domain constraint only at reflectance edges. In Section 4.2 we discuss how these edges are obtained. The weights wa and wb represent the quality of image mapping. Tab is a warp operator that maps view Ia to Ib and Tcb is a warp operator that maps view Ic to Ib. We use the same warp operators to map reflectance Ra and Rc. The energy formulation is based on the idea introduced in [58] and further explored in [59].

However, we have modified this energy for our specific purpose. The first modification is introduced in the form of a weight, 1 − ζmn for the first energy term. By introducing such a weight, reflectance is enhanced while it is being propagated to dense views. The second modification is making use of two neighboring views, in the angular domain, and their sparse reflectance in the energy formulation. In [59], the authors consider only one previous view in temporal or angular domain. By making use of two nearest neighbors we ensure better consistency among views and also faster convergence for energy minimization. A detailed discussion of the sparse reflectance propagation is presented in the supplementary material.

In [10], the authors have established relationship between image statistics and the perception of lightness and gloss. Perceptual experiments indicate that by modifying the skewness of subbands of luminance histogram, perception of gloss can be altered. In [12], the authors observe that positive subband coefficients correspond to bright features like highlights, and the negative values represent features like crevices and holes. We use this observation to first identify and then extract these bright regions using sifting-based operations.

The positive component of the subbands of the image luminance channel corresponds to its bright regions. We first sift the positive components for the complete image (Figure 6(a)) to identify the bright regions, using sifting operation: L(AAP, κl), where κl > 1. We then look for all the pixels which got modified due to this operation to obtain a binary mask (Fig. 6(b)). In the next step we use invert sifting operation only in the masked region: L(AAP, κl), where κl < 0. By performing such a step we reduce the highlights present in the image. The difference between the original image and the image with reduced highlights Fig. 6(d) is the highlight or the pseudo-specular layer (Fig. 6(c)).

Here, we explore various operations and their effects. We illustrate appearance changes due to editing of shading, reflectance, and specularity layers, as well as their combinations.

Intrinsic Shading Layer: of an image contains information on object geometry and illumination. By applying sifting operations on this layer we can enhance or suppress the object shape and geometry details. Here we mostly focus on presenting the outcome of boost operation (refer to Section 5). The reduce operation typically leads to the opposite effect. By boosting the high-frequency, low-amplitude coefficients in the shading layer, we can enhance the fine-level surface details, such as wrinkles and bumps (Fig. 4(b)).

Intrinsic Reflectance Layer: of an image contains the color and texture information. Similarly to shading, we can apply band-sifting like operations on this layer to enhance color details. By boosting the low-frequency coefficients in the reflectance layer we can make the scene look more vivid (Fig. 4(c)). Texture colors can be made more pronounced by boosting high-frequency coefficients (Fig. 4(f)).

Intrinsic Specularity Layer: Humans make use of shine and gloss information to classify objects into different categories. By making an object look more shiny, one can make the material appear more plastic or metal like. By boosting all coefficients in the specularity layer we can increase the overall specularity of the objects, thereby making it look less diffuse and more metallic (Fig. 4(d)).

Multiple Intrinsic Layers: By sifting the combination of intrinsic layers we can achieve other interesting appearance edits such as wet paint (Fig. 4(f)), wet-oily/metallic look (Fig. 5(a)), and weathering (Fig. 5). The weathering effect can be further enhanced by desaturating the chroma channels of reflectance layer (see Figure 7 second row).

Please note that some effects mentioned till now can also be partly achieved by simply sifting the luminance channel of an image [12]. However, in the latter case only moderate boost or reduce factors are typically applicable due to possible unnatural look or even explicit artifacts (Fig. 5). The obvious reason for such artifacts is poorer selectivity of edited signal in the luminance channel. As shown in Fig. 5(a) and (c), sifting shading and specularity is more robust than sifting the entire luminance with similar factor. Similarly, more convincing weathered look can be achieved by boosting shading, and reducing reflectance and specularity, which is not possible when a single luminance channel is edited (see Fig. 5(d), Figure 8). Moreover, all our sifting examples are performed on pixel intensity channel given by, In=r2+g2+b2, where r, g, and b are the color channels, instead of luminance L = 0.2126r + 0.7152g + 0.0722b. We observed that by using In instead of luminance L, sifting becomes more robust against color artifacts. In the supplementary material, we provide additional results showing how the range of edits is increased in our case as compared to [12].

By making use of multiple intrinsic layers we produce new appearance effects which were not possible by simple 2D image filtering to the best of our knowledge. Please note that attempts have been made to achieve such appearance editing using only images [5, 62]. However, that involved partial to full inverse rendering.

Translucency: is one of the important effects in this category. As proposed by [7], inverting the high-frequency components of shading layer makes an object appear more translucent. Inspired by their work, we propose techniques which add up to produce a realistic translucent look. By inverting the high-frequency coefficients of both shading and reflectance layers we achieve a certain degree of translucency. We can enhance this effect further by smoothing and desaturating the chroma channels of the reflectance layer. In order to make translucency effect look more realistic we take inspiration from the work of [5], and in-paint the background within the object boundaries (Figures 9 and 10). By refraining from such in-painting and using a more moderate boosting coefficients we can achieve a pearl-like look (Fig. 4(e)). In [63] the authors have discussed how to reconstruct the background, behind occlusions, using synthetic aperture refocusing. The synthetic aperture technique can also be used for moderately sparse light fields. In [64] the authors get rid of reflections and occlusions to separate an image into occlusions/reflections and a clear background. For our transparent appearance edit, where the background is typically distorted due to refraction and blurred, a rough approximation of background is enough. We believe that such ideas for background reconstruction can greatly improve the realism of our transparent edits. We leave detailed investigations in this regard as future work.

Wetness: appearance edit is inspired from a recent work of Shimano et al. [66], where the authors conclude two fundamental characteristics of a wet surface appearance: darkening and spectral sharpening. We perform darkening on reflectance layer by reducing its intensity, spectral sharpening is achieved by increasing the saturation of chroma channels of reflectance layer. We further enhance this effect by amplifying the fine shape details of an object by manipulating the shading layer (Figure 11).

Depth-guided Selective Filtering: In case of light field, we can create depth maps for a given scene using off-the-shelf depth estimation techniques. We can then make use of these depth maps to selectively target objects at different depths for all the effects mentioned previously e.g., by modulation of editing magnitude (effectively, the coefficient κ in Eq. 11) as a function of depth as demonstrated in Figure 12. Please note that the appearance editing is more pronounced for the objects in front.

Since intrinsic image decomposition is a well-established research problem on its own, in this section we evaluate our decomposition approach as presented in Section 4 with respect to the state-of-the-art methods for single images [20], RGB-D [15], video [17], stereo-image pairs [16, 51], and light fields [2, 3].

Note that we do not compare against multi-view methods like [71, 72] since they use a 360∘ coverage of the object, which require that either the environment map or the true geometry is given, and each of their scenes contains a single object. We further exclude works like [73] since they require a single known light source, single object, and user interaction.

Figs. 2 and 3 compare our results with respect to [2, 15–17, 20], where typically we produce more consistent and shading-free reflectance for sparsely sampled light-field data. For example in Fig. 3, other methods label reflectance edges as shading specially on the eyes and numbers. Note that our results are not only consistent among views, but also within each view and do not exhibit patch-like artifacts. Recently, Bonneel et al. [58] showed promising results on improving consistency across views for the reflectance layer as produced by existing methods. Yet, achieving consistency within an image had remained a challenging task, which we address using our optimization scheme. We further show in the supplementary material and video results that our light-field decomposition is more consistent than simply applying Bonneel et al. consistency scheme to the per-view intrinsic image decomposition. Figure 13 shows that unlike [51], in our results the missing correspondence between views due to occlusions does not result in artifacts, and we recover consistent reflectance.

Figure 14 compares our method with [3]. Both [2] and [3] leverage the dense light-field structure and rely on its small baseline, and hence would not handle sparse light fields (10–20 cm baselines). Nevertheless, we compare our performance with these methods by applying our intrinsic image decomposition on a sparse subsample of dense light-field data.

Here we evaluate the performance of our appearance editing framework using intrinsic image layers estimated by our method. Fig. 9 presents our Translucency operation on stereo images of [69] (15 cm baseline). We compare the results when using intrinsic layers estimated by our method versus the ground truth. While our method has misclassified some of the reflectance edges as shading edges, the perceptual quality of the final result has not particularly suffered. Furthermore, Fig. 7 presents results of applying our full framework using intrinsic layers estimated by our method on three different scenes.

In this section, we discuss the performance of our complete pipeline for an example 1D (only horizontal parallax) light field composed of 101 × 1 views of spatial resolution 960 × 720 (Fig. 2 Reading Room).

The interactive interface allows a user to play with different parameters for a single image. Please note that many of the high-level appearance edits (such as “old” look shown in Fig. 8) involve combinations of parameters that remain in certain relation while performing manipulations. To simplify the editing task we provide a single slider to perform such aggregated parameter changes as demonstrated in the supplemental video. The intuitive interface enables us to quickly find the parameter values for the desired appearance which can later be applied to all the views. Due to our consistent intrinsic layer decomposition, regularity is maintained between views for different appearance modifications.

We could further simplify the user navigation over the parameter choices by developing a collaborative editing system [74] that registers parameter configurations that are often selected by the users. Alternatively, crowd sourcing can be used to learn such meaningful parameter configurations [75]. We could then enable sampling from such parameter distributions to obtain meaningful appearance variations, possibly supported by image-gallery-style user interfaces [76]. We relegate such interaction scenarios as future work.

While our translucent appearance effect performs well on the object itself and produces the desired result, cast shadows and reflections of the object on its surrounding also need to be accordingly corrected. Figure 15 demonstrates a failure case example where the object has taken a translucent appearance that no longer matches its reflection.

Finally, we believe improving specular layer extraction and using a multi-illuminant illumination estimation method to be good avenues for future work.