Appearance is one of the elements that constitutes the unique visual aspect of an object. In recent years, considerable research has been conducted on appearance [1, 2]. A typical example of appearance research is the measurement, analysis, and reproduction of object gloss, which is perceived as a shiny or matte appearance. There is tremendous interest in the perception of gloss. As a result of a number of factors, including advances in technologies such as computer graphics and the development of new experimental techniques and measurements, efforts are underway to analyze the complex interplay between variables such as illumination and surface reflectance properties to determine the perception of gloss by the observer [3–10].

In recent years, opportunities for individuals to take photographs have increased dramatically with the reach of digital content such as on smartphones and social networking services. The higher resolution, brightness, and saturation of on-board cameras and display screens have made it possible to easily reproduce high-quality digital images. On the other hand, captured image data are encoded and compressed before being stored or transmitted. The joint photographic experts group (JPEG) [11] is a representative lossy compression method that has been actively used in various media as an international standard encoding method. Tanaka et al. showed that gloss, one of the appearance features, can be degraded in JPEG image coding [12]. They also showed that even if the color, brightness, and size of an object are faithfully reproduced on a display, the appearance can change depending on the material [13]. In general, appearance can easily change during the reproduction of digital images.

In this paper, the authors focus on the perception of bumpiness of object surfaces. In nature, object surfaces have a certain degree of bumpiness. Bumpiness is an appearance that affects a wide range of products, including metals, leather, and clothing. When the bumpy shape of an object changes, its appearance changes as well. Figure 1 shows an example of how JPEG image compression significantly changes the perceived bumpiness. Thus, because the bumpiness of an image can change, a method to edit the appearance in images is required.

Tanaka et al. proposed “PuRet” as a method to edit the appearance of objects in images [14]. This method focuses on the physiological phenomenon in which the pupil diameter decreases when observing the appearance of a material on the surface of an object, and a tone-mapping algorithm enhances the perceived appearance of the material by contracting the pupil diameter based on a retinal response model. Using this method, it is possible to emphasize the appearance of the material of the entire image, but it is impossible to edit only a specific component of the appearance of the material, such as gloss. An example of a method for editing the specific appearance of an object in an image is “band-sifting” [15], proposed by Boyadzhiev et al.. They demonstrated that sub-bands of spatial frequencies in an image can represent specific appearance information, and they edited this information by increasing or decreasing the signal in the sub-bands. By decomposing the image into frequencies and modulating it based on frequency, amplitude, and sign, it is possible to manipulate not only glossiness but also various other components of appearance. However, this manipulation requires trial and error during the editing process because the physical function is not clearly understood. In a previous study, the authors proposed a method using color space and object reflectance characteristics to edit the glossiness of the object surfaces of various materials in digital images [5]. This method was applied to test images containing objects of various materials. The effectiveness of the method was confirmed by subjective evaluation based on visual inspection and objective evaluation based on the image statistics proposed in Refs. [6–9]. However, because the method considered only glossiness, it could not edit other appearance components, including bumpiness. Ho et al. studied the perception of “roughness” [16, 17], which is a perceptual quality similar to that of bumpiness. They conducted an experiment using computer graphic (CG) images as stimulus images in which the shape and size of the bumps as well as albedo were fixed. The experimental results suggest that roughness is perceived by the proportion of shadows, mean pixel luminance, variation in pixel luminance, and texture contrast, irrespective of illumination or viewpoint angle. Although these studies on the perception of roughness have been conducted through psychological evaluation experiments using CG, the perceived sense of roughness may differ depending on whether the subject is a real object or a CG image because objects in nature have a complex combination of factors such as reflective characteristics, shape, motion, and illumination, affecting the perception of the observer.

The purpose of this study was to propose a method for editing bumpiness using images of real objects made of various materials as subjects. Here, “bumpiness” is defined as the bumpy appearance of an object’s surface. To this end, first, the relationship between the statistics obtained from the images and the sensibility value of the bumpiness perceived by humans was analyzed. Based on the results of these analyses, a bumpiness editing method was proposed and its effectiveness was experimentally verified.

A subjective evaluation experiment was conducted to analyze the factors affecting the perception of bumpiness. In this experiment, bumpiness was defined as the bumpy appearance of an object’s surface, and each observer was asked to rate the bumpiness based on the subjectivity of each observer. In Section 2.1, the experimental method of the subjective evaluation experiment and in Section 2.2, the results of the analysis of the experimental results, are described.

Based on the results of the analysis of bumpiness perception in Section 2.2, bumpiness perception can be modified by modulating the power spectrum in the low- and middle-frequency bands. The bumpy appearance-editing method of Figure 6 was proposed for this purpose.

In processing color images, the proposed method first converts the RGB image into the YCbCr color space, and then decomposes it into the Y component (luminance) and CbCr component (color difference) as follows: (1)YCbCr=0.2990.5870.114−0.16874−0.331260.5000.500−0.41869−0.08131RGB+0128128.

The Y -component is then processed using Fourier transform to obtain the power spectrum, which modulates the intensity in the frequency range of 5–65 cpi based on the results of the regression model constructed in Section 2.2. To modulate the Mean, Std, and Top, the intensity of the power spectrum was amplified to emphasize, and subsequently attenuated to suppress bumpiness. The modulated power spectrum was then processed using inverse Fourier transform to obtain the Y component, which was then combined with CbCr and color difference components to obtain the color image with edited bumpiness. In this study, two methods of modulating the power spectrum were examined. The first method modulated the intensity uniformly within the applicable range of the power spectrum (Method 1) according to Eq. (2), and the degree of modulation was controlled by the scale factor a. (2)L′a,f=a⋅L5≤frequency ≤65[cpi]Lotherwise , where L is the intensity of the power spectrum before modulation; L^′ is the intensity of the power spectrum after modulation; and f is the spatial frequency. Bumpiness is emphasized when the scale factor a is a > 1, whereas it is suppressed when a < 1.

The second modulation method considered contrast sensitivity (Method 2). As shown in Figure 7, the peak of the contrast sensitivity curve overlapped the range of the average difference in the power spectrum intensity. Thus, contrast sensitivity also had an effect on the perception of bumpiness and modulated the intensity of the power spectrum along the contrast sensitivity curve. After modulation, the luminance of the entire image was adjusted to produce an image with modulated bumpiness perception. The contrast sensitivity curve is expressed by Eq. (3) [19],(3)Wf=(K+a⋅fc)⋅e−bfK where f represents the spatial frequency, and the constants a, b, c, and K are a = 75, b = 0.2, c = 0.9, and K = 46. To align the parameters of the scale factor a of Method 1 to the scale of the modulation equation, Method 2 was designed as shown in Eq. (4). (4)L′b,f=Wf−1Wmax−1⋅b−1+15≤f≤65cpiLotherwise. where L and L^′ are the intensities of the power spectrum before and after modulation; W_max is the maximum value of the contrast sensitivity curve; b is the scale factor; and f is the spatial frequency. The degree of modulation of bumpiness was controlled using scale factor b. The bumpiness was enhanced when b > 1 or suppressed when b < 1.

The results of modulation using Eq. (4) are shown in Figure 8. The luminance of the entire image was too high when bumpiness was suppressed and vice versa. To avoid this problem, the luminance components were adjusted to match the average luminance of the edited image to that of the original image using Eq. (5). (5)Vi,j′=Vi,j+1N∑Vsrci,j−1N∑Vi,j, where V _{i, j}^′ is the luminance value of each pixel after luminance adjustment; V _{i, j} is the luminance value of each pixel after editing the bumpiness using Eq. (4); V _{srci, j} is the luminance value of each pixel in the original image; and N is the number of pixels in the image. The results of adjusting the luminance component according to Eq. (5) are shown in Figure 9. The image generated after adjusting the luminance component according to Eq. (5) constitutes the output of Method 2.

In this section, the verification of the effectiveness of the editing method through subjective evaluation is discussed. The images used in the subjective evaluation experiment are presented in Fig. A1. The results of editing using the band-sifting algorithm [14] are also used for comparison.

Figures 10–12 show examples of the results of editing bumpiness using Methods 1 and 2. In all three figures, (a) and (b) display the results suppressed by Method 1; (d) and (e) display the results enhanced by Method 1; (f) and (g) display the results suppressed by Method 2; and (h) and (i) display the results enhanced by Method 2; (j) displays the evaluated values of bumpiness from (a) to (i), and (k) displays the value of unnaturalness evaluated on the images of (a) to (i), and the error bars indicate standard errors. In this experiment, to facilitate the interpretation of bumpiness, the scale factors a and b shown in Eqs. (1) and (3) were set to 1.25 and 1.50, respectively, to enhance the bumpiness and to 0.75 and 0.50, respectively, to suppress it. The subjects were seven students in their 20s, and the experiment was conducted in the same environment used for the subjective evaluation experiment described in Section 2.1. The subjects were asked to evaluate two items: “bumpiness” and “unnaturalness”. Here, unnaturalness is defined as an appearance contrary to nature that does not exist in reality. Subjects were asked to conduct the experiment twice for each item and rate bumpiness and unnaturalness on a 5-point scale, whereby the average of the scores from all subjects was used for the evaluation of bumpiness and unnaturalness of the image. The results of all images and script used in this experiment can be found at the URL shown in the appendix.

As shown in Figs. 10–12, most images used in the experiment demonstrate that the scale factor of Methods 1 and 2 control the sense of bumpiness. In addition, in all edited images, the bumpiness could be edited within an acceptable range, as the unnaturalness rating was never higher than 4. As seen in Figs. 11(k) and 12(k), there are no differences in the unnaturalness results of Methods 1 and 2 applied to bumpiness suppression; however, in the bumpiness enhancement process, there is a large increase in the evaluated value of unnaturalness in Method 2.

Figure 13 shows the results of applying the proposed method to an original image with low bumpiness evaluation value. Methods 1 and 2 control the perception of bumpiness by amplifying components that are considered to affect the perception of bumpiness. Therefore, the proposed method cannot amplify the bumpiness perception of the original image. Thus, an image with original low bumpiness is still perceived as such. The power spectrum of the original image in Figure 14 has strong intensity in the low-frequency band and weak intensity in the middle- and high-frequency bands. It can be confirmed that the suppression process of Method 1 does not work well for such images, and that the suppression process of Method 2 is more effective than that of Method 1. As the peak of the contrast sensitivity curve overlaps the peak of the frequency band with high intensity in the power spectrum, the suppression of bumpiness in the entire image can be effectively suppressed by emphasizing the degree of suppression along the contrast sensitivity rather than by changing the power spectrum uniformly.

Several test images were used to compare the proposed method with band-sifting. Figure 15 shows an example of the application of the proposed method and band-sifting. Bumpiness is emphasized by amplifying the high-frequency high-amplitude negative-sign (HHN) component and the high-frequency low-amplitude negative-sign (HLN) component. Amplifying the HHN and HLN components emphasizes the shadows on the surface of the object and the bumpiness of the surface. As a result, there is no significant difference between Fig. 15(b) edited by Method 1 and Fig. 15(c) edited by band-sifting in terms of the perceived bumpiness and unnaturalness. However, as shown in Figure 16(a), amplifying the HHN and HLN components did not improve the perceived bumpiness of an image formed by the reflection of light (Fig. 16(b)). To enhance the bumpiness of the original image in Fig. 16(a), it is necessary to amplify the HHP and HLP components. The results of editing bumpiness using band-sifting based on the amplification of the HHN and HLN components are not consistent. However, the proposed method can consistently edit bumpiness even when the bumpiness is attributed to shading or light reflection.

In this section, the limitations of the proposed method are discussed. As shown in Fig. 13, the proposed method cannot enhance the bumpiness of an image with a low degree of bumpiness because both Methods 1 and 2 enhance the bumpiness by amplifying the intensity in the range of 5–65 cpi of the power spectrum. Therefore, the proposed method cannot sufficiently enhance the images with low bumpiness. For images with low bumpiness, it is necessary to use the power spectrum. Adding bumpiness would require considering the shape and other factors, making the process difficult.

Figure 17 shows the results of applying the proposed method to an image with large amplitude and size of bumps and low randomness. In this case, the suppression process is effective because the evaluated value of the sense of bumpiness is lower than that of the original image, thus suppressing the sense of bumpiness. However, when bumpiness is enhanced, there is no significant change in the evaluated value of bumpiness compared to the original image. It is also observed that the enhancement process in Method 2 significantly increases the unnaturalness of the image. For an image with a structure, such as that shown in Fig. 17, the suppression process works effectively, but the enhancement process cannot increase the sense of bumpiness. In addition, the enhancement process may increase the unnaturalness, which is a drawback of the proposed method.

In this study, a method for analyzing and editing bumpiness perception is proposed using texture images created by cropping only the surface of an object. Therefore, the proposed method did not perform well on natural images. The results of applying the proposed method to natural images are shown in Figure 18. Looking at the area of the stone wall, it appears that the bumpiness perception was edited according to the scale factor. However, the suppression process caused background buildings and birds to appear blurry, and the area of the sea, which does not have a bumpy shape in Method 1, was edited unnaturally. In addition, the enhancement process in Method 1 also unnaturally edited the sea area with no irregularity. These results indicate that the proposed method is not effective on natural images. This may be due to the fact that the analysis in this study was conducted on texture images, and that the range of statistics and spatial frequencies that have a large impact on the perception of bumpiness in natural images may be different from those in texture images, and also the proposed method was applied to the entire image.

In this study, focusing on the bumpiness of object surfaces in images, a method was proposed to edit the bumpiness of these surfaces. First, to analyze the factors influencing the perception of bumpiness, the subjects were asked to evaluate the bumpiness of 50 texture images. The bumpiness perception obtained from the experimental results of the subjective evaluation experiment, as well as the image statistics obtained from the multiscale images, were analyzed and evaluated, and a regression model was constructed using Mean, Std, and Top. The model showed high accuracy, demonstrating that these statistics had an influence on bumpiness perception. In addition, as a common result, the frequency band from 5–65 cpi had a large influence on the perception of bumpiness. Based on these results, we proposed a method to edit the sense of bumpiness.

In color images, the proposed method first transforms the RGB color space to YCbCr color space. Two modulation methods were proposed for the Fourier transform of the Y component, which is the luminance component, to modulate the Mean, Std, and Top. The first method modulates the intensity of the power spectrum uniformly in the frequency band of 5–65 cpi. The second method modulates the intensity of the power spectrum by varying the degree of modulation along the contrast sensitivity curve. Upon applying inverse Fourier transform to the modulated Y-component back to the RGB color space using the color difference component CbCr, an image is generated with edited bumpiness. Subjective evaluation experiments were conducted on the images to which the proposed method was applied to evaluate the bumpiness and unnaturalness of the images after bumpiness-editing. The effectiveness of the proposed method was confirmed through visual evaluation of many images.

Two modulation methods were designed for the proposed method. In Method 1, both the enhancement and suppression processes edit the bumpiness without significantly increasing the unnaturalness. However, suppression does not work effectively for images wherein the intensity is high in the low-frequency band of the power spectrum and low in the middle- and high-frequency bands. In contrast, in Method 2, the suppression process works effectively for such images. However, Method 2 has the disadvantage of being prone to artifacts.

The proposed method may not be effective for images with low bumpiness or images with structures such as Fig. 17. In addition, when applied to natural images, unnatural images can be generated. This is because the proposed method was designed to analyze texture images, and the results of the analysis reflect this. Because the proposed method is applied to the entire image, it is necessary to make it applicable only to areas where objects and textures exist. Furthermore, the study of roughness is not limited to 2D images, but has been applied to 2.5D prints [20]. According to Kadyrova et al. the attributes that observers look for when evaluating the naturalness of 2.5D prints are color, roughness, gloss, elevation, and lightness. Further development of our research to achieve more natural 2.5D and 3D prints is also a future challenge.