According to [5], the perception of complex visual patterns and objects appears from neural activity as it is transformed through a cascade of areas in the cerebral cortex. Neurons in the primary visual cortex (V1) are selective for local orientation and spatial scale of visual input [11–13]. Downstream regions contain neurons selective for more complex attributes, which is approximately achieved by assembling particular combinations of their upstream afferents. Considering the ubiquity of orientation selectivity in primary visual cortex [14], it is intuitive to make the assumption that its computational purpose is to represent the local orientation of edges.

Furthermore, over the past decades, the mainstream view in both the biological and computational vision communities is that later stages of processing should somehow combine these local edge elements to construct more extensive contours, eventually leading to shapes, forms, and objects [15]. Until recently, most researches on object recognition were built around this paradigm, as well as much of the study of mid-level pattern perception, and physiological measurements in areas V2 and V4 of the ventral stream.

It can thus be concluded that local edges and contours, which are local structural information in images, are vital foundation for the following process of higher level semantic structure understanding of images. Edges and contours features are important elements that reveal the structure information of an image. Therefore, in the proposed model, a contour based estimator as well as a histogram of oriented gradient based estimator are designed to quantify the amount of structural change due to disruption. Details of these estimators will be given in the following sections.

Subjective test done in [9] about human material perception provides us with important clues about how to extract structural features and texture features separately. In [9], the subjective experiment is conducted in order to understand which features are useful for the recognition of material categories. In their experiments, images emphasizing local surface information and global structure information were generated separately with bilateral filtering, which is usually used as a non-linear, edge-preserving and noise-reducing smoothing filter for images.

More specifically, Sharan et al. followed the idea of Bae et al. [16] to extract the micro-structure of the surface by smoothing an image with bilateral filtering. Afterward, they utilized the residual image for further texture analysis. The residual image was obtained by subtracting the bilateral filtered results from the gray-scale versions of the original images to emphasize details of surface structure, which is an operation similar to high-pass filtering. In their subjective test, observers were asked to categorize those distorted images into ten material categories. Based on their results, they concluded that texture is the important attribute of material appearance, while information about surface micro-structure is often related to certain categories, i.e., higher level semantics.

Based on their conclusion, in this paper, bilateral filtering is used as a proxy to separate structure and texture information. In the proposed model, structure features related to shape are extracted from the response of bilateral filtering while texture features were extracted from the residual.

As discussed in the second section that local contours reveal the structure of an image, in this section, a contour based estimator is thus introduced. It has been confirmed that fragments of contours can be used to successfully understand semantics in images [19–21], which further showcases the importance of structure information in semantics related tasks. Since contours are important for global structure understanding, the NICE estimator is improved by using bilateral filter to emphasize important structural local elements. First of all, the edge maps are generated only on the responses of bilateral filtering using the Canny edge detector. For the reference and the degraded images, the obtained contour maps are then denoted as CBI and CBI ̂ correspondingly.

Before calculating the distance, to probe and expand the shapes contained in the image, the contour maps are subjected to morphological dilation with a 3 × 3 “plus-sign” shaped structuring element E. In line with the one-scale NICE estimator, the object score was computed by comparing the binary contours map of the reference and the test images. Afterward, the final contour error map is obtained by exerting point-wise exclusive-or (XOR) operation of the dilated binary images, since XOR is the commonly used operation for contours maps comparison. In the end, the overall BI-NICE score for a test image is defined as where NCBI is the number of contours elements, dH(X, Y ) denotes the Hamming distance between the X and Y , and CBI ⊗ E denotes the dilation operation of the contour map CBI with the morphological structuring element E.

Considering the fact that HOG [22] is a powerful shape related descriptor used in computer vision and image processing for the purpose of object detection, action recognition and so on, we extract HOG features from each response of bilateral filtering as a higher level structure feature. First, each image is divided into 8 × 8 cells/blocks. After calculating the histogram of each cell, spatial pooling strategy based on visual importance proposed in [20] is utilized to pool the dissimilarity values. This pooling strategy was presented based on the perception study that humans tend to perceive “poor” regions in an image more severely than the “good” ones.

Finally, the shape related estimator named bilateral HOG estimator (BI-HOG) is then defined as where H-HOGijR and H-HOGijD denote the histograms correspond to the cells at the ith row and jth column of the bilateral response of both the reference and distorted images. Bp is the lowest 60% of cells ranked by the dissimilarity values. De(X, Y ) denotes the euclidean distance between the two vectors X and Y .

To represent detail information in images, texture related features are considered in this section. Different from [9], instead of extracting micro-jet and micro-SIFT features, the LRI [23] texture descriptor is extracted in this paper with a size limit of K = 4 and a threshold T equaling to the standard deviation of the image divided by 2. Similar to BI-HOG, LRI texture descriptor is extracted based on 8 × 8 cells/blocks. After extracting the LRI descriptors from the residual of the bilateral filtering from both of the reference and degraded images, the texture based estimator named bilateral LRI estimator (BI-LRI) is then defined as where H-LRIijR and H-LRIijD denote the LRI feature histograms correspond to the cell at the ith row and jth column of the bilateral residual of both the reference and distorted images.

As discussed in the second section of this paper, different information plays different roles in different application scenarios. Therefore, we combine the three proposed estimators so that the weights of which can be tuned as parameters according to certain applications. The output of each estimator, which is the dissimilarity value calculated based on different features, is normalized to a range of [0,1]. Finally, the proposed BF-M model, which can also be utilized as a tasks-based parametric image metric, is designed as where α, β, γ are the aforementioned weights for fine-tuning the roles of the contour, shape, and texture based estimators, respectively and α + β + γ = 1. The configurations of these three estimators are set differently according to the specific task in our experiments and will be further discussed for the purpose of investigating the functionality of different information in images in the following section.

In utility assessment task, human observers estimate the usefulness of a natural image as a substitute for a reference. In such a task, structure information is important since the main purpose is to quantify the amount of useful information from an image instead of evaluating its quality. For example, as long as the license plate numbers of vehicles are captured by the surveillance camera, the image is useful for tracking them. More interestingly, according to what have been analyzed in [24] based on the results obtained on the CU-Nantes database, there is a linear relationship between perceived quality score and perceived utility score for images with quality scores under 30, while the one is non-linear for those whose quality scores are higher. It was concluded that observers evaluate very low quality images in terms of the ability to interpret the content. About why the relationship between them is non-linear for higher quality images, one possible explanation could be that texture information plays different role in task of utility and quality assessment for higher quality images. Because the higher the quality, the more details will be maintained. Disruption of texture, e.g., blurriness, is annoying for human observers while judging the quality of the image. For example, in Fig. 1, the image in the last row is one degraded image while the one in the second last row is its reference image. It can be observed from the last columns of the two rows (i.e., the residual of the correspondence images) that there are more texture information in the residual of the reference image than the one of the degraded images. For the task of quality assessment of high quality images, details are important but may not be the same case in utility assessment. Therefore, we also hypothesize that the roles of structure and texture information in the task of utility assessment vary with the quality.

To affirm the assumption that (1) structure information plays the main role in this task, (2) the role of texture and structure differs in different quality ranges, the proposed BF-M model is utilized as the utility estimator and is tested on the CU-Nantes database [24]. The CU-Nantes database consists of 9 reference gray-scale images and 235 distorted images. Each image was degraded by one of the five processes including JPEG compression, blocking, JPEG2000 with dynamic contrast-based quantization, texture smoothing (TS) and texture smoothing with high-pass filtering. To further check how the weights of different information vary with quality, one best configuration will be selected for each sub-interval divided according to the perceived quality score, i.e., the mean opinion score (MOS). To confirm the feasibility of using the proposed model as estimator, ReDLOG [30], most apparent distortion (MAD) [31] metric, multi-scale SSIM (MS-SSIM) [32], the visual information fidelity criterion (VIF) [33], the contours based image evaluation (NICE) [2] metric, the multi-scale version of NICE (MS-NICE) and the multi-scale difference of Gaussian utility (MS-DGU) [34] metrics are chosen to be the compared metrics for utility prediction performance evaluation.

In the experiment, since each sample in the database is labeled not only with the utility score but also with the quality score ranging from 1 to 5, we divide the whole range into quarters and optimize one configuration for each sub-range. Table I illustrates the correlation between objective and subjective scores in different quality interval along with the relative weights configuration. As it can be observed from Table I, for images locate in the quality range of [1,3], the proposed model performs the best with a configuration of α = 0.9, β = 0.1, γ = 0, while for higher quality images which locate in the range of [3,5], the model performs better with a higher weight for the texture estimator. Overall speaking, it can be concluded that structure plays a vital role in utility assessment, especially for lower quality images. Furthermore, it is also obvious that texture also plays certain role in evaluating the utility of higher quality images. It has been verified that the role of structure and texture information is different among different quality ranges in utility evaluation task.

For performance evaluation, best weights are selected for the three basic estimators for images with different quality according to Table I. The overall performance of the metrics is concluded in Table II. Among the compared metrics, the proposed BF-M performs the best. It is proven that the proposed model is qualified for the task of utility assessment.

To better understand why structure related information is more vital in the case of utility assessment, the edge maps and the extracted HOG descriptors are visualized in the second and third rows of Figure 3. In the figure, the first column is the reference image while the other two are the degraded image and the one in the second column has higher utility score than the third one (10.528 > −47.638). By only observing the edges and HOG maps, one can observe that the shapes of the “pumpkin lanterns” on the floor in the first and second columns are recognizable while the ones in the third column are not. It can be thus concluded that, for low quality distorted images, where most of the texture information is lost, structure is the most important information for judging its utility.

Texture synthesis is a broadly and commonly used technique for bit-rate saving in image, video compression, in-painting (e.g., used for error concealment or disoccluded regions filling for view synthesis in FTV system) and so on. The purpose of quality assessment for texture synthesized images is to estimate the perceived quality of the synthesized texture referring to the original texture in images. Therefore, the role of texture information is definitely more important than one of the structure information in such case.

In verifying what has been discussed above, we test the proposed BF-M model on the SynTEX Granularity database [25–27]. This database contains 21 reference textures and 105 synthesized texture images generated with five different texture synthesis algorithms. For BF-M, relatives parameters of the model are set as described in the previous section to obtain the most correlated objective scores with the MOS. According to [35], CWSSIM [36], WCWSSIM [37], parametric metric that proposed in [38] and STQA [35] are the four most promising metrics on the SynTEX Granularity database for evaluating the quality of synthesized texture. Therefore, the performance of the proposed model used as an estimator of perceived quality for texture synthesized images is tested on the same database and compared to these four methods.

During the experiment, when α = 0.2, β = 0.2, γ = 0.6 the proposed model gets the most consistent objective quality score with the subjective one. Since the weights for the texture estimator (i.e., BI-LRI) account for the greatest proportion, we can then draw the conclusion that texture is more important than structure in the task of quality assessment for texture synthesis. In addition, the overall performance of the model applied as quality estimator for texture synthesized images is concluded in Table III. Although BF-M does not outperform STQA, the performance is still comparable to the others. This result proves the feasibility of using the proposed model as a quality estimator for texture synthesized images.

For further interpreting why texture information is most important for quality assessment of synthesis texture, we visualized the edges, HOG, LRI maps and the error map between the LRI maps of the reference and the synthesis texture images in the second to fourth columns in Figure 4. In the figure, the first row corresponds to the reference image while the second and third rows correspond to synthesis texture images and the second one has higher perceived quality score (4.647 > 1.235). For better observation, when generating the visualized LRI maps, we select a slightly larger block size 16 × 16 and crop only the top left part of the image. In the visualized LRI map, each sub-figure is an LRI histogram representing the texture information of the local block. LRI is a statistical texture feature that considers inter-edge distance distribution along different angles, i.e., eight directions by comparing the current pixel value to the closest edge pixel value along each direction. The magnitude of each bin in the histogram is decided by the pixel number between the current pixel and the closest edge pixel along the direction and the sign of the bin is decided by comparing the two pixels’ value. Therefore, the more saturated the histogram, the smoother the block. By comparing the edges, HOG maps of the synthesis texture images with the ones of the original image, it is almost impossible for human observers to tell the difference between them, let alone asking them to judge which one is better synthesized. On the contrary, the LRI map can provide more clues about the statistical difference between the texture in images. By comparing the error maps calculated using Euclidean distance between the LRI histograms of the original image and the synthesized one, it can be observed that the overall error in the third row is larger than the one in the second row, which is consistent to the perceived quality score (bins in the error map larger than 1.2 are labeled with red color). It can be concluded from this sub-section that texture information is more important when task involves mainly fine-granular texture in the images. Since there is no clear main structure (e.g., boundaries of objects) in these images, details of these images (i.e., the texture) are the dominant factor in the task.

Depth-Image-Based-Rendering (DIBR) techniques are indispensable for three-dimensional (3D) video applications including 3D Television (3DTV) and free-viewpoint video. Views that synthesized with DIBR based techniques contain specific distortions like object shifting, incorrect rendering, flickering, blurriness and geometry distortion around disoccluded regions. Since HVS is more sensitive to local severe disruptions than the global consistent ones [39], we hypothesize that structure information plays greater role than texture information during the process of assessing the quality of synthesized views.

Targeting at evaluating the synthesized images’ quality properly, several metrics have been proposed to improve common used metrics. In [40], VSQA was proposed to improve SSIM with three visibility maps which help in characterizing complexity of the images. Federica et al. [41] presented the 3DswIM on the basis of statistical features of wavelet sub-bands. Considering the fact that multi-resolution image quality assessment approaches perform better than the single-resolution ones, Dragana [42] first deployed morphological wavelet decomposition for quality assessment of synthesized images named Morphological Wavelet Peak Signal-to-Noise Ratio metric (MW-PSNR). Later, they devised PSNR with morphological pyramids decomposition (MP-PSNR) instead of morphological wavelet decomposition to obtain better performances [43]. However, none of the aforementioned researches has verified the importance of structure information.

To verify our hypothesis, the proposed model is applied as a quality estimator and is tested on the IRCCyN/IVC-DIBR images database [28, 29]. Images from this database were generated from three multi-view video plus depth sequences including Book Arrival (1024 × 768, 16 cameras with 6.5 cm spacing), Lovebird1 (1024 × 768, 12 cameras with 3.5 cm spacing) and Newspaper (1024 × 768, 9 cameras with 5 cm spacing). Seven DIBR algorithms labelled as A1–A7 [44–49] processed the three sequences to generate four new virtual views for each of them. The database is composed of 84 synthesized views and 12 original frames extracted from the corresponding sequence along with subjective score in the form of MOS. The difference mean opinion score (DMOS) is then calculated measuring the subjective difference between the reference and synthesized images. In [42, 43], images synthesized with A1 are excluded from the experiment due to the significant shifting artifacts compared to the others. However, according to the MOS, images synthesized with A1 have better quality compared to the others, and thus are more similar to advanced synthesized algorithms. Since the main purpose of developing a quality metric is to evaluate the performance of synthesis algorithms, the tested database should be in line with the images/videos synthesized with the state-of-the-art synthesis algorithms to follow the trend. Based on the previous discussion, in our experiments, we include the image set generated by A1 and check the performance on the full IRCCyn/IVC DIBR database. As claimed in [42, 43, 50], MP-PSNR MW-PSNR performed the best among the state-of-the-art metrics designed for synthesized views. According to Dragana et al. [50], PSNR is more consistent with human judgment when calculated at higher morphological decomposition scales. They thus proposed a reduced version of the morphological multi-scale measures, which are reduced MP-PSNR, and reduced MW-PSNR correspondingly, by using only detail images from higher decomposition scales. The reduced versions outperform the full ones. Therefore, in this section, we mainly compare our proposed model with MW-PSNRfull, MP-PSNRfull MW-PSNRreduced and MP-PSNRreduced. To obtain the best performance of them, a 5 × 5 size of SE is used for MP-PSNR and a min Haar wavelet decomposition is used for MW-PSNR as reported in [50].

The overall performance of the metrics is concluded in Table IV. In the experiment, by setting α, β, γ to 0.5, 0.2 and 0.3, the performance of the our model peaks. This configuration indicates the fact that both structure and texture information play roles in evaluating the quality of synthesized views, and one of the structures is greater than the other. In another word, artifact that interferes the structure of the view is more annoying for HVS, which verifies our previous assumption. Moreover, according to Table IV, the proposed BF-M achieves 0.6980 value of PCC, which outperforms all of the compared metrics designed for synthesis images. Compared to the second best performing MP-PSNRredeuce our proposed model obtains a gain of 0.0247 in PCC, which verifies its capability of assessing the perceived quality of synthesized views.

In order to understand how different information loss affects the perceived quality in the scenario of quality evaluation for synthesized views, the edges, HOG and LRI maps of the synthesized image in the third row of Fig. 1 and one of its original images are made visible in Figure 5. By only comparing the edges and HOG map, one can easily notice the geometric distortion around the face of the girl, specially at the right part where the entire regions are blurred. Therefore, it is obvious that structure related information is more important in this task since the deformation of objects’ shapes caused by synthesized algorithms is more eye catchy and can be well captured by structure related descriptors. More interestingly, by comparing the right part of the two LRI maps, one can easily notice the big differences of the histograms in that part. Due to the blurriness introduced by the DIBR algorithms, texture information has been modified, and start become annoying. That is why BI-LRI accounts for 20% in this task.

In summary, the optimized configurations of BI-NICE, BI-HOG, and BI-LRI in tasks of utility assessment, quality assessment for synthesized texture and views are concluded in Figure 6. The setting of the weights is selected according to the performance of the proposed model tested on the Cu-Nantes, SynTEX, and IRCCyN-DIBR database correspondingly. According to the optimized setting, two main conclusions can be made:

Although our model manages to quantify the structural/textural errors for certain cases in different tasks, there are still some cases that our model fails to handle. Examples are shown in Fig. 7: (1) For quality assessment of synthesized views, since BI-LRI and BI-HOG are calculated at block level, the global shifting artifact has been over-penalized to some extend. For example, in the fist row of Fig. 7, the three images are the reference, synthesized image and the error map (the darker the color the more errors there are) between them correspondingly. There is an obvious shift of the object but is not noticeable by only checking the synthesized image. It is obvious that this shifting artifact is durable for human observers compared to the severe local geometric distortions, but it is over-penalized by our model. This can be improved by block matching before dissimilarity calculation; (2) For quality assessment of synthesis texture image, our proposed BI-LRI estimator is not rotation invariant enough, meaning that it is not invariant enough for acceptable rotation of textures. For example, in the second row of Fig. 7, where the first column is the original image and the rest are two synthesized texture images with slightly different subjective quality score. The characteristic of this “goldWeave” is that it consists of a set of regular small geometric figures. Slight rotation of the geometric figure does not affect significantly the overall perceived quality, but will be over-penalized by our model. In this paper, the LRI descriptor is chosen mainly for its capability of being easily visualized, which makes it easier for analysis. It can be improved significantly by using more advanced texture descriptor, e.g., code-book based model.

As the main goal of this paper is to explore the role of structure and texture information in different tasks, simple structure and texture descriptors, which are easier, clearer for visualization, are selected. However, with more powerful structural and texture relative features or measures, the proposed model has the capability to achieve better performance. In order to check how far BF-M can be improved as well as to verify whether the weights will change with different features, in this sub-section, we take quality assessment of synthesized view as an example and replace the simple descriptors with more complicated ones.

More specifically, the mid-level contour descriptor based measure ST-IQA proposed in [51] (other structure related measures like [52, 53] could be utilized too) is used to replace the structure related estimators BI-NICE and BI-HOG as BI-ST, while the spatial contrast sensitivity (CSF) based texture descriptor proposed in [54] is used to replace the texture related estimator BI-LRI as BI-CSF. Thus, equation (4) can be modified as and the performance of the BF-Mnew is summarized in Table V. The optimized performance of BF-Mnew is obtained while (α + β) is set as 0.8 and γ as 0.2. This best-fit configuration is similar to the optimum configuration of BF-M as described in the previous section. The conclusion that structural information plays the major role in the task of synthesized views’ quality assessment is still solid in this case. As it can be seen from the table, the performance of BF-M is much better than the one of BF-M with more advanced structural/texture estimators. It can be thus concluded that, (1) the weights of structural and texture estimators do not fluctuate with different features/measures; (2) the performance of the proposed model can be improved significantly with more advanced structure/textures measures.