After the success of AlexNet [1] in ImageNet [2] Challenge 2012, deep convolutional neural networks have become an indispensable tool in computer vision. From the early successes in image classification, these networks have been used in different computer vision tasks such as object detection, object tracking [3, 4] and image segmentation [5–8]. Image classification is the task of assigning a class label to a particular image, but merely assigning a class label does not provide information on the understanding of the entire scene. Human visual systems are naturally trained to develop a deeper understanding simply by looking at the image. Semantic segmentation is the task of assigning a class label to a particular pixel and grouping together similar pixels. Semantic segmentation provides a deeper understanding of the context of the entire image. Several architectures based on convolutional neural networks have been developed for image classification over the last decade. An example of semantic segmentation is shown in Figure 1 (Ref. [9]). Some of the most popular ones are ResNet [10], DenseNet [11], VGGNets [12], GoogleNet [13], EfficientNets [14] among others. Recently, next-generation ConvNeXts [15] have been proposed to take on transformers, where ResNet architectures have made some design changes to mimic patchifying. In these architectures, depth-wise convolutions and inverted bottlenecks, larger kernel sizes permit global receptive power and micro-designs like fewer activations and normalization layers.

Transformers have been popular in the field of natural language processing. Recently, computer vision researchers have achieved state-of-the-art results on the task of image classification outperforming most CNN based architectures. Popular architectures include the vision transformer (VIT) [16] and the shifted window transformer (SWIN) [17]. The core idea of VIT is to split the input image into patches followed by vectorization. The vectors are then followed by dense layers with shared parameters. The next step is positional encoding, which is used to represent structural information. The vectors along with a token for classification are passed through a series of multihead self-attention layers followed by dense layers, which constitute the transformer encoder network. The SWIN transformer also transforms the image into patches that are passed through a linear transform. The SWIN transformer uses small patches in the first transformer layer and merges into larger patches in the deeper transformer layers. Then the concept of shifted window-based self-attention is applied followed by a series of transformers with limited attention and merging layers followed by a linear dimensionality reducers. One of the recent advances is the data-efficient image transformer (DEIT), which uses the concept of distillation and the attention mechanism [18].

Generally, deep learning-based models are data-dependent and need a large amount of data to develop robust models, and as a result, the quality of data is an important parameter when training these models. One of the most critical challenges in the deployment of machine learning-based systems in the real world is that during testing time, if there is a shift in distribution of the data, the system is vulnerable to failures. Recent work [19–21] has demonstrated that image quality is a very important attribute in developing a machine learning-based system involving images. Multiple studies by Hendrycks et al. [22–24] in the ImageNet database have demonstrated that perturbations and distribution shifts in images have a significant impact on the performance of deep learning-based computer vision models.

One of the lesser explored areas in deep learning, in computer vision systems is the impact of colour on the performance of these networks. Deep learning networks such as CNNs [25] and generative adversarial networks (GANs) [26, 27] have shown promising results in converting grayscale images into colour images, and also some approaches have been proposed in the area of demosaicing [28, 29]. Recent studies [30–32] have shown that colour information has a significant impact on image classification tasks. Colour parameters such as the hue angle shift [33] have shown a significant impact on the performance of state-of-the-art deep neural networks trained on pristine ImageNet data. Colour information has been exploited successfully in the past by image segmentation algorithms [34–36]. Kantipudi [37] et al. have shown that colour channels can be exploited to attack deep learning systems. Previous studies have shown that CNNs trained on ImageNet are biased towards texture [38]. Much current research is focused on the robustness [39–43] of deep learning research; therefore,it is important to investigate the effect of colour and image quality on the robustness of these deep learning methods. To the best of our knowledge, there has been very little work exploring the impact of colour information on modern deep learning-based semantic segmentation networks.

In this paper, we try to study the robustness of deep learning-based semantic segmentation models [44]. One of the first challenges is to identify and inspect quality and colour based parameters which are likely to have an impact on the performance of deep learning based semantic segmentation models and generate a dataset to bench-mark the performance. The parameters used for our study include color space information, ISONoise, gamut, hue angle shift, saturation, contrast, brightness, etc. to name a few. The proposed dataset is built using the standard ADE20K [9, 45]. We tested some of the CNN- and transformer-based methods for semantic segmentation to gain insight on how these models respond to inputs which have been perturbed from the distribution on which they are trained. With more and more real-life tasks being deployed based on deep learning trained computer vision models, understanding the robustness parameters of these models is of paramount importance. The rest of the paper is organized as follows: we briefly describe the methods and architectures used in this study, and then we describe in detail the dataset generation and investigations conducted.

For this work, we have studied a few state-of-the-art semantic segmentation networks and their backbones. We have included CNN- and transformer-based methods for this study. For our analysis, we have used models pre-trained in ADE20K from the MMSegementation [46] model zoo.

In this section, we describe the performance measure used for our analysis. Common problems in semantic segmentation include mismatched class labels, getting inconspicuous classes, mismatched relationships, among others to name a few. For semantic segmentation, each and every pixel in the image is assigned a class label. To measure performance, we calculate the ratio of the number of pixel labels identified by the semantic segmentation algorithm to the pixel annotations from the ground truth. Let GT and Pred be the ground truth and the predicted segmented maps, respectively. Let T_N be the total number of pixels in the ground truth and T_matched be the total number of pixels where the ground truth and the predicted segmentation map labels are in a pair. Pixel accuracy (PA) is the ratio between T_matched and T_N. For this analysis, we report the mean pixel accuracy for 2000 images considered the dataset and the values are reported as a percentage.

One of the key challenges, to our knowledge, is that there are no data to support the study of how image quality and colour affect the process of semantic segmentation in images. ADE20K is one of the most challenging benchmarks in semantic segmentation, so we decided to build our image quality and colour dataset on the classes of the ADE20K dataset by using 2000 of their labeled validation datasets, and we only just modified those colour and quality parameters so that the boundaries of segmentation are not altered. We have used the pixel-level accuracy between the predicted and ground-truth maps to evaluate different semantic segmentation techniques. Table I shows the average pixel classification accuracy for all 2000 pristine images in the dataset. The main observation is that transformer-based models have better accuracy than CNN models, but the UPerNet-ConvNeXt combination has competitive performance. For all the methods, as expected, ResNet-101 performs better than the ResNet-50 backbones.

In this paper, we have created synthetic data based on the popular ADE20K dataset to understand the impact of colour information and quality parameters on the performance of semantic segmentation networks. We have performed comparative studies ranging from basic FCNs to the most recent state-of-the-art transformer-based methods and attempted to get an estimate of robustness of these methods under different colour and image quality-based modifications. The general observation is that transformer-based methods show more robustness to poor-quality data compared to their CNN-based counterparts. However, transformer-based methods can be difficult to train and may require significant computing resources. There has to be a trade-off between robustness and model complexity. One consistent observation is that pyramid scene parsing networks (PSPNets) and unified perceptual parsing networks (UPerNets) with ResNet-50 and ResNet-101 backbone perform significantly better than FCN on these backbones, with ResNet-101 outperforming ResNet-50. Colour channel information is not widely studied in the context of deep neural networks and the current era is based on deep learning, and one of the main objectives of this study is to find ways in which these modern techniques match to quality and colour parameters and motivate researchers to incorporate colour channels into the architecture engineering process. Real-time computer vision systems have been deployed in critical areas such as surgery, autonomous driving, etc. to name a few, errors in scene understanding can lead to catastrophe. To the best of our knowledge, we have listed some of the key quality and colour parameters which need to be investigated in detail by both the colour imaging and deep learning community to build safer systems.