Since the advent of photography, the colorization of grayscale images has been a topic of considerable interest. This technology can provide additional semantic information, enhancing the readability and interpretability of image content while also improving visual effects. Traditional grayscale image colorization methods typically require users to manually provide color and image information for the process [1–3]. However, these approaches are labor-intensive and carry the risk of inaccuracies due to user-provided erroneous color information.

Driven by the rapid development and technological breakthroughs of deep learning, automatic colorization of grayscale images has become an important research topic in recent years. Early convolutional neural network (CNN) architectures for colorization used simple and straightforward designs [4–6], primarily consisting of networks with increased depth achieved by stacking multiple convolutional layers. Although these architectures were well designed, they required large datasets for effective learning, limiting their practicality in scenarios with limited data availability. In subsequent studies, some approaches reformulated the colorization problem as a classification task by learning color distributions from large-scale natural image datasets [7]. Other methods employed pixel histogram modeling to capture multimodal color possibilities and avoid single-point estimation [8]. In addition, research combining local and global semantic features has effectively improved color consistency for objects such as buildings and the sky [9]. Exemplar-based methods, on the other hand, transfer colors from reference images to grayscale inputs, thereby enhancing the realism of specific objects in street scenes [10].

In CNN-based colorization methods, user inputs in the form of dots or doodles are often incorporated [11–14]. However, this approach is associated with an increased workload and requires a certain level of expertise from the user. Consequently, the process can be time-consuming. The employment of generative adversarial networks (GANs) or variational autoencoders is a common practice in achieving diverse colorization. The GANs utilize a competitive framework in which the generator endeavors to generate colors that are indistinguishable to the discriminator while the discriminator’s objective is to differentiate between genuine and generated colors [15–19]. However, GAN-based methods often exhibit suboptimal performance when dealing with objects that have consistent colors. Additionally, they are characterized by high computational cost and significant resource consumption.

Notwithstanding the elevated memory demands associated with the training process, multipath neural networks demonstrate a remarkable ability to accurately capture semantic information [9, 20–24]. By capitalizing on both local and global image features, these networks achieve enhanced colorization precision in grayscale images, thereby improving the overall quality of the colorization process. Transformer-based models [25–27] have recently garnered considerable attention due to their capacity to extract salient image features through multihead attention mechanisms. Another notable approach is the diffusion model [28, 29], which incorporates incremental noise during training to augment image diversity through denoising. However, both methods are highly data-dependent and necessitate substantial datasets for effective implementation.

With the advancement of generative models, diffusion models and multimodal conditional control methods have been introduced into the field of automatic colorization, further enhancing detail and texture. However, recent studies have also revealed several limitations. First, the one-to-many nature of mapping grayscale to color persists: models in open-domain street scenes tend to generate low-saturation or conservative colors, with limited ability to recover rare hues, as shown in Figure 1 [28, 30, 31]. Second, insufficient semantic understanding often leads to incorrect color predictions when the model encounters uncommon objects or distinctive signs, reducing the realism of the results [9, 28]. Furthermore, although exemplar-based methods can improve colorization accuracy, they are highly dependent on the structural similarity and alignment quality of the reference image. When significant differences exist, these methods can cause color shifts or unnatural transfers [10].

Another challenge lies in the trade-off between controllability and stability. Although conditions such as text descriptions, strokes, and reference images allow users greater control over colorization results, current methods still suffer from issues like color bleeding and unstable condition alignment [32, 33]. On the data side, most existing approaches rely on natural image datasets such as ImageNet and Places, which lack training sources specifically tailored for street scene colorization. As a result, their generalization ability in cross-domain applications, such as low-light environments and historical photographs, remains limited [31]. Although some studies have begun to adopt street scene datasets, such as Cityscapes [34] and Mapillary Vistas [35], these datasets were initially designed for semantic segmentation and autonomous driving, rather than being optimized for colorization tasks.

In terms of colorization evaluation, existing assessments still mainly rely on the peak signal-to-noise ratio and the structural similarity index. However, these pixel-level metrics cannot adequately reflect the realism and usability of street scene colorization [36–40], for instance, whether traffic light colors are correctly reproduced or how the results affect autonomous driving tasks. Therefore, future research should not only establish benchmark datasets specifically for street scene colorization but also introduce evaluation metrics grounded in human visual perception. Furthermore, integrating human subjective assessments with task-oriented performance measures will be essential for a more comprehensive evaluation of the practical value of these models in real-world applications.

In image colorization, a thorough understanding of semantic information is crucial to ensure the authenticity of the results. Such understanding enables sensible color assignments, for example, recognizing that cats are unlikely to be blue while leaves are typically green. In the realm of image segmentation, network architectures proposed in [41–45] exhibit a close correlation with semantic and spatial location information despite adopting divergent design approaches. Based on these concepts, this study proposes a similar framework to support and enhance image colorization.

To meet the application needs of autonomous driving and intelligent transportation, the automatic colorization of grayscale street scene images has gradually attracted increasing attention. However, in the extant literature on grayscale image colorization, user-provided information or reference images are often required, with limited emphasis placed on road-specific colorization. This study introduces an automated method for colorizing grayscale road images. Image segmentation was incorporated into the design to address the challenge of road colorization due to the presence of various artificial objects. By capturing multiple local textures and objects and integrating this information into the colorization network, our model can effectively colorize elements such as buildings, trucks, and the sky without human intervention.

Although diffusion models have gained popularity in colorization, our CNN-based model exhibits superior learning capability, achieving improved performance on both small and large datasets. Our approach leverages the CIELAB color space to predict chromaticity components of an image. The proposed model comprises three key elements: a contextual network, a spatial network, and a colorization network. The contextual network learns semantic information about objects, the spatial network identifies their positions, and the colorization network integrates this information to generate the final colorized output. Experimental results demonstrate that our method outperforms the state-of-the-art diffusion model, achieving superior colorization accuracy and diversity.

The motivation and objective of this study are to enable the practical application of colorization in real-world scenarios while delivering high-quality results. However, due to the diversity of real-world colors, this task presents considerable challenges. Our research primarily employs the proposed model training architecture to first validate its effectiveness on a specific dataset, with subsequent work focusing on conducting a more comprehensive investigation and optimization of the model’s generalization capability.

This section presents a comprehensive account of the proposed CNN architecture illustrated in Fig. 1. The architecture comprises three principal components: the context pathway, the spatial pathway, and the colorization pathway. The context path furnishes data regarding the immediate content of the image, including the sky, building, and tree. In contrast, the spatial path provides information regarding the exact spatial position of these contextual elements within the image. The outputs of the context and spatial paths were integrated after important features were extracted by Efficient Channel Attention (ECA-Net) [46] and Squeeze and Excitation (SE-Net) [47], which enhanced channel-wise attention by adaptively weighting feature maps based on their importance, thereby facilitating effective semantic segmentation before being passed to the coloring path. A detailed explanation of ECA-Net and SE-Net is provided in Section 2.3. The incorporation of semantic information into the colorization network yielded three primary benefits: (1) improved precision in color prediction, (2) mitigation of color overflow problems, and (3) enhanced diversity in color applications.

In the early stages of this study, a single-path architecture was tested, but the recognition and segmentation performance on grayscale images was found to be suboptimal. This was likely due to the absence of color information, which typically provided important cues for semantic discrimination. To address this limitation, a dual-path architecture was adopted. The contextual path, based on an ImageNet pretrained model, enhanced semantic understanding and improved generalization across diverse scenes. The spatial path focused on preserving edge and structural details, thereby helping to prevent color bleeding during the colorization process.

In the proposed model, the process of downsampling (DS) was achieved through the implementation of convolution (Conv2d) [48], a layer that extracts spatial features by sliding filters over the input to generate feature maps. For upsampling (US), transposed convolution (ConvT2d) [49] was employed, which reverses the convolution operation to increase spatial resolution in a learnable manner. Following each convolutional layer, batch normalization (BN) [50] was applied to normalize the activations within each mini-batch, thereby accelerating training and improving model stability. To introduce nonlinearity, the rectified linear unit (ReLU) [51] was used.

The CNN pathways were connected and optimized via an end-to-end training process. The entire framework operated in a CIELAB-type color space, which consists of three channels: the lightness channel (L∗) and two chromatic channels (a∗ and b∗). The input of the three pathways was a grayscale image. The input grayscale image was derived from an RGB-to-grayscale conversion based on the ITU-R BT.709 standard (Y709 = 0.2126R + 0.7152G + 0.0722B), which is closely aligned with the sRGB standard. The outputs from the contextual and spatial paths represented semantic segmentation results while the chromatic path produced the image planes for a∗ and b∗ color channels, separately. Since the primary objective of image colorization is to generate perceptually realistic chromatic channels from a grayscale image, this study adopted the method proposed by Iizuka [9], where the input grayscale image is treated as the lightness channel in the LAB color space transformation and the calculation of color differences. The final colorized images are represented in the sRGB color space. The conversion from CIELAB to sRGB was carried out in OpenCV-Python using the cvtColor function with the COLOR_Lab2RGB flag. This process involved two steps: (1) transforming CIELAB values into CIE 1931 XYZ values with the D65 illuminant as the reference white and (2) converting the XYZ values into sRGB in compliance with the IEC 61966-2-1:1999 standard [52].

To leverage the superior feature extraction capabilities of the pretrained ResNet50 [53] model, this study adopted the recommended input size of 224 × 224 × 3 in the semantic segmentation network. However, using the same resolution in the coloring network may result in the loss of details of small objects (such as brake lights and traffic signals). To address this issue, this study adopted a high-resolution input of 896 × 896 × 3 in the coloring network to preserve the details of small objects and improve coloring accuracy. This study assumed that the semantic information of large objects is sufficient to provide the overall contextual guidance required for coloring.

Figure 2 illustrates the colorization results on the GTA5 validation set, highlighting the model’s ability to colorize small objects with diverse color distributions accurately. Leveraging image segmentation, the model achieved realistic colorization for categories such as pedestrians and vehicles. The entire process was fully automated, requiring no human intervention.

Previous research on the perception of color differences in large printed images [58] demonstrated that statistical measures of extreme color deviations correlate more strongly with perceived image color differences than mean color differences do. The results of image colorization using the proposed methods, in terms of the mean and 95th percentile of CAM16-UCS color differences (denoted as ΔECAM16-UCS) [59] between the colorized images and the corresponding ground-truth images, are presented in Table IV. The global color difference refers to the mean and the 95th percentile of ΔECAM16-UCS computed across all pixels in the image. In contrast, the traffic light color difference refers explicitly to the mean and the 95th percentile of ΔECAM16-UCS computed only over pixels corresponding to traffic signal lights. In Methods 1 and 2, a ResNet50 pretrained model was employed to enhance model diversity and to ensure optimal performance even with limited training data. The results are illustrated in Figure 3(a). In Methods 3 and 4, the ability to colorize small objects, such as red traffic signs with traffic horns, was enhanced by increasing the resolution and reducing the perceptual loss as illustrated in Fig. 3(b). Figure 3(c) demonstrates that the ability to colorize artificial objects can be enhanced by instance normalization and by increasing the chroma of the output images by a factor of 1.3 in Methods 4–6.

To further contextualize these findings, it is necessary to compare the method with representative prior studies. Zabari proposed a text-guided latent diffusion framework for image colorization, which integrates Cold Diffusion with a CLIP (Contrastive Language–Image Pretraining; Radford et al., 2021) [60]-based ranking mechanism to provide flexible and diverse results, albeit at a relatively high computational cost. In contrast, Iizuka designed a CNN-based architecture that performs colorization by fusing global scene priors with local features through image recognition. Their model benefits from implicit semantic guidance via scene classification, enabling natural colorization across a wide variety of images. As shown in Figure 4, the proposed model (Method 6) is compared with Zabari’s diffusion model [29] and Iizuka’s model [9] for image colorization. The model’s performance was evaluated using 2000 training samples and 500 validation samples. Although implementations employed a greater number of training samples, this strategy is not consistently practical because the data collection process is often characterized by its labor-intensive and time-consuming nature, particularly in real-world applications. Notably, the ability to achieve competitive results with a reduced dataset underscores the efficiency of the proposed method and indicates its robust generalization capabilities while substantially reducing training resource demands.

The results indicate that elements such as trees and artifacts are not effectively colorized in the Zabari and Iizuka models. Although the sky exhibits some blue tones, the colorized regions remain imprecise. The proposed model has been demonstrated to exhibit superior performance in identifying both natural and artificial objects. The results of the ΔECAM16-UCS color difference comparison for various semantic categories are presented in Table V.

The results indicate that the proposed model achieves superior color restoration for categories such as sky and trees. This improvement can be attributed to the relatively consistent spatial positions, features, and color distributions of elements like the sky, trees, and vegetation.

In terms of artificial objects, the model employed in this study utilizes a multipath neural network, enabling precise identification of the necessity for distinct colors to denote varied semantic objects.

In the context of image segmentation, the proposed model’s categorization of vehicles, including cars, trucks, and buses, enables the discernment of color variations across different categories. This capability is supported by the neural network’s learning process, which integrates diverse annotated data. Analysis revealed that the color differences between buses and trucks were within acceptable limits. However, the outcome for cars was less optimal, likely due to the inability to distinguish between different cars solely through image segmentation when multiple cars were present in a single image. However, the Zabari and Iizuka models demonstrated even poorer performance, with the color of the cars often being different from the ground truth and being unable to distinguish the saturated red of the brake lights.

In the case of roads, walls, and sidewalks, these features in the image were very close to each other. Therefore, image segmentation is necessary to determine the location of the road and the presence of people. The proposed model outperformed the competitor’s model, as it improved the visibility of roads, walls, and sidewalks.

In the coloring of small objects, such as traffic lights and brake lights, the proposed model significantly outperformed other models due to the use of a high-resolution U-Net architecture in the colorization path, which preserved the loss of image encoding by concatenating the encoder and decoder information.

Coloring buildings presents a unique challenge compared to other artifacts due to the wide range of styles and colors found in this category. To address this challenge, we propose a model for segmenting images into various categories, thereby enhancing learning. This approach aimed to enhance the diversity of building colors and minimize the impact on other categories. The results of the proposed model demonstrated the efficacy of this method, surpassing the performance of different models.

The ΔECAM16-UCS calculation method in this study is as follows. First, convert both the colorized image and the ground-truth image from the sRGB space to the CAM16-UCS J’a’b’ space, using a D65 reference white and an adapting field luminance of 20 cd/m2 under dim surround conditions. Then, compute the Euclidean distance between them in the J’a’b’ space. Statistics are computed only for pixels belonging to the corresponding category in the segmentation mask (i.e., ΔECAM16-UCS is calculated for all pixels of each object), and no low-pass filtering is applied, primarily to preserve image detail. However, such pixel-wise statistics cannot account for perceptual phenomena such as visual masking or color assimilation. To further validate whether the model aligns with human subjective perception and to address these limitations, psychophysical experiments can serve as a valuable extension of this study.

The proposed method has two primary limitations. First, although the coloring accuracy of small objects can be improved by using a high-resolution U-Net structure, certain inaccuracies still persist. For instance, as shown in Figure 5(a), sign lights that are originally yellow may be incorrectly colored as green in the output. Second, referring to Figs. 5(b) and 5(c), in more complex scenes, the model sometimes applies unnatural colors, such as gray or red, which negatively impacts the overall realism of the image.

Although the proposed model performs well on the synthetic image dataset GTA5, it is still necessary to further validate its performance on real-world images. The BDD100K [61] dataset was released by the Berkeley Artificial Intelligence Research laboratory in collaboration with the Berkeley DeepDrive Industrial Consortium. It is one of the largest and most diverse publicly available datasets of driving videos. The dataset consists of 720p-resolution driving videos collected across multiple regions of the United States, including metropolitan areas such as New York City and the San Francisco Bay Area. The model (Method 6) proposed in this study achieves satisfactory results in terms of color filling performance on real-world scene datasets as shown in Figure 6. However, compared to the colorized outputs, the color differences from the ground-truth images are still relatively high. For memory colors (e.g., trees, roads, sky), the colorization results are satisfactory, likely because these categories exhibit relatively consistent canonical colors across scenes. An additional observation is that sky regions exhibit lower chroma, likely due to a dataset-induced bias in GTA5, where skies tend to appear less vividly blue. In the coloring of small objects, the red color of brake lights is consistently present, indicating that the high-resolution input of the proposed model effectively enhances the coloring results, particularly ensuring accuracy in coloring small objects on the road. Although the coloring of nonmemory colors is less accurate, the overall image still demonstrates noticeable color diversity. Additionally, based on the color filling performance of the proposed model, transfer learning can be applied to train the model on different application datasets, thereby improving color filling performance and reducing color differences. Furthermore, incorporating panoptic segmentation may also enhance the coloring of vehicles.

In this study, a novel automatic image colorization method was proposed, which integrates a multipath neural network with semantic segmentation to enhance the accuracy of color prediction. The experimental results on the GTA5 dataset demonstrated that the proposed method significantly improved color fidelity and object edge preservation compared to existing CNN and diffusion models. Using a high-resolution training dataset, the proposed method achieved small global color differences on 4500 test images relative to the ground truth, with mean ΔECAM16-UCS = 2.7 and 95th percentile ΔECAM16-UCS = 7.3. Moreover, even when trained on a small dataset, the method consistently outperformed the CNN and diffusion models across all categories. However, in highly complex scenes, this method did not produce ideal coloring results.

Future work will focus on improving image segmentation through two main directions. One direction is the application of panoptic segmentation to achieve more comprehensive scene understanding. Another direction is the incorporation of semantic guidance for objects with characteristic colors, such as taxis or airplanes of specific airlines. These strategies are expected to reduce color misclassification and further enhance color accuracy and diversity in complex scenes. In addition, existing color difference formulas are considered insufficient to fully capture human perception in complex images. To address this limitation, a preliminary experimental framework has been designed to recruit participants for subjective evaluations. Specifically, participants will compare the colorized results generated by the deep learning model with the corresponding ground-truth color images and provide naturalness ratings. This evaluation aims to assess the perceptual performance of colorization models from the perspective of human visual perception. This study can help us gain a deeper understanding of the key aspects of perceived color differences in image colorization.

Code and model availability: The source code and trained model can be provided upon reasonable request for research use. Requests can be directed to D10822501@mail.ntust.edu.tw or plsun@mail.ntust.edu.tw. Redistribution and commercial use are prohibited.