The architectural details of SOTA ALPR systems are often kept confidential for commercial purposes, and older implementations are often outdated and not maintained. To provide a methodological framework for the assessment of ALPR systems performance, due to the limited availability of implementations, the following three ALPR systems are used in this study:

Training of the models was not part of this work, i.e. pre-trained models were used to examine their robustness. The observed performance variations under different simulated distortion types offer valuable insights for future work aimed at modifying the models to enhance their robustness against various distortions.

ALPR in Unconstrained Scenarios, published by Silva and Jung [12], has a well-documented architecture and design details. It was designed to address challenging image-capture scenarios, such as oblique camera views in which the vehicle’s license plate is significantly tilted. It comprises four main components (Figure 1). The vehicle detection component uses the pre-trained YOLOv2 [36] network, modified to merge all vehicle-related classes into a single entity and discard other classes. YOLOv2 was chosen for its balance of speed and precision [12]. The license plate detection component uses Warped Planar Object Detection NETwork (WPOD-NET) to locate the license plate and regress one affine transformation per detection, producing bounding box coordinates and affine transformation parameters. This network consists of a total of 21 convolutional layers (filter size 3 × 3), followed by ReLU activation functions, and four max pooling layers (size 2 × 2 with stride 2). The final detection head is divided into two branches. The first branch uses the Softmax activation function to predict the probability that the license plate is located in the current block of pixels. The second branch uses the identity activation function to regress the six affine parameters. The areas with object probability higher than the specified thresholds are predicted to be the license plates and the affine parameters are used for their rectification, transforming the license plate to resemble a front view. Finally, the OCR network recognises the characters on the rectified license plate. This network is based on a modified YOLO network [6]. Silva and Jung [12] trained the WPOD-NET network using 196 images from three different datasets. Various data augmentation techniques were applied to all training images. The OCR network was based on a modified YOLO network [6], and was trained with a large number of synthetic and augmented images to improve robustness to regional variations in license plate formats [12].

HyperLPR [34] is an open-source ALPR system designed to read Chinese license plates. Although it is not a part of a specific research publication, it has been used in various ALPR studies on adversarial attacks on ALPR systems [37, 38]. Information about its architecture was derived from scientific publications and analysis of the available implementation. The model is composed of three convolutional layers with increasing filter sizes (3 × 3 × 32, 3 × 3 × 64, and 3 × 3 × 128), followed by batch normalisation, ReLU activation, and 2 × 2 max pooling layers. The mentioned components of the network are used for detecting license plates, and the output is fed into a network with four gated recurrent units of 256 units each, a dropout layer, and a softmax output layer normalising an 84-unit probability distribution (corresponding to the number of possible license plate characters). There is no information on the dataset used for the training, and the training implementation is not available.

UltimateALPR-SDK, a SOTA commercial solution [35], is known to be the fastest ALPR implementation with high precision and accuracy [39]. During experimentation, it performed recognition significantly faster than the other two models. According to its documentation, the model is trained on license plates from more than 150 countries, primarily in Europe but also in the USA, Canada, Russia, Indonesia and others. There is no public information on its architecture and the code is protected in a way that makes it impossible to deduce it, making it a “black box”. The open-source version used in this study constrains the last character of the license plate. For example, if the plate shows AB123CD, the model outputs AB123C∗.

To ensure diversity of the origins and formats of the license plates, a comprehensive search for available datasets was conducted. Six datasets (Table I) from four different continents were obtained, containing license plates with different formats, font styles, and colours, acquired under various conditions (night, day, background scene complexity, etc.).

The Caltech Cars 1999 dataset [19] consists of parked cars images acquired from the rear. There is no information available on image capture. All images were acquired during daytime, from approximately the same distance, without camera tilt. The license plates vary in colour, format, and number of characters, and consist of a combination of Arabic numerals and Latin characters.

The Application-Oriented License Plate Recognition (AOLP) dataset [20] contains images divided into three groups depending on the complexity of the intended application: 681 images for the access control (AC) group, 757 images for the law enforcement (LE) group, and 611 images for the road patrol (RP) group. The images were captured with different imaging devices and have different spatial resolutions and illumination conditions. The imaging devices were not disclosed. However, it is revealed that the AC and LE image groups were collected with fixed cameras at entry points or roadside, while the images of the RP group were captured with mobile devices. The license plates comprise black Latin characters and Arabic numerals on a white background, with each plate having a fixed length of six characters and numerals.

The Croatian (CRO) license plate dataset [21] contains images acquired with an Olympus Cmedia C-2040ZOOM digital camera. Most images of cars were taken from the rear, with no significant tilt. They were all captured at a very close distance, with few images taken at night.

The PKU dataset [40] consists of images captured under various conditions and is divided into five groups (G1–G5) depending on the acquisition conditions. G1 contains 810 images of cars on highways, captured during the day, while G2 consists of 700 images of cars and trucks on highways during daytime, with some glare from the sunlight present. In G3, there are 743 images of cars and trucks on highways captured during night. G4 contains 572 images of cars and trucks on city roads, both in the day and at night. Finally, G5 contains 1152 images of cars and trucks at intersections with crosswalks. There is no information on the capture devices, but it is discernable that different devices were used for different groups (G1–G5). The license plates consist of white characters on a blue background and follow a fixed format: one Chinese character, one Latin character, followed by five characters, a combination of Latin characters and Arabic numerals.

The RodoSol-ALPR dataset [41] consists of 20,000 images captured by static cameras located at toll booths. Half of the dataset are images of cars, while the other half are images of motorcycles. The license plates of cars are formatted in a single line, while those of motorcycles are in two lines. There are variations in colour, including black, blue, or red characters on a white background, and white characters on a red background. Since the chosen models in this study do not handle two-line plates, images of motorcycles were excluded, resulting in 10,000 images used from this dataset. There is no information available on the capture devices. The acquisition conditions vary in terms of illumination, vehicle position in the scene, and camera distance.

The UFPR-ALPR dataset [18] was collected in the following way: three different cameras were mounted inside a vehicle (GoPro Hero4 Silver, Huawei P9 Lite and iPhone 7 Plus). The vehicle then followed 150 vehicles, 120 cars (passenger cars, buses, trucks, vans), and 30 motorcycles. For each tracked vehicle, 30 images were acquired, resulting in a total of 4500 images. The background was complex and there were variations in the illumination conditions (e.g. shadows). In this work, all 30 images are treated as separate images, as they vary significantly with respect to camera-to-license plate distance and illumination conditions. The license plates contain seven characters in fixed format: three Latin characters followed by four Arabic numerals. The dataset includes license plates with black characters on a white background and white characters on a red background. The car license plates are formatted in a single line, while the motorcycle license plates are formatted in two lines. The latter were excluded from this study, resulting in 3600 images used from this dataset. The images were acquired in daylight with clear weather conditions.

There are certain limitations and restrictions that must be addressed in all presented datasets. First, it is not reasonable to expect that datasets with a small number of images, as well as those having similar and not-so-complex acquisition conditions (Caltech Cars 1999, CRO, AOLP), to be sufficient for training SOTA deep learning ALPR architectures. The other three datasets (PKU, RodoSol-ALPR, and UFPR-ALPR) could serve as valuable sources for training such architectures, especially UFPR-ALPR, which has significant variations in the acquisition conditions. Perhaps the most important limitation is the fact that the initial technical quality is not characterised. In other words, the exact capture conditions and distortions present in the images are unknown (i.e., noise level, lens blur, optical distortions, image processing and compression artifacts, etc.). In particular, for this study, characterising the noise originating from the imaging pipeline would allow the quantification of the total image noise present in the images (original and simulated). To the best of our knowledge, no relevant dataset published so far includes characterised technical quality.

Given the known generalisation capabilities of ALPR systems [25], it is of interest to investigate how different weather distortions influence the performance of ALPR systems. This can be approached in two ways: either by acquiring images in various weather conditions, or by simulating the distortions introduced by adverse weather conditions on existing license plate datasets. The first is less favourable, since it would involve capture of the same scene from a fixed point under different conditions. In the second approach, the recognition performance is first determined on the original images, then systematic and controlled simulated weather distortions are added to them. The weakness of this approach lies in the fact that simulated weather distortions deviate from real-world scenarios. For example, simulating rain by adding a rain mask to the image [42] does not account for the potential reduction in scene brightness due to clouds.

In this work, rain (Figure 2(b)), brightness (Fig. 2(c)), fog (Fig. 2(d)), frost (Fig. 2(e)), and snow (Fig. 2(f)) were simulated on images as weather distortions. The brightness distortion simulates camera overexposure, which reduces the performance of license plate detection algorithms due to reduced contrast, clipping, and loss of details in highlights [43]. The presence of fog, snow or rain represents significant challenges for modern ALPR systems [31, 44], as they can potentially occlude important parts of the scene. Frost on the camera lens is another type of distortion that affects the performance of SOTA object detection algorithms [24]. The brightness, fog, frost, and snow distortions were applied using the image corruption library introduced by Michaelis et al. [24] with five intensity levels defined in the library. Rain distortion was applied using Adobe After Effects [45] software, a well-known tool for developing deraining algorithms [42]. Three different rain configurations were created depending on the presence and direction of the wind (no wind W0, wind in one direction W1, and wind in the opposite direction W2). The parameters used to create rain-distorted images were defined by trial-and-error: the number of raindrops (10,000), the size of the raindrops (level four), and the speed of the rainfall (5000). In cases where wind is present, the wind intensity was 2000 and the wind direction variations 20%.

The simulation of the imaging pipeline was chosen to systematically add read noise to the datasets (Fig. 2(g)), and read noise was modeled using Image Systems Evaluation Toolkit (ISET) [46]. Noise is generated during the process of analog-to-digital signal conversion [47]. The experimental configuration followed that of Plavac et al. [48], where 15 noisy images were generated for each original image, with read noise levels ranging from 0.2 to 14.2 mV in 1 mV increments.

The distortion algorithms used are popular and widely used in the research community to investigate the robustness of DL-based models [49–51] and to develop deraining algorithms [42]. However, certain limitations in their realism and consistency should be noted. Across different types of distortions (Fig. 2), it is obvious that each distortion damaged the original image in a different way and to a different degree for the same intensity level. The algorithm used for fog distortion does not account for scene depth, despite fog intensity being a function of depth [52]. Additionally, it significantly reduces scene brightness, making it resemble smoke more than fog. The frost distortion algorithm introduces non-monotonic changes in image contrast as distortion levels increase. The contrast gradually decreases from levels one to three, but increases at levels four and five. Such inconsistencies can cause ambiguities in the performance evaluation, i.e. changes in ALPR system accuracy could be attributed to these inconsistencies rather than to the distortion level itself. The snow distortion does not fully convey a real snowfall, even though snow is falling, it is not seen on the ground that remains green (vegetation). Additionally, the occlusion of the license plate appears to be more pronounced at distortion level four than at level five. The rain distortion is simplified compared to what could be achieved with physically-based rain rendering algorithms. While the modeled read noise provides grounds for a comprehensive evaluation of noise impact, the compound pre-existing noise level from capture affects the interpretation of results.

Various metrics have been proposed in literature to evaluate the performance of ALPR systems [6, 12, 22]. In this work, the performance of ALPR is assessed using the following measures:

The assumption that an empty output indicates a license plate detection error was derived during trials, but may not always be true. For instance, if the detected bounding box contains only part of the license plate, the recognition might be entirely correct for that partial detection (e.g., half of the characters). Using the current evaluation method, this scenario would be classified as a license plate recognition error. The common methodology in literature is to evaluate the performance of the license plate detection component using Intersection over Union (IoU) metric [22]. This was not feasible in this work since ground truth bounding boxes for license plate detection were available for only one out of the six datasets used.

The above-mentioned metrics were used to evaluate the performance of the three models on original and distorted images. Since the ALPR in Unconstrained Scenarios model is capable of detecting Latin characters and Arabic numerals, the PKU dataset was not part of this model evaluation. The HyperLPR model was developed specially to recognise Chinese license plates; therefore, it is used only on the PKU dataset. For the UltimateALPR-SDK model, modifications were made because its predictions are restricted, and the last character was ignored in the evaluation for all datasets used. The model also skips the first character in the PKU dataset (Chinese alphabet letter), which was accounted for during evaluation of performance. This left the performance evaluation of PKU dataset based on the remaining five characters.

Since the ALPR in Unconstrained Scenarios model only supports license plates containing Latin alphabet characters, the PKU dataset was not included in its performance evaluation. The performance of ALPR in Unconstrained Scenarios and UltimateALPR-SDK models on all datasets, except the PKU dataset, is shown in Table II. On average, the ALPR in Unconstrained Scenarios model showed good performance with an average TA of 78.33%. The highest TA was achieved with the RP group of AOLP dataset, while the lowest TA was observed in the AC group of images from the same dataset. This performance drop is attributed to the close proximity of the camera to the cars in the AC group, leading to car detection failures with YOLOv2-based vehicle detection module, which is reflected by LPDE of 29.52%.

Compared to other two models, the UltimateALPR-SDK model demonstrated very high performance with all datasets (Tables II and III). Except for RodoSol-ALPR and the LP group of AOLP dataset, the model achieved TA above 90% for all datasets used in the study. Similar to previous models, the license plate recognition component was the most error-prone component, while the detection component performed perfectly in the Caltech Cars 1999 dataset and two groups of the PKU dataset. It is important to note that this model had an advantage over the other two models because the last character in the model output was restricted. Therefore, the evaluation was based on the remaining characters. In the case of the PKU dataset, the first (Chinese) letter was also skipped.

HyperLPR was trained to recognize Chinese license plates. Therefore, the performance analysis focuses solely on the PKU dataset. Table III shows that the model performed exceptionally well in all five groups of PKU dataset, with an average TA of 97.83% which was expected, since the model was evaluated on the license plates in the same regional format as the ones used to train the model.

It can be seen that the license plate recognition component was usually more prone to errors compared to the license plate detection component in all the models used, with exceptions in the AC and LE groups of AOLP dataset for ALPR in Unconstrained Scenarios model. The YOLO-based vehicle detection component used in this model tends to fail with detection when dealing with images with great vehicle proximity.