The detection and recognition of objects is essential for the operation of autonomous vehicles and robots. Designing and predicting the performance of camera systems intended to supply information to neural networks and vision algorithms is nontrivial. Optimization has to occur across many parameters, such as focal length, f-number, pixel and sensor size, exposure regime and transmission schemes. As such numerous metrics are being explored to assist with these design choices. Detectability index (SNRI) is derived from signal detection theory as applied to imaging systems and is used to estimate the ability of a system to statistically distinguish objects [1], most notably in the medical imaging and defense fields [2]. A new metric is proposed, Contrast Signal to Noise Ratio (CSNR), which is calculated simply as mean contrast divided by the standard deviation of the contrast. This is distinct from contrast to noise ratio which uses the noise of the image as the denominator [3,4]. It is shown mathematically that the metric is proportional to the idealized observer for a cobblestone target and a constant may be calculated to estimate SNRI from CSNR, accounting for target size. Results are further compared to Contrast Detection Probability (CDP), which is a relatively new objective image quality metric proposed within IEEE P2020 to rank the performance of camera systems intended for use in autonomous vehicles [5]. CSNR is shown to generate information in illumination and contrast conditions where CDP saturates and further can be modified to provide CDP-like results.
Autonomous vehicles rely on the detection and recognition of objects within images to successfully navigate. Design of camera systems is non-trivial and involves trading system specifications across many parameters to optimize performance, such as f-number, focal length, CFA choice, pixel, and sensor size. As such, tools are needed to evaluate and predict the performance of such cameras for object detection. Contrast Detection Probability (CDP) is a relatively new objective image quality metric proposed to rank the performance of camera systems intended for use in autonomous vehicles. Detectability index is derived from signal detection theory as applied to imaging systems and is used to estimate the ability of a system to statistically distinguish objects, most notably in the medical imaging and defense fields. A brief overview of CDP and detectability index is given after which an imaging model is developed to compare and explore the behavior of each with respect to camera parameters. Behavior is compared to matched filter detection performance. It is shown that, while CDP can yield a first order ranking of camera systems under certain constraints, it fails to track detector performance for negative contrast targets and is relatively insensitive.
The automotive industry formed the initiative IEEE-P2020 to jointly work on key performance indicators (KPIs) that can be used to predict how well a camera system suits the use cases. A very fundamental application of cameras is to detect object contrasts for object recognition or stereo vision object matching. The most important KPI the group is working on is the contrast detection probability (CDP), a metric that describes the performance of components and systems and is independent from any assumptions about the camera model or other properties. While the theory behind CDP is already well established, we present actual measurement results and the implementation for camera tests. We also show how CDP can be used to improve low light sensitivity and dynamic range measurements.
The recent established goal of autonomous driving cars, motivates the discussion about safety relevant performance parameters in the automotive industry. The majority of currently accepted key performance indicators (KPIs) do not allow a good prediction over the system performance along a safety relevant critical effect chain. A breakdown of the functional system down to component and sensor levels makes this KPI problem evident. We will present a methodology for sensor performance prediction by a probabilistic approach, on the basis of significant critical use cases. As a result the requirement engineering along the effect chain especially for safety relevant processes appears transparent and understandable. Specific examples from the field of image quality will concentrate on the proposal of a new KPI, the contrast detection probability (CDP). This proposal is currently under discussion within the P2020 work group on automotive image quality and challenges known KPIs such as SNR, especially with respect their effects on automotive use cases.