IS&T | Library

SOLAS: Superpositioning an Optical Lens in Automotive Simulation

79 23

Automotive Simulation
Optical Quality
Point Spread Function
Optical Simulation
Fisheye

Daniel Jakab, Julian Barthel, Alexander Braun, Reenu Mohandas, Brian Michael Deegan, Mahendar Kumbham, Dara Molloy, Fiachra Collins, Anthony Scanlan, Ciarán Eising

DOI

10.2352/EI.2025.37.15.AVM-101

Volume 37

Issue 15

Abstract

Automotive Simulation is a potentially cost-effective strategy to identify and test corner case scenarios in automotive perception. Recent work has shown a significant shift in creating realistic synthetic data for road traffic scenarios using a video graphics engine. However, a gap exists in modeling realistic optical aberrations associated with cameras in automotive simulation. This paper builds on the concept from existing literature to model optical degradations in simulated environments using the Python-based ray-tracing library KrakenOS. As a novel pipeline, we degrade automotive fisheye simulation using an optical doublet with +/-2◦ Field of View(FOV), introducing realistic optical artifacts into two simulation images from SynWoodscape and Parallel Domain Woodscape. We evaluate KrakenOS by calculating the Root Mean Square Error (RMSE), which averaged around 0.023 across the RGB light spectrum compared to Ansys Zemax OpticStudio, an industrial benchmark for optical design and simulation. Lastly, we measure the image sharpness of the degraded simulation using the ISO12233:2023 Slanted Edge Method and show how both qualitative and measured results indicate the extent of the spatial variation in image sharpness from the periphery to the center of the degradations.

Digital Library: EI

Published Online: February 2025

Image Segmentation: Inducing graph-based learning

38 14

GNN
Semantic Segmentation
Fisheye image

Aryan Singh, Pepijn Van de Ven, Ciarán Eising, Patrick Denny

DOI

10.2352/EI.2025.37.15.AVM-104

Volume 37

Issue 15

Abstract

This study explores the potential of graph neural networks (GNNs) to enhance semantic segmentation across diverse image modalities. We evaluate the effectiveness of a novel GNN-based U-Net architecture on three distinct datasets: PascalVOC, a standard benchmark for natural image segmentation, Wood-Scape, a challenging dataset of fisheye images commonly used in autonomous driving, introducing significant geometric distortions; and ISIC2016, a dataset of dermoscopic images for skin lesion segmentation. We compare our proposed UNet-GNN model against established convolutional neural networks (CNNs) based segmentation models, including U-Net and U-Net++, as well as the transformer-based SwinUNet. Unlike these methods, which primarily rely on local convolutional operations or global self-attention, GNNs explicitly model relationships between image regions by constructing and operating on a graph representation of the image features. This approach allows the model to capture long-range dependencies and complex spatial relationships, which we hypothesize will be particularly beneficial for handling geometric distortions present in fisheye imagery and capturing intricate boundaries in medical images. Our analysis demonstrates the versatility of GNNs in addressing diverse segmentation challenges and highlights their potential to improve segmentation accuracy in various applications, including autonomous driving and medical image analysis. Code Available at GitHub.

Digital Library: EI

Published Online: February 2025

FlexEye - Application Specific Quality-scalable ISP Tuning

89 45

Image-Signal-Processor (ISP)
Quality of Service (QoS)
Computer Vision
Evolutionary Algorithms
ISP Tuning

Sumbal Akram, Muhammad Abdullah, Khuzaeymah Nasir, Shaharyar Yaqub, Rehan Ahmed, Rehan Hafiz

DOI

10.2352/EI.2025.37.15.AVM-109

Volume 37

Issue 15

Abstract

As AI becomes more prevalent, edge devices face challenges due to limited resources and the high demands of deep learning (DL) applications. In such cases, quality scalability can offer significant benefits by adjusting computational load based on available resources. Traditional Image-Signal-Processor (ISP) tuning methods prioritize maximizing intelligence performance, such as classification accuracy, while neglecting critical system constraints like latency and power dissipation. To address this gap, we introduce FlexEye, an application-specific, quality-scalable ISP tuning framework that leverages ISP parameters as a control knob for quality of service (QoS), enabling trade-off between quality and performance. Experimental results demonstrate up to 6% improvement in Object Detection accuracy and a 22.5% reduction in ISP latency compared to state of the art. In addition, we also evaluate Instance Segmentation task, where 1.2% accuracy improvement is attained with a 73% latency reduction.

Digital Library: EI

Published Online: February 2025

Data optimization strategies for collaborative perception

55 17

collaborative perception
V2X perception
object detection
data optimization

Besma Abdali, Quentin Picard, Maryem Fadili

DOI

10.2352/EI.2025.37.15.AVM-110

Volume 37

Issue 15

Abstract

Collaborative perception for autonomous vehicles aims to overcome the limitations of individual perception. Sharing information between multiple agents resolve multiple problems, such as occlusion, sensor range limitations, and blind spots. One of the biggest challenge is to find the right trade-off between perception performance and communication bandwidth. This article proposes a new cooperative perception pipeline based on the Where2comm algorithm with optimization strategies to reduce the amount of transmitted data between several agents. Those strategies involve a data reduction module in the encoder part for efficient selection of the most important features and a new representation of messages to be exchanged in a V2X manner that takes into account a vector of information and its positions instead of a high-dimensional feature map. Our approach is evaluated on two simulated datasets, OPV2V and V2XSet. The accuracy is increased by around 7% with AP@50 on both datasets and the communication volume is reduced by 89.77% and 92.19% on V2XSet and OPV2V respectively.

Digital Library: EI

Published Online: February 2025

TIOVX Apps – A new approach to development with OpenVX

20 1

OpenVX
Gstreamer
Heterogeneous SoCs
Automotive
ADAS
TI
Pipelining
Linux

Rahul Ravikumar, Abhay Chirania, Shyam Jagannathan, Jesse Villarreal

DOI

10.2352/EI.2025.37.15.AVM-111

Volume 37

Issue 15

Abstract

OpenVX is an open standard for accelerating computer vision applications on a heterogeneous platform with multiple processing elements. OpenVX is accepted by Automotive industry as a go-to framework for developing performance-critical, power-optimized and safety compliant computer vision processing pipelines on real-time heterogeneous embedded SoCs. Optimizing OpenVX development flow becomes a necessity with ever growing demand for variety of vision applications required in both Automotive and Industrial market. Although OpenVX works great when all the elements in the pipeline is implemented with OpenVX, it lacks utilities to effectively interact with other frameworks. We propose a software design to make OpenVX development faster by adding a thin layer on top of OpenVX which simplifies construction of an OpenVX pipeline and exposes simple interface to enable seamless interaction with other frameworks like v4l2, OpenMax, DRM etc....

Digital Library: EI

Published Online: February 2025

Revisiting Birds Eye View Perception Models with Frozen Foundation Models: DINOv2 and Metric3Dv2

31 7

Foundation Models
Birds Eye View
Vehicle Segmentation
DINOv2
Metric3Dv2

Seamie Hayes, Ganesh Sistu, Ciaran Eising

DOI

10.2352/EI.2025.37.15.AVM-112

Volume 37

Issue 15

Abstract

Birds Eye View perception models require extensive data to perform and generalize effectively. While traditional datasets often provide abundant driving scenes from diverse locations, this is not always the case. It is crucial to maximize the utility of the available training data. With the advent of large foundation models such as DINOv2 and Metric3Dv2, a pertinent question arises: can these models be integrated into existing model architectures to not only reduce the required training data but surpass the performance of current models? We choose two model architectures in the vehicle segmentation domain to alter: Lift-Splat-Shoot, and Simple-BEV. For Lift-Splat-Shoot, we explore the implementation of frozen DINOv2 for feature extraction and Metric3Dv2 for depth estimation, where we greatly exceed the baseline results by 7.4 IoU while utilizing only half the training data and iterations. Furthermore, we introduce an innovative application of Metric3Dv2’s depth information as a PseudoLiDAR point cloud incorporated into the Simple-BEV architecture, replacing traditional LiDAR. This integration results in a +3 IoU improvement compared to the Camera-only model.

Digital Library: EI

Published Online: February 2025

Minimizing Occlusion Effect on Multi-view Camera Perception in BEV with Multi-sensor Fusion

49 16

Multi-Sensor Fusion
Bird’s Eye View (BEV)
Occluded
Image Data
Vehicle Segmentation

Sanjay Kumar, Hiep Truong, Sushil Sharma, Ganesh Sistu, Tony Scanlan, Eoin Grua, Ciarán Eising

DOI

10.2352/EI.2025.37.15.AVM-113

Volume 37

Issue 15

Abstract

Autonomous driving technology is rapidly evolving, offering the potential for safer and more efficient transportation. However, the performance of these systems can be significantly compromised by the occlusion on sensors due to environmental factors like dirt, dust, rain, and fog. These occlusions severely affect vision-based tasks such as object detection, vehicle segmentation, and lane recognition. In this paper, we investigate the impact of various kinds of occlusions on camera sensor by projecting their effects from multi-view camera images of the nuScenes dataset into the Bird’s-Eye View (BEV) domain. This approach allows us to analyze how occlusions spatially distribute and influence vehicle segmentation accuracy within the BEV domain. Despite significant advances in sensor technology and multi-sensor fusion, a gap remains in the existing literature regarding the specific effects of camera occlusions on BEV-based perception systems. To address this gap, we use a multi-sensor fusion technique that integrates LiDAR and radar sensor data to mitigate the performance degradation caused by occluded cameras. Our findings demonstrate that this approach significantly enhances the accuracy and robustness of vehicle segmentation tasks, leading to more reliable autonomous driving systems. https: // youtu. be/ OmX2NEeOzAE

Digital Library: EI

Published Online: February 2025

A Comparative Study of NeRF and 3D Gaussian Splatting for Automotive and Edge Applications

84 23

NeRF
3D Gaussian splatting
Automotive applications
Edge applications

Mary Raymond, Ganesh Sistu

DOI

10.2352/EI.2025.37.15.AVM-114

Volume 37

Issue 15

Abstract

This paper presents a comparative study of Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) models within the context of automotive and edge applications. Both models demonstrate potential for novel view synthesis but encounter challenges related to real-time rendering, memory limitations, and adapting to changing scenes. We assess their performance across key metrics, including rendering rate, training time, memory usage, image quality for novel viewpoints, and compatibility with fisheye data. While neither model fully meets all automotive requirements, this study identifies the gaps that need to be addressed for each model to achieve broader applicability in these environments.

Digital Library: EI

Published Online: February 2025

LiDAR Panoptic Segmentation for Autonomous Driving: A Survey

58 33

Panoptic Segmentation
Instance Segmentation
Clustering
Sensor Fusion
3D perception
Autonomous driving
Survey

Aditya Dusi, Bassam Helou

DOI

10.2352/EI.2025.37.15.AVM-115

Volume 37

Issue 15

Abstract

This survey provides a comprehensive overview of LiDAR-based panoptic segmentation methods for autonomous driving. We motivate the importance of panoptic segmentation in autonomous vehicle perception, emphasizing its advantages over traditional 3D object detection in capturing a more detailed and comprehensive understanding of the environment. We summarize and categorize 42 panoptic segmentation methods based on their architectural approaches, with a focus on the kind of clustering utilized: machine learned or non-learned heuristic clustering. We discuss direct methods, most of which use single-stage architectures to predict binary masks for each instance, and clustering-based methods, most of which predict offsets to object centers for efficient clustering. We also highlight relevant datasets, evaluation metrics, and compile performance results on SemanticKITTI and panoptic nuScenes benchmarks. Our analysis reveals trends in the field, including the effectiveness of attention mechanisms, the competitiveness of center-based approaches, and the benefits of sensor fusion. This survey aims to guide practitioners in selecting suitable architectures and to inspire researchers in identifying promising directions for future work in LiDAR-based panoptic segmentation for autonomous driving.

Digital Library: EI

Published Online: February 2025

Flex-PLACE: Flexible robot base placement for industrial inspection

67 25

Quality Inspection
Industrial application
Robot base placement

Vanessa Staderini, Tobias Glück, Andreas Kugi

DOI

10.2352/EI.2025.37.15.AVM-116

Volume 37

Issue 15