The performance of autonomous agents in both commercial and consumer applications increases along with their situational awareness. Tasks such as obstacle avoidance, agent to agent interaction, and path planning are directly dependent upon their ability to convert sensor readings into scene understanding. Central to this is the ability to detect and recognize objects. Many object detection methodologies operate on a single modality such as vision or LiDAR. Camera-based object detection models benefit from an abundance of feature-rich information for classifying different types of objects. LiDAR-based object detection models use sparse point clouds, where each point contains accurate 3D position of object surfaces. Camera-based methods lack accurate object to lens distance measurements, while LiDAR-based methods lack dense feature-rich details. By utilizing information from both camera and LiDAR sensors, advanced object detection and identification is possible. In this work, we introduce a deep learning framework for fusing these modalities and produce a robust real-time 3D bounding box object detection network. We demonstrate qualitative and quantitative analysis of the proposed fusion model on the popular KITTI dataset.
Modern warehouses utilize fleets of robots for inventory management. To ensure efficient and safe operation, real-time localization of each agent is essential. Most robots follow metal tracks buried in the floor and use a grid of precisely mounted RFID tags for localization. As robotic agents in warehouses and manufacturing plants become ubiquitous, it would be advantageous to eliminate the need for these metal wires and RFID tags. Not only do they suffer from significant installation costs, the removal of wires would allow agents to travel to any area inside the building. Sensors including cameras and LiDAR have provided meaningful localization information for many different positioning system implementations. Fusing localization features from multiple sensor sources is a challenging task especially when the target localization task’s dataset is small. We propose a deep-learning based localization system which fuses features from an omnidirectional camera image and a 3D LiDAR point cloud to create a robust robot positioning model. Although the usage of vision and LiDAR eliminate the need for the precisely installed RFID tags, they do require the collection and annotation of ground truth training data. Deep neural networks thrive on lots of supervised data, and the collection of this data can be time consuming. Using a dataset collected in a warehouse environment, we evaluate the performance of two individual sensor models for localization accuracy. To minimize the need for extensive ground truth data collection, we introduce a self-supervised pretraining regimen to populate the image feature extraction network with meaningful weights before training on the target localization task with limited data. In this research, we demonstrate how our self-supervision improves accuracy and convergence of localization models without the need for additional sample annotation.
Inventory management and handling in warehouse environments have transformed large retail fulfillment centers. Often hundreds of autonomous agents scurry about fetching and delivering products to fulfill customer orders. Repetitive movements such as these are ideal for a robotic platform to perform. One of the major hurdles for an autonomous system in a warehouse is accurate robot localization in a dynamic industrial environment. Previous LiDAR-based localization schemes such as adaptive Monte Carlo localization (AMCL) are effective in indoor environments and can be initialized in new environments with relative ease. However, AMCL can be influenced negatively by accumulated odometry drift, and is also reliant primarily on a single modality for scene understanding which limits the localization performance. We propose a robust localization system which combines multiple sensor sources and deep neural networks for accurate real-time localization in warehouses. Our system employs a novel deep neural network architecture consisting of multiple heterogeneous deep neural networks. The overall architecture employs a single multi-stream framework to aggregate the sensor information into a final robot location probability distribution. Ideally, the integration of multiple sensors will produce a robust system even when one sensor fails to produce reliable scene information.