Sunday 09 March 2025
The devil is in the details when it comes to representing images as LiDAR data, a crucial task for autonomous vehicles and other applications that rely on 3D scene understanding. Researchers have been focusing on developing loss functions to distill pre-trained 2D image representations into 3D models, but overlooked design elements have been holding them back.
A team of scientists has identified three fundamental design choices that are more critical than the loss function itself: the LiDAR coordinate system, quantization according to the existing input interface, and data utilization. By addressing these often-overlooked aspects, they have achieved significant performance improvements in 3D semantic segmentation and object detection tasks.
The first design choice is the LiDAR coordinate system. Previous work has used cylindrical coordinates without considering their side effects when combined with sparse convolution layers, leading to spatial quantization errors. The team found that using Cartesian coordinates instead results in more accurate representations of the 3D scene.
Quantization is another critical aspect. Existing methods have been discarding unsynchronized data, limiting the use of only a small portion of data that is temporally synced across sensors. By utilizing unsynchronized data itself, the team was able to achieve better results.
Data utilization is also key. The team’s approach involves accumulating point clouds from consecutive frames and aggregating them in global coordinates. This allows for more accurate representation of dynamic scenes, such as moving objects or changing environments.
The researchers demonstrated their method on the nuScenes dataset, achieving 16% improvement in 3D semantic segmentation and 13% improvement in 3D object detection compared to existing methods. They also showed that their approach is consistent across different pre-trained 2D backbones and frame gaps.
To achieve these improvements, the team developed a new module called Positive Pair Mining (PPM). PPM aggregates consecutive point clouds, removes ground points, clusters non-ground points, tracks moving clusters, applies cluster-wise ICP, and constructs positive pairs of 3D-2D. This module is designed to alleviate data redundancy and transform moving points from inter-frames into the nearest keyframe.
The team’s findings highlight the importance of considering design elements beyond loss functions in image-to-LiDAR representation learning. By addressing these often-overlooked aspects, researchers can achieve significant performance improvements and develop more accurate 3D scene understanding models for applications such as autonomous driving and robotics.
Cite this article: “Unlocking Accurate Image-to-LiDAR Representations: A Study on Design Choices”, The Science Archive, 2025.
Lidar, Image-To-Lidar Representation Learning, 3D Semantic Segmentation, Object Detection, Autonomous Vehicles, Robotics, Positive Pair Mining, Nuscenes Dataset, Loss Functions, Pre-Trained 2D Backbones
