Abstract

This paper introduces Scene Completeness-Aware Depth Completion (SCADC) to complete raw lidar scans into dense depth maps with fine and complete scene structures. Recent sparse depth completion for lidars only focuses on the lower scenes and produces irregular estimations on the upper because existing datasets, such as KITTI, do not provide groundtruth for upper areas. These areas are considered less important since they are usually sky or trees of less scene understanding interest. However, we argue that in several driving scenarios such as large trucks or cars with loads, objects could extend to the upper parts of scenes. Thus depth maps with structured upper scene estimation are important for RGBD algorithms. SCADC adopts stereo images that produce disparities with better scene completeness but are generally less precise than lidars, to help sparse lidar depth completion. To our knowledge, we are the first to focus on scene completeness of sparse depth completion. We validate our SCADC on both depth estimate precision and scene-completeness on KITTI. Moreover, we experiment on less-explored outdoor RGBD semantic segmentation with scene completeness-aware D-input to validate our method.

Video

Framework

The framework is shown as follows.

Our motivation is to obtain more structured upper scene measurements for depth completion-based method without the guidance of upper scene groundtruth. Meanwhile, we obtain more accurate lower scene depth from the semi-dense depth groundtruth data. SCADC exploits raw depth from both the depth completion-based block and the other stereo matching branch. Attentional Point Confidence (APC) module learns the attention score for the lidar. To perform probabilistic fusion, only measuring the weight for one modality is sufficient. We then combine the two raw maps from the two branches. Last, we use a stacked hourglass network ro regress depth values.

Structure of APC and SAConv we used in this work. APC learns confidence score from SAConv, under the guidance of a pre-computed confidence map. SAConv is a sparse convolution method to perform efficent convolution over sparse input.

Results

(Left) Qualitative results of stereo matching (PSMNet), SSDC (direct lidar completion), and our SCADC on KITTI Depth Completion validation set. We show driving scenarios of large trucks beside and cars with loads. Vehicle structures extend to upper scenes. SSDC fails to regress upper structures. Shape distortion of PSMNet could be seen in highlights (a) Bicycle contour. (b) Bridge structure bleeds into the background and creates irregular estimations. (Right) Comparison on KITTI Depth Completion test set.} Results of other works are directly from KITTI official site. Other depth completion based method only obtain messy and unstructured upper scenes.

We last show the applicability of the more structured upper scene resutls. We perform semantic segmention for KITTI and get consistent label maps.

Citation