2D–3D fusion approach for improved point cloud segmentation

Abstract

Semantic segmentation of point clouds with deep learning (DL) has shown significant potential. However, existing DL algorithms struggle with accurately segmenting categories with fewer instances or similar shapes. To address this issue, this paper proposes a 2D–3D fusion approach (Point-YOLO) to improving semantic segmentation accuracy of point clouds. The proposed method captures images from virtual cameras within point clouds and conducts image semantic segmentation with YOLO. Then, the image segmentation results are fused with point cloud segmentation results obtained from point-based DL methods (e.g., PointNet, PointNet++) for improved point cloud segmentation. The Point-YOLO approach improved mean class Accuracy by 14.52 % on the S3DIS dataset and 26.49 % on the underground dataset compared to PointNet++. The mean Intersection over Union for minority categories such as doors, windows, and air ducts improved by 37.65 %, 19.35 %, and 75.16 %, respectively. The proposed method also performed well for state-of-the-art algorithms such as PointNext and PointVector.