Mobile robotic multi-view photometric stereo

IF 10.6 1区地球科学 Q1 GEOGRAPHY, PHYSICAL

ISPRS Journal of Photogrammetry and Remote Sensing Pub Date : 2025-03-08 DOI:10.1016/j.isprsjprs.2025.02.012

Suryansh Kumar

{"title":"Mobile robotic multi-view photometric stereo","authors":"Suryansh Kumar","doi":"10.1016/j.isprsjprs.2025.02.012","DOIUrl":null,"url":null,"abstract":"<div><div>Multi-View Photometric Stereo (MVPS) is a popular method for fine-detailed 3D acquisition of an object from images. Despite its outstanding results on diverse material objects, a typical MVPS experimental setup requires a well-calibrated light source and a monocular camera installed on an immovable base. This restricts the use of MVPS on a movable platform, limiting us from taking MVPS benefits in 3D acquisition for mobile robotics applications. To this end, we introduce a new mobile robotic system for MVPS. While the proposed system brings advantages, it introduces additional algorithmic challenges. Addressing them, in this paper, we further propose an incremental approach for mobile robotic MVPS. Our approach leverages a supervised learning setup to predict per-view surface normal, object depth, and per-pixel uncertainty in model-predicted results. A refined depth map per view is obtained by solving an MVPS-driven optimization problem proposed in this paper. Later, we fuse the refined depth map while tracking the camera pose w.r.t the reference frame to recover globally consistent object 3D geometry. Experimental results show the advantages of our robotic system and algorithm, featuring the local high-frequency surface detail recovery with globally consistent object shape. Our work is beyond any MVPS system yet presented, providing encouraging results on objects with unknown reflectance properties using fewer frames without a tiring calibration and installation process, enabling computationally efficient robotic automation approach to photogrammetry. The proposed approach is nearly 100 times computationally faster than the state-of-the-art MVPS methods such as Kaya et al., (2023), Kaya et al., (2022) while maintaining the similar results when tested on subjects taken from the benchmark DiLiGenT MV dataset (Li et al., 2020). Furthermore, our system and accompanying algorithm is data-efficient, i.e., it uses significantly fewer frames at test time to perform 3D acquisition<span><span><sup>1</sup></span></span></div></div>","PeriodicalId":50269,"journal":{"name":"ISPRS Journal of Photogrammetry and Remote Sensing","volume":"223 ","pages":"Pages 15-27"},"PeriodicalIF":10.6000,"publicationDate":"2025-03-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"ISPRS Journal of Photogrammetry and Remote Sensing","FirstCategoryId":"5","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0924271625000620","RegionNum":1,"RegionCategory":"地球科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"GEOGRAPHY, PHYSICAL","Score":null,"Total":0}

引用次数: 0

Abstract

Multi-View Photometric Stereo (MVPS) is a popular method for fine-detailed 3D acquisition of an object from images. Despite its outstanding results on diverse material objects, a typical MVPS experimental setup requires a well-calibrated light source and a monocular camera installed on an immovable base. This restricts the use of MVPS on a movable platform, limiting us from taking MVPS benefits in 3D acquisition for mobile robotics applications. To this end, we introduce a new mobile robotic system for MVPS. While the proposed system brings advantages, it introduces additional algorithmic challenges. Addressing them, in this paper, we further propose an incremental approach for mobile robotic MVPS. Our approach leverages a supervised learning setup to predict per-view surface normal, object depth, and per-pixel uncertainty in model-predicted results. A refined depth map per view is obtained by solving an MVPS-driven optimization problem proposed in this paper. Later, we fuse the refined depth map while tracking the camera pose w.r.t the reference frame to recover globally consistent object 3D geometry. Experimental results show the advantages of our robotic system and algorithm, featuring the local high-frequency surface detail recovery with globally consistent object shape. Our work is beyond any MVPS system yet presented, providing encouraging results on objects with unknown reflectance properties using fewer frames without a tiring calibration and installation process, enabling computationally efficient robotic automation approach to photogrammetry. The proposed approach is nearly 100 times computationally faster than the state-of-the-art MVPS methods such as Kaya et al., (2023), Kaya et al., (2022) while maintaining the similar results when tested on subjects taken from the benchmark DiLiGenT MV dataset (Li et al., 2020). Furthermore, our system and accompanying algorithm is data-efficient, i.e., it uses significantly fewer frames at test time to perform 3D acquisition¹

查看原文本刊更多论文

移动机器人多视点测光立体

多视点光度立体（MVPS）是从图像中获取物体细节的一种流行方法。尽管它在不同的物质对象上取得了出色的结果，但典型的MVPS实验装置需要一个校准良好的光源和一个安装在固定基座上的单目摄像机。这限制了MVPS在移动平台上的使用，限制了我们在移动机器人应用的3D采集中使用MVPS的好处。为此，我们提出了一种新的移动mvp机器人系统。虽然提出的系统带来了优势，但它引入了额外的算法挑战。针对这些问题，在本文中，我们进一步提出了一种移动机器人mvp的增量方法。我们的方法利用监督学习设置来预测模型预测结果中的每个视图表面法线、对象深度和每个像素的不确定性。通过求解一个由mvp驱动的优化问题，得到每个视图的精细化深度图。随后，我们在跟踪相机姿态的同时融合了精细的深度图，以恢复全局一致的物体三维几何形状。实验结果表明，该机器人系统和算法具有局部高频表面细节恢复和全局一致的目标形状的优势。我们的工作超越了目前任何MVPS系统，使用更少的帧，在未知反射特性的物体上提供令人鼓舞的结果，而无需繁琐的校准和安装过程，从而实现了计算效率高的机器人自动化摄影测量方法。所提出的方法比最先进的mvp方法（如Kaya等人，（2023），Kaya等人，（2022））的计算速度快近100倍，同时在对取自基准勤勉MV数据集的受试者进行测试时保持相似的结果（Li等人，2020）。此外，我们的系统和相应的算法具有数据效率，即在测试时使用更少的帧来执行3D获取1

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

ISPRS Journal of Photogrammetry and Remote Sensing 工程技术-成像科学与照相技术

CiteScore

21.00

自引率

6.30%

发文量

273

审稿时长

40 days

期刊介绍： The ISPRS Journal of Photogrammetry and Remote Sensing (P&RS) serves as the official journal of the International Society for Photogrammetry and Remote Sensing (ISPRS). It acts as a platform for scientists and professionals worldwide who are involved in various disciplines that utilize photogrammetry, remote sensing, spatial information systems, computer vision, and related fields. The journal aims to facilitate communication and dissemination of advancements in these disciplines, while also acting as a comprehensive source of reference and archive. P&RS endeavors to publish high-quality, peer-reviewed research papers that are preferably original and have not been published before. These papers can cover scientific/research, technological development, or application/practical aspects. Additionally, the journal welcomes papers that are based on presentations from ISPRS meetings, as long as they are considered significant contributions to the aforementioned fields. In particular, P&RS encourages the submission of papers that are of broad scientific interest, showcase innovative applications (especially in emerging fields), have an interdisciplinary focus, discuss topics that have received limited attention in P&RS or related journals, or explore new directions in scientific or professional realms. It is preferred that theoretical papers include practical applications, while papers focusing on systems and applications should include a theoretical background.