Measuring posture and volume of meals for meal-assisting robotics

Abstract

Accurate volume measurement and posture estimation of meals have significant applications in meal-assisting robotics, food engineering, and food analysis. Traditional multi-view image acquisition techniques have proven effective in reconstructing the 3D morphology of meals. However, these methods encounter significant challenges when applied to real-time posture and volume estimation for meal-assisting robots due to computational complexity and time constraints. Furthermore, the multi-view image acquisition methods require precise calibration and synchronization of multiple cameras, which can be cumbersome and impractical in dynamic environments of meal-assisting robots. Moreover, the irregular shapes of dinner plates and complex rheological properties of fluid and solid foods pose substantial hurdles to achieving accurate measurements. Aiming at the above problems, this paper proposes a new method for fitting, posture estimation, and volume measurement of multiple classes of foods from a single viewpoint (FPV-MCFs). The method utilizes the RGB-D images of meals from a single viewpoint as input to reconstruct and fit different kinds of meals in three dimensions and then estimates the posture and volume of each meal separately by combining with the geometric models of meals. Specifically, for the non-Newtonian fluid sticky meals (non-Newtonian FSM) and non-Newtonian fluid-solid interaction sticky meals (non-Newtonian FSISM), the principal component analysis (PCA), iterative closest point algorithm (ICP), and optimization method with chamfer distance as the objective function are used in this paper to fit the point cloud of meals into a plate-like sector geometric model. For block meals (BM) and diced mixed meals (DMM), the least squares and randomized sampling consistency (RANSAC) algorithms are used to fit them to get the sphere and super-ellipsoid models, respectively. Finally, the volume and posture of each meal are estimated by combining the geometric approach with the FPV-MCFs algorithm, respectively. To evaluate the performance of the FPV-MCFs algorithm, some comprehensive measurement experiments of the actual volumes and actual postures of multiple meals are carried out, which cover single classes, mixed classes, and different orientations. The experimental results show that the FPV-MCFs algorithm exhibits smaller absolute relative deviations and average deviations in both volume measurement and posture estimation of meals. Specifically, the FPV-MCFs algorithm achieves 2.95% and 2.53% in $\overline{ARE\left(V\right)}$ and $\overline{\epsilon \beta }$ metrics for non-Newtonian FSM or non-Newtonian FSISM, respectively, and 6.6 ms and 1.2 ms in processing time metrics for DM and DMM, respectively. Moreover, experiments involving different voxel numbers and various orientations of meals have demonstrated that the proposed algorithm boasts strong robustness. This study can provide important guidance and practical application value for robotic arm food fetching planning in meal-assisting robotics and food engineering.