Fast geometric deep learning for intraoperative soft tissue deformation estimation: Towards real-time AR guidance in liver surgery

The real-time computation of the intraoperative spatial positioning of soft tissues, particularly those not visible within the body, such as blood vessels, is crucial for augmented reality navigation systems. Conventional biomechanical models face challenges in real-time computation and the acquisition of boundary conditions. A novel deep learning framework is proposed, integrating an optimized PointNet++ architecture for modelling liver and vascular deformation. The framework utilizes multi-scale feature extraction, lightweight self-attention mechanisms, and residual feature propagation to predict vascular displacement fields and normal vectors. A hybrid loss function that integrates Chamfer distance and MSE losses improves geometric consistency and deformation accuracy. The proposed approach, utilizing finite element method (FEM)-simulated datasets of liver stretching procedures, exhibits enhanced performance with root mean square errors (RMSE) of 2.78 ± 0.69 mm for hepatic veins and 1.81 ± 0.74 mm for portal veins. This method surpasses conventional techniques by 37.5% in accuracy and reduces inference time to 0.25 seconds. The optimized network exhibits a computation speed that is 83.9% faster than leading non-rigid registration algorithms. Subsequent tumour localization experiments demonstrate a targeting accuracy of 3.2 mm via vascular topology analysis, confirming clinical relevance. This research develops an effective framework for predicting deformation in real-time, providing a significant advancement for navigation in AR-guided hepatobiliary surgery.