3D colour multimodality fusion imaging and preoperative augmented reality for neurosurgeons training

Neurosurgery residents face significant challenges in acquiring comprehensive surgical skills due to limited cadaveric resources and demanding clinical schedules, while anatomy textbooks and neuroimaging remain valuable references. Developing a precise three-dimensional (3D) mental representation from two-dimensional (2D) neuroimaging remains a critical hurdle, even for experienced ones. 3D multimodality fusion imaging (MFI) and augmented reality (AR) techniques have made it easier to understand the complex anatomy of the brain and shorten the learning curve. Conventional 3D-MFI, while valuable, suffers from uniform coloration, obscuring anatomical boundaries and functional subdivisions critical for surgical planning. The 3D colour brain functional partitioning MFI is a new educational tool that has emerged as a transformative resource for neurosurgeons.¹

3D colour brain segmentation technique was developed and applied to 33 patients with extracerebral tumours by integrating 3D-T1-weighted MRI, diffusion tensor imaging and CT angiography, named 3D-colour-MFI.¹ A neurosurgical resident, trained in software, used 3DSlicer and FreeSurfer to generate conventional 3D and colour-MFI models for all patients, enabling accurate tumour localisation, identification of peritumoural structures, and clarification of neurovascular relationships.

Another 15 neurosurgical residents were trained to use 3DSlicer, allowing them to manipulate 3D-MFI models. They simulated surgeries with conventional 3D and colour-MFI models, respectively, comparing to intraoperative videos to assess anatomical accuracy and procedural realism between the two modalities, and the differences were documented in questionnaires. Additionally, surgical simulation screenshots were used to discuss procedures with experienced surgeons and conduct preoperative communication with patients.

Neurosurgery residents who viewed the intraoperative video noted that the 3D-colour-MFI model significantly enhanced their understanding of surgical anatomy, helping them accurately identify brain structures and functions and facilitating more effective communication with patients, increasing patient trust and reducing patient anxiety. Experienced surgeons also provided positive feedback, stating that the 3D-colour-MFI model improved their ability to visualise complex anatomical structures more clearly.

Integrating 3D neuroimaging with intraoperative observations is crucial for residents' learning and effective contributions during procedures. 3D-colour-MFI enhances surgical understanding and prepares trainees to become proficient neurosurgeons. This pilot project integrates 3D-colour-MFI into neurosurgical education, improving lesion localisation, peritumoural tissue assessment and patient communication. Although powerful free 3D medical software such as 3DSlicer, ITK-SNAP and FreeSurfer are available, proficiency in neuroimaging and neurosurgery remain essential for creating 3D-colour-MFI models. Additionally, multiple software training sessions are typically required to effectively implement and promote the use of 3D-colour-MFI in other centres.

In all, the 3D-colour-MFI adds colour coding to differentiate brain areas and fuses multiple image modalities. This helps residents in understanding complex brain structures, refining tumour localisation and facilitating preoperative discussions. Despite being a small, single-centre study focused on extracerebral tumours, the 3D-colour-MFI approach holds transformative potential for neurosurgical training and patient care. Expanding the use of 3D-colour-MFI in various training centres, coupled with standardised training programmes, could further enhance its application in neurosurgical practice. Future studies should also expand into the field of intracerebral tumours.

Xiaolin Hou: Conceptualization; methodology; writing – original draft. Ruxiang Xu: Project administration; supervision; writing – review and editing. Dongdong Yang: Investigation; visualization; validation. Dingjun Li: Data curation; formal analysis.