Accuracy of Artificial Intelligence for Gatekeeping in Referrals to Specialized Care.

Abstract

Importance: Integrating artificial intelligence (AI) technologies into gatekeeping holds significant potential, as it efficiently handles repetitive tasks and can process large amounts of information quickly.

Objective: To develop and assess the accuracy of an AI model that enhances the gatekeeping process for referrals to specialized care.

Design, setting, and participants: This diagnostic study comprised referrals from primary care to endocrinology, gastroenterology, proctology, rheumatology, and urology from a retrospective administrative database of patients in Brazil between June 2016 and April 2019. Analysis was performed between December 2022 and July 2024.

Main outcomes and measures: The algorithm's development and testing comprised 2 stages. Multiple AI models were initially evaluated to train and test the algorithm for categorizing referrals as authorizing or requiring additional information. Subsequently, the model's performance was assessed against an independent set of referrals. Additionally, the current (human) evaluations of gatekeepers were evaluated against the standard. The reference standard was the consensus of 2 physicians with extensive experience. Accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC-ROC) were assessed.

Results: The electronic system retrieved 45 039 eligible referrals for the development stage (mean [SD] patient age, 51.9 [15.8] years; 25 458 women [56.5%]). An algorithm utilizing word embeddings and a neural network proved the most effective. In the second phase, 1750 referrals (350 for each specialty) showed a 32% authorization rate according to the reference standard. The AI model achieved an overall accuracy of 0.716 (95% IC, 0.694-0.737), with a sensitivity of 0.542 (95% CI, 0.501 to 0.582) and specificity of 0.801 (95% CI, 0.777 to 0.822). Regarding individual specialties, rheumatology exhibited the highest accuracy (0.811; 95% IC, 0.767-0.849), while proctology had the lowest (0.649; 95% IC, 0.597-0.697). The overall AUC-ROC was 0.765 (95% IC, 0.742-0.788). When compared against the consensus standard, the AI model had higher accuracy and specificity and lower sensitivity than the current approach.

Conclusions and relevance: In this diagnostic study of referral data, a novel AI model effectively distinguished between referrals that warranted immediate authorization and those that required further information with moderate accuracy; it had higher specificity and lower sensitivity than gatekeepers decisions. Implementing this AI model in the gatekeeping process should combine human judgment and AI support to optimize the referral process.