Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting: the DECIPHER study.

IF 4.4 Q1 HEALTH CARE SCIENCES & SERVICES

BMJ Health & Care Informatics Pub Date : 2025-07-25 DOI:10.1136/bmjhci-2025-101433

Ben Bloom, Adrian Haimovich, Jason Pott, Sophie L Williams, Michael Cheetham, Sandra Langsted, Imogen Skene, Raine Astin-Chamberlain, Stephen H Thomas

{"title":"Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting: the DECIPHER study.","authors":"Ben Bloom, Adrian Haimovich, Jason Pott, Sophie L Williams, Michael Cheetham, Sandra Langsted, Imogen Skene, Raine Astin-Chamberlain, Stephen H Thomas","doi":"10.1136/bmjhci-2025-101433","DOIUrl":null,"url":null,"abstract":"Objectives: Identifying whether there is a traumatic intracranial bleed (ICB+) on head CT is critical for clinical care and research. Free text CT reports are unstructured and therefore must undergo time-consuming manual review. Existing artificial intelligence classification schemes are not optimised for the emergency department endpoint of classification of ICB+ or ICB-. We sought to assess three methods for classifying CT reports: a text classification (TC) programme, a commercial natural language processing programme (Clinithink) and a generative pretrained transformer large language model (Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting (DECIPHER)-LLM).Methods: Primary objective: determine the diagnostic classification performance of the dichotomous categorisation of each of the three approaches.Secondary objective: determine whether the LLM could achieve a substantial reduction in CT report review workload while maintaining 100% sensitivity.Anonymised radiology reports of head CT scans performed for trauma were manually labelled as ICB+/-. Training and validation sets were randomly created to train the TC and natural language processing models. Prompts were written to train the LLM.Results: 898 reports were manually labelled. Sensitivity and specificity (95% CI)) of TC, Clinithink and DECIPHER-LLM (with probability of ICB set at 10%) were respectively 87.9% (76.7% to 95.0%) and 98.2% (96.3% to 99.3%), 75.9% (62.8% to 86.1%) and 96.2% (93.8% to 97.8%) and 100% (93.8% to 100%) and 97.4% (95.3% to 98.8%).With DECIPHER-LLM probability of ICB+ threshold of 10% set to identify CT reports requiring manual evaluation, CT reports requiring manual classification reduced by an estimated 385/449 cases (85.7% (95% CI 82.1% to 88.9%)) while maintaining 100% sensitivity.Discussion and conclusion: DECIPHER-LLM outperformed other tested free-text classification methods.","PeriodicalId":9050,"journal":{"name":"BMJ Health & Care Informatics","volume":"32 1","pages":""},"PeriodicalIF":4.4000,"publicationDate":"2025-07-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12306305/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"BMJ Health & Care Informatics","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1136/bmjhci-2025-101433","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}

引用次数: 0

Abstract

Objectives: Identifying whether there is a traumatic intracranial bleed (ICB+) on head CT is critical for clinical care and research. Free text CT reports are unstructured and therefore must undergo time-consuming manual review. Existing artificial intelligence classification schemes are not optimised for the emergency department endpoint of classification of ICB+ or ICB-. We sought to assess three methods for classifying CT reports: a text classification (TC) programme, a commercial natural language processing programme (Clinithink) and a generative pretrained transformer large language model (Digitalizing English-language CT Interpretation for Positive Haemorrhage Evaluation Reporting (DECIPHER)-LLM).

Methods: Primary objective: determine the diagnostic classification performance of the dichotomous categorisation of each of the three approaches.

Secondary objective: determine whether the LLM could achieve a substantial reduction in CT report review workload while maintaining 100% sensitivity.Anonymised radiology reports of head CT scans performed for trauma were manually labelled as ICB+/-. Training and validation sets were randomly created to train the TC and natural language processing models. Prompts were written to train the LLM.

Results: 898 reports were manually labelled. Sensitivity and specificity (95% CI)) of TC, Clinithink and DECIPHER-LLM (with probability of ICB set at 10%) were respectively 87.9% (76.7% to 95.0%) and 98.2% (96.3% to 99.3%), 75.9% (62.8% to 86.1%) and 96.2% (93.8% to 97.8%) and 100% (93.8% to 100%) and 97.4% (95.3% to 98.8%).With DECIPHER-LLM probability of ICB+ threshold of 10% set to identify CT reports requiring manual evaluation, CT reports requiring manual classification reduced by an estimated 385/449 cases (85.7% (95% CI 82.1% to 88.9%)) while maintaining 100% sensitivity.

Discussion and conclusion: DECIPHER-LLM outperformed other tested free-text classification methods.

查看原文本刊更多论文

数字化英语CT解释阳性出血评估报告：破译研究。

目的：确定头部CT上是否存在外伤性颅内出血（ICB+）对临床护理和研究具有重要意义。自由文本CT报告是非结构化的，因此必须经过耗时的人工审查。现有人工智能分类方案未针对ICB+或ICB-分类的急诊科终点进行优化。我们试图评估CT报告分类的三种方法：文本分类（TC）程序，商业自然语言处理程序（clininthink）和生成预训练的变形大语言模型（数字化英语CT解释阳性出血评估报告(DECIPHER)-LLM）。方法：主要目的：确定三种方法的二分类诊断分类性能。次要目标：确定LLM是否能够在保持100%灵敏度的同时大幅减少CT报告审查工作量。头颅CT扫描的匿名放射学报告被手工标记为ICB+/-。随机创建训练集和验证集来训练TC和自然语言处理模型。写提示是为了训练法学硕士。结果：898份报告手工标记。TC、clininithink和DECIPHER-LLM （ICB概率设为10%）的敏感性和特异性（95% CI）分别为87.9%（76.7% ~ 95.0%）和98.2%(96.3% ~ 99.3%),75.9%（62.8% ~ 86.1%）和96.2%(93.8% ~ 97.8%),100%（93.8% ~ 100%）和97.4%（95.3% ~ 98.8%）。通过将cipher - llm的ICB+阈值概率设置为10%来识别需要人工评估的CT报告，需要人工分类的CT报告估计减少了385/449例（85.7% (95% CI 82.1%至88.9%)），同时保持100%的敏感性。讨论与结论：DECIPHER-LLM优于其他经过测试的自由文本分类方法。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊