Methods of Information in Medicine最新文献

{"title":"The Completeness of the Operating Room Data.","authors":"Päivi Nurmela, Minna Marjetta Mykkänen, Ulla-Mari Kinnunen","doi":"10.1055/a-2566-7958","DOIUrl":"https://doi.org/10.1055/a-2566-7958","url":null,"abstract":"<p><p>Background In the operating theatre, a large collection of data is collected at each visit. Some of this data is patient information, some is related to resource management, which is linked to hospital finances. Poor quality data leads to poor decisions, impacting patient safety and the continuity of care. Objectives The study aimed to evaluate the completeness of data documented within surgical operations, and based on the results, the goal is to improve data quality and identify data management improvement ideas. Methods The study was a quantitative evaluation of 33,684 surgical visits, focusing on data omissions. The organization identified 58 operating room data variables related to visits, procedures, resources, and personnel. Data completeness was evaluated for 36 variables, excluding 47 visits with missing 'Complete' flags. Data preprocessing was done using Python and Pandas, with pseudonymization of personnel names. Data analyzing was done using the R programming language. Data omissions were coded as '1' for missing values and '0' for others. Summary variables were created to indicate the number of personnel and procedure, and data omissions per visit. Results The average completeness of the operating room data was 98%, which is considered excellent. However, seven variables - the start and end date and time of anesthesia, type of treatment, personnel group and assistant information - had completeness below the 95% target level. 34% of the surgical visits contained at least one data omission. In the yearly comparison the completeness values of variables were statistically significantly higher in 2022 compared to 2023. Conclusion By ensuring existing quality assurance practices, verifying internal data maintenance and verifying and standardizing documenting practices the organization can achieve net benefits through improved data completeness, enhancing patient records, financial information and management. Improved data quality will also benefit national and international registries. Keywords: data, patient data, quality, health information system, operating room.</p>","PeriodicalId":49822,"journal":{"name":"Methods of Information in Medicine","volume":" ","pages":""},"PeriodicalIF":1.3,"publicationDate":"2025-03-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143732739","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Towards a national information model for medication orders in Sweden.","authors":"Sofie Holmeland, Tobias Blomberg, Andreas Mårtensson, Sabine Koch","doi":"10.1055/a-2546-4092","DOIUrl":"https://doi.org/10.1055/a-2546-4092","url":null,"abstract":"<p><strong>Background: </strong>Semantic interoperability among health information systems (HISs), in particular electronic health records (EHRs), is crucial for informed healthcare decisions and patient access to vital health data. However, inconsistent medication information and limited health data exchange contribute to medication errors worldwide. While Sweden offers various solutions for health information exchange, there is a limitation in the exchange of medication orders and a lack of understanding the structure of medication orders among EHRs, highlighting the need for further exploration of the structure of medication orders.</p><p><strong>Objectives: </strong>This study aims to develop a common information model of medication orders for EHRs to be used in the Swedish context.</p><p><strong>Methods: </strong>An explorative qualitative design study was conducted. Documents and reference models of how medication orders are structured were collected, and semi-structured interviews were conducted with five purposefully selected participants with insight into how medication orders are structured in Swedish EHRs. The data were analyzed using information needs analysis, information structure analysis, and code systems, classifications, and terminology analysis.</p><p><strong>Results: </strong>The following information areas were identified for a medication order: medication, medication indication, way of administration, medication order details, and dosage. These information areas were conceptualized into a developed Unified Modeling Language Class Diagram information model with defined classes, attributes, and data types. The resulting information model provides a representation of how medication orders are depicted in EHRs in Sweden and is aligned with existing national information models such as the National Medication List, while still providing additional information related to medication order details.</p><p><strong>Conclusions: </strong>The developed information model could potentially provide a national standardized model for medication orders, contributing to enhance semantic interoperability and improving data exchange across various HISs. This could enhance data consistency, reducing the risk of medication errors and thereby improving patient safety.</p>","PeriodicalId":49822,"journal":{"name":"Methods of Information in Medicine","volume":" ","pages":""},"PeriodicalIF":1.3,"publicationDate":"2025-02-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143525129","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Harnessing Advanced Machine Learning Techniques for Microscopic Vessel Segmentation in Pulmonary Fibrosis Using Novel Hierarchical Phase-Contrast Tomography (HiP-CT) Images.","authors":"Pardeep Vasudev, Moucheng Xu, Mehran Azimbagarad, Shahab Aslani, Yufei Wang, Robert Chapman, Hannah Coleman, Christopher Werlein, Claire Walsh, Peter Lee, Paul Tafforeau, Joseph Jacob","doi":"10.1055/a-2540-8166","DOIUrl":"https://doi.org/10.1055/a-2540-8166","url":null,"abstract":"<p><strong>Background: </strong>Fibrotic lung disease is a progressive illness that causes scarring and ultimately respiratory failure, with irreversible damage by the time its diagnosed on computed tomography imaging. Recent research postulates the role of the lung vasculature on the pathogenesis of the disease, and with the recent development of high-resolution hierarchical phase contrast tomography (HiP-CT), we have the potential to understand and detect changes in the lungs long before conventional imaging. However, to gain quantitative insight into vascular changes you first need to be able to segment the vessels before further downstream analysis can be conducted. Aside from this, HiP-CT generates large volume, high resolution data which is time consuming and expensive to label.</p><p><strong>Objectives: </strong>This project aims to qualitatively assess the latest machine learning methods for vessel segmentation in HiP-CT data to enable label propagation as the first step for imaging biomarker discovery, with the goal to identify early-stage interstitial lung disease amenable to treatment, before fibrosis begins.</p><p><strong>Methods: </strong>Semi-supervised learning has become a growing method to tackle sparsely labelled datasets due to its leveraging of unlabelled data. In this study we will compare 2 semi-supervised learning methods; Seg PL, based on pseudo labelling and MisMatch, using consistency regularisation against state of the art supervised learning method, in nnU-Net, on vessel segmentation in sparsely labelled lung HiP-CT data.</p><p><strong>Results: </strong>On initial experimentation, both MisMatch and SegPL showed promising performance on qualitative review. In comparison with supervised learning, both MisMatch and SegPL showed better on out of distribution performance within the same sample (different vessel morphology and texture vessels), though supervised learning provided more consistent segmentations for well represented labels in the limited annotations.</p><p><strong>Conclusion: </strong>Further quantitative research is required to better assess the generalisability of these findings, though they show promising first steps towards leveraging this novel data to tackle fibrotic lung disease.</p>","PeriodicalId":49822,"journal":{"name":"Methods of Information in Medicine","volume":" ","pages":""},"PeriodicalIF":1.3,"publicationDate":"2025-02-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143450734","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Deciphering Abbreviations in Malaysian Clinical Notes Using Machine Learning.","authors":"Ismat Mohd Sulaiman, Awang Bulgiba, Sameem Abdul Kareem, Abdul Aziz Latip","doi":"10.1055/a-2521-4372","DOIUrl":"10.1055/a-2521-4372","url":null,"abstract":"<p><strong>Objective: </strong> This is the first Malaysian machine learning model to detect and disambiguate abbreviations in clinical notes. The model has been designed to be incorporated into MyHarmony, a natural language processing system, that extracts clinical information for health care management. The model utilizes word embedding to ensure feasibility of use, not in real-time but for secondary analysis, within the constraints of low-resource settings.</p><p><strong>Methods: </strong> A Malaysian clinical embedding, based on Word2Vec model, was developed using 29,895 electronic discharge summaries. The embedding was compared against conventional rule-based and FastText embedding on two tasks: abbreviation detection and abbreviation disambiguation. Machine learning classifiers were applied to assess performance.</p><p><strong>Results: </strong> The Malaysian clinical word embedding contained 7 million word tokens, 24,352 unique vocabularies, and 100 dimensions. For abbreviation detection, the Decision Tree classifier augmented with the Malaysian clinical embedding showed the best performance (F-score of 0.9519). For abbreviation disambiguation, the classifier with the Malaysian clinical embedding had the best performance for most of the abbreviations (F-score of 0.9903).</p><p><strong>Conclusion: </strong> Despite having a smaller vocabulary and dimension, our local clinical word embedding performed better than the larger nonclinical FastText embedding. Word embedding with simple machine learning algorithms can decipher abbreviations well. It also requires lower computational resources and is suitable for implementation in low-resource settings such as Malaysia. The integration of this model into MyHarmony will improve recognition of clinical terms, thus improving the information generated for monitoring Malaysian health care services and policymaking.</p>","PeriodicalId":49822,"journal":{"name":"Methods of Information in Medicine","volume":" ","pages":""},"PeriodicalIF":1.3,"publicationDate":"2025-02-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143025162","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Leveraging Guideline-Based Clinical Decision Support Systems with Large Language Models: A Case Study with Breast Cancer.","authors":"Solène Delourme, Akram Redjdal, Jacques Bouaud, Brigitte Seroussi","doi":"10.1055/a-2528-4299","DOIUrl":"https://doi.org/10.1055/a-2528-4299","url":null,"abstract":"<p><strong>Background: </strong>Multidisciplinary tumor boards (MTBs) have been established in most countries to allow experts collaboratively determine the best treatment decisions for cancer patients. However, MTBs often face challenges such as case overload, which can compromise MTB decision quality. Clinical decision support systems (CDSSs) have been introduced to assist clinicians in this process. Despite their potential, CDSSs are still underutilized in routine practice. The emergence of large language models (LLMs), such as ChatGPT, offers new opportunities to improve the efficiency and usability of traditional clinical decision support systems (CDSSs).</p><p><strong>Objectives: </strong>OncoDoc2 is a guideline-based CDSS developed using a documentary approach and applied to breast cancer management. This study aims to evaluate the potential of LLMs, used as question-answering (QA) systems, to improve the usability of OncoDoc2 across different prompt engineering techniques (PETs).</p><p><strong>Methods: </strong>Data extracted from breast cancer patient summaries (BCPSs), together with questions formulated by OncoDoc2, were used to create prompts for various LLMs, and several PETs were designed and tested. Using a sample of 200 randomized BCPSs, LLMs and PETs were initially compared on their responses to OncoDoc2 questions using classic metrics (accuracy, precision, recall, and F1 score). Best performing LLMs and PETs were further assessed by comparing the therapeutic recommendations generated by OncoDoc2, based on LLM inputs, to those provided by MTB clinicians using OncoDoc2. Finally, the best performing method was validated using a new sample of 30 randomized BCPSs.</p><p><strong>Results: </strong>The combination of Mistral and OpenChat models under the enhanced zero-shot PET showed the best performance as a question-answering system. This approach gets a precision of 60.16%, a recall of 54.18%, an F1 Score of 56.59%, and an accuracy of 75.57% on the validation set of 30 BCPSs. However, this approach yielded poor results as a CDSS, with only 16.67% of the recommendations generated by OncoDoc2 based on LLM inputs matching the gold standard.</p><p><strong>Conclusions: </strong>All the criteria in the OncoDoc2 decision tree are crucial for capturing the uniqueness of each patient. Any deviation from a criterion alters the recommendations generated. Despite a good accuracy rate of 75.57% was achieved, LLMs still face challenges in reliably understanding complex medical contexts and be effective as CDSSs.</p>","PeriodicalId":49822,"journal":{"name":"Methods of Information in Medicine","volume":" ","pages":""},"PeriodicalIF":1.3,"publicationDate":"2025-01-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143069247","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"The Significance of Information Quality for the Secondary Use of the Information in the National Health Care Quality Registers in Finland.","authors":"Anna Frondelius, Ulla-Mari Kinnunen, Vesa Jormanainen","doi":"10.1055/a-2511-7866","DOIUrl":"10.1055/a-2511-7866","url":null,"abstract":"<p><strong>Background: </strong> The aim of the national health care quality registers is to monitor, assess, and improve the quality of care. The information utilized in quality registers must be of high quality to ensure that the information produced by the registers is reliable and useful. In Finland, one of the key sources of information for the quality registers is the national Kanta services.</p><p><strong>Objectives: </strong> The objective of the study was to increase understanding of the significance of information quality for the secondary use of the information in the national health care quality registers and to provide information on whether the information quality of the national Kanta services supports the information needs of the national quality registers, and how information quality should be developed.</p><p><strong>Methods: </strong> The research data were collected by interviewing six experts responsible for national health care quality registers, and it was analyzed using theory-driven qualitative content analysis based on the DeLone and McLean model.</p><p><strong>Results: </strong> Based on the results, the relevance of the information in the Kanta services met the information needs of the national quality registers. However, due to the limited amount of structured information and deficiencies in the completeness of the information, relevant information could not be fully utilized. Deficiencies in information quality posed challenges in information retrieval and hindered drawing conclusions in reporting. Challenges in information quality did not diminish the intention to use the information when information was considered relevant. Solutions to improve information quality included structuring, development of documentation practices, patient information systems and quality assurance, as well as collaboration among stakeholders.</p><p><strong>Conclusion: </strong> The Kanta services' information is relevant for the national health care quality registers, but developing the quality of the information, especially in terms of structures and completeness, is the key to fully enabling the secondary use of this information.</p>","PeriodicalId":49822,"journal":{"name":"Methods of Information in Medicine","volume":" ","pages":""},"PeriodicalIF":1.3,"publicationDate":"2025-01-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142957856","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Cross-lingual Natural Language Processing on Limited Annotated Case/Radiology Reports in English and Japanese: Insights from the Real-MedNLP Workshop.","authors":"Shuntaro Yada, Yuta Nakamura, Shoko Wakamiya, Eiji Aramaki","doi":"10.1055/a-2405-2489","DOIUrl":"10.1055/a-2405-2489","url":null,"abstract":"<p><strong>Background: </strong> Textual datasets (corpora) are crucial for the application of natural language processing (NLP) models. However, corpus creation in the medical field is challenging, primarily because of privacy issues with raw clinical data such as health records. Thus, the existing clinical corpora are generally small and scarce. Medical NLP (MedNLP) methodologies perform well with limited data availability.</p><p><strong>Objectives: </strong> We present the outcomes of the Real-MedNLP workshop, which was conducted using limited and parallel medical corpora. Real-MedNLP exhibits three distinct characteristics: (1) limited annotated documents: the training data comprise only a small set (∼100) of case reports (CRs) and radiology reports (RRs) that have been annotated. (2) Bilingually parallel: the constructed corpora are parallel in Japanese and English. (3) Practical tasks: the workshop addresses fundamental tasks, such as named entity recognition (NER) and applied practical tasks.</p><p><strong>Methods: </strong> We propose three tasks: NER of ∼100 available documents (Task 1), NER based only on annotation guidelines for humans (Task 2), and clinical applications (Task 3) consisting of adverse drug effect (ADE) detection for CRs and identical case identification (CI) for RRs.</p><p><strong>Results: </strong> Nine teams participated in this study. The best systems achieved 0.65 and 0.89 F1-scores for CRs and RRs in Task 1, whereas the top scores in Task 2 decreased by 50 to 70%. In Task 3, ADE reports were detected by up to 0.64 F1-score, and CI scored up to 0.96 binary accuracy.</p><p><strong>Conclusion: </strong> Most systems adopt medical-domain-specific pretrained language models using data augmentation methods. Despite the challenge of limited corpus size in Tasks 1 and 2, recent approaches are promising because the partial match scores reached ∼0.8-0.9 F1-scores. Task 3 applications revealed that the different availabilities of external language resources affected the performance per language.</p>","PeriodicalId":49822,"journal":{"name":"Methods of Information in Medicine","volume":" ","pages":""},"PeriodicalIF":1.3,"publicationDate":"2024-10-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142114054","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Deep Learning for Predicting Progression of Patellofemoral Osteoarthritis Based on Lateral Knee Radiographs, Demographic Data, and Symptomatic Assessments.","authors":"Neslihan Bayramoglu, Martin Englund, Ida K Haugen, Muneaki Ishijima, Simo Saarakkala","doi":"10.1055/a-2305-2115","DOIUrl":"10.1055/a-2305-2115","url":null,"abstract":"<p><strong>Objective: </strong>In this study, we propose a novel framework that utilizes deep learning and attention mechanisms to predict the radiographic progression of patellofemoral osteoarthritis (PFOA) over a period of 7 years.</p><p><strong>Material and methods: </strong>This study included subjects (1,832 subjects, 3,276 knees) from the baseline of the Multicenter Osteoarthritis Study (MOST). Patellofemoral joint regions of interest were identified using an automated landmark detection tool (BoneFinder) on lateral knee X-rays. An end-to-end deep learning method was developed for predicting PFOA progression based on imaging data in a five-fold cross-validation setting. To evaluate the performance of the models, a set of baselines based on known risk factors were developed and analyzed using gradient boosting machine (GBM). Risk factors included age, sex, body mass index, and Western Ontario and McMaster Universities Arthritis Index score, and the radiographic osteoarthritis stage of the tibiofemoral joint (Kellgren and Lawrence [KL] score). Finally, to increase predictive power, we trained an ensemble model using both imaging and clinical data.</p><p><strong>Results: </strong>Among the individual models, the performance of our deep convolutional neural network attention model achieved the best performance with an area under the receiver operating characteristic curve (AUC) of 0.856 and average precision (AP) of 0.431, slightly outperforming the deep learning approach without attention (AUC = 0.832, AP = 0.4) and the best performing reference GBM model (AUC = 0.767, AP = 0.334). The inclusion of imaging data and clinical variables in an ensemble model allowed statistically more powerful prediction of PFOA progression (AUC = 0.865, AP = 0.447), although the clinical significance of this minor performance gain remains unknown. The spatial attention module improved the predictive performance of the backbone model, and the visual interpretation of attention maps focused on the joint space and the regions where osteophytes typically occur.</p><p><strong>Conclusion: </strong>This study demonstrated the potential of machine learning models to predict the progression of PFOA using imaging and clinical variables. These models could be used to identify patients who are at high risk of progression and prioritize them for new treatments. However, even though the accuracy of the models were excellent in this study using the MOST dataset, they should be still validated using external patient cohorts in the future.</p>","PeriodicalId":49822,"journal":{"name":"Methods of Information in Medicine","volume":" ","pages":"1-10"},"PeriodicalIF":1.3,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11495941/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140854286","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Development and Validation of a Natural Language Processing Algorithm to Pseudonymize Documents in the Context of a Clinical Data Warehouse.","authors":"Xavier Tannier, Perceval Wajsbürt, Alice Calliger, Basile Dura, Alexandre Mouchet, Martin Hilka, Romain Bey","doi":"10.1055/s-0044-1778693","DOIUrl":"10.1055/s-0044-1778693","url":null,"abstract":"<p><strong>Objective: </strong>The objective of this study is to address the critical issue of deidentification of clinical reports to allow access to data for research purposes, while ensuring patient privacy. The study highlights the difficulties faced in sharing tools and resources in this domain and presents the experience of the Greater Paris University Hospitals (AP-HP for Assistance Publique-Hôpitaux de Paris) in implementing a systematic pseudonymization of text documents from its Clinical Data Warehouse.</p><p><strong>Methods: </strong>We annotated a corpus of clinical documents according to 12 types of identifying entities and built a hybrid system, merging the results of a deep learning model as well as manual rules.</p><p><strong>Results and discussion: </strong>Our results show an overall performance of 0.99 of F1-score. We discuss implementation choices and present experiments to better understand the effort involved in such a task, including dataset size, document types, language models, or rule addition. We share guidelines and code under a 3-Clause BSD license.</p>","PeriodicalId":49822,"journal":{"name":"Methods of Information in Medicine","volume":" ","pages":"21-34"},"PeriodicalIF":1.3,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11495938/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140040727","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Does Differentially Private Synthetic Data Lead to Synthetic Discoveries?","authors":"Ileana Montoya Perez, Parisa Movahedi, Valtteri Nieminen, Antti Airola, Tapio Pahikkala","doi":"10.1055/a-2385-1355","DOIUrl":"10.1055/a-2385-1355","url":null,"abstract":"<p><strong>Background: </strong>Synthetic data have been proposed as a solution for sharing anonymized versions of sensitive biomedical datasets. Ideally, synthetic data should preserve the structure and statistical properties of the original data, while protecting the privacy of the individual subjects. Differential Privacy (DP) is currently considered the gold standard approach for balancing this trade-off.</p><p><strong>Objectives: </strong>The aim of this study is to investigate how trustworthy are group differences discovered by independent sample tests from DP-synthetic data. The evaluation is carried out in terms of the tests' Type I and Type II errors. With the former, we can quantify the tests' validity, i.e., whether the probability of false discoveries is indeed below the significance level, and the latter indicates the tests' power in making real discoveries.</p><p><strong>Methods: </strong>We evaluate the Mann-Whitney U test, Student's <i>t</i>-test, chi-squared test, and median test on DP-synthetic data. The private synthetic datasets are generated from real-world data, including a prostate cancer dataset (<i>n</i> = 500) and a cardiovascular dataset (<i>n</i> = 70,000), as well as on bivariate and multivariate simulated data. Five different DP-synthetic data generation methods are evaluated, including two basic DP histogram release methods and MWEM, Private-PGM, and DP GAN algorithms.</p><p><strong>Conclusion: </strong>A large portion of the evaluation results expressed dramatically inflated Type I errors, especially at levels of <i>ϵ</i> ≤ 1. This result calls for caution when releasing and analyzing DP-synthetic data: low <i>p</i>-values may be obtained in statistical tests simply as a byproduct of the noise added to protect privacy. A DP Smoothed Histogram-based synthetic data generation method was shown to produce valid Type I error for all privacy levels tested but required a large original dataset size and a modest privacy budget (<i>ϵ</i> ≥ 5) in order to have reasonable Type II error levels.</p>","PeriodicalId":49822,"journal":{"name":"Methods of Information in Medicine","volume":" ","pages":"35-51"},"PeriodicalIF":1.3,"publicationDate":"2024-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11495942/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141977081","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}