Diagnostic and prognostic research最新文献

{"title":"A scoping review of causal methods enabling predictions under hypothetical interventions.","authors":"Lijing Lin, Matthew Sperrin, David A Jenkins, Glen P Martin, Niels Peek","doi":"10.1186/s41512-021-00092-9","DOIUrl":"10.1186/s41512-021-00092-9","url":null,"abstract":"<p><strong>Background: </strong>The methods with which prediction models are usually developed mean that neither the parameters nor the predictions should be interpreted causally. For many applications, this is perfectly acceptable. However, when prediction models are used to support decision making, there is often a need for predicting outcomes under hypothetical interventions.</p><p><strong>Aims: </strong>We aimed to identify published methods for developing and validating prediction models that enable risk estimation of outcomes under hypothetical interventions, utilizing causal inference. We aimed to identify the main methodological approaches, their underlying assumptions, targeted estimands, and potential pitfalls and challenges with using the method. Finally, we aimed to highlight unresolved methodological challenges.</p><p><strong>Methods: </strong>We systematically reviewed literature published by December 2019, considering papers in the health domain that used causal considerations to enable prediction models to be used for predictions under hypothetical interventions. We included both methodologies proposed in statistical/machine learning literature and methodologies used in applied studies.</p><p><strong>Results: </strong>We identified 4919 papers through database searches and a further 115 papers through manual searches. Of these, 87 papers were retained for full-text screening, of which 13 were selected for inclusion. We found papers from both the statistical and the machine learning literature. Most of the identified methods for causal inference from observational data were based on marginal structural models and g-estimation.</p><p><strong>Conclusions: </strong>There exist two broad methodological approaches for allowing prediction under hypothetical intervention into clinical prediction models: (1) enriching prediction models derived from observational studies with estimated causal effects from clinical trials and meta-analyses and (2) estimating prediction models and causal effects directly from observational data. These methods require extending to dynamic treatment regimes, and consideration of multiple interventions to operationalise a clinical decision support system. Techniques for validating 'causal prediction models' are still in their infancy.</p>","PeriodicalId":72800,"journal":{"name":"Diagnostic and prognostic research","volume":"5 1","pages":"3"},"PeriodicalIF":0.0,"publicationDate":"2021-02-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7860039/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"10291708","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Multivariate prediction of mixed, multilevel, sequential outcomes arising from in vitro fertilisation.","authors":"Jack Wilkinson,&nbsp;Andy Vail,&nbsp;Stephen A Roberts","doi":"10.1186/s41512-020-00091-2","DOIUrl":"https://doi.org/10.1186/s41512-020-00091-2","url":null,"abstract":"<p><p>In vitro fertilisation (IVF) comprises a sequence of interventions concerned with the creation and culture of embryos which are then transferred to the patient's uterus. While the clinically important endpoint is birth, the responses to each stage of treatment contain additional information about the reasons for success or failure. As such, the ability to predict not only the overall outcome of the cycle, but also the stage-specific responses, can be useful. This could be done by developing separate models for each response variable, but recent work has suggested that it may be advantageous to use a multivariate approach to model all outcomes simultaneously. Here, joint analysis of the sequential responses is complicated by mixed outcome types defined at two levels (patient and embryo). A further consideration is whether and how to incorporate information about the response at each stage in models for subsequent stages. We develop a case study using routinely collected data from a large reproductive medicine unit in order to investigate the feasibility and potential utility of multivariate prediction in IVF. We consider two possible scenarios. In the first, stage-specific responses are to be predicted prior to treatment commencement. In the second, responses are predicted dynamically, using the outcomes of previous stages as predictors. In both scenarios, we fail to observe benefits of joint modelling approaches compared to fitting separate regression models for each response variable.</p>","PeriodicalId":72800,"journal":{"name":"Diagnostic and prognostic research","volume":" ","pages":"2"},"PeriodicalIF":0.0,"publicationDate":"2021-01-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1186/s41512-020-00091-2","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"38840971","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?","authors":"David A Jenkins, Glen P Martin, Matthew Sperrin, Richard D Riley, Thomas P A Debray, Gary S Collins, Niels Peek","doi":"10.1186/s41512-020-00090-3","DOIUrl":"10.1186/s41512-020-00090-3","url":null,"abstract":"<p><p>Clinical prediction models (CPMs) have become fundamental for risk stratification across healthcare. The CPM pipeline (development, validation, deployment, and impact assessment) is commonly viewed as a one-time activity, with model updating rarely considered and done in a somewhat ad hoc manner. This fails to address the fact that the performance of a CPM worsens over time as natural changes in populations and care pathways occur. CPMs need constant surveillance to maintain adequate predictive performance. Rather than reactively updating a developed CPM once evidence of deteriorated performance accumulates, it is possible to proactively adapt CPMs whenever new data becomes available. Approaches for validation then need to be changed accordingly, making validation a continuous rather than a discrete effort. As such, \"living\" (dynamic) CPMs represent a paradigm shift, where the analytical methods dynamically generate updated versions of a model through time; one then needs to validate the system rather than each subsequent model revision.</p>","PeriodicalId":72800,"journal":{"name":"Diagnostic and prognostic research","volume":" ","pages":"1"},"PeriodicalIF":0.0,"publicationDate":"2021-01-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7797885/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"38807699","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Protocol for the development and validation of a risk prediction model for stillbirths from 35 weeks gestation in Australia.","authors":"Jessica K Sexton,&nbsp;Michael Coory,&nbsp;Sailesh Kumar,&nbsp;Gordon Smith,&nbsp;Adrienne Gordon,&nbsp;Georgina Chambers,&nbsp;Gavin Pereira,&nbsp;Camille Raynes-Greenow,&nbsp;Lisa Hilder,&nbsp;Philippa Middleton,&nbsp;Anneka Bowman,&nbsp;Scott N Lieske,&nbsp;Kara Warrilow,&nbsp;Jonathan Morris,&nbsp;David Ellwood,&nbsp;Vicki Flenady","doi":"10.1186/s41512-020-00089-w","DOIUrl":"https://doi.org/10.1186/s41512-020-00089-w","url":null,"abstract":"<p><strong>Background: </strong>Despite advances in the care of women and their babies in the past century, an estimated 1.7 million babies are born still each year throughout the world. A robust method to estimate a pregnant woman's individualized risk of late-pregnancy stillbirth is needed to inform decision-making around the timing of birth to reduce the risk of stillbirth from 35 weeks of gestation in Australia, a high-resource setting.</p><p><strong>Methods: </strong>This is a protocol for a cross-sectional study of all late-pregnancy births in Australia (2005-2015) from 35 weeks of gestation including 5188 stillbirths among 3.1 million births at an estimated rate of 1.7 stillbirths per 1000 births. A multivariable logistic regression model will be developed in line with current Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines to estimate the gestation-specific probability of stillbirth with prediction intervals. Candidate predictors were identified from systematic reviews and clinical consultation and will be described through univariable regression analysis. To generate a final model, elimination by backward stepwise multivariable logistic regression will be performed. The model will be internally validated using bootstrapping with 1000 repetitions and externally validated using a temporally unique dataset. Overall model performance will be assessed with R², calibration, and discrimination. Calibration will be reported using a calibration plot with 95% confidence intervals (α = 0.05). Discrimination will be measured by the C-statistic and area underneath the receiver-operator curves. Clinical usefulness will be reported as positive and negative predictive values, and a decision curve analysis will be considered.</p><p><strong>Discussion: </strong>A robust method to predict a pregnant woman's individualized risk of late-pregnancy stillbirth is needed to inform timely, appropriate care to reduce stillbirth. Among existing prediction models designed for obstetric use, few have been subject to internal and external validation and many fail to meet recommended reporting standards. In developing a risk prediction model for late-gestation stillbirth with both providers and pregnant women in mind, we endeavor to develop a validated model for clinical use in Australia that meets current reporting standards.</p>","PeriodicalId":72800,"journal":{"name":"Diagnostic and prognostic research","volume":"4 1","pages":"21"},"PeriodicalIF":0.0,"publicationDate":"2020-12-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1186/s41512-020-00089-w","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"38726380","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Lung cancer screening by low-dose computed tomography: a cost-effectiveness analysis of alternative programmes in the UK using a newly developed natural history-based economic model.","authors":"Edward Griffin,&nbsp;Chris Hyde,&nbsp;Linda Long,&nbsp;Jo Varley-Campbell,&nbsp;Helen Coelho,&nbsp;Sophie Robinson,&nbsp;Tristan Snowsill","doi":"10.1186/s41512-020-00087-y","DOIUrl":"https://doi.org/10.1186/s41512-020-00087-y","url":null,"abstract":"<p><strong>Background: </strong>A systematic review of economic evaluations for lung cancer identified no economic models of the UK setting based on disease natural history. We first sought to develop a new model of natural history for population screening, then sought to explore the cost-effectiveness of multiple alternative potential programmes.</p><p><strong>Methods: </strong>An individual patient model (ENaBL) was constructed in MS Excel® and calibrated against data from the US National Lung Screening Trial. Costs were taken from the UK Lung Cancer Screening Trial and took the perspective of the NHS and PSS. Simulants were current or former smokers aged between 55 and 80 years and so at a higher risk of lung cancer relative to the general population. Subgroups were defined by further restricting age and risk of lung cancer as predicted by patient self-questionnaire. Programme designs were single, triple, annual and biennial arrangements of LDCT screens, thereby examining number and interval length. Forty-eight distinct screening strategies were compared to the current practice of no screening. The primary outcome was incremental cost-effectiveness of strategies (additional cost per QALY gained).</p><p><strong>Results: </strong>LDCT screening is predicted to bring forward the stage distribution at diagnosis and reduce lung cancer mortality, with decreases versus no screening ranging from 4.2 to 7.7% depending on screen frequency. Overall healthcare costs are predicted to increase; treatment cost savings from earlier detection are outweighed by the costs of over-diagnosis. Single-screen programmes for people 55-75 or 60-75 years with ≥ 3% predicted lung cancer risk may be cost-effective at the £30,000 per QALY threshold (respective ICERs of £28,784 and £28,169 per QALY gained). Annual and biennial screening programmes were not predicted to be cost-effective at any cost-effectiveness threshold.</p><p><strong>Limitations: </strong>LDCT performance was unaffected by lung cancer type, stage or location and the impact of a national screening programme of smoking behaviour was not included.</p><p><strong>Conclusion: </strong>Lung cancer screening may not be cost-effective at the threshold of £20,000 per QALY commonly used in the UK but may be cost-effective at the higher threshold of £30,000 per QALY.</p>","PeriodicalId":72800,"journal":{"name":"Diagnostic and prognostic research","volume":"4 1","pages":"20"},"PeriodicalIF":0.0,"publicationDate":"2020-12-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1186/s41512-020-00087-y","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"38351106","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Critical appraisal and external validation of a prognostic model for survival of people living with HIV/AIDS who underwent antiretroviral therapy.","authors":"Junfeng Wang,&nbsp;Tanwei Yuan,&nbsp;Xuemei Ling,&nbsp;Quanmin Li,&nbsp;Xiaoping Tang,&nbsp;Weiping Cai,&nbsp;Huachun Zou,&nbsp;Linghua Li","doi":"10.1186/s41512-020-00088-x","DOIUrl":"https://doi.org/10.1186/s41512-020-00088-x","url":null,"abstract":"<p><strong>Background: </strong>HIV/AIDS remains a leading cause of death worldwide. Recently, a model has been developed in Wenzhou, China, to predict the survival of people living with HIV/AIDS (PLWHA) who underwent antiretroviral therapy (ART). We aimed to evaluate the methodological quality and validate the model in an external population-based cohort.</p><p><strong>Methods: </strong>Prediction Model Risk of Bias Assessment Tool (PROBAST) was used to assess the risk of bias of the Wenzhou model. Data were from the National Free Antiretroviral Treatment Program database. We included PLWHA treated between February 2004 and December 2019 in a tertiary hospital in Guangzhou city, China. The endpoint was all-cause deaths and assessed until January 2020. We assessed the discrimination performance of the model by Harrell's overall C-statistics and time-dependent C-statistics and calibration by comparing observed survival probabilities estimated with the Kaplan-Meier method versus predicted survival probabilities. To assess the potential prediction value of age and gender which were precluded in developing the Wenzhou model, we compared the discriminative ability of the original model with an extended model added with age and gender.</p><p><strong>Results: </strong>Based on PROBAST, the Wenzhou model was rated as high risk of bias in three out of the four domains (selection of participants, definition of outcome, and methods for statistical analysis) mainly because of the misuse of nested case-control design and propensity score matching. In the external validation analysis, 16758 patients were included, among whom 743 patients died (mortality rate 11.41 per 1000 person-years) during follow-up (median 3.41 years, interquartile range 1.64-5.62). The predictor of HIV viral load was missing in 14361 patients (85.7%). The discriminative ability of the Wenzhou model decreased in the external dataset, with the Harrell's overall C-statistics being 0.76, and time-dependent C-statistics dropping from 0.81 at 6 months to 0.48 at 10 years after ART initiation. The model consistently underestimated the survival, and the level was 6.23%, 10.02%, and 14.82% at 1, 2, and 3 years after ART initiation, respectively. The overall and time-dependent discriminative ability of the model improved after adding age and gender to the original model.</p><p><strong>Conclusion: </strong>The Wenzhou prognostic model is at high risk of bias in model development, with inadequate model performance in external validation. Thereby, we could not confirm the validity and extended utility of the Wenzhou model. Future prediction model development and validation studies need to comply with the methodological standards and guidelines specifically developed for prediction models.</p>","PeriodicalId":72800,"journal":{"name":"Diagnostic and prognostic research","volume":"4 1","pages":"19"},"PeriodicalIF":0.0,"publicationDate":"2020-11-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1186/s41512-020-00088-x","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"38687532","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A study protocol for a predictive algorithm to assess population-based premature mortality risk: Premature Mortality Population Risk Tool (PreMPoRT).","authors":"Laura C Rosella, Meghan O'Neill, Stacey Fisher, Mackenzie Hurst, Lori Diemert, Kathy Kornas, Andy Hong, Douglas G Manuel","doi":"10.1186/s41512-020-00086-z","DOIUrl":"10.1186/s41512-020-00086-z","url":null,"abstract":"<p><strong>Background: </strong>Premature mortality is an important population health indicator used to assess health system functioning and to identify areas in need of health system intervention. Predicting the future incidence of premature mortality in the population can facilitate initiatives that promote equitable health policies and effective delivery of public health services. This study protocol proposes the development and validation of the Premature Mortality Risk Prediction Tool (PreMPoRT) that will predict the incidence of premature mortality using large population-based community health surveys and multivariable modeling approaches.</p><p><strong>Methods: </strong>PreMPoRT will be developed and validated using various training, validation, and test data sets generated from the six cycles of the Canadian Community Health Survey (CCHS) linked to the Canadian Vital Statistics Database from 2000 to 2017. Population-level risk factor information on demographic characteristics, health behaviors, area level measures, and other health-related factors will be used to develop PreMPoRT and to predict the incidence of premature mortality, defined as death prior to age 75, over a 5-year period. Sex-specific Weibull accelerated failure time models will be developed using a Canadian provincial derivation cohort consisting of approximately 500,000 individuals, with approximately equal proportion of males and females, and about 12,000 events of premature mortality. External validation will be performed using separate linked files (CCHS cycles 2007-2008, 2009-2010, and 2011-2012) from the development cohort (CCHS cycles 2000-2001, 2003-2004, and 2005-2006) to check the robustness of the prediction model. Measures of overall predictive performance (e.g., Nagelkerke's R²), calibration (e.g., calibration plots), and discrimination (e.g., Harrell's concordance statistic) will be assessed, including calibration within defined subgroups of importance to knowledge users and policymakers.</p><p><strong>Discussion: </strong>Using routinely collected risk factor information, we anticipate that PreMPoRT will produce population-based estimates of premature mortality and will be used to inform population strategies for prevention.</p>","PeriodicalId":72800,"journal":{"name":"Diagnostic and prognostic research","volume":"4 1","pages":"18"},"PeriodicalIF":0.0,"publicationDate":"2020-11-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7640636/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"38691683","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A study protocol for the development of a multivariable model predicting 6- and 12-month mortality for people with dementia living in residential aged care facilities (RACFs) in Australia.","authors":"Ross Bicknell, Wen Kwang Lim, Andrea B Maier, Dina LoGiuidice","doi":"10.1186/s41512-020-00085-0","DOIUrl":"10.1186/s41512-020-00085-0","url":null,"abstract":"<p><strong>Background: </strong>For residential aged care facility (RACF) residents with dementia, lack of prognostic guidance presents a significant challenge for end of life care planning. In an attempt to address this issue, models have been developed to assess mortality risk for people with advanced dementia, predominantly using long-term care minimum data set (MDS) information from the USA. A limitation of these models is that the information contained within the MDS used for model development was not collected for the purpose of identifying prognostic factors. The models developed using MDS data have had relatively modest ability to discriminate mortality risk and are difficult to apply outside the MDS setting. This study will aim to develop a model to estimate 6- and 12-month mortality risk for people with dementia from prognostic indicators recorded during usual clinical care provided in RACFs in Australia.</p><p><strong>Methods: </strong>A secondary analysis will be conducted for a cohort of people with dementia from RACFs participating in a cluster-randomized trial of a palliative care education intervention (IMPETUS-D). Ten prognostic indicator variables were identified based on a literature review of clinical features associated with increased mortality for people with dementia living in RACFs. Variables will be extracted from RACF files at baseline and mortality measured at 6 and 12 months after baseline data collection. A multivariable logistic regression model will be developed for 6- and 12-month mortality outcome measures using backwards elimination with a fractional polynomial approach for continuous variables. Internal validation will be undertaken using bootstrapping methods. Discrimination of the model for 6- and 12-month mortality will be presented as receiver operating curves with c statistics. Calibration curves will be presented comparing observed and predicted event rates for each decile of risk as well as flexible calibration curves derived using loess-based functions.</p><p><strong>Discussion: </strong>The model developed in this study aims to improve clinical assessment of mortality risk for people with dementia living in RACFs in Australia. Further external validation in different populations will be required before the model could be developed into a tool to assist with clinical decision-making in the future.</p>","PeriodicalId":72800,"journal":{"name":"Diagnostic and prognostic research","volume":" ","pages":"17"},"PeriodicalIF":0.0,"publicationDate":"2020-10-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7538167/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"38571066","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Development of risk prediction models to predict urine culture growth for adults with suspected urinary tract infection in the emergency department: protocol for an electronic health record study from a single UK university hospital.","authors":"Patrick Rockenschaub,&nbsp;Martin J Gill,&nbsp;David McNulty,&nbsp;Orlagh Carroll,&nbsp;Nick Freemantle,&nbsp;Laura Shallcross","doi":"10.1186/s41512-020-00083-2","DOIUrl":"https://doi.org/10.1186/s41512-020-00083-2","url":null,"abstract":"<p><strong>Background: </strong>Urinary tract infection (UTI) is a leading cause of hospital admissions and is diagnosed based on urinary symptoms and microbiological cultures. Due to lags in the availability of culture results of up to 72 h, and the limitations of routine diagnostics, many patients with suspected UTI are started on antibiotic treatment unnecessarily. Predictive models based on routinely collected clinical information may help clinicians to rule out a diagnosis of bacterial UTI in low-risk patients shortly after hospital admission, providing additional evidence to guide antibiotic treatment decisions.</p><p><strong>Methods: </strong>Using electronic hospital records from Queen Elizabeth Hospital Birmingham (QEHB) collected between 2011 and 2017, we aim to develop a series of models that estimate the probability of bacterial UTI at presentation in the emergency department (ED) among individuals with suspected UTI syndromes. Predictions will be made during ED attendance and at different time points after hospital admission to assess whether predictive performance may be improved over time as more information becomes available about patient status. All models will be externally validated for expected future performance using QEHB data from 2018/2019.</p><p><strong>Discussion: </strong>Risk prediction models using electronic health records offer a new approach to improve antibiotic prescribing decisions, integrating clinical and demographic data with test results to stratify patients according to their probability of bacterial infection. Used in conjunction with expert opinion, they may help clinicians to identify patients that benefit the most from early antibiotic cessation.</p>","PeriodicalId":72800,"journal":{"name":"Diagnostic and prognostic research","volume":" ","pages":"15"},"PeriodicalIF":0.0,"publicationDate":"2020-09-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1186/s41512-020-00083-2","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"38518405","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease.","authors":"Alexander Pate,&nbsp;Richard Emsley,&nbsp;Matthew Sperrin,&nbsp;Glen P Martin,&nbsp;Tjeerd van Staa","doi":"10.1186/s41512-020-00082-3","DOIUrl":"https://doi.org/10.1186/s41512-020-00082-3","url":null,"abstract":"<p><strong>Background: </strong>Stability of risk estimates from prediction models may be highly dependent on the sample size of the dataset available for model derivation. In this paper, we evaluate the stability of cardiovascular disease risk scores for individual patients when using different sample sizes for model derivation; such sample sizes include those similar to models recommended in the national guidelines, and those based on recently published sample size formula for prediction models.</p><p><strong>Methods: </strong>We mimicked the process of sampling <i>N</i> patients from a population to develop a risk prediction model by sampling patients from the Clinical Practice Research Datalink. A cardiovascular disease risk prediction model was developed on this sample and used to generate risk scores for an independent cohort of patients. This process was repeated 1000 times, giving a distribution of risks for each patient. <i>N</i> = 100,000, 50,000, 10,000, <i>N</i> _min (derived from sample size formula) and <i>N</i> _epv10 (meets 10 events per predictor rule) were considered. The 5-95th percentile range of risks across these models was used to evaluate instability. Patients were grouped by a risk derived from a model developed on the entire population (population-derived risk) to summarise results.</p><p><strong>Results: </strong>For a sample size of 100,000, the median 5-95th percentile range of risks for patients across the 1000 models was 0.77%, 1.60%, 2.42% and 3.22% for patients with population-derived risks of 4-5%, 9-10%, 14-15% and 19-20% respectively; for <i>N</i> = 10,000, it was 2.49%, 5.23%, 7.92% and 10.59%, and for <i>N</i> using the formula-derived sample size, it was 6.79%, 14.41%, 21.89% and 29.21%. Restricting this analysis to models with high discrimination, good calibration or small mean absolute prediction error reduced the percentile range, but high levels of instability remained.</p><p><strong>Conclusions: </strong>Widely used cardiovascular disease risk prediction models suffer from high levels of instability induced by sampling variation. Many models will also suffer from overfitting (a closely linked concept), but at acceptable levels of overfitting, there may still be high levels of instability in individual risk. Stability of risk estimates should be a criterion when determining the minimum sample size to develop models.</p>","PeriodicalId":72800,"journal":{"name":"Diagnostic and prognostic research","volume":" ","pages":"14"},"PeriodicalIF":0.0,"publicationDate":"2020-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1186/s41512-020-00082-3","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"38492354","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}