Li Ge , Zhongkai Wang , Charles C. Liu , Spencer Childress , Jeremy Wildfire , George Wu
{"title":"评估多中心试验中二元或连续结果中央统计监测方法的性能:模拟研究","authors":"Li Ge , Zhongkai Wang , Charles C. Liu , Spencer Childress , Jeremy Wildfire , George Wu","doi":"10.1016/j.cct.2024.107580","DOIUrl":null,"url":null,"abstract":"<div><h3>Background</h3><p>Quality study monitoring is fundamental to patient safety and data integrity. Regulators and industry consortia have increasingly advocated for risk-based monitoring (RBM) and central statistical monitoring (CSM) for more effective and efficient monitoring. Assessing which statistical methods underpin these approaches can best identify unusual data patterns in multi-center clinical trials that may be driven by potential systematic errors is important.</p></div><div><h3>Methods</h3><p>We assessed various CSM techniques, including cross-tests, fixed-effects, mixed-effects, and finite mixture models, across scenarios with different sample sizes, contamination rates, and overdispersion via simulation. Our evaluation utilized threshold-independent metrics such as the area under the curve (AUC) and average precision (AP), offering a fuller picture of CSM performance.</p></div><div><h3>Results</h3><p>All CSM methods showed consistent characteristics across center sizes or overdispersion. The adaptive finite mixture model outperformed others in AUC and AP, especially at 30% contamination, upholding high specificity unless converging to a single-component model due to low contamination or deviation. The mixed-effects model performed well at lower contamination rates. However, it became conservative in specificity and exhibited declined performance for binary outcomes under high deviation. Cross-tests and fixed-effects methods underperformed, especially when deviation increased.</p></div><div><h3>Conclusion</h3><p>Our evaluation explored the merits and drawbacks of multiple CSM methods, and found that relying on sensitivity and specificity alone is likely insufficient to fully measure predictive performance. The finite mixture method demonstrated more consistent performance across scenarios by mitigating the influence of outliers. In practice, considering the study-specific costs of false positives/negatives with available resources for monitoring is important.</p></div>","PeriodicalId":10636,"journal":{"name":"Contemporary clinical trials","volume":null,"pages":null},"PeriodicalIF":2.0000,"publicationDate":"2024-05-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Assessing the performance of methods for central statistical monitoring of a binary or continuous outcome in multi-center trials: A simulation study\",\"authors\":\"Li Ge , Zhongkai Wang , Charles C. Liu , Spencer Childress , Jeremy Wildfire , George Wu\",\"doi\":\"10.1016/j.cct.2024.107580\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><h3>Background</h3><p>Quality study monitoring is fundamental to patient safety and data integrity. Regulators and industry consortia have increasingly advocated for risk-based monitoring (RBM) and central statistical monitoring (CSM) for more effective and efficient monitoring. Assessing which statistical methods underpin these approaches can best identify unusual data patterns in multi-center clinical trials that may be driven by potential systematic errors is important.</p></div><div><h3>Methods</h3><p>We assessed various CSM techniques, including cross-tests, fixed-effects, mixed-effects, and finite mixture models, across scenarios with different sample sizes, contamination rates, and overdispersion via simulation. Our evaluation utilized threshold-independent metrics such as the area under the curve (AUC) and average precision (AP), offering a fuller picture of CSM performance.</p></div><div><h3>Results</h3><p>All CSM methods showed consistent characteristics across center sizes or overdispersion. The adaptive finite mixture model outperformed others in AUC and AP, especially at 30% contamination, upholding high specificity unless converging to a single-component model due to low contamination or deviation. The mixed-effects model performed well at lower contamination rates. However, it became conservative in specificity and exhibited declined performance for binary outcomes under high deviation. Cross-tests and fixed-effects methods underperformed, especially when deviation increased.</p></div><div><h3>Conclusion</h3><p>Our evaluation explored the merits and drawbacks of multiple CSM methods, and found that relying on sensitivity and specificity alone is likely insufficient to fully measure predictive performance. The finite mixture method demonstrated more consistent performance across scenarios by mitigating the influence of outliers. In practice, considering the study-specific costs of false positives/negatives with available resources for monitoring is important.</p></div>\",\"PeriodicalId\":10636,\"journal\":{\"name\":\"Contemporary clinical trials\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":2.0000,\"publicationDate\":\"2024-05-23\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Contemporary clinical trials\",\"FirstCategoryId\":\"3\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S1551714424001630\",\"RegionNum\":3,\"RegionCategory\":\"医学\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"MEDICINE, RESEARCH & EXPERIMENTAL\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Contemporary clinical trials","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1551714424001630","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"MEDICINE, RESEARCH & EXPERIMENTAL","Score":null,"Total":0}
Assessing the performance of methods for central statistical monitoring of a binary or continuous outcome in multi-center trials: A simulation study
Background
Quality study monitoring is fundamental to patient safety and data integrity. Regulators and industry consortia have increasingly advocated for risk-based monitoring (RBM) and central statistical monitoring (CSM) for more effective and efficient monitoring. Assessing which statistical methods underpin these approaches can best identify unusual data patterns in multi-center clinical trials that may be driven by potential systematic errors is important.
Methods
We assessed various CSM techniques, including cross-tests, fixed-effects, mixed-effects, and finite mixture models, across scenarios with different sample sizes, contamination rates, and overdispersion via simulation. Our evaluation utilized threshold-independent metrics such as the area under the curve (AUC) and average precision (AP), offering a fuller picture of CSM performance.
Results
All CSM methods showed consistent characteristics across center sizes or overdispersion. The adaptive finite mixture model outperformed others in AUC and AP, especially at 30% contamination, upholding high specificity unless converging to a single-component model due to low contamination or deviation. The mixed-effects model performed well at lower contamination rates. However, it became conservative in specificity and exhibited declined performance for binary outcomes under high deviation. Cross-tests and fixed-effects methods underperformed, especially when deviation increased.
Conclusion
Our evaluation explored the merits and drawbacks of multiple CSM methods, and found that relying on sensitivity and specificity alone is likely insufficient to fully measure predictive performance. The finite mixture method demonstrated more consistent performance across scenarios by mitigating the influence of outliers. In practice, considering the study-specific costs of false positives/negatives with available resources for monitoring is important.
期刊介绍:
Contemporary Clinical Trials is an international peer reviewed journal that publishes manuscripts pertaining to all aspects of clinical trials, including, but not limited to, design, conduct, analysis, regulation and ethics. Manuscripts submitted should appeal to a readership drawn from disciplines including medicine, biostatistics, epidemiology, computer science, management science, behavioural science, pharmaceutical science, and bioethics. Full-length papers and short communications not exceeding 1,500 words, as well as systemic reviews of clinical trials and methodologies will be published. Perspectives/commentaries on current issues and the impact of clinical trials on the practice of medicine and health policy are also welcome.