Research Synthesis Methods最新文献

{"title":"Personalized treatment hierarchies in Bayesian network meta-analysis.","authors":"Augustine Wigle, Erica E M Moodie","doi":"10.1017/rsm.2026.10089","DOIUrl":"https://doi.org/10.1017/rsm.2026.10089","url":null,"abstract":"<p><p>Network meta-analysis (NMA) is an increasingly popular evidence synthesis tool that can provide a ranking of competing treatments, also known as a treatment hierarchy. Treatment-covariate interactions (TCIs) can be included in NMA models to allow relative treatment effects to vary with covariate values. We show that in an NMA model that includes TCIs, treatment hierarchies should be created with a particular covariate profile in mind. We outline the typical approach for creating a treatment hierarchy in standard Bayesian NMA and show how a treatment hierarchy for a particular covariate profile can be created from an NMA model that estimates TCIs. We demonstrate our methods using a real network of studies for the treatment of major depressive disorder.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":" ","pages":"1-7"},"PeriodicalIF":6.1,"publicationDate":"2026-05-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147830765","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"How comprehensive is comprehensive? Prevalence of searches beyond peer-reviewed journal articles and bibliographic sources: a cross-sectional study of systematic reviews.","authors":"Jane O'Sullivan, Sarah Dawson, Chris Cooper, Julian Piers Thomas Higgins","doi":"10.1017/rsm.2026.10086","DOIUrl":"https://doi.org/10.1017/rsm.2026.10086","url":null,"abstract":"<p><p>To examine the extent to which information sources other than journal articles are sought for systematic reviews. Cross-sectional study of published systematic reviews. We examined all published systematic reviews included in MEDLINE in a 4-week period in 2019. Both systematic reviews and protocols of reviews were eligible for inclusion. (1) Number and types of information sources sought in systematic reviews; (2) proportion of reviews that explicitly searched for study reports other than journal articles; (3) proportion of reviews that searched resources containing study reports other than journal articles. A total of 1,262 systematic reviews fulfilled the eligibility criteria. The median number of information resources searched for all systematic reviews was 4. Of the 1,262 reviews, study reports other than journal articles were sought in 40% (<i>n</i> = 502) of systematic reviews (97% (<i>n</i> = 64) of Cochrane reviews and 37% (<i>n</i> = 438) of non-Cochrane reviews). Trial registers were searched in 88% of Cochrane reviews and 21% of non-Cochrane reviews. In 99.3% (<i>n</i> = 1,253) of all the systematic reviews, the searches performed had the potential to identify study reports other than journal articles. Between a third and a half of systematic reviews search for study reports other than journal articles. Systematic review searches often search resources that include study reports other than journal articles, whether or not the reviewers explicitly sought them.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":" ","pages":"1-11"},"PeriodicalIF":6.1,"publicationDate":"2026-05-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147831576","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Evaluation of the replicability of systematic reviews with meta-analyses of the effects of health interventions.","authors":"Daniel G Hamilton, Joanne E McKenzie, Phi-Yen Nguyen, Melissa L Rethlefsen, Steve McDonald, Sue E Brennan, Fiona M Fidler, Julian P T Higgins, Raju Kanukula, Sathya Karunananthan, Lara J Maxwell, David Moher, Shinichi Nakagawa, David Nunan, Peter Tugwell, Vivian A Welch, Matthew J Page","doi":"10.1017/rsm.2025.10064","DOIUrl":"10.1017/rsm.2025.10064","url":null,"abstract":"<p><p>Systematic reviews are often characterized as being inherently replicable, but several studies have challenged this claim. The objective of the study was to investigate the variation in results following independent replication of literature searches and meta-analyses of systematic reviews. We included 10 systematic reviews of the effects of health interventions published in November 2020. Two information specialists repeated the original database search strategies. Two experienced review authors screened full-text articles, extracted data, and calculated the results for the first reported meta-analysis. All replicators were initially blinded to the results of the original review. A meta-analysis was considered not 'fully replicable' if the original and replicated summary estimate or confidence interval width differed by more than 10%, and meaningfully different if there was a difference in the direction or statistical significance. The difference between the number of records retrieved by the original reviewers and the information specialists exceeded 10% in 25/43 (58%) searches for the first replicator and 21/43 (49%) searches for the second. Eight meta-analyses (80%, 95% CI: 49-96) were initially classified as not fully replicable. After screening and data discrepancies were addressed, the number of meta-analyses classified as not fully replicable decreased to five (50%, 95% CI: 24-76). Differences were classified as meaningful in one blinded replication (10%, 95% CI: 1-40) and none of the unblinded replications (0%, 95% CI: 0-28). The results of systematic review processes were not always consistent when their reported methods were repeated. However, these inconsistencies seldom affected summary estimates from meta-analyses in a meaningful way.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":"17 3","pages":"538-556"},"PeriodicalIF":6.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13126212/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147697140","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Reporting and evaluation of assumptions and certainty of evidence in network meta-analyses.","authors":"Kansak Boonpattharatthiti, Kanyaphak Chueadi, Phiyanuch Thimkorn, Deborah M Caldwell, Nathorn Chaiyakunapruk, Teerapon Dhippayom","doi":"10.1017/rsm.2025.10045","DOIUrl":"10.1017/rsm.2025.10045","url":null,"abstract":"<p><p>Network meta-analysis (NMA) facilitates the comparison of multiple treatments by integrating both direct and indirect evidence. Applications of NMA in medical decision making have grown exponentially. However, the validity of NMA findings depends on key assumptions: homogeneity, transitivity, and consistency. A lack of consistent assessment of these assumptions potentially compromises the reliability of NMA outcomes. The objective of this study is to evaluate the extent to which researchers address NMA assumptions and report the assessment of evidence certainty in NMA publications. A total of 22,079 studies were identified from PubMed, Embase, and Cochrane CENTRAL (January 2010-August 2024). A sample of 393 NMAs was calculated to represent this population and randomly selected. Data on study characteristics, NMA assumptions, and the certainty of evidence were extracted and analyzed descriptively. Of the 393 NMAs, 71.8% were published between 2020 and 2024. Homogeneity was assessed in 300 (76.3%) NMAs, transitivity in 45 (11.5%) NMAs, and consistency in 265 (67.4%) NMAs. The certainty of evidence was assessed in 110 (28.0%) studies, predominantly using GRADE (71 NMAs; 18.1%) or CINeMA (29 NMAs; 7.4%). NMAs published in journals with high-impact factors more frequently evaluate these aspects than those published in low-impact journals. The assessment of NMA assumptions is inconsistently reported across studies, particularly for transitivity and consistency assumptions. Our findings highlight the need for standardized protocols or reporting guidelines to ensure these assessments are conducted and transparently reported.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":"17 3","pages":"557-566"},"PeriodicalIF":6.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13126206/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147696871","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Reflections on the I-squared index for measuring inconsistency in meta-analysis.","authors":"Julian P T Higgins, José A López-López","doi":"10.1017/rsm.2025.10062","DOIUrl":"10.1017/rsm.2025.10062","url":null,"abstract":"<p><p>The I-squared index was proposed in 2002 as a measure to help understand the consistency of study results in a meta-analysis. It was developed to overcome some of the limitations of existing measures, principally the chi-squared test for heterogeneity and the between-study variance as estimated in a random-effects meta-analysis. I-squared measures approximately the proportion of total variability in results that is due to true heterogeneity rather than random error; it is also conveniently interpreted as a measure of inconsistency in the results of the studies. The index has become extremely widely used, although it is often misinterpreted as an absolute measure of the amount of heterogeneity, which it is not. Here, we discuss the I-squared index and the different ways it can be defined, computed, and interpreted. We introduce a new interpretation of I-squared as a weighted sum of squares, which we propose may be helpful when setting up simulation studies. We discuss some of the extensions and repurposes that have been proposed for I-squared and offer some recommendations on the appropriate use of the index in practice.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":"17 3","pages":"389-402"},"PeriodicalIF":6.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13126227/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147696931","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Evaluating differences in latent means across studies: Extending meta-analytic confirmatory factor analysis with the analysis of means.","authors":"Suzanne Jak, Mike W-L Cheung, Selcuk Acar, Reuben Kindred","doi":"10.1017/rsm.2025.10057","DOIUrl":"10.1017/rsm.2025.10057","url":null,"abstract":"<p><p>Meta-analytic confirmatory factor analysis (CFA) is a type of meta-analytic structural equation modeling (MASEM) that is useful for evaluating the factor structure of measurement scales based on data from multiple studies. Modeling the factor structure is just one example of the many potentially interesting research questions. Analyzing covariance matrices allows for the evaluation of measurement properties across studies, such as whether indicators are functioning the same across studies. For example, are some indicators more indicative of the common factor in certain types of studies than in others? The additional analysis of means of the observed variables opens up many other research questions to consider such as: \"Are there mean differences in mental health between clinical and non-clinical samples?\" To answer such questions, it is necessary to analyze both the covariance and the mean structure of the indicators. In this paper, we present, illustrate, and evaluate a method to incorporate the means of variables in the MASEM analyses of such datasets. We focus on meta-analytic CFA, with the aim of testing differences in latent means across studies. We provide illustrations of the comparison of latent means across groups of studies using two empirical datasets, for which data and analysis scripts are provided online. The performance of the new model was tested in a small-scale simulation study. The results showed adequate performance under the tested conditions. Finally, we discuss how the proposed method relates to other analysis options such as multigroup or multilevel structural equation modeling.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":"17 3","pages":"498-516"},"PeriodicalIF":6.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13126220/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147697120","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Strategizing AI utilization for psychological literature screening: A comparative analysis of machine learning algorithms and key factors to consider.","authors":"Lars König, Steffen Zitzmann, Martin Hecht","doi":"10.1017/rsm.2025.10053","DOIUrl":"10.1017/rsm.2025.10053","url":null,"abstract":"<p><p>With the rapid growth of scholarly literature, efficient artificial intelligence (AI)-aided abstract screening tools are becoming increasingly important. This study evaluated 10 different machine learning (ML) algorithms used in AI-aided screening tools for ordering abstracts according to their estimated relevance. We focused on assessing their performance in terms of the number of abstracts required to screen to achieve a sufficient detection rate of relevant articles. Our evaluation included articles screened with diverse inclusion and exclusion criteria. Crucially, we examined how characteristics of the screening data-such as the proportion of relevant articles, the overall frequency of abstracts, and the amount of training data-impacted algorithm effectiveness. Our findings provide valuable insights for researchers across disciplines, highlighting key factors to consider when selecting an ML algorithm and determining a stopping point for AI-aided screening. Specifically, we observed that the algorithm combining the logistic regression (LR) classifier with the sentence-bidirectional encoder representations from transformers (SBERT) feature extractor outperformed other algorithms, demonstrating both the highest efficiency and the lowest variability in performance. Nonetheless, the algorithm's performance varied across experimental conditions. Building on these findings, we discuss the results and provide practical recommendations to assist users in the AI-aided screening process.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":"17 3","pages":"451-482"},"PeriodicalIF":6.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13126213/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147696940","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Bayes factor hypothesis testing in meta-analyses: Practical advantages and methodological considerations.","authors":"Joris Mulder, Robbie C M van Aert","doi":"10.1017/rsm.2025.10060","DOIUrl":"10.1017/rsm.2025.10060","url":null,"abstract":"<p><p>Bayesian hypothesis testing via Bayes factors offers a principled alternative to classical <i>p</i>-value methods in meta-analysis, particularly suited to its cumulative and sequential nature. Unlike <i>p</i>-values, Bayes factors allow for quantifying support both for and against the existence of an effect, facilitate ongoing evidence monitoring, and maintain coherent long-run behavior as additional studies are incorporated. Recent theoretical developments further show how Bayes factors can flexibly control Type I error rates through connections to e-value theory. Despite these advantages, their use remains limited in the meta-analytic literature. This article provides a critical overview of their theoretical properties, methodological considerations-such as prior sensitivity-and practical advantages for evidence synthesis. Two illustrative applications are provided: one on statistical learning in individuals with language impairments, and another on seroma incidence following post-operative exercise in breast cancer patients. New tools supporting these methods are available in the open-source R package BFpack.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":"17 3","pages":"589-623"},"PeriodicalIF":6.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13126231/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147697107","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Clustered flexible calibration plots for binary outcomes using random effects modeling.","authors":"Lasai Barreñada, Bavo De Cock Campo, Laure Wynants, Ben Van Calster","doi":"10.1017/rsm.2025.10046","DOIUrl":"10.1017/rsm.2025.10046","url":null,"abstract":"<p><p>Evaluation of clinical prediction models across multiple clusters, whether centers or datasets, is becoming increasingly common. A comprehensive evaluation includes an assessment of the agreement between the estimated risks and the observed outcomes, also known as calibration. Calibration is of utmost importance for clinical decision making with prediction models, and it often varies between clusters. We present three approaches to take clustering into account when evaluating calibration: (1) clustered group calibration (CG-C), (2) two-stage meta-analysis calibration (2MA-C), and (3) mixed model calibration (MIX-C), which can obtain flexible calibration plots with random effects modeling and provide confidence interval (CI) and prediction interval (PI). As a case example, we externally validate a model to estimate the risk that an ovarian tumor is malignant in multiple centers (<i>N</i> = 2489). We also conduct a simulation study and a synthetic data study generated from a true clustered dataset to evaluate the methods. In the simulation study, MIX-C and 2MA-C (splines) gave estimated curves closest to the true overall curve. In the synthetic data study, MIX-C produced cluster-specific curves closest to the truth. Coverage of the PI across the plot was best for 2MA-C with splines. We recommend using 2MA-C with splines to estimate the overall curve and 95% PI and MIX-C for cluster-specific curves, especially when the sample size per cluster is limited. We provide ready-to-use code to construct summary flexible calibration curves, with CI and PI to assess heterogeneity in calibration across datasets or centers.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":"17 3","pages":"567-588"},"PeriodicalIF":6.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13126218/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147697123","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Assessing the properties of the prediction interval in random-effects meta-analysis.","authors":"Péter Mátrai, Tamás Kói, Zoltán Sipos, Nelli Farkas","doi":"10.1017/rsm.2025.10055","DOIUrl":"10.1017/rsm.2025.10055","url":null,"abstract":"<p><p>Random-effects meta-analysis is a widely applied methodology to synthesize research findings of studies related to a specific scientific question. Besides estimating the mean effect, an important aim of the meta-analysis is to summarize the heterogeneity, that is, the variation in the underlying effects caused by the differences in study circumstances. The prediction interval is frequently used for this purpose: a 95% prediction interval contains the true effect of a similar new study in 95% of the cases when it is constructed, or in other words, it covers 95% of the true effects distribution on average in repeated sampling. In this article, after providing a clear mathematical background, we present an extensive simulation investigating the performance of all frequentist prediction interval methods published to date. The work focuses on the distribution of the coverage probabilities and how these distributions change depending on the amount of heterogeneity and the number of involved studies. Although the single requirement that a prediction interval has to fulfill is to keep a nominal coverage probability on average, we demonstrate why the distribution of coverages should not be disregarded. We show that for meta-analyses with small number of studies, this distribution has an unideal, asymmetric shape. We argue that assessing only the mean coverage can easily lead to misunderstanding and misinterpretation. The length of the intervals and the robustness of the methods concerning the non-normality of the true effects are also investigated.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":"17 3","pages":"517-537"},"PeriodicalIF":6.1,"publicationDate":"2026-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13126221/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147697104","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}