{"title":"Searchsmart.org: Guiding researchers to the best databases and search systems for systematic reviews and beyond.","authors":"Michael Gusenbauer","doi":"10.1002/jrsm.1746","DOIUrl":"10.1002/jrsm.1746","url":null,"abstract":"<p><p>When searching for scholarly documents, researchers often stick with the same familiar handful of databases. Yet, just beyond these limited horizons lie dozens of alternatives with which they could search more effectively, whether for quick lookups or thorough searches in systematic reviews or meta-analyses. Searchsmart.org is a free website that guides researchers to particularly suitable search options for their particular disciplines, offering a wide array of resources, including search engines, aggregators, journal platforms, repositories, clinical trials databases, bibliographic databases, and digital libraries. Search Smart currently evaluates the coverage and functionality of more than a hundred leading scholarly databases, including most major multidisciplinary databases and many that are discipline-specific. Search Smart's primary use cases involve database-selection decisions as part of systematic reviews, meta-analyses, or bibliometric analyses. Researchers can use up to 583 criteria to filter and sort recommendations of databases and the interfaces through which they can be accessed for user-friendliness, search rigor, or relevance. With specific pre-defined filter settings, researchers can quickly identify particularly suitable databases for Boolean keyword searching and forward or backward citation searching. Overall, Search Smart's recommendations help researchers to discover knowledge more effectively and efficiently by selecting the more suitable databases for their tasks.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141915611","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Uncertain about uncertainty in matching-adjusted indirect comparisons? A simulation study to compare methods for variance estimation.","authors":"Conor O Chandler, Irina Proskorovsky","doi":"10.1002/jrsm.1759","DOIUrl":"10.1002/jrsm.1759","url":null,"abstract":"<p><p>In health technology assessment, matching-adjusted indirect comparison (MAIC) is the most common method for pairwise comparisons that control for imbalances in baseline characteristics across trials. One of the primary challenges in MAIC is the need to properly account for the additional uncertainty introduced by the matching process. Limited evidence and guidance are available on variance estimation in MAICs. Therefore, we conducted a comprehensive Monte Carlo simulation study to evaluate the performance of different statistical methods across 108 scenarios. Four general approaches for variance estimation were compared in both anchored and unanchored MAICs of binary and time-to-event outcomes: (1) conventional estimators (CE) using raw weights; (2) CE using weights rescaled to the effective sample size (ESS); (3) robust sandwich estimators; and (4) bootstrapping. Several variants of sandwich estimators and bootstrap methods were tested. Performance was quantified on the basis of empirical coverage probabilities for 95% confidence intervals and variability ratios. Variability was underestimated by CE + raw weights when population overlap was poor or moderate. Despite several theoretical limitations, CE + ESS weights accurately estimated uncertainty across most scenarios. Original implementations of sandwich estimators had a downward bias in MAICs with a small ESS, and finite sample adjustments led to marked improvements. Bootstrapping was unstable if population overlap was poor and the sample size was limited. All methods produced valid coverage probabilities and standard errors in cases of strong population overlap. Our findings indicate that the sample size, population overlap, and outcome type are important considerations for variance estimation in MAICs.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142338038","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Caspar J Van Lissa, Eli-Boaz Clapper, Rebecca Kuiper
{"title":"A tutorial on aggregating evidence from conceptual replication studies using the product Bayes factor.","authors":"Caspar J Van Lissa, Eli-Boaz Clapper, Rebecca Kuiper","doi":"10.1002/jrsm.1765","DOIUrl":"10.1002/jrsm.1765","url":null,"abstract":"<p><p>The product Bayes factor (PBF) synthesizes evidence for an informative hypothesis across heterogeneous replication studies. It can be used when fixed- or random effects meta-analysis fall short. For example, when effect sizes are incomparable and cannot be pooled, or when studies diverge significantly in the populations, study designs, and measures used. PBF shines as a solution for small sample meta-analyses, where the number of between-study differences is often large relative to the number of studies, precluding the use of meta-regression to account for these differences. Users should be mindful of the fact that the PBF answers a qualitatively different research question than other evidence synthesis methods. For example, whereas fixed-effect meta-analysis estimates the size of a population effect, the PBF quantifies to what extent an informative hypothesis is supported in all included studies. This tutorial paper showcases the user-friendly PBF functionality within the bain R-package. This new implementation of an existing method was validated using a simulation study, available in an Online Supplement. Results showed that PBF had a high overall accuracy, due to greater sensitivity and lower specificity, compared to random-effects meta-analysis, individual participant data meta-analysis, and vote counting. Tutorials demonstrate applications of the method on meta-analytic and individual participant data. The example datasets, based on published research, are included in bain so readers can reproduce the examples and apply the code to their own data. The PBF is a promising method for synthesizing evidence for informative hypotheses across conceptual replications that are not suitable for conventional meta-analysis.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142491729","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Zero- and few-shot prompting of generative large language models provides weak assessment of risk of bias in clinical trials.","authors":"Simon Šuster, Timothy Baldwin, Karin Verspoor","doi":"10.1002/jrsm.1749","DOIUrl":"10.1002/jrsm.1749","url":null,"abstract":"<p><p>Existing systems for automating the assessment of risk-of-bias (RoB) in medical studies are supervised approaches that require substantial training data to work well. However, recent revisions to RoB guidelines have resulted in a scarcity of available training data. In this study, we investigate the effectiveness of generative large language models (LLMs) for assessing RoB. Their application requires little or no training data and, if successful, could serve as a valuable tool to assist human experts during the construction of systematic reviews. Following Cochrane's latest guidelines (RoB2) designed for human reviewers, we prepare instructions that are fed as input to LLMs, which then infer the risk associated with a trial publication. We distinguish between two modelling tasks: directly predicting RoB2 from text; and employing decomposition, in which a RoB2 decision is made after the LLM responds to a series of signalling questions. We curate new testing data sets and evaluate the performance of four general- and medical-domain LLMs. The results fall short of expectations, with LLMs seldom surpassing trivial baselines. On the direct RoB2 prediction test set (n = 5993), LLMs perform akin to the baselines (F1: 0.1-0.2). In the decomposition task setup (n = 28,150), similar F1 scores are observed. Our additional comparative evaluation on RoB1 data also reveals results substantially below those of a supervised system. This testifies to the difficulty of solving this task based on (complex) instructions alone. Using LLMs as an assisting technology for assessing RoB2 thus currently seems beyond their reach.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142034719","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Lars König, Steffen Zitzmann, Tim Fütterer, Diego G Campos, Ronny Scherer, Martin Hecht
{"title":"An evaluation of the performance of stopping rules in AI-aided screening for psychological meta-analytical research.","authors":"Lars König, Steffen Zitzmann, Tim Fütterer, Diego G Campos, Ronny Scherer, Martin Hecht","doi":"10.1002/jrsm.1762","DOIUrl":"10.1002/jrsm.1762","url":null,"abstract":"<p><p>Several AI-aided screening tools have emerged to tackle the ever-expanding body of literature. These tools employ active learning, where algorithms sort abstracts based on human feedback. However, researchers using these tools face a crucial dilemma: When should they stop screening without knowing the proportion of relevant studies? Although numerous stopping rules have been proposed to guide users in this decision, they have yet to undergo comprehensive evaluation. In this study, we evaluated the performance of three stopping rules: the knee method, a data-driven heuristic, and a prevalence estimation technique. We measured performance via sensitivity, specificity, and screening cost and explored the influence of the prevalence of relevant studies and the choice of the learning algorithm. We curated a dataset of abstract collections from meta-analyses across five psychological research domains. Our findings revealed performance differences between stopping rules regarding all performance measures and variations in the performance of stopping rules across different prevalence ratios. Moreover, despite the relatively minor impact of the learning algorithm, we found that specific combinations of stopping rules and learning algorithms were most effective for certain prevalence ratios of relevant abstracts. Based on these results, we derived practical recommendations for users of AI-aided screening tools. Furthermore, we discuss possible implications and offer suggestions for future research.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142454357","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Livia Puljak, Andrija Babić, Ognjen Barčot, Tina Poklepović Peričić
{"title":"Evolving use of the Cochrane Risk of Bias 2 tool in biomedical systematic reviews.","authors":"Livia Puljak, Andrija Babić, Ognjen Barčot, Tina Poklepović Peričić","doi":"10.1002/jrsm.1756","DOIUrl":"10.1002/jrsm.1756","url":null,"abstract":"","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142491730","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Kylie E Hunter, Mason Aberoumand, Sol Libesman, James X Sotiropoulos, Jonathan G Williams, Jannik Aagerup, Rui Wang, Ben W Mol, Wentao Li, Angie Barba, Nipun Shrestha, Angela C Webster, Anna Lene Seidler
{"title":"The Individual Participant Data Integrity Tool for assessing the integrity of randomised trials.","authors":"Kylie E Hunter, Mason Aberoumand, Sol Libesman, James X Sotiropoulos, Jonathan G Williams, Jannik Aagerup, Rui Wang, Ben W Mol, Wentao Li, Angie Barba, Nipun Shrestha, Angela C Webster, Anna Lene Seidler","doi":"10.1002/jrsm.1738","DOIUrl":"10.1002/jrsm.1738","url":null,"abstract":"<p><p>Increasing concerns about the trustworthiness of research have prompted calls to scrutinise studies' Individual Participant Data (IPD), but guidance on how to do this was lacking. To address this, we developed the IPD Integrity Tool to screen randomised controlled trials (RCTs) for integrity issues. Development of the tool involved a literature review, consultation with an expert advisory group, piloting on two IPD meta-analyses (including 73 trials with IPD), preliminary validation on 13 datasets with and without known integrity issues, and evaluation to inform iterative refinements. The IPD Integrity Tool comprises 31 items (13 study-level, 18 IPD-specific). IPD-specific items are automated where possible, and are grouped into eight domains, including unusual data patterns, baseline characteristics, correlations, date violations, patterns of allocation, internal and external inconsistencies, and plausibility of data. Users rate each item as having either no issues, some/minor issue(s), or many/major issue(s) according to decision rules, and justification for each rating is recorded. Overall, the tool guides decision-making by determining whether a trial has no concerns, some concerns requiring further information, or major concerns warranting exclusion from evidence synthesis or publication. In our preliminary validation checks, the tool accurately identified all five studies with known integrity issues. The IPD Integrity Tool enables users to assess the integrity of RCTs via examination of IPD. The tool may be applied by evidence synthesists, editors and others to determine whether an RCT should be considered sufficiently trustworthy to contribute to the evidence base that informs policy and practice.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141970257","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Laura Caquelin, Pauline Badra, Lucas Poulain, Bruno Laviolle, Moreno Ursino, Clara Locher
{"title":"Meta-analyses of phase I dose-finding studies: Application for the development of protein kinase inhibitors in oncology.","authors":"Laura Caquelin, Pauline Badra, Lucas Poulain, Bruno Laviolle, Moreno Ursino, Clara Locher","doi":"10.1002/jrsm.1747","DOIUrl":"10.1002/jrsm.1747","url":null,"abstract":"<p><p>This study aimed to assess the feasibility of applying two recent phase I meta-analyses methods to protein kinase inhibitors (PKIs) developed in oncology and to identify situations where these methods could be both feasible and useful. This ancillary study used data from a systematic review conducted to identify dose-finding studies for PKIs. PKIs selected for meta-analyses were required to have at least five completed dose-finding studies involving cancer patients, with available results, and dose escalation guided by toxicity assessment. To account for heterogeneity caused by various administration schedules, some studies were divided into study parts, considered as separate entities in the meta-analyses. For each PKI, two Bayesian random-effects meta-analysis methods were applied to model the toxicity probability distribution of the recommended dose and to estimate the maximum tolerated dose (MTD). Meta-analyses were performed for 20 PKIs including 96 studies corresponding to 115 study parts. The median posterior probability of toxicity probability was below the toxicity thresholds of 0.20 for 70% of the PKIs, even if the resulting credible intervals were very wide. All approved doses were below the MTD estimated for the minimum toxicity threshold, except for one, for which the approved dose was above the MTD estimated for the maximal threshold. The application of phase I meta-analysis methods has been feasible for the majority of PKI; nevertheless, their implementation requires multiple conditions. However, meta-analyses resulted in estimates with large uncertainty, probably due to limited patient numbers and/or between-study variability. This calls into question the reliability of the recommended doses.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141892471","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Judith Logan, Jenaya Webb, Nalini K Singh, Nailisa Tanner, Kathryn Barrett, Margaret Wall, Benjamin Walsh, Ana Patricia Ayala
{"title":"Scoping review search practices in the social sciences: A scoping review.","authors":"Judith Logan, Jenaya Webb, Nalini K Singh, Nailisa Tanner, Kathryn Barrett, Margaret Wall, Benjamin Walsh, Ana Patricia Ayala","doi":"10.1002/jrsm.1742","DOIUrl":"10.1002/jrsm.1742","url":null,"abstract":"<p><p>A thorough literature search is a key feature of scoping reviews. We investigated the search practices used by social science researchers as reported in their scoping reviews. We collected scoping reviews published between 2015 and 2021 from Social Science Citation Index. In the 2484 included studies, we observed a 58% average annual increase in published reviews, primarily from clinical and applied social science disciplines. Bibliographic databases comprised most of the information sources in the primary search strategy (n = 9565, 75%), although reporting practices varied. Most scoping reviews (n = 1805, 73%) included at least one supplementary search strategy. A minority of studies (n = 713, 29%) acknowledged an LIS professional and few listed one as a co-author (n = 194, 8%). We conclude that to improve reporting and strengthen the impact of the scoping review method in the social sciences, researchers should consider (1) adhering to PRISMA-S reporting guidelines, (2) employing more supplementary search strategies, and (3) collaborating with LIS professionals.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"141970256","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Alexander Pachanov, Catharina Münte, Julian Hirt, Dawid Pieper
{"title":"Development and validation of a geographic search filter for MEDLINE (PubMed) to identify studies about Germany.","authors":"Alexander Pachanov, Catharina Münte, Julian Hirt, Dawid Pieper","doi":"10.1002/jrsm.1763","DOIUrl":"10.1002/jrsm.1763","url":null,"abstract":"<p><p>While geographic search filters exist, few of them are validated and there are currently none that focus on Germany. We aimed to develop and validate a highly sensitive geographic search filter for MEDLINE (PubMed) that identifies studies about Germany. First, using the relative recall method, we created a gold standard set of studies about Germany, dividing it into 'development' and 'testing' sets. Next, candidate search terms were identified using (i) term frequency analyses in the 'development set' and a random set of MEDLINE records; and (ii) a list of German geographic locations, compiled by our team. Then, we iteratively created the filter, evaluating it against the 'development' and 'testing' sets. To validate the filter, we conducted a number of case studies (CSs) and a simulation study. For this validation we used systematic reviews (SRs) that had included studies about Germany but did not restrict their search strategy geographically. When applying the filter to the original search strategies of the 17 SRs eligible for CSs, the median precision was 2.64% (interquartile range [IQR]: 1.34%-6.88%) versus 0.16% (IQR: 0.10%-0.49%) without the filter. The median number-needed-to-read (NNR) decreased from 625 (IQR: 211-1042) to 38 (IQR: 15-76). The filter achieved 100% sensitivity in 13 CSs, 85.71% in 2 CSs and 87.50% and 80% in the remaining 2 CSs. In a simulation study, the filter demonstrated an overall sensitivity of 97.19% and NNR of 42. The filter reliably identifies studies about Germany, enhancing screening efficiency and can be applied in evidence syntheses focusing on Germany.</p>","PeriodicalId":226,"journal":{"name":"Research Synthesis Methods","volume":null,"pages":null},"PeriodicalIF":5.0,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142454358","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"生物学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}