Evaluating the current methodological practices and issues in existing literature in pooling complex surveys: a systematic review.

Abstract

Background: Pooling data from complex survey designs is increasingly used in the health and medical sciences. However, current methodological practices are not well documented in the literature while performing the pooling strategy. We aimed to review related pooling studies and evaluate the quality of pooling within the framework of specific methodological guidelines, particularly when combining complex surveys such as Demographic & Health Surveys (DHS) and Multiple Indicator Cluster Surveys (MICS).

Methods: We performed a systematic literature search focusing on studies utilizing the pooling method with DHS and MICS survey data. These studies were selected from those published between 2010 and 2021 and were retrieved from electronic databases (PubMed and Scopus) in accordance with pre-defined inclusion criteria. Then, we extracted 355 studies for the final review and evaluated the reporting quality of the pooling strategy while considering some methodological issues.

Results: The majority of studies (81.4%) reported using a pooled (one-stage) approach, while 11.8% used a separate (two-stage) approach, and 6.8% used both approaches. Approximately 63.3% of studies did not clearly describe their pooling strategy. Only 3.4% of the studies mentioned the variable harmonization process, while 66.9% addressed dealing with heterogeneity between surveys. All studies that used the separate (two-stage) approach conducted a meta-analytic procedure, while 38.1% of studies using the pooled approach employed a multilevel model. More than half of the studies (55.6%) mentioned the use of clustered standard errors. The Delta method, Bootstrap, and Taylor linearization were each applied in 11.1% of the studies for variance estimation. Survey weights, primary sampling unit (PSU) or cluster, and strata were used together in 30.5% of the studies. Survey weights were employed by 69.8%, PSU or cluster by 43.8%, and the strata variable by 31.7%. Sensitivity analysis was conducted in 16% of the studies.

Conclusions: Our study revealed that fundamental methodological issues associated with pooling complex survey databases, such as the selection of pooling procedures, data harmonization, accounting for cycle effects, quality control checks, addressing heterogeneity, selecting model effects, utilizing survey design variables, and dealing with missing values, etc., were inadequately reported in the included studies. We recommend authors, readers, reviewers, and editors examine pooling studies more attentively and utilize the customized checklist developed by our study to assess the quality of future pooling studies.