Comparing drug policies: An assessment of the validity and reliability of the global drug policy index

Abstract

Background

The Global Drug Policy Index (GDPI) is an instrument that attempts to comparatively evaluate national drug policies at a global scale. This paper assesses the validity and reliability of this new Index, speaking to the wider question of whether such a comparative evaluation can be achieved in a methodologically robust manner.

Method

We review the validity of the Index through analysis of the conceptual logic of the GDPI. Construct validity is assessed using Cronbach’s alpha statistics alongside exploratory factor analysis (EFA). The reliability of the Index is assessed using uncertainty analysis where we systematically analyse how Index rankings vary across simulations with randomly perturbed weighting schemes. In addition, we explore the reliability of experts’ assessment of policy implementation, reporting on a common vignette undertaken by all country-expert evaluators.

Results

The analysis of conceptual validity is a reproduction of the logic supporting the contention that the project’s underpinning normative document (a report produced by the UN system coordination Task Team on the Implementation of the UN System Common Position on drug-related matters) represents a solid starting point of what will be a sustained, iterative research process to develop a valid conceptual and operational basis for comparative evaluation of national drug policies. The empirical analysis of construct validity reveals that operationalising the Common Position creates a set of variables with a coherent multidimensional structure that is amenable to aggregation into an overall index. While the performance of states under simulations of different weighting schemes was highly consistent, country expert evaluation in developing state codes on policy implementation can be inconsistent, even when they are provided with a common description designed to capture variance on their rating scale. Cronbach’s alpha indicates that the variables encompassed by the GDPI measure a coherent construct, while EFA results provide support for three of the a priori dimensions used in the creation of the Index.

Conclusions

We conclude that the GDPI project demonstrates that robust comparative evaluation of drug policy at a global scale is possible. However, we also reveal that this project will necessarily be subject to continuous refinement – especially when it comes to standardising expert country evaluations. We outline practical challenges and suggestions for future work in this direction.