Unpack the black box of pilot sampling in policy experimentation: A qualitative comparative analysis of China's public hospital reform

Abstract

Governments increasingly use policy experimentation programs to seek solutions for complex problems. Because randomization and controllability are unrealistic for real-world policy experiments, how subnational pilots are selected is crucial for generating sound evidence for national replication. However, the received wisdom on pilot sampling is thin and paradoxical. While some studies suggest that policymakers prefer to select regions with favorable conditions, others contend that securing representativeness remains the principal concern when it comes to pilot selection. This study resolves the paradox by elucidating the logic of selecting pilots in large policy experimentation programs. We focus on China's huge public hospital reform program and through a novel research design that combines comparative qualitative analysis and illustrative case studies we seek to explain the strategy for pilot selection. Our analyses reveal five distinctive pathways of pilot sampling: piloting for challenge, piloting for advancement, piloting for innovation, piloting for action, and piloting for regional generalization. Each modality represents a specific experimental purpose. We reveal that piloting serves as a versatile governance tool that can fulfill multiple functions in complex reforms.