The Choice of Policy Comparisons and Relevant Evidence and Approach to Selection Bias Are Keys to the Debate

Abstract

Patrick Wolf and I use the same framework for evaluating universal vouchers/ESAs, from Levin (2002), but we come to different conclusions. Why is that? In my view, this is because I disagree with his choices about the counterfactual policies and the relevant evidence, and his interpretation of that evidence with respect to selection bias. All of this is at the heart of the debate.

First, his counterfactual is not the status quo. His analysis includes references to the public education system as it was all the way back in the mid-1800s. Some of his analysis might have been reasonable up through the early 1990s but the choice landscape changed long ago. More than a third of students were attending schools other than their neighborhood public schools before the recent cascade of vouchers, mostly because of expanding public school open enrollment, charter schools, and publicly supported virtual/homeschooling. Given these recent and significant expansions of public school choice, it is difficult to see why we would compare universal vouchers to a form of universal traditional public education that has faded away.

Such a comparison seems like a straw man and significantly changes the interpretation of evidence. Public school choice programs, as I explained in my initial essay, seem to have some of the same advantages as vouchers (e.g., positive competitive effects), while also avoiding some of their disadvantages (e.g., social cohesion and negative achievement participation effects).

Now, consider the evidence on vouchers. Wolf's summary averages the modest positive average treatment effects of small-scale, targeted programs with the negative effects of more relevant statewide programs. Two issues arise here, the first being the central role of eligibility targeting. Almost every study in Wolf's review pertains to programs targeted to low-income students. But the policy we are debating is universal school vouchers. This is a problem because, as I explained, there is a general research consensus that school choice works better for disadvantaged students (Chen and Harris 2023; Cohodes and Roy 2024). This means almost all the studies he is citing are overly optimistic about the effects we can expect from universal voucher programs.

To reinforce his argument, Wolf quotes Howard Fuller as saying, “School choice is widespread in America, unless you are poor.” Setting aside that this quote is also dated (see above on increasingly widespread public school choice), this raises the question, if the goal were to increase choice for disadvantaged students, then why not target the program to those students? A more reasonable case can be made for a targeted program than for universal programs, given the targeted programs are more effective and less costly.

The word “universal” with today's vouchers also signals the programs’ lack of geographic boundaries. Until the last decade, essentially all U.S. voucher studies were in individual cities. Unfortunately, the effects of small programs rarely scale up, in part because they are piloted in locations where we expect them to be effective. This is such a significant issue that the U.S. Department of Education's Institute of Education Sciences has (or at least had) an entirely separate funding category for research involving scale up. Once these important distinctions, of both scale and income/urbanicity targeting, are made, it is difficult to make the case for vouchers on academic achievement grounds.

Our third disagreement pertains to voucher evidence on high school graduation and college-going. He relies on studies that suffer from significant sources of potential selection bias. Unlike most public policies, where access to programs among eligible people is guaranteed, selection threatens voucher studies in two directions. In addition to families choosing schools, private schools can select their own students. This has hamstrung researchers trying to understand program effects, forcing them to rely mostly on matching based on observable differences. The only statewide studies where we see positive effects on long-term outcomes like college-going are those where selection bias is a real concern.

Wolf does not mention that the most important study in the debate—his own study of Louisiana (Erickson et al. 2021)—found no average effect on college-going. This study is the most convincing of all because it is statewide and students were randomized to voucher receipt. This critical study finds no effect on college-going even though targeted to the disadvantaged students who are supposed to benefit most. The authors also find large negative effects from this program on student achievement.

Perhaps the strongest evidence Wolf presents for the pro-universal-voucher/ESA side is that parents seem more satisfied with private schools than public schools. Parent satisfaction is an important outcome, but selection also makes this evidence difficult to interpret. As I have emphasized, private schools have two built-in advantages over public schools: (1) they can select their students; and (2) they can teach religion. In the first case, those who successfully gain admission to private schools will give them higher ratings because they are tailored to their private needs and reduce interaction with the more disruptive, less “desirable,” and expensive-to-serve students who are rejected. It is difficult to know whether this shaping of the student body is the main reason for higher satisfaction.

We can similarly expect families who want more religious teaching to use their vouchers to choose religious schools and give them higher ratings, too. Is this good for education overall, according to Levin's framework? To the extent we are interested in the private good elements of education, and liberty generally, it would seem so, but it would also raise questions about another of Levin's criteria: social cohesion. As I emphasized, the evidence on vouchers and social cohesion is quite limited.

On this point, Wolf writes “many European countries have achieved a high level of social cohesion by instituting universal private school choice.” While he provides no citation to support this statement, it is important to note that these programs are designed very differently than the current universal vouchers/ESAs in the U.S. In particular, those schools are required to teach the state/national curriculum and students are generally required to take the national exams. Such rules almost certainly do help facilitate social cohesion, but they have been actively rejected in today's U.S. voucher context. European vouchers are more like the charter schools that, as I explained at the beginning of this response, are the main alternative to vouchers.

Overall, as I originally argued, it still seems difficult to make a case for universal vouchers/ESAs based using standard policy analysis. Public school choice has a stronger academic track record, especially at scale and for a broad range of students. Again, while the policy has some merit on liberty grounds, it poses risks both for academic achievement and social cohesion.

Epilogue: I had submitted my first piece before vouchers became a part of federal law as part of President Trump's Big Beautiful Bill. This policy allows states to opt in to a tax credit program that will allow people to donate to scholarship granting organizations (SGOs) that, in turn, will grant vouchers to eligible state students. This is a good example of the type of policy that is difficult to justify based on the empirical evidence and Levin's criteria. From an equity standpoint, the program is only slightly targeted—being available to almost 90% of Americans by income. While the details of this new program are not available, there is little reason to expect any better results than what I have described so far.