Getting Policy Right Takes Time

Posted

RALEIGH — Well, never mind, then. According to the only large-scale, randomized controlled trial of a statewide preschool initiative — Tennessee’s pre-K program —intervening early to boost long-term academic performance doesn’t work.

Thousands of low-income students in Tennessee were randomly assigned to the pre-K program or to a control group. While there were some academic gains for participants at the kindergarten stage, by the third grade the benefits disappeared. In fact, pre-K participants actually scored lower than their otherwise-comparable peers.

States aren’t just wasting taxpayer money, then. Preschool programs are actively harming their intended beneficiaries. If North Carolina leaders truly care about disadvantaged children, they should shut down our state’s pre-K program immediately.

Is that what you were expecting me to say? Do you think such a conclusion would be reasonable based on a single study, however well-constructed? If your answer to either or both of these questions is yes, I would respectfully urge you to consider carefully what it means to apply scientific concepts and findings to public policy.

One of those concepts is that the scientific method isn’t really about proving propositions true. As a great 20th-century philosopher of science, Karl Popper, argued persuasively, science is really about proving propositions false. Although a scientific theory may seem highly useful in explaining the past and predicting the future, there is always the possibility that some valid future experiment will disprove it.

But what does falsification look like in the real world? Another great 20th-century philosopher of science, Thomas Kuhn, argued persuasively that the model of a broadly accepted scientific theory being disproved by a single experiment is unrealistic. What really happens is that the defenders of the theory assume the experiment is flawed in some way, or only applies to some situations.

Then what happens, and what indeed ought to happen, is that scientists see a puzzle to be solved. They replicate the contradictory experiment. If the finding is the same, they seek to explain it without abandoning the established theory. Others do the same.

If the contradictions persist and proliferate, a crisis comes. A good analogy would be flowing water and a dam. Sometimes the dam holds. But if the pressure grows strong enough, it bursts — and in the case of science, a new paradigm is formed.

To return to the question of preschool intervention, the Tennessee study, an earlier and comprehensive evaluation of Head Start, and new statistical work by the Brookings Institution’s Russ Whitehurst all suggest little to no benefit. But you’ll find a number of other, smaller-scale studies pointing the other way.

Similarly, while dozens of scholarly studies show that school choice and competition are associated with academic and other benefits, initial studies of statewide voucher programs showed scant or even negative effects, although subsequent findings were more promising.

In response to such evidence, it would be reasonable to temper one’s short-run expectations and recognize that scaling up policy ideas can be challenging. But it would not be reasonable to abandon those ideas altogether. That’s not how science works. And it’s certainly not how public policy works.

John Hood (@JohnHoodNC) is chairman of the John Locke Foundation and appears on “N.C. Spin,” broadcast statewide Fridays at 7:30 p.m. and Sundays at 12:30 p.m. on UNC-TV.

Comments

No comments on this story | Please log in to comment by clicking here
Please log in or register to add your comment