I was criticized by that famous nutjob Diane Ravitch:
The second post links to a pretty hilarious review of our study (http://pulse.ncpolicywatch.org/2018/06/12/n-c-states-hyped-voucher-study-tells-us-nothing-about-n-c-s-voucher-program/). The writer, Kris Nordstrom, writes as if he’s uncovered a nefarious plot. Sorry, it’s just a propensity score analysis paper written by some nerds.
Over several paragraphs he breathlessly states that the private schools in our sample were a non-random sample and that alarmingly:
Unfortunately, one would have to carefully read the report to reach these conclusions.
Uh, yes, that’s typically what I expect when I write a paper and then circulate it publicly. That people will read it before jumping to conclusions. Crazy, amirite?
He clearly hasn’t read the paper. He falsely claims that PEFNC (a NC school-choice interest group) recruited students for the study. Instead, they (among others) helped us recruit private schools to participate in the study, as is very clearly explained on p.11 of the manuscript. Is this a major limitation? Absolutely, and one that we have carefully laid out in the paper and in every public presentation about our results. The study would have been impossible to pull off without the support of private-school interest groups in the state to help convince private schools to participate and let us test their students.
Critics like to mention PEFNC as some sort of right-wing bogeyman in order to dismiss the paper. My guess is that the major selection issue here is not PEFNC but the quality of the private schools. Principals at schools where students are doing well wouldn’t mind if we came in and tested their kids; principals whose schools are a shitshow would think, no way I’m letting NC State researchers test my kids, who knows what they’ll find?
Nordstrom and Ravitch don’t understand the very basics of educational research. It is extraordinarily difficult to conduct a K-12 study with both strong internal and strong external validity. It’s why the major the shift in the social sciences during the past decade has been away from external validity towards internal validity. People finally woke up and realized that it makes little sense to generate an empirical result that is perfectly representative of a population, if that empirical result is garbage.
Without detailed data on private schools throughout the state, it’s not possible to state a definitive conclusion about our study’s external validity. But if pressed, I would say our results suggest that giving kids a voucher and having them attend a well-run private school yields educational benefits over remaining in their public school. That’s very different from Nordstrom’s assertion that you can’t conclude anything at all from our study due to the school selection issue. But then I think carefully about these things.
Ravitch also takes a swipe at us because the Walton Foundation funded part of the project. Yep, and so did the Z. Smith Reynolds Foundation, which many consider a left-leaning foundation. We were purposeful in seeking funding from a variety of groups, precisely so that no one would accuse us of being in a funder’s pockets. But facts have never gotten in the way of Ravitch’s ravings before, have they?
Then there is this gem from Ravitch:
The researchers might give some thought to their professional standards and ethics. They have embarrassed their profession.
Or maybe we just did a data analysis that conflicts with your rabid ideology?
Going through life foaming at the mouth must be exhausting.