In an important and sad new paper Meyer et al. demonstrate the phenomena in a series of 16 experiments which show that unease with experiments is replicable and general. The authors, for example, ask 679 people in a survey to rate the appropriateness of three interventions designed to reduce hospital infections. The three interventions are: Badge (A): The director decides that all doctors who...
Educational RCT’s are uninformative
From the abstract: There are a growing number of large-scale educational randomized controlled trials (RCTs). Considering their expense, it is important to reflect on the effectiveness of this approach. We assessed the magnitude and precision of effects found in those large-scale RCTs commissioned by the UK-based Education Endowment Foundation and the U.S.-based National Center for Educational...
IES is offering its Summer Cluster-Randomized Trials Institute again
Dear Colleague, We will be offering the IES Summer Research Training Institute again this summer at Northwestern University from July 8–18, 2019. I hope you have found the previous training you received from the Summer CRT Institute to be valuable. As a previous participant in the Training Institute, I strongly encourage you to recommend applying to the Training Institute to your colleagues...
Student evaluations of teaching are not only unreliable, they are significantly biased against female instructors
To answer this question, we apply nonparametric permutation tests to data from a natural experiment at a French university (the original study by Anne Boring is here), and a randomized, controlled, blind experiment in the US (the original study by Lillian MacNell, Adam Driscoll and Andrea N. Hunt is here). We confirm and extend the studies’ main conclusion: Student evaluations of teaching (SET)...
IES training on RCTs
I attended this several years ago, and it was one of the best training experiences I’ve had:
You will learn about much more than just RCTs.
One problem with RCTs: results get exaggerated via site selection
“Site selection bias” can occur when the probability that a program is adopted or evaluated is correlated with its impacts. I test for site selection bias in the context of the Opower energy conservation programs, using 111 randomized control trials involving 8.6 million households across the United States. Predictions based on rich microdata from the first 10 replications substantially overstate...
Problems with double-blind RCTs
Perhaps the most influential and rigorous of these early studies was the Good Friday experiment, conducted in 1962 by Walter Pahnke, a psychiatrist and minister working on a Ph.D. dissertation under Leary at Harvard. In a double-blind experiment, twenty divinity students received a capsule of white powder right before a Good Friday service at Marsh Chapel, on the Boston University campus; ten...
2015 Northwestern Summer Research Training Institute on Cluster-Randomized Trials
This is an excellent training opportunity:
RCT of police body cameras
Hard to argue with this graph: