Evidence of p-hacking in ed policy research and the What Works Clearinghouse

I am really looking forward to this paper:

This article presents a pre-analysis plan for analyzing the evidential value in a selection of policy research taken from scholarly journals and two research clearinghouses run by the federal government. The analysis will collect p-values from selected studies and estimate the evidential value that they represent using the newly introduced p-curve. This article outlines a precise data collection routine, a set of decision rules for including p-values in the analysis sample, and exact hypothesis tests to be used.
[…]
The studies taken into the sample will be:
A. The 20 most recent articles in the Journal of Policy Analysis and Management (JPAM), The Journal of Human Resources (JHR) and Education Evaluation and Policy Analysis (EEPA) that estimate causal parameters in real data.
B. The entire group of studies included in the What Works Clearinghouse’s (WWC) Single Study Review section in the following categories that meet the clearinghouse’s standards of evidence (with and without qualifications): http://ies.ed.gov/ncee/wwc/
a. Dropout Prevention
b. Early Childhood Education
c. Postsecondary Education
d. School Choice
e. Teacher Incentives
C. The entire group of causal studies included in the Clearinghouse for Labor Evaluation and Research (CLEAR) database that have been labeled high or moderate strength. http://clear.dol.gov/study_database
FacebookTwitterGoogle+Share