I predict pre-registration will make some people very unhappy.
Register Studies. Did the researcher execute the research and analysis activities as originally proposed in a recognized study registry? Did the registration describe key elements of the study protocol, including a limited number of primary outcomes? Are any deviations from those plans clearly documented and their rationale explained?
Focus on Meaningful Outcomes. Does the intervention affect learning, achievement, or attainment outcomes that are broadly understood as important to student success? Are the gains large enough to matter? Do these gains persist over time? Are potential differences in impact by key student characteristics explored?
Identify Core Components. What are the core components of the intervention? Is there a clear description of how core components of the intervention are meant to affect outcomes? Do the results of the research help us understand which of those components may be most important in achieving impact?
Analyze Cost. Did the researcher measure the cost of core components of the intervention relative to the control or comparison condition? Does that measure include all resources that might be needed to achieve similar impacts?
Support Scaling Up. If an efficacy trial yielded positive effects, does the study consider the transferability of its findings to other settings or their generalizability to other populations of interest? If an effectiveness trial, are important new settings and populations included and variation in impacts examined? Among interventions where evidence of effectiveness is found, are opportunities for scaling the approach, including commercialization, discussed?
Document Implementation. Were data on the implementation of the intervention collected? Was variation between implementation and impact explored and discussed? Were key hurdles and needed resources identified? How hard will it be to implement the intervention in other venues?