Understanding variation in treatment effects in education

A new report issued by IES:

A key purpose of rigorous evaluations of education programs and interventions is to inform policy
choices. Typically, such assessments focus on the overall or average treatment effect of the
intervention on key outcomes. However, there are also important program and policy questions
that pertain to variation in treatment effects across subgroups of study participants, as defined by
their baseline characteristics, local area contexts, and program experiences. Variation in effects has
important implications for educational practice—and for facilitating the most efficient use of
limited resources—by informing decisions about how to best target specific interventions and
suggesting ways to improve the design or implementation of the tested interventions.
Understanding variation in effects is also critical to assessing how findings from a particular study
or set of studies may be generalized to broader educational environments.

The purpose of this report is to (1) summarize the research literature on quantitative methods for
assessing how impacts of educational interventions on instructional practices and student learning
differ across students, educators, and schools; and (2) provide technical guidance about the use and
interpretation of these methods. The goal is not to provide a comprehensive literature review or a
detailed description of the considered methods, but rather to provide summary information (with
some mathematical formulation) and references to recent papers that can be used as a starting
point for those interested in learning more. The intended audience is education researchers with
an intermediate to advanced knowledge of quantitative research methods who seek a unified
introduction to modern approaches for understanding variation in treatment effects to apply in
their own studies. This introductory section may also be useful to policymakers and educators who
want to know more about the types of questions related to variation in effects that can be addressed
in evaluations that they may be planning.

The report begins with a discussion of potential reasons why treatment effects can vary across
individuals and sites. The rest of the report is then structured around six interrelated
methodological research questions (topics) that cover a range of quantitative methods to get inside
the “black box” of mechanisms that drive the overall impact findings.

Six methodological research questions (topics) that guide the report

  1. What are treatment effects for subgroups defined by baseline characteristics of students, teachers, and sites?
  2. To whom do the results of this evaluation generalize?
  3. What mediating factors account for treatment effects on longer-term outcomes?
  4. What are treatment effects for subgroups defined by individuals’ post-baseline experiences, such as the nature and amount of services received by treatment group members?
  5. Do treatment effects vary along the distribution of an outcome measure, such as a student achievement test score?
  6. What impact estimation methods are appropriate when treatment effects are heterogeneous?