Power analyses for matching

A colleague at RTI asked me about power analyses for PSM, specifically, should he assume dependent samples for his power analysis?

Recall that with dependent samples, we pair observations at the analysis stage. For example, we might calculate the change over time for two observations of the same individual, or compare husbands’ and wives’ perception of their marriages.

With matching, however, we (sometimes) pair observations at the design stage. That is, we find a control unit with a similar propensity as a treated unit, put them in the analysis sample, then go back to the full sample to continue grabbing matched pairs. But at the analysis stage, we simply compare two independent samples, the treated and the controls. We don’t actually match units when estimating the difference in Y between the two groups. The best way to think about this is that both random assignment and matching are used to create to comparable groups; once created, we can use any kind of analysis to estimate the difference in outcomes (e.g., t-tests, OLS, etc.).

So a conservative approach would be to estimate power for a two-sample t-test. This is conservative, because many researchers would argue that you should use a doubly-robust design, by running a covariate control model on the matched sample using the covariates from the matching stage. Adding covariates will increase your power as the R-square increases.

Two good resources for power analysis are Optimal Design and PowerUp.