A Difference-in-Differences Approach for Estimating Survey Mode Effects

: As surveys increasingly rely on new modes, it is important that researchers understand
how mode influences survey responses. Two common designs for identifying mode effects are
cross-sectional approaches and experiments. But cross-sectional designs risk a combination of
omitted variable bias and post-treatment bias when conditioned on respondent characteristics that
are themselves mode sensitive. In theory, experiments can obviate these biases, but only when
the experiment occurs in tightly-controlled settings that avoid differential uptake across modes.
Considering the costliness and paucity of such experiments, in this paper, I propose a differencein-differences approach for estimating mode effects. Leveraging mixed-mode panel surveys,
mode effects can be identified by comparing changes in responses for panelists who switch
modes across waves to those who remain in the same modes. Difference-in-differences offers a
cost-free alternative to experiments and potentially large bias reduction gains vis-à-vis widelyutilized cross-sectional designs. I apply the difference-in-differences approach by estimating the
effects of completing live interviews vs. web surveys on racial attitudes and political knowledge
in the 2016-2020 ANES and on cognitive functioning measures in the 1992-2020 Health and
Retirement Study.