Investigating the effects of panel conditioning on the measurement of attitudes in the Life in Australia™ panel

As evermore survey research is undertaken via online panels, including probability-based ones, an issue facing researchers is the possibility that data quality could be changed by panel conditioning effects that may arise from repeatedly interviewing the same respondents over time. This may especially occur when the same survey questions are asked repeatedly of the same panellists over data collection waves. This paper presents findings from a study investigating the impact of panel conditioning on the reported attitudes of panellists in Australia’s only probabilitybased online panel – Life in Australia™. The focus of this study is to look for changes in attitudinal reporting over time consistent with the theorised explanations provided by the Cognitive Stimulus Theory (CST) of Sturgis et al. (2009). CST hypothesises that repeated exposure to similar questions may lead to a nonnegligible change in attitudes amongst some panellists, which will manifest as a crystallisation of expressed attitudes between the first wave of data collection and subsequent waves. This crystallisation of attitudes is hypothesised to lead, over time, to an increase in response validity and an increase in the inter-wave consistency of reported attitudes. We find mixed results for the contention that panel conditioning leads to more valid responses being given over time. While we observe a statistically significant increase in the validity of responses for two of the five attitudinal constructs measured, two show no significant change and one shows a statistically significant decline in validity. We also find no clear evidence that panellists’ answers become more consistent as a result of repeatedly responding to the same questions over time. Across a battery of 33-items, panellists are almost as likely to demonstrate less consistency in their responses as more consistency. Even where we do see increases in response consistency this cannot be simply interpreted as support for the CST. We show that other factors such as the innate invariability of some of the attitudes being measured, as well as potentially harmful panel conditioning effects such as interwave freezing and satisficing, may also be contributing factors to the level of response invariance that we observe. Another potentially beneficial effect arising from panel conditioning would be if there was a reduction in social desirability (SD) bias over time. This would accord with the theorised expectation that as panellists become more comfortable completing questionnaires, more familiar with repeated survey questions, and more confident and comfortable with the research panel environment, that they also become more willing to report less socially desirable behaviours and attitudes. However, we find evidence to the contrary and observe instead a small but non-negligible overall increase in socially desirable responding (SDR) across all 13 items we examine. The size of this effect is differential. When responding to a set of questions prone to SD bias, the amount of change observed is significantly different for various subgroups of panel members. The latent characteristics that seem to mediate changes in SDR over time include panellists’ sociodemographic characteristics such as age and the socioeconomic status of their place of residence, personality traits relating to conscientiousness and agreeableness, as well as the level of political activity engaged in. A panellist’s starting out point in answering a set of items that are prone to SD bias is also a main predictor of the amount of subsequent change in the level of SDR. We put forward a possible alternative theoretical explanation, related to a sense of panel belongingness/attachment, as to why panel conditioning might bring about an increase rather than a decrease in SDR. This study contributes to the emerging body of knowledge about panel conditioning effects in probability-based online panels and to the discussion as to whether such effects might be deemed beneficial or detrimental to data quality. Overall, we find little evidence in support of the Cognitive Stimulus Theory and, as such, little support for the assertion that panel conditioning has a beneficial impact on the quality of the attitudinal data reported by panellists when answering the same questions over various waves of a survey program.

Methods Paper No. 2/2024 An electronic publication downloaded
from