I’m still sifting through the aftermath of the Presidential election and the controversial accusations of bias in polling and predictions leveled against various pollsters on all sides of the political spectrum. For data geeks, the question of bias in polling is a source of endless fascination. As someone whose profession involves mostly non-political surveys, however, I zeroed in on a basic methodological question:
Regardless of polling firm, what polling method showed the least bias in predicting the election?
According to a detailed post by Nate Silver, the answer is clear, and should make anyone who relies on survey or polling data sit up and take notice:
All things being equal, online surveys showed 40 percent less bias than live telephone interviewers, and an astonishing 72 percent less bias than automated telephone “robopolls.”
The direction of the bias generally tilted in favor of the Republican candidate, and Silver offers several plausible explanations for this pattern of bias, some of which are situational (e.g. the late-breaking impact of Hurricane Sandy supposedly in favor of the Obama campaign).
However, in my view the most intriguing explanations are methodological, For example, selection bias is inherent in landline-based polling to the extent that it excludes voters who solely use cellphones. If as many assert, Obama voters fall disproportionately into this category, then a Romney-skewed sample would have distorted the results of any landline-based poll. Also, the jury is still out on the accuracy of automated robopolls, with a recent study suggesting that their effectiveness is tied to exposure to prior exposure to a live interviewer.
As I understand the data, the pattern that stands out is the relative lack of bias in the online survey method. This finding suggests that online polls may actually be the psychometric method of choice when conducting large-scale surveys, rather than simply the option often chosen for its relatively low cost or other pragmatic factors.
As Silver notes, there will always be a place for including multiple methods in one’s strategy in order to cross-check patterns in the data and refine interpretation. But the polls for the 2012 election strengthens the case for survey strategies that are built around a sturdy online backbone.
Leave A Comment