Not all Polls are created equal

On Thanksgiving, a cousin of mine emailed me this Zogby Poll. As I researched it myself, I discovered that the controversy surrounding this poll provides an immediate case study of many of the topics we have covered in class.

The poll itself is a series of 8 questions addressed to Obama voters. The results demonstrated that Obama voters knew significantly less negatives about Obama then they did about McCain. John Zegiler, the Polls commissioner, claims on his website HowObamaGotElected that these results prove media bias...

"On Election day twelve Obama voters were interviewed extensively right after they voted to learn how the news media impacted their knowledge of what occurred during the campaign. These voters were chosen for their apparent intelligence/verbal abilities and willingness to express their opinions to a large audience. The rather shocking video below seeks to provide some insight into which information broke through the news media clutter and which did not....Because obviously interviewing a relative handful of Obama voters, while interesting, is hardly scientific proof of anything, we also commissioned a Zogby telephone poll which asked the very same questions (as well as a few others) with similarly amazing results..."

...and proudly displays the results in the following fashion:

512 Obama Voters 11/13/08-11/15/08 MOE +/- 4.4 points

97.1% High School Graduate or higher, 55% College Graduates

Results to 12 simple Multiple Choice Questions

57.4% could NOT correctly say which party controls congress (50/50 shot just by guessing)

71.8% could NOT correctly say Joe Biden quit a previous campaign because of plagiarism (25% chance by guessing)

82.6% could NOT correctly say that Barack Obama won his first election by getting opponents kicked off the ballot (25% chance by guessing)

88.4% could NOT correctly say that Obama said his policies would likely bankrupt the coal industry and make energy rates skyrocket (25% chance by guessing)

56.1% could NOT correctly say Obama started his political career at the home of two former members of the Weather Underground (25% chance by guessing).

And yet.....

Only 13.7% failed to identify Sarah Palin as the person on which their party spent $150,000 in clothes

Only 6.2% failed to identify Palin as the one with a pregnant teenage daughter

And 86.9 % thought that Palin said that she could see Russia from her "house," even though that was Tina Fey who said that!!

Only 2.4% got at least 11 correct.

Only .5% got all of them correct. (And we "gave" one answer that was technically not Palin, but actually Tina Fey)

However, as other media sources were quick to point out, the methodology of this poll is deeply flawed. The relatively non-partisan WSJ noted that the presentation of the results was misleading:

Here’s an alternative way of presenting the findings that’s at least as valid. There were 10 questions that were meant to be answered with the name of one of the candidates. Five of them covered events that had surfaced in the three months before the election: McCain’s inability to say how many homes he owned; Palin’s pregnant teenage daughter; Palin’s wardrobe budget; Obama’s intent to redistribute wealth; and Biden’s comment that Obama would be tested by an international crisis. On those questions, more than half of Obama voters correctly identified the involved candidate (the percentage for the last question was lowest, at 53.3%; on the others, it was in the 80s).

...

...and cast in doubt the selection of the questions in proving that Obama voters did not know much about the man they where voting for:

Apart from the exclusion of McCain supporters and whether the poll’s questions were representative of overall campaign knowledge, the poll didn’t demonstrate that the news media favored Obama, or that any media distortions “got Obama elected.” No questions addressed how voters got their information or how the answers to the questions influenced their vote. It may well be that supporters of each candidate gravitate toward media that downplay the shortcomings of their own candidates and highlight those of opponents — or simply that they retain knowledge that conforms with their world view.

The significantly more partisan blogosphere was considerably more harsh, and accused the questions themselves of being misleading.

The wisdom of "The 20 Questions about Polls" advice to look at the sponsor of the poll never rang truer.

Further, even if we assume that there are no methodological issues with this poll, there is the simple causation/correlation issue. As we discussed extensively in the beginning of the semester, people flock to "like media." Conservatives tend to watch Fox News and liberals MSNBC. So to, cognitive psychology has documented the existence of confirmation bias, or the phenomenon where individuals pay more attention to examples that confirm their preexisting beliefs and assume these examples are more common. This is a misuse of a heuristic called the "representative heuristic", where individuals assume that single example represents the character of a larger group. Thus, two people can watch the exact same coverage and come away with different memories. The poll does not prove that media coverage was biased, rather that Obama supporters were biased in favor of Obama.

In an attempt to correct this last issue, Ziegler commissioned a poll asking both McCain and Obama voters the same questions about the negatives of both candidates. In this case, we can assume the effect of "like media" is negated. The results:

Here are the highlights:

35 % of McCain voters got 10 or more of 13 questions correct.

18% of Obama voters got 10 or more of 13 questions correct.

McCain voters knew which party controls congress by a 63-27 margin.

Obama voters got the “congressional control” question wrong by 43-41.

Those that got "congressional control" correct voted 56-43 for McCain.

Those that got "congressional control" wrong voted 65-35 for Obama.

The 35% number is very important. It shows, assuming there are no methodological problems with the poll, a big IF, that even McCain supporters did not know that much negative about Obama. If the phenomenon of "like media" is to be blamed for the Obama-supporter's answers, then we should expect the McCain people to do much better on Obama Negatives. The problem is that Zeigler has not released the exact details of the last poll, question by question. But this new, companion poll does at least begin to remove the "like-media" issue, and perhaps (if we assume the poll to be fair) demonstrates that the media was not vocal enough about the issues expressed in the poll's questions.