Polling New Jersey Republicans

I've been reading a lot about polling and survey design lately, so I thought I'd comment on this article from The Huffington Post, which seems a bit misleading.

The author, Rachel Weiner, cites the results of a recent poll of New Jersey residents, which suggest that one-third of New Jersey Republicans do not believe that President Obama was born in the United States. The results also indicate that 14 percent of New Jersey Republicans believe that President Obama is the Anti-Christ.

Let's break this down a bit.

The sample size for the poll was 500, which means that the margin of error is +/- 4.5%. The pollster also notes that "[o]ther factors, such as refusal to be interviewed and weighting, may introduce additional error that is more difficult to quantify." Thus, nonresponse bias may have contributed an even greater degree of uncertainty.

It's important to point out that this was a poll of "likely New Jersey voters." The sample contained only 165 Republicans, putting the margin of error closer to +/- 7.5% for this group. (For those of you who are interested, here is a good margin of error calculator.)

When we begin to break a small sample size down into different subgroups, the results are subject to a significantly higher degree of uncertainty -- in this case, with the additional caveat about nonresponse bias.

Interestingly, The Huffington Post failed to note that, according to this same poll, roughly one-third of New Jersey Democrats believe that President Bush had advance knowledge of the 9/11 terrorist attacks. And 6 percent of Democrats believe that we should eliminate public education as well as the entire federal government.

We need to be very careful how we interpret results like these from polls with such small sample sizes. You can't simply look at the cross-tabulations and assume that the margin of error remains constant. It doesn't.