Sullivan and Public Policy Polling

Today, Andrew Sullivan linked to this survey from Public Policy Polling, which seems to show that 52 percent of Republicans believe ACORN stole the election for President Obama. I've written to Sullivan before criticizing the results of another survey from Public Policy Polling.

A few points:

First, Public Policy Polling isn't exactly an independent polling agency. Josh Krausharr of Politico writes:

Though there’s little to indicate the firm’s Democratic affiliation on its website — clients are listed, but without partisan identification — [pollster Tom Jensen] said PPP makes no secret of its politics.

This certainly doesn't mean that Public Policy Polling is doing anything nefarious. But shouldn't this conflict of interest be noted? What if the situation were reversed, and a Republican Party-affiliated pollster found that 52 percent of Democrats believed that the Republican National Committee manipulated the election results?

Second, Public Policy Polling often gets results that are out of line with other mainstream pollsters.

For example, in the very same survey that Sullivan links to, Public Policy Polling reports that only 77 percent of African-Americans approve of the job that President Obama is doing. Gallup currently puts the figure at 95 percent for this demographic group (with a monthly average of approximately 92 percent, and a comparable yearly average).

Third, only 33 percent of the 1066 respondents in this survey identified themselves as Republicans. Thus, the sample size for Republicans is about 352, and the sampling error around 52 percent is +/- 5 percentage points. Public Policy Polling notes that "[o]ter factors, such as refusal to be interviewed and weighting, may introduce additional error that is more difficult to quantify." So, the overall margin of error around the 52 percent figure could potentially be quite high.

Perhaps more importantly, Public Policy Polling uses Interactive Voice Response (IVR) technology to conduct its telephone interviews. As Brian Schaffner of Pollster points out, the use of IVR technology in polling is still quite controversial.

Schaffner explains:

[O]ne of the reasons for concerns with IVR polling is that citizens with only a cell phone cannot be reached by these pollsters and these citizens now comprise at least one-fifth of the population. Yet, while the cell-only problem may generally be an issue for IVR technology (and for live interview pollsters who aren't calling cell phones), it is less of a problem for polling on elections, and particularly in low turnout elections.

. . .

Where these polls may run into greater challenges is when they attempt to make inferences about the American public rather than registered (or likely) voters.

I think Sullivan should really consider some of these points before he links to another survey from Public Policy Polling without providing any context.

Update: The NYT will not publish the results of IVR polls.