Internal campaign polls -- a brief primer

Amy Klobuchar's pollster, Anna Greenberg, released a poll earlier this month showing Klobuchar ahead of Kennedy 50-42. The release seems to have disappeared from the Greenberg Quinlan Rosner website, but an excerpt is still up at DFL Senate.

The DFL Senate folks write, "keep in mind this is Democratic polling." Chris Cillizza also offers caution:

Since the numbers were provided by Klobuchar's pollster, we take them with a grain of salt. But if Greenberg's numbers are anywhere close to where public sentiment actually lies . . .

I thought I should offer some thoughts on exactly what the bias is that everyone is worried about -- so you can judge a little better how much salt to add.

First of all, let's clearly state that pollsters who work for partisan campaigns do not fudge their numbers, nor do they generally ask questions in such a way as to tilt the results in any particular direction (with a major exception I'll get to below).

Although playing fast and loose with proper polling methodology to produce favorable results might occasionally be tempting in the context of a particular campaign, it would be extremely unwise in the long run. In order to recruit new clients every election cycle, pollsters have to develop and maintain a reputation for accuracy. No campaign wants to spend thousands of dollars for questionable numbers. Candidates and campaign managers want and need the best possible estimate of where they truly stand.

Further, pollsters are extremely unlikely to conspire with campaigns to release information that does not represent what they found. Remember that there is a world of professional campaign staffers who move from one campaign to another every cycle and who gossip and trade stories among themselves. It would raidly become known within the political community (and then within the political press corps) if a particular firm was willing to misreport data. Again, that would tremendously weaken a firm's ability to attract new clients in the long run -- who wants a pollster that no one outside the campaign will believe?

So, they don't use bad methodology and they don't lie -- so why don't we treat these polls the same way that we treat media polls? The concern is not misreporting, but selective reporting.

Pollsters who are working for a campaign don't release results unless they and the campaign agree that it's in the campaign's best interest to do so. When news media organizations put a poll in the field, they report on its results no matter what they find. Campaign pollsters are likely to report results for only those questions where their candidates scored impressively. News media organizations will report the results for every question they ask -- and they'll report the demographic breakdowns for every question too. Campaign pollsters will highlight the demographic differences that are most dramatic or provide the best "hook" for favorable media stories. For example:

Among Independent voters, Klobuchar leads 47 - 34 percent. At 30 percent of the electorate, the Independent vote in Minnesota clearly has much to say on the outcome of this race.

(from the Greenberg press release as quoted by DFL Senate)

A couple of other things to note: campaign pollsters will trumpet minor poll-to-poll differences which could just be statistical noise. News media organizations . . . well, news media organizations do that too.

Campaign pollsters are also willing to show off the results of an internal poll even if it is different from all of their previous polls and all other polls in the race -- especially if it suggests that their candidate is doing better -- despite the fact that all of the other results suggest that this poll is a statistical outlier (and these are smart folks, they know this). News organizations will also tend to ignore evidence suggesting that their poll is so different that it is probably an outlier (not that these are not smart folks, but few political reporters have the statistical training that practically all professional pollsters have).

Finally, that exception I was talking about earlier -- where pollsters do try to "tilt" the polling data by the way they ask questions. Internal campaign polls commonly "test" campaign messages that are being used either by their own client's campaign or by the opponents. Different pollsters do this in different ways, but one approach is as follows:

A horse race question is asked at pretty much the beginning of the poll ("if the election were being held today, would you vote for Amy Klobuchar, the Democrat, or Mark Kennedy, the Republican?"). Then the questioner provides the respondent with some information:

"I'd like to read you a statement about Amy Klobuchar. Amy Klobuchar has refused to stand behind our troops and has called for a removal of our forces from Iraq."

And then the horse race question is asked again. By looking at how many people change their votes before and after the statement, the pollster tries to judge what the most effective messages are likely to be in the campaign. Note that pollsters frequently test the campaign messages of their opponents so they can see how much their own candidate might be hurt, so if you happen to get called to respond to a poll, there is no way you can tell during the phone call which side is doing the polling. In fact, sometimes they'll do the attack, then the horse race question, then the campaign's response, then a third horse race question.

Because of the concerns I cited before about protecting their reputation, pollsters aren't going to publish the horse race numbers that come after the test messages. What they'll do is advise the campaigns to focus their campaigning on the most effective messages. If those messages do truly resonate with the population, the pollster will get to release nice numbers for their candidate later on.

So, bottom line: I think we can conclude with reasonable confidence that Klobuchar was ahead of Kennedy by more than the margin of error when this poll was taken. (I'm assuming the MoE was in the usual 2-3 percent range, and by "reasonable confidence" I'm taking into account that there's a 5% chance that even a perfectly-conducted poll will miss the true measurement by more than the MoE). But if a later Greenberg poll shows the race to be tightening, I wouldn't expect them to let us know.

Full disclosure: As one of my first jobs out of college, I worked for two of Anna Greenberg's partners (her dad Stan and Al Quinlan) back in the 1990 election cycle at a firm headed by Stan Greenberg and Celinda Lake. I haven't interacted with any of those folks since then except for trading some emails recently with Mystery Pollster (who was then a young analyst working for Stan and Celinda -- he's got his own firm now).

posted by DK @ 3:57 PM permalink