How to ask questions

Polinaut is asking for input on 'Select A Candidate', a little online tool that MPR developed. Here's the idea:

So we came up with the idea of Select A Candidate to make it more interesting -- dare I say: fun? -- for people to get to know the candidates. With Select A Candidate you take a quiz asking you your opinion on a set of issues (I like to keep it around 12-15), hit SUBMIT and then see a stacked list of the order of all candidates and how they matched you. You'll also see a cumulative list of how all the candidates stacked up with everyone taking the quiz (it's a poor man's poll but you hard-core candidate supporters can forget about trying to up the score for your candidate. It won't work that way). And you can see what issues are driving the electorate.

Now, I don't want to go too overboard in critiquing this particular project, because it's not really intended to be very serious. Not many people are going to answer a couple of questions at the MPR website, view the results, and decide who to vote for while ignoring all other information that comes their way. That being said, the project does provide a nice jumping off point for some thoughts about polling, public opinion, campaign issues, and question wording. (After all, MPR is apparently looking for some help with polling anyway.)

So, to continue with Select a Candidate:

There's still some work to do and this is a good time for you to get involved. In the interest of transparency, here's the quiz, so far, for the governor's race. Add to it, modify it with your suggestions, make corrections.

1. Do you support or oppose a constitutional amendment that would define marriage -- or its legal equivalent -- as between one man and one woman only?
2. Do you support or oppose an increase in the state's gasoline tax to pay for transportation projects in Minnesota?
3. Which of the following statements most closely matches your view on the issue of health care in Minnesota?

[more questions omitted, you get the idea]

I could choose any of these questions to focus on, but let's take the first one. The problem is that this question frames the issue in a very specific way -- and I suspect that many opponents of the amendment would characterize this language as fairly friendly to the Pawlenty/Bachmann point of view. I've thrown together a few alternatives, each of which frame the issue differently.

a. Should the Minnesota Legislature devote a significant portion of its time this session to debating a proposed constitutional amendment that would define marriage as between one man and one woman only?

b. Do you support or oppose an amendment to the state constitution that would prohibit civil unions between gay couples?

c. State law currently defines marriage as between one man and one woman only. Do you support adding this definition to the state constitution through a consitutional amendment?

d. Should the state legislature prevent Minnesotans from voting on whether to pass a constitutional amendment which would define marriage or its legal equivalent as between one man and one woman only?

The point here is not to criticize Polinaut over question wording. Sure, I could start a vicious, protracted, and totally pointless debate in the blogosphere by charging that the original question is unfair, that one of these alternatives would be better, and that this is another example of the conservative/liberal bias in the MSM. (And in fact, that seems to be the usual direction for this sort of discussion.)

Instead, let's take a step back and look at the big picture. The problem does not have to do with the wording of the question. There are lots of ways to ask about any issue, and reasonable people can differ over how the issue ought to be framed by the question. The problem has to do with the premise that a single question can accurately capture public opinion (or a candidate's opinion) on any given issue. That puts a lot of weight on the wording of that one question -- more weight than most questions can handle.

Look at it this way. What if we asked 500 randomly chosen voters each of the questions above? Chances are good that we would not get an identical distribution of responses. But we wouldn't therefore conclude that one question was better than the others. The proper conclusion would be that public opinion on this issue is complex and not easily captured by any single one of these questions standing on its own. Further examination of how voters responded differently as the questions changed would reveal more about where the electorate stands than any one of the polls taken on its own.

Candidates also hold complex opinions, though one might be more likely to get consistent answers on this set of questions. But if, for example, you interviewed each of the candidates for an hour and asked them nothing but questions about the role of government as it relates to marriage and personal relationships, you'd end up with a much more nuanced picture of their views than you would from their prepared thirty-second soundbites on the Bachmann amendment.

Here's one interesting conclusion from this line of reasoning: we practically always have a weaker understanding of state-level public opinion than we do of national-level public opinion. There are lots of different polling organizations going into the field all the time at the national level, resulting in a number of differently worded questions about any major issue. With some careful analysis of how different wording produces different responses, we can learn quite a bit more than we can from any single survey. (Not that anyone in the media ever does this -- reporters can't seem to handle writing about more than one poll at a time, for some reason.)

Further, looking at multiple polls simultaneously makes it easier to identify those polls which fall outside the 'margin of error'. I won't get into the statistical principles here, but even if everything is done perfectly polls should miss a true measurement by more than the margin of error about once every 20 times. If you look at a lot of polls together though, the outliers are easier to spot.

At the state level, on the other hand, there aren't lots of different questions asked about gay rights, professional sports stadiums, light rail, or any other issue of the day simply because there aren't a lot of people doing the asking. And those that are doing the asking aren't asking very often. This is not to suggest that the Strib or the Humphrey Institute do a poor job on their polls -- on the contrary, I think they're quite good. (In fact, most of the professional political pollsters hired by state parties and campaigns are pretty good too.) But due to logistical and cost considerations, these outfits can only do statewide polls occasionally, and when they do they can only ask so many questions.

I'm not saying that we can't learn anything from state-level polls. Rather, I am saying that we learn something from each question that is asked. The more questions we ask, the more we can learn. And when we can't ask as many questions, we ought to be rather more cautious about the confidence we express in our analysis. (BTW, if you want to increase your overall learning from public opinion polls, I highly recommend regular reading of Mystery Pollster and Political Arithmetik.)

One important note: the entire discussion above is directed toward polling about issues. Polling about election outcomes ("if the election were held today, would you vote for . . .") and frequently repeated items like approval ratings are much less susceptible to framing effects from question wording.

Back to Select A Candidate, which is not a poll, and is not trying to be one. I think the change that I would most like to see is some other way of comparing the voter's choices to the candidate's positions. Here's one idea -- instead of coming up with questions yourself, take the actual text of candidate's statements about an issue and put them side-by-side with the names removed -- and ask the voters to select which statement they agree with most. Then, at the end, the voters can see a scoresheet of how many times they agreed with Pawlenty, Hatch, Lourey, Kelley, Doran, etc.

Not that this is perfect either -- it would take a lot more time to put together, for one thing. For another, using the candidates' rhetoric may make it more difficult to distinguish between differing stances. So I'm not necessarily suggesting that this is the best option. But such an approach would reduce the extent to which the framing of the questions affects the voter's comparison of the candidates.

posted by DK @ 12:19 AM permalink