Art and Science

16 Sep 2004|Christoper Ireland

We all woke up this morning to learn that Bush leads Kerry in the polls by 13 pts (Gallup). No, make that Kerry leads Bush by 1 pt. (Harris Interactive). Oops, no make that Bush leads Kerry by 1 pt (Pew Research). In statistical terms, the difference between the Harris and Pew data is insignificant, but the difference between them and the Gallup poll is a whopper.

How could this happen? All these firms are reputable, high quality research organizations with plenty of relevant experience in this field. Their sample sizes are large and their polling methods are sound. Is one side right? And if so, which one?

Most people think that quantitative data is the most objective form of research. They imagine that it’s a refined science with hard-coded rules that make it replicable within a small margin of error. But quantitative surveys have their “art” elements as well. And this is where the polls cited earlier diverge.

In this case, the “art” is the assumptions that determine who is a “likely voter,” along with the rationale behind the weighting of the data to match a national sample. You see, these polls are not just adding up all the “Kerry” and “Bush” votes–they are modifying the final responses based on what they believe is valid. All of them are technically “right” depending on what you believe to be true about people. There’s nothing unethical or shady about this–but if you care about these things–you should check to see who’s assumptions match your own.

While you may not be running for president anytime soon, if you’re in business you probably have plenty of quantitative research programs going. It’s not a bad idea to check out how they’re being conducted. Obviously, you want to make sure they’ve got the science part right, but don’t forget to ask about the art.

prev next