News

By Patrick Barritt, Associate Consultant

Over the past few weeks and months, and this week to come, leading up to the election, you’re going to see a lot of news outlets offer you “what the polls are saying.” Today, we’ll give you a quick guide on how to review this data with a healthy degree of skepticism. 

Before we get to the guide, let’s go over how poll data is collected. Typically, data is conducted via mobile or desktop surveys, and two key questions are asked: 

“If the vote were today, who would you vote for?” 

“Who do you think will win the election?” 

These sound similar, but they offer very different kinds of data — the first being predictive of one’s own behavior, and the other predicting the outcome of a future. It is frequently more accurate to ask people what they believe others will do than it is to ask them what they themselves will do. Doing this helps reduce biases and social desirability. 

Secondly, polls reflect only the opinions in that exact moment (aka a snapshot study) and don’t account for the impact of external forces like social pressures and campaign influence. 

So with this in mind, lets go over our Skeptic’s Guide to Poll Results. 

Is it a national poll, or a state poll?

• State polls are more helpful, since that’s how the US presidential election works. National polls would be useful if the popular vote was what determined the winner.
• Remember, national polls are not helpful, since elections aren’t won by the popular vote (if that were the case, polls are typically quite accurate). 
• It’s good practice to look at the figures in key battle/swing states. Keep an eye on key battleground states like Arizona, Florida, Georgia, Michigan, Minnesota, North Carolina, Pennsylvania and Wisconsin (according to Politico.com).

Who is in the sample?

• This information can be hard to find, but is very important to knowing how reliable a datasource is. A dataset should be as representative of the overall population as possible. This is particularly difficult to do across all 50 states, so analysts will weight the data. Weighting can become problematic, as it allows some numbers to become inflated and over-represent information provided by small samples. 
• it is very difficult to accurately reflect national demographics, age, race, gender identity, students/graduates, and so on. On top of that, not all eligible voters actually go out and vote, so creating a representative sample of known national demographics may not be fully reflective of the ultimate voting population. Therefore, the data must be weighted in order to balance toward a more representative sample. 

When was the data collected?

• It can take a week to 10 days to analyze datasets as large as many of these polls. So when you see the numbers, they’re already old, and aren’t accounting for recent media coverage, campaign advertisements, etc.  

Who ran the poll?

• Was it conducted by a third party trying to convey unbiased data to inform the public, or are these numbers pushing an opinion?

Which data points are the most important?

• Rather than placing a lot of focus on a single poll or measurement, it’s better to look at the average across the more reputable pollsters.

What is the margin of error (MoE) of the sample?

• As described by the Pew Research Center, “The margin of sampling error describes how close we can reasonably expect a survey result to fall relative to the true population value. A margin of error of plus or minus 3 percentage points at the 95% confidence level means that if we fielded the same survey 100 times, we would expect the result to be within 3 percentage points of the true population value 95 of those times.”
• The above explanation means that there is potential for percentages to overlap. Typically, the MoE for polls is around 3%. This can be significantly higher when looking at demographic subgroups or minority groups because the sample sizes will be smaller. Therefore their responses need to be “weighted” up.

All in all, polling is helpful as a guide, but not so helpful as a hard-and-fast prediction of what an outcome of something like a presidential election will be. An oft-touted argument made against polls is the predictions made during the 2016 presidential race, where Hilary Clinton was predicted in most polls to be the likely winner. This “prediction” was ultimately incorrect as Donal Trump won the presidency, but that doesn’t reveal the whole story. The polls represent voter count, and in 2016, they had Hilary ahead by 3.2 points. By the final tally of the popular vote, Hilary Clinton won by 2.1%. Viewing the numbers this way, the polls were actually quite accurate.

At RedSky we run poll-like surveys all the time, and cast acute attention to the sampling techniques and testing methodologies we use to ensure we don’t run into the biases presented in this guide. So if you want to talk to us about how we do that, feel free to get in touch. For now, please exercise your rights and go vote!