Questionable Polling: What to Look Out For

At SoonerPoll, we think it is important to provide the public with information about polls they may encounter that are not based on solid, scientific methods. An informed public should have the information it needs to distinguish between questionable polls and polls they can rely on.

We often receive calls and emails about the results of various polls reported by the media. People want to know if they can take the results of these polls seriously. Were they really based upon solid scientific research methods? With the help of several research associations, we have compiled information about some non-scientific polls of which an informed public should be wary.

Science or Entertainment?

Many people these days are using the term “poll” or “survey,” but these terms may not always be used these days to indicate a scientifically-conducted poll or survey.

Do not be fooled.

One good indication is if a newspaper article doesn’t report the sample size and the margin of error of the poll, then it was NOT conducted scientifically and was printed for “entertainment” purposes only. Many times, the sample sizes are very small and any conclusions should be highly questionable.

This is an injustice to the typical reader who might believe the poll was a scientific poll, but nonetheless some disreputable pollsters will try and make it look that way.

Internet Surveys

A growing trend in public opinion polls has been that of Internet surveys. For the most part, these surveys are not conducted in a scientific manner. Results of such polls are often misleading.

Because there are many Americans who do not have access to or are not regular users of the Internet, online polls that claim to measure public opinion in a scientific manner can hardly be expected to do so with accuracy.

Another problem with online polling is the fact that respondents decide for themselves whether or not to participate. In a scientifically valid poll, respondents are targeted by a carefully designed sampling process. Also, people may choose to respond more than once to an Internet poll, further compromising its validity.

Read the American Association of Public Opinion Research’s statement about online polling.

Push Polls

A push poll is where, using the guise of opinion polling, disinformation about a candidate or issue is planted in the minds of those being ‘surveyed’. Push polls are designed to shape, rather than measure, public opinion.  A type of political telemarketing, push polls are only effective if they can call more voters than just a representative sample (the goal of legitimate pollsters).

But, not all questions that seem negative are part of push polls. As Kathy Frankovic, Director of Surveys for CBS News noted, “Candidate organizations sometimes do actual polls that contain negative information about the opposing candidate. These polls, which are not push polls, are conducted for the same reasons market and advertising researchers do their work: to see what kinds of themes and packages move the public”.

In the advertisers’ case, they want to figure out the best way to reach buyers; candidate pollsters need to motivate voters. Polls done for campaign research are full-length, with more topics than just questions about the opponent, and include demographic questions that allow researchers to categorize respondents. Interviewers will not ask to speak to anyone by name, but are calling a sample of randomly selected telephone numbers.

How do you distinguish a push poll from the legitimate poll?  According to Mark Blumenthal, the Mystery Pollster, the proof is in the intent: If the sponsor intends to communicate a message to as many voters as possible rather than measure opinions or test messages among a sample of voters, it qualifies as a “push poll”.

We can usually identify a true push poll by a few characteristics that serve as evidence of that intent. “Push pollsters” (and Mystery Pollster hates that term) aim to reach as many voters as possible, so they typically make tens or even hundreds of thousands of calls. Real surveys usually attempt to interview only a few hundred or perhaps a few thousand respondents (though not always).

Push polls typically ask just a question or two, while real surveys are almost always much longer and typically conclude with demographic questions about the respondent (such as age, race, education, income).

The information presented in a true push poll is usually false or highly distorted, but not always.  A call made for the purposes of disseminating information under the guise of survey is still a fraud – and thus still a “push poll” – even if the facts of the “questions” are technically true or defensible.

Accusations of push polling have become a political attack in itself in recent elections with candidates attacking each other over polls their opponent’s pollster conducted that asked negative questions.  In most cases the polls were legitimate, but each side decided to take advantage of the situation to merely attack their opponent.  In the end, the listening public is typically left in doubt about what was fraud and what was legitimate , and more skeptical and less willing to participate the next time legitimate pollsters, like SoonerPoll, come calling.

For the record, SoonerPoll is a public opinion research firm and does NOT engage in push polling of any type.

University-Organized Surveys for Class Credit.

Surveys conducted by university students as part of a class assignment can be flawed for a number of reasons.

For example, in January 2004, the Modesto Bee reported that a professor at California State University, Stanislaus, required students in his class to conduct public opinion interviews by telephone to receive credit for a course, but failed to provide resources and oversight or to validate that the students carried out the interviews.

The Bee reported that student interviewers received only an hour of training and were expected to pay for long distance charges. Several students came forward to say they falsified their interviews because they were pressed for time and would have to make lengthy long distance telephone calls at their own expense, according to the story. The falsified results of the public opinion poll were introduced as evidence in the Scott Petersen murder trial for possible change of venue in the trial.

Polls You Can Count On

SoonerPoll is a member of the American Association for Public Opinion Research and abides by the Code of Professional Ethics and Practices as set out by that organization. Since SoonerPoll does not outsource any aspect of the polling process, we are able to ensure that our high standards are met throughout the duration of every research project we conduct. It is through the consistent maintenance of those high standards that SoonerPoll has emerged as one of the most trusted, respected researchers in the industry.