The illusion of opinion polls

Posted on June 15th 2009  

Mass media (radio, TV, newspapers) seem not to be very critical towards opinion polls. They simply accept poll results as news facts, and publish them without checking them. This overview of methodological problems of polls may help journalists to put opinion polls in a proper perspective.

Opinion polls play an ever increasing role in the public debate about important issues in society. These polls attempt to measure public opinion. Apparently, mass media en policy makers believe that the poll results accurately reflect attitudes of people. Therefore, it is not uncommon that these results form the basis of policy decisions. Unfortunately, many opinion polls suffer from major methodological flaws, making their outcome unreliable.

Many, if not all, opinion polls in The Netherlands are based on online surveys or online panels. The popularity of these online polls is not surprising. The Internet provides a simple means of getting access to a large group of potential participants. Questionnaires can be distributed at very low costs. No interviewers are needed, and there are no mailing and printing costs. Surveys can be launched very quickly. Little time is lost between the moment the questionnaire is ready and the start of the fieldwork.

At first sight, online polls seem to have much in common with other types of polls. It is just another mode of data collection. Questions are not asked face-to-face or by telephone, but over the Internet. What is different for many online polls, however, is that the scientific principles of survey research have not been applied. This may lead to methodological problems that may seriously affect the survey outcomes.

Some of these methodological problems are described here. They can roughly be divided in two types:

About public opinion

Is there such a thing as public opinion? No, it is an artefact! According to Bishop (2005), public opinion is an illusion. Opinion polls only seem to serve the interests of the political and journalistic elites. Their most important function is perhaps to impose the illusion that public opinion really exists. This public opinion should simply the sum of a number of individual opinions.

With an opinion there is no such thing as a true value. An opinion question attempts to measure a subjective state of a person that cannot be observed by another means. The opinion only exists in the mind of the person. People often do not have formed an opinion about a specific issue. They only start to think about it when confronted with the question. According to the theory of the memory based model people collect all kinds of information from the media and in contacts with other people. Much of this information is stored in memory without paying attention to it. When they have to answer an opinion question, they may recall some of the relevant information stored in memory. Due to the limitations of the human memory, only part of the information is used. This is the information that immediately comes to mind when the question is asked. This is often information that only recently has been stored in memory. Therefore, the memory based model is able to explain why people seem to be unstable in their opinions. Their answer may easily change due to the way the issue was recently covered in the media.

Asking opinion questions

It will be clear that it is not easy to measure the opinion of people on a certain issue. Moreover the way opinion questions are asked in a poll also may seriously affects the way they are answered.

Opinion questions may address topics about which non-respondents may not yet have made up their mind. They may even lack sufficient information for a balanced judgment. Questionnaire designers may sometime provide additional information in the question text. Such information may influence participants in a specific direction. Saris (1997) performed an experiment to show the dangers of changes in the question text. He measured the opinion of the Dutch about increasing the power of the European Parliament.

Respondents were randomly assigned one of these two questions:

Question 1
An increase of the powers of the European Parliament will be at the expense of the national Parliament.

Do you think the powers of the European Parliament should be increased?

Question 2
Many problems cross national borders. For example, 50% of the acid rain in The Netherlands comes from other countries.

Do you think the powers of the European Parliament should be increased?

In case people were offered the question on the left, 33% answered ‘yes’ and 42% answered ‘no’. In case they were offered the question in the right, 53% answered ‘yes’ and only 23% answered ‘no’. These substantial differences are not surprising, as the explanatory text on the left stresses a negative aspect and the text on the right stresses a positive aspect.

The answer to an option question may also be affected by the context in which it is asked. Tiemeijer (2008) mentions an example were the answers to a specific questions were affected by a previous question. The Eurobarometer is an opinion survey in all member states of the European Union held since 1973. The European Commission uses this survey to monitor the evolution of public opinion in the Member States. This may help in making policy decision. The following question was asked in 2007:

Taking everything into consideration, would you say that the country has on balance benefited or not from being a member of the European Union?

Eurobarometer 2007

It turned out that 69% of the respondents were of the opinion that the country had benefited from the EU. A similar question was included at the same time in the Dutch opinion poll Peil.nl (Maurice de Hond). However, the question was preceded by another question that asked respondents to select the most important disadvantages of being a member of the EU. Among the items in the list were the fast extension EU, the possibility of Turkey becoming a member state, the, introduction of the Euro, the waste of money by the European Commission, the loss of identity of the member states, the lack of democratic rights of citizens, veto rights of member states, and possible interference of the European Commission with national issues. As a result, only 43% of the respondents considered membership of the EU beneficial.

Under-coverage

Most opinion polls in The Netherlands use online panels. Examples are Peil.nl (conducted by Maurice de Hond) and Politieke Barometer (conducted by Synovate). All these polls suffer from under-coverage: People without Internet access will never be able to participate in such a poll. If people with Internet differ from those without Internet, polls results will be biased.

According to Eurostat (2007) Internet access is high in The Netherlands. More than four out of five households have an Internet connection. Internet coverage is also high in the Scandinavian countries Sweden and Denmark. Coverage is very low in the Balkan countries Romania and Bulgaria. Only approximately one out of five households there has Internet access. Notwithstanding the high Internet penetration in The Netherlands, there are still groups where access is lower than average. This particularly relates to the low-educated, the elderly and non-natives. See Bethlehem (2008) for a more detailed analysis.

Self-selection

The principles of probability sampling are fundamental for modern survey research. The first ideas emerged a little more than a century ago. For an overview, see e.g. Bethlehem (2009). Reliable estimates can only be computed if a participants are selected by means of probability sampling. Furthermore, only then the accuracy of estimates can be computed.

Unfortunately, most only polls are not based on probability sampling. The survey questionnaire is simply put on the web. Participants are those people who happen to have Internet, visit the website and decide to participate. The poll organization is not in control over the selection process. Therefore, no unbiased estimates can be computed nor can the accuracy of estimates be determined. These polls are called self-selection polls.

Various market research organizations used self-selection opinion polls to predict the outcome of the general elections in The Netherlands in 2006. The results of the three major polls are summarized by Bethlehem (2008). For example, Peil.nl (Maurice de Hond) made a wrong prediction for the Labour Party and the right-wing populists (Wilders). And a survey carried out by Statistics Netherlands (based on probability sampling) produced correct predictions.

Probability sampling has the additional advantage that it provides protection against certain groups in the population attempting to manipulate the outcomes of the poll. Self-selection does not have this safeguard. An example of this effect could be observed in the election of the 2005 Book of the Year award (Dutch: NS Publieksprijs), a high-profile literary prize. The winning book was determined by means of a poll on a website. People could vote for one of the nominated books or mention another book of their choice. More than 90,000 people participated in the survey. The winner turned out to be the new interconfessional Bible translation launched by the Netherlands and Flanders Bible Societies. This book was not nominated, but nevertheless an overwhelming majority (72%) voted for it. This was due to a campaign launched by (among others) Bible societies, a Christian broadcaster and Christian newspaper. Although this was all completely within the rules of the contest, the group of voters could clearly not be considered representative for the Dutch population.

It should be stressed that increasing the sample size will not decrease the problems caused by under-coverage and self-selection. An example of a large online opinion poll in The Netherlands was 21minuten.nl, a survey supposed to supply answers to questions about important problems in Dutch society. Within a period of six weeks in 2006 about 170.000 people completed the online questionnaires. The news program NOVA of the Dutch public television station NPO 3 presented the results of this poll and claimed them to be representative because of the large number of participants. This statement is simply untrue.

Conclusion

The American Association for Public Opinion Research (AAPOR) is the leading professional association for public opinion researchers. It has warned its members against the use of self-selection polls:

Only when a Web-based survey adheres to established principles of scientific data collection can it be characterized as representing the population from which the sample was drawn. But if it uses volunteer respondents, allows respondents to participate in the survey more than once, or excludes portions of the population from participation, it must be characterized as unscientific and is unrepresentative of any population.

American Association for Public Opinion Research

A study across 19 online panels of Dutch market research organizations showed that most of them use self-selection, see Vonk et al. (2006). It became also clear that participants structurally differ on average from the Dutch population.

Poll organizations sometimes attempt to correct for the negative effects of under-coverage and self-selection by applying so-called weighting adjustment procedure. Participants are assigned weights. Those in under-represented groups get a larger weight than those in over-represented groups. Bethlehem (2008) shows that such a correction procedure is absolutely no guarantee that the poll results will be reliable.

In conclusion, one can say that many opinion polls are quick and dirty. They are quick because they are carried out by means of the Internet. And they are dirty because their outcomes are unreliable.

Literature

Read on