Accuracy in Polling

When I first started doing political polling in 1998 I knew that whatever the results were they would be controversial: one party, one leader would fare better than the other and in a charged political climate the messenger would be attacked.  My company was the lone (and much maligned) voice predicting a PLP victory in 1998 solely due to polling data but our methods were vindicated when the results came in. I knew then that  an important part of my focus was necessarily to ensure international best practices were followed at all levels and there was integrity in the results I published.

It is unfortunate that the MindMaps political survey falls short of this mark, rendering many of its findings inaccurate. The international standard question on “approval ratings” is based on a simple construct: “do you approve or disapprove”. The answer is either “yes”, “no” or “not sure”. This is the format used by CNN, IPSOS, Gallup, Pew, Zogby and every other reputable polling company world-wide. There are two reasons for this: (1) it allows comparisons of polls by different companies and it allows comparisons of leaders, for example and (2) it allows for clarity of a response, with little interpretation.

Comparison of polls is critical in a democracy since they effectively act as informal policing of quality control. If different companies ask the same questions yet get widely different answers the public will want to know why and questions of competence and manipulation will arise.  Further, by asking the same question worldwide we can compare approval ratings for Bermuda’s leaders with other leaders and gain greater insight into the drivers of public sentiment.

In contrast, the MindMaps survey, by using a scale of 1 to 5 (in industry speak it is called a Likert scale) asks respondents about the intensity of their approval, not simply whether or not they approve or disapprove. Why they would violate such a fundamental tenet of political polling is disconcerting since it raises questions about both their competence and motives.  With this approach, there are two positive and two negative responses and a mid-point of uncertainty, producing exactly the ambiguity the international standard question format was designed to prevent.  More importantly, those who have opted for the mid-point have their views discounted altogether , being placed in neither the positive nor negative category.  But what can you say about these respondents’ approval or disapproval of a leader? In fact, the MindMaps survey is not an assessment of approval ratings of leaders and should not be published as such – it simply tells us how strongly people approve or disapprove their leadership. And these are two quite distinct issues.

A different example may help illustrate the point. If I were to ask how strongly do you approve an amendment to the Human Rights Act to include sexual orientation I am certain I would get quite a different answer than if I simply asked if you approve its inclusion or not.

An equally important issue is the integrity in data collection. Any company which allows its staff to hand out forms to friends and colleagues to complete violates the central tenet of polling—randomness in data collection—and thus renders the data useless. As a result, any findings published or presented to clients will have no validity. For companies and political parties making decisions on bad data, they may well be in for some costly surprises.

Political polling is an important part of informing public opinion and giving the public a voice on many things political. Good polling provides reliable actionable data that can both inform strategies and help mobilize support. Bad polling, no matter how well it is packaged, provides no insight and will inevitably lead to flawed strategies because of its flawed premise. We need to move to a higher standard.

Note: This was first published as a Facebook Note December 2011.