Tuesday, February 08, 2005

Now That's A Survey!

It's not generally appreciated just how difficult it is to run a well-done survey. The wording of the questions, the order of the questions, and even the order of the answers all can introduce bias into the data collected. Then there's the methodology in selecting the sample size and the selection of sample members from the larger population. How the survey is presented and in what venue and using what method of communication (face-to-face, phone call, direct mail, etc.) is also of concern. To get it right takes enormous effort and those groups that can do it well are well paid, deservedly.

The Gallup Organization is literally synonymous in this country with polls and surveys. Their latest poll release deals with Bush's approval ratings which is an ongoing kind of event. They basically ask the same set of questions regularly over time and compare the results to plot trends. The latest release of the poll shows Bush's approval rating has increased in some of the more contentious areas and quite significantly. His overall approval rating has climbed to 57%, the highest this year. Of course, the year's still young.

While the numbers are encouraging for his supporters, it's the presentation of the findings of this survey that I'd like to applaud at this point. The conclusions of the survey are presented, of course, but so is the methodology. The number and type of respondents are listed followed by the actual survey questions asked. This is truly a poll you can 1) take yourself and 2) decide for yourself whether the questions are misleading or biased. (My take is they're neither. Gallup is good at what they do.) There are other polls whose results are being touted whose makers could learn a lot from this kind of report. In this age of information access, it's no longer sufficient to announce the big numbers and not allow people to see the data.