|
| |
Reporting the Horse Race: To Poll or Not to Poll?
From: Columbia University
| By:
Steven S. Ross |
EDITOR'S INTRODUCTION |
Few things have become more familiar in campaign news coverage than polls of who's ahead, who's behind and other "horse race" aspects of political races. Polling has evolved from serving as a tool for political campaigns to being the campaign, according to Steven S. Ross, a polling expert and professor of journalism at Columbia University's Graduate School of Journalism. Today's polls help determine campaign tactics, shape the issue stances of politicians and create interest in the race. |
 ampaign polling by the media now borders on the unethical. It's not time to stop polling, but it is time for a new code of ethical practices. The media simply conducts too many bad polls and provides too much uncritical reporting on bad polls conducted by others--including academic institutions. |
Don't get me wrong. I'm all for media polling. But many polls, if not most, are badly designed, badly executed and badly reported. It takes money and talent to poll well. Many media organizations are short of both. Their polls may provide "newsworthy" results only because they are done poorly enough to be "surprising." |
That's a shame, because polling drives almost all of politics today:
- Candidates conduct early, confidential polls to show to their potential funders.
- Candidates poll to find out what their positions are or should be on issues--the same way toothpaste companies poll to find features customers might pay for. This is not quite as obnoxious as it sounds. Various candidates occupy various places in the political spectrum, and use polls more to fine-tune than to make wholesale image transplants. Likewise, toothpaste companies know their business and use market research to refine their product features and to target their competitors.
- During the campaign, candidates "track" their progress with weekly or daily polls, to see how the public responds to their actions and to the actions of their opponents.
|
News organizations must do their own polls, in part to mimic the candidates' internal polling and thus put their actions into context. |
Media horse races
Unfortunately, the media usually chooses to use polls for "horse race" reporting that distracts readers from the shallowness of their reporting on issues. |
In a sense, polling is not simply part of a political campaign. Polling is the campaign--the hidden campaign that few actually see. Long before Election Day, the candidates' carefully planted impressions--guided by polling--have usually decided the outcome. |
The media marvels at how the trailing candidate almost always closes the gap as Election Day approaches--not enough to win, but enough to make the race seem close. What's really happening is that the leading candidate will try to play it safe. Better to let the opponent get close than to announce some new initiative or commit a campaign gaffe that could cost the election! How does the candidate know how far he or she leads the pack? Polling. |
That's why "horse race" polling is both important and misleading. It is important because it helps newspapers understand when a candidate moves into this kind of "prevent defense." It misleads by describing what would happen "if the election were held today." The election isn't being held "today." It is being held on Election Day. |
News organizations tend not to follow their own polling data. As of May 2000, for example, Pat Buchanan had received far more media attention than Ralph Nader--although polls suggested that more American voters would vote for Nader than for Buchanan. (A Zogby poll reported Nader with 5.7 percent to Buchanan's 3.6 percent in early April. While the two remained barely distinguishable statistically, Nader was certainly not behind Buchanan.) |
The Minneapolis Star-Tribune treated Jesse Ventura's candidacy as a joke until he polled about 25 percent of the electorate. He ended up winning the Minnesota governorship with 37 percent of the vote in a three-way race. |
Accuracy in polls
Typical polls are also much less accurate than readers--and editors--believe. Voters are getting harder to sample:
- Most people (some pollsters say 80 percent or more) refuse to talk to pollsters. Today, pollsters have to make five calls to get one valid response--versus two or three calls a decade ago.
- The strong economy allows people to move more frequently, changing their voting address. This makes it more difficult for pollsters to confirm likely voters.
- Movement of people and a new, strong trend toward quasi marriage and racial intermarriage have made it more difficult to "profile" a voting district--determining voters' age and race to draw a representative sample. If 25 black females from a certain district are polled, do they stand for 3,000 likely voters in the district's population or 4,000? If 4,000, then their answers might be magnified in the poll 33 percent more than if they stand for 3,000. Is a family counted as Hispanic if only the husband is Hispanic?
- Voters' opinions are often quite fluid and changeable. When times are good, voters tend to worry about a lot of issues, but not too much about any of them. Small changes in wording can generate different impressions. A Washington Post/ABC poll in early April suggested that the public trusts George W. Bush to do a better job than Al Gore in handling the national economy (47 to 41 percent). But two-thirds of the respondents also said that Gore would do a good job "keeping the economy strong." In late March, a Pew survey had Gore and Bush even (44 to 43 percent) on the general question, but Bush ahead 41 percent to 25 percent on dealing with rising gasoline prices.
- As politicians know too well, voters forget. So do editors. The Vanishing Voter Project at Harvard's Kennedy School of Government found in February 2000 (just after the New Hampshire primary) that 40 percent of the electorate said it knew whether Bush "favors or opposes requiring people to register all the guns they own." Of that 40 percent, only 24 percent actually got it right--that Bush opposes gun registration. By April, 24 percent said they knew the answer, while only 13 percent actually did.
|
Statistical fallibility
Most important, even if a perfect sample of voters can be polled, the results will still not mirror the real world with exactitude. It's the way God made numbers. |
This sounds bizarre at first, but think about what happens when we "poll" a penny about voting heads or tails. We understand intuitively that if a perfectly balanced coin is tossed many thousands of times, the poll will end up even. Half the time the penny will vote "heads" and half the time it will vote "tails." |
We also understand that if we toss the penny only 10 times it may vote seven heads to three tails. We call this a "winning streak." Thousands of tosses have many such streaks, which tend to cancel each other out. |
Choosing a person to be polled is like tossing a penny. One person may say Bush (Heads!). A second person, demographically identical, might choose Gore (Tails!). The pollster might talk to one and not the other. To make sure the streaks cancel out, the pollster must talk to hundreds of voters. |
Not so intuitively, it turns out that such "luck" is predictable. The mathematician Jacques Bernoulli lived between 1654 and 1705, so he never met a senator. But he understood the process. He said that if we poll the coin 1,000 times, we will rarely get a 500-500 tie. But 19 times out of 20 we will get a result between 465 and 535. |
Statisticians call that "19 times out of 20" the confidence level. If you divide 19 by 20, you get 0.95, or 95 percent. The difference between 465 and 500 (or 500 and 535) represents the margin of error--35, in our example. The 35 divided by 1,000 (the size of the sample) is 0.035, or 3.5 percent. |
When reported for the first time, polls usually carry a disclaimer based loosely on Bernoulli's math. A poll of 1,000 people would say that 95 percent of the time the reported results fall within 3 percent of the results that could be expected if the entire electorate were polled. |
That statement is roughly in line with one approved more than a decade ago by the American Statistical Association (ASA). It actually overstates the perfection (Bernoulli's formula would widen the error limits to plus or minus 3.5 percent). Also in line with the ASA, some news organizations note the larger error limits for subsamples. They never say how much larger, however. Thus, if a newspaper polls 1,000 people and 500 are women, the error limit for women's opinions is plus or minus 5 percent, at a 95 percent confidence level. For 100 black females in the sample, the error limit reaches more than 10 percent! |
Remember, this assumes a perfectly drawn sample, which can never be. Furthermore, Bernoulli is fairly generous. Other mathematicians note that the error margin gets larger as the sample gets closer to a 50-50 split. The error margin also increases as the choices increase--in a three-way or four-way race, for instance. |
The "gold standard" for polling is a confidence level of 95 percent--19 times out of 20 the poll will be within a certain percent of reality. With so many polls, however, some are bound to be wrong--even if they are well done by a reputable firm. No poll I've read over the past decade points this out. |
The Wall Street Journal and NBC News, for instance, conducted a joint poll throughout the Dukakis-Bush presidential race of 1988. On October 18, they reported a poll of more than 1,300 likely voters showing Bush ahead by 55 to 38 percent. No other large poll showed such a huge gap between the two candidates. Part of the reason may have had to do with the timing--the poll was taken just after a debate that Bush was deemed to have won. My review of all polls taken during that last campaign month suggests that the poll simply fell among the 1 in 20 outside the error limit. The real gap was probably 50 percent to 43 percent. Of course, the 55 percent to 38 percent gap was widely reported, drying up the last of Dukakis's campaign donation stream. |
Reporting bad polls
It is bad enough that news organizations don't include all the relevant statistical information in their own stories. They tend not to include any such information when they report on polls by others. |
This fools the readers, who assume that news organizations apply their own good news judgment to determine whether the original report is worthy of repetition. In fact, just the opposite is true. A poorly done poll is more prone to error, and thus more likely to produce "surprising" results. Among editors and reporters, another word for "surprising" is "newsworthy." |
The New York Times runs a regular "campaign briefing" column, usually compiled by B. Drummond Ayres Jr. Particularly dangerous, these compilations simplify results in one or two paragraphs. On April 28, for instance, the Times reported that Bush was leading Gore by 8 points in Ohio--47 percent to 39 percent. The poll, taken by the University of Cincinnati, covered only 531 likely voters and was taken over a long period--April 5 through 22. The Times reported the polling techniques, but it should not have reported the poll at all. |
On April 28, the Times also reported an Elway Research poll taken April 11-12 in the state of Washington showing Bush ahead 39 percent to 38 percent--a statistical dead heat, but surprising because the state had been considered Gore territory. The poll included only 400 voters and reported a margin of error of plus or minus 5 percent (6 percent under Bernoulli's Law). Gore could actually be comfortably ahead--and he was, in some polls. |
Better polling practices
We've developed an online calculator that lets readers figure all these polling statistics for themselves. We've also developed an ethics code that should be easy for news organizations to implement, at least on their website. |
The public does like horse races and tends to lose interest when the race is won. The Joan Shorenstein Center on the Press, Politics and Public Policy at Harvard has been polling voters weekly since last fall, asking what (if any) political issues they have been following and discussing among themselves. The center found that public interest was very low throughout the fall of 1999, rose during the spring primaries and then fell sharply when the major party nominees emerged. |
It is up to the press to go beyond the horse race. Perhaps we can start by actually engaging the public in the issues in a meaningful way, and debunking the cheap shots and sound bites that have become commonplace in political ads. |
|
| |
