With polls drawing increased attention in the closing weeks of the presidential race, perhaps it is no surprise that when supporters of one candidate do not like the numbers they are seeing, they tend to blame the messenger.
In 2004, Democratic Web sites were convinced that the polls were biased for President George W. Bush, saying they showed an implausible gain in the number of voters identifying as Republicans. But in fact, the polls were very near the actual result.
Mr. Bush defeated John Kerry by 2.5 percentage points, slightly better than the one- or two-point lead that he had on average in the final polls. Surveys of voters leaving polling places that year found an equal number of voters describing themselves as Democrats and Republicans, also close to what the polls had predicted.
Since President Obama gained ground in the polls after the Democratic National Convention, it has been the Republicans’ turn to make the same accusations of bias. Some have said the polls are “oversampling” Democrats and producing results that are biased in Mr. Obama’s favor. One Web site, unskewedpolls.com, contends that even Fox News is part of the racket in what it says is a “trend of skewed polls that oversample Democratic voters to produce results favorable for the president.”
The criticisms are largely unsound, especially when couched in terms like “oversampling,” which implies pollsters are deliberately rigging their samples.
Pollsters, at least if they are following the industry’s standard guidelines, do not choose how many Democrats, Republicans or independent voters to put into their samples — any more than they choose the number of voters for Mr. Obama or Mitt Romney. Instead, this is determined by the responses of the voters that they reach after calling random numbers from telephone directories or registered voter lists.
Data suggest that polling in presidential elections has no history of partisan bias, at least not on a consistent basis. There have been years, like 1980 and 1994, when the polls did underestimate the standing of Republicans. But there have been others, like 2000 and 2006, when they underestimated the standing of Democrats.
To examine for bias, it is useful to examine all presidential polls since 1972, the year when the number of presidential polls began to proliferate.
In the 10 presidential elections from 1972 on, there have been five years (1976, 1980, 1992, 1996 and 2004) in which the national presidential polls of likely voters overestimated the standing of the Democratic candidate. However, there were also four years (1972, 1984, 1988 and 2000) in which they overestimated the standing of the Republican. Finally, there was 2008, when the average of likely voter polls showed Mr. Obama winning by 7.3 percentage points, his exact margin of victory over John McCain.
In all but three years, the partisan bias in the polls was small, with the polling average coming within 1.5 percentage points of the actual result. (I use the term “bias” in a statistical sense, meaning simply that the results tended to miss toward one direction.)
The first major exception was 1980, when late polls showed Ronald Reagan leading Jimmy Carter by only two or three percentage points on average — but Mr. Reagan won by almost 10 points. There were some complicating factors that year: the first and only debate between Mr. Carter and Mr. Reagan was held very late in the election cycle, perhaps too late to be captured by the polls. In addition, that race had a third-party candidate, John Anderson, an independent, and third-party candidates contribute significantly to polling volatility. Also, some private polls of the campaign showed Reagan with a much wider advantage.
Still, it is hard to make too many excuses for the polls: 1980 was probably the worst year for them since 1948, when the Gallup poll favored the Republican candidate, Gov. Thomas E. Dewey of New York, but the Democratic incumbent, Harry S. Truman, won instead.
In 1980, the miss was in Reagan’s favor, meaning that the polls had a Democratic bias. But one does not have to go back to 1948 to find a year when they had a Republican bias instead. In 2000, national polls showed Mr. Bush winning the popular vote by about three percentage points — but Al Gore narrowly won the popular vote.
The other year in which the polls were reasonably poor was 1996, when most of the national polls projected President Bill Clinton to win re-election by double digits, but he defeated Bob Dole by 8.5 percentage points. The results received little attention since Mr. Clinton’s victory was not in any real doubt before or after the election. But the polls had a Democratic bias that year, as they had in 1980.
Over the long term, however, the polls have been about as likely to miss in either direction. Since 1980, they have overestimated the Democratic candidate’s margin by an average of 0.9 percentage points and by a median of 0.3 percentage points. These errors are so modest that they cannot really be distinguished from statistical noise.
We can also evaluate whether there was bias in the polls of Senate races. In some ways, this is a much richer data set because there are different candidates and different conditions in each of the 33 or 34 states that hold Senate contests every two years.
As in the case of presidential polls, there have been years in which most of the Senate polls missed in the same direction. Senate polls had a Democratic bias in 1992 and 1994 but a Republican bias in 1998, 2000 and 2006.
(A Republican bias, although it was very modest, shows up in 2010. The two Senate races that the FiveThirtyEight forecasts “called” wrong in 2010 were Colorado and Nevada, where the polls had Republicans as favored but where Democrats won instead.)
But as in the case of the presidential polls, the years in which the Senate polls missed in either direction have tended to cancel one another out. On average across 240 Senate races since 1990, the polls have had a Republican bias of just 0.4 percentage points, a trivial number that is of little meaning statistically.
On the whole, it is reasonably impressive how unbiased the polls have been. In both presidential and Senate races, the bias has been less than a full percentage point over the long run, and it has run in opposite directions.
That does not mean the pollsters will necessarily get this particular election right. Years like 1980 suggest that there are sometimes errors in the polls that are much larger than can be explained through sampling error alone. The probability estimates attached to the FiveThirtyEight forecasts are based on how the polls have performed historically in practice, and not how well they claim to do in theory.
But if there is such an error, the historical evidence suggests that it is about equally likely to run in either direction.