Feeds:
Posts
Comments

Archive for November 7th, 2008

I have often expressed my admiration for Nate Silver’s wonderful polling aggregate/analysis website, 538. However, there are at least two other very popular sites which aggregate polls : Real Clear Politics and Pollster. How did Nate do in comparison to them?

First of all, all three sites were pretty good in calling the winner of each state. 538 only got Indiana — and possibly one of Nebraska’s congressional districts — wrong. Nate called these two slightly for McCain, and Obama won. RCP missed these two as well; they also got North Carolina wrong. Pollster missed both these as well as Missouri. However all these states were really close and it would be unfair to read too much into them.

Time for a more thorough analysis. I will do the following thing – list all the IMPORTANT states, see by how much 538, RCP and Pollster were off in their predicted margins, and calculate who made the least error.

I define an important state as any state in which the final actual margin was 15 points or less. Thus, this definition includes all the swing states as well many of the ‘safe’ states. The reason I only take these states is that it seems somewhat more important for a polling site to get these right. Besides, it saves me some labor.

State            EVs       Obama Margin       538 error      RCP error   Pollster error

AZ                10                      -8.6                  3.7               5.1               3.7

CO                 9                        7.8                  1.2               2.3                 .2

FL                 27                        2.5                   .8                 .7                 .8

GA                15                       -5.5                 1.8               1.5               2.6

IN                 11                          .9                 2.4               2.3                2.1

IA                   7                         9.3                 2.4               6.0               3.6

MN               10                       10.2                   .1                .4                2.2

MS                  6                       -13.8                 2.5             2.4               3.3

MO                 11                      -.2                     0                  .5               1.3

MT                  3                        -2.5                   .2                1.3               .3

NV                  5                        12.4                 7.5               5.9               5.3

NH                  3                         9.5                   .3                1.1              2.4

NJ                   15                      14.6                 .1                 .9                 .9

NM                  5                       14.7                 5.0                 7.4             5.8

NC                 15                          .4                  .6                  .8                0

ND(*)              3                        -8.6               5.9                9.6               9.3

OH                 20                         4                   .6                 1.5               .9

PA                  21                      10.3               2.2                 3.0               3.1

SC                  8                        -8.9                  .8                1.1                .4

SD                   3                       -8.5                  .2                 .2                 .5

TX                  34                     -11.7                 .9                1.3               1.4

VA                 13                          5.5                 .1                 1.1                .1

WV                  5                       -13.1              3.3                 4.1              1.7

WI                 10                         13.9              2.4                 2.9              1.9

Ok, so who is the winner?

One way to see that is to simply take the median error. Pollster has a median error of 1.8, while RCP gets a surprisingly good 1.5. But 538? It’s median error is an amazing 1.05.

A slightly different measure is to take the root-mean-square error. This number is 3.1 for Pollster and 3.58 for RCP. The clear winner, once again, is 538, with a root mean square error of only 2.71.

Other methods, such as weighting the errors by electoral votes, only increases 538’s advantage.

Nate Silver rocks.

Read Full Post »