The Ranking Assumption in 2012 using Intrade Data and in 2014 using Upshot, 538, and PredictWise data

Post Mortem for the 2014 Senate Elections: November 5, 2014

Of the ten states followed below, the Democrats won only MI and NH. Using the final ranking on November 4, 2014, for Upshot and 538 (below), the ranking assumption says that if the Democrats lost NC, which they did, they should also have lost all the states ranked below NC, which they did. So given these two rankings, the ranking assumption is perfect. Note that both Upshot and 538 had probabilities greater than 50 percent for the Democrats winning NC and KS (counting the independent candidate in KS as a Democrat). Even though they were wrong in the sense that the Democrats lost both states, this has no effect on the performance of the ranking assumption. It was perfect even though Upshot and 538 were not.

PredictWise had NC ranked above NH (below). Given that the Democrats lost NC, the ranking assumption says that they should also have lost NH, which they did not. So the ranking assumption made one error using the PredictWise ranking.

Loosely speaking, in the Senate elections there was a wave of Republican support not completely captured in the probabilities. This wave of support was across the country, affecting all probabilities in the same direction, which is consistent with the ranking assumption. The probabilities were not independent, nor even close to being independent. Note that the Democrats almost lost Virginia, where the probabilities were close to 100 percent that the Democrats would win (not collected below).

The Ranking Assumption in 2014 using Upshot, 538, and PredictWise Data

Background

The paper, Interpreting the Predictive Uncertainty of Elections, Journal of Politics, April 2009, provides an interpretation of the uncertainty that exists on election morning as to who will win. The interpretation is based on the theory that there are a number of possible conditions of nature than can exist on election day, of which one is drawn. Political betting markets and sites like Upshot and 538 provide a way of trying to estimate this uncertainty. (Polling standard errors do not provide estimates of this type of uncertainty. They estimate sample-size uncertainty, which can be driven close to zero with a large enough sample.)

This paper also introduces a "ranking assumption," which puts restrictions on the possible conditions of nature that can exist on election day. Take as an example the vote in each state for the Democratic candidate for Senate. Rank the states by the probability that the Democratic candidate wins the state. The ranking assumption says that if the Democrats win state i, they win every state ranked above state i.

Given some ranking for the Senate elections in 2014, the ranking assumption can be tested by simply looking to see after the fact if the Democrats won a state ranked lower than one they lost (contrary to the ranking assumption). I am going to collect probabilities each Tuesday morning from Upshot, 538, and PredictWise. The November 4, 2014, data will be used to test the ranking assumption. The test is a test of the joint hypothesis that the Upshot or 538 or PredictWise probabilities are right and the ranking assumption is right.

November 4, 2014, 6:00 am

I have been following 10 states (see below), where the independent candidate in KS is counted as a Democrat. The final ranking for Upshot (before voting begins) of the probability that the Democrats win the state is:.

Upshot

state	prob
MI	96
NH	75
NC	73
KS	54
GA	40
AL	35	pivot for 50 Democratic seats
IA	34
CO	20
LA	16
AR	11

30 = Upshot probability that the Democrats get 50 or more seats.

The final ranking for 538 is:

538

state	prob
MI	99
NH	79
NC	69
KS	53
IA	30
CO	28	pivot for 50 Democratic seats
AL	26
GA	25
LA	19
AR	4

23.8 = 538 probability that the Democrats get 50 or more seats.

The final ranking for PredictWise is as of 11:00 pm on November 3, 2014, except for KS, which is as of October 28, 2014. (PredictWise did not report values for KS on November 3.) The ranking is:
PredictWise

state prob
MI 100.0
NC 85.0
NH 74.8
KS 48.0 as of October 28, 2014
IA 38.4
CO 23.1 pivot for 50 Democratic seats
AL 23.0
GA 6.9
LA 4.5
AR 0.2

For PredictWise the probability that the Democrats get 50 or more seats under the assumption that the independent candidate in KS is counted as a Democrat is not computed.

October 28, 2014

All but 10 states have probabilities close to 0 or 1 on Upshot and 538. The 10 states for Upshot ranked by probabilities for the Democrats are (the independent candidate in KS is counted as a Democrat): Upshot

state prob
MI 97
NH 74
NC 71
KS 58
GA 48
IA 37 pivot for 50 Democratic seats
AL 29
CO 17
AR 15
LA 6

30 = Upshot probability that the Democrats get 50 or more seats.

The 10 states for 538 ranked by probabilities for the Democrats are:
538

state prob
MI 98
NH 82
NC 70
KS 51
GA 50
IA 34 pivot for 50 Democratic seats
AL 34
CO 22
LA 19
AR 15

36.7 = 538 probability that the Democrats get 50 or more seats.

The 10 states for PredictWise ranked by probabilities for the Democrats are:
PredictWise

state prob
MI 100.0
NH 85.4
NC 76.6
KS 48.0
GA 38.7
IA 28.5 pivot for 50 Democratic seats
CO 17.2
AL 8.7
AR 4.0
LA 2.0

For PredictWise the probability that the Democrats get 50 or more seats under the assumption that the independent candidate in KS is counted as a Democrat is not computed.

October 21, 2014

All but 10 states have probabilities close to 0 or 1 on Upshot and 538. The 10 states for Upshot ranked by probabilities for the Democrats are (the independent candidate in KS is counted as a Democrat): Upshot

state prob
MI 92
NH 81
NC 81
KS 47
GA 39
IA 37 pivot for 50 Democratic seats
CO 32
AR 23
AL 22
LA 8

34 = Upshot probability that the Democrats get 50 or more seats.

The 10 states for 538 ranked by probabilities for the Democrats are:
538

state prob
MI 93
NH 81
NC 71
KS 54
GA 35
IA 35 pivot for 50 Democratic seats
CO 30
AR 25
LA 22
AL 22

36.8 = 538 probability that the Democrats get 50 or more seats.

The 10 states for PredictWise ranked by probabilities for the Democrats are:
PredictWise

state prob
MI 99.8
NH 82.9
NC 70.4
KS 39.0
GA 35.3
IA 28.7 pivot for 50 Democratic seats
CO 24.7
AL 11.8
LA 8.6
AR 5.9

For PredictWise the probability that the Democrats get 50 or more seats under the assumption that the independent candidate in KS is counted as a Democrat is not computed.

October 14, 2014

October 7, 2014

September 29, 2014

All but 9 states have probabilities close to 0 or 1 on Upshot and 538. The 9 states for Upshot ranked by probabilities for the Democrats are (the independent candidate in KS is counted as a Democrat):
Upshot

state prob
NH 82
NC 81
MI 81
KS 56
IA 39
CO 39 pivot for 50 Democratic seats
LA 28
AL 28
AR 19

33 = Upshot probability that the Democrats get 50 or more seats.
The 9 states for 538 ranked by probabilities for the Democrats are:
538

state prob
NH 82
NC 81
MI 77
KS 57
IA 43
CO 43 pivot for 50 Democratic seats
LA 30
AL 29
AR 26

39.0 = 538 probability that the Democrats get 50 or more seats.

Post Mortem for the 2012 Election: November 7, 2012

The final ranking at 6am on November 6, 2012, repeated from below (see also Table 1), was:

state prob votes sumvotes
MN 90.6 10 217
NV 88.1 6 223
PA 79.0 20 243
WI 77.5 10 253
IA 70.0 6 259
OH 68.8 18 277 pivot
NH 68.0 4
VA 55.8 13
CO 54.1 9
FL 35.0 29
NC 20.5 15

The ranking assumption was perfect. Obama won all these states except NC. Note that Intrade was wrong about FL regarding the winner. The ranking assumption is, however, fine because it says that if Obama wins FL, he wins every state above FL, which he did.
The ranking assumption did well for the Senate races, where it was off by one. The following is the Intrade ranking as of 6am on November 6, 2012, for the Senate races that were in play. The price for each state is the price that the Democratic candidate wins:

state prob
FL 91.2
CT 88.0
MA 84.0
OH 83.2
IN 80.0
VA 78.2
MO 72.7
WI 60.0
MT 47.4
NV 20.1
ND 19.0

The Democrats won every state except NV. Given that the Republicans won NV, the ranking assumption says that the Democrats should have lost ND, and this is the one error. Otherwise, the Democrats won MT and every state above it. Note that Intrade was wrong about the winner in MT, but again this is not a problem for the ranking assumption. Both Intrade and the ranking assumption were wrong about ND.

The Ranking Assumption in 2012 using Intrade Data

Background
The paper, Interpreting the Predictive Uncertainty of Elections, Journal of Politics, April 2009, provides an interpretation of the uncertainty that exists on election morning as to who will win. The interpretation is based on the theory that there are a number of possible conditions of nature than can exist on election day, of which one is drawn. Political betting markets like Intrade provide a way of trying to estimate this uncertainty. (Polling standard errors do not provide estimates of this type of uncertainty. They estimate sample-size uncertainty, which can be driven close to zero with a large enough sample.)
This paper also introduces a "ranking assumption," which puts restrictions on the possible conditions of nature that can exist on election day. Take as an example the vote in each state for the Democratic candidate for president. Rank the states by the probability that the candidate wins the state. The ranking assumption says that if the candidate wins state i, he or she wins every state ranked above state i.
Given some ranking, the ranking assumption can be tested by simply looking to see after the fact if the candidate won a state ranked lower than one he or she lost. In the paper the assumption was tested for the 2004 and 2008 presidential elections using Intrade probabilities at 6am Eastern standard time on the day of the election. The test is thus a test of the joint hypothesis that the Intrade probabilities are right and the ranking assumption is right. Using the Intrade probabilities, the ranking assumption was perfect in 2004 and off by one in 2008. In 2008 Missouri was ranked above Indiana, and Obama won Indiana and lost Missouri. Both of these elections were very close. Obama won Indiana with 50.477 percent of the two-party vote and lost Missouri with 49.937 percent of the two-party vote. Otherwise, 2008 was perfect.
Evidence is also presented in the paper, although this is not a test of the ranking assumption, that Intrade traders use the ranking assumption to price various contracts.
2012 Election
I plan to collect Intrade probabilities at 6am Eastern standard time on six days: September 11, 25, October 9, 23, 30, and November 6. The November 6 data will be used to test the ranking assumption. The data as they are collected are presented in Table 1.
September 11, 2012
All but 11 states on Intrade either have probabilities close to 0 or 1 or have essentially no trading. The 11 states, ranked by probabilities for the Democratic candidate, are:

state prob votes sumvotes
MN 82.5 10 217
PA 80.1 20 237
NH 71.0 4 241
WI 68.0 10 251
NV 67.9 6 257
CO 64.8 9 266
OH 61.9 18 284 pivot
IA 59.7 6
VA 58.9 13
FL 47.0 29
NC 28.3 15

61.0 = Intrade probability that the Democratic candidate wins the presidential election.
"sumvotes" is the sum of the electoral votes of all the states ranked above the state plus the state's vote. 270 votes are needed to win. You can see that Ohio is the pivot state. If Obama takes Ohio and all the states ranked above it, he gets 284 votes. Of the states ranked below Ohio, he could also win by not taking Ohio and taking any one of the others.
There is evidence that the Intrade traders are using the ranking assumption. According to the ranking assumption, the probability that Obama wins overall is the probability that he wins the pivot state, Ohio, which is 61.9. The price of the contract on Intrade that the Democratic candidate wins the presidential election (in the electoral college) is 61.0, close to 61.9. If the ranking assumption were not being used, the overall probablilty would be much higher if, say, the probabilities were thought to be independent. The ranking assumption says that Obama is not going to take, say, Iowa or Virginia unless he also takes Ohio, so the probabilities for Iowa and Virginia do not matter. The probability he wins overall is just the probability he wins Ohio. If the probabilities were independent, the probability of an overall win would be 82.3 percent (computed numerically).
September 25, 2012
The results are:

state prob votes sumvotes
MN 92.2 10 217
PA 87.0 20 237
NV 79.8 6 243
WI 79.0 10 253
NH 75.6 4 257
OH 72.6 18 275 pivot
IA 65.1 6
VA 64.3 13
CO 61.1 9
FL 55.0 29
NC 42.6 15

72.4 = Intrade probability that the Democratic candidate wins the presidential election.
Ohio is still pivot, and it seems clear that the Intrade traders are using the ranking assumption. If the probabilities were thought to be independent, the overall probability would be 93.1 percent (computed numerically).
October 9, 2012
The results are:

state prob votes sumvotes
MN 90.0 10 217
PA 87.4 20 237
NV 76.7 6 243
NH 74.8 4 247
WI 70.0 10 257
IA 67.7 6 263
OH 66.6 18 281 pivot
VA 63.0 13
CO 61.0 9
FL 49.9 29
NC 30.0 15

62.7 = Intrade probability that the Democratic candidate wins the presidential election.
Ohio is still pivot. If the ranking assumption were being used by the Intrade traders, the probability of winning overall should be 66.6, the probability for Ohio. It is in fact 62.7, so a discrepancy of 3.9. In the two earlier samples above the discrepancies were 0.9 and 0.2. If the probabilities were thought to be independent, the overall probability would be 89.0 percent (computed numerically). It is thus surprising that the overall probability of 62.7 is smaller than both the probability for Ohio and the overall probability if the state probabilities were independent. Could be a thin market problem or market manipulation of the overall probability contract.
October 23, 2012
The results are:

state prob votes sumvotes
NV 85.0 6 213
MN 81.0 10 223
PA 75.2 20 243
WI 71.0 10 253
IA 57.8 6 259
OH 57.0 18 277 pivot
NH 53.5 4
VA 48.0 13
CO 45.5 9
FL 30.0 29
NC 18.4 15

61.7 = Intrade probability that the Democratic candidate wins the presidential election.
Ohio is still pivot. If the ranking assumption were being used by the Intrade traders, the probability of winning overall should be 57.0, the probability for Ohio. It is in fact 61.7, so a discrepancy of -4.7. This compares to a positive 3.9 on October 9, 2012. If the probabilities were thought to be independent, the overall probability would be 68.1 percent (computed numerically). So in this case the overall probability is in between what the ranking assumption would imply and what the independence assumption would imply.
October 30, 2012
The results are:

state prob votes sumvotes
MN 92.0 10 217
PA 82.0 20 237
NV 80.0 6 243
WI 71.2 10 253
IA 61.6 6 259
OH 58.8 18 277 pivot
NH 58.3 4
CO 47.8 9
VA 45.1 13
FL 30.1 29
NC 22.0 15

62.0 = Intrade probability that the Democratic candidate wins the presidential election.
Ohio is still pivot. If the ranking assumption were being used by the Intrade traders, the probability of winning overall should be 58.8, the probability for Ohio. It is in fact 62.0, so a discrepancy of -3.2. If the probabilities were thought to be independent, the overall probability would be 73.9 percent (computed numerically).
November 6, 2012
The results are:

state prob votes sumvotes
MN 90.6 10 217
NV 88.1 6 223
PA 79.0 20 243
WI 77.5 10 253
IA 70.0 6 259
OH 68.8 18 277 pivot
NH 68.0 4
VA 55.8 13
CO 54.1 9
FL 35.0 29
NC 20.5 15

67.9 = Intrade probability that the Democratic candidate wins the presidential election.
Ohio is still pivot. If the ranking assumption were being used by the Intrade traders, the probability of winning overall should be 68.8, the probability for Ohio. It is in fact 67.9, so a discrepancy of 0.9. If the probabilities were thought to be independent, the overall probability would be 82.7 percent (computed numerically).
This ranking will be used to test the ranking assumption. The probabilities are at 6am on the day of the election. The ranking assumption (combined with the Intrade probabilities) says, for example, that if Obama wins Colorado, Romney will not win Virginia, New Hampshire, or Ohio. If, on the other hand, Romney wins New Hampshire, the ranking assumption says that he will also win Virginia and Colorado.
Table 1 gives more details.

state	prob
MI	100.0
NC	85.0
NH	74.8
KS	48.0	as of October 28, 2014
IA	38.4
CO	23.1	pivot for 50 Democratic seats
AL	23.0
GA	6.9
LA	4.5
AR	0.2

state	prob
MI	99.8
NH	82.9
NC	70.4
KS	39.0
GA	35.3
IA	28.7	pivot for 50 Democratic seats
CO	24.7
AL	11.8
LA	8.6
AR	5.9

state	prob	votes	sumvotes
MN	90.6	10	217
NV	88.1	6	223
PA	79.0	20	243
WI	77.5	10	253
IA	70.0	6	259
OH	68.8	18	277	pivot
NH	68.0	4
VA	55.8	13
CO	54.1	9
FL	35.0	29
NC	20.5	15

state	prob
FL	91.2
CT	88.0
MA	84.0
OH	83.2
IN	80.0
VA	78.2
MO	72.7
WI	60.0
MT	47.4
NV	20.1
ND	19.0