Data On Coaches' Challenges

To calibrate our model for coaches' challenges, we collected the data displayed below from 71 games that we watched either in whole or in part, from October 23, 2005 through February 5, 2006, for a total of 3,683 minutes of playing time—the equivalent of 61.4 regulation games. We recorded every challengeable ruling that had a realistic possibility of being reversed if challenged by a coach or Replay Assistant. We are making the data set available to anyone who wants to use it, provided that published results using the data include a link to the source.

The first column in the data set is the Type of ruling. (A key is given below.) The second column is our subjective probability that the ruling would be reversed if challenged. The third column equals 1 if the ruling was challenged, and 0 otherwise. The fourth column equals 1 if the ruling was reversed, and 0 otherwise.

Types of rulings:

1=Complete pass ruled on the field, but the pass might have been incomplete.
2=Fumble ruled on the field, but the ballcarrier might have been down by contact.
3=Defensive pass interference ruled on the field, but maybe the pass was tipped.
4=Incomplete pass ruled on the field, but the pass might have been complete.
5=QB fumble ruled on the field, but his arm might have been going forward for an incomplete pass.
6=Ballcarrier ruled not to have crossed the goal line, but maybe he did.
7=Receiver didn't get both feet in bounds according to the ruling on the field, but maybe he did.
8=Interception ruled on the field, but the pass might have been incomplete.
9=Ballcarrier runs for extra yardage, but maybe he was already down by contact.
10=Complete pass ruled on the field, but maybe the receiver didn't get both feet in bounds.
11=Ballcarrier scores according to the ruling on the field, but maybe he didn't cross the plane.
12=Spot of ball could be wrong.
13=Ballcarrier didn't step on the sideline according to the ruling on the field, but maybe he did.
14=Touchback ruled on the field, but maybe the kicking team downed the ball before it reached the end zone.
15=Defender got control of the fumble before it went O.B., according to the ruling on the field, but maybe he didn't.
16=Forward lateral ruled on the field, but maybe the pass was actually backward.
17=QB arm going forward for incomplete pass, according to the ruling on the field, but maybe it was a fumble.
18=Defender fails to intercept because the ball hit the ground, according to the ruling on the field, but maybe he caught it.
19=Interceptor didn't get both feet inbounds, according to the ruling on the field, but maybe he did.
20=Fumble O.B., according to the ruling on the field, but the ball might have gone through the endzone for a touchback.
21=TD ruled on the field, but maybe before scoring the ballcarrier fumbled through endzone for a touchback.
22=Punt team downs kick near goal line, according to the ruling on the field, but maybe it's a touchback.
23=Complete pass and then fumble ruled on the field, but maybe it was incomplete.
24=Interceptor runs for extra yardage, but maybe he was already down by contact.

To illustrate the interpretation of the data below, consider the sixth row: with 2:47 remaining in the 1st quarter of the game between the Jets and Atlanta on 10/24, the ruling on the field was fumble, but the ballcarrier might have been down by contact. Our subjective probability of reversal was 0.50. The ruling was challenged, and upheld.

 2 .05 0 0  % den@nyg 10/23 13:46 in 1st
 3 .15 1 0  % den@nyg 10/23 4:42 in 2nd
 4 .30 1 0  % den@nyg 10/23 3:22 in 2nd
 5 .80 1 1  % den@nyg 10/23 13:22 in 4th
 1 .10 0 0  % sd@phil 10/23 10:33 in 3rd
 2 .50 1 0  % nyj@atl 10/24 2:47 in 1st
 1 .95 1 1  % nyj@atl 10/24 0:55 in 2nd
 6 .10 0 0  % nyj@atl 10/24 7:20 in 3rd
 5 .40 1 1  % nyj@atl 10/24 0:54 in 3rd
 4 .30 1 0  % wash@nyg 10/30 11:31 in 4th
 1 .70 1 1  % phil@den 10/30 10:02 in 1st
 7 .80 1 1  % phil@den 10/30 0:38 in 1st
 1 .50 1 0  % phil@den 10/30 14:21 in 4th
 1 .05 1 0  % buf@ne 10/30 2:00 in 2nd
 8 .05 0 0  % buf@ne 10/30 6:31 in 3rd
 5 .05 0 0  % buf@ne 10/30 12:55 in 4th
 1 .80 1 0  % buf@ne 10/30 9:45 in 4th
 1 .05 0 0  % buf@ne 10/30 6:06 in 4th
 1 .20 0 0  % balt@pitt 10/31 14:00 in 1st
24 .95 1 1  % balt@pitt 10/31 13:42 in 3rd
10 .05 0 0  % sd@jets 11/6 6:14 in 4th
 1 .20 0 0  % pitt@gb 11/6 5:45 in 2nd
11 .10 1 0  % phil@wash 11/6 4:17 in 2nd
 1 .15 1 0  % phil@wash 11/6 1:14 in 2nd
 4 .30 1 0  % phil@wash 11/6 7:16 in 3rd
12 .20 0 0  % indi@ne 11/7 4:00 in 1st
10 .20 0 0  % indi@ne 11/7 0:08 in 2nd
 1 .15 0 0  % indi@ne 11/7 10:15 in 4th
 1 .10 1 0  % indi@ne 11/7 5:53 in 4th
 5 .40 1 1  % mn@nyg 11/13 2:57 in 3rd
13 .50 1 0  % wash@tb 11/13 7:36 in 2nd
 2 .20 1 0  % wash@tb 11/13 14:44 in 3rd
 1 .20 1 0  % wash@tb 11/13 1:31 in 4th
11 .05 1 0  % wash@tb 11/13 0:58 in 4th
10 .30 1 1  % cleve@pitt 11/13 7:22 in 1st
23 .10 1 0  % cleve@pitt 11/13 0:41 in 3rd
 7 .30 1 0  % phil@nyg 11/20 6:20 in 2nd
 8 .20 1 0  % pitt@balt 11/20 1:07 in 4th
 2 .30 0 0  % nyj@den 11/20 4:11 in 3rd
11 .10 0 0  % kc@hou 11/20 8:38 in 4th
 8 .20 1 0  % mn@gb 11/21 1:04 in 2nd
11 .60 1 1  % atl@det 11/24 1:10 in 1st
 9 .90 1 1  % den@dal 11/24 12:32 in 2nd
14 .15 1 1  % den@dal 11/24 10:28 in 2nd
12 .10 1 0  % den@dal 11/24 5:40 in 2nd
15 .50 0 0  % den@dal 11/24 14:54 in 4th
12 .75 1 1  % den@dal 11/24 9:49 in 4th
 8 .15 0 0  % ne@kc 11/27 5:00 in 1st
 8 .05 0 0  % ne@kc 11/27 2:00 in 4th
 1 .30 1 0  % nyg@sea 11/27 1:07 in 2nd
12 .20 1 0  % nyg@sea 11/27 4:44 in 4th
10 .20 1 0  % nyg@sea 11/27 1:59 in 4th
23 .30 1 1  % nyg@sea 11/27 11:28 in OT
 1 .10 1 0  % nyg@sea 11/27 9:35 in OT
 1 .60 1 1  % no@nyj 11/17 1:37 in 2nd
 1 .40 0 0  % nyj@ne 12/4 1:30 in 3rd
12 .10 1 0  % nyj@ne 12/4 10:51 in 4th
 7 .80 1 1  % oak@sd 12/4 11:15 in 2nd
12 .20 0 0  % sea@phil 12/5 3:40 in 2nd
 5 .15 1 0  % sea@phil 12/5 3:44 in 3rd
 1 .25 0 0  % sea@phil 12/5 1:28 in 3rd
 1 .05 1 0  % ok@nyj 12/11 14:54 in 4th
10 .05 1 0  % kc@dal 12/11 1:48 in 2nd
10 .20 1 0  % det@gb 12/11 0:43 in 1st
 2 .10 1 0  % no@atl 12/12 11:19 in 1st
 6 .90 1 1  % no@atl 12/12 6:06 in 2nd
 2 .80 1 1  % no@atl 12/12 4:12 in 3rd
 1 .20 0 0  % tb@ne 12/17 3:55 in 1st
 5 .10 1 0  % kc@nyg 12/17 10:53 in 1st
12 .10 0 0  % kc@nyg 12/17 5:00 in 2nd
 9 .30 1 0  % kc@nyg 12/17 12:52 in 4th
12 .20 1 0  % den@buf 12/17 10:10 in 1st
 1 .50 1 1  % den@buf 12/17 5:03 in 3rd
 1 .50 0 0  % car@no 12/18 11:30 in 1st
16 .10 1 0  % car@no 12/18 11:52 in 2nd
 7 .10 0 0  % dal@wash 12/18 13:28 in 1st
 2 .10 1 0  % dal@wash 12/18 7:55 in 4th
 4 .30 1 0  % atl@chi 12/18 1:59 in 2nd
24 .05 1 1  % atl@chi 12/18 12:10 in 4th
11 .40 0 0  % gb@balt 12/19 13:25 in 4th
 1 .60 1 0  % nyg@wash 12/24 0:10 in 2nd
 6 .05 1 0  % chi@gb 12/25 12:19 in 2nd
 4 .10 1 0  % chi@gb 12/25 11:16 in 2nd
 4 .10 1 0  % chi@gb 12/25 2:06 in 2nd
 8 .05 0 0  % chi@gb 12/25 0:29 in 2nd
 2 .90 1 1  % ne@nyj 12/26 5:12 in 1st
17 .95 1 1  % den@sd 12/31 4:57 in 4th
 2 .90 1 1  % nyg@oak 12/31 7:50 in 1st
 4 .30 1 1  % nyg@oak 12/31 14:16 in 2nd
10 .05 1 0  % nyg@oak 12/31 0:19 in 2nd
 6 .50 1 0  % nyg@oak 12/31 2:31 in 4th
 2 .70 1 1  % buf@nyj 1/1 13:23 in 1st
13 .10 1 0  % wash@phil 1/1 12:26 in 3rd
12 .20 1 0  % stl@dal 1/1 0:14 in 2nd
 4 .05 1 0  % stl@dal 1/1 6:13 in 4th
 9 .20 1 0  % wash@tb 1/7 4:15 in 1st
18 .05 0 0  % jax@ne 1/7 8:09 in 1st
 1 .40 1 1  % pitt@cin 1/8 13:37 in 3rd
 4 .10 0 0  % pitt@cin 1/8 5:17 in 3rd
19 .85 1 1  % ne@den 1/14 6:10 in 2nd
20 .10 1 0  % ne@den 1/14 0:47 in 3rd
 8 .10 1 1  % pitt@indi 1/15 5:33 in 4th
 2 .20 1 1  % car@chi 1/15 4:35 in 1st
21 .95 1 1  % car@chi 1/15 12:27 in 4th
 2 .20 1 1  % pitt@den 1/22 6:38 in 1st
17 .10 0 0  % pitt@den 1/22 9:18 in 2nd
22 .80 1 1  % pitt@den 1/22 6:00 in 3rd
 4 .30 0 0  % sea@pitt 2/5 12:23 in 2nd
11 .05 1 0  % sea@pitt 2/5 1:55 in 2nd
 2 .95 1 1  % sea@pitt 2/5 8:26 in 4th