nonparametric estimation in the dynamic bradley-terry model

Nonparametric Estimation in the Dynamic Bradley-Terry Model

Heejong Bong* Wanshan Li* Shamindra Shrotriya* Alessandro RinaldoDepartment of Statistics and Data Science

Carnegie Mellon University

Abstract

We propose a time-varying generalization ofthe Bradley-Terry model that allows for non-parametric modeling of dynamic global rank-ings of distinct teams. We develop a novel es-timator that relies on kernel smoothing to pre-process the pairwise comparisons over timeand is applicable in sparse settings where theBradley-Terry may not be fit. We obtainnecessary and su�cient conditions for theexistence and uniqueness of our estimator.We also derive time-varying oracle boundsfor both the estimation error and the excessrisk in the model-agnostic setting where theBradley-Terry model is not necessarily thetrue data generating process. We thoroughlytest the practical e↵ectiveness of our modelusing both simulated and real world data andsuggest an e�cient data-driven approach forbandwidth tuning.

1 Introduction and Prior Work

1.1 Pairwise Comparison Data and theBradley-Terry Model

Pairwise comparison data are very common in dailylife, especially in cases where the goal is to rank sev-eral objects. Rather than directly ranking all objectssimultaneously, it is usually much easier and more ef-ficient to first obtain results of pairwise comparisonsand then use them to derive a global ranking across allindividuals in a principled manner. Since the requiredglobal rankings are not directly observable, developinga statistical framework for the estimating rankings is achallenging unsupervised learning problem. One such

*Equal contribution.

Proceedings of the 23

rdInternational Conference on Artificial

Intelligence and Statistics (AISTATS) 2020, Palermo, Italy.

PMLR: Volume 108. Copyright 2020 by the author(s).

statistical model for deriving global rankings using pair-wise comparisons was presented in the classic paper(Bradley and Terry, 1952), and thereafter commonlyreferred to as the Bradley-Terry model in the literature.A similar model was also studied by Zermelo (Zermelo,1929). The Bradley-Terry model is one of the mostpopular models to analyze paired comparison data dueto its interpretability and computational e�ciency inparameter estimation. The Bradley-Terry model alongwith its variants has been studied and applied in vari-ous ranking applications across many domains. Thisincludes the ranking of sports teams (Masarotto andVarin, 2012; Cattelan et al., 2013; Fahrmeir and Tutz,1994), scientific journals (Stigler, 1994; Varin et al.,2016), and the quality of several brands (Agresti, 2013;Radlinski and Joachims, 2007), to name a few.

In order to introduce the Bradley-Terry model, supposethat we have N distinct teams, each with a positivescore s

i

, i 2 [N ] := {1, . . . , N}, quantifying its propen-sity to be picked or win over other items. The modelpostulates that the comparisons between di↵erent pairsare independent and the results of the comparisonsbetween a given pair, say team i and team j, are inde-pendent and identically distributed Bernoulli randomvariables, with winning probability defined as

P(i beats j) =si

si

+ sj

, 8 i, j 2 [N ] (1)

A common way to parametrize the model is to set, foreach i, s

i

= exp(�i

), where (�1, . . . , �N

) are real pa-rameters such that

P

i2[N ] �i

= 0 (this latter conditionis to ensure identifiability). In this case, equation (1)is usually expressed as logit(P(i beats j)) = �

i

� �j

,where, for x 2 (0, 1), logit(x) := log x

1�x

.

1.2 The time-varying (dynamic)Bradley-Terry Model

In many applications it is very common to observepaired comparison data spanning over multiple (dis-crete) time periods. A natural question of interest isthen to understand how the global rankings vary overtime i.e. dynamically. For example, in sports ana-


lytics the performance of teams often changes acrossmatch rounds and thus explicitly incorporating thetime-varying dependence into the model is crucial. Inparticular the paper (Fahrmeir and Tutz, 1994) consid-ers a state-space generalization of the Bradley-Terrymodel to analyze sports tournaments data. In a similarmanner, bayesian frameworks for the dynamic Bradley-Terry model are studied further in (Glickman, 1993;Glickman and Stern, 1998; Lopez et al., 2018). Suchdynamic ranking analysis is becoming increasingly im-portant because of the rapid growth of openly availabletime-dependent paired comparison data.

Our main focus in this paper is to tackle the prob-lem of generalizing the Bradley-Terry model to thetime-varying setting with statistical guarantees. Ourapproach estimates the changes in the Bradley-Terrymodel parameters over time nonparametrically. Thisenables the derivation of time-varying global dynamicrankings with minimal parametric assumptions. Un-like previous time-varying Bradley-Terry estimationapproaches, we seek to establish guarantees in themodel-agnostic setting where the Bradley-Terry modelis not the true data generating process. This is incontrast to more assumption-heavy parametric frequen-tist dynamic Bradley-Terry models including (Cattelanet al., 2013). Our method is also computationally e�-cient compared to some state-space methods includingbut not limited to (Fahrmeir and Tutz, 1994; Glickmanand Stern, 1998; Maystre et al., 2019).

2 Time-varying Bradley-Terry Model

2.1 Model Setup

In our time-varying generalization of the originalBradley-Terry model we assume that N distinct teamsplay against each other at possibly di↵erent times, overa given time interval which, without loss of generality, istaken to be [0, 1]. The result of a game between team iand team j at time t is determined by the timestampedwinning probability p

ij

(t) := P(i defeats j at time t)which we assume arises from a distinct Bradley-Terrymodel, one for each time point. In detail, for each i, j,and t

logit(pij

(t)) = �i

(t) � �j

(t), 8 i, j 2 [N ], t 2 [0, 1] (2)

where �(t) = vec(�1(t), �2(t), . . . , �N

(t)) 2 RN is anunknown vector such that

P

i

�i

(t) = 0.

We observe the outcomes of M timestamped pairwisematches among the N teams {(i

m

, jm

, tm

) : m 2 [M ]}.Here (i

m

, jm

, tm

) indicates that team im

and team jm

played at time tm

, where t1 t2 . . . tM

. Theresult of the m-th match can be expressed in a N ⇥ N

data matrix X(m) as follows:

X(m)ij

=

8

>

>

>

<

>

>

>

:

1(i defeats j at tm

)

⇠ Bernoulli(pij

(tm

))for i = i

m

and j = jm

1 � X(m)ji

for i = jm

and j = im

0 elsewhere(3)

Our goal is to estimate the underlying parameters �(t)where t 2 [0, 1], and then derive the correspondingglobal ranking of teams. In order to make the estima-tion problem tractable we assume that the parameters�(t) vary smoothly as a function of time t 2 [0, 1].It is worth noting that the naive strategy of estimat-ing the model parameters separately at each observeddiscrete time point on the original data will su↵erfrom two major drawbacks: (i) it will in general notguarantee smoothly varying estimates and, perhapsmore importantly, (ii) computing the maximum likeli-hood estimator (MLE) of the parameters in the staticBradley-Terry model may be infeasible due to sparsityin the data (e.g., at each time point we may observeonly one match), as we discuss below in 4. To overcomethese issues we propose a nonparametric methodologywhich involves kernel-smoothing the observed data overtime.

2.2 Estimation

Our approach in estimating time-varying global rank-ings is described in the following three step procedure:

1. Data pre-processing: Kernel smooth the pairwisecomparison data across all time periods. This isused to obtain the smoothed pairwise data at eachtime t:

X(t) =M

X

m=1

Wh

(tm

, t)X(m), (4)

where Wh

is an appropriate kernel function withbandwidth h, which controls the extent of datasmoothing. The higher the value of h is, thesmoother X

ij

(t) is over time.

2. Model fitting: Fit the regular Bradley-Terry modelon the smoothed data X

ij

(t). The model estimatesthe performance of each team at time t using theestimated score vector

b�(t) = arg min�:P

i

�

i

(t)=0bR(�; t) (5)

minimizing the negative log-likelihood risk

bR(�; t) =X

i,j

:i 6=j

Xij

(t) log(1 + exp(�j

(t) � �i

(t)))P

i,j

:i 6=j

Xij

(t)

(6)

Heejong Bong*, Wanshan Li*, Shamindra Shrotriya*, Alessandro Rinaldo

3. Derive global rankings : Rank each team at time tby its score from �(t).

We observe that if t = t1 = t2 = · · · = tM

then step1 reduces to the original (static) Bradley-Terry model.In this case,

Xij

(t) = Wh

(t, t)M

X

m=1

1(im

= i, jm

= j)

= Wh

(t, t)Xij

(7)

where Xij

= #{i defeated j}. Thus, fitting the modelon X(t) in step 2 is equivalent to the original method ondata X. In this sense our proposed ranking approachrepresents a time-varying generalization of the originalBradley-Terry model.

This data pre-processing is a critical step in our methodand is similar to the approach adopted in (Zhou et al.,2010) where it was used in the context of estimatingsmoothed time varying undirected graphs. This ap-proach has two main advantages. First, applying kernelsmoothing on the input pairwise comparison data en-ables borrowing of information across timepoints. Insparse settings, this reduces the data requirement ateach time point to meet necessary and su�cient con-ditions required for the Bradley-Terry model to havea unique solution as detailed in Section 4. Second,kernel smoothing is computationally e�cient in a sin-gle dimension and is readily available in open sourcescientific libraries.

3 Our Contributions

Our main contributions in this paper are summarizedas follows:

1. We obtain necessary and su�cient conditions forthe existence and uniqueness for our time-varyingestimator b�(t) for each t 2 [0, 1] simulataneously.See Section 4.

2. We extend the results of Simons and Yao (1999)and obtain statistical guarantees for our proposedmethod in the form of convergence results of theestimated model parameters uniformly over alltimes. We express such guarantees in the form oforacle inequalities in the model-agnostic settingwhere the Bradley-Terry model is not necessarilythe true data generating process. See Section 5.

3. We apply our estimator with an data-driven tuned(by LOOCV) hyperparameter successfully to simu-lations and to real life applications including acomparison to 5 seasons of NFL ELO ratings.See Section 6 and Section 7.

4 Existence and uniqueness ofsolution

The existence and uniqueness of solutions for model (5)is not guaranteed in general. This is an innate prop-erty of the original Bradley-Terry model (Bradley andTerry, 1952). As pointed out in (Ford, 1957) existenceof the MLE for the Bradley-Terry model parametersdemands a su�cient amount of pairwise comparisondata so that there is enough information of relativeperformance between any pair of two teams for param-eter estimation purposes. For example, if there is ateam which has never been compared to the others,there is no information which the model can exploitto assign a score for the team. As such its derivedrank could be arbitrary. In addition if there are severalteams which have never outperformed the others thenthe Bradley-Terry model would assign negative infinityfor the performance of these teams. It would not bepossible to compare amongst them for global rankingpurposes. In all such cases, the model parameters arenot estimable.

Ford (1957) derived the necessary and su�cient con-dition for the existence and uniqueness of the MLEin the original Bradley-Terry model. Below we showhow this condition can also be adapted to guaranteethe existence and uniqueness of the solution in ourtime-varying Bradley-Terry model. The condition canbe stated for each time t in terms of the correspondingkernel-smoothed data X(t) as follows:

Condition 4.1. In every possible partition of theteams into two nonempty subsets, some team i in thefirst set and some team j in the second set satisfyX

ij

(t) > 0. Or equivalently, for each ordered pair (i, j),there exists a sequence of indices i0 = i, i1, . . . , in = jsuch that X

i

k�1ik(t) > 0 for k = 1, . . . , n.

Remark 1. If we regard [|X(t)|ij

] as the adjacency ma-trix of a (weighted) directed graph, then Condition 4.1is equivalent to the strong connectivity of the graph.

Under condition 4.1 we obtain the following existenceand uniqueness theorem on the solution set of the time-varying Bradley-Terry model.

Theorem 4.1. If the smoothed data X(t) satisfiesCondition 4.1, then the solution of (5) uniquely existsat time t.

Hence, in the proposed time-varying Bradley-Terrymodel we do not require the strong conditions of (Ford,1957) to be met at each time point, but simply requirethe aggregated conditions in Theorem 4.1 to hold. Thisis a significant weakening of the data requirement. Forexample, even if one team did not play any game in amatch round – a situation that would preclude the MLEin the standard Bradley-Terry model – it is still possible


to assign a rank to this team in their missing round,as long a game is recorded in another round (with atleast one win and one loss). In this sense, the kernel-smoothing of the data in our time-varying Bradley-Terry model reduces the required sample complexityfor a unique solution.

5 Statistical Properties of theTime-varying Bradley-Terry Model

5.1 Preliminaries

Existing results (Simons and Yao, 1999; Negahbanet al., 2017) demonstrate the consistency of the esti-mated static Bradley-Terry scores provided that thedata were generated from the Bradley-Terry model.However this assumption may be too restrictive fordata generation processes in real world applications. Inthe rest of this section, we will consider model-agnostictime-varying settings where the Bradley-Terry model isnot necessarily the true pairwise data generating model.In order to investigate the statistical properties of theproposed estimator, we impose the following relativelymild assumptions.

Assumption 5.1. Each pair of teams (i, j) play T (i,j)

times at time points {t(i,j)k

, k = 1, 2, . . . , T (i,j)}, whereeach T (i,j) > 0 satisfy the following conditions, for fixedconstants T > 0 and 0 < D

m

1 DM

:

1. T (i,j) > T ;

2. for every interval (a, b) ⇢ [0, 1],

bDm

(b � a)T (i,j)c |{k : t(i,j)k

2 (a, b)}|dD

M

(b � a)T (i,j)e.(8)

We remark that the second condition further impliesthat

t(i,j)1 1

Dm

T (i,j), t(i,j)

T

(i,j) � 1 � 1

Dm

T (i,j),

1

DM

T (i,j) t(i,j)

k+1 � t(i,j)k

1

Dm

T (i,j)

(9)

for k = 1, 2, . . . , T (i,j) � 1.

Assumption 5.1 allows for di↵erent team pairs to playagainst each other a di↵erent number of times andfor the game times to be spaced irregularly, thoughin a controlled manner. To enable statistical analyseson time-varying quantities, we further require thatthe winning probabilities satisfy a minimal degree ofsmoothness and that their rate of decay is controlled.

Assumption 5.2. For any i, j, the function t 2[0, 1] 7! p

ij

(t) is Lipschitz with universal constant Lp

and uniformly bounded below pmin > 0 which is depen-dent to N and T .

The quantity pmin need not be bounded away from 0 asfunction of T and N . However, in order to guaranteeestimability of the model parameters in time-varyingsettings, we will need to control the rate at which it isallowed to vanish. See Theorem 5.1 below.

Finally, we assume that the kernel used to smooth overtime satisfy the following regularity conditions, whichare quite standard in the nonparametric literature.

Assumption 5.3. The kernel function W :(�1, 1) ! (0, 1) is a symmetric function such that

Z 1

�1W (x)dx = 1,

Z 1

�1|x|W (x)dx < 1

V(W ) < 1, V(| · |W ) < 1(10)

where V(f(x)) is the total variation of a function f .For each s, t 2 [0, 1] we further write

Wh

(s, t) =1

hW

✓

s � t

h

◆

. (11)

It is easy to see that these conditions imply

kWk1 = supx

W (x) < 1. (12)

Thus, without loss of generality, we assume thatkWk1 1; the general case can be handled by modi-fying the constants accordingly. The use of kernels sat-isfying the above assumptions is standard in nonpara-metric problems involving Holder continuous functionsof order 1, such as the winning probabilities functionof Assumption 5.2.

5.2 Existence and uniqueness of solutions

Simons and Yao (1999) showed that the necessary andsu�cient condition for the existence and uniqueness ofthe MLE in the original Bradley-Terry model is satisfiedasymptotically almost surely under minimal assump-tions. Below, we show that this type of result can beextended to our more general time-varying settings.

Theorem 5.1.

P(Condition 4.1 is satisfied at every t)

� 1 � 4N exp

✓

�NT

2pmin

◆

.(13)

Remark 2. As we remarked above, pmin needs not bebounded away from zero, but is allowed to vanish slowlyenough in relation to N and T so that condition (5.1)

is fulfilled as long as 1pmin

= o⇣

NT

2 logN

⌘

.

5.3 Oracle Properties

In our general agnostic time-varying setting theBradley-Terry model is not assumed to be the true


data generating process. It follows that, for each t,there is no true parameter to which to compare theestimator defined in (5). Instead, we may compareit to the projection parameter �⇤(t) 2 RN , which isthe best approximation to the winning probabilities attime t using the dynamic Bradley Terry model; see (2).In detail, the oracle parameter is defined as

�⇤(t) = arg min�:P

i

�

i

(t)=0R(�; t) (14)

where

R(�; t) =1�

N

2

�

X

i,j

:i 6=j

pij

(t) log(1 + exp(�j

(t) � �i

(t)))

(15)We note that, when the winning probabilities obey aBradley-Terry model, the projection parameter corre-sponds to the true model parameters.

Next, for each fixed time t 2 (0, 1) and h > 0, we intro-duce two quantities, namely M(t) and �

h

(t), that canbe thought of as conditions numbers of sort, a↵ectingdirectly both the estimation and prediction accuracyof the proposed estimator. In detail, we set

M(t) = maxi,j

:i 6=j

exp(�⇤i

(t) � �⇤j

(t))

and

�h

(t) = maxi

X

j

:j 6=i

�

�

�

�

�

Tij

(t)

Ti

(t)� 1

N � 1

�

�

�

�

�

.

where Tij

(t) = Xij

(t) + Xji

(t) and Ti

(t) =P

j

:j 6=i

Tij

(t). The ratio M(t) quantifies the maximaldiscrepancy in winning scores among all possible pairsat time t, and, as shown in Simons and Yao (1999),determines the consistency rate of the MLE in the tra-ditional Bradely-Terry model (see also the commentsfollowing Remark 3 below). The quantity �

h

(t) is in-stead a measure of regularity in how evenly the teamsplay against each other. In particular �

h

(t) = 0, for allt and h when there is a constant number of matchesamong each pair of teams, for each time. Since weallow for the possibility of di↵erent number of matchesbetween teams and across time, it becomes necessaryto quantity such degree of design irregularity.

In order to verify the quality of the proposed estimatorb�(t), we will consider high-probability oracle bound

on estimation error kb�(t) � �⇤(t)k1. In the followingtheorems, we present both a point-wise and a uniformin t version of this bound in the asymptotic regime ofT, N ! 1 and under only minimal assumptions onthe ground-truth winning probabilities p

ij

(t)’s.

Theorem 5.2. Let � = �(T, N, pmin) be the probabilitythat Condition 4.1 fails and suppose that the kernelbandwidth is chosen as

h = max

(

1

T 1+⌘

,

✓

36(1 � pmin) log N

C2s

Dm

(N � 1)T

◆

13

)

,

for any ⌘ > 0 and some universal constant Cs

depend-ing only to D

m

, DM

, and W . Then, for each fixedtime t 2 (0, 1) and su�ciently large N and T ,

k�(t) � �⇤(t)k1 48M(t)�

�h

(t) + Cs

h�

(16)

with probability at least 1 � 2N

� � as long as the righthand side is smaller than 1

3 .

Next, we strengthen our previous result, which is validfor each fixed time t, to a uniform guarantee over theentire time course.

Theorem 5.3. Let � = �(T, N, pmin) be the probabilitythat Condition 4.1 fails and suppose that the kernelbandwidth is chosen as

h = max

(

1

T 1+⌘

,

✓

36(1 � pmin) log(NT 3+3⌘)

C2s

Dm

(N � 1)T

◆

13

)

and that W is LW

-Lipschitz. Then, for su�cientlylarge N and T ,

supt2[0,1]

k�(t) � �(t)⇤k1 48 supt2[0,1]

M(t)�

�h

(t) + Cs

h�

(17)

with probability at least 1 � 2h3

N

� � as long as the righthand side is smaller than 1

3 .

Remark 3. The rate of point-wise convergence for theestimation error implied by the previous result is

M(t)

0

@�h

(t) + max

8

<

:

1

T 1+⌘

,

r

log N

NT

!

23

9

=

;

1

A, (18)

while the rate for uniform convergence is

M(t)

0

@�h

(t) + max

8

<

:

1

T12+⌘

,

r

log(NT 1+⌘)

NT

!

23

9

=

;

1

A.

(19)

Importantly, as we can see in the previous results, theproposed time varying estimator �(t) is consistent onlyprovided that the design regularity parameter �

h

(t)goes to zero. Of course, if all teams play each other aconstant number of times, then �

h

(t) = 0 automatically.In general, however, the impact of the design on theestimation accuracy needs to be assessed on a case-by-case basis.

The rate (18) should be compared with the convergencerate to the true parameters under the static BradleyTerry model, which (Simons and Yao, 1999) show tobe O

p

(maxi,j

:i 6=j

exp(�⇤i

��⇤j

)p

log N/NT ). Thus, notsurprisingly, in the more challenging dynamic settings


with smoothly varying winning probabilities the estima-tion accuracy decreases. The exponent of 2

3 in the rate(18) matches the familiar rate for estimating Holdercontinuous function of order 1.

From (18) and (19) we observe that the desired oracleproperty on estimated parameters requires rate con-straints on M(t). These constraints appear to be strongassumptions without a direct connection to p

ij

(t)’s inour model-agnostic setting. Instead, we circumventthis issue by introducing a more interpretable condi-tion number K (or pmin), dependent on N, T , givenby

K = exp

✓

1

pmin

◆

and proving that for each fixed time t 2 (0, 1) ourdesired oracle property follows from a bound on M(t).

Theorem 5.4. Under the conditions in Theorem 5.2

k�(t) � �⇤(t)k1 72K�

�h

(t) + Cs

h�

(20)

and under the conditions Theorem 5.3

supt2[0,1]

k�(t) � �(t)⇤k1 72K supt2[0,1]

�

�h

(t) + Cs

h�

(21)

with probability at least 1 � 2N

� �.

Remark 4. We note that assuming pmin to be boundedaway from 0 ensures sup

t2[0,1] k�(t) � �⇤(t)k1 ! 0,with high probability. This assumption means that noteam is uniformly dominated by or dominates others(since this implies 1 � pmin(t) is bounded away from 1).This is a reasonable assumption in real-world data suchas sports match histories where teams are screened to becompetitive with each other. Therefore it is reasonableto only consider matches between teams that do nothave vastly di↵erent skills, which is reflected in winningprobabilities that are bounded away from {0, 1}.

In summary, our proposed method achieves high-probability oracle bounds on the estimation error inour general model-agnostic time-varying setting. Weprovide the proofs for the stated theorems in AppendixSection 9.1.

6 Experiments

We compare our method with some other methodson synthetic data⇤. We consider both cases wherethe pairwise comparison data are generated from theBradley-Terry model and from a di↵erent, nonparamet-ric model.

⇤Code available at https://github.com/shamindras/

bttv-aistats2020

6.1 Bradley-Terry Model as the True Model

First we conduct simulation experiments in which theBradley-Terry model is the true model. Given thenumber of teams N and the number of time points M ,the synthetic data generation process is as follows:

1. For i 2 [N ], simulate �i

2 RM as described below;

2. For 1 i < j N and t 2 [M ], set nij

(t) and

simulate X(t) by xij

(t) ⇠ Binom⇣

nij

(t), 1/{(1 +

exp[�j

(t) � �i

(t)]}⌘

and xji

(t) = nij

(t) � xij

(t).

For each i 2 [N ], we generate �i

2 RM from a Gaussianprocess GP(µ

i

(t), �i

(t, s)) as follows:

1. Set the values of the mean process µi

(t) for all t 2[M ] and get mean vector µ

i

= (µi

(1), . . . , µi

(M));

2. Set the values of the variance process �i

(t, s) at(s, t) 2 [M ]2, to derive ⌃

i

2 RM⇥M ;

3. Generate a sample �i

from Normal(µi

, ⌃i

).

In our experiment, we generate the parameter � via aGaussian process for N = 50, M = 50 and n

ij

(t) = 1for all t. See appendix for full details. We comparethe true �, the win rate, and � by di↵erent methodsin Fig. 1.

0 10 20 30 40 50t

�2

�1

0

1

2

3

��

True ��

0 10 20 30 40 50t

�2

�1

0

1

2

3

�

Dynamic Bradley-Terry, Gaussian Kernel

0 10 20 30 40 50t

0.2

0.4

0.6

0.8

1.0

Win

Rate

Win Rate

0 10 20 30 40 50t

�2

0

2

�

Original Bradley-Terry

Figure 1: Comparison of � and di↵erent estimators.First row: true � (left), � with our dynamic BT (right);second row: win rate (left), original BT (right).

We use LOOCV to select the kernel parameter h. TheCV curve can be found in Section 6 of Appendix. Inthis relatively sparse simulated data we see that ourdynamic Bradley-Terry estimator � recovers the com-prehensive global ranking of each team in line with the

https://github.com/shamindras/bttv-aistats2020

https://github.com/shamindras/bttv-aistats2020


Estimator Rank Di↵ LOO Prob LOO nllWin Rate 3.75 0.44 -

Original BT 3.75 0.37 0.56Dynamic BT 2.29 0.37 0.55

Table 1: Comparison of Di↵erent estimators with re-sults based on 20 repeats. Rank Di↵. is the averageabsolute di↵erence between the estimated and truerankings of teams. LOO Prob means average leave-one-out prediction error of win/loss. LOO nll meansaverage leave-one-out negative log-likelihood.

original Bradley-Terry model. We also observe thatdue to the kernel-smoothing that our estimator hasrelatively more stable paths over time.

Table 1 compares the three estimators across key met-rics. The results are averaged over 20 repeats. Asexpected, in this sparse data setting, our dynamicBradley-Terry method performs better than the origi-nal Bradley-Terry model.

6.2 Model-Agnostic Setting

In our second experiment we adopt a model-agnosticsetting where we assume that Bradley-Terry modelis not necessarily the true data generating model, asdescribed in Section 5.1. With the same notation, for1 i < j N and t 2 [M ] we first simulate p

ij

(t),and then set n

ij

(t) and simulate X(t) by xij

(t) ⇠Binom

⇣

nij

(t), pij

(t)⌘

and xji

(t) = nij

(t) � xij

(t). To

generate a smoothly changing pij

(t), we again use Gaus-sian process. Specifically, first we generate p

ij

(t) for1 i < j N and t 2 [M ] from a Gaussian process.Then we scale those p

ij

(t)’s uniformly to make thevalues fall within a range [p

l

, pu

]. In our experimentwe set M = 50, N = 50, [p

l

, pu

] = [0.05, 0.95] andnij

(t) = 1 for all t. The projection parameter �⇤, the

win rate, and b� by di↵erent methods are compared inFig. 2. Again the kernel parameter h in our model isselected by LOOCV.

Estimator Rank Di↵ LOO Prob LOO nllWin Rate 10.68 0.49 -

Original BT 10.70 0.49 0.71Dynamic BT 5.48 0.49 0.68

Table 2: Comparison of di↵erent estimators when theunderlying model is not the Bradley-Terry model.

By comparing curves in Fig. 2, we note that our estima-tor � recovers the global rankings better than the winrate and the original Bradley-Terry model, and pro-duces relatively more stable paths over time. The sameconclusion is confirmed by Table 2, which compares thethree estimators in some metrics with 20 repetitions.

0 10 20 30 40 50t

�0.6

�0.4

�0.2

0.0

0.2

0.4

�

�

0 10 20 30 40 50t

�0.8

�0.6

�0.4

�0.2

0.0

0.2

0.4

�

Dynamic Bradley-Terry, Gaussian Kernel

0 10 20 30 40 50t

0.2

0.3

0.4

0.5

0.6

0.7

Win

Rate

Win Rate

0 10 20 30 40 50t

�1.5

�1.0

�0.5

0.0

0.5

1.0

�

Original Bradley-Terry

Projection

Figure 2: Comparison of �⇤ and di↵erent estimatorswhen the underlying model is not the Bradley-Terrymodel. First row: projection �⇤ (left), our dynamicBT (right); second row: win rate (left), original BT(right).

Remark 5. In this sparse data setting where nij

(t) isfairly small, the original Bradley-Terry model performsworse than our model for two reasons: 1. Condition 4.1can fail to hold at some time points, whence the MLEdoes not exist; 2. even when the MLE exists, it canfluctuate significantly over time because of the relativelysmall sample size at each time point. As we show inSection 9.3.4 in the appendix, when M = 50, N =50 and n

ij

(t) = 1, the MLE exists with fairly highfrequency. Still, our model performs much better thanthe original Bradley-Terry model.

Remark 6. Since our method is aimed at obtaining accu-rate estimates of smoothly changing beta/rankings withstrong prediction power, it may fail to capture somechanges in rankings, especially when these changes arerelatively small (as in the present case). However thewinrate and original Bradley-Terry methods performmuch worse, as they appear to miss some true rankingchanges while returning many false change points.

Additional details about experiments, including run-ning time e�ciency in simulated settings, can be foundin the Appendix Section 9.3.

7 Application - NFL Data

In order to test our model in practical settings we con-sider ranking National Football League (NFL) teamsover multiple seasons. Specifically we source 5 seasonsof openly available NFL data from 2011-2015 (inclu-sive) using the nflWAR package (Yurko et al., 2018).Each NFL season is comprised of N = 32 teams play-


rank2011 2012 2013 2014 2015

ELO BT ELO BT ELO BT ELO BT ELO BT

1 GB GB NE HOU SEA SEA SEA DEN SEA CAR2 NE SF DEN ATL SF DEN NE ARI CAR DEN3 NO NO GB SF NE NO DEN NE ARI NE4 PIT NE SF CHI DEN KC GB SEA KC CIN5 BAL DET ATL GB CAR SF DAL DAL DEN ARI6 SF BAL SEA NE CIN NE PIT GB NE GB7 ATL PIT NYG DEN NO IND BAL PHI PIT MIN8 PHI HOU CIN SEA ARI CAR IND SD CIN KC9 SD CHI BAL BAL IND ARI ARI DET GB PIT10 HOU ATL HOU IND SD CIN CIN KC MIN SEA

Av. Di↵. 4.2 5.0 3.5 4.3 3.4

Table 3: BT within season vs. ELO NFL top 10 rankings. Blue: perfect match, yellow: top 10 match. Ourdynamic BT model is fitted on 16 rounds of each season, and the ranking of a season is based on the ranking atthe last round.

ing M = 16 games each over the season. This meansthat at each point in time t the pairwise comparisonmatrix based on scores across all 32 teams is sparselypopulated with only 16 entries. We fit our time-varyingBradley Terry estimator over all 16 rounds in the seasonusing a standard Gaussian Kernel and tune h using theLOOCV approach described in section 9.2. In order togauge whether the rankings produced by our model arereasonable we compare our season-ending rankings (fitover all games played in that season) with the relevantopenly available NFL ELO ratings (Paine, 2015). Thetop 10 season-ending rankings from each method acrossNFL seasons 2011-2015 are summarized in Table 3.

Based on Table 3 we observe that we roughly matchbetween 6 to 10 of the top 10 ELO teams consistentlyover all 5 seasons. There is often misalignment withspecific ranking values across both ranking methods.We note that the unlike our method, the NFL ELOrankings use pairwise match data and also additionalfeatures including an adjustment for margin of victory.This demonstrates an advantage of our model in onlyrequiring the minimal time-varying pairwise match dataand smoothness assumptions to deliver comparableresults to this more feature rich ELO ranking method.Furthermore, since our model aggregates data acrosstime it can provide a reasonable minimalist rankingbenchmark in modern sparse time-varying data settingswith limited “expert knowledge” e.g. e-sports.

8 Conclusion

We propose a time-varying generalization of theBradley-Terry model that captures temporal depen-dencies in a nonparametric fashion. This enables themodeling of dynamic global rankings of distinct teamsin settings in which the parameteres of the ordinaryBradley Terry model would not be estimable.

From a theoretical perspective we adapt the results of(Ford, 1957) to obtain the necessary and su�cient con-

dition for the existence and uniqueness of our Bradley-Terry estimator in the time-varying setting. We extendthe previous analysis of (Simons and Yao, 1999) toderive oracle inequalities on for our proposed methodfor both the estimation error and the excess risk undersmoothness conditions on the winning probabilities.The resulting rates of consistency are of nonparametrictype.

From an implementation perspective we provide a gen-eral strategy for tuning the kernel bandwidth hyperpa-rameter using an e�cient data-driven approach specificto our unsupervised time-varying setting. Finally, wetest the practical e↵ectiveness of our estimator underboth simulated and real world settings. In the lattercase we separately rank 5 consecutive seasons of openNational Football League (NFL) team data (Yurkoet al., 2018) from 2011-2015. Our NFL ranking resultscompare favourably to the well-accepted NFL ELOmodel rankings (Paine, 2015). We thus view our non-parametric time-varying Bradley-Terry estimator as auseful benchmarking tool for other feature-rich time-varying ranking models since it simply relies on theminimalist time-varying score information for model-ing.


References

Agresti, A. (2013). Categorical data analysis. WileySeries in Probability and Statistics. Wiley-Interscience[John Wiley & Sons], Hoboken, NJ, third edition.

Bradley, R. A. and Terry, M. E. (1952). Rank analysisof incomplete block designs. I. The method of pairedcomparisons. Biometrika, 39:324–345.

Cattelan, M., Varin, C., and Firth, D. (2013). Dy-namic Bradley-Terry modelling of sports tournaments.J. R. Stat. Soc. Ser. C. Appl. Stat., 62(1):135–150.

Fahrmeir, L. and Tutz, G. (1994). Dynamic stochasticmodels for time-dependent ordered paired comparisonsystems. Journal of the American Statistical Associa-tion, 89(428):1438–1449.

Ford, Jr., L. R. (1957). Solution of a ranking problemfrom binary comparisons. Amer. Math. Monthly, 64(8,part II):28–33.

Glickman, M. E. (1993). Paired comparison modelswith time-varying parameters. ProQuest LLC, AnnArbor, MI. Thesis (Ph.D.)–Harvard University.

Glickman, M. E. and Stern, H. S. (1998). A state-spacemodel for national football league scores. Journal ofthe American Statistical Association, 93(441):25–35.

Lopez, M. J., Matthews, G. J., and Baumer, B. S.(2018). How often does the best team win? A uni-fied approach to understanding randomness in NorthAmerican sport. Ann. Appl. Stat., 12(4):2483–2516.

Masarotto, G. and Varin, C. (2012). The rankinglasso and its application to sport tournaments. Ann.Appl. Stat., 6(4):1949–1970.

Maystre, L., Kristof, V., and Grossglauser, M. (2019).Pairwise comparisons with flexible time-dynamics. InProceedings of the 25th ACM SIGKDD InternationalConference on Knowledge Discovery & Data Mining,pages 1236–1246.

Negahban, S., Oh, S., and Shah, D. (2017). Rankcentrality: ranking from pairwise comparisons. Oper.Res., 65(1):266–287.

Paine, N. (2015). NFL Elo Ratings AreBack! https://fivethirtyeight.com/features/

nfl-elo-ratings-are-back/.

Radlinski, F. and Joachims, T. (2007). Active explo-ration for learning rankings from clickthrough data. InProceedings of the 13th ACM SIGKDD InternationalConference on Knowledge Discovery and Data Mining,KDD ’07, pages 570–579, New York, NY, USA. ACM.

Raghavan, P. (1988). Probabilistic construction of de-terministic algorithms: approximating packing integerprograms. Journal of Computer and System Sciences,37(2):130–143.

Simons, G. and Yao, Y.-C. (1999). Asymptotics whenthe number of parameters tends to infinity in theBradley-Terry model for paired comparisons. Ann.Statist., 27(3):1041–1060.

Stigler, S. M. (1994). Citation patterns in the journalsof statistics and probability. Statist. Sci., 9:94–108.

Varin, C., Cattelan, M., and Firth, D. (2016). Statis-tical modelling of citation exchange between statisticsjournals. J. Roy. Statist. Soc. Ser. A, 179(1):1–63.

von Luxburg, U. (2007). A tutorial on spectral clus-tering. Stat. Comput., 17(4):395–416.

Yurko, R., Ventura, S., and Horowitz, M. (2018).nflwar: A reproducible method for o↵ensiveplayer evaluation in football. arXiv preprintarXiv:1802.00998.

Zermelo, E. (1929). Die Berechnung der Turnier-Ergebnisse als ein Maximumproblem der Wahrschein-lichkeitsrechnung. Math. Z., 29(1):436–460.

Zhou, S., La↵erty, J., and Wasserman, L. (2010). Timevarying undirected graphs. Mach. Learn., 80(2-3):295–319.

https://fivethirtyeight.com/features/nfl-elo-ratings-are-back/

https://fivethirtyeight.com/features/nfl-elo-ratings-are-back/

nonparametric estimation in the dynamic bradley-terry model

Documents