29

Diagonal Orthant Multinomial Probit Models

James E. Johndrow Kristian Lum David B. DunsonDuke University Virginia Tech Duke University

Abstract

Bayesian classification commonly relies onprobit models, with data augmentation al-gorithms used for posterior computation.By imputing latent Gaussian variables, onecan often trivially adapt computational ap-proaches used in Gaussian models. How-ever, MCMC for multinomial probit (MNP)models can be inefficient in practice dueto high posterior dependence between latentvariables and parameters, and to difficultiesin efficiently sampling latent variables whenthere are more than two categories. To ad-dress these problems, we propose a new classof diagonal orthant (DO) multinomial mod-els. The key characteristics of these modelsinclude conditional independence of the la-tent variables given model parameters, avoid-ance of arbitrary identifiability restrictions,and simple expressions for category probabil-ities. We show substantially improved com-putational efficiency and comparable predic-tive performance to MNP.

1 Introduction

This work is motivated by the search for an alterna-tive to the multinomial logit (MNL) and multinomialprobit (MNP) models that is more amenable to effi-cient Bayesian computation, while maintaining flexi-bility. Historically, the MNP has been preferred forBayesian inference in polychotomous regression, sincethe data augmentation approach of Albert and Chib[1993] leads to straightforward Gibbs sampling. Ef-ficient methods for Bayesian inference in the MNLare a more recent development. A series of pro-posed data-augmentation methods for Bayesian in-ference in the MNL dates at least to O’Brien and

Appearing in Proceedings of the 16th International Con-ference on Artificial Intelligence and Statistics (AISTATS)2013, Scottsdale, AZ, USA. Volume 31 of JMLR: W&CP31. Copyright 2013 by the authors.

Dunson [2004], who use a student t data augmen-tation scheme with the latent t variables expressedas scale mixtures of Gaussians. The scale and de-grees of freedom in the t are chosen to provide a nearexact approximation to the logistic density. Holmesand Held [2006] represent the logistic distribution as ascale-mixture of normals where the scales are a trans-formation of Kolmogorov-Smirnov random variables.citet fruhwirth2009improved propose an alternativedata-augmentation scheme which approximates a log-Gamma distribution with a mixture of normals, result-ing in conditionally-conjugate updates for regressioncoefficients. Polson et al. [2012] develop a novel data-augmented representation of the likelihood in a lo-gistic regression using Polya-Gamma latent variables.With a normal prior, the regression coefficients haveconditionally-conjugate posteriors. Their method hasthe advantage that the latent variables can be sampleddirectly via an efficient rejection algorithm without theneed to rely on additional auxiliary variables.

Although the work outlined above has opened MNLto Gibbs sampling, the distributions of the latent vari-ables are either exotic or complex scale-mixtures ofnormals for which multivariate analogues are not sim-ple to work with. As such, the extension to analysis ofmultivariate unordered categorical data, nonparamet-ric regression, and other more complex situations is notstraightforward. In contrast, the multinomial probit(MNP) model is trivially represented by a set of Gaus-sian latent variables, allowing access to a wide range ofmethods developed for Gaussian models. MNP is alsoa natural choice for Bayesian estimation because of thefully-conjugate updates for model parameters and la-tent variables. However, because the latent Gaussiansare not conditionally independent given regression pa-rameters, mixing is poor and computation does notscale well.

Three characteristics of the multinomial probit modellead to challenging computation and render it of lim-ited use in complex problems: the need for identifyingrestrictions, the specification of non-diagonal covari-ance matrices for the residuals when it is not well-motivated, and high dependence between latent vari-

30

Diagonal Orthant Multinomial Probit Models

ables. Because our proposed method addresses allthree of these issues, we review each of them here andsummarize the pertinent literature.

1.1 Identifying Restrictions

Many choices of identifying restrictions for the MNPhave been suggested, and the choice of identifying re-strictions has important implications for Bayesian in-ference (Burgette and Nordheim [2012]). Consider thestandard multinomial probit, where here we assume asetting where covariates are constant across levels ofthe response (i.e. a classification application):

yi = j ⇔ uij =∨k

uik

uij = xiβj + εij

εi ∼ N(0,Σ)

where∨

is the max function. The uij ’s are referredto as latent utilities, after the common economic in-terpretation of them as unobserved levels of welfarein choice models. We make the distinction between aclassification application as presented above and thechoice model common to the economics literature inwhich each category has its own set of covariates (i.e.uij = xijβ + εij). A common approach to identifyingthe model is to choose a base category and take differ-ences. Suppose we select category 1 as the base. Wethen have the equivalent model:

ui1 = 0

uij = xi(βj − β1) + εij − εi1

The u’s are a linear transformation M of the originallatent utilities, so ui,2:J ∼ N(xi(β2:J−β1),MTΣM).Early approaches to Bayesian computation in theMNP placed a prior on Σ = MTΣM . Even thisparametrization is not fully identified, and requiresthat one of the variances be fixed (usually to one)to fully identify the model. McCulloch and Rossi[1994] ignored this issue and adopted a parameter-expanded Gibbs sampling approach by placing aninverse-Wishart prior on Σ. McCulloch et al. [2000]developed a prior specification on covariance matriceswith the upper left element restricted to one. Imai andVan Dyk [2005] use a working parameter that is resam-pled at each step of the Gibbs sampler to transform be-tween identified and unidentified parameters, with im-proved mixing, though the resulting full conditionalsare complex and the sampling requires twice the num-ber of steps as the original McCullough and Rossi sam-pler. Zhang et al. [2006] used a parameter-expandedMetropolis-Hastings algorithm to obtain samples froma correlation matrix, resulting in an identified modelthat restricts the scales of the utilities to be the same.

Zhang et al. [2008] extended their algorithm to multi-variate multinomial probit models.

Whatever prior specification and computational ap-proach is used, the selection of an arbitrary base cate-gory is an important modeling decision that may haveserious implications for mixing (Burgette and Hahn[2010]). The class probabilities in the MNP are linkedto the model parameters by integrating over subsets ofRJ . The choice of one category as base results in theseregions having different geometries, causing an asym-metry in the effect of translations of the elliptical mul-tivariate normal density on the resulting class proba-bilities. To address this issue, Burgette and Nordheim[2012] and Burgette and Hahn [2010] rely on an alter-native identifying restriction, circumventing the needto select an arbitrary base category. Although mixingis improved and the model has an appealing symme-try in the representation of categorical probabilities asa function of latent variables, it does not address theissue of high dependence between latent variables, andthe proposed Gibbs sampler is quite complex (thoughcomputationally no slower than simpler algorithms).

1.2 Dependence in Idiosyncratic Errors

The multinomial probit model arose initially in theeconometrics literature (see e.g. Hausman and Wise[1978]), where dependence between the errors in thelinear utility models is often considered desirable. Ineconometrics and marketing, interest often centers onpredicting the effects of changes in product prices onproduct shares. If the errors have a diagonal covari-ance matrix, the model has the “independence of irrel-evant alternatives” (IIA) property (see Train [2003]),which is often considered too restrictive for character-izing the substitution patterns between products in amarket. Because early applications for the MNP werelargely of the marketing and econometrics flavor, it hasbecome standard to specify a model with dependencein the errors without much attention to whether it isscientifically motivated. We find little compelling rea-son for this assumption for most applications outsideof economics and marketing. If the application doesnot motivate dependence in errors, the resulting modelwill certainly have higher variance than a model withindependent errors, and the additional dependence be-tween latent variables will negatively impact mixing.In this case, the applications we have in mind are notspecifically economics/marketing applications, and assuch we will generally assume that the errors are con-ditionally independent given regression parameters.

31

James E. Johndrow, Kristian Lum, David B. Dunson

1.3 Dependence Between Latent Variables

Except in the special case of marketing applicationswhere there are covariates that differ with the levelof the response variable (such as the product price),an identified MNP must have J − 1 sets of regressioncoefficients, with one set restricted to be zero. If weassume β1 = 0 and an identity covariance matrix, wecan actually sample all J latent variables for each ob-servation i. Note this is equivalent to setting one classto have utility that is identically zero and samplingthe remaining utility differences from the transformeddistribution described in section 2.1; however, the in-tractability of the high dependence between utilities ismuch clearer using this alternative parametrization.

To implement data augmentation MCMC, we mustsample the latent utilities conditional on regressioncoefficients from a multivariate normal distribution re-stricted to the space where ui,yi =

∨k uik. Albert and

Chib [1993] suggest rejection sampling; but this tendsto be extremely inefficient, particularly as J grows. Allsubsequent algorithms have sampled uij | ui,−j in se-quence. With an identity covariance matrix for latentutilities conditional on regression parameters, we canreduce this to a two-step process:

1. Set bi =∨k 6=yi uik. Sample u[i,yi] ∼

N[b,∞)(xiβyi , 1).

2. Sample u[i,j] ∼ N(−∞,u[i,yi]](xiβ[i,j], 1) indepen-

dently for all j 6= yi.

Even in simple cases, mixing tends to be poor becausethe truncation region for each latent Gaussian dependson the current value in the Markov chain for the otherlatent Gaussians, resulting in high dependence. Wereiterate that this issue is not simply a consequenceof the choice of a non-diagonal covariance matrix; itis true for any choice of covariance matrix for the ε’s.While recent work on developing a Hamiltonian MCscheme for jointly sampling from truncated multivari-ate normal distributions appears promising as an alter-native to the standard Gibbs sampling approach (Pak-man and Paninski [2012]), the need to update eachlatent variable conditional on the others has histori-cally been a substantial contibutor to the inefficiencyof Bayesian computation for MNP.

In this paper we propose Diagonal Orthant Multino-mial models (DO models), a novel class of models forunordered categorical response data that admits la-tent variable representations (including a Gaussian la-tent variable representation for the DO-probit model),does not require the selection of a base category, andin which the latent variables are conditionally inde-pendent given regression parameters and thus may be

updated in a block, greatly improving mixing and com-putational scaling. The remainder of the paper is orga-nized as follows. In section 2, we introduce DO models,and show that a unique solution to the likelihood equa-tions can be obtained from independent binary regres-sions. We also illustrate the relationship of DO-logisticto MNL and DO-probit to MNP. We give several in-terpretations for regression coefficients in DO models,and explain why the model parameters are often easierto interpret than in the MNP. In section 3, we outlinea simple algorithm for Bayesian computation in theDO-probit and discuss extensions to the basic regres-sion setting. In section 4, we compare the DO-probit,MNP, and MNL in simulation studies. In section 5,we apply both methods to a real dataset and showthat they are virtually indistinguishable in prediction.In section 6 we conclude and discuss potential futuredirections for this work.

2 The Diagonal Orthant MultinomialModel

The Diagonal Orthant Multinomial (DO) class of mod-els represent an unordered categorical variable as a setof binary variables. Let y be unordered categoricalwith J levels and suppose γ[1:J] are independent bi-nary variables. Define y = j ⇔ γj = 1 ∪ γk =0 ∀ k 6= j. Binary variables have a well-known la-tent variable representation. Let zj ∼ f(µj , σ) wheref is a location-scale density with location parameterµj and common scale σ, and set γj = 1 ⇔ zj > 0.However, for our purposes, we must ensure that onlyone γj is one, and thus we restrict the z’s to belong tothe set:

Ω =

J⋃j=1

z ∈ RJ : zj > 0, zk < 0, k 6= j

By the Radon-Nikodym theorem, the joint distributionof the z’s is:

f(z) =1(z ∈ Ω)

∏Jj=1 f(zj − µj)∫

RJ1(z ∈ Ω)

J∏j=1

f(zj − µj)dz

If we let f = φ(·), where φ(·) is the univariate normalpdf, the result is a probit analogue that we refer to asDO-Probit.

The joint probability density of z’s in DO-probit isthat of a J-variate normal distribution with identity

32

Diagonal Orthant Multinomial Probit Models

covariance that is restricted to the regions of RJ withone sign positive and the others negative. This is easyto visualize in R2, where the density is a bivariate nor-mal with correlation zero and unit variance restrictedto the second and fourth quadrants.

The density in the two dimensional case is shown inFigure 1. In higher dimensions, the restriction willdefine orthants over which the density is nonzero. Themarginal distribution of any two latent variables willalways be restricted to orthants that are diagonallyapposed rather than adjacent, hence the designationDiagonal Orthant Multinomial model.

Figure 1: The joint density of the z’s in the DO-probit forthe 2-category case with zero means (left panel). The rightpanel shows approximate semispherical regions covering 95percent of the total probability in the 3-category case withzero means for all categories.

The probability measure for z induces a probabilitymeasure on y. The categorical probabilities are easilycalculated as:

Pr(y = j) =(1− F (−µj))

∏k 6=j F (−µk)∑J

j=1(1− F (−µj))∏k 6=j F (−µk)

= ψj(µ1, . . . , µJ)

where F (·) is the CDF corresponding to f . Clearly, iff = φ, then F = Φ(·) is the standard normal CDF.While strictly speaking, the DO model with a full set ofJ category-specific intercepts and J×p regression coef-ficients is not identified, in the next section we suggesta simple identifying restriction that, critically, allowsthe parameters for the DO-model to be estimated viaindependent binary regressions, providing substantialcomputational advantages.

2.1 Interpretation of Regression Coefficientsand Relationship to Multinomial Logit

In a regression context, DO models define an alterna-tive link function in a GLM for unordered categori-cal/multinomial data, where:

yi ∼ Categorical(pi)

pi =(ψ1(xi,β[1:J]), ψ2(xi,β[1:J]), . . . , ψJ(xi,β[1:J])

)

The conditional class probabilities in the DO mod-els have a useful correspondence with the multinomiallogit. The class probabilities in the regression prob-lem, Pr(yi = j | xi,β[1:J]) = ψj(xi,β[1:J]), are givenby:

(1 − F (−xiβj))∏

l 6=j F (−xiβl)∑Js=1(1 − F (−xiβs))

∏l 6=j F (−xiβl)

The ratio of probabilities for two classes j and k istherefore:

[(1−F (−xiβj))∏l6=j F (−xiβl)]

[∑Js=1(1−F (−xiβs))

∏l 6=s F (−xiβl)]

[(1−F (−xiβk))∏l 6=k F (−xiβl))]

[∑Js=1(1−F (−xiβs))

∏l6=s F (xiβl)]

The denominators and all but one of the terms in theproducts in the numerators cancel, leaving:

ψj(xi,β[1:J])

ψk(xi,β[1:J])=

(1− F (−xiβj))F (−xiβk)

(1− F (−xiβk))F (xiβj)

=(1−F (−xiβj))/F (−xiβj)(1−F (−xiβk))/F (−xiβk)

which is a function of only βj and βk. Recall that forthe multinomial logit model, the ratio of class proba-bilities for classes j and k also depends only on βj andβk and is given by:

log

(Pr(yi = j)

Pr(yi = k)

)= xiβj − xiβk

In DO-probit, F (·) = Φ(·), so the log relative class

probability, log(

Pr(yi=j)Pr(yi=k)

), is:

log

(Φ(−xiβj)

1 − Φ(−xiβj)

)− log

(Φ(−xiβk)

1 − Φ(−xiβk)

)While not as convenient as the linear expression

arising from the multinomial logit, this quantity isnonetheless easily calculated and provides a direct re-lationship between the coefficients in the multinomiallogit and those in the DO-probit. This is a substantialadvantage over the multinomial probit model, in whichthe class probabilities do not have a closed form.

The more interesting case arises when we choose a lo-gistic distribution for the zj ’s in the DO model, givingthe DO-logistic model. Here, F (t) = 1

1+e−t is the lo-gistic CDF, and the log probability ratio for two cate-gories is:

log

(p(j)i

p(k)i

)= log

(1

1+exiβk

1 − 1

1+exiβk

)− log

( 1

1+exiβj

1 − 1

1+exiβj

)

= log

1

1+exiβkexiβj

1+exiβj

1

1+exiβj

exiβk

1+exiβk

(1)

= xiβj − xiβk

33

James E. Johndrow, Kristian Lum, David B. Dunson

where p(j)i = Pr(yi = j). This is identical to the

log probability ratios in the multinomial logit. Evi-dently, the DO-logistic is an alternative form of theMNL. In the next section, we suggest an identifyingrestriction for the DO model that is much more con-venient than the use of an arbitrary base category inMNL models and treats all of the categories identically.We also note that using the approach in O’Brien andDunson [2004] one can easily do computation for theDO-logistic model by introducing a scale parameter forthe latent variables that is mixed over a Gamma den-sity. This immediately suggests an alternative Gibbssampling algorithm for an MNL-like model that in-volves only one additional sampling step relative tothe DO-probit.

2.2 Identification

Closer inspection of (1) reveals something rather strik-ing about the representation of category probabilitiesin DO models. Consider (1) in an intercepts-onlymodel:

log

(p(j)i

p(k)i

)= log

(1

1+eµkeµj

1+eµj

11+eµj

eµk1+eµk

)

As presented thus far, DO is an under-identified gen-eralized linear model, and thus there will be multiplesets of parameters µ1, . . . , µJ that maximize the like-lihood (inifinitely many, in fact). However, all of thesolutions to the likelihood equations µ1, . . . , µJ mustsatisfy:

log

(pjpk

)= µj − µk

for any j, k ∈ 1, . . . , J, a simple consequence of MLEinvariance. Yet (1) suggests that we might identify themodel and obtain a very useful approach to estimationsimply by recognizing the connection with binary lo-gistic regression. Consider an intercept-only binary

logistic regression with response y(j)i = 1(yi = j) and

let p(B)j = Pr(yj = 1). We have that:

log

(p(B)j

1− p(B)j

)= µ

(B)j

and so for two independent logistic regressions of yjand yk on an intercept, we get:

log

(p(B)j (1− p(B)

k )

p(B)k (1− p(B)

j )

)= µ

(B)j − µ(B)

k

But since

p(B)j

1− p(B)j

=

eµ(B)j

1+eµ(B)j

1

1+eµ(B)j

we have that:

log

1

1+eµ(B)k

eµ(B)j

1+eµ(B)j

1

1+eµ(B)j

eµ(B)k

1+eµ(B)k

= µ(B)j − µ(B)

k

Therefore the collection µ(B)1 , . . . , µ

(B)J - that is,

the MLEs from independent binary regressions ony1, . . . , yJ - is a valid solution to the likelihood equa-

tions for the DO model. Since the MLEs µ(B)j for any

j are unique, this solution is also unique, and is in factidentical to the solution that results from imposingthe restriction

∑Jj=1 pj = 1 in the DO logistic model.

A similar argument goes through for the general DOmodel. This further hints at the strategy we employfor Bayesian computation, which is identical to thatused for J independent binary regressions. Of course,this was by construction; DO models were conceivedof and designed expressly to allow for Bayesian com-putation using a set of J independent latent variables.

Note that while this is conceptually similar to thederivation of the multinomial logit model from inde-pendent binary regressions, in that case the response

for each binary regression is defined by y(j)i = 1 if

yi = j and y(j)i = 0 if yi = b, where b is the base cate-

gory, and thus the binary regressions are on a subset ofthe observations and defined against an arbitrary basecategory. For DO models, we perform binary regres-sions on all of the observations and do not require abase category. We also have the advantage of an addi-tional interpretation for the estimated parameters asrelating to the marginal probability of each category.

2.3 Relationship of DO-probit toMultinomial Probit

Both MNP and DO-probit link probability vectors tolatent Gaussian variables by integrating multivariatenormal densities over particular regions. For reasonsdiscussed earlier, we consider only cases in which thelatent Gaussian variables are conditionally indepen-dent given parameters. To simplify the exposition, weconsider an intercepts-only MNP and a correspondingDO-probit model with J category-specific mean pa-rameters and the identifying restriction presented inthe previous section. Suppose we have an MNP withidentity covariance and restrict the mean of the firstutility to be zero. If we take µ2 = −0.75, the result-ing category probabilities for the trivial two-category

34

Diagonal Orthant Multinomial Probit Models

model are (0.7115, 0.2885). The equivalent DO-probitmodel will have parameters µ1 = Φ−1(0.7115) =0.5578 and µ2 = Φ−1(0.2885) = −0.5578. The twomodels give identical category probabilities, but theyarrive at them differently. Figure 2 shows the con-tours of the bivariate normal distribution for the la-tent utilities in the MNP (left panel) and the trun-cated bivariate normal distribution in the DO-probit(right panel). The MNP integrates above and belowthe line y = x (shown on the figure) to calculate prob-abilities, whereas DO-probit integrates over the secondand fourth quadrants.

Figure 2: Contours and integration regions for equivalenttwo-category MNP and DO-probit models with categoryprobabilities (0.7115, 0.2885).

Of course, we can define a 1-1 function between MNPswith J categories, an identity covariance matrix, andµ1 = 0 and DO-probit with the marginal MLE restric-tion outlined in the previous section. The MNP prob-abilities for such a model are defined by the intractableintegrals:

pj = Pr(y = j) =

∫ ∞−∞

φ(zj − µj)

×[∫ zj

−∞φ(zJ − µJ)dzJ . . .

∫ zj

−∞φ(z1)dz1

]dzj

for j = 1, . . . , J . The integral has no analytic form. Assuch, marginalizing over the latent variables in MNPis computationally costly, and thus the latent variablesare usually conditioned on in MCMC algorithms ratherthan integrating them out. This leads to high depen-dence and inefficient computation. In contrast, onecan marginalize out the latent variables in DO-probitand obtain analytic expressions for the category prob-abilities, a significant benefit of our approach.

One can approximate the MNP integrals by quadra-ture or simulation, giving a set of probabilitiesp1, . . . , pJ . We can then find the equivalent parametersof a DO-probit by simply inverting the probabilities,i.e. µj = Φ−1(pj) for all j. Note that while cate-gory probabilities are increasing in µ for both models,the MNP does not have the simple interpretation ofthe regression parameters as relating to the marginalcategory probabilities, and cannot be estimated fromindependent binary regressions as can the DO-probit.

3 Computation

Bayesian computation for the DO-probit is verystraightforward. With a N(0, cI), c ∈ R+ prior on theregression coefficients, the entire algorithm consists ofGibbs steps:

1. Sample zi,yi | βyi ∼ N+(xTi βyi , 1) and zi,k ∼N−(xTi βk, 1) for k 6= yi, where N+ and N− repre-sents a normal distribution truncated below andabove by zero, respectively.

2. Sample βk | z ∼ N(µ, S) with S = (xTx+1/cI)−1

and µ = xTz·,k for k ∈ 1, . . . ,K.

Moreover, because of the latent Gaussian data aug-mentation scheme used to estimate the model, thereare many alternatives to normal priors on regressioncoefficients. In classification problems with p covari-ates, the dimension of the parameter space is J × p(or J × (p− 1) in the MNL and MNP), where J is thenumber of possible values of y. Also, the larger the J ,the less information is provided by observing yi = j.Thus, even with modest p and fairly large n, one shouldconsider priors on regression coefficients with robustshrinkage properties, particularly when prediction isan important goal. The local-global family of shrink-age priors have the scale-mixture representation:

βjl ∼ N(0, τ2φ2jl)

where we have adapted the notation to the regres-sion classification context with j ∈ 1, . . . , J andl ∈ 1, . . . , p. Here, τ is a global shrinkage parameterand φjl are local shrinkage parameters correspondingto βjl. There is a substantial literature on priors for τand the φjl’s that favor aggressively shrinking most ofthe coefficients toward zero while retaining the sparsesignals. For example, choosing independent C+(0, 1)priors on τ and the φjl’s gives the horseshoe prior(Carvalho et al. [2010]), where C+(0, 1) is a standardhalf-Cauchy distribution. Local-global priors can beemployed in our model, with the only additional com-putational burden relative to the continuous responsecase being the imputation of the latent variables.

It is equally straightforward to apply other priors onregression parameters. Bayesian variable selection andmodel averaging can be performed by specifying point-mass mixture priors on βjl and employing a stochasticsearch variable selection algorithm (see Hoeting et al.[1999] for an overview of these methods). A nonpara-metric prior on regression parameters can be obtainedby specifying Gaussian process (GP) priors on the la-tent variables. The simplest approach is to specify Jindependent GP’s corresponding to the J sets of la-tent variables (one for each possible value of y). In

35

James E. Johndrow, Kristian Lum, David B. Dunson

MNP, this leads to difficult computation, and thus itis often necessary to use posterior approximations tomake computation tractable (see Girolami and Rogers[2006]). The superior computational properties of DO-probit suggest that exact posterior computation usingMCMC may be feasible with GP priors for problemsof meaningful scale.

4 Simulation Studies

We conducted a series of simulation studies to assessthe performance and properties of DO-probit and tocompare it with MNP. We first simulated data froma MNP with n × p design matrix x with identity co-variance matrix. We used n = 2000, p = 2, and J = 5levels of the response. The (J−1) category-specific in-tercepts were sampled from N(0, .5) and the (J − 1)pcoefficients were sampled from N(0, 1). We then fiteither MNP or DO-probit by Gibbs sampling. Thechains were run for 10,000 iterations each, in everycase starting from β = 0. We repeated the simulationand subsequent fitting 10 times. Note that the param-eter expansion algorithms designed to improve mixingare irrelevant for fitting this MNP model since we donot need to sample a covariance matrix. We choosecategory 1 as the base category for fitting.

Figure 3 shows histograms of lag-10, lag-25, and lag-100 autocorrelations for the Markov chains for β fromMNP and DO-probit across the 10 simulations. Au-tocorrelations are much lower at all three lag lengths.This is a critical aspect of the performance of Bayesianstochastic algorithms. Higher autocorrelations requiremuch longer run times to achieve the same effectivesample sizes. In more complex models, the autocor-relations for MNP can be prohibitively high. In thefollowing section, we show that lower autocorrelationsare a feature of the DO-probit that persists in real ap-plications with more complex priors on the coefficients.

Table 1 shows the mean in-sample misclassificationrate (using the posterior mode as the prediction) fromeach of the ten simulations. The results are quite com-parable for the two models, suggesting that DO-probitand MNP are exchangeable in regard to their perfor-mance as classifiers.

5 Applications

We consider the glass identification data from the UCImachine learning website (Frank and Asuncion [2010]).There are 214 observations in the data and the re-sponse (class of the glass sample) is unordered cate-gorical with seven possible levels, of which six are ob-served. There are nine continuous covariates. Thusa DO-probit classifier has sixty parameters (6 × 9 re-

Figure 3: Distribution of Lag-10, -25, and -100 autocorre-lations across all simulations and parameters for DO-probit(top) and MNP (bottom)

Simulation MNP DO-probit1 0.30 0.302 0.36 0.373 0.51 0.514 0.32 0.325 0.42 0.426 0.49 0.497 0.49 0.488 0.36 0.379 0.42 0.4210 0.45 0.45

Table 1: Comparison of misclassification rates for each of10 simulated data sets.

gression coefficients and 6 intercepts). Because n isnot large relative to p in this case, we use a Horseshoeshrinkage prior on the β’s (see Carvalho et al. [2010]and the discussion in section 3).

A boxplot of posterior samples for β is shown in theleft panel of figure 4, and a corresponding plot for theMNP with identity covariance matrix is presented inthe right panel. Red colored boxes indicate parametersthat are considered nonzero on the basis of the crite-ria suggested in Carvalho et al. (let κjk = 1/(1−τ2φ2

jk),and consider βjk nonzero if κjk, the posterior meanof κjk, is > 0.5). Recent work has shown that thisinclusion criterion has optimal properties under a 0-1loss function (Datta and Ghosh [2012]). Note that ofthe 60 coefficients in the DO-probit, 56 are effectivelyshrunk to zero, whereas of the 50 coefficients in theMNP, 39 of them are effectively shrunk to zero. Inaddition, of the 9 covariates, 6 of them have all 6 coef-ficients shrunk to zero in the DO-probit, which is theequivalent of excluding that covariate entirely. In theMNP, 5 of these 6 covariates also have all 5 coefficients

36

Diagonal Orthant Multinomial Probit Models

effectively shrunk to zero.

Figure 4: Boxplots of posterior samples for β1:p for MNPand DO-probit. Red boxes indicate that the coefficient isconsidered nonzero by the criterion described above.

Figure 5 shows boxplots of the lag 1 through lag 50posterior autocorrelations in the Markov chains for βfrom the DO-probit and MNP models. The boxesshow the interquartile range and the black dots themedians. This confirms that much lower autocorre-lations in the DO-probit persist in real applicationswith larger number of parameters and more complexhierarchical modeling structures. Note that the resid-ual autocorrelation for DO-probit is due mainly to theMetropolis-Hastings steps used to sample the scale pa-rameters for the horseshoe prior, and that parameterexpansion or slice sampling could improve the mixing,see Scott [2010]. Table 2 shows quantiles of the ef-fective sample size for DO-probit and MNP estimatedon the glass data. The median effective sample sizefor DO-probit is 3.85 times that for MNP, even in thisrelatively simple modeling context. The run time periteration for DO-probit was 1.06 times that of MNP, adifference that is entirely due to the larger number ofparameters for DO-probit.

twenty

10% 25% 50% 75% 90%DO probit 101 250 1204 3176 6085

MNP 78 128 216 464 835

Table 2: Quantiles of effective sample sizes for β parame-ters from DO-probit and MNP using 15,000 MCMC itera-tions with 1000 iteration burn-in.

We assessed out-of-sample prediction on the datasetby randomly holding out 10 percent of observationsand estimating the model on the remaining 90 per-

cent, a process that was repeated twenty times. Weformed out of sample predictions by taking the poste-rior mode of the predicted class for each observationin the test set. The full set of 60 (50) coefficients wereused for prediction in the DO-probit (MNP) models,which is standard practice for shrinkage priors. Table3 shows the best, median, and worst misclassificationrates for the two models. The results show that thetwo classifiers are equivalent. Note that for over halfof the test datasets, the misclassification rates for thetwo models were identical.

best median worstProbit 0.22 0.38 0.58

DO-Probit 0.22 0.37 0.60

Table 3: Summary of misclassification rates for 20 randomholdouts of glass identification data.

Figure 5: Boxplots of lag-1 through lag-50 autocorrela-tions for MCMC samples of β1:p from MNP and DO-probit.

6 Discussion

The DO-probit model provides an attractive alterna-tive to the multinomial logit and multinomial pro-bit models that overcomes the major limitations ofeach while retaining many of their attractive proper-ties. Like MNP, DO-probit admits a Gaussian latentvariable representation, allowing for simple conjugateupdates for regression parameters and application ofnumerous methods designed for multivariate Gaussiandata, a feature that MNL lacks. However, our modeldoes not suffer from the poor mixing and high auto-correlations in MCMC samples that make MNP prac-tically infeasible for use in high-dimensional applica-tions. Unlike MNP and MNL, our model does not re-quire a base category for identification. However, class

37

James E. Johndrow, Kristian Lum, David B. Dunson

probabilities for DO-probit marginal of latent variablesare easily calculated, and relative class probabilitiesare functions only of the regression parameters corre-sponding to the compared classes, an attractive featureshared with MNL.

The DO-probit link provides a number of possibleavenues for future work. The Gaussian latent vari-able representation of the model allows for extensionto nonparametric regression via specifying a Gaussianprocess prior on the latent variables. Another interest-ing possibility would be to explore a novel class of dis-crete choice models by allowing dependence betweenlatent variables, the analogue of a non-diagonal co-variance matrix in the MNP. Multivariate unorderedcategorical response data with covariates is a particu-larly challenging context in which to develop compu-tationally tractable Bayesian methods. One could po-tentially allow for dependence in multivariate cases byspecifying a prior on structured covariance matrices forthe latent Gaussian variables. Another possible appli-cation would be to time series of polychotomous vari-ables by specifying AR models on the latent variables.In complex modeling situations such as these, fullyBayesian estimation using DO-probit may be straight-forward, whereas the computational challenges of theMNP make these extensions, while theoretically pos-sible, practically infeasible.

7 Acknowledgments

This work was supported by Award NumberR01ES017436 from the National Institute of Environ-mental Health Sciences. The content is solely the re-sponsibility of the authors and does not necessarilyrepresent the official views of the National Instituteof Environmental Health Sciences or the National In-stitutes of Health. This work has been partially sup-ported by DTRA CNIMS Contract HDTRA1-11-D-0016-0001 and Virginia Tech internal funds.

References

J.H. Albert and S. Chib. Bayesian analysis of binaryand polychotomous response data. Journal of theAmerican statistical Association, 88(422):669–679,1993.

L.F. Burgette and P.R. Hahn. Symmetric bayesianmultinomial probit models. Duke University Statis-tical Science Technical Report, pages 1–20, 2010.

L.F. Burgette and E.V. Nordheim. The trace restric-tion: An alternative identification strategy for thebayesian multinomial probit model. Journal of Busi-ness & Economic Statistics, 30(3):404–410, 2012.

C.M. Carvalho, N.G. Polson, and J.G. Scott. Thehorseshoe estimator for sparse signals. Biometrika,97(2):465–480, 2010.

J. Datta and J.K. Ghosh. Asymptotic properties ofbayes risk for the horseshoe prior. Bayesian Analy-sis, 7(4):771–792, 2012.

A. Frank and A. Asuncion. UCI ma-chine learning repository, 2010. URLhttp://archive.ics.uci.edu/ml.

M. Girolami and S. Rogers. Variational bayesian multi-nomial probit regression with gaussian process pri-ors. Neural Computation, 18(8):1790–1817, 2006.

J.A. Hausman and D.A. Wise. A conditional probitmodel for qualitative choice: Discrete decisions rec-ognizing interdependence and heterogeneous prefer-ences. Econometrica: Journal of the EconometricSociety, pages 403–426, 1978.

J.A. Hoeting, D. Madigan, A.E. Raftery, and C.T.Volinsky. Bayesian model averaging: a tutorial. Sta-tistical science, pages 382–401, 1999.

Chris C. Holmes and Leonhard Held. Bayesian aux-iliary variable models for binary and multinomialregression. Bayesina Analysis, 1(1):145–168, 2006.

K. Imai and D.A. Van Dyk. A bayesian analysis of themultinomial probit model using marginal data aug-mentation. Journal of Econometrics, 124(2):311–334, 2005.

R. McCulloch and P.E. Rossi. An exact likelihoodanalysis of the multinomial probit model. Journalof Econometrics, 64(1):207–240, 1994.

R.E. McCulloch, N.G. Polson, and P.E. Rossi. Abayesian analysis of the multinomial probit modelwith fully identified parameters. Journal of Econo-metrics, 99(1):173–193, 2000.

S.M. O’Brien and D.B. Dunson. Bayesian multivariatelogistic regression. Biometrics, 60(3):739–746, 2004.

Ari Pakman and Liam Paninski. Exact hamiltonianmonte carlo for truncated multivariate gaussians.arXiv preprint arXiv:1208.4118, 2012.

Nick G. Polson, James G. Scott, and Jesse Windle.Bayesian inference for logistic models using Polya-Gamma latent variables. ArXiv e-prints, 2012.

J.G. Scott. Parameter expansion in local-shrinkagemodels. arXiv preprint arXiv:1010.5265, 2010.

K.E. Train. Discrete choice methods with simulation.Cambridge university press, 2003.

38

Diagonal Orthant Multinomial Probit Models

X. Zhang, W.J. Boscardin, and T.R. Belin. Samplingcorrelation matrices in bayesian models with corre-lated latent variables. Journal of Computational andGraphical Statistics, 15(4):880–896, 2006.

X. Zhang, W.J. Boscardin, and T.R. Belin. Bayesiananalysis of multivariate nominal measures usingmultivariate multinomial probit models. Computa-tional statistics & data analysis, 52(7):3697–3708,2008.