Heterogeneous Multi-output Gaussian Process PredictionPablo Moreno-Muñoz1 Antonio Artés-Rodríguez1 Mauricio A. Álvarez2

1Universidad Carlos III de Madrid, Spain 2University of Sheffield, UK{pmoreno, antonio}@tsc.uc3m.es mauricio.alvarez@sheffield.ac.uk

IntroductionA novel extension of multi-output Gaussian processes (MOGPs) for handling heterogeneous outputs (bi-nary, real, categorical, . . . ). Each output has its own likelihood distribution and we use a MOGP prior tojointly model the parameters in all likelihoods as latent functions. We are able to obtain tractable variationalbounds amenable to stochastic variational inference (SVI).

Multi-output GPsWe will use a linear model of corregionalisation type of covariance function for expressing correlationsbetween latent parameter functions fd,j(x) (LPFs).

Each LPF is a linear combination of independent latent functions U = {uq(x)}Qq=1. Each uq(x) is assummedto be drawn from a GP prior such that uq(·) ∼ GP(0, kq(·, ·)), where kq can be any valid covariance function.

fd,j(x) =

Q∑q=1

Rq∑i=1

aid,j,quiq(x),

We assume that Rq = 1, meaning that the corregionalisation matrices are rank-one. In the literature suchmodel is known as the semiparametric latent factor model.

Heterogeneous Likelihood ModelConsider a set of output functions Y = {yd(x)}Dd=1, with x ∈ Rp, that we want to jointly model using GPs.Let y(x) = [y1(x), y2(x), · · · , yD(x)]> be a vector-valued function. If outputs are conditionally independentgiven the vector of parameters θ(x) = [θ1(x), θ2(x), · · · , θD(x)]>, we may define

p(y(x)|θ(x)) = p(y(x)|f(x)) =D∏

d=1

p(yd(x)|θd(x)) =D∏

d=1

p(yd(x)|̃fd(x)),

where f̃d(x) = [fd,1(x), · · · , fd,Jd(x)]> ∈ RJd×1 are the set of LPFs that specify the parameters in θd(x) for anarbitrary number D of likelihood functions.

Variational BoundsSparse Approximations in MOGPs: We define the set of M inducing variables per latent function uq(x)as uq = [uq(z1), · · · , uq(zM)]>, evaluated at a set of inducing inputs Z = {zm}Mm=1 ∈ RM×p. We also defineu = [u>1 , · · · ,u>Q]> ∈ RQM×1. We approximate the posterior p(f ,u|y,X) as follows:

p(f ,u|y,X) ≈ q(f ,u) = p(f |u)q(u) =D∏

d=1

Jd∏j=1

p(fd,j|u)Q∏

q=1

q(uq),

Variational Inference: Exact posterior inference is intractable in our model due to the presence of an arbi-trary number of non-Gaussian likelihoods. We use variational inference to compute a lower bound L forthe marginal log-likelihood log p(y), and for approximating the posterior distribution p(f ,u|D).

L =D∑

d=1

Eq(f̃d)

[log p(yd(xn)|̃fd)

]−

Q∑q=1

KL(q(uq)||p(uq)

)Acknowledgements: PMM is supported by a doctoral FPI grant (BES2016-077626) under the project Macro-ADOBE (TEC2015-67719-P), MINECO, Spain. AARacknowledges the projects ADVENTURE (TEC2015-69868-C2-1-R), AID (TEC2014-62194-EXP) and CASI-CAM-CM (S2013/ICE-2845). MAA has been partiallyfinanced by the Engineering and Physical Research Council (EPSRC) Research Projects EP/N014162/1 and EP/R034303/1.

Results Code→ github.com/pmorenoz/HetMOGPMissing Gap Prediction: We predict observations in one output (binary classification) using training information from anotherone (Gaussian regression). Multi-output test-NLPD value: 32.5±0.2×10−2 / Single-output test-NLPD value: 40.51±0.08×10−2.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−6

−4

−2

0

2

4

6

Real Input

RealOutput

Output 1: Gaussian Regression

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.2

0.4

0.6

0.8

1

Real Input

Bin

ary

Ou

tpu

t

Output 2: Binary Classification

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

0

0.2

0.4

0.6

0.8

1

Real Input

Bin

ary

Ou

tpu

t

Single Output: Binary Classification

London House Price: Complete register of properties sold in the Greater London Countyduring 2017. All properties addresses were translated to latitude-longitude points. For eachspatial input, we considered two observations, one binary (property type) and one real(sale price).

-0.51 -0.34 -0.17 -0.0 0.16 0.3351.29

51.37

51.45

51.53

51.61

51.69

Longitude

Latitude

Property Type

FlatOther

-0.51 -0.34 -0.17 -0.0 0.16 0.3351.29

51.37

51.45

51.53

51.61

51.69

Longitude

Latitude

Sale Price

79K£

167K£

351K£

738K£

1.5M£

-0.51 -0.34 -0.17 -0.0 0.16 0.3351.29

51.37

51.45

51.53

51.61

51.69

Longitude

Latitude

Probability of Flat House

0

0.2

0.4

0.6

0.8

1

-0.51 -0.34 -0.17 -0.0 0.16 0.3351.29

51.37

51.45

51.53

51.61

51.69

Longitude

Latitude

Log-price Variance

0.3

0.6

0.9

1.2

1.5

1.8

2.1

2.4

TEST-NLPD[LONDON] Bernoulli Heteroscedastic Global

HetMOGP 6.38± 0.46 10.05± 0.64 16.44± 0.01ChainedGP 6.75± 0.25 10.56± 1.03 17.31± 1.06

Human Behavior Data: Wemodel human behavior in psy-chiatric patients. Our datacomes from a medical study thatuses the monitoring app (eB2).

Monday

Tuesday

Wednesday

Thursday

Friday

Saturday

Sunday

0

0.2

0.4

0.6

0.8

1

Output1:

Binary

Presence/Absence at Home

Monday

Tuesday

Wednesday

Thursday

Friday

Saturday

Sunday

−4

−2

0

2

4

Output2:

Log-distance

Distance from Home (Km)

Monday

Tuesday

Wednesday

Thursday

Friday

Saturday

Sunday

0

0.2

0.4

0.6

0.8

1

Output3:

Binary

Use/non-use of Whatsapp

ConclusionsWe present a MOGP model for handling heterogeneous obser-vations that is able to work on large scale datasets. Experimentalresults show relevant improvements with respect to indepen-dent learning.

ReferencesY. W. Teh et al. Semiparametric latent factor models. AISTATS, 2005M. A. Álvarez et al., Sparse convolved Gaussian processes for multi-output regres-sion. NIPS, 2008J. D. Hadfield, MCMC methods for multi-response GLMMs. JSS, 2010J. Hensman et al., Gaussian processes for big data. UAI, 2013A. Saul et al., Chained Gaussian processes. AISTATS, 2016

Likelihood Linked Parameters

Gaussian µ(x) = f , σ(x)

Het. Gaussian µ(x) = f1, σ(x) = exp(f2)

Bernoulli ρ(x) = exp(f)1+exp(f)

Categorical ρk(x) =exp(fk)

1+∑K−1

k′=1exp(fk′ )

Poisson λ(x) = exp(f)

Gamma a(x) = exp(f1), b(x) = exp(f2)