hyperbolic neur al ne tworks - 2020 conference06-15...poincaré embeddings for learning hierarchical...

Hyperbolic Neural NetworksHyperbolic Neural Networks

Use hyperbolic space instead of Euclidean spacefor embedding data with a latent hierarchical structure

imag

e so

urce

: htt

p://

insp

irehe

p.ne

t/re

cord

/135

5197

/plo

ts

The volume of a ball growsexponentially with its

radius!

Use hyperbolic space instead of Euclidean spacefor embedding data with a latent hierarchical structure

imag

e so

urce

: htt

p://

insp

irehe

p.ne

t/re

cord

/135

5197

/plo

ts

The volume of a ball growsexponentially with its

radius!

Use hyperbolic space instead of Euclidean spacefor embedding data with a latent hierarchical structure

Imag

e so

urce

: htt

p://

prio

r.si

gchi

.org

Similarly as for a tree: the number of nodesgrows exponentially with the tree depth!

Image source: http://prior.sigchi.org

Hot topic in ML since 

Poincaré Embeddings for LearningHierarchical Representations, Nickel & Kiela, (NIPS 2017)

Use hyperbolic space instead of Euclidean spacefor embedding data with a latent hierarchical structure

Poincaré BallPoincaré Ball

Poincaré BallPoincaré Ball

Poincaré BallPoincaré Ball

Our contributionsOur contributions

Image sources: stackexchange.com , wikipedia.org

exp (v)x

Our contributionsOur contributions

Our contributionsOur contributions

Our contributionsOur contributions

Our contributionsOur contributions

Riemannian OptimizationRiemannian Optimization

Both Euclidean and hyperbolic parameters

Riemannian SGD:

Riemannian gradient:

x ← exp (−η∇ L), x ∈xc

xR D c

n

∇ L =xR (1/λ ) ∇ L, conformal factor λ =x

c 2x x

c

1 − c∥x∥2

2

exp (v)x

Image source: stackexchange.com

ExperimentsExperiments

All word and sentence embeddings have dimension 5.

ExperimentsExperiments

ExperimentsExperiments

ExperimentsExperiments

THANK YOU!THANK YOU!

hyperbolicdeeplearning.com

Please visit our website:

Octavian Ganea is currently looking for postdoctoral positions!

Matrix-vector multiplication

We define:

Nice properties:

When the curvature c goes to zero, it recovers the usualmatrix multiplication!

lim M (x) =c→0⊗ c Mx

Matrix-vector multiplication