on computational thinking, inferential thinking and data ... · computational thinking and...

66
On Computational Thinking, Inferential Thinking and Data Science Michael I. Jordan University of California, Berkeley August 7, 2018

Upload: ngodien

Post on 19-Aug-2019

248 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

On Computational Thinking, Inferential Thinking and Data Science

Michael I. JordanUniversity of California, Berkeley

August 7, 2018

Page 2: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Some Perspective• What’s new?

– not just large data sets, but data sets containing abundant data on each individual– where “individual” can be a person, a gene, a region of the sky, a habitat, etc– and where there are long tails

Page 3: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Some Perspective• What’s new?

– not just large data sets, but data sets containing abundant data on each individual– where “individual” can be a person, a gene, a region of the sky, a habitat, etc– and where there are long tails

• What’s challenging?– timely, trustable, and transparent inference and decision-making at the level of individuals

Page 4: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Some Perspective• What’s new?

– not just large data sets, but data sets containing abundant data on each individual– where “individual” can be a person, a gene, a region of the sky, a habitat, etc– and where there are long tails

• What’s challenging?– timely, trustable, and transparent inference and decision-making at the level of individuals

• What’s required?– not just a library of algorithms, but a blend of computational thinking and inferential thinking

Page 5: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

A Job Description, circa 2015

• Your Boss: “I need a Big Data system that will replace our classic service with a personalized service”

Page 6: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

A Job Description, circa 2015

• Your Boss: “I need a Big Data system that will replace our classic service with a personalized service”

• “It should work reasonably well for anyone and everyone; I can tolerate a few errors but not too many dumb ones that will embarrass us”

Page 7: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

A Job Description, circa 2015

• Your Boss: “I need a Big Data system that will replace our classic service with a personalized service”

• “It should work reasonably well for anyone and everyone; I can tolerate a few errors but not too many dumb ones that will embarrass us”

• “It should run just as fast as our classic service”

Page 8: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

A Job Description, circa 2015

• Your Boss: “I need a Big Data system that will replace our classic service with a personalized service”

• “It should work reasonably well for anyone and everyone; I can tolerate a few errors but not too many dumb ones that will embarrass us”

• “It should run just as fast as our classic service”• “It should only improve as we collect more data; in

particular it shouldn’t slow down”

Page 9: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

A Job Description, circa 2015

• Your Boss: “I need a Big Data system that will replace our classic service with a personalized service”

• “It should work reasonably well for anyone and everyone; I can tolerate a few errors but not too many dumb ones that will embarrass us”

• “It should run just as fast as our classic service”• “It should only improve as we collect more data; in

particular it shouldn’t slow down”• “There are serious privacy concerns of course, and

they vary across the clients”

Page 10: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

What’s Required

• Data Science requires a thorough blending of computational thinking and inferential thinking

Page 11: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

What’s Required

• Data Science requires a thorough blending of computational thinking and inferential thinking

• Computational thinking means (inter alia)– abstraction, modularity, scalability, robustness, etc.

Page 12: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

What’s Required

• Data Science requires a thorough blending of computational thinking and inferential thinking

• Computational thinking means (inter alia)– abstraction, modularity, scalability, robustness, etc.

• Inferential thinking means (inter alia)– considering the real-world phenomenon behind the data– considering the sampling pattern that gave rise to the data– developing procedures that will go “backwards” from the

data to the underlying phenomenon

Page 13: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

The Challenges are Daunting

• The core theories in computer science and statistics developed separately and there is an oil and water problem

• Core statistical theory doesn’t have a place for runtime and other computational resources

• Core computational theory doesn’t have a place for statistical risk

Page 14: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

• Inference under privacy constraints• Inference under communication constraints• Lower bounds, the variational perspective and

symplectic integration

Outline

Page 15: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Part I: Inference and Privacy

with John Duchi and Martin Wainwright

Page 16: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

• Individuals are not generally willing to allow their personal data to be used without control on how it will be used and now much privacy loss they will incur

• “Privacy loss” can be quantified (say) via differential privacy

• We want to trade privacy loss against the value we obtain from “data analysis”

• The question becomes that of quantifying such value and juxtaposing it with privacy loss

Privacy and Data Analysis

Page 17: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Privacy

query

database

Page 18: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Privacy

query

database

Page 19: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Privacy

query

database privatizeddatabase

Q

Page 20: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Privacy

query

database

query

privatizeddatabase

Q

Page 21: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Privacy

query

database

query

privatizeddatabase

Q

Page 22: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Privacy

query

database

query

privatizeddatabase

Q

✓ ✓Classical problem in differential privacy: show that and are close under constraints on Q

Page 23: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Inference

query

database

Page 24: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Inference

query

databasePS

Page 25: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Inference

query

databaseP

query

S

✓ ✓

Page 26: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Inference

query

database

Classical problem in statistical theory: show that and are close under constraints on

P

query

S

S

Page 27: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Privacy and Inference

query

database

query

privatizeddatabase

Q

The privacy-meets-inference problem: show that and are close under constraints on and on Q

query

S

P

S✓

Page 28: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Local Privacy

Private

Page 29: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Local Privacy

PrivateChannel

Individuals with private dataEstimator

Page 30: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

-private Minimax risk

Private Minimax Risk

Best -private channel

Minimax risk under privacy constraint

• Parameter of distribution• Family of distributions• Loss measuring error• Family of private channels

Page 31: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Proportions =

Vignette: Private Mean Estimation

Example: estimate reasons for hospital visitsPatients admitted to hospital for substance abuseEstimate prevalence of different substances

1 Alcohol1 Cocaine0 Heroin0 Cannabis0 LSD0 Amphetamines

= .45= .32= .16= .20= .00= .02

Page 32: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Proposition:

Vignette: Mean Estimation

Consider estimation of mean , witherrors measured in -norm, for

Minimax rate

(achieved by sample mean)

Page 33: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Proposition:

Private minimax rate for

Vignette: Mean Estimation

Consider estimation of mean , witherrors measured in -norm, for

Effective sample sizeNote:

Page 34: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

• Fixed-design regression• Convex risk minimization• Multinomial estimation• Nonparametric density estimation

• Almost always, the effective sample size reduction is:

Additional Examples

Page 35: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Optimal mechanism?

Non-privateobservation

Idea 1: add independent noise(e.g. Laplace mechanism)

Problem: magnitude much too large(this is unavoidable: provably sub-optimal)

[Dwork et al. 06]

Page 36: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Optimal mechanism

Non-privateobservation

Page 37: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Optimal mechanism

Non-privateobservation

View 1 View 2

• Draw uniformly in

(closer: 3 overlap) (farther: 2 overlap)

Page 38: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Optimal mechanism

Non-privateobservation

View 1 View 2

• Draw uniformly in

• With probability

choose closer of and to

(closer: 3 overlap) (farther: 2 overlap)

• otherwise, choose farther

Page 39: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Empirical evidence

Estimate proportion of emergency room visits involving different substances

Data source: Drug Abuse

Warning NetworkSample size n

Page 40: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Computation and Inference

• How does inferential quality trade off against classical computational resources such as time and space?

Page 41: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Computation and Inference

• How does inferential quality trade off against classical computational resources such as time and space?

• Hard!

Page 42: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Computation and Inference: Mechanisms and Bounds

• Tradeoffs via convex relaxations– linking runtime to convex geometry and risk to convex

geometry• Tradeoffs via concurrency control

– optimistic concurrency control • Bounds via optimization oracles

– number of accesses to a gradient as a surrogate for computation

• Bounds via communication complexity• Tradeoffs via subsampling

– bag of little bootstraps, variational consensus Monte Carlo

Page 43: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Part II: Variational, Hamiltonian and Symplectic Perspectives on Acceleration

with Andre Wibisono, Ashia Wilson and Michael Betancourt

Page 44: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

The Important Role of Optimization

• Optimization has been playing an increasingly important role in Data Science

Page 45: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

The Important Role of Optimization

• Optimization has been playing an increasingly important role in Data Science

• It not only supplies algorithms

Page 46: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

The Important Role of Optimization

• Optimization has been playing an increasingly important role in Data Science

• It not only supplies algorithms• But it also supplies lower bounds, and thereby

fundamental understanding

Page 47: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

The Important Role of Optimization

• Optimization has been playing an increasingly important role in Data Science

• It not only supplies algorithms• But it also supplies lower bounds, and thereby

fundamental understanding

• But, perhaps surprisingly, optimization is still an immature field, and open problems abound

Page 48: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Interplay between Differentiation and Integration

• The 300-yr-old fields: Physics, Statistics– cf. Lagrange/Hamilton, Laplace expansions, saddlepoint

expansions• The numerical disciplines

– e.g.,. finite elements, Monte Carlo

Page 49: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Interplay between Differentiation and Integration

• The 300-yr-old fields: Physics, Statistics– cf. Lagrange/Hamilton, Laplace expansions, saddlepoint

expansions• The numerical disciplines

– e.g.,. finite elements, Monte Carlo• Optimization?

Page 50: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Interplay between Differentiation and Integration

• The 300-yr-old fields: Physics, Statistics– cf. Lagrange/Hamilton, Laplace expansions, saddlepoint

expansions• The numerical disciplines

– e.g.,. finite elements, Monte Carlo• Optimization?

– to date, almost entirely focused on differentiation

Page 51: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Accelerated gradient descent

Setting: Unconstrained convex optimization

minx∈Rd

f (x)

I Classical gradient descent:

xk+1 = xk − β∇f (xk)

obtains a convergence rate of O(1/k)

I Accelerated gradient descent:

yk+1 = xk − β∇f (xk)

xk+1 = (1− λk)yk+1 + λkyk

obtains the (optimal) convergence rate of O(1/k2)

Page 52: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Accelerated methods: Continuous time perspective

I Gradient descent is discretization of gradient flow

Xt = −∇f (Xt)

(and mirror descent is discretization of natural gradient flow)

I Su, Boyd, Candes ’14: Continuous time limit of acceleratedgradient descent is a second-order ODE

Xt +3

tXt +∇f (Xt) = 0

I These ODEs are obtained by taking continuous time limits. Isthere a deeper generative mechanism?

Our work: A general variational approach to acceleration

A systematic discretization methodology

Page 53: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Bregman LagrangianDefine the Bregman Lagrangian:

L(x , x , t) = eγt+αt

(Dh(x + e−αt x , x)− eβt f (x)

)I Function of position x , velocity x , and time t

I Dh(y , x) = h(y)− h(x)− 〈∇h(x), y − x〉is the Bregman divergence

I h is the convex distance-generating function

I f is the convex objective function

I αt , βt , γt ∈ R are arbitrary smooth functions

I In Euclidean setting, simplifies to damped

Lagrangian

x y

h(x)

h(y)

Dh(y, x)

Ideal scaling conditions:

βt ≤ eαt

γt = eαt

Page 54: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Bregman Lagrangian

L(x , x , t) = eγt+αt

(Dh(x + e−αt x , x)− eβt f (x)

)

Variational problem over curves:

minX

∫L(Xt , Xt , t) dt

t

x

Optimal curve is characterized by Euler-Lagrange equation:

d

dt

{∂L∂x

(Xt , Xt , t)

}=∂L∂x

(Xt , Xt , t)

E-L equation for Bregman Lagrangian under ideal scaling:

Xt + (eαt − αt)Xt + e2αt+βt[∇2h(Xt + e−αt Xt)

]−1∇f (Xt) = 0

Page 55: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Bregman Lagrangian

L(x , x , t) = eγt+αt

(Dh(x + e−αt x , x)− eβt f (x)

)

Variational problem over curves:

minX

∫L(Xt , Xt , t) dt

t

x

Optimal curve is characterized by Euler-Lagrange equation:

d

dt

{∂L∂x

(Xt , Xt , t)

}=∂L∂x

(Xt , Xt , t)

E-L equation for Bregman Lagrangian under ideal scaling:

Xt + (eαt − αt)Xt + e2αt+βt[∇2h(Xt + e−αt Xt)

]−1∇f (Xt) = 0

Page 56: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

General convergence rate

TheoremTheorem Under ideal scaling, the E-L equation has convergencerate

f (Xt)− f (x∗) ≤ O(e−βt )

Proof. Exhibit a Lyapunov function for the dynamics:

Et = Dh

(x∗, Xt + e−αt Xt

)+ eβt (f (Xt)− f (x∗))

Et = −eαt+βtDf (x∗,Xt) + (βt − eαt )eβt (f (Xt)− f (x∗)) ≤ 0

Note: Only requires convexity and differentiability of f , h

Page 57: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Naive discretization doesn’t work

xk+1 =p

k + pzk +

k

k + pxk

zk = arg minz

{Cpk(p−1)〈∇f (xk), z〉+

1

εDh(z , zk−1)

}

Cannot obtain a convergence guarantee, and empirically unstable

-2 -1 0 1 2-2

-1

0

1

2

k0 20 40 60 80 100 120 140 160

f(x

k)

0

1

2

3

4

5

6

7

8

Page 58: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Mysteries

• Why can’t we discretize the dynamics when we are using exponentially fast clocks?

• What happens when we arrive at a clock speed that we can discretize?

• How do we discretize once it’s possible?

Page 59: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Mysteries

• Why can’t we discretize the dynamics when we areusing exponentially fast clocks?

• What happens when we arrive at a clock speed thatwe can discretize?

• How do we discretize once it’s possible?

• The answers are to be found in symplecticintegration

Page 60: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Symplectic Integration

•  Consider discretizing a system of differential equations obtained from physical principles

•  Solutions of the differential equations generally conserve various quantities (energy, momentum, volumes in phase space)

•  Is it possible to find discretizations whose solutions exactly conserve these same quantities?

•  Yes! –  from a long line of research initiated by Jacobi, Hamilton,

Poincare’ and others

Page 61: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Towards A Symplectic Perspective • We’ve discussed discretization of Lagrangian-based

dynamics• Discretization of Lagrangian dynamics is often fragile

and requires small step sizes• We can build more robust solutions by taking a Legendre

transform and considering a Hamiltonian formalism:

Page 62: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Symplectic Integration of Bregman Hamiltonian

Page 63: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Symplectic vs Nesterov

10-8

10-4

100

104

1 10 100 1000 10000

Nesterov

Symplectic

f(x)

Iterations

p = 2, N = 2, C = 0.0625, ε = 0.1

Page 64: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Symplectic vs Nesterov

10-8

10-4

100

104

1 10 100 1000 10000

NesterovSymplectic

f(x)

Iterations

p = 2, N = 2, C = 0.0625, ε = 0.25

Page 65: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Discussion• Data and inferential problems will be everywhere in

computer science, and will fundamentally change the field

• Many conceptual and mathematical challenges arising in taking this effort seriously, in addition to systems challenges and “outreach” challenges

• Facing these challenges will require a rapprochement between computational thinking and inferential thinking

• This effort is just beginning!

Page 66: On Computational Thinking, Inferential Thinking and Data ... · computational thinking and inferential thinking • Computational thinking means (inter alia) – abstraction, modularity,

Reference

•  Wibisono, A., Wilson, A. and Jordan, M. I. (2016). A variational perspective on accelerated methods in optimization. Proceedings of the National Academy of Sciences, 133, E7351-E7358.