Swarat ChaudhuriRice University

Biplab DekaRakesh Kumar

UIUC

Principled Approximations in Probabilistic Programming

The computing stack (approximation)

AlgorithmsCompiler and runtime

Architecture

The APPROX view: with probabilitiesand approximations!

The computing stack

AlgorithmsCompiler and runtimeArchitecture

The APPROX view: with probabilitiesand approximations!

Probabilistic program

Program synthesis

Hardware for sampling

DPMM Clustering

• In each iteration, update the cluster assignments of data points one at a time

• For each data point, compute the probability of it belonging to all existing clusters and an unseen cluster

• Sample from this probability distribution

Gaussian Dirichlet Process Mixture Model (DPMM)

Gibbs Sampling Algorithm

Cluster 3 Cluster 2

Cluster 1

Cluster 2

Cluster 1

Cluster 2

Cluster 1

New Cluster

Model

Sampling in Hardware

Prefix Sum Comparators

PRNG

Encoder

CLK

Prob

abili

ties

Sam

ple

32b 35b 1b

3b

32b

Idea: exploit errors!

Robustness to hardware faultsStuck-at Faults Transient Faults

Sampler

Clustering using

DPMM

Deka, Biplab. “On Fault Tolerance of Hardware Samplers”. Masters Thesis, University of Illinois at Urbana Champaign, 2014

Approximating compilers

Traditional compiler:• Input: Deterministic program• Goal: Executable semantically equivalent to source • Method: Syntax-guided translation

Approximating compiler:• Input: Probabilistic program• Goal:

– Satisfy basic boolean invariants– Minimize quantitative error

• Method: Program synthesis

Probabilistic programs

heightMan = Gaussian(177,64);heightWoman = Gaussian(164,64);assume(heightWoman > heightMan);return heightMan

Source: Tutorial on Infer.NET by John Winn and Tom Minka

Addition: assertions, angelic nondeterminism

More complex example: clustering

Probabilistic programming ++

• Random variables X: range over distributions

• Deterministic variables y– Either holes or temporaries

• Functions f(X1,..., Xk, y1,..., yk)– Can map random variables to deterministic ones

• Expectation, probability

• Assertions • Pareto-optimality goals

Example

X = Gaussian(??, 10);assume (X < 10);c = Pr(X > 0);assert (c > 0.7);minimize (c);

One synthesis algorithm in [CCS14]Based on probabilistic abstract interpretation

Hole

[CCS14] Chaudhuri, Clochard, Solar-Lezama. Bridging boolean and quantitative synthesis using smoothed proof search. POPL 2014.

Use in approximation

Holes = degree of approximation

Assertions = invariants, hard bounds

Declarative error minimization

Deterministic temporaries track resource

consumption

Synthesis = Compilation

Thank you!

Questions?