boaz barak – microsoft research partially based on joint work with jonathan kelner (mit) and david...

Boaz Barak – Microsoft Research

Partially based on joint work with Jonathan Kelner (MIT) and David Steurer (Cornell)

Sum of Squares Proofs andThe Quest Towards Optimal

Algorithms

SURGEON GENERAL’S WARNING:Talk contains only “pseudo theorems”

The space of algorithms is very rich

- a different algorithm for every problem

A pipe dream… One algorithm for all problems?

Answer 1: Yes for non-interesting reasons.

Theorem: There is a single universal time algorithm that solves every combinatorial search problem that has an algorithm.

Proof: Enumerate over all Turing machines and try them all..

Answer 2: Maybe there is a deeper reason..Few unifying principles underlie many algorithms.

Convexity, matroid structure, submodularity, algebraic identities, ..

Hope: An optimal meta algorithm:

if doesn’t solve some then no efficient algorithm can solve .

An efficient algorithm such that for a large class of problems,

Why care?

A pipe dream… One algorithm for all problems?

Unified approach to classifying easy vs. hard problems

• Hardness*/algorithmic results are often very challenging.

• Proving specific algorithm fails much easier than ruling out all algorithms.

• Understanding why problem easy/hard rather than laundry-list of results.

A generic “meta algorithm” for polynomial optimization [Shor’87,Nesterov’00,Parillo’00,Lasserre’01]

Underlying principle: An efficient way to prove that is to find a representation .

Algorithmic version of works related to Hilbert’s 17th problem [Artin 27,Krivine64,Stengle74]

Used in many applications including quantum information theory, control theory, automated theorem proving, game theory, and more…

Can it be an optimal meta algorithm?

Image credit: Chakraborty et al

The Sum of Squares Algorithm

Generalizes many known algorithms.

Empirically seems to work well. Theoretical analysis lags far behind.

See also Laurent’s talk

This talk:• Dual views of SOS alg:Positivstellensatz/SOS proofs “Pseudo distributions”

• Example application: Dictionary Learning / Sparse Coding

• Relation to Khot’s Unique Games Conjecture

Image credits: Georghiades et al, Mairal et al, Singer, Kindler et al

Polynomial EquationsProblem: Given set of polynomial equations in , find s.t. for all or

prove that none exists.

Extremely general framework:

All poly’s of degree 1: Linear programming

Single quadratic: Least squares / eigenvalue problem

Poly’s of degree >1: Captures a great many problems, some NP hard

Can similarly encode many other problems, including SAT, 3COL, Max-Cut, etc…

for

𝑥1 𝑥2=𝑥1𝑥7=𝑥1𝑥9=𝑥2 𝑥3…=𝑥11𝑥12=0𝑥1+𝑥2+…+𝑥12=5

Example: If is an -vertex graph, the equations for , and for all , are equivalent to the graph containing an independent set (aka stable set) of size .

1

9

2

310

4

5

8

6

712

11

Given parameter , poly eq’s , either find a “degree pseudo-solution” to , or prove that none exists.

Since problem is NP hard, sometimes need to converge.

Running time:

Relaxation of actual solution: degree = “solution quality”: a degree pseudo-solution is also degree pseudo-solution. As , converges* to actual solution.

Empirically often converges much faster, still not very well understood.

Hope: For many natural classes of polynomial equations, if there is some poly-time algorithm to solve , then SOS converges for .

Some evidence and intuition, still not at the level of a precise conjecture.

Next: What is a “proof” and what is a “pseudo-solution”

Problem: Given set of polynomial equations in , find s.t. for all or prove that none exists.

SOS Algorithm – high level view:

SOS ProofsAn SOS proof deduces polynomial inequalities using the following rules:

For , we write if there is (static*) SOS proof that only uses polynomials of degree (a.k.a. degree SOS proof).

SOS Theorem [Shor,Nesterov,Parillo,Lasserre]: If then the proof can be found in time using semidefinite programming.

⊩𝑃2≥0{𝑃 ≥0 ,𝑄≥0 }⊩ {𝑃+𝑄≥0 ,𝑃𝑄≥0 }

Positivstellensatz [Krivine’64,Stengle’74]: If implies then this has a deg SOS proof for some (sufficiently large) .

SOS proofs surprisingly powerful:

• Capture many standard tools such as Cauchy-Schwarz, Hölder, etc.. and even more advanced notions (e.g., Isoperimetric results on Boolean cube)

• Only examples of assertions that robustly* require large degree to prove are non-constructive, shown using probabilistic method.

What is a “pseudo-solution”?

Problem: Given set of polynomial equations in , find s.t. for all or prove that none exists.

Intuitively: Degree pseudo-solution attempts to our computational knowledge on the solutions of : how much information about the solution can we gather in time

As grows the computational knowledge converges to true solution.Next:

1) Detour: defining statistical knowledge via distributions

2) Definition of pseudo solutions.

Given parameter , poly eq’s , either find a “degree pseudo-solution” to , or prove that none exists.

SOS Algorithm – high level view:

Find degree SOS proof that

Detour: Knowledge as a distributionSuppose I pick some and gradually reveal to you information about .

𝑥

Can capture your knowledge of as distribution over the potential values of from your perspective.

Initially, you don’t know anything, and so your knowledge is modeled by the uniform distribution over

As you learn more information, you adjust your distribution accordingly.

Detour: Knowledge as a distributionSuppose I pick some and gradually reveal to you information about .

Can capture your knowledge of as distribution over the potential values of from your perspective.

Initially, you don’t know anything, and so your knowledge is modeled by the uniform distribution over

As you learn more information, you adjust your distribution accordingly.

“”

“”

“”

𝑥

i.e., all information is known, but we are computationally bounded and can’t make all possible logical inferences from it.

Can we do the same for computational knowledge?

Computational knowledge as a pseudo-distributionSuppose I give all information about but in computationally hard to

decode form

𝑥

Example: You get graph s.t. is unique max independent set of .

for

𝑥1 𝑥2=𝑥1𝑥7=…=𝑥11 𝑥12=0𝑥1+𝑥2+…+𝑥12=5

1

9

2

310

45

8

6

712

11

Want “distribution-like object” to model knowledge of after computation steps.

is a degree- pseudo distribution consistent with if:

• The value is defined for every deg poly

• For every , if then

• Using linearity, can be specified by numbers.

Notes: • is not an actual distribution: is undefined if is not deg poly

An SOS proof deduces polynomial inequalities using the following rules:

For , we write if there is a (static*) SOS proof that only uses polynomials of degree (a.k.a. degree SOS proof).

⊩{𝑃2≥0 }{𝑃 ≥0 ,𝑄≥0 }⊩ {𝑃+𝑄≥0 ,𝑃𝑄≥0 }

is a degree- pseudo distribution consistent with if:

• The value is defined for every deg poly

• For every , if then

SOS Theorem: there’s an time algorithm that on input equations outputs either:(1) A derivation of or(2) A degree pseudo distribution consistent with

Goal: Given examples of form , recover

Find the “right” representation of observed data

Previous best (rigorous) results: [Spielman-Wang-Wright ’12, Arora-Moitra-Ge ‘13, Agrawal-Anandkumar-Jain-Netrapalli-Tandon ‘13]

Theorem [B-Kelner-Steurer’14]: is sufficient* (even in non-independent, overcomplete, coherent case)

Let set of unit vectors.

LOTS of work: important primitive in Machine Learning, Vision, Neuroscience...

Example Application: Dictionary Learning / Sparse Coding

[Olhausen-Field ’96]

[Mairal-Bach-Ponce-Sapiro ’10]

(Use minimization, incoherence; see also Candes, Gilbert talks)

Goal: Given examples of form , recover

Proof overview: Using examples find a polynomial s.t.

Let set of unit vectors.

Example Application: Dictionary Learning / Sparse Coding

Theorem [B-Kelner-Steurer’14]: is sufficient*

(have no control over local maxima)

’s global maxima on sphere are (approximately) (*)

can recover ’s from a pseudo-distribution over such maxima.

Proof of (*) uses low degree SOS arguments

Our choice for : empirical variance of

Can show (*) using Hölder-type inequalities.

Unique Games ConjectureConjecture [Khot ‘02]: Certain problem known as “Unique Games” is NP

hard.(morally) equivalent formulation [Raghavendra-Steurer ‘08]: Given

, NP hard to distinguish between:

Many implications to complexity: Implies hardness results problems across many domains including constraint satisfaction, cut and routing, scheduling, algebra and more. If true then yields optimality of many classical algorithms (e.g. Grothendieck, Cheeger-

Alon-Milman, Geomans-Williamson, etc..).Fascinating connections to areas including probability, geometry, metric embeddings, social choice theory, etc.. E.g. Khot/Arora/O’Donnell’s talks ; surveys [Khot, ‘10 ‘10 ‘14], [Trevisan ‘12]

vs.

The ₩ question: Is the UGC true?

Main reasons to believe UGC:

Can’t refute it: Don’t know of an algorithm that solves it.

Want it to be true: Gives very clean picture of complexity landscape.

Algorithmic attacks on UGC:

“Basic semidefinite program”: Analog of Cheeger-Alon-Milman, Grothendieck,

Lovász function, Geomans-Williamson.

“Eigenspace enumeration”: Brute force search in top eigenspace of adj. matrix

SOS Algorithm: Generalizes both

Solves random instances …[Arora-Khot-Kolla-Steurer-Tulsiani-Vishnoi’05]

but not all instances [Khot-Vishnoi’04]

Solves KV instance[Kolla’10] [Arora-B-Steurer-’10]

..and every instance in subexp time[B-Gopalan-Håstad-Meka-Raghavendra-Steurer’12]

..but not much better

know an algorithm, don’t have proof

Solves KV+BGHMRS instances[B-Brandao-Harrow-Kelner-Steurer-Zhou ‘12]

[B-Kelner-Steurer ‘14].. candidate algorithm to solve all instances

sort of

Conclusions

• Sum of Squares is a powerful algorithmic framework that can yield strong results for the right problems.

• Has (yet to be explored) potential to resolve longstanding problems.

• Only beginning to build the tools to analyze it –for both upper bounds and lower bounds.

• Connection of

proof complexity pseudo-distributions approximation algorithms

might be useful beyond SOS.

• Personal prediction: will see much progress on this in the next few years.

Maybe talk on refuting UGC via SOS at ICM 2018?

boaz barak – microsoft research partially based on joint work with jonathan kelner (mit) and david...

Documents