how much restricted isometry is needed in nonconvex matrix ... › media › slides › nips ›...
TRANSCRIPT
![Page 1: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/1.jpg)
How Much Restricted Isometry is Needed in
Nonconvex Matrix Recovery?
1
Cédric
Josz
Javad
Lavaei
Somayeh
Sojoudi
Richard Y.
Zhang
Neural Information Processing Systems 2018Wed Dec 5th 5 - 7 PM @ Room 210 & 230 AB
Poster #46
![Page 2: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/2.jpg)
2
Nonconvex matrix recovery (Burer & Monteiro 2003)
Recommendation
engines
Phase retrieval
Power system
state estimation
Cluster
analysis
![Page 3: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/3.jpg)
3
Nonconvex matrix recovery (Burer & Monteiro 2003)
1. Express low-rank matrix as product of factors
table of movie ratingsmovie genres
user preferences
![Page 4: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/4.jpg)
4
Nonconvex matrix recovery (Burer & Monteiro 2003)
2. Minimize least-squares loss of linear model
specific
table elements
known
ratingstable of movie ratings
![Page 5: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/5.jpg)
5
Spurious local minima
Global minimum
X = UUT
0
![Page 6: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/6.jpg)
6
Exact recovery guarantee (Bhojanapalli et al. 2016)
δ-Restricted isometry property (δ-RIP)
If δ < 1/5, then no spurious local minima.
See also (Ge et al. 2017; Li & Tang 2017; Zhu et al. 2017)
![Page 7: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/7.jpg)
7
Local search is guaranteed to succeed.
Exact recovery guarantee (Bhojanapalli et al. 2016)
See also (Ge et al. 2017; Li & Tang 2017; Zhu et al. 2017)
δ-Restricted isometry property (δ-RIP)
If δ < 1/5, then no spurious local minima.
![Page 8: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/8.jpg)
8
δ-Concentration inequality
If δ very small, then no spurious local minima.
Exact recovery guarantee (Ge et al. 2016)
See also (Ge et al. 2015; 2017; Sun et al. 2015; 2016; Park et al. 2017; etc.)
Similar idea drives many proofs
![Page 9: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/9.jpg)
9
If δ < 1/5, then no spurious local min.
![Page 10: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/10.jpg)
10
If δ < 1/5, then no spurious local min.
=1.0
=1.0
![Page 11: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/11.jpg)
11
If δ < 1/5, then no spurious local min.
=1.0
=1.0
< 1/5
< 1/5
![Page 12: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/12.jpg)
12
If δ < 1/5, then no spurious local min.
Preserve lengths with <10% distortion
=1.0
=1.0
<1.1
>0.9
< 1/5
< 1/5
![Page 13: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/13.jpg)
13
Can this be significantly improved?
δ0 11/5
Good ???
NO
• Problem hard.
• Specific to algorithm.
• Proof idea is limited.
Yes
• Problem easy.
• Agnostic to algorithm.
• Proof idea is powerful.
If δ < 1/5, then no spurious local min.
![Page 14: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/14.jpg)
14
Previous attempts all stuck at 1/5 (Bhojanapalli et al. 2016) (Ge et al. 2017) (Li & Tang 2017) (Zhu
et al. 2017) etc.
Can this be significantly improved?
δ0 11/5
Good ???
NO
• Problem hard.
• Specific to algorithm.
• Proof idea is limited.
Yes
• Problem easy.
• Agnostic to algorithm.
• Proof idea is powerful.
If δ < 1/5, then no spurious local min.
![Page 15: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/15.jpg)
15
Can this be significantly improved?
δ0 11/5
Good Bad
1/2
NO
• Problem hard.
• Specific to algorithm.
• Proof idea is limited.
If δ ≥ 1/2, many counterexamples.
If δ < 1/5, then no spurious local min.
Contribution 1.
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
![Page 16: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/16.jpg)
16
If δ < 1/2, then no spurious local min.
δ0 11/2
Good Bad
If δ ≥ 1/2, many counterexamples.
Contribution 2.
Let rank r = 1.
Zhang, Sojoudi, Lavaei. Submitted to JMLR (2018)
![Page 17: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/17.jpg)
Counterexample with δ = 1/2
17
Satisfies ½-RIP.
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
Zhang, Sojoudi, Lavaei, Submitted to JMLR (2018)
![Page 18: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/18.jpg)
Counterexample with δ = 1/2
18
Satisfies ½-RIP. Ground truth
z = (1,0)
Spurious local min
x = (0,1/√2)
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
Zhang, Sojoudi, Lavaei, Submitted to JMLR (2018)
![Page 19: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/19.jpg)
Counterexample with δ = 1/2
• 100,000 trials w/ SGD
• 87,947 successful
• 12% failure rate
19
Satisfies ½-RIP. Ground truth
z = (1,0)
Spurious local min
x = (0,1/√2)
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
Zhang, Sojoudi, Lavaei, Submitted to JMLR (2018)
![Page 20: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/20.jpg)
Counterexample with δ = 1/2
• 100,000 trials w/ SGD
• 87,947 successful
• 12% failure rate
20
Satisfies ½-RIP. Ground truth
z = (1,0)
Spurious local min
x = (0,1/√2)
Generalization to
arbitrary rank-1
ground truth
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
Zhang, Sojoudi, Lavaei, Submitted to JMLR (2018)
![Page 21: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/21.jpg)
Proof idea. Counterexamples via convex optimization
21Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
Key insight. Relax into a semidefinite program
![Page 22: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/22.jpg)
Main Result 1. Counterexamples are almost everywhere
22
Theorem 1 (Zhang, Josz, Sojoudi, Lavaei 2018).
Given x, z not colinear and nonzero,
there exists a counterexample that
• satisfy δ-RIP and 1/2 ≤ δ < 1
• has z as ground truth
• has x as spurious local min.
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
![Page 23: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/23.jpg)
Main Result 1. Counterexamples are almost everywhere
23
Theorem 1 (Zhang, Josz, Sojoudi, Lavaei 2018).
Given x, z not colinear and nonzero,
there exists a counterexample that
• satisfy δ-RIP and 1/2 ≤ δ < 1
• has z as ground truth
• has x as spurious local min.
Take-away.
If δ-RIP with δ ≥ 1/2, then
expect spurious local minima.
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
![Page 24: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/24.jpg)
24
Conjecture (Zhang, Josz, Sojoudi, Lavaei 2018).
If δ-RIP with δ < 1/2, then no spurious local min.
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
![Page 25: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/25.jpg)
Main Result 2. Sharp RIP-based guarantee
25Zhang, Sojoudi, Lavaei. Submitted to JMLR (2018)
Theorem 2 (Zhang, Sojoudi, Lavaei 2018).
If δ-RIP with δ < 1/2 and r =1, then no spurious local
min.
Proof for rank-1 case
![Page 26: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/26.jpg)
26
Ongoing work.
Generalization to rank-r
Theorem 2 (Zhang, Sojoudi, Lavaei 2018).
If δ-RIP with δ < 1/2 and r =1, then no spurious local
min.
![Page 27: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/27.jpg)
27
Practical implications?
δ-RIP with 1/2 ≤ δ < 1
![Page 28: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/28.jpg)
28
“Engineered”
spurious local
minimum
Global minimum
0
xbad
xgood
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
![Page 29: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/29.jpg)
29
0
xbad
xgood
2. Start SGD at
3. Make 10k SGD steps, measure error
error = | xfinal - xgood |
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
xinit = (1-γ) xbad + γ Gaussian
1. Select γ in [0,1]
![Page 30: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/30.jpg)
max
95%
median
5%
min
error
(1k trials)
xinit = xbad xinit ~ Gaussian
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
γ
Example 1
100% success rate
![Page 31: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/31.jpg)
Example 1
100% success rate
max
95%
median
5%
min
error
(1k trials)
xinit = xbad xinit ~ Gaussian
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
γ
100%
failure
![Page 32: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/32.jpg)
Example 1
100% success rate
max
95%
median
5%
min
error
(1k trials)
xinit = xbad xinit ~ Gaussian
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
γ
100%
failure
100%
success
![Page 33: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/33.jpg)
max95%
median
5%
min
error
(1k trials)
γxinit = xbadxinit = w
Zhang, Josz, Sojoudi, Lavaei, NeurIPS (2018)
Example 2
<95% success rate
100%
failure
>5%
failure
![Page 34: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/34.jpg)
34
Practical implications?
δ-RIP with 1/2 ≤ δ < 1
100%
success
no
spurious
local min
spurious
local min> 0%
failure
Limitations of “no spurious local min” guarantees
![Page 35: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/35.jpg)
δ0 11/5
Good Bad
1/2
How Much Restricted Isometry is Needed in
Nonconvex Matrix Recovery?
35
If δ ≥ 1/2, many counterexamples.
R.Y. Zhang, C. Josz, S. Sojoudi, J. Lavaei, NeurIPS (2018)
Wed Dec 5th 5 - 7 PM @ Room 210 & 230 AB
Poster #46
![Page 36: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/36.jpg)
How Much Restricted Isometry is Needed in
Nonconvex Matrix Recovery?
36
If δ < 1/2, then no spurious local min (?)
δ0 11/2
Good Bad
If δ ≥ 1/2, many counterexamples.
R.Y. Zhang, C. Josz, S. Sojoudi, J. Lavaei, NeurIPS (2018)
Wed Dec 5th 5 - 7 PM @ Room 210 & 230 AB
Poster #46
![Page 37: How Much Restricted Isometry is Needed in Nonconvex Matrix ... › media › Slides › nips › 2018 › 517cd(05-15... · How Much Restricted Isometry is Needed in Nonconvex Matrix](https://reader033.vdocument.in/reader033/viewer/2022060209/5f04784d7e708231d40e2087/html5/thumbnails/37.jpg)
How Much Restricted Isometry is Needed in
Nonconvex Matrix Recovery?
37
If δ < 1/2, then no spurious local min (?)
δ0 11/2
Good Bad
If δ ≥ 1/2, many counterexamples.
R.Y. Zhang, C. Josz, S. Sojoudi, J. Lavaei, NeurIPS (2018)
Wed Dec 5th 5 - 7 PM @ Room 210 & 230 AB
Poster #46
Limitations of “no spurious local min” guarantees