Upload File

matrix completion with als - stanford...

  • Home
  • Documents
  • Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD
17
Reza Zadeh Matrix Completion with ALS @Reza_Zadeh | http://reza-zadeh.com

Upload: others

Post on 04-Dec-2020

1 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Reza Zadeh

Matrix Completion with ALS

@Reza_Zadeh | http://reza-zadeh.com

Page 2: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Collaborative Filtering Goal: predict users’ movie ratings based on past ratings of other movies

R  =  

1  ?  ?  4  5  ?  3  ?  ?  3  5  ?  ?  3  5  ?  5  ?  ?  ?  1  4  ?  ?  ?  ?  2  ?  

Movies  

Users  

Page 3: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Don’t mistake this with SVD.

Both are matrix factorizations, however SVD cannot handle missing entries.

Page 4: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Optimization problem

Page 5: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Alternating Least Squares 1.  Start with random A1, B1 2.  Solve for A2 to minimize ||R – A2B1

T|| 3.  Solve for B2 to minimize ||R – A2B2

T|| 4.  Repeat until convergence

R A = BT

Page 6: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Attempt 1: Broadcast All

Page 7: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Attempt 2: Data Parallel

Page 8: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Attempt 3: Fully Parallel

Page 9: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

ALS on Spark Cache 2 copies of R in memory, one partitioned by rows and one by columns Keep A & B partitioned in corresponding way Operate on blocks to lower communication

R A = BT

Matei Zaharia,"Joey Gonzales,"Virginia Smith

Page 10: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

ALS Results

4208

481 297 0

1000

2000

3000

4000

5000

Tota

l Tim

e (s)

Mahout / Hadoop Spark (Scala) GraphLab (C++)

Page 11: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

State of the Spark ecosystem

Page 12: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Most active open source community in big data

200+ developers, 50+ companies contributing

Spark Community

Giraph Storm

0

50

100

150

Contributors in past year

Page 13: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Project Activity M

apRe

duce

YA

RN HD

FS

Stor

m

Spar

k

0

200

400

600

800

1000

1200

1400

1600

Map

Redu

ce

YARN

HDFS

St

orm

Spar

k

0

50000

100000

150000

200000

250000

300000

350000

Commits Lines of Code Changed

Activity in past 6 months

Page 14: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Continuing Growth

source: ohloh.net

Contributors per month to Spark

Page 15: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Conclusions

Page 16: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Spark and Research Spark has all its roots in research, so we hope to keep incorporating new ideas!

Page 17: Matrix Completion with ALS - Stanford Universitystanford.edu/~rezab/slides/sparksummit2015/lec5.pdf · Don’t mistake this with SVD. Both are matrix factorizations, however SVD

Conclusion Data flow engines are becoming an important platform for numerical algorithms While early models like MapReduce were inefficient, new ones like Spark close this gap More info: spark.apache.org


Multimedia Databases SVD II. SVD - Detailed outline Motivation Definition - properties Interpretation Complexity Case studies SVD properties More case

Top Productivity Working Hacks by Jan Rezab

6 The SVD and Image Compression - BYU ACME · 6 The SVD and Image Compression Lab Objective: The Singular Value Decomposition (SVD) is an incredibly useful matrix factor-ization that

Factorbird: a Parameter Server Approach to Distributed ...stanford.edu/~rezab/slides/factorbird_slides_shannon.pdf · Factorbird: a Parameter Server Approach to Distributed Matrix

Lessons from the Netflix prize - cse.iitk.ac.in prize.pdf · The road to the Netflix Prize •SVD •SVD++ •Temporal SVD •Temporal SVD++ •Highly recommended reading: Madrigal

Social Trends & Data - Jan Rezab at Engage Prague 2016

Distributing Matrix Computations with Spark MLlibstanford.edu/~rezab/slides/reza_mllib_maryland.pdf · 2014-10-22 · Distributing Matrix Computations with Spark MLlib. A General

Matrix Computations and Neural Networks in Sparkstanford.edu/~rezab/slides/stratany2015.pdf(inevitable at scale) » Even worse: stragglers (node not failed, but slow) ... Spark Computing

4. Matrix Methods for Analysis of Structure in Data …dennis/552_Notes_4.pdf · 2016-02-21 · ATM 552 Notes: Matrix Methods: EOF, SVD, ETC. D.L.Hartmann Page 68 Copyright Dennis

Matrix Completion and Large-scale SVD Computationsweb.stanford.edu/~hastie/TALKS/Iceland_2015.pdf · Matrix Completion and Large-scale SVD Computations Trevor Hastie Stanford Statistics

Singular Value Decomposition (SVD)ajitvr/CS763_Spring2017/SVD.pdfSingular value Decomposition •For any m x n real matrix A, the SVD consists of matrices U,S,V which are always real

Learning with Matrix Parameters - Stanford Universitystanford.edu/~rezab/nips2013workshop/slides/nati_slides.pdfLearning with Matrices •Matrices occur explicitly or implicitly in

Singular Value Decomposition - Computer Graphicsgraphics.stanford.edu/courses/cs205a/assets/lecture_slides/svd.pdf · Motivation SVD Pseudoinverses Low-Rank Approx. Matrix Norms Regularization

Faster Matrix Completion Using Randomized SVD · runtime benefit. Besides, as sparse matrix is processed in matrix completion, special technique should be devised to make the randomized

Bi-cross-validation of the SVD and the non-negative matrix ...statweb.stanford.edu/~owen/reports/cvsvd.pdfcompares cross-validation and the bootstrap. Hoff (2007) treats the SVD under

Matrix Completion and Low-Rank SVD via Fast Alternating ...rezab/papers/fastals.pdf · Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares Trevor Hastie Rahul Mazumder

A Singularly Valuable Decomposition: The SVD of a Matrixreichel/courses/intr.num.comp.1/fall11/lecture7/svd.pdf · A Singularly Valuable Decomposition: The SVD of a Matrix Dan Kalman

SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method

Hierarchical Decomposition of 2D/3D Images, Based on SVD …...through Hierarchical SVD (HSVD), based on the SVD for a matrix of size 2 2. In the paper are given the HSVD algorithms

Faster Matrix Completion Using Randomized SVD · The problem of matrix completion, or estimating missing values in a matrix, occurs in many areas of engineering and applied science

Lecture 15 Symmetric matrices, quadratic forms, matrix norm, and SVD

Singular Value Decomposition and Applicationshcso/EE4016_svd.pdf · 2020. 3. 27. · Singular Value Decomposition (SVD) SVD is a useful tool to decompose a matrix : (1) where . contains

Image Compression using SVD and DCT - University … · Image Compression using SVD and DCT Math 2270-003 Spring 2012 ... Matrix? Matrix? Notes Image File = Header + RGB / GrayScale

Chapter 11 Applications of SVD and Pseudo-inversescis515/cis515-11-sl11.pdf · 412 CHAPTER 11. APPLICATIONS OF SVD AND PSEUDO-INVERSES Note that AA is a symmetric matrix, one of the

A Review of a Singularly Valuable Decomposition: The SVD ...morrow/336_13/papers/reed.pdf · A Review of a Singularly Valuable Decomposition: The SVD of a Matrix Reed Tillotson June

Matrix Completion and Low-Rank SVD via Fast Alternating

Singular Value Decomposition (matrix factorization) · 2020. 5. 6. · Singular Value Decomposition The SVD is a factorization of a !×#matrix into $=&’(! ... Example: Image compression

Matrix Completion and Large-scale SVD Computationshastie/TALKS/SVD_hastie.pdf · Matrix Completion and Large-scale SVD Computations Trevor Hastie Stanford Statistics joint with Rahul

Multimedia Databases SVD II. Optimality of SVD Def: The Frobenius norm of a n x m matrix M is (reminder) The rank of a matrix M is the number of independent

A Singularly Valuable Decomposition: The SVD of a Matrixlerman/math5467/svd.pdf · A Singularly Valuable Decomposition: The SVD of a Matrix Dan Kalman The American University Washington,

#EngageBali 2016: Jan Rezab, Founder of Socialbakers

A FAST ALGORITHM FOR STRUCTURED LOW-RANK MATRIX … · 2016-01-29 · value decomposition (SVD) of the dense large-scale matrix at each iteration. Several strategies have been introduced

Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear

Spark and Matrix Factorization - Stanford Universitystanford.edu/~rezab/slides/reza_codeneuro.pdf · Spark Computing Engine ... » The world only lets you make make RDDs such that

Matrix Completion and Low-Rank SVD via Fast Alternating Least … · 2020-04-29 · Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares Trevor Hastie [email protected]