sketching as a tool for numerical linear algebra (part 2)

Sketching as a Tool for Numerical LinearAlgebra

(Part 2)

David P. Woodruffpresented by Sepehr Assadi

o(n) Big Data Reading GroupUniversity of Pennsylvania

February, 2015

Sepehr Assadi (Penn) Sketching for Numerical Linear Algebra Big Data Reading Group 1 / 21

GoalNew survey by David Woodruff:

I Sketching as a Tool for Numerical Linear AlgebraTopics:

I Subspace EmbeddingsI Least Squares RegressionI Least Absolute Deviation RegressionI Low Rank ApproximationI Graph SparsificationI Sketching Lower Bounds


http://researcher.watson.ibm.com/researcher/files/us-dpwoodru/wNow.pdf

IntroductionYou have “Big” data!

I Computationally expensive to deal withI Excessive storage requirementI Hard to communicateI . . .


IntroductionYou have “Big” data!

I Computationally expensive to deal withI Excessive storage requirementI Hard to communicateI . . .

Summarize your dataI Sampling

F A representative subset of the dataI Sketching

F An aggregate summary of the whole data


ModelInput:

I matrix A ∈ Rn×d

I vector b ∈ Rn .Output: function F(A,b, . . .)

I e.g. least square regressionDifferent goals:

I Faster algorithmsI StreamingI Distributed


Linear SketchingInput:

I matrix A ∈ Rn×d

Let r n and S ∈ Rr×n be a random matrixLet S ·A be the sketchCompute F(S ·A) instead of F(A)


Linear Sketching (cont.)Pros:

I Compute on a r × d matrix instead of n × dI Smaller representation and faster computationI Linearity:

F S · (A + B) = S ·A + S ·BF We can compose linear sketches !

Cons:I F(S ·A) is an approximation of F(A)


Approximate `2-regressionInput:

I matrix A ∈ Rn×d (full column rank)I vector b ∈ Rn

I parameter 0 < ε < 1Output x ∈ Rd :

‖Ax− b‖2 ≤ (1 + ε) arg minx‖Ax− b‖2


Subspace EmbeddingDefinition (`2-subspace embedding)A (1± ε) `2-subspace embedding for a matrix A ∈ Rn×d is a matrixS for which for all x ∈ Rn

‖SAx‖22 = (1± ε) ‖Ax‖2

2

Actually subspace embedding for column space of A


Previous SessionOblivious `2-subspace embedding

I The distribution from which S is chosen is oblivious to AI One very common tool: Johnson-Lindenstrauss transform (JLT)I Immediately approximate `2-regression problem


TodayNon-oblivious `2-subspace embedding

I The distribution from which S is chosen depends on AI One very common tool: Leverage Score SamplingI Can still be used to approximate `2-regression problem


Leverage ScoresThin Singular Value Decomposition (SVD) of A:

I An×d = Un×d ·Σd×d ·Vd×dI U is an orthonormal basis of column space of A

Leverage Score of i-th row of A:

`i =∥∥∥U(i)

∥∥∥2

Properties:I Independent of the basis (property of the column space)I Forms a probability distribution (by simple normalization)I Let H = A(ATA)−1AT , then `2i = Hi,i


Leverage Score SamplingDefinition (SampleRescale(n, s, p))We define the procedure S = SampleRescale(n, s, p), ifSs×n = D ·Ω, where each row of Ω is a random basis vector in Rn

chosen according to the probability distribution p, and D is a diagonalmatrix where Di,i = 1/√pjs if ej is chosen for i-th row of Ω.

Leverage Score Sampling (p = LS-Sampling(A, β)):I p = (p1, . . . , pn) is a probability distribution satisfying

pi ≥ β · `2i /d, where `i is the i-th leverage score of An×dI Compute S = SampleRescale(n, s, p)I Return S ·A


Subspace Embedding via LS-SamplingTheoremLet s = Θ(d log d

βε2 ), S = SampleRescale(n, s, p) forp = LS-Sampling(A, β), and U be an orthonormal matrix of thecolumn space of A; then with probability 0.99, simultaneously for alli ∈ [d],

1− ε ≤ σ2(S ·U) ≤ 1 + ε

It immediately implies subspace embedding


Subspace Embedding via LS-Sampling (cont.)TheoremLet s = Θ(d log d

βε2 ), S = SampleRescale(n, s, p) forp = LS-Sampling(A, β), and U be an orthonormal matrix of thecolumn space of A; then with probability 0.99, simultaneously for alli ∈ [d],

1− ε ≤ σ2(S ·U) ≤ 1 + ε

Proof.Matrix Chernoff: Suppose X1, . . . ,Xs are independent copies ofsymmetric matrix X ∈ Rd×d with E[X] = 0, and ‖X‖ ≤ γ, and∥∥∥E[XTX]

∥∥∥ ≤ s2 and let W = 1s∑s

i=1 Xi ; then

Pr(‖W‖ > ε) ≤ 2d · exp(−sε2/(2s2 + 2γε/3)

)


Linear Regression via LS-SamplingTheoremLet s = Θ(d log d

βε2 ), S = SampleRescale(n, s, p) forp = LS-Sampling(A, β), and x = arg minx ‖SAx− Sb‖, then withprobability 0.99,



Linear Regression via LS-Sampling (cont.)Theorem (Approximate Matrix Multiplication)For an orthonormal matrix Cn×m, an arbitrary vector dn×1, andprobabilities p = (p1, . . . , pn) such that:

pk ≥β∣∣∣C(k)

∣∣∣2‖C‖F

let S = SampleRescale(n, s, p); then, with probability 0.99:

∥∥∥(SC)T (Sd)−CTd∥∥∥

F≤ O(

√1sβ ) ‖C‖F ‖d‖F

Warning: this statement is neither general nor precise!see [DKM06]


Linear Regression via LS-Sampling (cont.)TheoremLet s = Θ(d log d

βε2 ), S = SampleRescale(n, s, p) forp = LS-Sampling(A, β), and x = arg minx ‖SAx− Sb‖, then withprobability 0.99,


Proof.On the board.


Approximating Leverage ScoresComputing leverage scores is as hard as solving the regressionproblem!Can we approximate them?

I For β = 1/2, in time O(nd log n + d3) [DMIMW12]I Improved to O(nnz(A) log n + d3) [CW13]


Questions?


Kenneth L Clarkson and David P Woodruff.Low rank approximation and regression in input sparsity time.In Proceedings of the forty-fifth annual ACM symposium onTheory of computing, pages 81–90. ACM, 2013.

Petros Drineas, Ravi Kannan, and Michael W Mahoney.Fast monte carlo algorithms for matrices i: Approximating matrixmultiplication.SIAM Journal on Computing, 36(1):132–157, 2006.

Petros Drineas, Malik Magdon-Ismail, Michael W Mahoney, andDavid P Woodruff.Fast approximation of matrix coherence and statistical leverage.The Journal of Machine Learning Research, 13(1):3475–3506,2012.


sketching as a tool for numerical linear algebra (part 2)

Documents