computer vision: models, learning and inference convexity ...cv192/wiki.files/cv192...cv/ml people)...

Computer Vision: Models, Learning and Inference–

Convexity; Least Squares; Robustness; Some Statistics

Oren Freifeld and Ron Shapira-Weber

Computer Science, Ben-Gurion University

April 7, 2019

www.cs.bgu.ac.il/~cv192/ Needful Things (ver. 1.00) Apr 7, 2019 1 / 82

1 Convexity

2 Linear Model, Least Squares and Weighted Least Squares

3 Outliers, “Wrong” Assumptions, and Robustness

4 Some Statistics and M-Estimators

Convexity

Definition (convex function from I ⊂ R to R)

Let I ⊂ R be an interval.f : I → R is called convex if, ∀x1, x2 ∈ I, ∀t ∈ (0, 1),

f(tx1 + (1− t)x2) ≤ tf(x1) + (1− t)f(x2) .

In words: any chord connecting two values of f is always not below thegraph of f .

Example

f(x) = x2 is convex.

Example

f(x) = |x| is convex.

Convexity

Definition (strictly-convex function from I ⊂ R to R)

Let I ⊂ R be an interval.f : I → R is called strictly convex if, ∀x1 6= x2 ∈ I, ∀t ∈ (0, 1),

f(tx1 + (1− t)x2) < tf(x1) + (1− t)f(x2)

In words: any chord connecting two values of f is always above thegraph of f .

A strictly-convex function is convex; the converse is false.

Example

f(x) = x2 is strictly convex.

Counterexample

f(x) = |x| is convex but not strictly convex.

Convexity

A differentiable f : I → R is convex ⇐⇒

f(x) ≥ f(y) + f ′(y)(x− y) ∀x, y ∈ I

and it is strictly convex ⇐⇒

f(x) > f(y) + f ′(y)(x− y) ∀x, y ∈ I , x 6= y

a differentiable convex function f is not below its tangents.

a strictly differentiable convex function f is above its tangents.

f is convex and differentiable & f ′(x) = 0 ⇒ x is a global minimum of f .

f ′′ exists & f ′′ ≥ 0 (resp. f ′′ > 0) on I ⇒ f is convex (strictly convex).

Convexity

A differentiable f : I → R is convex ⇐⇒

f(x) ≥ f(y) + f ′(y)(x− y) ∀x, y ∈ I

f(x) > f(y) + f ′(y)(x− y) ∀x, y ∈ I , x 6= y

a differentiable convex function f is not below its tangents.

a strictly differentiable convex function f is above its tangents.

f is convex and differentiable & f ′(x) = 0 ⇒ x is a global minimum of f .

f ′′ exists & f ′′ ≥ 0 (resp. f ′′ > 0) on I ⇒ f is convex (strictly convex).

Convexity

Definition (convex combination (of points))∑mi=1 αixi, a linear combination of {xi}mi=1 ⊂ Rn, is called convex if all

the αi’s are nonnegative and∑m

i=1 αi = 1.

Definition (convex set)

S ⊂ Rn is called convex if it is closed under convex combinations.

Convexity

Definition (convex function from a convex subset of Rn to R)

Let S ⊂ Rn be convex. f : S → R is called convex if,∀x1,x2 ∈ S,∀t ∈ (0, 1),

f(tx1 + (1− t)x2) ≤ tf(x1) + (1− t)f(x2)

and it is called strictly convex if, ∀x1 6= x2 ∈ I, ∀t ∈ (0, 1),

f(tx1 + (1− t)x2) < tf(x1) + (1− t)f(x2)

Convexity

Let S ⊂ Rn be convex. A differentiable f : S → R is convex ⇐⇒

f(x) ≥ f(y) +∇f(y)(x− y) ∀x,y ∈ S

f(x) > f(y) +∇f(y)(x− y) ∀x,y ∈ S ,x 6= y

f is convex and differentiable & ∇f(x) = 0 ⇒ x is a global minimum off .

Convexity

Definition (Hessian)

the Hessian matrix of a twice-differentiable f : Rn → R is

∂x21

∂x1 ∂x2· · · ∂2f

∂x1 ∂xn

∂x2 ∂x1

∂x22· · · ∂2f

∂x2 ∂xn...

.... . .

∂xn ∂x1

∂xn ∂x2· · · ∂2f

∂x2n

Convexity

A twice-differentiable f : S → Rn is convex (resp. strictly convex) ⇐⇒its Hessian matrix is SPSD (SPD) on the interior of S.

Convexity

A weighted sum, with nonnegative weights, of convex functions is convex.A weighted sum, with nonnegative weights, of strictly convex functions isstrictly convex (if we ignore the degenerate case that all the weights arezero).

Example

If (xi)ni=1 ⊂ R and (wi)

ni=1 ⊂ R≥0 (and the wi’s do not depend on the

xi’s) then

f : R→ R f(µ) =∑N

i=1wi(µ− xi)2

is strictly convex while

g : R→ R g(µ) =∑N

i=1wi|µ− xi|

is convex.

Convexity

Every norm is a convex function.

Example

Let p ≥ 1.f : Rn → R≥0, f : x 7→ ‖x‖`p is convex.

Example

Let Q be an n× n SPD matrix.f : Rn → R≥0, f : x 7→ ‖x‖Q , (xTQx)1/2 is convex.

Convexity

Some Operations that Preserve Convexity of Functions

Nonnegative weighted sum of convex functions is convex

Example

f : Rn → R≥0 f : x 7→ ‖x‖2`2 ,n∑i=1

Composition with an affine function.

Example

f : Rn → R≥0 f : x 7→ ‖Ax+ b‖`p A ∈ Rm×n b ∈ Rm

Pointwise maximum and supremum.

Example

f(x) = maxi∈{1,2,3}

{f1(x), f2(x), f3(x)}

Convexity

Convex Optimization

Convex functions are “easy” to optimize.

Sometimes they have closed-form solutions for their minimizer(s).

Exercise

f : R→ R f : x 7→ ax2 + bx+ c a ∈ R>0 b, c ∈ R

Show that arg minx f(x) = − b2a (as you saw in high school).

But, more importantly, even whey have no closed-form minimizers it isstill “easy”.

Textbook: Boyd and Vandenberghe’s Convex Optimization. Also checkout Boyd’s video lectures at Stanford’s youtube channel.

Convexity

Convex Optimization

The take-home message: while learning convex optimization is veryuseful, unless you want to specialize in that field, it often suffices (forCV/ML people) to focus on how to:

recognize the problem is convex; ortransform a non-convex problem to a convex one; orapproximate a non-convex problem using a convex one; orsolve a non-convex problem by iterating between convex subproblems.

E.g., this is how Convex Optimization is taught by Boyd at Stanford.

Linear Model, Least Squares and Weighted Least Squares

Linear Model

Measurements: {yNi=1} ⊂ Rd

Hi is a known matrix, possibly dependent on i.

Residual of a linear model:

ri︸︷︷︸d×1

= Hi︸︷︷︸d×k

θ︸︷︷︸k×1

− yi︸︷︷︸d×1

When we discussed optical flow for gray-scale images, d was 1.

Least-Squares Estimation in a Linear Model

θLS , arg minθ

N∑i=1

‖ri‖2︸︷︷︸‖r‖2

H ,[HT

1 · · · HTN

]T ∈ R(Nd)×k

y ,[yT1 · · · yTN

]T ∈ R(Nd)×1

r ,[rT1 · · · rTN

]T= Hθ − y =∈ R(Nd)×1 (3)

The cost function, from Rk to R>0, is convex; it’s strictly convex ifrank(H) = k.A minimizer satisfies

HTHθLS = HTy (4)

and is unique if rank(H) = k; i.e., if HTH is invertible.

Weighted Least-Squares

θWLS , arg minθ

N∑i=1

wi‖ri‖2︸︷︷︸‖W 1/2(Hθ−y)‖2

W = diag(w1, . . . , w1︸︷︷︸d times

, w2, . . . , w2︸︷︷︸d times

, wN , . . . , wN︸︷︷︸d times

W 1/2 = diag(√w1, . . . ,

√w1︸︷︷︸

d times

,√w2, . . . ,

√w2︸︷︷︸

d times

,√wN , . . . ,

√wN︸︷︷︸

d times

HTWHθWLS = HTWy (7)

Outliers, “Wrong” Assumptions, and Robustness

Least-Squares Estimation is not Robust

10 5 0 5 1010

70Least Squares

Alsotrue for weighted least squares.

Robust Least-Squares

Measurements: {yNi=1} ⊂ Rd

Residual: ri︸︷︷︸d×1

= Hi︸︷︷︸d×k

θ︸︷︷︸k×1

− yi︸︷︷︸d×1

Wantarg min

ρ(‖ri‖)

where ρ : R→ R>0 is differentiable

ψ(x) = dρ(x)dx is called the influence function.

If ρ is of the form x 7→ x2 then this is least squares.

Robust Error Function

Example (Geman-McClure robust error function)

ρ(x) = x2/(x2 + σ2)

30 20 10 0 10 20 30x

Geman-McClure Robust Error Function: ρ(x) = x2/(x2 + σ2)

σ= 0. 5

σ= 10

σ= 20

Influence Function: Derivative of the Error Function

Example (Derivative of the Geman-McClure robust error function)

ψ(x) =d

dxρ(x) = (2σ2x)/(x2 + σ2)2

30 20 10 0 10 20 30x

1.5Influence Function: ψ(x) = (2σ2x)/(x2 + σ2)2

σ= 0. 5

σ= 10

σ= 20

Definition (quasiconvex function))

Let S ⊂ Rn be a convex set. f : S → R is called quasiconvex if all itssublevel sets

Sα = {x|f(x) ≤ α} ,

for α ∈ R, are convex.

Example

Geman-McClure robust error function:

x→ x2

x2 + σ2

If S ⊂ R and f : S → R is quasiconvex then f is unimodal.

A quasiconvex function is usually still easy to minimize. But . . .

A weighted sum, with nonnegative weights, of quasiconvex functions isusually not quasiconvex.

Example

If (xi)ni=1 ⊂ R and (wi)

ni=1 ⊂ R≥0 (and the wi’s do not depend on the

xi’s) then

f : R→ R f(µ) =

N∑i=1

wiρ(µ− xi) ,

where ρ is the Geman-McClure robust error function, is usually notunimodal.

Minimizing a Nonconvex Robust Cost Function

⇒ when ρ(x) is more robust than |x| then optimization is hard. . .

Example for possible approaches:Assuming ρ is differentiable, can try gradient-based methods, possibly withgraduated optimization such as

graduated non-convexity (minimize (1− t)fconvex + tf); or“annealing” (e.g., gradually decrease σ in the GM robust error function)

Iterative Reweighted Least Squares (IRLS)

Figure taken from Wikipediawww.cs.bgu.ac.il/~cv192/ Needful Things (ver. 1.00) Apr 7, 2019 25 / 82

Convexity

Results for a Gradient-Based Method in the Global ApproachFigure from Michael Black’s PhD, 1992

Convexity

Iterative Reweighted Least-Squares

Idea: use w(x) = ψ(x)/x

Start with an LS solution, then alternate between computing weightsbased on the residual errors, and a WLS solution using fixed weightsfrom the previous iteration.

Initialization: solve

HTHθ[0]IRLS = HTy (8)

Alternate

1 Set w[k]i =

ψ(∥∥∥r[k]

∥∥∥)∥∥∥r[k]i

∥∥∥ where r[k]i =Hiθ

[k]IRLS − yi

2 Solve

HTW [k]Hθ[k+1]IRLS =HTW [k]y (9)

Convexity

ERLS =∑i

ρ(‖ri‖)

∇θERLS =∑i

ψ(‖ri‖)∇θ‖ri‖want= 0 (10)

EIRLS =∑i

w(ri)‖ri‖2 (11)

E[k+1]IRLS =

w(r[k]i )‖ri‖2 =

ψ(∥∥∥r[k]i ∥∥∥)∥∥∥r[k]i ∥∥∥ ‖ri‖2 ≈

ψ(∥∥∥r[k]i ∥∥∥) ‖ri‖

0 = ∇θE[k+1]IRLS ≈

ψ(∥∥∥r[k]i ∥∥∥)∇θ‖ri‖

Convexity

ERLS =∑i

ρ(‖ri‖)

∇θERLS =∑i

ψ(‖ri‖)∇θ‖ri‖want= 0 (10)

EIRLS =∑i

w(ri)‖ri‖2 (11)

E[k+1]IRLS =

w(r[k]i )‖ri‖2 =

ψ(∥∥∥r[k]i ∥∥∥)∥∥∥r[k]i ∥∥∥ ‖ri‖2 ≈

ψ(∥∥∥r[k]i ∥∥∥) ‖ri‖

0 = ∇θE[k+1]IRLS ≈

ψ(∥∥∥r[k]i ∥∥∥)∇θ‖ri‖

Convexity

10 5 0 5 1010

70Least Squares

Convexity

10 5 0 5 1010

70Iterative Rewighted Least Squares, iter 1

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

10 5 0 5 1010

Convexity

Recall Lucas-Kanade

N∑i=1

g(x,xi)

(∇xI(xi, t)

[u(x)v(x)

]+ It(xi, t)

=∥∥∥W 1

2ε∥∥∥2 = εTWε =

N∑i=1

√g(x,xi)∇xI(xi, t)︸︷︷︸Hi:1×2

[u(x)v(x)

]︸︷︷︸θ:2×1

+√g(x,xi)It(xi, t)︸︷︷︸

−yi∈R

W = diag([g(x,x1) . . . g(x,xN )

]) (not the same W from IRLS)

W12 = diag(

[ √g(x,x1) . . .

√g(x,xN )

]) (13)[

u(x)v(x)

, arg minu(x),v(x)

εTWε (14)

Convexity

Robust Lucas-Kanade

ERLK =

N∑i=1

√g(x,xi)∇xI(xi, t)︸︷︷︸

Hi:1×2

[u(x)v(x)

]︸︷︷︸θ:2×1

+√g(x,xi)It(xi, t)︸︷︷︸

−yi∈R︸︷︷︸ri∈R

Convexity

Robust Affine Lucas-Kanade

[u(xi)v(xi)

[xi − x yi − y 1 0 0 0

0 0 0 xi − x yi − y 1

]︸︷︷︸

A(x,xi),

θ1θ2θ3θ4θ5θ6

︸︷︷︸θ,

ERALK =

N∑i=1

√g(x,xi)∇xI(xi, t)A(x,xi)︸︷︷︸

Hi:1×6

θ +√g(x,xi)It(xi, t)︸︷︷︸

−yi∈R︸︷︷︸ri∈R

Convexity

Random Sample Consensus (RANSAC)

A non-deterministic algorithm

For generality, write the model as f(xi,θ) = yi

Stick to the least-squares formulation

Algorithm: given data {xi,yi}Ni=1, alternate between

Pick a very small random subset of the data. Its cardinality should suffice forestimating the parameters (e.g., you can’t estimate a line from a singlepoint). Fit a least-squares model to it.

For each data point in the original set, compute∥∥∥f(xi, θ)− yi

∥∥∥. Using a

threshold, reject outliers.

Once a certain iteration achieves enough inliers (points not rejected),the algorithm stops.

Convexity

RANSAC and Optical Flow

For HS-type models, RANSAC is inapplicable.

For LK-type models, it fits only if the neighborhood is sufficiently large.For example, if we try to find a single affine flow for the entire image.

Some Statistics and M-Estimators

Definition (sample mean for R-valued data)

The sample mean of {xi}Ni=1 ⊂ R is

N∑i=1

xi (18)

Definition (sample mean for Rn-valued data)

The sample mean of {xi}Ni=1 ⊂ Rn is

N∑i=1

xi (19)

Definition (sample mean for R-valued data)

The sample mean of {xi}Ni=1 ⊂ R is

N∑i=1

xi (18)

Definition (sample mean for Rn-valued data)

The sample mean of {xi}Ni=1 ⊂ Rn is

N∑i=1

xi (19)

The Sample Mean as a Minimizer

The sample mean minimizes the sum of squared Euclidean distances:

x = arg minµ∈R

N∑i=1

(xi − µ)2 . (20)

Example

Let x1, x2 ∈ R. Then

x =x1 + x2

2= arg min

µ∈R(x1 − µ)2 + (x2 − µ)2 . (21)

The sample mean minimizes the sum of squared scaled Euclideandistances:

x = arg minµ∈R

N∑i=1

(xi − µ)2

σ2σ > 0 . (22)

Proof.

N∑i=1

(xi − µ)2

N∑i=1

(xi − µ)2

arg minµ∈R

N∑i=1

(xi − µ)2 = arg minµ∈R

N∑i=1

(xi − µ)2 = x

The sample mean minimizes the sum of squared scaled Euclideandistances:

x = arg minµ∈R

N∑i=1

(xi − µ)2

σ2σ > 0 . (22)

Proof.

N∑i=1

(xi − µ)2

N∑i=1

(xi − µ)2

arg minµ∈R

N∑i=1

(xi − µ)2 = arg minµ∈R

N∑i=1

(xi − µ)2 = x

Example

Let x1, x2 ∈ R. Then

x =x1 + x2

2= arg min

µ∈R

(x1 − µ)2

(x2 − µ)2

σ2. (23)

σ = 2:

30 20 10 0 10 20 30

(µ− x1)2/σ2

30 20 10 0 10 20 30

(µ− x2)2/σ2

30 20 10 0 10 20 30

both funcs

30 20 10 0 10 20 30

sum both funcs

Σi(xi −µ)2/σ2

sample mean

σ = 4:

30 20 10 0 10 20 30

(µ− x1)2/σ2

30 20 10 0 10 20 30

(µ− x2)2/σ2

30 20 10 0 10 20 30

both funcs

30 20 10 0 10 20 30

sum both funcs

Σi(xi −µ)2/σ2

sample mean

σ = 4, N = 3:

30 20 10 0 10 20 30

all 3 funcs

30 20 10 0 10 20 30

sum all funcs

Σi(xi −µ)2/σ2

sample mean

σ = 4, N = 4:

30 20 10 0 10 20 30

all 4 funcs

30 20 10 0 10 20 30

sum all funcs

Σi(xi −µ)2/σ2

sample mean

σ = 4, N = 4:

30 20 10 0 10 20 30

all 5 funcs

30 20 10 0 10 20 30

sum all funcs

Σi(xi −µ)2/σ2

sample mean

σ = 4, N = 7 (not robust to outliers):

30 20 10 0 10 20 30

all 7 funcs

30 20 10 0 10 20 30

sum all funcs

Σi(xi −µ)2/σ2

sample mean

x = arg minµ∈Rn

N∑i=1

∥∥∥∥ 1

σ(xi − µ)

∥∥∥∥2`2

. (24)

Outline of the proof:

(i) Show that, ∇µE(µ), the gradient of E(µ) =∑N

∥∥ 1σ (xi − µ)

∥∥2`2

w.r.t. µ =[µ1 . . . µn

]T, is proportional to[ ∑N

i=1(µ1 − xi,1) . . .∑N

i=1(µn − xi,n)]

where xi,j is the j-th entry of xi.

(ii) Solve ∇µE(µ) = 0

x = arg minµ∈Rn

N∑i=1

∥∥∥∥ 1

σ(xi − µ)

∥∥∥∥2`2

. (24)

Outline of the proof:

(i) Show that, ∇µE(µ), the gradient of E(µ) =∑N

∥∥ 1σ (xi − µ)

∥∥2`2

w.r.t. µ =[µ1 . . . µn

]T, is proportional to[ ∑N

i=1(µ1 − xi,1) . . .∑N

i=1(µn − xi,n)]

where xi,j is the j-th entry of xi.

(ii) Solve ∇µE(µ) = 0

More generally, let Q be an SPD n-by-n matrix.

x = arg minµ∈Rn

N∑i=1

‖xi − µ‖2Q (26)

where‖xi − µ‖2Q = (xi − µ)TQ(xi − µ)

If Q = σ−2In×n ∝ In×n this reduces to the previous problem:

arg minµ∈Rn

N∑i=1

∥∥∥∥ 1

σ(xi − µ)

∥∥∥∥2`2

Informal Definition (Statistic)

A statistic is a function that depends only on the data.

Trivially, a function of a statistic is also a statistic.

Example

S1(x1, . . . ,xN ;N) =∑N

i=1 xi ∈ Rn is a statistic of (x1, . . . ,xN ) ⊂ Rn.

The sames goes for x = 1N

∑Ni=1 xi = 1

N S1(x1, . . . ,xN ;N).

Example

S2(x1, . . . ,xN ;N) =∑N

i=1 xixTi ∈ Rn×n is a statistic of

(x1, . . . ,xN ) ⊂ Rn. The sames goes for 1N S2(x1, . . . ,xN ) and

∑Ni=1 xix

Ti )− xxT

Example

S1(x1, . . . ,xN ;N) =∑N

∑Ni=1 xi = 1

N S1(x1, . . . ,xN ;N).

Example

S2(x1, . . . ,xN ;N) =∑N

∑Ni=1 xix

Ti )− xxT

Example

S1(x1, . . . ,xN ;N) =∑N

∑Ni=1 xi = 1

N S1(x1, . . . ,xN ;N).

Example

S2(x1, . . . ,xN ;N) =∑N

∑Ni=1 xix

Ti )− xxT

Definition (k-th order statistic)

The k-th order statistic of {xi}Ni=1 ⊂ R is the k-th-smallest value among

{xi}Ni=1. It is denoted by x(k). Thus, x(1) ≤ x(2) ≤ . . . ≤ x(N).

Example

x(1) = min {x1, . . . , xN} and x(N) = max {x1, . . . , xN}.

Definition (order statistics)

The ordered N -tuple of the sorted values,

(x(1), x(2), . . . , x(N)) , (27)

is called the order statistics of {xi}Ni=1.

Example

(x(1), x(2), . . . , x(N)) , (27)

Example

(x(1), x(2), . . . , x(N)) , (27)

For simplicity, let us assume N is odd.

Definition (Sample median)

If N is odd, then the sample median of {xi}Ni=1 ⊂ R, is

x(N+12

) (28)

If N is even, there are several different ways to define the samplemedian; when N is large then they usually become similar.

For simplicity, let us assume N is odd.

Definition (Sample median)

If N is odd, then the sample median of {xi}Ni=1 ⊂ R, is

x(N+12

) (28)

If N is even, there are several different ways to define the samplemedian; when N is large then they usually become similar.

The Sample Median as a Minimizer

The sample median minimizes the sum of scaled `1 distances:

x(N+12 ) = arg min

N∑i=1

|xi −m|σ

σ > 0 . (29)

We may take this as the definition of the sample median (in which case,we don’t need to worry if N is even or odd).

σ = 2:

30 20 10 0 10 20 30

|µ− x1|/σ

30 20 10 0 10 20 30

|µ− x2|/σ

30 20 10 0 10 20 30

both funcs

30 20 10 0 10 20 30

sum both funcs

Σi|xi −µ|/σsample median

σ = 4:

30 20 10 0 10 20 30

|µ− x1|/σ

30 20 10 0 10 20 30

|µ− x2|/σ

30 20 10 0 10 20 30

both funcs

30 20 10 0 10 20 30

sum both funcs

σ = 4, N = 3:

30 20 10 0 10 20 30

all 3 funcs

30 20 10 0 10 20 30

sum all funcs

σ = 4, N = 4:

30 20 10 0 10 20 30

all 4 funcs

30 20 10 0 10 20 30

sum all funcs

σ = 4, N = 4:

30 20 10 0 10 20 30

all 5 funcs

30 20 10 0 10 20 30

sum all funcs

σ = 4, N = 7 (more robust to outliers):

30 20 10 0 10 20 30

all 7 funcs

30 20 10 0 10 20 30

60sum all funcs

The Sample Median is More Robust Than the Sample Mean

Since `1 is more robust than `2, this interpretation in terms ofoptimization problems explains why the sample median is more robustthan the sample mean.

Median Filtering

Replace pixel (i, j) with the median of the value in, say, a 5 by 5,neighborhood around it. This is a nonlinear operation.

Images taken from Wikipediawww.cs.bgu.ac.il/~cv192/ Needful Things (ver. 1.00) Apr 7, 2019 62 / 82

Trimmed Average

The median is also an extreme case of the truncated (or trimmed) average:

N − 2N0

N−N0∑i=N0

x(i) (30)

We will return to the trimmed average later when we discuss Robust PCA(PCA is a dimensionality-reduction technique which we will discuss aswell).

M-Estimators

More generally, estimators that are defined as minimizers of sumsfunctions of the data are called M-estimators. These include, amongother things,∑N

i=1 ρ(ri) we saw previously.Maximum-Likelihood Estimators (MLE):

arg maxθ

N∏i=1

p(xi;θ) = arg minθ

N∑i=1

− log p(xi;θ)

MLE enjoy many desired properties and satisfy various asymptotic optimalitycriteria – but may suffer from outliers. Some robust estimators achievenear-optimality when there are no outliers, and suffer little in their presence.

M -estimators can be defined even when the space is nonlinear — wewill see some examples.

M-Estimators

arg maxθ

N∏i=1

N∑i=1

− log p(xi;θ)

M-Estimators

arg maxθ

N∏i=1

N∑i=1

− log p(xi;θ)

M-Estimators

arg maxθ

N∏i=1

N∑i=1

− log p(xi;θ)

The Sample Mean as an MLE

The sample mean maximizes the likelihood under a Gaussian likelihoodmodel:

x = arg maxµ∈R

N∏i=1

N (xi;µ, σ2) . (31)

Remark

∫R· · ·∫R

N∏i=1

N (xi;µ, σ2) dx1 · · · dxN = 1

∫RN (xi;µ, σ

2) dµ = 1 ∀i but

N∏i=1

N (xi;µ, σ2) dµ 6= 1 ∀N > 1

Example

Let (x1, x2) ∼ N (x1;µ, σ2)N (x2;µ, σ

x =x1 + x2

2= arg max

µN (x1;µ, σ

2)N (x2;µ, σ2) (32)

σ = 2:

30 20 10 0 10 20 30

(2σ2π)−1/2exp(− (µ− x1)2/σ2)

30 20 10 0 10 20 30

(2σ2π)−1/2exp(− (µ− x2)2/σ2)

30 20 10 0 10 20 30

both funcs

30 20 10 0 10 20 30

0.000005

0.000000

0.000005

0.000010

0.000015

product of both funcs

(2πσ2)−n/2∏iexp(− 0. 5(xi −µ)2/σ2)

sample mean

σ = 4:

30 20 10 0 10 20 30

(2σ2π)−1/2exp(− (µ− x1)2/σ2)

30 20 10 0 10 20 30

(2σ2π)−1/2exp(− (µ− x2)2/σ2)

30 20 10 0 10 20 30

both funcs

30 20 10 0 10 20 30

0.00001

0.00000

0.00001

0.00002

0.00003

(2πσ2)−n/2∏iexp(− 0. 5(xi −µ)2/σ2)

sample mean

σ = 4, N = 3:

30 20 10 0 10 20 30

all 3 funcs

30 20 10 0 10 20 30

1e 7 product all funcs

(2πσ2)−n/2∏iexp(− 0. 5(xi −µ)2/σ2)

sample mean

σ = 4, N = 4:

30 20 10 0 10 20 30

all 4 funcs

30 20 10 0 10 20 30

(2πσ2)−n/2∏iexp(− 0. 5(xi −µ)2/σ2)

sample mean

σ = 4, N = 4:

30 20 10 0 10 20 30

all 5 funcs

30 20 10 0 10 20 30

2.51e 11 product all funcs

(2πσ2)−n/2∏iexp(− 0. 5(xi −µ)2/σ2)

sample mean

σ = 4, N = 7 (not robust to outliers):

30 20 10 0 10 20 30

all 7 funcs

30 20 10 0 10 20 30

(2πσ2)−n/2∏iexp(− 0. 5(xi −µ)2/σ2)

sample mean

More generally, let Σ be an SPD n-by-n matrix.

The inverse of an SPD matrix is also SPD.

x = arg maxµ∈Rn

N∏i=1

(2π)n/2|Σ|1/2exp

2‖xi − µ‖2Σ−1

where‖xi − µ‖2Σ−1 = (xi − µ)TΣ−1(xi − µ),

is called the Mahalanobis distance.

|Σ| is the determinant of Σwww.cs.bgu.ac.il/~cv192/ Needful Things (ver. 1.00) Apr 7, 2019 73 / 82

The Sample Median as an MLE

The sample median maximizes the likelihood under a Laplace-distributionlikelihood model

x = arg maxµ∈R

N∏i=1

f(xi;µ, σ) . (34)

f(x;µ, σ) =1

2σexp

(−|x− µ|

30 20 10 0 10 20 30

(2σ)−1exp(− |µ− x1|/σ)

30 20 10 0 10 20 30

(2σ)−1exp(− |µ− x2|/σ)

30 20 10 0 10 20 30

both funcs

30 20 10 0 10 20 30

(2σ)−1/n∏iexp(− |xi −µ|/σ)

sample median

σ = 4, N = 3:

30 20 10 0 10 20 30

all 3 funcs

30 20 10 0 10 20 30

0.0002

0.0001

0.0000

0.0001

0.0002

0.0003

0.0004

product all funcs

(2σ)−1/n∏iexp(− |xi −µ|/σ)

sample median

σ = 4, N = 4:

30 20 10 0 10 20 30

all 4 funcs

30 20 10 0 10 20 30

0.00001

0.00000

0.00001

0.00002

0.00003

product all funcs

(2σ)−1/n∏iexp(− |xi −µ|/σ)

sample median

σ = 4, N = 4:

30 20 10 0 10 20 30

all 5 funcs

30 20 10 0 10 20 30

0.000001

0.000000

0.000001

0.000002

0.000003

product all funcs

(2σ)−1/n∏iexp(− |xi −µ|/σ)

sample median

σ = 4, N = 7 (more robust to outliers than the Gaussian):

30 20 10 0 10 20 30

all 7 funcs

30 20 10 0 10 20 30

(2σ)−1/n∏iexp(− |xi −µ|/σ)

sample median

Definition (sample correlation matrix)

∑Ni=1 xix

Ti ∈ Rn×n is called the sample correlation matrix of

{xi}Ni=1 ⊂ Rn

Definition (sample covariance matrix)

∑Ni=1(xi − x)(xi − x)T is called the sample covariance matrix of

{xi}Ni=1 ⊂ Rn

N∑i=1

(xi − x)(xi − x)T =

N∑i=1

)− xxT

N∑i=1

(xi − x)(xi − x)T = 1N

N∑i=1

(xi − x)xTi = 1N

N∑i=1

xi(xi − x)T

Definition (sample correlation matrix)

∑Ni=1 xix

Ti ∈ Rn×n is called the sample correlation matrix of

{xi}Ni=1 ⊂ Rn

Definition (sample covariance matrix)

∑Ni=1(xi − x)(xi − x)T is called the sample covariance matrix of

{xi}Ni=1 ⊂ Rn

N∑i=1

(xi − x)(xi − x)T =

N∑i=1

)− xxT

N∑i=1

(xi − x)(xi − x)T = 1N

N∑i=1

(xi − x)xTi = 1N

N∑i=1

xi(xi − x)T

Both 1N

∑Ni=1 xix

Ti and 1

∑Ni=1(xi − x)(xi − x)T are symmetric n× n

matrices and their eigenvalues are always nonnegative.⇒ they are SPSD. If their eigenvalues are positive, then they are also SPD.

Version Log

7/4/2019, ver 1.00.

computer vision: models, learning and inference convexity ...cv192/wiki.files/cv192...cv/ml people)...

Documents

geometry of convex functions - convex optimization3.5it is...

lecture 8 - convex optimization 8 - convex optimization i...

polygons - concave or convexpolygons - concave & convex...

lecture 7: convex optimizations · convex optimization...

convex optimization -...

convex hull algorithms · convex hull: basic facts problem:...

disciplined convex-concave programming - arxiv · 1.3...

cv192, hw #4 - bgucv192/wiki.files/cv192_hw4.pdf ·...

convex optimization problems - university of...

convex optimization: modeling and...

chapter 8 convex · pdf filechapter 8 convex optimization...

convex hull problem presented by erion lin. outline convex...

lecture 9 – nonlinear programming models topics convex...

convex optimization - iit bombay · chapter 4 convex...

3d convex hulls - university of california, irvine · 2020....

convex optimization m2aspremon/pdf/mva/firstorder... · a....

convex geometry and functional analysis › ~lehec › gdl...

26 convex hull computationsctoth/handbook/chap26.pdf · 26...

an online portfolio selection algorithm with regret ... ·...

data science - convex optimization and application · 6data...