math 408a convergence of backtracking methodsburke/crs/408/... · math 408a convergence of...
TRANSCRIPT
![Page 1: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/1.jpg)
Outline
Math 408AConvergence of Backtracking Methods
Math 408A Convergence of Backtracking Methods
![Page 2: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/2.jpg)
Outline
Math 408A Convergence of Backtracking Methods
![Page 3: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/3.jpg)
Outline
Lipschitz Continuity
We say that F : Rn → Rm is Lipschitz continuous relative to aset D ⊂ Rn if there exists a constant K ≥ 0 such that
‖F (x)− F (y)‖ ≤ K‖x − y‖
for all x , y ∈ D.
Fact: Lipschitz continuity implies uniform continuity.
Examples:
1. f (x) = x−1 is continuous on (0, 1), but it is not uniformlycontinuous on (0, 1).
2. f (x) =√
x is uniformly continuous on [0, 1], but it is notLipschitz continuous on [0, 1].
Math 408A Convergence of Backtracking Methods
![Page 4: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/4.jpg)
Outline
Lipschitz Continuity
We say that F : Rn → Rm is Lipschitz continuous relative to aset D ⊂ Rn if there exists a constant K ≥ 0 such that
‖F (x)− F (y)‖ ≤ K‖x − y‖
for all x , y ∈ D.
Fact: Lipschitz continuity implies uniform continuity.
Examples:
1. f (x) = x−1 is continuous on (0, 1), but it is not uniformlycontinuous on (0, 1).
2. f (x) =√
x is uniformly continuous on [0, 1], but it is notLipschitz continuous on [0, 1].
Math 408A Convergence of Backtracking Methods
![Page 5: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/5.jpg)
Outline
Lipschitz Continuity
We say that F : Rn → Rm is Lipschitz continuous relative to aset D ⊂ Rn if there exists a constant K ≥ 0 such that
‖F (x)− F (y)‖ ≤ K‖x − y‖
for all x , y ∈ D.
Fact: Lipschitz continuity implies uniform continuity.
Examples:
1. f (x) = x−1 is continuous on (0, 1), but it is not uniformlycontinuous on (0, 1).
2. f (x) =√
x is uniformly continuous on [0, 1], but it is notLipschitz continuous on [0, 1].
Math 408A Convergence of Backtracking Methods
![Page 6: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/6.jpg)
Outline
Lipschitz Continuity
We say that F : Rn → Rm is Lipschitz continuous relative to aset D ⊂ Rn if there exists a constant K ≥ 0 such that
‖F (x)− F (y)‖ ≤ K‖x − y‖
for all x , y ∈ D.
Fact: Lipschitz continuity implies uniform continuity.
Examples:
1. f (x) = x−1 is continuous on (0, 1), but it is not uniformlycontinuous on (0, 1).
2. f (x) =√
x is uniformly continuous on [0, 1], but it is notLipschitz continuous on [0, 1].
Math 408A Convergence of Backtracking Methods
![Page 7: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/7.jpg)
Outline
Lipschitz Continuity
We say that F : Rn → Rm is Lipschitz continuous relative to aset D ⊂ Rn if there exists a constant K ≥ 0 such that
‖F (x)− F (y)‖ ≤ K‖x − y‖
for all x , y ∈ D.
Fact: Lipschitz continuity implies uniform continuity.
Examples:
1. f (x) = x−1 is continuous on (0, 1), but it is not uniformlycontinuous on (0, 1).
2. f (x) =√
x is uniformly continuous on [0, 1], but it is notLipschitz continuous on [0, 1].
Math 408A Convergence of Backtracking Methods
![Page 8: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/8.jpg)
Outline
Lipschitz Continuity and the Derivative
Fact:If F ′ exists and is continuous on a compact convex setD ⊂ Rm, then F is Lipschitz continuous on D.
Proof: By the MVT
‖F (x)− F (y)‖ ≤
(sup
z∈[x ,y ]‖F ′(z)‖
)‖x − y‖.
Lipschitz continuity is almost but not quite a differentiabilityhypothesis. The Lipschitz constant provides bounds on rate ofchange. For example, every norm is Lipschitz continuous, but notdifferentiable at the origin.
Math 408A Convergence of Backtracking Methods
![Page 9: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/9.jpg)
Outline
Lipschitz Continuity and the Derivative
Fact:If F ′ exists and is continuous on a compact convex setD ⊂ Rm, then F is Lipschitz continuous on D.
Proof: By the MVT
‖F (x)− F (y)‖ ≤
(sup
z∈[x ,y ]‖F ′(z)‖
)‖x − y‖.
Lipschitz continuity is almost but not quite a differentiabilityhypothesis. The Lipschitz constant provides bounds on rate ofchange. For example, every norm is Lipschitz continuous, but notdifferentiable at the origin.
Math 408A Convergence of Backtracking Methods
![Page 10: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/10.jpg)
Outline
Lipschitz Continuity and the Derivative
Fact:If F ′ exists and is continuous on a compact convex setD ⊂ Rm, then F is Lipschitz continuous on D.
Proof: By the MVT
‖F (x)− F (y)‖ ≤
(sup
z∈[x ,y ]‖F ′(z)‖
)‖x − y‖.
Lipschitz continuity is almost but not quite a differentiabilityhypothesis. The Lipschitz constant provides bounds on rate ofchange.
For example, every norm is Lipschitz continuous, but notdifferentiable at the origin.
Math 408A Convergence of Backtracking Methods
![Page 11: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/11.jpg)
Outline
Lipschitz Continuity and the Derivative
Fact:If F ′ exists and is continuous on a compact convex setD ⊂ Rm, then F is Lipschitz continuous on D.
Proof: By the MVT
‖F (x)− F (y)‖ ≤
(sup
z∈[x ,y ]‖F ′(z)‖
)‖x − y‖.
Lipschitz continuity is almost but not quite a differentiabilityhypothesis. The Lipschitz constant provides bounds on rate ofchange. For example, every norm is Lipschitz continuous, but notdifferentiable at the origin.
Math 408A Convergence of Backtracking Methods
![Page 12: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/12.jpg)
Outline
The Quadratic Bound Lemma (QBL)
Let F : Rn → Rm be such that F ′ is Lipschitz continuous on theconvex set D ⊂ Rn. Then
‖F (y)− (F (x) + F ′(x)(y − x))‖ ≤ K
2‖y − x‖2
for all x , y ∈ D where K is a Lipschitz constant for F ′ on D.
Math 408A Convergence of Backtracking Methods
![Page 13: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/13.jpg)
Outline
Lipschitz Continuity and the Quadratic Bound Lemma
Proof:
F (y)− F (x)− F ′(x)(y − x) =∫ 1
0F ′(x + t(y − x))(y − x)dt − F ′(x)(y − x)
=∫ 1
0[F ′(x + t(y − x))− F ′(x)](y − x)dt
‖F (y)− (F (x) + F ′(x)(y − x))‖ = ‖∫ 1
0[F ′(x + t(y − x))− F ′(x)](y − x)dt‖
≤∫ 1
0‖(F ′(x + t(y − x)− F ′(x))(y − x)‖dt
≤∫ 1
0‖F ′(x + t(y − x))− F ′(x)‖ ‖y − x‖dt
≤∫ 1
0Kt‖y − x‖2dt
= K2 ‖y − x‖2.
Math 408A Convergence of Backtracking Methods
![Page 14: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/14.jpg)
Outline
Theorem: Convergence of Backtracking
Let f : Rn → R and x0 ∈ R be such that f is differentiable on Rn with∇f Lipschitz continuous on an open convex set containing the set{x : f (x) ≤ f (x0)}. Let {xk} be the sequence satisfying xk+1 = xk
if∇f (xk) = 0; otherwise,
xk+1 = xk + tkdk , where dk satisfies f ′(xk ; dk) < 0,
and tk is chosen by the backtracking stepsize selection method.
Thenone of thefollowing statements must be true:
(i) There is a k0 such that ∇f ′(xk0) = 0.
(ii) f (xk)↘ −∞(iii) The sequence {‖dk‖} diverges (‖dk‖ → ∞).
(iv) For every subsequence J ⊂ N for which {dk : k ∈ J} is bounded, wehave
limk∈J
f ′(xk ; dk) = 0.
Math 408A Convergence of Backtracking Methods
![Page 15: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/15.jpg)
Outline
Theorem: Convergence of Backtracking
Let f : Rn → R and x0 ∈ R be such that f is differentiable on Rn with∇f Lipschitz continuous on an open convex set containing the set{x : f (x) ≤ f (x0)}. Let {xk} be the sequence satisfying xk+1 = xk
if∇f (xk) = 0; otherwise,
xk+1 = xk + tkdk , where dk satisfies f ′(xk ; dk) < 0,
and tk is chosen by the backtracking stepsize selection method. Thenone of thefollowing statements must be true:
(i) There is a k0 such that ∇f ′(xk0) = 0.
(ii) f (xk)↘ −∞(iii) The sequence {‖dk‖} diverges (‖dk‖ → ∞).
(iv) For every subsequence J ⊂ N for which {dk : k ∈ J} is bounded, wehave
limk∈J
f ′(xk ; dk) = 0.
Math 408A Convergence of Backtracking Methods
![Page 16: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/16.jpg)
Outline
Theorem: Convergence of Backtracking
Let f : Rn → R and x0 ∈ R be such that f is differentiable on Rn with∇f Lipschitz continuous on an open convex set containing the set{x : f (x) ≤ f (x0)}. Let {xk} be the sequence satisfying xk+1 = xk
if∇f (xk) = 0; otherwise,
xk+1 = xk + tkdk , where dk satisfies f ′(xk ; dk) < 0,
and tk is chosen by the backtracking stepsize selection method. Thenone of thefollowing statements must be true:
(i) There is a k0 such that ∇f ′(xk0) = 0.
(ii) f (xk)↘ −∞(iii) The sequence {‖dk‖} diverges (‖dk‖ → ∞).
(iv) For every subsequence J ⊂ N for which {dk : k ∈ J} is bounded, wehave
limk∈J
f ′(xk ; dk) = 0.
Math 408A Convergence of Backtracking Methods
![Page 17: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/17.jpg)
Outline
Theorem: Convergence of Backtracking
Let f : Rn → R and x0 ∈ R be such that f is differentiable on Rn with∇f Lipschitz continuous on an open convex set containing the set{x : f (x) ≤ f (x0)}. Let {xk} be the sequence satisfying xk+1 = xk
if∇f (xk) = 0; otherwise,
xk+1 = xk + tkdk , where dk satisfies f ′(xk ; dk) < 0,
and tk is chosen by the backtracking stepsize selection method. Thenone of thefollowing statements must be true:
(i) There is a k0 such that ∇f ′(xk0) = 0.
(ii) f (xk)↘ −∞
(iii) The sequence {‖dk‖} diverges (‖dk‖ → ∞).
(iv) For every subsequence J ⊂ N for which {dk : k ∈ J} is bounded, wehave
limk∈J
f ′(xk ; dk) = 0.
Math 408A Convergence of Backtracking Methods
![Page 18: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/18.jpg)
Outline
Theorem: Convergence of Backtracking
Let f : Rn → R and x0 ∈ R be such that f is differentiable on Rn with∇f Lipschitz continuous on an open convex set containing the set{x : f (x) ≤ f (x0)}. Let {xk} be the sequence satisfying xk+1 = xk
if∇f (xk) = 0; otherwise,
xk+1 = xk + tkdk , where dk satisfies f ′(xk ; dk) < 0,
and tk is chosen by the backtracking stepsize selection method. Thenone of thefollowing statements must be true:
(i) There is a k0 such that ∇f ′(xk0) = 0.
(ii) f (xk)↘ −∞(iii) The sequence {‖dk‖} diverges (‖dk‖ → ∞).
(iv) For every subsequence J ⊂ N for which {dk : k ∈ J} is bounded, wehave
limk∈J
f ′(xk ; dk) = 0.
Math 408A Convergence of Backtracking Methods
![Page 19: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/19.jpg)
Outline
Theorem: Convergence of Backtracking
Let f : Rn → R and x0 ∈ R be such that f is differentiable on Rn with∇f Lipschitz continuous on an open convex set containing the set{x : f (x) ≤ f (x0)}. Let {xk} be the sequence satisfying xk+1 = xk
if∇f (xk) = 0; otherwise,
xk+1 = xk + tkdk , where dk satisfies f ′(xk ; dk) < 0,
and tk is chosen by the backtracking stepsize selection method. Thenone of thefollowing statements must be true:
(i) There is a k0 such that ∇f ′(xk0) = 0.
(ii) f (xk)↘ −∞(iii) The sequence {‖dk‖} diverges (‖dk‖ → ∞).
(iv) For every subsequence J ⊂ N for which {dk : k ∈ J} is bounded, wehave
limk∈J
f ′(xk ; dk) = 0.
Math 408A Convergence of Backtracking Methods
![Page 20: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/20.jpg)
Outline
Corollaries
Corollary 1:If the sequences {dk} and {f (xk)} are bounded, then
limk→∞
f ′(xk ; dk) = 0.
Corollary 2: If dk = −∇f ′(xk)/‖∇f (xk)‖ is the Cauchy directionfor all k, then every accumulation point, x, of the sequence {xk}satisfies ∇f (x) = 0.
Math 408A Convergence of Backtracking Methods
![Page 21: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/21.jpg)
Outline
Corollaries
Corollary 1:If the sequences {dk} and {f (xk)} are bounded, then
limk→∞
f ′(xk ; dk) = 0.
Corollary 2: If dk = −∇f ′(xk)/‖∇f (xk)‖ is the Cauchy directionfor all k, then every accumulation point, x, of the sequence {xk}satisfies ∇f (x) = 0.
Math 408A Convergence of Backtracking Methods
![Page 22: Math 408A Convergence of Backtracking Methodsburke/crs/408/... · Math 408A Convergence of Backtracking Methods. Outline Theorem: Convergence of Backtracking Let f : Rn!R and x 0](https://reader036.vdocument.in/reader036/viewer/2022071005/5fc297756f03b77c3c5494ab/html5/thumbnails/22.jpg)
Outline
Corollaries
Corollary 3: Let us further assume that f is twice continuouslydifferentiable and that there is a β > 0 such that, for all u ∈ Rn,β‖u‖2 < uT∇2f (x)u on {x : f (x) ≤ f (x0)}. If the BasicBacktracking algorithm is implemented using the Newton searchdirections,
dk = −∇2f (xk)−1∇f (xk),
then every accumulation point, x, of the sequence {xk} satisfies∇f (x) = 0.
Math 408A Convergence of Backtracking Methods