trust region method
TRANSCRIPT
![Page 1: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/1.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 1/18
Trust Region MethodsPart I
Andrew R. [email protected]
Mathematical SciencesIBM T.J. Watson Research Center
February 2007, Montreal
1
![Page 2: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/2.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 2/18
Overview
1 Trust-Region/Modelling Methods
![Page 3: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/3.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 3/18
Trust-Region/Modelling Methods
What is a trust-region method?
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Problem: Minimize
−10x 21 + 10x 22 + 4 sinx 1x 2
− 2x 1 + x
41
3
![Page 4: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/4.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 4/18
Trust-Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 0
T R i /M d lli M h d
![Page 5: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/5.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 5/18
Trust-Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 1
T t R i /M d lli M th d
![Page 6: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/6.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 6/18
Trust-Region/Modelling Methods
What is a trust-region method?
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 2
6
Trust Region/Modelling Methods
![Page 7: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/7.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 7/18
Trust-Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 3 = x 2
Trust Region/Modelling Methods
![Page 8: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/8.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 8/18
Trust-Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 4
8
Trust-Region/Modelling Methods
![Page 9: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/9.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 9/18
Trust-Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 5
9
Trust-Region/Modelling Methods
![Page 10: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/10.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 10/18
Trust Region/Modelling Methods
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
The model and trust region around x 6 (x 7 = x ∗)
10
Trust-Region/Modelling Methods
![Page 11: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/11.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 11/18
st g o / o g t o s
What is a trust-region method? (continued)
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Six iterations —different initial point
11
Trust-Region/Modelling Methods
![Page 12: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/12.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 12/18
g / g
Basic Algorithm
Initialize: x 0, ∆
Compute Model: mk ( )
Compute Step: Compute s k from
mins ≤∆
mk (x k + s )
Trust-region Update: ρ = f (x k )−f (x k +s k )mk (x k )−mk (x k +s k )
If ρ > 0.75 ∆ ← 2.0∆
If 0.25 < ρ < 0.75 ∆ ← ∆
If ρ < 0.25 ∆ ← 0.5∆
Accept x k + s k
Accept x k + s k
Reject x k + s k
12
Trust-Region/Modelling Methods
![Page 13: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/13.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 13/18
/
Assumptions to prove convergence
On the problem
Smooth f ∈ C
2
Bounded Below f bounded below
Bounded Hessian xx f bounded above
13
Trust-Region/Modelling Methods
![Page 14: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/14.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 14/18
Assumptions to prove convergence
On the model (∀k )
Smooth mk ∈ C 2
Interpolatesm
k (x k ) =
f (x k )
Interpolates Gradient x mk (x k ) = x f (x k )
Bounded Hessianmaxx ∈Bk
xx mk (x ) bounded above where,Bk = {x ∈ n | x − x k k ≤ ∆k }
13
![Page 15: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/15.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 15/18
![Page 16: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/16.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 16/18
Trust-Region/Modelling Methods
![Page 17: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/17.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 17/18
Standard Model in the differentiable case
Typical trust region or line search method builds linear or
quadratic model of the objective function f .The model has to satisfy Taylor-like error bounds.Second Order
|f (x ) − m(x )| ≤ O(∆3)
|f
(x
) − m
(x
)| ≤ O(∆2
)|2
f (x ) − 2m(x )| ≤ O(∆)
In fact it typically is a first (or second) order Taylor series
approximation.In derivative based methods constants in O depend only on f
(and its derivatives).
By reducing the trust region or step size one guarantees better
accuracy.14
Trust-Region/Modelling Methods
![Page 18: Trust Region Method](https://reader031.vdocument.in/reader031/viewer/2022021117/577d35151a28ab3a6b8f8840/html5/thumbnails/18.jpg)
8/8/2019 Trust Region Method
http://slidepdf.com/reader/full/trust-region-method 18/18
Standard Model in the differentiable case
Typical trust region or line search method builds linear or
quadratic model of the objective function f .The model has to satisfy Taylor-like error bounds.Second Order
|f (x ) − m(x )| ≤ O(∆3)
|f
(x
) − m
(x
)| ≤ O(∆2
)|2
f (x ) − 2m(x )| ≤ O(∆)
In fact it typically is a first (or second) order Taylor series
approximation.In derivative based methods constants in O depend only on f
(and its derivatives).
By reducing the trust region or step size one guarantees better
accuracy.14