![Page 1: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/1.jpg)
Data processing and networks optimization
Part II: Optimization (Basics)
Pierre Borgnat1, Jean-Christophe Pesquet2, Nelly Pustelnik1
1 ENS Lyon – Laboratoire de Physique – CNRS UMR [email protected], [email protected]
2 LIGM – Univ. Paris-Est – CNRS UMR [email protected]
![Page 2: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/2.jpg)
2/29
Hilbert spaces
A (real) Hilbert space H is a complete real vector space endowed with an
inner product 〈· | ·〉. The associated norm is
(∀x ∈ H) ‖x‖ =√
〈x | x〉.
◮ Particular case: H = RN (Euclidean space with dimension N).
![Page 3: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/3.jpg)
2/29
Hilbert spaces
A (real) Hilbert space H is a complete real vector space endowed with an
inner product 〈· | ·〉. The associated norm is
(∀x ∈ H) ‖x‖ =√
〈x | x〉.
◮ Particular case: H = RN (Euclidean space with dimension N).
2H is the power set of H, i.e. the family of all subsets of H.
![Page 4: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/4.jpg)
3/29
Hilbert spaces
Let H and G be two Hilbert spaces.A linear operator L : H → G is bounded (or continuous) if
‖L‖ = sup‖x‖H≤1
‖Lx‖G < +∞
![Page 5: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/5.jpg)
3/29
Hilbert spaces
Let H and G be two Hilbert spaces.A linear operator L : H → G is bounded (or continuous) if
‖L‖ = sup‖x‖≤1
‖Lx‖ < +∞
![Page 6: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/6.jpg)
3/29
Hilbert spaces
Let H and G be two Hilbert spaces.A linear operator L : H → G is bounded (or continuous) if
‖L‖ = sup‖x‖≤1
‖Lx‖ < +∞
◮ In finite dimension, every linear operator is bounded.
![Page 7: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/7.jpg)
3/29
Hilbert spaces
Let H and G be two Hilbert spaces.A linear operator L : H → G is bounded (or continuous) if
‖L‖ = sup‖x‖≤1
‖Lx‖ < +∞
◮ In finite dimension, every linear operator is bounded.
B(H,G): Banach space of bounded linear operators from H to G.
![Page 8: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/8.jpg)
4/29
Hilbert spaces
Let H and G be two Hilbert spaces.Let L ∈ B(H,G). Its adjoint L∗ is the operator in B(G,H) defined as
(∀(x , y) ∈ H × G) 〈y | Lx〉G = 〈L∗y | x〉H .
![Page 9: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/9.jpg)
4/29
Hilbert spaces
Let H and G be two Hilbert spaces.Let L ∈ B(H,G). Its adjoint L∗ is the operator in B(G,H) defined as
(∀(x , y) ∈ H× G) 〈y | Lx〉 = 〈L∗y | x〉 .
![Page 10: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/10.jpg)
4/29
Hilbert spaces
Let H and G be two Hilbert spaces.Let L ∈ B(H,G). Its adjoint L∗ is the operator in B(G,H) defined as
(∀(x , y) ∈ H× G) 〈Lx | y〉 = 〈x | L∗y〉 .
![Page 11: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/11.jpg)
4/29
Hilbert spaces
Let H and G be two Hilbert spaces.Let L ∈ B(H,G). Its adjoint L∗ is the operator in B(G,H) defined as
(∀(x , y) ∈ H× G) 〈Lx | y〉 = 〈x | L∗y〉 .
Example:
If L : H → Hn : x 7→ (x , . . . , x)
then L∗ : Hn → H : y = (y1, . . . , yn) 7→n∑
i=1
yi
Proof:
〈Lx | y〉 = 〈(x , . . . , x) | (y1, . . . , yn)〉 =n∑
i=1
〈x | yi 〉 =
⟨x |
n∑
i=1
yi
⟩
![Page 12: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/12.jpg)
4/29
Hilbert spaces
Let H and G be two Hilbert spaces.Let L ∈ B(H,G). Its adjoint L∗ is the operator in B(G,H) defined as
(∀(x , y) ∈ H× G) 〈Lx | y〉 = 〈x | L∗y〉 .
◮ We have ‖L∗‖ = ‖L‖.
◮ If L is bijective (i.e. an isomorphism ) then L−1 ∈ B(G,H) and
(L−1)∗ = (L∗)−1.
◮ If H = RN and G = R
M then L∗ = L⊤.
![Page 13: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/13.jpg)
5/29
Functional analysis: definitions
Let f : H → ]−∞,+∞] where H is a Hilbert space.
◮ The domain of f is dom f = {x ∈ H | f (x) < +∞}.
◮ The function f is proper if dom f 6= ∅.
Domains of the functions ?
x
f (x)
x
f (x)
δ
![Page 14: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/14.jpg)
5/29
Functional analysis: definitions
Let f : H → ]−∞,+∞] where H is a Hilbert space.
◮ The domain of f is dom f = {x ∈ H | f (x) < +∞}.
◮ The function f is proper if dom f 6= ∅.
Domains of the functions ?
x
f (x)
dom f = R
(proper)
x
f (x)
δ
![Page 15: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/15.jpg)
5/29
Functional analysis: definitions
Let f : H → ]−∞,+∞] where H is a Hilbert space.
◮ The domain of f is dom f = {x ∈ H | f (x) < +∞}.
◮ The function f is proper if dom f 6= ∅.
Domains of the functions ?
x
f (x)
dom f = R
(proper)
x
f (x)
δ
dom f =]0, δ](proper)
![Page 16: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/16.jpg)
6/29
Functional analysis: definitions
Let C ⊂ H.The indicator function of C is
(∀x ∈ H) ιC (x) =
{0 if x ∈ C
+∞ otherwise.
Example : C = [δ1, δ2]f (x) = ι[δ1,δ2](x)
δ1 xδ2
![Page 17: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/17.jpg)
7/29
Convergence in Hilbert spaces
Let H be a Hilbert space.Let (xn)n∈N be a sequence in H and x ∈ H.
◮ (xn)n∈N converges strongly to x if
limn→+∞
‖xn − x‖ = 0.
It is denoted by xn → x .◮ (xn)n∈N converges weakly to x if
(∀y ∈ H) limn→+∞
〈y | xn − x〉 = 0.
It is denoted by xn ⇀ x .
Remark: xn → x ⇒ xn ⇀ x .In a finite dimensional Hilbert space, strong and weak convergences areequivalent.
![Page 18: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/18.jpg)
7/29
Convergence in Hilbert spaces
Let S be a subset of a Hilbert space H.
◮ S is bounded if it is included in a ball.
◮ S is closed if the limit of every converging sequence of elements ofS belongs to S .
◮ S is compact if, from every sequence (xn)n∈N of H, one can extract
a subsequence (xnk )k∈N which converges to a point of S .
◮ If S is compact, then it is closed and bounded.◮ The converse property holds, when H is finite dimensional.
![Page 19: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/19.jpg)
8/29
Limits inf and sup
Let (ξn)n∈N be a sequence of elements in [−∞,+∞].
Its infimum limit is lim inf ξn = limn→+∞ inf{ξk
∣∣ k ≥ n}∈ [−∞,+∞]
and its supremum limit is lim sup ξn = limn→+∞ sup{ξk
∣∣ k ≥ n}
∈
[−∞,+∞].
![Page 20: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/20.jpg)
8/29
Limits inf and sup
Let (ξn)n∈N be a sequence of elements in [−∞,+∞].
Its infimum limit is lim inf ξn = limn→+∞ inf{ξk
∣∣ k ≥ n}∈ [−∞,+∞]
and its supremum limit is lim sup ξn = limn→+∞ sup{ξk
∣∣ k ≥ n}
∈
[−∞,+∞].
◮ lim sup ξn = − lim inf(−ξn)
◮ limn→+∞ ξn = ξ ∈ [−∞,+∞] if and only if lim inf ξn = lim sup ξn = ξ.
![Page 21: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/21.jpg)
9/29
Epigraph
Let f : H → ]−∞,+∞]. The epigraph of f is
epi f ={(x , ζ) ∈ dom f × R
∣∣ f (x) ≤ ζ}
![Page 22: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/22.jpg)
9/29
Epigraph
Let f : H → ]−∞,+∞]. The epigraph of f is
epi f ={(x , ζ) ∈ dom f × R
∣∣ f (x) ≤ ζ}
x
f (x) = |x |
epif
xδ−δ
f (x) = ι[−δ,δ](x)
epif
![Page 23: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/23.jpg)
10/29
Lower semi-continuity
Let f : H → ]−∞,+∞].
f is a lower semi-continuous (l.s.c.) function at x ∈ H if, for every se-quence (xn)n∈N of H,
xn → x ⇒ lim inf f (xn) ≥ f (x).
![Page 24: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/24.jpg)
10/29
Lower semi-continuity
Let f : H → ]−∞,+∞].f is a lower semi-continuous function on H if and only if epi f is closed
◮ l.s.c. functions ?
x
f (x)
x
f (x)
![Page 25: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/25.jpg)
10/29
Lower semi-continuity
Let f : H → ]−∞,+∞].f is a lower semi-continuous function on H if and only if epi f is closed
◮ l.s.c. functions ?
f (x)
xx
f (x)
![Page 26: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/26.jpg)
10/29
Lower semi-continuity
Let f : H → ]−∞,+∞].f is a lower semi-continuous function on H if and only if epi f is closed
◮ l.s.c. functions ?
x
f (x)
x
f (x)
![Page 27: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/27.jpg)
10/29
Lower semi-continuity
Let f : H → ]−∞,+∞].f is a lower semi-continuous function on H if and only if epi f is closed
◮ l.s.c. functions ?
x
f (x)
x
f (x)
![Page 28: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/28.jpg)
10/29
Lower semi-continuity
Let f : H → ]−∞,+∞].f is a lower semi-continuous function on H if and only if epi f is closed
◮ l.s.c. functions ?
x
f (x)
x
f (x)
![Page 29: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/29.jpg)
11/29
Lower semi-continuity
◮ Every continuous function on H is l.s.c.
◮ Every finite sum of l.s.c. functions is l.s.c.
◮ Let (fi )i∈I be a family of l.s.c functions.supi∈I fi is l.s.c.
![Page 30: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/30.jpg)
12/29
Minimizers
Let S be a nonempty set of a Hilbert space H.Let f : S → ]−∞,+∞] be a proper function and let x ∈ S .
◮ x is a local minimizer of f if there exists an open neigborhood O ofx such that
(∀x ∈ O ∩ S) f (x) ≤ f (x).
◮ x is a (global) minimizer of f if
(∀x ∈ S) f (x) ≤ f (x).
![Page 31: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/31.jpg)
12/29
Minimizers
Let S be a nonempty set of a Hilbert space H.Let f : S → ]−∞,+∞] be a proper function and let x ∈ S .
◮ x is a strict local minimizer of f if there exists an open neigborhoodO of x such that
(∀x ∈ (O ∩ S) \ {x}) f (x) < f (x).
◮ x is a strict (global) minimizer of f if
(∀x ∈ S \ {x}) f (x) < f (x).
![Page 32: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/32.jpg)
13/29
Existence of a minimizer
Weierstrass theorem
Let S be a nonempty compact set of a Hilbert space H.Let f : S → ]−∞,+∞] be a proper l.s.c function such that dom f ∩S 6= ∅.Then, there exists x ∈ S such that
f (x) = infx∈S
f (x).
![Page 33: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/33.jpg)
14/29
Existence of a minimizer
Let H be a Hilbert space. Let f : H → ]−∞,+∞].f is coercive if lim‖x‖→+∞ f (x) = +∞.
![Page 34: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/34.jpg)
14/29
Existence of a minimizer
Let H be a Hilbert space. Let f : H → ]−∞,+∞].f is coercive if lim‖x‖→+∞ f (x) = +∞.
Coercive functions ?
x
f (x)
+∞
x
f (x)
+∞
x
f (x)
+∞
![Page 35: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/35.jpg)
14/29
Existence of a minimizer
Let H be a Hilbert space. Let f : H → ]−∞,+∞].f is coercive if lim‖x‖→+∞ f (x) = +∞.
Coercive functions ?
x
f (x)
+∞
x
f (x)
+∞
x
f (x)
+∞
![Page 36: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/36.jpg)
15/29
Existence of a minimizer
Theorem
Let H be a finite dimensional Hilbert space.Let f : H → ]−∞,+∞] be a proper l.s.c. coercive function.Then, the set of minimizers of f is a nonempty compact set.
![Page 37: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/37.jpg)
16/29
Convex set
C ⊂ H is a convex set if
(∀(x , y) ∈ C 2)(∀α ∈]0, 1[) αx + (1− α)y ∈ C
Convex sets ?
C CC
![Page 38: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/38.jpg)
16/29
Convex set
C ⊂ H is a convex set if
(∀(x , y) ∈ C 2)(∀α ∈]0, 1[) αx + (1− α)y ∈ C
Convex sets ?
C CC
![Page 39: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/39.jpg)
17/29
Minimizers over convex sets
Let H be a Hilbert space and let f : H → ]−∞,+∞] be a proper function.
f is Gateaux differentiable at x ∈ dom f if there exists ∇f (x) ∈ H suchthat
(∀y ∈ H) 〈∇f (x) | y〉 = limα→0α6=0
f (x + αy)− f (x)
α.
Theorem
Let C be a nonempty convex subset of a Hilbert space H. Let f : C →]−∞,+∞] be Gateaux differentiable at x ∈ C . If x is a local minimizer off , then
(∀y ∈ C ) 〈∇f (x) | y − x〉 ≥ 0.
If C is a vector space or x ∈ int (C ), then the condition reduces to
∇f (x) = 0.
![Page 40: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/40.jpg)
18/29
Convex function: definitions
f : H → ]−∞,+∞] is a convex function if
(∀(x , y) ∈ H2
)(∀α ∈]0, 1[)
f (αx + (1− α)y) ≤ αf (x) + (1− α)f (y)
![Page 41: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/41.jpg)
18/29
Convex function: definitions
f : H → ]−∞,+∞] is a convex function if
(∀(x , y) ∈ H2
)(∀α ∈]0, 1[)
f (αx + (1− α)y) ≤ αf (x) + (1− α)f (y)
Convex functions ?
x
f (x) = |x |
x
f (x) =√|x |
+∞ otherwise)
xδ−δ
f (x) = ι[−δ,δ](x)
(0 if x ∈ [−δ, δ]
![Page 42: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/42.jpg)
18/29
Convex function: definitions
f : H → ]−∞,+∞] is a convex function if
(∀(x , y) ∈ H2
)(∀α ∈]0, 1[)
f (αx + (1− α)y) ≤ αf (x) + (1− α)f (y)
Convex functions ?
x
f (x) = |x |
x
f (x) =√|x |
+∞ otherwise)
xδ−δ
f (x) = ι[−δ,δ](x)
(0 if x ∈ [−δ, δ]
![Page 43: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/43.jpg)
19/29
Convex functions: definition
f : H → ]−∞,+∞] is convex ⇔ its epigraph is convex.
![Page 44: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/44.jpg)
19/29
Convex functions: definition
f : H → ]−∞,+∞] is convex ⇔ its epigraph is convex.
x
f (x) = |x |
epif
x
f (x) =√|x |
epif
xδ−δ
f (x) = ι[−δ,δ](x)
epif
![Page 45: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/45.jpg)
19/29
Convex functions: definition
f : H → ]−∞,+∞] is convex ⇔ its epigraph is convex.
x
f (x) = |x |
epif
x
f (x) =√|x |
epif
xδ−δ
f (x) = ι[−δ,δ](x)
epif
![Page 46: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/46.jpg)
19/29
Convex functions: definition
f : H → ]−∞,+∞] is convex ⇔ its epigraph is convex.
x
f (x) = |x |
epif
x
f (x) =√|x |
epif
xδ−δ
f (x) = ι[−δ,δ](x)
epif
◮ If f : H → ]−∞,+∞] is convex, then dom f is convex.
◮ f : H → [−∞,+∞[ is concave if −f is convex.
![Page 47: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/47.jpg)
20/29
Convex functions: properties
◮ Every finite sum of convex functions is convex.
◮ Let (fi )i∈I be a family of convex functions. supi∈I fi is convex.
◮ Γ0(H) : class of convex, l.s.c., and proper functions from H to
]−∞,+∞].
◮ ιC ∈ Γ0(H) ⇔ C is a nonempty closed convex set.Proof: epiιC = C × [0,+∞[.
![Page 48: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/48.jpg)
21/29
Strictly convex functions
Let H be a Hilbert space. Let f : H → ]−∞,+∞].
f is strictly convex if
(∀x ∈ dom f )(∀y ∈ dom f )(∀α ∈]0, 1[)
x 6= y ⇒ f (αx + (1− α)y) < αf (x) + (1− α)f (y).
![Page 49: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/49.jpg)
21/29
Strictly convex functions
Let H be a Hilbert space. Let f : H → ]−∞,+∞].
f is strictly convex if
(∀x ∈ dom f )(∀y ∈ dom f )(∀α ∈]0, 1[)
x 6= y ⇒ f (αx + (1− α)y) < αf (x) + (1− α)f (y).
Strictly convex functions ?
x
f (x)
x
f (x)
x
f (x)
![Page 50: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/50.jpg)
21/29
Strictly convex functions
Let H be a Hilbert space. Let f : H → ]−∞,+∞].
f is strictly convex if
(∀x ∈ dom f )(∀y ∈ dom f )(∀α ∈]0, 1[)
x 6= y ⇒ f (αx + (1− α)y) < αf (x) + (1− α)f (y).
Strictly convex functions ?
x
f (x)
x
f (x)
x
f (x)
![Page 51: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/51.jpg)
22/29
Characterization of twice differentiable convex functions
Let H be a Hilbert space.Let f : H → ]−∞,+∞] be a twice (Frechet) differentiable function on itsdomain. Assume that dom f is a convex set.
◮ f is convex if and only if, for every x ∈ dom f ,
(∀z ∈ H)⟨z | ∇2f (x)z
⟩≥ 0.
◮ If, for every x ∈ dom f ,
(∀z ∈ H \ {0})⟨z | ∇2f (x)z
⟩> 0,
then f is strictly convex.
![Page 52: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/52.jpg)
23/29
Minimizers of a convex function
Theorem
Let H be a Hilbert space. Let f : H → ]−∞,+∞] be a proper convexfunction such that µ = inf f > −∞.
◮
{x ∈ H
∣∣ f (x) = µ}is convex.
◮ Every local minimizer of f is a global minimizer.
◮ If f is strictly convex, then there exists at most one minimizer.
![Page 53: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/53.jpg)
24/29
Existence and uniqueness of a minimizer
Theorem
Let H be a Hilbert space and C a closed convex subset of H. Let f ∈ Γ0(H)such that dom f ∩ C 6= ∅.If f is coercive or C is bounded, then there exists x ∈ C such that
f (x) = infx∈C
f (x).
If, moreover, f is strictly convex, this minimizer x is unique.
![Page 54: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/54.jpg)
25/29
Exercise 1
Provide an example of a function f : R → R and a nonempty set C ⊂ R
such that
◮ f is nonconvex
◮ C is convex
◮ f + ιC is convex.
![Page 55: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/55.jpg)
26/29
Exercise 2
1. Let f : H → ]−∞,+∞] be a convex function.
Prove that for every ζ ∈ R, the lower level set
lev≤ζ f ={x ∈ H
∣∣ f (x) ≤ ζ}
is convex.
2. Show that the converse is false by providing an example of anonconvex function the lower level sets of which are all convex.
![Page 56: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/56.jpg)
27/29
Exercise 3
Let A ∈ RM×N and z ∈ R
M . Let f : RN → R : x 7→ ‖Ax − z‖ and letg : RN → R : x 7→ ‖Ax − z‖2.
1. Prove that f and g are convex.
2. Give a necessary and sufficient condition on A for g to be strictlyconvex.
3. Can f be strictly convex ?
4. Find the minimizers of g .
5. What are the minimizers of f ?
![Page 57: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/57.jpg)
28/29
Exercise 4
Let y ∈ R. Show that
f : R → R : x 7→ log(1 + exp(−yx))
is convex. When is it strictly convex ?
![Page 58: Dataprocessingandnetworksoptimization PartII:Optimization ...perso.ens-lyon.fr/nelly.pustelnik/Cours_M2/pdf/cours_M2_systemCompl... · 2/29 Hilbert spaces A (real) Hilbert space H](https://reader033.vdocument.in/reader033/viewer/2022042003/5e6eb897621cbf3b1f0e4d59/html5/thumbnails/58.jpg)
29/29
Exercise 5
Let H be a Hilbert space and let f : H → ]−∞,+∞] be a convexfunction. Let g be the perspective function of f defined as
(∀(x , t) ∈ H× R) g(x , t) =
{t f (x/t) if t > 0
+∞ otherwise.
1. How is the epigraph of g related to the epigraph of f ?
2. Deduce that g is a convex function.
3. As a consequence of this result, show that the Kullback-Leiblerdivergence defined as(∀x = (x(i))1≤i≤N ∈ R
N)(∀y = (y (i))1≤i≤N ∈ R
N)
h(x , y) =
{∑N
i=1 x(i) ln(x(i)/y (i)) if (x , y) ∈ ( ]0,+∞[N)2
+∞ otherwise,
is convex.