subdifferential properties of quasiconvex and pseudoconvex ......introduced in aussel et al. (ref....

17
JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS: Vol. 97, No. I, pp. 29-45, APRIL 1998 Subdifferential Properties of Quasiconvex and Pseudoconvex Functions: Unified Approach1 D. AUSSEL2 Communicated by M. Avriel Abstract. In this paper, we are mainly concerned with the characteriza- tion of quasiconvex or pseudoconvex nondifferentiable functions and the relationship between those two concepts. In particular, we charac- terize the quasiconvexity and pseudoconvexity of a function by mixed properties combining properties of the function and properties of its subdifferential. We also prove that a lower semicontinuous and radially continuous function is pseudoconvex if it is quasiconvex and satisfies the following optimality condition: 0sdf(x) =f has a global minimum at x. The results are proved using the abstract subdifferential introduced in Ref. 1, a concept which allows one to recover almost all the subdiffer- entials used in nonsmooth analysis. Key Words. Nonsmooth analysis, abstract subdifferential, quasicon- vexity, pseudoconvexity, mixed property. 1. Preliminaries Throughout this paper, we use the concept of abstract subdifferential introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth analysis. The aim of such an approach is to show that a lot of results concerning the subdifferential prop- erties of quasiconvex and pseudoconvex functions can be proved in a unified way, and hence for a large class of subdifferentials. We are interested here in two aspects of convex analysis: the charac- terization of quasiconvex and pseudoconvex lower semicontinuous functions and the relationship between these two concepts. 1The author is indebted to Prof. M. Lassonde for many helpful discussions leading to a signifi- cant improvement in the presentation of the results. 2AssistantProfessor, Mathematiques Appliquees, Universite de Perpignan, Perpignan, France. 29 0022-3239/98/0400-0029$15.00/0 © 1998 Plenum Publishing Corporation

Upload: others

Post on 24-Aug-2021

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS: Vol. 97, No. I, pp. 29-45, APRIL 1998

Subdifferential Properties of Quasiconvex andPseudoconvex Functions: Unified Approach1

D. AUSSEL2

Communicated by M. Avriel

Abstract. In this paper, we are mainly concerned with the characteriza-tion of quasiconvex or pseudoconvex nondifferentiable functions andthe relationship between those two concepts. In particular, we charac-terize the quasiconvexity and pseudoconvexity of a function by mixedproperties combining properties of the function and properties of itssubdifferential. We also prove that a lower semicontinuous and radiallycontinuous function is pseudoconvex if it is quasiconvex and satisfiesthe following optimality condition: 0sdf(x) =f has a global minimumat x. The results are proved using the abstract subdifferential introducedin Ref. 1, a concept which allows one to recover almost all the subdiffer-entials used in nonsmooth analysis.

Key Words. Nonsmooth analysis, abstract subdifferential, quasicon-vexity, pseudoconvexity, mixed property.

1. Preliminaries

Throughout this paper, we use the concept of abstract subdifferentialintroduced in Aussel et al. (Ref. 1). This definition allows one to recoveralmost all classical notions of nonsmooth analysis. The aim of such anapproach is to show that a lot of results concerning the subdifferential prop-erties of quasiconvex and pseudoconvex functions can be proved in a unifiedway, and hence for a large class of subdifferentials.

We are interested here in two aspects of convex analysis: the charac-terization of quasiconvex and pseudoconvex lower semicontinuous functionsand the relationship between these two concepts.

1The author is indebted to Prof. M. Lassonde for many helpful discussions leading to a signifi-cant improvement in the presentation of the results.

2Assistant Professor, Mathematiques Appliquees, Universite de Perpignan, Perpignan, France.

290022-3239/98/0400-0029$15.00/0 © 1998 Plenum Publishing Corporation

Page 2: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

Throughout this paper, d stands for the following abstract notion ofsubdifferential.

Definition 1.1. See Ref. 1. We call subdifferential, denoted by d, anyoperator which associates a subset Sf(x) of X* to any lower semicontinuousf: X-**R u {+00} and any xeX, and satisfies the following properties:

(P1) df(x) = {x*eX*\<x*,y-xy+f(x)<f(y), VyeX},whenever f is convex;

(P2) Oedf(x), whenever xedom f is a local minimum of f;(P3) d(f+g)(x)<df(x) + 8g(x), whenever g is a real-valued convex

continuous function which is 3-differentiable at x.

Here, g is d-differentiable at x means that both dg(x) and d(-g)(x) arenonempty. We say that a function f is 3-subdifferentiable at x when df(x)is nonempty.

This abstract subdifferential allows one to recover, by a unique defini-tion, a large class of subdifferentials. Among this class, let us mention someexamples (for precise definitions, see Ref. 1 and references therein):

the Clarke-Rockafellar subdifferential dCRf;the lower and upper Dini subdifferentials <3D- f and <3D+ f;

30 JOTA: VOL. 97, NO. 1, APRIL 1998

The paper is organized as follows. Section 2 is devoted to the study ofquasiconvex functions. We establish a characterization based on a mixedproperty combining properties of the function and properties of its subdiff-erential. The case of quasiconcave and quasiaffine functions, for which anal-ogous characterizations are obtained, is treated in Section 3. In Section 4,we generalize to the nonsmooth setting a result of Crouzeix and Ferland(Ref. 2) which describes the precise relation between quasiconvexity andpseudoconvexity. Moreover, this allows one to prove in an elegant way twocharacterizations of pseudoconvexity. Finally, we point out in Section 5 thatthe continuity assumption can be replaced in some results by the radiallynonconstant hypothesis, already used by Diewert in 1981 (Ref. 3).

In all the sequel, X denotes a Banach space, X* its topological dual,and < •, • > the duality pairing. Most functionsf: X-+R u { + 00} consideredin this paper will be (at least) lower semicontinuous (l.s.c. in short), anddom f will denote the domain off

For a point-to-set map A: X-+X*, we let

Page 3: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

the lower Hadamard subdifferential dH- f, also called contingentsubdifferential;the Frechet subdifferential dF f;the Lipschitz smooth subdifferential dLS f, also called proximal subdiff-erential d" f whenever X is a Hilbert space.

The verification that these subdifferentials satisfy properties (P1) to (P3) isstraightforward (see Ref. 4) and does not require any assumption on theBanach space. The use of the word d-differentiable is justified by the follow-ing observation:

a function is <3°~-differentiable

[resp. dH--differentiable, dF-differentiable]

at x if and only if it is Gateaux [resp. Hadamard, Frechet]

differentiable at x.

In all the sequel, we will focus our attention on the class of subdifferen-tials which satisfy properties (P1) to (P3) and one of the following inclusions:

Let us recall the definition of the Clarke-Rockafellar subdifferential and thedefinition of the upper Dini subdifferential which will be helpful for thesequel,

(one can take «=f(u) whenever f is l.s.c.);

JOTA: VOL. 97, NO. 1, APRIL 1998 31

This assumption is not needed in all the proofs and in any case is not reallyrestrictive, since the Clarke-Rockafellar and the upper Dini subdifferentialsare the biggest (in the sense of inclusion) among the classical subdifferentials.In particular, one has

Page 4: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

Definition 1.2. See Ref. 1.3 A norm || • || on X is said to be d-smoothif all the (real-valued, convex, continuous) functions of the following formare 3-differentiable:

(i) df[a,b](x) •= min,6[a,fc] ||x-c||2,

where [a, b] is a closed segment in X,(ii) A2(x):=I.nnn\\x-vn\\2,

where 2 n J u M = 1, ^«>0, and (i>«) converges in X.

We say that a Banach space admits a d-smooth renorm if it admits anequivalent norm which is 3-smooth. Let us give some interesting examplesof such d-smooth norms:

(a) a norm is ^"-smooth iff it is dfl~-differentiable on ^f\{0}, thatis, according to a previous remark, iff it is Gateaux-differentiableon X\{0] (elementary proof);

(b) any norm is dc/J-smooth since the functions rff0ifc] and A2 are locallyLipschitz.

Concerning (a), we note that the same equivalence, with respect toHadamard or Frechet differentiability, is true for d — dH- or d = dF. So, anyseparable [resp. reflexive] Banach space admits a <3D--smooth [resp. dF-smooth] renorm; see, e.g., Ref. 6.

Finally, we need the following result which is a special case of the mean-value inequality stated in Ref. 1.

Proposition 1.1. Let X be a Banach space with a 3-smooth renorm,and let f:X-»R u {+00} be a l.s.c. function. For any aedom f and beXsuch that f(a)<f(b), there exist ce[a, b[ and sequences (xn) converging toc and (x*), x*edf(xn), such that

2. Two Characterizations of Quasiconvexity

This section is devoted to the study of quasiconvex functions. Moreprecisely, we extract from the proof of the main result of Aussel et al. (Ref. 7,Theorem 1) a mixed property (Qs) combining properties of the function

3Since this paper was submitted, J. N. Corvellec, M. Lassonde, and the author have shownthat (i) can be deduced from (ii); see Proposition 2.3 in Ref. 5.

JOTA: VOL. 97, NO. 1, APRIL 199832

for every d=c + t(b-a) with t>0.

Page 5: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

and properties of the subdifferential. Using the same arguments as in Ref. 7,Theorem 1, we show that this mixed property characterizes the quasi-convexity of l.s.c. functions. This characterization will play a central role inthe sequel.

Let us recall that a function is quasiconvex if, for any x, yeX and anyz e [ x , y ] , one has

JOTA: VOL. 97, NO. 1, APRIL 1998 33

It is well known that, in the differentiable case, quasiconvex functions satisfy

In the nondifferentiable case, this property becomes

Our first result asserts that the following slightly stronger mixed propertyactually characterizes the quasiconvexity of l.s.c. functions:

Theorem 2.1. Let X be a Banach space with a 5-smooth renorm, andlet f: X ->R u { + 00} be a l.s.c. function. Then, the following assertions areequivalent:

(i) f is quasiconvex;

Proof. (i) =» (ii) Case dfcdCRf. Let x,yedom df and x*edf(x)such that

Hence, there exists e > 0 such that, for every ne N*, one can find xneBe/n(x)[and then y — xneBe(y — x)] and tne ]0, 1/n[ which satisfy

Since f is quasiconvex, the previous inequality implies that, for every te [0, 1 ],we have

and hence, by lower semicontinuity of f,

(i) => (ii) Case df c BD+f. The proof is very simple. Indeed, if x, yand x*edD+f(x) satisfy fD+(x,y-x)>0, then f(z)>f(x) for some

Page 6: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

It satisfies property (Q) for d = dCR but is not quasiconvex.In previous works, Hassouni (Ref. 8, locally Lipschitz function) and

Penot and Quang (Ref. 9, continuous function) characterize quasiconvexityusing property (0. But property (Qs) is proved to be the good expressionin the more general setting of l.s.c. functions.

In the following result, we show that, for a continuous function orradially continuous function (that is, continuous on each segment), bothproperties are equivalent.

Proposition 2.1. Let X be a Banach space with a d-smooth renorm.Every l.s.c. radially continuous function which satisfies (0 is quasiconvex.

Proof. Let x, yeX, and let ze]x, y[ be such thatf(z)>max[f(x), f ( y ) ] . Applying Proposition 1.1 to the points x and z, weobtain two sequences (an) and (a*), with an-+ae[x, z[, a*edf(an), and<a*,c —an„>>(), for every neN and every ce[z,y]. Then, property (0implies that f(an)<f(c), and hence, using the lower semicontinuity of f,f(a)<minc6[:j,] f ( c ) . By the same argument, the inequality f(z) >f(y)implies the existence of a point b in ]z, y] such that f(b)<min<.6[x,_-| f ( c ) .Consequently, we have

34 JOTA: VOL. 97, NO. 1, APRIL 1998

ze ]x, y[, and by quasiconvexity of f,

(ii) => (i) We essentially reproduce the first part of the proof of Ref.7, Theorem 1. Let x, yedom f and

According to Proposition 1.1, there exist sequences (xn) and (x*) such thatxn->xe[x, z[, x*edf(xn), and <x*,y-xn>>0, VneM. Hypothesis (ii)implies that, for every neN, the point zn defined on ]xn,y] by zn =Axn + ( l -A)y satisfies f ( z n ) < f ( y ) , and hence by lower semicontinuityf(z)<f(y). D

Obviously, every l.s.c. quasiconvex function also satisfies property (0.But the converse is false, in general, for l.s.c. functions. Consider for examplethe function f defined by

Page 7: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

The equivalence between the quasiconvexity of a function and thequasimonotonicity of its subdifferential has been studied in Aussel et al.(Refs. 7, 1) for d e d C R and in Luc (Ref. 10) for d = d C R . The aim of the nexttwo results is to show that the implication

f quasiconvex => df quasimonotone

can be stated without any regularity assumption on f and that the reverseimplication can be seen as a consequence of Theorem 2.1.

Proposition 2.2. Let X be a Banach space. Then, the Clarke-Rockafel-lar subdifferential and the upper Dini subdifferential of any quasiconvexfunctionf: X-*R u {+00} are quasimonotone.

Proof. The case of the Clarke-Rockafellar subdifferential consists ofa slight refinement of the second part of the proof of Ref. 7, Theorem 1.We include the proof for completeness. Suppose that f is quasiconvex, andlet x, yedom dCR f and x*edCR f(x) be such that <x*,y-x>>0. It issufficient to prove that fT(>>, x -y) < 0. For every e > 0, there exists 7 e ]0, e[such that <x*. v-x>>0, VveBr(y). Let fix veBr(y). Since f\x, v-x)is strictly positive, there exist e'e]0, e-yl, u t € B f ( x ) , a e B f ( f ( x ) ) , andre ]0, 1[ which satisfy

JOTA: VOL. 97, NO. 1, APRIL 1998 35

The case d = dCR is given in Penot and Quang (Ref. 9).A point-to-set map A: X-+X* is quasimonotone if, for any x, yeX, the

following holds:

According to Proposition 1.1 applied to point a = a + t(z-a)e ]a, z[ andpoint y, and using property (0, there exists a'e[a, z[ such thatf(a') <f(b).The desired contradiction is obtained, since

The functionfbeing radially continuous, there exists

These inequalities, according to the quasiconvexity assumption, yield

Moreover, from the choice of y and e', the direction UB - v is an element ofBf(x-y). Thus, summing up the previous steps, for any e>0, there existsy>0 such that, for any veBr(y), any /Je5y(f(>0)> f(v)<P, and any

Page 8: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

Consequently, if x*edD+f(x) is such that <x*,>>-.x>>0, we immediatelyobtain thatf(x) <f(y), and hence that fD+(y, x-y)<0, thus showing thatdD+ f is a quasimonotone point-to-set map. D

Theorem 2.2. Let X be a Banach space with a <3-smooth renorm, andlet f: .X-Ru {+00} be a l.s.c function. Then, f is quasiconvex iff df isquasimonotone.

Proof. Since the abstract subdifferential df is assumed to be includedin dCR for SD+ f, the "only if" part follows from Proposition 2.2.

To prove the "if" part, we follow the lines of the second part of theproof of Ref. 1, Theorem 5.4. Let us assume that df is quasimonotone.According to Theorem 2.1, we just have to show that the l.s.c. functionfsatisfies property (Qs). Let xedom df, jedom f(x^y), and ze[x,y[ suchthat f(z)>f(y). Applying Proposition 1.1 to y and z, we get a sequence(yn) converging to ye ]z,y] and a sequence (y*) such that y * e d f ( y n ) and

36 JOTA: VOL. 97, NO. 1, APRIL 1998

te]0, 1[, one can find a direction w = uv — veBe(x — y) such that

which clearly implies the desired inequality f1(y, x-y)<,0.For the case of the upper Dini subdifferential, as an immediate conse-

quence of the quasiconvexity of the function f, we have

or equivalently that

By quasimonotonicity of df, we have

for all n and all x * e d f ( x ) . Then,

Thus, the function f satisfies property (Qs), and the proof is complete. D

3. Quasiconcave and Quasiaffine Functions

In this section, we consider the case of quasiconcave and quasiaffinefunctions for which analogous mixed characterizations are proved usingappropriate mixed properties.

Page 9: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

The characterization of the quasiconcavity of a function f by the property(27) cannot be deduced in general from Theorem 2.1. Indeed, consideringthe function (—f) in place of a functionfin Theorem 2.1 yields a charac-terization of the quasiconcavity of f in terms of —3(—f), which in generalis different from df. The function f(x) = ^/\x\ is a standard counterexamplefor dCR, since dCR f(0) = R and dcr(-f)(0) = 0; [however, the subdifferen-tial equality dCR f=-dCR(-f) is true if f is assumed to be locally Lipschitz(see, e.g., Ref. 11).

Proposition 3.1. Let X be a Banach space with a 5-smooth renorm,and let f: X->R u {+00} be a continuous function. Then,f is quasiconcaveif and only if, for any x, y of X, the function f satisfies

A function f is said to be quasiconcave if -f is quasiconvex and it issaid to be quasiaffine if f and -f are quasiconvex.

Let us consider the following mixed property

JOTA: VOL. 97, NO. 1, APRIL 1998 37

Proof. It follows the same line as the proof of Theorem 2.1. Let usassume that f satisfies property ( Q - ) and that x, yeX and ze[x, y[ are suchthat f(z) <f(y). From Proposition 1.1, there exist a sequence (an) converg-ing to ae[z,y[ and a sequence (a*), a*edf(an), which verify

Let t1 and t2 be some positive numbers, with 0<t1 < t2, such that

and define two sequences (xn) and (zn) by

For n large enough, we have <a*, xn - an > < 0, and hence according to prop-erty ( Q - ) , f ( z n ) > f ( x n ) . Finally, since the functionfis continuous, we getf(*)>f(*).

Conversely, assume that f is quasiconcave. Let xedomdf, yeX,ze[x, y], and x*edf(x) such that <x*, y-x) <0. lfdfc:dCKf, then the pointsx and y satisfy fT(x, x->')>0, and hence there exists e>0 such that, forany n e N , one can find xneBf/n(x) and tne]Q, \/n[ satisfying

Page 10: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

and hence, sincefis upper semicontinuous,f(z) >f(>>).Ifdf<=dD+f, then we havefD+(x, x—y)>0, and hence for every «, there

exists tne ]0, \fn[ verifyingf(x + tn(x-y)) >f(x). But f is quasiconcave andxe]x + tn(x-y),y[, for all «. Consequently f ( z ) > f ( y ) andfsatisfies prop-erty (fiT). D

Let us mention that one can easily find counterexamples which showthat, for the "if" part of the proposition, the continuity hypothesis cannotbe replaced by a lower or upper semicontinuity hypothesis. Consider forexample the indicator function of the set {0, 1} or the indicator function ofthe set R\{0}.

As an immediate consequence of Theorem 2.1 and Proposition 3.1, weobtain the following characterization of quasiamne functions.

Corollary 3.1. Let X be a Banach space with a 3-smooth renorm, andletf: A r-+Ru{+oo}bea continuous function. Then, the following assertionsare equivalent:

(i) fis quasiaffine;

4. Pseudoconvexity

The original definition of pseudoconvexity by Mangasarian in thedifferentiable setting was extended by many authors in the following way

JOTA: VOL. 97, NO. 1, APRIL 199838

For any n, the two points xn and zn = kxn + (1 - A )y [with A defined by z =Ax + (l -A)y] are on the segment line ]xn + tn(xn—y),y[. Using the quasicon-cavity off we have

Indeed, the conjunction of the mixed properties (Qs) and (Qs ) is equivalentto

3x*edf(x): <**, rf> > 0=> fXid: t*-*f(x + td) is nondecreasing on K,

which is exactly assertion (ii). Another equivalent way to describe assertion(ii) of the corollary is

Using the corresponding expression in the differentiable case, Martos (Ref.12) characterized quasiaffine functions in finite-dimensional spaces.

Page 11: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

JOTA: VOL. 97, NO. 1, APRIL 1998 39

(see for instance Ref. 9): A functionfis said to be pseudoconvex if, for anyx, yeX, one has

3x*edf(x):<x*,y-xy*Q =>f(*) £f(?)•

As in the differentiable case, every pseudoconvex function f satisfies thefollowing fundamental properties:

(a) every local minimizer offis a global minimizer;(b) Oedf(x) => f has a global minimum at x.

Unfortunately, the relation between quasiconvexity and pseudoconvexity isnot simple. For example, the function f ( x ) = x3. is quasiconvex and notpseudoconvex, whereas the function denned by

Proof, (i) =*• (ii). From the definition of pseudoconvexity, it is obvi-ous that, if Oedf(x), then

is pseudoconvex, with e.g. d = dCR, but not quasiconvex.Extending a result of Crouzeix and Ferland (Ref. 2) stated for differen-

tiable functions, the following theorem gives the precise relation betweenpseudoconvexity and quasiconvexity for l.s.c. radially continuous functions.

Theorem 4.1. Let X be a Banach space with a ^-smooth renorm, andletf: X->Rv {+00} be a l.s.c. and radially continuous function. Then, thefollowing assertions are equivalent:

(i) fis pseudoconvex;(ii) fis quasiconvex and (Qedf(x) => fhas a global minimum at x).

So x is a global minimum of the functionf. On the other hand, the functionfis l.s.c., radially continuous and satisfies property (Q) of Section 2, sinceevery pseudoconvex function satisfies property (Q). Then, from Proposition2.1,f is quasiconvex.

(ii) => (i). Let xedomdf, yeX, and x*edf(x) such that<x*, y — xy ^0. If Qedf(x), then x is a global minimum offand in particularone has f(x)<f(y). Otherwise [Q$df(x)], there exists deX such that<x*, dy>0. Now let us define the sequence (yn) by

Page 12: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

Theorem 4.2. Let X be a Banach space with a d-smooth renorm, andletf: A"-*Ru {+°o} be a l.s.c. and radially continuous function. Then, thefollowing assertions are equivalent:

We claim that 0 is not an element of df(y).Indeed, since f](x,y-x)>0, there exist e>0, x'eBc(x), and re]0, 1[

such that

40 JOTA: VOL. 97, NO. 1, APRIL 1998

For every neN, the point yn satisfies

Using Theorem 2.1, we obtain that, for every n,f(yn)>f(x) and by radialcontinuity o f f , f ( y ) > f ( x ) . D

In finite dimensions, for d = dD+ and for an upper semicontinuous func-tion, the implication (ii) => (i) is given in Komlosi (Ref. 13).

Now using the relation between quasiconvexity and pseudoconvexity(Theorem 4.1) and the characterization of quasiconvexity by the quasimono-tonicity of its subdifferential (Theorem 2.2), it is possible to give two charac-terizations of l.s.c. radially continuous pseudoconvex functions.

Let us recall that a point-to-set map A: X->X* is said to bepseudomonotone if, for any x, yeX, one has

Proof, (i) => (ii). Let jcedom df, yeX, and x*edf(x) such that<**,y-x}>0. From the definition of pseudoconvexity, it follows that

f(x)<f(y). But, according to Theorem 4.1, the functionfis quasiconvex.Consequently, for every z e [ x , y ] , f ( z ) < f ( y ) ,

(ii) => (i). This is obvious.(i) => (Hi) Case df<= dCRf. Let us suppose, for a contradiction, that f

is pseudoconvex and that dfis not pseudomonotone. This means that thereexist x, _yedom df, x*edf(x), and y*edf(y) such that

Page 13: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

From Theorem 4.1, the function f is quasiconvex and then we havef ( x ' ) < f ( y } - Since f is pseudoconvex, this last inequality implies that, forevery aedf(y), the real <(a,y-x'y is strictly positive, which proves theclaim.

Now, let us observe that there exists r\ > 0 such that <x*, u — x> > 0,for every ueB^(y). Then, the pseudoconvexity offimplies that, for everyueBr,(y),f(u)>f(*). Since <y*, x-y}>0, we also obtain thatf(x) > f ( y ) .Consequently, y is a local minimizer of f and from property (P2), 0 is anelement of df(y), which contradicts the claim.

(i) => (iii) Case dfc. dD+f. We use a direct proof. Let x e dom df, y eX,and x*edf(x) with <x*, ^-x>>0. Then, there exists re]0, 1[ such thatf(x + T(y — x))>f(x). Sincefis quasiconvex (by Theorem 4.1), we obtainf(y) >f(x). Now, the pseudoconvexity of the function implies that, for everyy*edf(y), we have <f*, x-y}<Q, thus proving that 3fis pseudomonotone.

(iii) =*• (i). Using Theorem 4.1 again, we will prove thatfis pseudo-convex. Indeed, the point-to-set map df is pseudomonotone, and hencequasimonotone. According to Theorem 2.2, the functionfis then quasicon-vex. On the other hand, if x is not a minimizer off there exists yeX suchthat f(y)<f(x). Applying Proposition 1.1, one can find aedomdfanda*edf(a) such that <a*, x-a) >0 and then, by the pseudomonotonicity ofdf, <x*, x-a>>0 for every x*edf(x). Hence, 0 is not an element of df(x).Consequently, f satisfies

5. Radially Nonconstant Functions

Following Komlosi (Ref. 14), we say that a function f is radially non-constant (r.n.c. for short) if one cannot find any line segment on which f isconstant; i.e.,

The purpose of this section is to give a different light to some of thepreceding results by proving them under the r.n.c. assumption, instead ofthe continuity (or radial continuity) assumption.

JOTA: VOL. 97, NO. 1, APRIL 1998 41

and the proof is complete. D

The idea of the proof of (i) => (iii) for dCR is due to Penot and Quang(Ref. 9).

Page 14: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

and hence by pseudoconvexity f(x)-f(y)<f(z).The function f beingr.n.c., there exists ze ]x, y[ such thatf(z) >f(x) =f(y). Let Ae ]f(*),f(£)[.By semicontinuity off, we havef(H)>A for all u in a neighborhood Fofz. Let ;ce ]x, z[ be such that ]x, z[c Kn ]x, z[. Sincefis r.n.c., there existsae ]x, z[ satisfyingf(a) ^f(z). Now, let us suppose thatf(a) <f(z). Accord-ing to Proposition 1.1, there exist a sequence (bn~) converging to be[a, z[and a sequence (£>*), b*edf(bn) for which

Since f is pseudoconvex and l.s.c., it follows that f(y)>f(b). Butbe]x, z[<=V, and thenf(6)>A>f(j). Consequently, we havef(a)>f(z).Using the same arguments one can prove that this is impossible too, thussupplying the desired contradiction. D

As an immediate consequence of Proposition 5.1, we obtain the follow-ing relation between pseudoconvexity and quasiconvexity.

Corollary 5.1. Let X be a Banach space with a d-smooth renorm.Every l.s.c., radially nonconstant, and pseudoconvex function isquasiconvex.

Proof. Iffis a l.s.c., r.n.c., and pseudoconvex function, then accordingto Proposition 5.1, its subdifferential is pseudomonotone and hencequasimonotone. By Theorem 2.2, the function is then quasiconvex. D

42 JOTA: VOL. 97, NO. 1, APRIL 1998

Proposition 5.1. Let X be a Banach space with a 3-smooth renorm,and letf: X-^Ru { + 00} be a l.s.c. and radially nonconstant function. Iffis pseudoconvex, then df is pseudomonotone.

Let us remark that Proposition 5.1 does not allow one to recover theimplication (i) => (iii) of Theorem 4.2, since there exists continuous (andconvex) functions which are not radially nonconstant.

Proof. Assume, for a contradiction, that f is pseudoconvex and thatdf is not pseudomonotone. So, there exist x, yedomdf, x*edf(x), andy*edf(y) such that <x*, >>-x>>Oand (y*,y-x><0. Sincefis pseudocon-vex, these inequalities imply thatf(^)^f(x) andf(*)>f(7). Moreover,for any ze ]jc, y], we have

Page 15: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

Obviously every strictly quasiconvex [resp. strictly pseudoconvex] func-tion is quasiconvex [resp. pseudoconvex]. Diewert (Ref. 3) observed that,conversely, every quasiconvex r.n.c. function is strictly quasiconvex.

A consequence of the next result is that the same holds true for pseudo-convex functions.

Proposition 5.2. Let X be a Banach space with a 3-smooth renorm,and letf: X-*R u {+00} be a l.s.c. function. Then, the following assertionsare equivalent:

(i) fis pseudoconvex and radially nonconstant;(ii) f is strictly quasiconvex and strictly pseudoconvex.

Proof, (i) => (ii). According to Corollary 5.1, the radially noncon-stant function f is quasiconvex and hence strictly quasiconvex. To provethatfis strictly pseudoconvex, let us assume, for a contradiction, that thereexist xedom df, yedomf, and x*edf(x) such that <;c*,_y-x>^0 andf(x)^.f(y). Consequently, for any ze]x,y], we have <x*, z-xy^.0, andhence f(z)>f(x) =f(y). But this contradicts the strict quasiconvexity off.

(ii) =*• (i). This is obvious, since every strictly quasiconvex functionis radially nonconstant. D

6. Final Remarks

(i) Throughout this paper, two different assumptions are made. Thefirst one is the inclusion of the subdifferential operator in the Clarke-Rock-afellar or upper Dini subdifferential [Assumption (H)]. The other one, isthe existence of a d-smooth renorm.

A swift reading is sufficient to mention that these hypotheses are notnecessary in all the proofs. They are, in a sense, complementary. Indeed, inmost equivalence proofs, one hypothesis allows to show an implication andthe other is used to prove the converse.

JOTA: VOL. 97, NO. 1, APRIL 1998 43

We say that a functionf:X-»R u{+oo} is:

(a) strictly quasiconvex iff, for any x, yeX and any ze ]x, y[, one has

(b) strictly pseudoconvex iff, for any x, yeX, the following holds:

Page 16: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

To be more precise, a direct or indirect use of Proposition 1.1 requiresthe existence of a 3-smooth renorm, whereas Assumption (H) is necessaryfor proofs based on the definition of dD+ or BCR.

(ii) Another way to characterize generalized convexity and to describethe relationships between different kinds of generalized convexity is to usecriteria and properties based on the notion of generalized derivative insteadof the subdifferential. In recent works [Komlosi (Refs. 14, 15), Penot (Ref.16), etc.], some abstract approach with generalized derivatives are developed.Since some subdifferentials covered by our abstract definition are not definedby a generalized derivative (the Frechet and proximal subdifferentials, forexample), these two approaches have to be considered as complementarypoints of view.

References

1. AUSSEL, D., CORVELLEC, J. N., and LASSONDE, M., Mean-Value Property andSubdifferential Criteria for Lower Semicontinuous Functions, Transactions of theAmerican Mathematical Society, Vol. 347, pp. 4147-4161, 1995.

2. CROUZEIX, J. P., and FERLAND, J. A., Criteria for Quasiconvexity and Pseudo-convexity: Relationships and Comparisons, Mathematical Programming, Vol. 23,pp. 193-205, 1982.

3. DIEWERT, W. E., Alternative Characterizations of Six Kinds of Quasiconcavityin the Nondifferentiable Case with Applications to Nonsmooth Programming, Gen-eralized Concavity in Optimization and Economics, Edited by S. Schaible andW. T. Ziemba, Academic Press, New York, New York, pp. 51-93, 1981.

4. AUSSEL, D., Theoreme de la Valeur Moyenne et Convexite Generalisee en AnalyseNon Reguliere, PhD Thesis, Universite Blaise Pascal, Clermont II, France, 1994.

5. AUSSEL, D., CORVELLEC, J. N., and LASSONDE, M., Nonsmooth ConstrainedOptimization and Multidirectional Mean- Value Inequalities, Research Report 96/04, Universite des Antilles et de la Guyane, 1996.

6. DEVILLE, R., GODEFROY, G., and ZIZLER, V., Smoothness and Renormings inBanach Spaces, Longman Scientific and Technical, London, England, 1993.

7. AUSSEL, D., CORVELLEC, J. N., and LASSONDE, M., Subdifferential Charac-terization of Quasiconvexity and Convexity, Journal of Convex Analysis, Vol. 1,pp. 195-201, 1994.

8. HASSOUNI, A., Sous-Differentiels des Fonctions Quasiconvexes, PhD Thesis,Universite Paul Sabatier, Toulouse, France, 1983.

9. PENOT, J. P., and QUANG, H. P., On Generalized Convexity of Functions andGeneralized Monotonicity of Set- Valued Maps, Journal of Optimization Theoryand Applications, Vol. 92, pp. 343-356, 1997.

10. Luc, D. T., Characterizations of Quasiconvex Functions, Bulletin of the Austral-ian Mathematical Society, Vol. 48, pp. 393-405, 1993.

44 JOTA: VOL. 97, NO. 1, APRIL 1998

Page 17: Subdifferential Properties of Quasiconvex and Pseudoconvex ......introduced in Aussel et al. (Ref. 1). This definition allows one to recover almost all classical notions of nonsmooth

11. ROCKAFELLAR, R. T., Generalized Directional Derivatives and Subgradients ofNonconvex Functions, Canadian Journal of Mathematics, Vol. 32, pp. 257-280,1980.

12. MARTOS, B., Nonlinear Programming: Theory and Methods, North-Holland,Amsterdam, Holland, 1975.

13. KOMLOSI, S., Some Properties of Nondifferentiable Pseudoconvex Functions,Mathematical Programming, Vol. 26, pp. 232-237, 1983.

14. KOMLOSI, S., Generalized Monotonicity and Generalized Convexity, Journal ofOptimization Theory and Applications, Vol. 84, pp. 361-376, 1995.

15. KOMLOSI, S., Monotonicity and Quasimonotonicity in Nonsmooth Analysis,Recent Advances in Nonsmooth Optimization, Edited by D. Z. Du, L. Qi, andR. S. Womersley, World Scientific Publishers, Singapore, pp. 101-124, 1994.

16. PENOT, J. P., Generalized Convexity in the Light of Nonsmooth Analysis, RecentDevelopments in Optimization, Lecture Notes in Economics and MathematicalSystems, Springer, Berlin, Germany, Vol. 429, pp. 269-290, 1995.

JOTA: VOL. 97, NO. 1, APRIL 1998 45