norms and inner products

7/31/2019 Norms and Inner Products

1/4

Norms and inner products

Vector norms

A vector norm on a (real or complex) vector space V is a measure x of the size of the vector x. Anorm must satisfy the following simple axioms:

1. Ifx = 0 then x > 0.2. cx = |c| x.3. x + y x + y.

The last axiom is called the Triangle Inequality, because in the usual picture of vector addition x, y, andx + y are the sides of a triangle.

x

+ y

y

x

On Rn or Cn there is a family of important examples of vector norms, namely the p-norms:

xp =

|xi|pp

, ifp 1. (1)

When p = 2 this is the familiar euclidean norm (distance). As p in (1) we get the -norm or themax-norm:

x = max |xi| . (2)Here, for comparison of the norms, we picture the sets {x | xp = 1}, for the three most importantp-norms: p = 1, 2, and .

x2

|| || = 1x1

|| || = 1x

|| || = 1

With each norm we get different estimates of closeness, but not different notions of convergence.That is, if a sequence of vectors converges in one norm that is, the norm of the difference betweenthe sequential terms and the limit tends to 0 then the sequence converges to the same limit in everyother vector norm.

This is because of the following fact: If and are two different norms on a finite dimensionalspace then there are constants c and c such that, for all x we have the inequalities

x c x and x c x . (3)For example, for the p-norms we have the following specific constants:

x2

x1

n x2

x x2 n x

x x1 n x(4)

1


2/4

If all norms on a finite-dimensional vector space are equivalent, then why is there more than onenorm? In applications, the specific estimates afforded by one norm may lead to better results than thoseafforded by another. For example, in image processing it has been observed in many many experimentalresults that estimates using the 2-norm lead to poorer quality than those using the 1-norm. This mayseem paradoxical, but bear in mind that in practice the limit is only a theoretical construct. Youalways have to make do with some particular approximation. The measure of how good a particular

approximation is depends very much on your means of measurement.By the way, the p-norms can be defined on spaces of functions, using the integral instead of the sum.

This is a very useful application of the idea of norms, especially to differential equations and numericalintegration. However the different norms are not equivalent to one another on these spaces. A sequenceof functions may converge in one norm but not another. We can see a hint of this in the inequalities (4)above: the constants depend on the dimension.

Inner products

The 2-norm is associated with a more refined measure of closeness, the inner product. The motivatingexample of an inner product is the familiar dot product from calc III a function of 2 variables fromwhich we can compute both length and angle. So, if x, y Rn then we learned in calc III that

yTx =

xiyi = x2 y2 (x, y). (5)

In particular, x2

=xTx. To generalize this when we have vectors in Cn we use the formula

yx =

xiyi. (6)

Although this is not always real and hence the latter part of formula (5) does not work we do stillhave that x

2=

xx for x Cn.

More generally, we define an inner product on a (real or complex) vector space V to be a function oftwo variables x, y satisfies the following axioms:

1.

x, y

=

y, x

.

2. x + y, z = x, z + y, z and x, y + z = x, y + x, z.3. cx,y = c x, y and x,cy = c x, y.4. Ifx = 0 then x, x > 0.

Note that ifV is a real vector space then the complex conjugate is to be ignored, since the conjugate ofa real number is real (and conversely).

One of the most important consequences of these axioms is the Cauchy-Schwarz Inequality:

|x, y|2 x, x2 y, y2 . (7)

As with the dot product, we can use an inner product to define a norm, by the rule

x =

x, x (8)

Even tho all norms are equivalent from the point of view of convergence, not all norms can be definedby an inner product. So, even tho norms are all analytically equivalent, they are geometrically quitedifferent as we saw in figure 2 above. A geometric property of norms which distinguishes those definedby inner products from all other norms is the Parallelogram Law:

x + y2 + x y2 = 2 x2 + 2 y2 . (9)

Let me stress again that this law is valid for the 2-norm, but not for any other p-norm. We can picturethis as saying that the sum of the squares of the diagonals of a parallelogram equals the sum of thesquares of the four sides:

2


3/4

y

+ y

x y

y

x

x

x

In a (real or complex) vector space with an inner product we say that vectors x and y are orthogonal(or perpendicular) in case their inner product is 0. We write x y in this case. For a real vector spaceorthogonality really means what you imagine: the vectors are at right angles to one another. Howeverin a complex vector space it has a slightly more subtle meaning. For example, in C1 that is, the usualcomplex plane the inner product of two complex numbers x and y is simply xy, which is never 0unless either x or y is. On the other hand C1 can also be pictured as R2. The angle between x and y asreal vectors is the real part ofxy. This phenomenon persists in higher dimensions: the real inner productof vectors x, y Cn is the real part of their complex inner product. Hence ifx y as complex vectorsthen certainly x y as real vectors, but the latter condition is in general weaker than the former.

Matrix norms

The set of (real or complex) matrices of a given size is a (real or complex) vector space. If we considermatrices of size m n, say, then this is a vector space of dimension mn. Hence it makes sense to talkabout matrix norms. However, there is more to the algebra of matrices than simply addition and scalarmultiplication. There is also matrix multiplication, and we want out matrix norms to reflect this.

More precisely, by a matrix norm we mean a norm (in the sense above) on the space of matrix whichsatisfies the additional axiom

AB A B . (10)Here are two simple examples: the max norm

|A| = maxi,j

|Ai,j | ; (11)

and the Frobenius norm AF =

i,j

|Ai,j |2. (12)

In the context of vector norms we used the notation and 2 for these definitions, but in thecontext of matrix norms we reserve this notation for the more important notions of operator norms.

An operator norm is defined by means of the relative effect of an operator on a vector, as measuredby some vector norm. So, each vector norm x gives rise to an associate operator norm by either of the(equivalent) rules

A = maxx=0

Axx = maxx=1 Ax . (13)

The operator norm is not always easy to compute, but it has the useful theoretical condition numberproperty

Ax A x . (14)In certain cases it is easy to compute the p-operator norm. For p = we have that the (operator)

-norm of a matrix A is the maximum of the (vector) 1-norms of all its rows:

A = maxi row i ofA1 . (15)

For p = 1 we have that the (operator) 1-norm of a matrix A is the maximum of the (vector) 1-norms ofall its columns:

A = maxj column j ofA1 . (16)

Thus A = A1.

3


4/4

Here are some inequalities which we will find very useful. (In each of these, assume A is an n nmatrix.)

Ax2

AF x2A

2= A

2

n1/2 A2

A1

n1/2 A2

n1/2

A

2

A

n1/2

A

2

n1 A A1 n AA

1 AF n1/2 A2

(17)

The operator 2-norm is not as easy to compute, and hence we sometimes use the Frobenius norm as asubstitute.

Hermitian, symmetric, unitary, and orthogonal matrices

A hermitian matrix A satisfies A = A. When A is real and hermitian we use the terminology symmetricmatrix, instead. Now if x, y = yx then for any matrix A we have that Ax,y = x,Ay. Henceanother way to say that A is hermitian is to say that it is selfadjoint:

A = A Ax,y = x,Ay . (18)In particular, ifA is hermitian then Ax,x is real. This is because

Ax,x = x,Ax = Ax, x.We say that a hermitian matrix is positive definite in case

Ax, x > 0 for all nonzero x. (19)This is such an important property that we abbreviate it as hpd, for complex matrices, or spd for realones.

Finally, we say that Q is unitary ifQ = Q1. We use the term orthogonal ifQ is real and unitary.To say that Q is unitary (or orthogonal) is equivalent to saying that it preserves the (standard) innerproduct:

Q = Q1

Qx,Qy = x, y . (20)

Homework problems: due Monday, 6 February

1. Suppose that A is hermitian.

(a) Show that the eigenvalues ofA are real. (Hint: IfAx = x consider Ax,x.)(b) Show that ifA is hpd then its eigenvalues are positive. Is the converse true?

(c) Suppose that v1 and v2 are eigenvectors for A, with corresponding eigenvalues 1 and 2.Show that v1 is orthogonal to v2.

2. Suppose that Q is unitary.

(a) Show that if is an eigenvalue ofQ then || = 1. (Hint: IfQx = x consider Qx,Qx.)(b) Suppose that v1 and v2 are eigenvectors for Q, with corresponding eigenvalues 1 and 2. Is

it true that v1 is orthogonal to v2?

3. Show that rk(A) = 1 if and only if there are column vectors x and y such that A = xyT. Showthat in this case AF = A2 = x2 y2.

4. We say that an n n matrix A is strictly upper triangular if all its entries on or below the maindiagonal are 0. Show that ifA is strictly upper triangular then An = 0. What does this say aboutthe eigenvalues ofA?

5. Show that ifA is both unitary and upper triangular then it is diagonal. In this case what can yousay about its diagonal entries?

4

norms and inner products

Documents