introduction to signal spaces - college of engineering, michigan

Introduction to Signal Spaces


Selin AviyenteDepartment of Electrical and Computer Engineering

Michigan State University

January 12, 2010


Motivation

Outline

1 Motivation

2 Vector Space

3 Inner Product Space

4 Normed Vector Space

5 L2 and l2 Spaces

6 Basis

7 Fourier Series


Motivation

Signal Expansion

Our goal in this course is to expand any signal x fromsome space S in terms of elementary signals {φi} such asx =

∑i αiφi .

The set {φi} should be complete for S.The expansion coefficients, αi should be sparse, explainthe signal with as few coefficients as possible. Attractive forsignal processing applications like compression.There should be a fast algorithm to compute αi from thesignal.{φi} should be a meaningful set of functions, e.g.sinusoids.


Vector Space

Outline

1 Motivation

2 Vector Space



5 L2 and l2 Spaces

6 Basis

7 Fourier Series


Vector Space

Vector Space

A linear vector space X is a collection of vectors (over thecomplex or real field) together with two operations; vectoraddition and scalar multiplication, which for all x , y ∈ X andα ∈ C/R satisfy the following:

1 Commutativity: x + y = y + x for all x, y ∈ X .2 Associativity: x + (y + z) = (x + y) + z.3 Distributivity: α(x + y) = αx + αy.4 Additive Identity: There is a zero vector such that x + 0 = x.5 Additive Inverse: For each x ∈ X there is a unique vector−x such that x +−x = 0.

6 Multiplicative Identity: 1x = x.


Vector Space

Examples

Set of real numbers (R): It is a real vector space withaddition defined in the usual way and multiplication byscalars defined as ordinary multiplication. The null vectoris the real number zero.n-dimensional real or complex vector spaces: Rn,Cn.Lp[0, T ] = {f (t)| ∫ |f (t)|pdt < ∞}, 1 ≤ p < ∞. Used forrepresenting continuous time signals.lp = {x [n]|∑ |x [n]|p < ∞}, 1 ≤ p < ∞. Used for discretetime signals.C[0, T ]: The collection of all real-valued continuousfunctions on the interval [0, T ].The collection of functions that are piecewise constantbetween integers.Is R+ a vector space?


Vector Space

Subspace

A subspace, S, of a vector space V is a subset of V that isclosed under vector addition and scalar multiplication.

Example: Is X1 = {x ∈ C[0, T ] :∫ T

0 x(t)dt = 1} asubspace of C[0, T ]?Answer: No, if x, y ∈ X1, then check whether x + y ∈ X1.

∫ T

0x(t) + y(t)dt =

∫ T

0x(t)dt +

∫ T

0y(t)dt = 2 6= 1 (1)


Inner Product Space

Outline

1 Motivation

2 Vector Space



5 L2 and l2 Spaces

6 Basis

7 Fourier Series


Inner Product Space

Inner Product

An inner product of two vectors x, y ∈ V on a vector spaceis a complex-valued scalar assigned to the two vectors andsatisfies the following properties:

1 < x, y >= (< y, x >)∗.2 < αx, y >= α < x, y >.3 < x + y, z >=< x, z > + < y, z >.4 < x, x >≥ 0, < x, x >= 0 iff x = 0.

An inner product space (pre-Hilbert space) is a linearvector space in which an inner product can be defined forall elements of the space and a norm is given by‖x‖ = (< x, x >)1/2.Cauchy-Schwarz Inequality: For all x, y in an inner productspace, | < x, y > | ≤ ‖x‖‖y‖. Equality holds if and only ifx = λy or y = 0.


Inner Product Space

Inner Product Space

A complete inner product space is called a Hilbert space.Completeness means that every sequence of vectors inthe vector space converge to a vector in the vector space,e.g. the space of rational numbers is not complete.A sequence of vectors {xn} is called a Cauchy sequence if‖xn − xm‖ → 0 as n, m →∞. If every Cauchy sequence inV converges to a vector in V , then V is called complete.Example: V = CN , < x, y >= yHx =

∑Ni=1 xiy∗i .

Example: V = L2[a, b], < x, y >=∫ b

a x(t)y∗(t)dt .In this course, the space of signals we will consider will bea Hilbert space.


Normed Vector Space

Outline

1 Motivation

2 Vector Space



5 L2 and l2 Spaces

6 Basis

7 Fourier Series


Normed Vector Space

Normed Vector Space

A norm is a function (‖ · ‖ : V → R) such that the followingproperties hold:

1 ‖x‖ ≥ 0 with equality iff x = 0.2 ‖αx‖ = |α|‖x‖.3 Triangle Inequality: ‖x + y‖ ≤ ‖x‖+ ‖y‖ with equality iff

x = αy.

The norm measures the size of a vector. A vector spacetogether with a norm function is called the normed vectorspace.The notion of length (norm) gives meaning to the distancebetween two vectors in V , d(x, y) = ‖x− y‖.


Normed Vector Space

Normed Vector Space: Examples

For example, V = RN , ‖x‖ =√∑N

i=1 x2i .

V = lp,‖x‖ = (∑

n xp[n])1/p.The normed space C[a, b] consists of continuous functionson the interval [a, b] together with the norm‖x‖ = maxa≤t≤b|x(t)|.You can define different norms for the same vector spaceyielding different normed spaces.A norm can be defined directly from an inner product,though not every norm can be defined from an innerproduct.


L2 and l2 Spaces

Outline

1 Motivation

2 Vector Space



5 L2 and l2 Spaces

6 Basis

7 Fourier Series


L2 and l2 Spaces

L2 Space

For an interval a ≤ t ≤ b, the space L2([a, b]) is the set ofall square integrable functions defined on a ≤ t ≤ b, i.e.L2([a, b]) = {f : [a, b] → C;

∫ ba |f (t)|2dt < ∞}.

This corresponds to the space of all signals whose energyis finite.Functions that are discontinuous are allowed as membersof this space.The space L2[a, b] is infinite dimensional.If a = 0 and b = 1, then f (t) = 1/t is an example of afunction that does not belong to L2[0, 1].The inner product on L2[a, b] is defined as< f , g >=

∫ ba f (t)g∗(t)dt .


L2 and l2 Spaces

l2 Space

For many applications, the signal is already discrete. Insuch cases, we represent the signal as a sequence.The space l2 is the set of all sequences with

∑n |xn|2 < ∞.

The inner product on this space is defined as< x , y >=

∑n xny∗n .


Basis

Outline

1 Motivation

2 Vector Space



5 L2 and l2 Spaces

6 Basis

7 Fourier Series


Basis

Span and Basis

A necessary and sufficient condition for the set of vectorsx1, x2, . . . , xn to be linearly independent is that theexpression

∑nk=1 αkxk = 0 implies αk = 0 for all

k = 1, 2, 3, . . . , n.The span of set S ⊂ V is the subspace of V containing alllinear combinations of vectors in S. WhenS = {x1, x2, . . . , xN}, span(S) = {∑N

i=1 αixi}.A subset of linearly independent vectors {x1, x2, . . . , xN} iscalled a basis for V when its span equals to V . In thiscase, we say that the dimension of V is equal to N. V isinfinite-dimensional if it contains an infinite number oflinearly independent vectors.Any two bases for a finite-dimensional vector spacecontain the same number of elements.


Basis

Basis Examples

Once you have a basis for a vector space you can expressany vector as a linear combination of the elements of thisset.Example: For R3, the basis is (1, 0, 0), (0, 1, 0), (0, 0, 1).Example: The space of infinite sequences, such as l2, isspanned by the infinite set {δ[n − k ]}k ∈ Z. Since they arelinearly independent, the space is infinite-dimensional.Example: The space of second order polynomials, P2 isspanned by {1, x , x2}.Note: In infinite dimensional spaces, the concept of basisis a bit different since we have not defined infinite sums.


Basis

Orthogonality

The vectors x and y are said to be orthogonal if< x, y >= 0.The collection of vectors, ei ,i = 1, . . . , N, is said to beorthogonal if each ei has unit length and < ei , ej >= δij .Two subspaces V1 and V2 of V are said to be orthogonal ifeach vector in V1 is orthogonal to every vector in V2.


Basis

Orthogonal and Orthonormal Basis

A set of vectors {x1, x2, . . . , } form an orthonormal basis forV if they span V and < xi , xj >= δij .In this course, we will work with infinite dimensionalfunction spaces such as L2 and l2.In Hilbert space, orthonormal bases are more useful. Anyorthonormal set in a Hilbert space can be extended to forman orthonormal basis.Fourier Series Theorem: Let {xn} be an orthonormal set ina Hilbert space, H. Then the following statements areequivalent:

1 {xn} is an orhonormal basis.2 For any x ∈ H, x =

∑n < x , xn > xn.

3 For any two vectors, < x , y >=∑

n < x , xn >< y , yn >.4 For any x , ‖x‖2 =

∑n | < x , xn > |2.


Basis

Orthogonal Projection Theorem

Suppose V0 is a subspace of an inner product space V .Suppose {e1, e2, . . . , eN} is an orthonormal basis for V0.Let v ∈ V and define v0 as the orthogonal projection of vonto V0, then v0 =

∑Ni=1 < v , ei > ei and it is the vector

closest to v in the space V0. Moreover, the error v − v0 isorthogonal to v0.Example: V = R2 and V0 = R, then v0 =< v , e1 > e1. InR2, the inner product is the dot product. Therefore,v0 = vcos(θ).In this course, we will consider orthogonal projections ininfinite dimensional vector spaces. Wavelet expansion isequivalent to projecting a signal onto different subspaces.


Basis

Gram-Schmidt Orthogonalization

Given any countable linearly independent set {yn} in aninner product space, it is always possible to construct anorthonormal set from it.We can construct an orthonormal set {xn} as follows:

1 x1 = y1‖y1‖ .

2 xk+1 =yk+1−

∑ki=1<yk+1,xi>xi

‖yk+1−∑k

i=1<yk+1,xi>xi‖ .


Fourier Series

Outline

1 Motivation

2 Vector Space



5 L2 and l2 Spaces

6 Basis

7 Fourier Series


Fourier Series

Fourier Analysis

There are many practical reasons for expanding a functionas a trigonometric sum. If f (t) is a signal, then adecomposition of f into a trigonometric sum gives adescription of its component frequencies.The space of functions considered are periodic functions,L2[−T/2, T/2] with the inner product< f , g >= 1

T

∫f (t)g∗(t)dt .

The functions {ejω0kt} form an orthonormal basis.By Fourier Theory, they span the space. Orthonormalitycan be proven as follows:

< ejω0kt , ejω0lt > =1T

∫ T/2

−T/2ejω0(k−l)tdt

=1T

[ej(k−l)ω0T/2

j(k − l)ω0− e−j(k−l)ω0T/2

j(k − l)ω0]

=

{1, k = l

sinc(π(k − l)) = 0, k 6= l(2)

Therefore, any f (t) ∈ L2[−T/2, T/2] can be written asf (t) =

∑k F [k ]ejkω0t where F [k ] = 1

T

∫f (t)e−jkω0tdt .


Fourier Series

If we need to find the best approximation of x(t) using Nterms in the summation, this is equivalent to finding theorthogonal projection of x(t) onto a subspace spanned by{1, ejω0t , . . . , ejω0Nt} and the expansion coefficients that willminimize the error are given by < x(t), ejkω0t >.Fourier transform is an extension for aperiodic signals andcan be defined as F (ω) =< f (t), ejωt >.


Fourier Series

Shannon’s Sampling Theorem

If f (t) is continuous and bandlimited to ωm, then f (t) isuniquely defined by its samples taken at the samplingfrequency of 2ωm (the minimum sampling frequency,Nyquist rate), with T = π/ωm.

f (t) =∞∑

n=−∞f (nT )sincT (t − nT )

sincT (t) =sin(πt/T )

πt/T(3)

This implies that the space of continuous and bandlimitedfunctions form a vector space (UT functions whose Fouriertransform is limited to [−π/T , π/T ]) andh(t − nT ) = sincT (t − nT ) form an orthogonal basis for thisspace.In this case, f (nT ) = 1

T < f (t), h(t − nT ) >.Proof: Fourier transform of h(t) is H(ω) = Trect(t/2π/T ).You first need to prove orthogonality (HW 1),< h(t − nT ), h(t − pT ) >= T δ(n − p).

< f (t), h(t − pT ) > =∑

n

f (nT ) < h(t − nT ), h(t − pT ) >

= Tf (pT )

f (pT ) =1T

< f (t), h(t − pT ) > (4)

introduction to signal spaces - college of engineering, michigan

Documents