signal vector spacesignal vector space - sfu.cadchlee/ensc832folder/lecture/vector-space.pdfvector...
TRANSCRIPT
Signal vector spaceSignal vector space
Def, Vector space, V over a field (scalar),F
• Vector addition, VxV->V, and scalar multiplication FxV-p>V, plus
• 8 axioms– 1) Commutativity of addition1) Commutativity of addition– 2) Associativity of addition– 3) Identity: There exists an element of V, denoted by 0, such that
v+0=v, for any element v of V.v 0 v, for any element v of V.– 4) Inverse: For each element v of V, there exists an element,
which we denote as –v, such that v + (-v) = 0– 5) Scalar associativity) y– 6) 1v=v– 7) a(v+u)=av+au– 8) (a+b)v=av+bv8) (a b)v av bv
Uniqueness of the identityUniqueness of the identityProposition: Let , be elements of such that . Then, 0.P f
v w V v w v w+ = =Proof:
( ) ( )0 0
v w vv v w v v
+ =− + + = − +
0 00 Q.E.D.
This proposition implies that the identity is unique.
ww
+ ==
Proposition: Let , be elements of such that 0. Then, .Proof:
v w V v w w v+ = = −
0( ) ( ) 00
v wv v w vw v
+ =− + + = − ++ = −
Q.E.D.This proposition implies that for each , its inverse is unique.w v
v= −
Basic Algebra: GroupBasic Algebra: Group
• A group is defined as a set of elements GA group is defined as a set of elements Gand a binary operation, denoted by · for which the following properties are satisfiedg p p– For any element a, b, in the set, a·b is in the
set.– The associative law is satisfied; that is for
a,b,c in the set (a·b)·c= a·(b·c)There is an identity element e in the set such– There is an identity element, e, in the set such that a·e= e·a=a for all a in the set.
– For each element a in the set there is anFor each element a in the set, there is an inverse element a-1 in the set satisfying a· a-1
= a-1 ·a=e.
Group: exampleGroup: example
• A set of non-singular n×n matrices of real numbers, with matrix multiplication
• Note; the operation does not have to be commutative to be a Group.
• Example of non-group: a set of non-Example of non group: a set of nonnegative integers, with +
Unique identity? Unique inverse of h l ?each element?
• a·x=a Then a-1·a·x=a-1·a=e so x=ea x a. Then, a a x a a e, so x e.• x·a=a
• a·x=e. Then, a-1·a·x=a-1·e=a-1, so x=a-1.
Abelian groupAbelian group
• If the operation is commutative the groupIf the operation is commutative, the group is an Abelian group.
The set of m×n real matrices with +– The set of m×n real matrices, with + .– The set of integers, with + .
Application?Application?
• In channel coding (for error correction orIn channel coding (for error correction or error detection).
Algebra: fieldAlgebra: field
A fi ld i t f t l t• A field is a set of two or more elements closed under two operations, + (addition) and * (multiplication) with the(addition) and (multiplication) with the following properties– F is an Abelian group under additionF is an Abelian group under addition– The set F−{0} is an Abelian group under
multiplication, where 0 denotes the identity p , yunder addition.
– The distributive law is satisfied: (α+β)∗γ = α∗γ+β∗γ
Immediately following propertiesImmediately following properties
Proposition: α∗β=0 implies α=0 or β=0Proposition: α∗β=0 implies α=0 or β=0Proposition: For any non-zero α, α∗0= 0
P f 0 0 1 (0 1) 1 Proof: α∗0 + α = α∗0 + α ∗1= α∗(0 +1)= α∗1=α; therefore α∗0 =0
P iti 0 0 0Proposition: 0∗0 =0Proof: For a non-zero α, its additive inverse is
0 0 ( ( ) ) 0 0 ( ) 0 non-zero. 0∗0=(α+(− α) )∗0 = α∗0+(− α)∗0 =0+0=0
Examples:Examples:
• the set of real numbersthe set of real numbers• The set of complex numbers
L t fi it fi ld (G l i fi ld ) ill b• Later, finite fields (Galois fields) will be studied for channel coding– E.g., {0,1} with + (exclusive OR), * (AND)
Vector space
• A vector space V over a given field F is a set of p gelements (called vectors) closed under and operation + called vector addition. There is also an operation * called scalar multiplication, which p p ,operates on an element of F (called scalar) and an element of V to produce an element of V. The following properties are satisfied:following properties are satisfied:– V is an Abelian group under +. Let 0 denote the additive
identity.For every v w in V and every α β in F we have– For every v,w in V and every α,β in F, we have
• (α∗β)∗v= α∗(β∗v)• (α+β)∗v= α∗v+β∗v
α∗( v+w)=α∗v+ α ∗wα∗( v+w) α∗v+ α ∗w• 1*v=v
Examples of vector spaceExamples of vector space
• Rn over RR over R• Cn over C
L R L C• L2 over R, L2 over C
SubspaceSubspace.
Let be a vector space and . If is also a vector space with the same operations as ,h S i ll d b f
V S VS V
V
⊂
then S is called a subspace of .
S i b if
V
S is a subspace if , v w S av bw S∈ ⇒ + ∈
Linear independence of vectorsLinear independence of vectors
1 2
Def)A set of vectors , ,..., are linearly independent iffnv v v V∈
BasisBasis
C id t ( fi ld)V F
0 0
Consider vector space over (a field).We say that a set (finite or infinite) is a basis, if * for every finite subset , the vectors in are linearly independent, and
V FB V
B B B⊂
⊂ * for every x
1 1
1 1
, it is possible to choose , ..., and , ...,
such thatn n
Va a F v v B
x a v a v
∈∈ ∈
= + +1 1 such that ... .
The sums in the above definition are all finite because without ddi i l h i
n nx a v a v+ +
f d iadditional structure the axioms of a vector space do not permit us to meaningfully speak about an infinite sum of vectors.
Vector spaceVector space
A set of vectors is said to span ifv v v V V∈1 2
1 2
A set of vectors , ,... is said to span if every vector is a linear combination of , , ..., .
n
n
v v v V Vu V v v v
∈∈
Example: nR
Finite dimensional vector spaceFinite dimensional vector space
• A vector space V is finite dimensional ifA vector space V is finite dimensional if there is a finite set of vectors u1, u2, …, un that span Vthat span V.
Finite dimensional vector spaceFinite dimensional vector space
Let V be a finite dimensional vector space. Then
If are linearly independent but do not span thenv v v V V• 1 2
1 2
If , ,..., are linearly independent but do not span , then has a basis with vectors ( ) that include , , .
m
m
v v v V Vn n m v v v
•>
1 2If , , ..., span anmv v v V•
1 2
d but are linearly dependent, then a subset of , ,..., is a basis for with vectors ( ) .mv v v V n n m<
Every basis of contains the same number of vectors. V•
Dimension of a finiate dimensional vector space.
Example: Rn and its Basis VectorsExample: R and its Basis Vectors
•••
Inner product space: for length and langle
Vector space over . Inner product is a mapping such that
V CV V C× →
*1) , ,2) , , ,
u v v uu v w u w v wα β α β
< >=< >< + >= < > + < >
* * (Conequently, , , , )3) , 0, with equality if and only if 0.
w u v w u w vu u v
α β α β< + >= < > + < >< > ≥ =
Example: RnExample: R
•••
•••
•••
•••
Follows the notion of orthogonality, ( i )norm (metric)
Def)
Orthonormal set and projection htheorem
Def)A non-empty subset of an inner product space is said to beorthonormal iff
Sorthonormal iff1) , , 1 and2) If , and , then , 0.
x S x xx y S x y x y
∀ ∈ < >=∈ ≠ < >=
Projection onto a finite dimensional bsubspace
Def) If is a subspace of an inner product space , and , the projection of on is defined to be a vector such that is orthogonal to all vectors in .S S
S V u V u Su V u S u u S
∈∈ ∈ −
Projection Theorem (Gallage
1
r Thm 5.1)Let be an n-demensional subspace of an inner product space and assume that , ,is an orthonormal basis of S Then any may be composed as where
nS Vu V u u u
φ φ∈ = +is an orthonormal basis of S. Then, any may be composed as where
and , 0 SS
SS
u V u u u
u S u s s S⊥
⊥
∈ = +
∈ = ∀ ∈
1
. Furthermore, is uniquely determined by
, .
S
nj jS j
u
u u φ φ=
= ∑ 1 j jj=∑
Projection onto a finite dimensional bsubspace
22 222 2
2 2
Form Pythagorean theorem .SSu u u⊥= +
2Norm bounds: 0
with equality on the right iff and equality on thh left iff is orthogonal to all vectors in
Su u
u Su S
≤ ≤
∈
1
equality on thh left iff is orthogonal to all vectors in .
Bessel's inequality: 0 ,njj
u S
u φ=
≤ ∑2 2 u≤
1 jj=∑ with equality on the right iff and equality on thh left iff is orthogonal to all vectors in .
u Su S
∈
Least squared error property: ,Su u u s s S− ≤ − ∀ ∈
Gram –Schmidt orthonormalizationGram Schmidt orthonormalization
{ }1
1
Consider linearly independent , ..., , and inner product space.We can construct an orthonormal set , ..., so that
n
n
s s VVφ φ
∈
∈
{ }1 1 { , ..., } , ...,n nspan s s span φ φ=
Gram-Schmidt Orthog. Procedure
ExamplesExamples
• Euclidian space Rn (over R)Euclidian space, R , (over R)• Cn, (over C, over R)
L2 f l f ti ( R)• L2 of real functions, (over R)• L2 of complex functions, (over C, over R)• Real random variables, (over R)• more abstract examples… more abstract examples…
Application: Detection in Gaussian noiseA di A 2Appendix A.2
Application: linear least square estimation of d t Xrandom parameter X
1 1 1Y h X W= +
( ) ( )2 2 2
and are uncorrelated. 0i i
Y h X WX W E X E W
= +
= =
Linear least square estimation of .X
Application: linear least square i i f destimation of random parameters
( ) ( )d l d 0
Y HX W
X W E X E W i j
= +
∀( ) ( )( )
and are uncorrelated. 0, ,
ˆDesign linear least square estimator s.t.i j i jX W E X E W i j
x y Ay
= = ∀
=
( ){ }2
1ˆ is minimized.n
i iiE X x Y
= − ∑
Application: linear least square i i f destimation of random parameters
( ) ( )and are uncorrelated 0
Y HX W
X W E X E W i j
= +
= = ∀( ) ( )( )
( ){ }2
and are uncorrelated. 0, ,
ˆDesign linear least square estimator s.t.
ˆ i i i i d
i j i j
n
X W E X E W i j
x y Ay
E X Y
= = ∀
=
∑ ( ){ }1ˆ is minimized.
Concept of sufficient information: projection onto the direction of H
i iiE X x Y
= − ∑
Concept of sufficient information: projection onto the direction of H.(Start the discussion with the real randomvariables first.)