linear algebra · 2020. 11. 30. · linear algebra iwe begin with recollections from the theory of...
TRANSCRIPT
![Page 1: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/1.jpg)
Linear Algebra
I We begin with recollections from the theory of linear algebra ⇒ Fields. Vector Spaces. Algebras
I Recast linear information processing as operators in the algebra of endomorphisms of a vector space
1
![Page 2: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/2.jpg)
Fields
Field (“Definition”)
A field F is a set where a sum and a multiplication are defined
I Define numbers and the operations we perform on them ⇒ Reals F = R. Complexes F = C
I There are two operations defined ⇒ The sum and the product
2
![Page 3: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/3.jpg)
Fields
Field (Definition)
A field F is a set with two binary operations: Addition (+) and multiplication (·). Such that:
⇒ Both are associative ⇒ α + (β + γ) = (α + β) + γ, and α · (β · γ) = (α · β) · γ
⇒ Both are commutative ⇒ α + β = β + α, and α · β = β · α
⇒ Both have identity elements ⇒ α + 0 = α, and α · 1 = α
⇒ Additive inverse ⇒ For all α ∈ F , there exists −α such that α + (−α) = 0
⇒ Multiplicative inverse ⇒ For all α ∈ F , α 6= 0, there exists α−1 such that α · (α−1) = 1
⇒ Distributive property ⇒ α · (β + γ) = (α · β) + (α · γ)
2
![Page 4: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/4.jpg)
Vector Spaces
Vector Space (“Definition”)
A vector space M over the field F is a set whose elements can be added together and can be also
multiplied by elements of the field F . A set of arrows.
I Define the signals we want to process ⇒ Vectors in Rn. Functions in L2([0, 1]). Sequences
I In addition to the field operations we add two more operations
⇒ The addition of signals and the multiplication of signals by scalars
3
![Page 5: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/5.jpg)
Vector Spaces
Vector Space (Definition)
A vector space M over the field F is a set with two operations: Vector addition (+) and scalar
multiplication (×). Such that
⇒ Vector addition is associative ⇒ x + (y + z) = (x + y) + z
⇒ Vector addition is commutative ⇒ x + y = y + x
⇒ Vector addition has an identity element ⇒ x + 0 = x
⇒ Vector addition has an inverse ⇒ For all x ∈ M, there exists −x such that x + (−x) = 0
3
![Page 6: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/6.jpg)
Vector Spaces
Vector Space (Definition)
A vector space M over the field F is a set with two operations: Vector addition (+) and scalar
multiplication (×). Such that
⇒ Scalar and field multiplication are compatible ⇒ α× (β × x) = (α · β)× x
⇒ Scalar multiplication has an identity element ⇒ 1× x = x
⇒ Distributive property w.r.t vector addition ⇒ α× (x + y) = (α× x) + (α× y)
⇒ Distributive property w.r.t field addition ⇒ (α + β)× x = (α× x) + (β × x)
3
![Page 7: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/7.jpg)
Associative Algebras
Associative Algebra (Definition)
An associative algebra A is a vector space with a bilinear map A×A→ A mapping (a, b)→ a ∗ b
and such that (a ∗ b) ∗ c = a ∗ (b ∗ c).
I An algebra with unity is one with an identity element 1 such that 1 ∗ a = a ∗ 1 = a
I The algebra is commutative if for all a, b ∈ A we have a ∗ b = b ∗ a
I Add a fifth operation to define the linear transformation of a signal ⇒ Endomorphisms
4
![Page 8: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/8.jpg)
Signals and Vector Spaces
I Signals are the entities we want to process ⇒ They are elements x of a vector space M
⇒ We can add two signals ⇒ y = x1 + x2
⇒ We can scale signals with elements of a field ⇒ y = α× x
I Set of vectors with n components ⇒ M = Rn ⇒ Graph signals. Or discrete signals
I Set of finite energy functions in [0, 1] ⇒ M = L2([0, 1]) ⇒ Graphon signals
I Sequences (discrete time). Functions in R (continuous time). Sequences with two indexes (images)
5
![Page 9: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/9.jpg)
Endomorphisms
I An endomorphism e is a linear map from the vector space M into itself
e(α1 × x1 + α2 × x2) = (α1 × ex1) + (α2 × ex2)
I If M = Rn ⇒ Square matrix multiplications ⇒ y = Ex
I If M = L2([0, 1]) ⇒ Linear functionals ⇒ y(u) =
∫ 1
0
E(u, v)x(v) dvM
x ex
End(M)
e
I The space of all the endomorphisms in M is End(M) ⇒ All Matrices. Or all linear functionals
6
![Page 10: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/10.jpg)
The Set of Endomorphisms is a Vector Space
I The set End(M) of endomorphisms of a vector space M is (another) vector space
I The sum operation yields the endomorphism e = e1 + e2 defined as
ex = e1x + e2x
I Scalar multiplication yields the endomorphism e′ = αe defined as
e′x = α× (ex)M
x ex
End(M)
e
I The set of square matrices of given dimension or the set of linear functionals is a vector space
7
![Page 11: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/11.jpg)
The Set of Endomorphisms is an Algebra
I It has more structure, though ⇒ End(M) is also an associative algebra with unity
I Product e = e1 ∗ e2 ⇒ Composition of endomorphisms ⇒ ex = e1(e2x)
I The product of two matrices ⇒ E = E1E2
I Composed functional ⇒ y(w) =
∫ 1
0
E1(w , v)
[∫ 1
0
E2(u, v)x(v)dv
]du
M
x ex
End(M)
e
I Linear Algebra ⇒ Process signals (vectors) by composing linear maps (endomorphisms)
8
![Page 12: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/12.jpg)
Signal Processing in the Algebra of Endomorphisms
I An endomorphism is the set of all linear transformations that can be applied to a signal
I There is no structure in the space of endomorphisms ⇒ Learning in End(M) does not scale
I Introducing structure ≡ Restricting the set of allowable endomorphisms
⇒ We accomplish this with an Algebra and a Representation to define sets of convolutional filters
9
![Page 13: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/13.jpg)
Algebraic Signal Processing
I The algebra of endomorphisms of a vector space does not leverage signal structure
I Convolutional filters use algebras and homomorphisms to restrict allowable linear transformations
10
![Page 14: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/14.jpg)
From Linear Algebra to Signal Processing
I The signals we want to process live in a vector space M
I Linear processing with elements e of the algebra End(M)
I This is too general ⇒ Restrict allowable operations
⇒ To those that represent another (more restrictive) algebra
I Have to define homomorphisms and representationsM
x ex
End(M)
e = ρ(a)
A
a
ρ
11
![Page 15: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/15.jpg)
Homomorphisms Between Algebras
Algebra Homomorphism
Let A and A′ be two algebras. A homomorphism is a map ρ : A→ A′ that preserves the operations
of A. I.e., for all a, b ∈ A we have that
⇒ The homomorphism preserves the sum ⇒ ρ(a+ b) = ρ(a) +′ ρ(b)
⇒ The homomorphism preserves the product ⇒ ρ(a ∗ b) = ρ(a) ∗′ ρ(b)
⇒ The homomorphism preserves the scalar product ⇒ ρ(α× a) = α×′ ρ(a)
I Doing operations on the algebra A is the same as carrying operations on the algebra A′
⇒ The converse need not be true. Algebra A′ may be “richer.” May have more elements
12
![Page 16: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/16.jpg)
Representations and Filters
Representation
Consider an algebra A, a vector space M, and a homomorphism ρ from A to End(M),
ρ : A→ End(M).
The pair (M, ρ) is said to be a representation of the associative algebra A
I Ties the abstract algebra A to concrete operations on signals that live in the vector space M
I We say that a ∈ A is a filter ⇒ The action of a on signal x produces the filtered signal y = ρ(a)x
13
![Page 17: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/17.jpg)
Algebra of Polynomials of a Single Variable
I A polynomial over the field F is an expression of the form ⇒ a =K∑
k=0
aktk
I The coefficients ak are elements of the field F . The sumK∑
k=0
and the powers tk are just symbols.
I The Algebra of polynomials over F is the set of all polynomials along with the operations
Scalar multiplication Vector sum Algebra product
(α× a) =K∑
k=0
(α · ak )tk (a + b) =K∑
k=0
(ak + bk )tk (a ∗ b) =K∑
k=0
[k∑
j=0
aj · bk−j
]tk
14
![Page 18: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/18.jpg)
Graph Signal Processing (GSP)
I Signals x in Vector space M = Rn ⇒ Endomorphisms End(M) = Rn×n (square matrices E)
I Processing with E is too general ⇒ Suppose that x is supported on a graph with shift operator S
I Define the homomorphism ρ from the algebra of polynomials to End(M) = Rn×n in which we map
a =K∑
k=0
aktk → ρ(a) =
K∑k=0
akSk
I Algebra of polynomials + Homomorphism ρ ≡ Graph signal processing on shift operator S
15
![Page 19: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/19.jpg)
Algebraic Signal Processing (ASP)
I An Algebraic SP model is a triplet (A,M, ρ)
I A is an Algebra with unity where filters h ∈ A live
⇒ It defines the rules of convolutional signal processing
I M is a vector space
⇒ The space containing the signals x we want to process
I ρ is a homomorphism from A to the endomorphisms of M
⇒ Instantiates the abstract filter h in the space End(M)
M
x ρ(h)x
End(M)
ρ(h)
A
h
ρ
I Any h ∈ A is a filter which operates on signals according to the homomorphism ⇒ y = ρ(h)x
16
![Page 20: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/20.jpg)
Polynomials in an Algebra and Polynomial Functions
17
![Page 21: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/21.jpg)
Polynomials in an Algebra
I Given an element of an algebra a ∈ A and a set of coefficients hk ∈ F , a polynomial is
pA(a) = h0 × 1 + h1 × a + h2 × (a ∗ a) + h3 × (a ∗ a ∗ a) + . . . =∑k
hk ak
I We know that pA(a) ∈ A because we start from a ∈ A and use algebra operations throughout
I The element pA(a) ∈ A is generated from a ∈ A using the operations of the algebra (×,+, ∗)
18
![Page 22: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/22.jpg)
Polynomials in a Different Algebra
I We use the algebra’s operations ⇒ If we change the algebra, the “same” polynomial is different
I For a′ ∈ A′ the “same” polynomial performs different operations to generate pA′(a′) ∈ A′
pA′ (a′) = h0 ×′ 1 +′ h1 ×′ a +′ h2 ×′ (a ∗′ a) +′ h3 ×′ (a ∗′ a ∗′ a) +′ . . . =
∑k
hk (a′)k
I We use the operations (×′,+′, ∗′) of the algebra A′.
19
![Page 23: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/23.jpg)
Polynomial Functions over a Field
I A related object is the polynomial function over the field F
I Consider given coefficients hk ∈ F and a variable λ ∈ F . The polynomial function pF takes values
pF (λ) = h0 · 1 + h1 · λ + h2 · (λ · λ) + h3 · (λ · λ · λ) + . . . =∑k
hk λk
I It is function of λ ⇒ pF : F → F . Defined in terms of the operations (·,+) of the field F .
I Resemblance to frequency responses is not coincidental
20
![Page 24: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/24.jpg)
Polynomials with Multiple Elements in a Commutative Algebra
I Generalize to multiple elements ⇒ For set of elements A = a1 . . . ar ⊆ A define the polynomial
pA(A) =∑
k1,...kr
hk1,...,kr ak11 . . . akrr
I Associated with the set of coefficients hk1,...,kr ∈ F . Algebra Operations. An element of the algebra
I For the same set of coefficients we define the polynomial function pF (λ1, . . . , λr ) = pF (L)
pF (L) =∑
k1,...kr
hk1,...,krλk11 . . . λkr
r
I A different (simpler) object.. A function with variables λi ∈ F . Operations performed on the field F
21
![Page 25: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/25.jpg)
Generators, Shift Operators, and Frequency Representations
I Algebraic Signal Processing is an abstraction of Convolutional Information Processing
I Three central components ⇒ generators, shift operators, and frequency representations
22
![Page 26: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/26.jpg)
Generators of an Algebra
Definition (Generators)
The set G ⊆ A generates A if all h ∈ A can be represented as polynomials of the elements of G,
h =∑
k1,...kr
hk1,...,kr g1k1 . . . gr
kr = pA(G)
I The elements g ∈ G are the generators of A. And h = pA (G) is the polynomial that generates h
⇒ Filters can be built from the generating set using the operations of the algebra
I Given the algebra, the generators are given ⇒ Filter h is completely specified by its coefficients
23
![Page 27: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/27.jpg)
Generators of the Algebras of Polynomials
I The algebra of polynomials of a single variable t is generated by the polynomial g = t
⇒ Algebra elements are expressions h =∑k
hktk ⇒ They can be generated as h =
∑k
hktk
I Algebra of polynomials of two variables x and y is generated by the polynomials g1 = x and g2 = y
⇒ Algebra elements are expressions h =∑k
hklxky l ⇒ Can be generated as h =
∑k
hklxky l
24
![Page 28: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/28.jpg)
Shift Operators
Definition (Shift Operators)
Let (M, ρ) be a representation of A and G⊆ A a generator set of A. We say S is a shift operator if
S = ρ(g), for some g ∈ G
The set S = {ρ(g), g ∈ G} is the shift operator set of the representation (M, ρ) of algebra A.
I Generators g of Algebra A mapped to shift operators S in the space End(M) of endomorphisms of M
25
![Page 29: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/29.jpg)
Polynomials on Shift Operators
Theorem (Filters as Polynomials on Shift Operators)
Let (M, ρ) be a representation of A with generators gi ∈ G and shift operators Si = ρ(gi ) ∈ S.
The representation ρ(h) of filter h is a polynomial on the shift operator set,
h = pA(G) =∑
k1,...kr
hk1,...,kr g1k1 . . . gr
kr ⇒ ρ(h) = pM(S) =∑
k1,...kr
hk1,...,kr S1k1 . . .Sr
kr
Proof: The theorem is true because the homomorphism ρ preserves the operations of the algebra
ρ(h) = ρ
( ∑k1,...kr
hk1,...,kr g1k1 . . . gr
kr
)=
∑k1,...kr
hk1,...,kr ρ(g1)k1 . . . ρ(gr )kr �
26
![Page 30: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/30.jpg)
Algebraic Signal Processing (ASP)
I ASP ⇒ Vector space ≡ Signals + Algebra ≡ Filters + Homomorphism ≡ Filter Instantiation.
I In principle, the homomorphism ρ : A→ End(M) has to be
specified for all possible filters h ∈ A.
I In reality, it suffices to specify ρ for the generator set
gi ⇒ Si = ρ(gi )
I Other filters are polynomials on gi ⇒ h = pA(G)
I Which instantiate to polynomials on Si ⇒ ρ(h) = pM(S) M
x
End(M)A
ρ(h)x
ρ(h)h
ρ
27
![Page 31: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/31.jpg)
Algebraic Signal Processing (ASP)
I ASP ⇒ Vector space ≡ Signals + Algebra ≡ Filters + Homomorphism ≡ Filter Instantiation.
I In principle, the homomorphism ρ : A→ End(M) has to be
specified for all possible filters h ∈ A.
I In reality, it suffices to specify ρ for the generator set
gi ⇒ Si = ρ(gi )
I Other filters are polynomials on gi ⇒ h = pA(G)
I Which instantiate to polynomials on Si ⇒ ρ(h) = pM(S) M
x
End(M)A
ρ(h1)x
ρ(h1)h1
ρ
ρ(h2)x
ρ(h2)h2
ρ
ρ(h3)x
ρ(h3)h3
ρ
27
![Page 32: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/32.jpg)
Algebraic Signal Processing (ASP)
I ASP ⇒ Vector space ≡ Signals + Algebra ≡ Filters + Homomorphism ≡ Filter Instantiation.
I In principle, the homomorphism ρ : A→ End(M) has to be
specified for all possible filters h ∈ A.
I In reality, it suffices to specify ρ for the generator set
gi ⇒ Si = ρ(gi )
I Other filters are polynomials on gi ⇒ h = pA(G)
I Which instantiate to polynomials on Si ⇒ ρ(h) = pM(S) M
x
End(M)A
ρ(g1)x
ρ(g1)g1
ρ
ρ(g2)x
ρ(g2)g2
ρ
27
![Page 33: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/33.jpg)
Graph Signal Processing
I GSP ⇒ Signals in Rn + Algebra of Polynomials + Homomorphism ρ defined by the map
h =K∑
k=0
hktk → ρ(h) =
K∑k=0
hkSk
I Equivalent to the (much) simpler specification of the homomorphism ⇒ ρ(t) = S
⇒ This is possible because the algebra of polynomials is generated by g = t
28
![Page 34: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/34.jpg)
Frequency Representation of an Algebraic Filter
Definition (Frequency Representation)
In an algebra A with generators gi ∈ G we are given the filter h expressed as the polynomial
h =∑
k1,...kr
hk1,...,kr g1k1 . . . gr
kr = pA(G)
The frequency representation of h over the field F 1 is the polynomial function with variables λi ∈ L
pF (L) =∑
k1,...kr
hk1,...,kr λ1k1 . . . λr
kr
I The two polynomials are different creatures ⇒ The frequency representation is a simpler object
1 The field is unspecified in the definition. But unless otherwise noted F refers to the field over which the vector space M is defined
29
![Page 35: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/35.jpg)
Three Polynomials (or More)
I The central components of an ASP model are three different polynomials
⇒ The filter. The filter’s instantiation on the space of endomorphisms The frequency response
I These three polynomials have the same coefficients. They are related. But similar though they look
⇒ They are different objects. They utilize different operations. They have different meanings.
30
![Page 36: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/36.jpg)
Polynomial 1: The Filter
P1: The Filter
I A polynomial on the elements gi of the generator set G of the algebra A
pA(G) =∑
k1,...kr
hk1,...,kr g1k1 . . . gr
kr
I Sum, product, and scalar product are the operations of the algebra A
I The abstract definition of a filter. Untethered to a specific signal model
31
![Page 37: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/37.jpg)
Polynomial 2 (or More): The Instantiation of the Filter
P2: The Instantiation of the Filter in the space of Endomorphisms End(M)
I A polynomial on the elements Si = ρ(gi ) of the shift operator set S
pM(S) =∑
k1,...kr
hk1,...,kr S1k1 . . .Sr
kr
I Sum, product, and scalar product are the operations of the algebra of Endomorphisms of M
I The concrete effect that a filter has on a signal x. Tethered to a specific signal model
I “Or more” ⇒ The same abstract filter can be instantiated in multiple signal models
32
![Page 38: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/38.jpg)
Polynomial 3: The Frequency Response
P3: The Frequency Response
I A polynomial function where the variables λi ∈ L take values on the field F
pF (L) =∑
k1,...kr
hk1,...,kr λ1k1 . . . λr
kr
I Sum and product are the operations of field F . ⇒ E.g, a polynomial with real variables
I Simpler representation of a filter. Untethered to a specific signal model (except for the field)
I The tool we use for analysis. ⇒ To explain discriminability, stability and transferability
33
![Page 39: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/39.jpg)
The Three (or More) Polynomials in GSP
(P1) Abstract filter ⇒ pA(t) =K∑
k=0
hktk . Abstract definition. Untethered to any specific graph
(P2) Filter instantiated on a graph ⇒ pM(S) =K∑
k=0
hkSk . Concrete instantiation. Tethered to S
⇒ On another graph ⇒ pM(S) =K∑
k=0
hk Sk . Concrete instantiation. Tethered to S
⇒ On a graphon ⇒ pM(TW ) =K∑
k=0
hkT(k)W . Concrete instantiation. Tethered to graphon W
(P3) Frequency response ⇒ pF (λ) =K∑
k=0
hkλk . Simple function of one variable. Same for all instances
34
![Page 40: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/40.jpg)
Convolutional Information Processing
I Algebraic filters are a generic abstraction of the common features of convolutional signal processing
I Graph, time, and image convolutions can be expressed as particular cases of algebraic filters
35
![Page 41: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/41.jpg)
Specification of ASP Models
I To specify an ASP model we need to specify
⇒ A vector space M where signals x live
⇒ An algebra A where convolutional filters h live
⇒ A homomorphism ρ mapping filters to End(M)
I The signal x is processed to the output ⇒ y = ρ(h)xM
x
End(M)A
hρ(h)
ρ
ρ(h)x
36
![Page 42: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/42.jpg)
Specification of ASP Models
I To specify an ASP model we need to specify
⇒ A vector space M where signals x live
⇒ An algebra A where convolutional filters h live
⇒ The images S = ρ(g) of the algebra generators g
I The signal x is processed to the output ⇒ y = ρ(h)xM
x
End(M)A
gρ(g)
ρ
36
![Page 43: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/43.jpg)
Graph Signal Processing (GSP)
Task
Process signals x that are supported on a graph with n nodes. A matrix representation of the
graph is given in the matrix S.
1
2
3
4
5
6
7
8
9
10
11x1
x2
x3
x4
x5
x6
x7
x8
x9
x10
x11
37
![Page 44: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/44.jpg)
Graph Signal Processing (GSP)
I GSP in the graph S is a particular case of ASP in which
⇒ M = Rn ⇒ Vectors with n components
⇒ A = P(t) ⇒ The algebra of polynomials h =∑k
hktk
⇒ Shift operator ρ(t) = S ⇒ Resulting in filters
ρ(h) = ρ
(∑k
hktk
)=∑k
hkSkRn
x
Rn×nP(t)
∑hk t
k
∑hkSk
ρ
∑hkSkx
tρ(t) = S
ρ
I Processing x with filter ρ(h) yields output ⇒ y = ρ(h)x = ρ
(∑k
hktk
)x =
∑k
hkSkx
38
![Page 45: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/45.jpg)
Graphon Signal Processing (WSP)
Task
Process signals x supported on a graphon W . The signals have finite energy. I.e, x ∈ L2([0, 1])
0 1 u
x(u)
39
![Page 46: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/46.jpg)
Graphon Signal Processing (WSP)
I WSP in the graphon W is a particular case of ASP
⇒ M = L2([0, 1]) ⇒ Finite-energy functions in [0, 1]
⇒ A = P(t) ⇒ The algebra of polynomials h =∑k
hktk
L2([0, 1])
x
End(L2([0, 1]))P(t)
∑hk t
k
∑hkT
kW
ρ
∑hkT
kW x
tρ(t) = TW
ρ
40
![Page 47: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/47.jpg)
Graphon Signal Processing (WSP)
I WSP in the graphon W is a particular case of ASP
⇒ Shift operator is ρ(t) = TW defined as
(Twx) (u) =
∫ 1
0
W (u, v)x(v)dv
⇒ This mapping of the generator t yields filters
ρ(h) = ρ
(∑k
hktk
)=∑k
hkTkw
where T kw represents k applications of Tw
L2([0, 1])
x
End(L2([0, 1]))P(t)
∑hk t
k
∑hkT
kW
ρ
∑hkT
kW x
tρ(t) = TW
ρ
I Processing x with filter ρ(h) yields output ⇒ y = ρ(h)x = ρ
(∑k
hktk
)x =
∑k
hkTkW x
40
![Page 48: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/48.jpg)
Discrete Time Signal Processing (DTSP)
Task
Process sequences X with values (X )n = xn for integer indexes n ∈ Z. The sequences have finite
energy. We say that X ∈ L2(Z)
41
![Page 49: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/49.jpg)
Discrete Time Signal Processing (DTSP)
I DTSP is a particular case of ASP in which
⇒ M = L2(Z) ⇒ Finite-energy sequences in Z
⇒ A = P(t) ⇒ The algebra of polynomials h =∑k
hktk
L2(Z)
X
End(L2(Z))P(t)
∑hk t
k
∑hkS
k
ρ
∑hkS
kX
tρ(t) = S
ρ
42
![Page 50: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/50.jpg)
Discrete Time Signal Processing (DTSP)
I DTSP is a particular case of ASP in which
⇒ Shift operator is a time shift ρ(t) = S such that
(SX )n = (X )n−1
⇒ This mapping of the generator t yields filters
ρ(h) = ρ
(∑k
hktk
)=∑k
hkSk
where Sk represents k applications of SL2(Z)
X
End(L2(Z))P(t)
∑hk t
k
∑hkS
k
ρ
∑hkS
kX
tρ(t) = S
ρ
I Processing X with ρ(h) yields ⇒(Y)n
=(ρ(h)X
)n
=
(∑k
hkSkX
)n
=∑k
hk(X)n−k
42
![Page 51: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/51.jpg)
Image Processing (IP)
Task
Process images, defined as sequences X with values (X )mn = xmn that depend on two integer
indexes m, n ∈ Z. The sequences have finite energy. We say that X ∈ L2(Z2)
43
![Page 52: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/52.jpg)
Image Processing (IP)
I IP is a particular case of ASP in which
⇒ M = L2(Z2) ⇒ Finite-energy sequences in Z2
⇒ A=P(x , y) ⇒ Two-letter polynomials h=∑k
hklxky l
L2(Z2)
X
End(L2(Z2))P(x, y)
∑hkl x
ky l∑
hklSkx S
ly
ρ
∑hklS
kx S
lyX
x ρ(x) = Sx
ρ
y
ρ(y) = Sy
ρ
44
![Page 53: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/53.jpg)
Image Processing (IP)
I IP is a particular case of ASP in which
⇒ Two shift operators ρ(x) = Sx and ρ(y) = Sy
(SxX )mn = (X )(m−1)n (SyX )mn = (X )m(n−1)
⇒ This mapping of the generators x and y yields filters
ρ(h) = ρ
(∑k
hktk
)=∑k
hklSkx S
ly
Skx or Sk
x represent k or l applications of Sx orSlL2(Z2)
X
End(L2(Z2))P(x, y)
∑hkl x
ky l∑
hklSkx S
ly
ρ
∑hklS
kx S
lyX
x ρ(x) = Sx
ρ
y
ρ(y) = Sy
ρ
I Processing X yields ⇒(Y)mn
=(ρ(h)X
)mn
=
(∑kl
hklSkx S
lyX
)mn
=∑k
hk(X)
(m−k)(n−l)
44
![Page 54: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/54.jpg)
ASP is a Generic Analysis Tool
I Algebraic SP encompasses Graph SP, graphon SP, Time SP, and Image SP as particular cases
⇒ Other particular cases exist. Notably, Group SP
I ASP provides a framework for fundamental analyses that hold for all forms of convolutional filters
45
![Page 55: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/55.jpg)
Algebraic Neural Networks
I We introduce Algebraic Neural Networks (AlgNNs) to generalize neural convolutional networks
46
![Page 56: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/56.jpg)
Algebraic Neural Networks
I Algebraic Neural Networks (AlgNNs) are stacked layered structures
⇒ Each layer consists of the algebraic signal model (A`,M`, ρ`) and σ` ⇒ nonlinearity and pooling
I The output at layer ` in the AlgNN is given by
x` = σ` (ρ`(a`)x`−1) with a` ∈ A`
I The operation is equivalent to
x` = Φ(x`−1,P`,S`)
⇒ P` ⊂ A` represents the properties of the filters and S` is the shifts associated to (M`, ρ`)
47
![Page 57: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/57.jpg)
Algebraic Neural Networks
I Algebraic Neural Network {(A`,M`, ρ`)}3`=1 architecture with three layers.
(A1,M1, ρ1)
(A2,M2, ρ2)
(A3,M3, ρ3)
x
y1 = ρ1(a1) x z1 = η1
[y1
]x1 = P1
[z1
]y1 z1
y2 = ρ2(a2) x1 z2 = η2
[y2
]x2 = P2
[z2
]y2 z2
y3 = ρ3(a3) x2 z3 = η3
[y3
]x3 = P3
[z3
]y3 z3
x1
x1
x2
x2
x3
48
![Page 58: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/58.jpg)
Convolutional Features
I The processing at each layer can be performed by a family of filters
xf` = σ`
(F∑g=1
ρ`(agf` )xg`−1
)
⇒ where agf` is the filter processing gth feature xg`−1 and F` is the number of features
I Layers may use (different) specific algebraic signal models (A`,M`, ρ`)
I Trainable parameters are the filters {afg` }`fg . Numerically, we train directly on ρ`(afg` ).
49
![Page 59: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/59.jpg)
Pooling
I The pooling increases the computational efficiency and improve the performance
I The operation is attributed to the composition operator σ` = P` ◦ η`
⇒ η` is a pointwise nonlinearity and P` is a pooling operator
I The operator σ` is only assumed to be Lipschitz and to have zero as a fixed point σ`(0) = 0
I The pooling operator P` projects elements from a given vector space M` to another M`+1
50
![Page 60: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/60.jpg)
Example 1: CNN
I Traditional CNNs particularize the algebraic signal model in AlgNNs as the typical signal model.
I Make M = CN and A the polynomial algebra in the variable t and module tN − 1
⇒ S = C is the cyclic shift operator satisfying C k = C k mod N .
I The f th feature at layer ` is given by
xf` = σ`
(F∑g=1
K−1∑i=0
hgfi C i
`xg`−1
)
I In this case P` is a sampling operator while η`(u) = max{0, u} is the ReLU.
51
![Page 61: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/61.jpg)
Example 2: GNN
I The GNNs particularize the algebraic signal model in AlgNNs as the graph signal model.
I Let M = CN with components xn of x ∈ M associated with graph nodes
I Let A be the polynomial algebra with elements a =∑K−1
k=0 hktk
⇒ The homomorphism filter is given by ρ(a) =∑K−1
k=0 hkSk
⇒ S ∈ CN×N is the graph matrix representation, e.g., adjacency matrix, Laplacian matrix, etc.
I The f th feature at layer ` is given by
xf` = σ`
(F∑g=1
K−1∑k=0
hgfk Skxg`−1
)
52
![Page 62: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/62.jpg)
Perturbation Models
I We define perturbations in the context of algebraic signal processing
53
![Page 63: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/63.jpg)
Representations and Shift Operators
I To define perturbations (deformations) in ASP we recall the notion of generator of an algebra
I Generators: The set G ⊆ A generates A if all a ∈ A are polynomial functions of elements of G
I Shift Operators: The set S of homomorphism images S = ρ(g) is the set of shift operators
I Definitions of generators and shift operators allows writing filters as polynomials on shift operators
ρ(a) = pM(ρ(G)
)= pM
(S)
= p(S)
54
![Page 64: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/64.jpg)
Perturbations in Algebraic Signal Models
I We define perturbations of Algebraic models as perturbations of shift operators ⇒ S = S + T(S)
I The ASP model (A,M, ρ) is consequently perturbed to the triplet (A,M, ρ) such that
ρ(a) = pM(ρ(g)
)= pM
(S)
That is, the polynomials that define filters are the same. But they use the perturbed shift operator
I Graphs ⇒ Shift operator S represents a graph ⇒ Perturbed operator S represents different graph
I Time ⇒ S represents translation equivariance ⇒ S represents quasi-translation equivariance
55
![Page 65: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/65.jpg)
Perturbation Definition: Remarks
I Our definition limits the perturbation of the homomorphism to the perturbation of the shift operator
⇒ Motivated by practice: seen in graph, discrete time, group and graphon signals
I The model perturbs the homomorphism ρ but not the algebra A ⇒ A define the type of operations
I Notice that a perturbation x = Tx acting on the signal x can be interpreted as a transformation of S
Sx = S(Tx) = (ST )x = Sx
56
![Page 66: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/66.jpg)
Perturbation Model: Deformations
I We will consider a first order generic model of small perturbation for the shift operator(s) S
I It includes an absolute or additive perturbation T0 and a relative or multiplicative perturbation T1S
T(S) = T0 + T1S
I The operators T0 and T1 are compact normal with norms satisfying that ‖T0‖ ≤ 1 and ‖T1‖ ≤ 1
I ‖T0‖ ≤ 1, ‖T1‖ ≤ 1 not strong requirements as our interest is on perturbations with ‖T(S)‖ � 1
57
![Page 67: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/67.jpg)
Example: perturbations of graph filters
I Apply the same filter h to the same signal x on different graphs shift operators S and S
H(S) x =K−1∑k=0
hkSkx H(S) x =K−1∑k=0
hk Skx
I Filter H(S) x ⇒ Coefficients hk . Input signal x. Instantiated on shift S
I Filter H(S) x ⇒ Same Coefficients hk . Same Input signal x. Instantiated on perturbed shift S
I Perturbed model S is the matrix S = T0 + T1S ⇒ T0 additive and T1 relative perturbations
58
![Page 68: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/68.jpg)
Stability Theorems
I We define the notion of stability in the context of algebraic signal processing
I We discuss the stability properties of algebraic filters and algebraic neural networks
59
![Page 69: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/69.jpg)
Stability: recalling basic notions and notation
I In the algebraic signal model (A,M, ρ) filters are defined by the operators ρ(a) ∈ End(M), a ∈ A
I If A is generated by the set G ⊂ A ⇒ a ∈ A can be written as a = p(g) with g ∈ G, p: polynomial
I Any filter H ∈ End(M) is defined by operators p(S) where S = ρ(g) ⇒ Filters are functions of S
I Filters are polynomial functions of the shift operators S ∈ S ⇒ We use denote filters as p(S)
60
![Page 70: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/70.jpg)
Stability
Stable Operators:
We say operator p(S) is stable if there exist constants C0, C1 > 0 such that
∥∥∥p(S)x− p(S)x∥∥∥ ≤ [C0 sup
S∈S‖T(S)‖+ C1 sup
S∈S
∥∥DT(S)∥∥+O
(‖T(S)‖2
)]∥∥x∥∥
for all x ∈M and DT(S) denoting the Frechet derivative of T.
I∥∥∥p(S)x− p(S)x
∥∥∥ is bounded by the size of the deformation. Measured by value and rate of change
I Stability is not a given ⇒ Counter examples in GNN and processing of time signals.
61
![Page 71: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/71.jpg)
Lipschitz and Integral Lipschitz Filters
I Filters are polynomials on shift operators ⇒ Isomorphic to polynomials with complex variables
I Lipschitz Filter: Polynomial p : C→ C is Lipschitz if ‖p(λ)− p(µ)‖ ≤ L0‖λ− µ‖ for some L0
I Integral Lipschitz: Polynomial p : C→ C is Integral Lipschitz if
∥∥∥∥λdp(λ)
dλ
∥∥∥∥ ≤ L1 for some L1
I Restricted attention to algebras with a single generator. Generalizations are cumbersome but ready
62
![Page 72: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/72.jpg)
Stability of Algebraic Filters
Theorem (Stability of Algebraic Filters)
A filter that is Lipschitz and Integral Lipschitz is stable, i.e.
∥∥∥p(S)x− p(S)x∥∥∥ ≤ [(1 + δ)
(L0 sup
S‖T(S)‖+ L1 sup
S‖DT(S)‖
)+O(‖T(S)‖2)
]‖x‖
I Good news ⇒ Algebraic filters can be made stable to perturbations
I Alas, we either have stability or discriminability. Integral Lipschitz Filter ⇒∥∥∥∥λdp(λ)
dλ
∥∥∥∥ ≤ L1
I Commutativity factor affects stability constant but does not generate instability
63
![Page 73: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/73.jpg)
Stability of Algebraic Neural Networks, Single Layer
Theorem (Stability of Algebraic Neural Networks, Single Layer)
Let Φ`(S, x) and Φ`(S, x) be the operators associated with layer ` of an Algebraic NN. If the layer
filters are Lipschitz and Integral Lipschitz,
∥∥∥Φ`(S, x)− Φ`(S, x)∥∥∥ ≤ [(1 + δ)
(L0 sup
S‖T(S)‖+ L1 sup
S‖DT(S)‖
)+O(‖T(S)‖2)
]‖x‖
I Good news ⇒ Algebraic NNs can be made stable to perturbations. It’s the same bound
I Individual layers lose discriminability. Integral Lipschitz Filter ⇒∥∥∥∥λdp(λ)
dλ
∥∥∥∥ ≤ L1
I Nonlinearity mixes frequency components ⇒ Recover discriminability in subsequent layers
64
![Page 74: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/74.jpg)
Stability of Algebraic Neural Networks, Multiple Layers
Theorem (Stability of Algebraic Neural Networks, Multi-Layer)
Let Φ(S, x) and Φ(S, x) be the operators associated with an Algebraic NN on L layers. If the layer
filters are Lipschitz and Integral Lipschitz,
∥∥∥Φ(S, x)− Φ(S, x)∥∥∥ ≤ L
[(1 + δ)
(L0 sup
S‖T(S)‖+ L1 sup
S‖DT(S)‖
)+O(‖T(S)‖2)
]‖x‖
I It is still the same bound ⇒ simply scaled by the number of layers L
65
![Page 75: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/75.jpg)
Stability of Algebraic Neural Networks, Multiple Generators
Theorem (Stability of Algebraic Neural Networks, Multiple Generators)
Let Φ(S, x) and Φ(S, x) be the operators associated with an Algebraic NN with M generators on
L layers. If the layer filters are Lipschitz and Integral Lipschitz,
∥∥∥Φ(S, x)− Φ(S, x)∥∥∥ ≤ ML
[(1 + δ)
(L0 sup
S‖T(S)‖+ L1 sup
S‖DT(S)‖
)+O(‖T(S)‖2)
]‖x‖
I It is still the same bound ⇒ simply scaled by the number of generators M and layers L
66
![Page 76: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/76.jpg)
Spectral Representations
I In this lecture we discuss of notion of spectral decompositions in algebraic signal models
67
![Page 77: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/77.jpg)
Recalling basic concepts
I Recalling algebraic signal model (A,M, ρ) ⇒ A: algebra, M: vector space and ρ: homomorphism
I Where (M, ρ) is a representation of A ⇒ Each representation of A defines an algebraic signal model
I Given (M1, ρ1) and (M2, ρ2) representations of A ⇒ we can define a direct sum (M1 ⊕M2, ρ)
I Where the action of ρ(a) on M1 ⊕M2 is given according to ρ(a)(x1 ⊕ x2) = (ρ1(a)x1 ⊕ ρ2(a)x2)
68
![Page 78: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/78.jpg)
Subrepresentations and Irreducible Representations
Definition (Subrepresentation)
Let (M, ρ) be a representation of A. Then, a representation (U, ρ) of A is a subrepresentation of
(M, ρ) if U ⊆ M and U is invariant under all operators ρ(a) for all a ∈ A, i.e. ρ(a)u ∈ U for all
u ∈ U and a ∈ A.
Definition (Irreducible Representations)
A representation (M, ρ) with M 6= 0 is irreducible or simple if the only subrepresentations of
(M, ρ) are (0, ρ) and (M, ρ).
69
![Page 79: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/79.jpg)
Intertwining Operators
Definition (Intertwining operators)
Let (M1, ρ1) and (M2, ρ2) be two representations of an algebra A. A homomorphism or interwining
operator φ : M1 → M2 is a linear operator which commutes with the action of A, i.e.
φ(ρ1(a)v) = ρ2(a)φ(v),
And, φ is said to be an isomorphism of representations if it is an isomorphism of vectors spaces.
I If (M1, ρ1) and (M2, ρ2) are isomorphic representations we use the notation (M1, ρ1) ∼= (M2, ρ2)
70
![Page 80: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/80.jpg)
Fourier Decompositions
Definition (Fourier decomposition)
For (A,M, ρ) we say that there is a spectral or Fourier decomposition of (M, ρ) if
(M, ρ) ∼=⊕
(Ui ,φi )∈Irr{A}
(Ui , φi )⊕m(Ui ,M)
where m(Ui ,M) indicates the number of irreducible subrepresentations of (M, ρ) that are iso-
morphic to (Ui , φi ). Any signal x ∈ M can be therefore represented by the map ∆ given by
∆ : M →⊕
i U⊕m(Ui ,M)i
x 7→ x
The projection of x in each Ui are the Fourier components represented by x(i).
71
![Page 81: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/81.jpg)
Fourier Decompositions
I It is worth pointing out that the Fourier decomposition of any representation is defined by two sums
(M, ρ) ∼=⊕
(Ui ,φi )∈Irr{A}(Ui , φi )⊕m(Ui ,M)
I Non isomorphic subrepresentations (external) and one (internal) for Isomorphic subrepresentations
I⊕
on the frequencies of the representation while⊕
on components associated to a given frequency
I ∆ is an intertwining operator ⇒ ∆(ρ(a)x) = ρ(a)∆(x) ⇒ convolution ρ(a)x = ∆−1(ρ(a)∆(x))
72
![Page 82: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/82.jpg)
Fourier Decompositions
I The projection of ρ(a)x on Ui is given by φi (a)x(i) where φi : A→ End(Ui ) is a homomorphism
I The collection of the projections of ρ(a)x on Ui for all i is the spectral representation of ρ(a)x
I The product in φi (a)x(i) translates into different operations depending on the dimension of Ui
I If dim(Ui ) = 1, the operator φi (a) is a scalar while if dim(Ui ) > 1 and finite φi (a) is a matrix
73
![Page 83: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/83.jpg)
Remarks
I The link between A and Fourier decomposition is exclusively given by φi (a) which is acting on x(i)
I Not possible by selecting filters in A to modify the spaces Ui in the spectral decomposition of (M, ρ)
I Two sources of differences between two operators ρ(a) and ρ(a) associated to (M, ρ) and (M, ρ)
I One source of difference can be modified by selecting subsets of A and this is embedded in φi and φi
I Second source of difference ⇒ the difference between the spaces Ui and Ui and cannot be modified
74
![Page 84: Linear Algebra · 2020. 11. 30. · Linear Algebra IWe begin with recollections from the theory of linear algebra )Fields. Vector Spaces. Algebras IRecast linear information processing](https://reader035.vdocument.in/reader035/viewer/2022071403/60f7ad45b635946c0d3affef/html5/thumbnails/84.jpg)
Examples
I In CNNs the filtering is defined by the polynomial algebra A = C[t]/(tN − 1), therefore, we have
ρ(a)x =N∑i=1
φi
(K−1∑k=0
hktk
)x(i)ui =
N∑i=1
K−1∑k=0
hk(e−
2πijN
)kx(i)ui ,
I ui (v) = 1√Ne
2πjviN is the ith column vector of the DFT matrix and φi (t) = e−
2πjiN its eigenvalue
I In GNNs the filtering is defined by the polynomial algebra A = C[t], therefore we have
ρ(a)x =N∑i=1
φi
(K−1∑k=0
hktk
)x(i)ui =
N∑i=1
K−1∑k=0
hkλik x(i)ui
I ui is the ith eigenvector of ρ(t) = S, which is the matrix graph, and φi (t) = λi its ith eigenvalue
75