chapter 0

7/13/2019 Chapter 0

1/32

ELE 774 - Adaptive Signal Processing 1

ELE 774 Adaptive

Signal Processing

Dr. Cenk Toker

Block F3, Room: 3304/Awww.ee.hacettepe.edu.tr/~toker/ELE774-Homepage.html

7/13/2019 Chapter 0

2/32


Course Content (Tentative) Textbook:

S.Haykin, Adaptive Filter Theory, 4th Ed., Prentice Hall, 2002

1. Introduction

BACKGROUND REVIEW

2. Discrete-time and random signals

3. Mathematical Tools

OPTIMAL LINEAR FILTERS

4. Wiener Filters5. Linear Prediction

ADAPTIVE FILTERING

6. Stochastic Gradient Descent Algorithms

7. Family of LMS Algorithms

8. Method of Least Squares

9. Recursive Least Squares (RLS) Algorithm10. Square-Root Algorithms

11. LMS and RLS Algorithms: Practical Issues

12. Kalman Filtering (?)

APPLICATIONS

13. Spectrum Estimation

14. Array Processing

7/13/2019 Chapter 0

3/32


Definition of filtering

A filteris commonly used to refer to a system that is designed toextract informationabout aprescribed quantity of interestfromnoisy data.

7/13/2019 Chapter 0

4/32


Applications

Communications; radar, sonar,

Control Systems; navigation,

Speech/Image Processing; echo and noise cancellation,biomedical engineering

Others; seismology, financial engineering, etc.

7/13/2019 Chapter 0

5/32


!!! Noise and errors are statisticalin nature !!!

We will use statistical tools.

Applications

7/13/2019 Chapter 0

6/32


Basic Kinds of Estimation

Filtering

(real-time operation)

Smoothing

(off-line operation)

Prediction

(real-time operation)

7/13/2019 Chapter 0

7/32


Filter

Linear Non-Linear

A filter is linearif the filtered,

smoothed or predicted quantity

at the output of the filter is alinear function of the observations

applied to the filter input.

Otherwise, it is non-linear.

Filter

u(t)

u(n)

y(t)

y(n)

Linear

Non-linear!!! Non-linear filters may be hard to

Analyse, if not impossible !!!

or

or

7/13/2019 Chapter 0

8/32


Optimum Filter

Definition: Solutionof an optimization problem wrt.

a certain criterionwithstatistical parameters.

Nonlinear:Maximum Likelihood (ML) sense (very difficult to implement)

Linear:

Minimum Mean Square Error (MMSE) sense

Wiener filters, (Stationary environment) Kalman filters, (Non-stationary environment)

Etc. (Any other criterion, e.g. ZF)

7/13/2019 Chapter 0

9/32


Adaptive Filters

Wiener Filter requires

Adaptive filteringcan overcome these disadvantages!

Recursive algorithm

No complete a priori information required Algorithm develops this information with increasing # of iterations.

If the environment is stationary converges to the Wiener soln.

non-stationary tracks the changes.

-a priori information of several statistics

-estimation (knowledge of the system)

is needed before filtering

-inversion of a huge matrix-computationally inefficient!

7/13/2019 Chapter 0

10/32


Analysis of Adaptive Filters

Rate of convergence (to the optimum Wiener soln.)

Misadjustment (deviation from the optimum Wiener

soln.)

Tracking (the variations in a non-stationary environment)

Robustness(to disturbances of small energy)

Computational Requirements/Cost

Numerical Properties (Numerical stability & accuracy)

7/13/2019 Chapter 0

11/32


Linear Filter Structure

Structure:

FIR(Finite-duration Impulse Response)

Transversal Filter (Tapped-delay line)

Lattice

Systolic array

IIR (Infinite-duration Impulse Response)

7/13/2019 Chapter 0

12/32


Transversal Filter

multiplier

adder

unit-delay

element

Convolution

Sum

7/13/2019 Chapter 0

13/32


Transversal Filter

xH

: Hermitian transposexH=(xT)*

7/13/2019 Chapter 0

14/32


Lattice Predictor

7/13/2019 Chapter 0

15/32


Lattice Predictor

Predictor order (# of stages): M

Forward prediction error

Backward prediction error

m: the mth reflection coefficient Input seq. u(n) is correlated, backward prediction error

b(n) is uncorrelated

Together with m, b(n) approximates d(n)(innovationsprocess).

7/13/2019 Chapter 0

16/32

ELE 774 - Adaptive Signal Processing 16y3

Systolic Array Boundary cell

Internal cell

s

7/13/2019 Chapter 0

17/32


Systolic Array

Represents a parallel computing network

Used for efficient pipelined operation

Matrix multiplication

Triangularisation

Back substitution (Matrix eqn. solving)

Example 3x3 matrix/3x1 vector

7/13/2019 Chapter 0

18/32


IIR Filter

May have stability problems,

We will prefer FIR filters for

Adaptive filtering.

7/13/2019 Chapter 0

19/32


Adaptive Filtering Algorithms

Error

Difference between

the filter output a desired response

Mean Square Error

Weighted Error Squares

+-

(n) : Errory(n)

d(n)

Filter

7/13/2019 Chapter 0

20/32



Stochastic Gradient Approach

Cost Function depends on Mean-Square Error criterion

(!!! Stochastic, depends on second order statistics !!!)

Solution of Wiener-Hopf Equations Results in Wiener soln. but with an iterative approach

Based on Method of Steepest Descent

Use instantaneous valuesinstead of expectations (LMS)

functioncost

of

gradient

size)(step

parameter

ratelearning

vector

weight-tapof

valueold

vector

weight-tapof

valueupdated Requires

Expectations

E{.}

signal

error

vector

input

-tap

size)(step

parameter

ratelearning

vector

weight-tapof

valueold

vector

weight-tapof

valueupdated

7/13/2019 Chapter 0

21/32



Least-Squares Estimation

Cost Function depends on sum of weighted error squares

Low computational complexity due to recursive operation

Three categories Standard RLS

Relies on Matrix Inversion Lemma

Numerically unstable, high computational complexity

Square-root RLS algorithm

Based on QR-decomposition

Numerically stable

Fast RLS algorithm

Exploits certain matrix structures to reduce complexity.

7/13/2019 Chapter 0

22/32


Applications Four Classes

Identification system identification

layered earth modeling

Inverse modeling deconvolution

adaptive and blind

equalisation Prediction

linear predictive coding

adaptive differential PCM

spectrum analysis

signal detection

Interference cancellation noise canceling

echo cancellation

adaptive beamforming

7/13/2019 Chapter 0

23/32


System Identification

Observing the output of aplant(system), given the inputsignal, tries to estimate theIR of the plant.

Filter coefficient are found byan adaptive algorithm.

7/13/2019 Chapter 0

24/32


Adaptive Equalization

Removes intersymbol

interference (ISI).

Filter coefficient are found

byan adaptive algorithm.

7/13/2019 Chapter 0

25/32


Adaptive Spectrum Estimation

Parametric (AR)

model

Linear AR filter

input: white noise output: observed

signal

aim: find the model

parameters by an

adaptive algorithm.

7/13/2019 Chapter 0

26/32


Adaptive Noise Cancellation

Electrocardiography (ECG)

Acoustic noise in speech

Active noise cancellation

(headphones)

7/13/2019 Chapter 0

27/32


Echo Cancellation Coupling due to imperfect

balancing in hybridtransformer creates an echoin analog telephone lines.

Echo signal can be estimated

by an adaptive filter and thesubtracted out.

7/13/2019 Chapter 0

28/32


Adaptive Beamforming

Multiple sensors

(antenna, microphone,

etc) used to steer the

beam to a specific

position.

Radar, sonar

Commun. systems,

Astrophysical

exploration, Biomedical signal

processing, etc.

7/13/2019 Chapter 0

29/32


Historical Notes

To understand a science it is necessary to know its history.

Auguste Comte (1798-1857)

Linear Estimation Theory

Method of least squares, Gauss, 1795 Minimum mean square error estimation, late 1930s

Discrete-time Wiener-Hopf equation, Levinson, 1947

Kalman filter, Swerling, 1958 and Kalman, 1960

Stochastic gradient algorithms, late 1950s Stochastic approximation, Robins and Monro, 1951

LMS algorithm, Widrow and Hoff, 1959

Gradient adaptive lattice (GAL) algorithm, Griffiths, 1977-8

7/13/2019 Chapter 0

30/32


Historical Notes

Recursive Least Squares Algorithm

Kalman filter, Godard Algorithm, Godard, 1974

Relationship between RLS and Kalman, Sayed and Kailath, 1994

QR decomposition based systolic array, Gentleman & Kung, 1981 Fast RLS algorithm, 1970s, Morf

Neural Networks

Logical calculus for neural networks, McCulloch and Pitts, 1943

Perceptron, Rosenblatt, 1958 Back-propagation algorithm, Rumelhart, et al., 1986

Radial basis function network, Broomhead and Lowe, 1988

7/13/2019 Chapter 0

31/32


Applications

Adaptive Equalisation, 1960s Zero-forcing equaliser, Lucky, 1965

MMSE equaliser, Gersho, 1969, Proakis&Miller, 1969

Godard Algorithm, Godard, 1974

Fractionally Spaced Equaliser (FSE), Brady, 1970

Decision Feedback Equaliser (DFE), Austin 1967, MMSE,Monsen, 1971.

Speech Coding

Maximum Likelihood speech prediction, Saito and Itakura, 1966 Linear Predictive Coding (LPC), Atal and Hanauer 1970-1

Adaptive Lattice Predictor, Nakhoul and Cossell, 1981

7/13/2019 Chapter 0

32/32


Applications

Spectrum Analysis, early 1900s

Maximum entropy method, Burg, 1967

Method of multiple windows, Thomson, 1982

Adaptive noise cancellation, started at 1965

Adaptive Beamforming

Intermediate Frequency (IF) sidelobe canceller, Howells, 1950

Control law for adaptive array antenna, Applebaum, 1966

Application of LMS, Widrow et al., 1967

Minimum Variance Distortionless Response (MVDR)

beamformer, Capon, 1969

chapter 0

Documents