ivm dip apendix filtering

8/10/2019 IVM DIP Apendix Filtering

1/48

Digital Image Processing

Hongkai XiongDepartment of Electronic Engineering

Shanghai Jiao Tong University
http://ivm.sjtu.edu.cn/


2/48

Today

Wiener Filter

Kalman Filter Particle Filter


3/48

Wiener Filter

In signal processing, theWiener filteris afilter proposed by Norbert Wiener during the1940s and published in 1949.

[1] Wiener, Norbert (1949).Extrapolation, Interpolation, andSmoothing of Stationary Time Series.
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/


4/48

The goal of the Wiener filter is to filter out noisethat has corrupted a signal. It is based on a

statistical approach. The design of the Wiener filter takes a different

approach. One is assumed to have knowledge ofthe spectral properties of the original signal andthe noise, and one seeks the linear time-invariant filter whose output would come asclose to the original signal as possible.

Wiener Filter Description


5/48

1. Assumption: signal and (additive) noise arestationary linear stochastic processes with

known spectral characteristics or knownautocorrelation and cross-correlation.

2. Requirement: the filter must be physicallyrealizable/causal.

3. Performance criterion: minimum mean-square error (MMSE)

Wiener Filter Description


6/48

Wiener Filter Problem Setup The input to the Wiener filter is assumed to be a

signal, , corrupted by additive noise, . The

output, , is calculated by means of a filter, ,using the following convolution:

is the original signal

is the noise

is the estimated signal

is the Wiener filters impulse response

)( ts

)(ts )(tn

)]()([*)()( tntstgts

where

)(ts

)(tn

)( ts

)(tg
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/


7/48

Wiener Filter Problem Setup The error is defined as:

is the delay of the Wiener filter

In other words, the error is the difference between the estimatesignal and the true signal shifted by

)(

)()( tstste

where

)()()(2)()( 222 tstststste


8/48

Depending on the value of , the problem can bedescribed as follows:

1. If then the problem is that of prediction(error is reduced when similar to a later value ofs)

2. If then the problem is that of filtering

(error is reduced when similar to s)3. If then the problem is that of smoothing

(error is reduced when similar to an earlier valueof s)

0

0

0

)(

)()( tstste


9/48

)]()([*)()(

tntstgts

dtntsgts )]()()[()(

Writing as a convolution integral:

We denote:is the observed signal

is the autocorrelation function ofis the autocorrelation function ofis the cross-correlation function of and

)()()( tntstx

sR

xR

xsR

)(ts)(tx

)(ts )(tx


10/48

Taking the expected value of thesquared error results in

)(

)(

)(2)()( 222

tstststste

ddRggdRgReE xxss )()()()()(2)0()( 2

If the signal and the noise are uncorrelated (i.e., the cross-correlation iszero), then this means that

snR

sxs RR

nsx RRR

h


11/48

Wiener Filter Solutions

where

The Wiener filter problem has solutions for threepossible cases:1. The case where a non-causal filter is acceptable

(requiring an infinite amount of both past andfuture data)

2. The case where a causal filter is desired (usingan infinite amount of past data)3. The finite impulse response (FIR) case where a

finite amount of past data is used.

where
http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/


12/48

Wiener Filter Solutions

where

The first case (non-causal) is simple to solve but is notsuited for real-time applications.

Wiener's main accomplishment was solving the case wherethe causality requirement (second case) is in effect, and inan appendix of Wiener's book also gave the FIR solution.

Non-causal simple to solve, not suitable forapplication

Causal main accomplishmentFIR used in discrete series


13/48

Wiener Filter SolutionsNon-causal solution:

s

x

sx esSsSsG )()()( ,

dRgReE sxs )()()0()( ,

2

Provided that is optimal, then the minimum mean-square error equationreduces to

)(tg


14/48

Wiener Filter SolutionsCausal solution:

)(

)()(

sS

sHsG

x

where consists of the causal part of

is the causal component of (i.e., the inverse Laplace transformof is non-zero only for ) is the anti-causal component of (i.e., the inverse Laplacetransform of is non-zero only for )

)(sH s

x

sxe

sS

sS

)(

)(,

)(sSx

)(sSx


15/48

Wiener Filter SolutionsFinite Impulse Response Wiener filter fordiscrete series:

The causal finite impulse response (FIR) Wiener filter,instead of using some given data matrix X and outputvector Y, finds optimal tap weights by using the statisticsof the input and output signals


16/48

In order to derive the coefficients of the Wiener filter,consider the signal w[n] being fed to a Wiener filter oforderN. The output of the filter is denotedx[n] which isgiven by the expression
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/commons/0/00/Wiener_block.svghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/


17/48

The residual error is denoted e[n] and is defined as

e[n] =x[n] s[n] (see corresponding block diagram).The Wiener filter is designed so as to minimize themean square error (MMSE criteria) which can bestated concisely as follows:


18/48

Assuming that w[n] and s[n] are each stationary andjointly stationary, the sequences known respectively asthe autocorrelation of w[n] and the cross-correlationbetween w[n] and s[n] can be defined as follows:

The derivative of the MSE may therefore be rewritten as


19/48

Letting the derivative be equal to zero results in

which can be rewritten in matrix form


20/48

Kalman Filter The Kalman filter, also known as linear quadratic

estimation(LQE), is an algorithm which uses a series

of measurements observed over time, containing noise(random variations) and other inaccuracies, andproduces estimates of unknown variables that tend to bemore precise than those that would be based on a single

measurement alone.


21/48

Kalman Filter: OverviewThe Kalman filter uses a system's dynamics model (e.g.,physical laws of motion), known control inputs to that

system, and multiple sequential measurements (such asfrom sensors) to form an estimate of the system'svarying quantities (its state) that is better than theestimate obtained by using any one measurement alone.As such, it is a common sensor fusion and data fusionalgorithm.


22/48

Kalman Filter: Application The Kalman filter is an efficient recursive filter that

estimates the internal state of a linear dynamic system

from a series of noisy measurements. It is used in a widerange of engineering and econometric applications fromradar and computer vision to estimation of structuralmacroeconomic models, and is an important topic incontrol theory and control systems engineering.


23/48

Kalman Filter: dynamic system model

Fk, the state-transition model;

Hk, the observation model;Qk, the covariance of the process noise;Rk, the covariance of the observation noise;Bk, the control-input model.
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/en/9/92/Kalman_filter_model.svghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/


24/48


Fkis the state transition model which is applied to the previous state xk1;

Bkis the control-input model which is applied to the control vector uk;

wkis the process noise which is assumed to be drawn from a zero meanmultivariate normal distribution with covariance Qk.


25/48


Hkis the observation model which maps the true state space into theobserved space andvkis the observation noise which is assumed to be zeromean Gaussian white noise with covariance Rk.

,
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/en/9/92/Kalman_filter_model.svghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/


26/48


The Kalman filter is a recursive estimator. This means thatonly the estimated state from the previous time step and

the current measurement are needed to compute theestimate for the current state.

The state of the filter is represented by two variables:

, theposterioristate estimate at time kgivenobservations up to and including at time k;

, theposteriorierror covariance matrix (a measureof the estimated accuracy of the state estimate).


27/48

Kalman Filter: Predict

Predicted state estimate:

Predicted estimate covariance:


28/48

Kalman Filter: Update

Innovation or measurement residual

Innovation (or residual) covariance

OptimalKalman gain

Updated state estimate

Updated estimate covariance


29/48

Kalman Filter: Invariants If the model is accurate, and the values for andaccurately reflect the distribution of the initial state values,then the following invariants are preserved: (all estimates

have a mean error of zero)


30/48

Kalman Filter: Estimation of the noise

covariances Qkand Rk

Practical implementation of the Kalman Filter is oftendifficult due to the inability in getting a good estimate of

the noise covariance matrices Qkand Rk. Extensiveresearch has been done in this field to estimate thesecovariances from data. One of the more promisingapproaches to doing this is called theAuto-covarianceLeast-Squares (ALS)technique that uses

autocovariances of routine operating data to estimate thecovariances.


31/48

Kalman Filter: An example applicationConsidering a model:

A truck on infinitely long straight rails. Initially, the truck is stationary atposition 0, but it is buffeted this way and that by random acceleration. Wemeasure the position of the truck every tseconds, but these

measurements are imprecise; we want to maintain a model of where thetruck is and what its velocity is.

The position and velocity of the truck are described by the linear state space

We show here how we derive the model from which we createour Kalman filter:

is the velocity, that is, the derivativeof position with respect to time.


32/48

Kalman Filter: An example application

We assume that between the (k 1) and ktimestep the truckundergoes a constant acceleration of akthat is normally distributed,with mean 0 and standard deviation a. From Newton's laws ofmotion we conclude that


33/48

Kalman Filter: An example application

At each time step, a noisy measurement of the true position of thetruck is made. Let us suppose the measurement noise vkis alsonormally distributed, with mean 0 and standard deviation z.

We know the initial starting stateof the truck with perfect precision,

so we initialize

To tell the filter that we knowthe exact position, we give it a

zero covariance matrix:


34/48

Particle Filter: In statistics, a particle filter, also known as a sequential

Monte Carlomethod (SMC), is a sophisticated modelestimation technique based on simulation.[1] Particle

filters are usually used to estimate Bayesian models inwhich the latent variables are connected in a Markov chainsimilar to a hidden Markov model (HMM), but typicallywhere the state space of the latent variables is continuousrather than discrete, and not sufficiently restricted to makeexact inference tractable

[1] Doucet, A.; De Freitas, N.; Gordon, N.J. (2001).Sequential Monte Carlo Methods in Practice. Springer


35/48

Particle Filter: Particle filters are the sequential analogue of Markov chain

Monte Carlo (MCMC) batch methods and are often similarto importance sampling methods.

Adventages:Well-designed particle filters can often be much faster thanMCMC.With sufficient samples, they approach the Bayesian

optimal estimate, so they can be made more accurate thaneither the EKF or UKF.Disadvantages:

When the simulated sample is not sufficiently large, theymight suffer from sample impoverishment.


36/48

Particle Filter: GoalThe particle filter aims to estimate the sequenceof hidden parameters,xkfor k= 0,1,2,3,, based

only on the observed data ykfor k= 0,1,2,3,.

All Bayesian estimates ofxkfollow from theposterior distributionp(xk| y0,y1,,yk). In

contrast, the MCMC(Markov chain Monte Carlo)or importance sampling approach would modelthe full posteriorp(x0,x1,,xk| y0,y1,,yk).

.


37/48

Particle Filter: ModelParticle methods assume and the observations can bemodeled in this form:

is a first order Markov process such that

and with an initial distribution .

The observations are conditionally independentprovided that are knownIn other words, each only depends on


38/48

Particle Filter: ModelOne example form of this scenario is

where both and are mutually independent and identicallydistributed sequences with known probability density functionsand and are known functions.

These two equations can be viewed as state space equations andlook similar to the state space equations for the Kalman filter.


39/48

Particle Filter: Monte Carlo approximation

Particle methods, like all sampling-based approaches,generate a set of samples that approximate the filtering

distribution

So, with P samples, expectations with respect to thefiltering distribution are approximated by

.


40/48

Particle Filter: Sequential ImportanceResampling (SIR)

Sequential importance resampling (SIR), the originalparticle filtering algorithm, is a very commonly used

particle filtering algorithm, which approximates thefiltering distribution by a weighted set of Pparticles

The importance weights are approximations to therelative posterior probabilities (or densities) of theparticles such that .


41/48


SIR is a sequential version of importance sampling. As in

importance sampling, the expectation of a functioncan be approximated as a weighted average

.


42/48

Particle Filter: Sequential ImportanceResampling (SIR)For a finite set of particles, the algorithm performance isdependent on the choice of theproposal distribution

The optimal proposal distributionis given as the targetdistribution

However, the transition prior is often used as importancefunction, since it is easier to draw particles (or samples) andperform subsequent importance weight calculations:

,


43/48

Particle Filter: Sequential ImportanceResampling (SIR)A single step of sequential importance resampling is asfollows:

1) For draw samples from theproposaldistribution

2) For update the importance weights up to anormalizing constant:

Note that when we use the transition prior as theimportance function,this simplifies to the following:

,


44/48


3) For compute the normalized importance weights:

4) Compute an estimate of the effective number of particles as

5) If the effective number of particles is less than a given threshold ,then perform resampling:a) Draw P particles from the current particle set with probabilities

proportional to their weights. Replace the current particle set with thisnew one.

b) For , set

,


45/48

Particle Filter:

Result of particle filtering (red line) based on observed data generatedfrom the blue line

,
http://localhost/var/www/apps/conversion/tmp/scratch_1//upload.wikimedia.org/wikipedia/commons/5/5f/Particle_filter_example_small.pnghttp://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/http://ivm.sjtu.edu.cn/


46/48

Particle Filter: "Direct version" algorithm

,


47/48

Other Particle Filter:

Auxiliary particle filter

Gaussian particle filterUnscented particle filterGauss-Hermite particle filterCost Reference particle filterRao-Blackwellized particle filterRejection-sampling based optimal particle filter


48/48

Thank You!

ivm dip apendix filtering

Documents