Lecture 16

Signals & SystemsIntroduction to Compressed Sensing

Adapted from:• M. Davenport, M. F. Duarte, Y. C. Eldar, G. Kutyniok, “Introduction to Compressed Sensing”, 2011• J. Romberg, “Imaging via Compressive Sampling”, IEEE Signal Processing Magazine, 2008• M. Davenport, “Compressed Sensing: Theory and Practice”

Ali Soltani-Farani

Abbas HosseiniAbbas Hosseini

Spring 2012

Lecture 16

Digital Revolutiong

If we sample a band-limited signal at twice its highest

frequency, then we can recover it exactly

Whittaker-Nyquist-Kotelnikov-Shannon

Sharif University of Technology, Department of Computer Engineering, signals & systems2

Lecture 16

Sensor Explosionp

Sharif University of Technology, Department of Computer Engineering, signals & systems3

Lecture 16

Data Delugeg

By 2011, ½ of digital universe will have no home

[The Economist – March 2010]

Sharif University of Technology, Department of Computer Engineering, signals & systems4

Lecture 16

Motivations

sample NC K Storesample Compress K Store

K

decompressN

Sharif University of Technology, Department of Computer Engineering, signals & systems5

Lecture 16

Motivations

Original Picture

Nonlinear ReconstructionUsing 10% of CoefficientsCoefficients

WaveletR i

Histogram of C ffi iRepresentation Coefficients

Sharif University of Technology, Department of Computer Engineering, signals & systems6

Lecture 16

Motivations

Why go to so much effort to acquire all the data when most f h t t ill b th ?of what we get will be thrown away?

Reducing number of SensorsReducing number of Sensors

Reducing measurement time

Very important in MRI

Reducing sampling ratesReducing sampling rates

Sharif University of Technology, Department of Computer Engineering, signals & systems7

Lecture 16

Compressed Sensingp g

Compressed Sensing is a method for:p g

Sampling Sparse signals with a rate much lower than

iproposed by Nyquist

Reconstructing signal using samples with quality

comparable to compressed signals

Sharif University of Technology, Department of Computer Engineering, signals & systems8

Lecture 16

Sparsity & k-Sparsityp y p y

5-Sparse Approximately Sparse

Sharif University of Technology, Department of Computer Engineering, signals & systems9

Lecture 16

What DO Compressing Algorithms DO?p g g

Transforming the signal to an orthonormal basis that most of the desired signals are sparse in thatdesired signals are sparse in that.

Taking K largest coefficients in that basis.

Sharif University of Technology, Department of Computer Engineering, signals & systems10

Lecture 16

Generalized Notion of Samplingp g

In common image sampling we measure values of each pixel. We can

look at this as:

Sharif University of Technology, Department of Computer Engineering, signals & systems11

Lecture 16

Generalized Notion of Samplingp g

Instead of a single pixel, take any linear function:

1 1 2 2

1 1

= , , = , , , = , m m

m m n n

y x y x y xY X

Sharif University of Technology, Department of Computer Engineering, signals & systems12

Lecture 16

Compressive Sensing [Donoho; Candes, Romberg, Tao - 2004]p g [ g ]

Sharif University of Technology, Department of Computer Engineering, signals & systems13

Lecture 16

Sparsity Through Historyp y g y

Willi f O G d Ri h (1795) Constantin CarathéodoryWilliam of Occam (1288-1348 AD)

“Entities must not bel i li d

Gaspard Riche (1795)

algorithm for estimating the parameters of a few

Constantin Carathéodory1907

Given a sum of K sinusoids we can recover from 2K+1multiplied

unnecessarily”the parameters of a few complex exponentials

( )( ) i i

kj t

ix t e

we can recover from 2K+1 random samples

1

( ) i

kj t

ii

x t e

Sharif University of Technology, Department of Computer Engineering, signals & systems14

1i 1i

Lecture 16

Sparsity Through Historyp y g y

Arne Beurling (1938)

Given a sum of K impulses we can recover from only a

Ben Tex (1965)Given a signal with

bandlimit B, we can corrupt i t l f l th 2 /Bpiece of the Fourier Transform an interval of length 2π/B

and still recover perfectly

1

( ) ( )k

i ii

x t t t

Sharif University of Technology, Department of Computer Engineering, signals & systems15

Lecture 16

Sparsityp y

N

1

j jj

x

ampl

es

K N

N S

a

Large Coefficients

Sharif University of Technology, Department of Computer Engineering, signals & systems16

Lecture 16

How can we exploit this prior knowledge of sparsity?p p g p y

Key Questions:y Q

How to design the sensing matrix, with minimum rows, while preser ing the str ct re of the original signal?preserving the structure of the original signal?

How to recover the original signal from the measurements?

Sharif University of Technology, Department of Computer Engineering, signals & systems17

Lecture 16

Matrix Designg

Restricted Isometry Property (RIP)

For any pair of k-sparse signals and 1x 2x

21 2 21 1

x x

2

1 2 2

1 1x x

Sharif University of Technology, Department of Computer Engineering, signals & systems18

Lecture 16

Random Measurements

Choose a random matrix:

Fill out the entries of with i.i.d samples from a sub-Gaussian

distribution

( log( ))M O k N k

Stable: Information preserving, robust to noise

Democratic: Each measurement has “equal weight” Democratic: Each measurement has “equal weight”

Universal: Will work with any fixed orthonormal basis

Sharif University of Technology, Department of Computer Engineering, signals & systems19

Lecture 16

Signal Recoveryg y

Given y x e

Find x

Ill-posed inverse problem

Sharif University of Technology, Department of Computer Engineering, signals & systems20

Lecture 16

Signal Recovery: g y

Sharif University of Technology, Department of Computer Engineering, signals & systems21

Lecture 16

Signal Recovery in noiseg y

Optimization based methods

1 2ˆ arg min s.t

Nx x y x

Greedy/Iterative Algorithms

Nx

OMP, StOMP, ROMP, CoSaMP, Thresh, SP, IHT

kx x1

0 12 2ˆ kx xx x C e C

k

Sharif University of Technology, Department of Computer Engineering, signals & systems22

Lecture 16

Compressive Sensing in Practicep g

• Tomography in medical imaging – each projection gives you a set of Fourier coefficients

– fewer measurements mean

� more patientsp

� sharper images

� less radiation exposure

• Wideband signal acquisition – framework for acquiring sparse, wideband signals

– ideal for some surveillance applications

• “Single-pixel” camera

Sharif University of Technology, Department of Computer Engineering, signals & systems23

Lecture 16

Single Pixel Camerag

Sharif University of Technology, Department of Computer Engineering, signals & systems24

Lecture 16

Image Acquisitiong q

Sharif University of Technology, Department of Computer Engineering, signals & systems25