B1 MiniP

B1 Engineering Computation Mini-Project

Information Engineering

David Murray, Paul Taylor

david.murray@eng.ox.ac.ukwww.robots.ox.ac.uk/∼dwm/Courses/3B1

Michaelmas 2010

1 / 54

B1 MiniP

1. Introduction

2 / 54

B1 MiniP

Approaches to explaining data

In all areas of engineering, and in all empirical sciences, there isa need to explain experimentally observed data

Some experiments designed to prove/disprove an hypothesis ...

... but most measure the variation of one quantity with another

Then some sort of model is devised which reproduces thatvariation.

Science progresses by experimentalists making observations ...

... followed by theoreticians explaining them by devising aphysically-plausible generative model

Engineers find the most satisfying generative models to bebased on the physics of the underlying processes.

3 / 54

B1 MiniP

Physics-based modelling

Eg, measure the |G| at different ω ...

1 1.5 2 2.5 3−4

−3

−2

−1

0

1

2

3

log(omega)

log

(|G

|)

1 1.5 2 2.5 3−4

−3

−2

−1

0

1

2

3

log(omega)

log

(|G

|)

1 1.5 2 2.5 3−4

−3

−2

−1

0

1

2

3

log(omega)

log

(|G

|)

Postulate 2nd order sys |G|(ω) = Aω20

/√(ω2

0 − ω2)2 + (2ζω0ω)2.

Quality of description then depends on model parameterszero-ω gain A, resonant ω0, and damping ζ.Physics-based, but distant from the inter-atomic forces givingrise to elasticity and viscosity.

4 / 54

B1 MiniP

Description rather than modelling

Physics-based modelling might be the ideal.

But often not feasible to devise a simple physical generativemodel with just a few parameters.

But still useful to describe observed data economically — interms of trends

There are two approaches ...

5 / 54

B1 MiniP

Parametric ModellingRepresent data by simple functions with a few parameters.Eg γ-ray spectroscopy.

450 500 550

400

600

800

1000

1200

1400

1600

1800

2000

Energy (keV)

Co

un

t

450 500 550

400

600

800

1000

1200

1400

1600

1800

2000

Energy (keV)

Co

un

t

450 500 550

400

600

800

1000

1200

1400

1600

1800

2000

Energy (keV)

Co

un

t

With isotope Without isotope DescriptionYou cannot explain the background — but describe it with aweakly quadratic functionDitto for the peak shape, but describe it with with a GaussianEnd up with a 6-parameter model

z(x) = [a + b(x − x0) + c(x − x0)2] + S exp

[− (x − x0)

2

2σ2

].

6 / 54

B1 MiniP

Non-parametric modellingIn kernel regression, for example, each observed datum has akernel function placed on it ... and a weighted sum formed

zfit(x) =∑n

i=1 ziKh(x − xi)∑ni=1 Kh(x − xi)

x

izz

x

z

A variety of kernel functions are used — top hats, triangularfunctions, quadratics, Gaussians and so on.Rather like using a Flexispine or French Curve to draw a curve.The data “speaks for itself”

7 / 54

B1 MiniP

What do we mean by a good fit?

Assume measurements zi at values xi of an independentvariable — the red dots... and a model with parameters p = (p1,p2, ...,pn) givingexpected values zfit(xi ,p) — the blue line

)ip,(xzfit

x

z

x i

z i

If uncertainty on each point is the same, the optimal leastsquares parameter set is

p∗ = arg minp

[C(p)] = arg minp

[∑i

(zi − zfit(xi , p))2

].

8 / 54

B1 MiniP

Reminder ... the optimal least squares parameter set is

p∗ = arg minp

[C(p)] = arg minp

[∑i

(zi − zfit(xi , p))2

].

To find the p∗ = (p1, . . . ,pn)∗ solve the n simultaneous equations

∂C∂p1

= 0,∂C∂p2

= 0, . . . ,∂C∂pn

= 0 ,

and check are that all ∂2C/∂p2i > 0

We want to fall into a minimum on the cost surface

p

p2

1

C

9 / 54

B1 MiniP

This mini-projectYou are going to investigate how to describe the shape anderosion over time of a heap of sand or spoil in flows of water.

Based on recent work in the Ocean and Coastal EngineeringGroup — described in the paper

To fit the data you will use a set of orthogonal basis functionscalled Hermite functions

You will represent data (i) in the absence of a physics-basedmodel, but (ii) using parametrized functions.

10 / 54

B1 MiniP

What has this got to do with Information Engineering?

Isn’t Information Engineering about Machine Learning, PatternRecognition, Computer Vision, Control, Robotics, etc?

Yes ...

... but at the heart of each of these is data fitting in one form oranother, usually maximimizing the conditional probability of aninterpretation I given the observed data D

P(I|D) =P(D|I)P(I)

P(D)

Bayes’ theorem shows this involves a a generative model P(D|I)

Example ... real-time visual scene recovery

11 / 54

B1 MiniP

Computer Vision and Augmented Reality

12 / 54

B1 MiniP

Computer Vision and Augmented Reality

MappingCamera Tracking

3D map fixed

Camera poses recovered

for each frame at 30Hz

3D map and selected

camera poses optimized

together

13 / 54

B1 MiniP

Project components

The research paper

The notes with coding suggestions and tasks2. Matlab revision & Orthogonality revision via Fourier3. Gram-Schmidt to obtain orthogonal basis functions4. Hermite functions5. Fitting mounds in 1D6. Fitting mounds in 2D7. Full least squares in 1D

Lab sessions

Your report

14 / 54

B1 MiniP

The paper

Gives motivation and explains how measurements were obtainedand how they are fitted — the notes suggest what bits to readand when

15 / 54

B1 MiniP

The test rig

In the UK Coastal Research Centre at Wallingford ...1 Pump water (steady or sinusoidal) up to .5 ms−1

2 Drain very carefully to fixed heights, capture contours3 Fill up again very carefully4 Repeat

Back in the lab ...1 Work out heights at mesh points2 Fit the heights with orthogonal basis functions

16 / 54

B1 MiniP

Why orthogonal basis functions?

The coefficients an are unique are can be determinedindependently of others

Suppose you model data using a non-orthogonal set, for example

f (x) = a0 + a1x + a2x2 + a3x3 + . . .

and you work out (using least squares) the coefficients a0 to a3for the first four terms.

Then you decide to add an extra term ...

Because this is a non-orthogonal set, adding in the extra termwill change all the coefficients a0 to a3 too.

More later ...

17 / 54

B1 MiniP

2. Revision: Orthogonality and Matlab

18 / 54

B1 MiniP

Orthogonal Basic Functions

Consider three vectors g0, g1, g2, lie at right angles to eachother.They form a set of orthogonal basis vectors in 3D.Any 3D vector f is described by a unique linear combo

f = a0g0 + a1g1 + a2g2 .

How would you find these unique coefficients for a particular f?Take the inner product of both sides with g0

f · g0 = a1g0 · g0 + a1g1 · g0 + a2g2 · g1 ,

and then exploit orthogonality

a0 =f · g0

g0 · g0.

The denominator is important — g0,1,2 are not unit vectors!

19 / 54

B1 MiniP

Possible to treat functions f the same way — make them up froma set of orthogonal basis functions.Suppose we have orthogonal “g-functions” ...

f (x) = a0g0(x) + a1g1(x) + a2g2(x) + · · · . (1)

How do we find a0 and so on?Define an inner product function, denote it 〈f ,g〉.

⇒〈f ,g0〉 = a0 〈g0,g0〉+ a1 〈g1,g0〉+ a2 〈g2,g0〉+ · · · .

But 〈g1,g0〉 = 0, and so on, so that

a0 =〈f ,g0〉〈g0,g0〉

and any an =〈f ,gn〉〈gn,gn〉

.

20 / 54

B1 MiniP

Example: Fourier Series

The series is

f (x) =12

A0 +∞∑

n=1

An cos(nωx) +∞∑

n=1

Bn sin(nωx) .

with coefficients

An =〈f , cn〉〈cn, cn〉

=2T

∫ +T/2

−T/2f (x) cos(nωx)dx n = 0,1,2, . . .

Bn =〈f , sn〉〈sn, sn〉

=2T

∫ +T/2

−T/2f (x) sin(nωx)dx n = 1,2, . . .

Probably not good for sand mounds as they are periodic!

But they allow us to revise some Matlab ...

21 / 54

B1 MiniP

Matlab script to test orthogonality

T = 1; % Define periodw = 2*pi/T; % Derive omega

% Integral limits are -T/2 to +T/2nint = 1000; % We’ll use a 1000 intervalsx = (-T/2:T/nint:T/2); % to give vector of x values

% Here we test cos(mwx) times sin(nwx) with m=3 and n=5m = 3; c = cos(m*w*x); % Because x is a vector, so is cn = 5; s = sin(n*w*x); % and so is scs = c .* s; % Compute element x element prod

result = trapz(x,cs) % Use Trapezium rule

Store script as myscript.m, then run from prompt

>> myscript

22 / 54

B1 MiniP

Much nicer to write functions rather than scriptsfunction [result,codeok] = fs_orthog(T,nint,m,n,code)

codeok = true; % assume code is ok until ...w = 2*pi/T; % Derive omegax = (-T/2:T/nint:T/2); % Make vector of x valuesif code(1) == ’c’ % Is the first code cos?v1 = cos(m*w*x); % Yes. Hence cos (m w x)

elseif code(1) == ’s’ % No. Is it sin?v1 = sin(m*w*x); % Yes. hence sin (m w x)

else % Is it neither c nor s?codeok = false; % Must be a bad code

endif code(2) == ’c’ % Is the second code cos?v2 = cos(n*w*x); % Yes. Hence cos (n w x)

elseif code(2) == ’s’ % etcv2 = sin(n*w*x);

elsecodeok = false;

endif codeok % Integrate if code is okresult = trapz(x,v1.*v2)/T;% Trapzm rule integrates

elseresult = 999; % Return a silly result

end

23 / 54

B1 MiniP

Much nicer to write functionsStore in fs orthog.m

function [result,codeok] = fs_orthog(T,nint,m,n,code)codeok = true; % assume code is ok until ...w = 2*pi/T; % Derive omegax = (-T/2:T/nint:T/2); % Make vector of x values

... blah blah blah ...

Called as follows ...

>> [integral, codeok]=fs_orthog(1,1000,3,3,’cc’);integral = 0.5000codeok = 1

Recall that helper functions are hidden: eg in file visible.m ...

function result = visible(a,b,c)result = helper(b,c)/a;

function h = helper (x,y)h = xˆ2 + yˆ2;

helper() cannot be accessed from elsewhere.24 / 54

B1 MiniP

3. Gram-Schmidt

25 / 54

B1 MiniP

Gram-Schmidt

The authors want to describe the height through a mound as

z(x) = a0g0(x) + a1g1(x) + a2g2(x) + . . . .

where the gm’s are orthogonal functions with certain properties.

The mother function:

g0(x) has its peak value when x is near to 0;

g0(x) > 0;

g0(x)→ 0 as |x | → ±∞.

They show that for any g(x) function with these properties∫ ∞−∞

dgdx

g(x)dx = 0 and∫ ∞−∞

g(odd deriv)g(even deriv)dx = 0

26 / 54

B1 MiniP

Gram-Schmidt

This almost looks like we could (i) define an inner product thisway and (ii) use the derivatives as an orthogonal set.

Yes to (i), but not quite for (ii) because∫ ∞−∞

g(odd deriv)g(another odd deriv)dx 6= 0

and similarly for any two even derivatives.

But the derivatives ARE linearly independent!And you know all about Gram-Schmidt for making an orthogonalbasis set from linearly independent functions

So Gram-Schmidt is your first major task ...

27 / 54

B1 MiniP

Gram-Schmidt v ’s are lin indep⇒orthogonal g’sFirst one is easy!

g0(x) = v0(x)

Then to make g1 orthogonal to g0

g1(x) = v1(x)− e10g0(x)〈g1,g0〉 = 〈v1,g0〉 − e10 〈g0,g0〉 = 0

⇒ e10 =〈v1,g0〉〈g0,g0〉

Next

g2(x) = v2(x)− e20g0(x)− e21g1(x)

e20 =〈v2,g0〉〈g0,g0〉

and e21 =〈v2,g1〉〈g1,g1〉

,

and so on.

28 / 54

B1 MiniP

For our function-type the problem is easierWe already know that eodd,even = 0 and eeven,odd = 0

g0(x) = v0(x)g1(x) = v1(x)− e10g0(x)g2(x) = v2(x)− e20g0(x)− e21g1(x)g3(x) = v3(x)− e30g0(x)− e31g1(x)− e32g2(x)

So Gram-Schmidt simplifies to

g0(x) = g(x)g1(x) = g(1)(x)g2(x) = g(2)(x)− e20g0(x)g3(x) = g(3)(x)− e31g1(x)g4(x) = g(4)(x)− e42g2(x)− e40g0(x)g5(x) = g(5)(x)− e53g3(x)− e51g1(x)

As before eij =⟨

g(i),gj

⟩/ ⟨gj ,gj

⟩.

29 / 54

B1 MiniP

What is the mother function?

Everything general up to now, but ...Eventually the mother function is chosen to be a Gaussian

g(x) = g0(x) = exp(−x2/2)

Which has derivatives

g(1) = −xg(x)g(2) = [−1 + x2]g(x)g(3) = [3x − x3]g(x)g(4) = [3− 6x2 + x4]g(x)g(5) = [−15x − 2x3 − x5]g(x)

and so on.

30 / 54

B1 MiniP

1st MAJOR TASK:

Write structured, intelligible Matlab code that achieves the following:1 Uses Gram-Schmidt to generates numerical values of the

orthogonal set g0(x), g1(x), and so on, over an appropriaterange of x values.

2 Prints out values of the e coefficients used.

3 Plots the functions in different colour on a single graph.

4 Verifies that all gm are orthogonal.

5 Computes for each the normalizing constant√〈gm,gm〉.

6 Normalizes the functions gm, and verifies their orthonormality.

31 / 54

B1 MiniP

4. Hermite Functions

32 / 54

B1 MiniP

Hermite FunctionsIt turns out that (once normalized) the orthogonal functions you(will) have just made have been spotted before — by CharlesHermite (1822-1901).

The Hermite functions are:

gn(x) ≡ ψn(x) =1

(n!2n√π)1/2 exp(−x2/2)Hn(x)

where the Hn are Hermite polynomials defined recursively as

H0(x) = 1, H1(x) = 2x , Hn+1(x) = 2xHn(x)− 2nHn−1(x) .

33 / 54

B1 MiniP

So this part of the project is to verify that the functions you workup using Gram-Schmidt are the same as Hermite functions ...

... and to test their orthonormality, and so on.

−6 −4 −2 0 2 4 6−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

x

Herm

ite functions 0

−5

Not much more to say about this task.

34 / 54

B1 MiniP

5. Fitting mounds in 1D

35 / 54

B1 MiniP

Why orthogonal basis functions? Again ...

We know that to approximate a continuous function zm(x) usingan orthogonal basis set, we write down the series

z(x) = a0ψ0(x) + a1ψ1(x) + . . .

and find the coefficients using

an = 〈ψn, zm〉/〈ψn, ψn〉 .

Because Hermite functions are orthonormal

an = 〈ψn, zm〉 =∫ ∞−∞

ψn(x)zm(x)dx .

Big Question: does this generate a least squares fit?

36 / 54

B1 MiniP

Does this generate a least squares fit? Oh yesLeast squares cost is:

C =

∫ ∞−∞

(z(x)− zm(x)

)2dx

=

∫ ∞−∞

z2 dx − 2∫ ∞−∞

z zm dx +

∫ ∞−∞

z2m dx

=(a2

0 + a21 + . . .

)− 2

∫ ∞−∞

z zm dx + Const

=(a2

0 + a21 + . . .

)− 2

∫ ∞−∞

(a0ψ0(x) + a1ψ1(x) + . . .) zm dx + Const

The a’s are the parameters

∂C/∂a0 = 2a0 − 2∫ ∞−∞

ψ0 zm dx = 0

∂C/∂a1 = 2a1 − 2∫ ∞−∞

ψ1 zm dx = 0

Each equation is independent and gives the general result

an =

∫ ∞−∞

ψn(x)zm(x)dx

37 / 54

B1 MiniP

1D Fitting

The aim now is to write a program that fits Hermite functions to1D cross sections through the data.

Rather than use real measurement at this point, you will usesynthesized measurements so that you can check your results.

The experimental measurements (synthetic or real) will not becontinuous, but will be samples of mound height taken atdifferent positions xi

But this is no different than sampling a continuous functionfor use in numerical integration!

Hence ...

38 / 54

B1 MiniP

The key recipe

1 Obtain a set of measurements zi(xi) at a number of differentpositions xi .

2 Evaluate ψn(xi) at those xi

3 Numerically integrate the products ψn(xi)zi(xi) over the range tofind an.

39 / 54

B1 MiniP

Results

−6 −4 −2 0 2 4 6−0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

x

Data

(re

d)

& O

rder(

0)

fit (b

lue)

−6 −4 −2 0 2 4 6−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

x

Data

(re

d)

& O

rder(

1)

fit (b

lue)

Term ψ0 Terms ψ0,ψ1

−6 −4 −2 0 2 4 6−0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

x

Data

(re

d)

& O

rder(

5)

fit (b

lue)

−6 −4 −2 0 2 4 6−0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

x

Data

(re

d)

& O

rder(

6)

fit (b

lue)

Terms ψ0-ψ5 Terms ψ0-ψ6

40 / 54

B1 MiniP

Scaling and offsetting

Bad news ... the Hermite functions assume that the functions arecentred on zero, and have a natural width of unity, whereas sandmounds don’t

To attempt to find the overall scaling and offset, the papersuggests fitting just a Gaussian to the data

z(x) = S exp[− (x − x0)

2

2σ2

]

To find the parameters you can use a non-linear optimizationprocess.

In my code I used the fminsearch() routine from Matlab,which runs the simplex method.There is advice how to use this in the notes — but you may useanother method

41 / 54

B1 MiniP

fminsearch()

You supply a vector of initial parameters, and fminsearch returnsa vector of optimized parameters.

% x is a vector of x positions% zmeas is a vector of measurements at those x positionsguess = [Sguess x0guess sigmaguess];optimized = fminsearch(@gaussiancost,guess,[],x,zmeas);

Of course you must also supply the definition of the cost function,here called gaussiancost().

function cost = gaussiancost(params,x,zmeas)zfit = params(1)*exp(-0.5*((x-params(2))/params(3)).ˆ2);vdiff = zmeas - zfit;cost = sum(vdiff.ˆ2);

42 / 54

B1 MiniP

6. Fitting in 2D

43 / 54

B1 MiniP

Fitting in 2D

You are given some help reading the data

Then you are on your own ...

1 Write code to fit the 2D data using Hermite functions.

2 Analyze several sets of data,

3 Generate the “ripple” – the differences between the fit and the data(as in paper figure 3).

44 / 54

B1 MiniP

7. Full Least Squares in 1D

45 / 54

B1 MiniP

Full least squares in 1D

If the mound is quite asymmetrical, fitting a Gaussian to estimatethe width and centre is not that good.

The idea here is to use the values from the Gaussian estimatesand values of an from the orthogonal Hermite functions as initialguesses into a full least squares fit

But there’s trouble — there are many local minima.

Can you devise a method which would find the global minimum?

46 / 54

B1 MiniP

Admin Stuff

47 / 54

B1 MiniP

Lab and drop-in sessions

TimesCohort X 11-1 Thursday Week 5 Lab

11-12 Thursday Week 6 Drop-in11-1 Thursday Week 7 Lab11-12 Thursday Week 8 Drop-in

Cohort Y 11-1 Friday Week 5 Lab11-12 Friday Week 6 Drop-in11-1 Friday Week 7 Lab11-12 Friday Week 8 Drop-in

Places: 6th floor Machines

You can work on your project anytime, anywhere, but you mustturn up to the labs so that we can monitor progress

48 / 54

B1 MiniP

Demonstrators

Demonstrators will be available to help you during scheduledsessions

They will:Help you understand the handoutConfirm that code is producing sensible resultsHelp with non-essential parts of programming such as input andoutput

They won’tWrite the essential code for youTell you what Matlab functions you should use

49 / 54

B1 MiniP

Project Report

No longer than 10 pages in length, single sided, must includealgorithms and code snippets

Templates will be produced for Latex and MSWord

Two printed copies to be handed in to the Faculty Office before5pm of Friday 1st week HT 2011.

50 / 54

B1 MiniP

Staying in touch

Important to keep in touch with Web sitewww.robots.ox.ac.uk/∼dwm/Courses/3B1

Pages will (by early Week 5) provideInitial codeUpdates and corrections to notes

51 / 54

B1 MiniP

Matlab for Students

The University has purchased a Matlab license for students touse on their own personal computers.

To download and install the software, go tohttps://register.oucs.ox.ac.uk/self/software

Any problems with this, please emailEric.Peasley@eng.ox.ac.uk

52 / 54

B1 MiniP

53 / 54

B1 MiniP

But you are Oxford Guinea Pigs..

... effortlessy superior

54 / 54