ltpda - max planck institute for gravitational · pdf fileltpda accountable and ... conj...

LTPDAaccountable and reproducible data analysis with MATLAB

M Hewitson for the LTP DA teamLIST WG2/3 Meeting, Heidelberg, 10-12-10

+----------------------------------------------------+| || || **** || ** || ------------- || //// / \\\\ || /// / \\\ || | / | || ** | +----+ / +----+ | ** || ***| | |//-------| | |*** || ** | +----+ /+----+ | ** || | / | || \\\ / /// || \\\\ // //// || ------------- || ** || **** || || Welcome to LTPDA Toolbox || || Version: 2.3.1 || Release: (R2010b) || Date: 09-12-10 || |+----------------------------------------------------+

Friday, December 10, 2010

M Hewitson, LIST WG2/3 Meeting, Heidelberg, 10-12-10

The Team

2



The Team

2

Martin HewitsonMiquel NofrariasIngo DiepholzAnneke MonskeyPhilip PetersonNatalia Korsakova

Mauro HuellerLuigi Ferraioli

Giuseppe Congedo

Eric Plagnol?

Marc Diaz Aguiló

Michele Armano

?

Adrien Grynagier



Why bother?

•LTPDA started to address needs of data analysis during operations of LPF mission•real-time analysis of experiments from each mission day

•Some initial requirements:•toolkit of fairly standard instrument characterisation

algorithms•results should be reproducible and accountable•reduce testing by building on a commercial product•ease of use (usable by non-experts)•simple data access for multiple users

3



System architecture

4

MATLAB



System architecture

4

MATLAB

utility functions

utility classes

user classes



System architecture

4

MATLAB

utility functions

utility classes

user classes

scripting layer GUI



System architecture

4

MATLAB

utility functions

utility classes

user classes

scripting layer GUI

mysql



System architecture

4

MATLAB

utility functions

utility classes

user classes

scripting layer GUI

mysql

php interface



System architecture

4

MATLAB

utility functions

utility classes

user classes

scripting layer GUI

mysqlwww

browser

php interface

apache/httpd



Objects, objects everywhere...

5

aoaoao algorithm ao




•LTPDA offers an object-oriented data analysis framework

5

aoaoao algorithm ao




•LTPDA offers an object-oriented data analysis framework•we encapsulate (describe) different data analysis

concepts with classes

5

aoaoao algorithm ao





concepts with classes•users instantiate (build) these classes to get ltpda

objects

5

aoaoao algorithm ao






objects•DA algorithms are methods (functions) of these different

classes

5

aoaoao algorithm ao






objects•DA algorithms are methods (functions) of these different

classes•users act on the objects using the methods

5

aoaoao algorithm ao



Example: The Analysis Object Class

•We want to encapsulate the concept of an analysis result•avoid storing images, text files, documents, etc

6



Example: The Analysis Object Class

•We want to encapsulate the concept of an analysis result•avoid storing images, text files, documents, etc

6

Analysis Object

Numerical

Data

Provenance

Processing

history

Additional

meta-data

- creator

- date

- IP address

- Hostname

- Operating System

- software versions

- Name

- ID number

- Comment

- pipeline file(s)

- Name

- Algorithm version

- Parameter list

- Creation date/time

- Input histories

- Name

- Numerical data

vectors

- Creation date/time

- Additional flags



What can I do with my AO?

7



What can I do with my AO?

7

>> methods ao

Contents detrend hist_gauss mdc1_ifo2acc_inloop rotate std abs dft hypot mdc1_ifo2cont_utn round straightLineFit acos diag ifft mdc1_ifo2control sDomainFit string angle diff imag mdc1_x2acc save submit ao display index mean scale sum asin dopplercorr integrate median scatterData sumjoin atan downsample interp min search svd atan2 dropduplicates interpmissing minus select svd_fit bilinfit dsmean inv mode setDescription t0 bin_data dx iplot mpower setDx table bsubmit dy iplotyy mrdivide setDy tan buildWhitener1D eig isprop mtimes setFs tdfit cat eq isvalid ne setMdlfile tfe char eqmotion join noisegen1D setName timeaverage cohere evaluateModel lcohere noisegen2D setPlotinfo timedomainfit complex exp lcpsd norm setProcinfo times compute export le normdist setT0 timeshift confint fft len nsecs setUUID transpose conj fftfilt linSubtract offset setX type consolidate filtSubtract lincom optSubtraction setXY uminus conv filter linedetect phase setXunits unwrap convert filtfilt linfit plot setY update copy find lisovfit plus setYunits upsample corr firwhiten ln polyfit setZ validate cos fixfs log polynomfit sign var cov fngen log10 power simplifyYunits viewHistory cpsd fromProcinfo lpsd psd sin whiten1D crbound fs lscov psdconf sineParams whiten2D created gapfilling lt pwelch smallvector_lincom x creator gapfillingoptim ltfe quasiSweptSine smallvectorfit xcorr csvexport ge ltp_ifo2acc rdivide smoother xfit ctranspose get max real sort xunits curvefit getdof mcmc rebuild spectrogram y delay gnuplot md5 removeVal spikecleaning yunits delayEstimate gt mdc1_cont2act_utn report split zDomainFit demux heterodyne mdc1_ifo2acc_fd resample spsd zeropad det hist mdc1_ifo2acc_fd_utn rms sqrt



Don’t touch me with that AO, I don’t know where it’s been!

8

ao




8

ao

name

history

version

data




8

ao

name

history

version

data

name

input histories

version

params




8

ao

name

history

version

data

name

input histories

version

params

name

input histories

version

params




8

ao

name

history

version

data

name

input histories

version

params

name

input histories

version

params

name

input histories

version

params



What clever algorithms you have!

9

ao

ao

algorithm

processing

get history

get history

Add algorithmhistory

ao



Show me where you’ve been!

10



Reliving history...

•All steps and configurations in the processing chain are stored in the history tree

•So, we can:• inspect that tree•export the commands from the tree•rebuild the object

11

obj.type(<file>)output commands needed to rebuild this object

robj = obj.rebuildrebuild this object



Other ltpda classes

12

ltpda_obj



Other ltpda classes

12

ltpda_objltpda_nuo ltpda_uo



Other ltpda classes

12

ltpda_objltpda_nuo ltpda_uo

history +20 others...



Other ltpda classes

12

ltpda_objltpda_nuo ltpda_uo plist

timehistory +20 others...



Other ltpda classes

12

ltpda_objltpda_nuo ltpda_uo plist

timehistory +20 others...

ltpda_uoh

collection filterbank smodel ltpda_tf timespan ssm ao pest matrix

rational parfrac pzmodelltpda_filter

miir mfir



Simple scripting interface

13

% Build test-mass noise modeltmnoise = ssm(plist('built-in', 'DIST_TM')); % Compute bode function of this modelpl = plist('inputs', 'DIST_TM_NOISE.tm1_x', ... 'outputs', 'DIST_TM.tm1_x', ... 'f', logspace(-5,0,1000));

tmbode = tmnoise.bode(pl); % Compute absolute value (drop the phase)tmabs = abs(tmbode); % Plot the resultiplot(tmabs, plist('linewidths', 2));




13



ssm constructor




13



ssm constructor

we have an ssm object




13



ssm constructor

build plist





13



ssm constructor

build plist

callssm/bode





13



ssm constructor

build plist

callssm/bode


take abs()




13



ssm constructor

build plist

callssm/bode


take abs()

plot it!



Help!

•LTPDA is extensively documented•in-line help in classes and methods•MATLAB documentation on many features•Detailed tutorial

14

>> help ao/abs ABS overloads the Absolute value method for Analysis objects. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% DESCRIPTION: ABS overloads the Absolute value operator for Analysis objects. CALL: ao_out = abs(ao_in); ao_out = abs(ao_in, pl); Parameter Sets VERSION: $Id: abs.m,v 1.48 2010/10/28 15:50:22 ingo Exp $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



Help!

•LTPDA is extensively documented•in-line help in classes and methods•MATLAB documentation on many features•Detailed tutorial

14

>> help ao/abs ABS overloads the Absolute value method for Analysis objects. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% DESCRIPTION: ABS overloads the Absolute value operator for Analysis objects. CALL: ao_out = abs(ao_in); ao_out = abs(ao_in, pl); Parameter Sets VERSION: $Id: abs.m,v 1.48 2010/10/28 15:50:22 ingo Exp $ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

click here



I love my mouse!

15



I really love my mouse!

16



Where can I put all these objects?

17

a = ao(1)submit(a)



Where can I put all these objects?

17

a = ao(1)submit(a)

M: running LTPDARepositoryManager/findConnectionsM: Matched 1 managed connectionsM: submitting 1 objects to repositoryM: got user id 2 for user: hewitsonM: submitting to [email protected]:ltpda_testM: submitting object: ao / NoneM: submitted ao object with ID: 86924, UUID: cc571f33-b899-4651-853d-8427c8f63943, name: None


mailto:[email protected]



How can I get them back again?

18

a = ao(plist('hostname', '130.75.117.67', ... 'database', 'ltpda_test', ... 'id', 86924));



Extending the toolbox

•The toolbox can be extended by:•building models

• users can create their own collections of built-in models • these act like factory constructors for different classes

•writing new methods and assemble them in an LTPDA Extension Module

19



Extending the toolbox

•The toolbox can be extended by:•building models

• users can create their own collections of built-in models • these act like factory constructors for different classes

•writing new methods and assemble them in an LTPDA Extension Module

19

STOC_DA_Module/│├── ao│ └── timesTwo.m│├── +stoc_da_module│ ├── @plot│ │ ├── plot.m│ │ └── savePlot.m│ └── @repository│ ├── getSTOCSimulationData.m│ └── repository.m└── tests



What else?

20



What else?

•Interfaces to other communities:•AO constructor which can talk to GEO data servers•AO constructor which can talk to LIGO (cds) data servers

20



What else?

•Interfaces to other communities:•AO constructor which can talk to GEO data servers•AO constructor which can talk to LIGO (cds) data servers

•Coming soon:•After many requests for improved plotting

• we will make dedicated plotter classes with different plot tools•Cleaner, more versatile built-in model structure•Support for extension modules

• module repository and easy browsing and installation from within MATLAB

•A new unit-testing infrastructure which will allow users to easily create unit tests for their built-in models and extension modules

20



Track record

•First serious release in April 2008•Last release August 2010•Next release early 2011•All releases are unit tested

•we run ~6000 unit tests every night•Some releases are system tested by professionals (FBK)

•We have made two formal deliveries to ESA

•We have run a few dedicated training sessions, and are willing to run more

21



Look at the size of that thing!

22

• Documentation• user-manual alone ~700

pages• ~1500 documented m-

files (functions)

• Testing• ~6000 unit tests• thousands of automated

system tests• 100s hand-run system

tests• ESA-run acceptance tests

• Code• 2000 m-files• ~200000 lines of MATLAB

code



Formalities

23

•LTPDA: • http://www.lisa.aei-hannover.de/ltpda/

•Bugs and features:• https://ed.fbk.eu/ltpda/mantis/


http://www.lisa.aei-hannover.de/ltpda/

http://www.lisa.aei-hannover.de/ltpda/

https://ed.fbk.eu/ltpda/mantis/

https://ed.fbk.eu/ltpda/mantis/


Get involved!

24

• [email protected]• http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltpda_releases

• for releases of LTPDA

• [email protected]• http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltpda_users

• for LTPDA users

• [email protected]• http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltpda_dev

• for core development team

• [email protected]• http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltpda_cvs

• for LTPDA CVS commit mails

• [email protected]• http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltp_da_meeting

• for meeting announcements






http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltpda_users

http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltpda_users



http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltpda_dev

http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltpda_dev



http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltpda_cvs

http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltpda_cvs



http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltp_da_meeting

http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/ltp_da_meeting

M Hewitson, LIST WG2/3 Meeting, Heidelberg, 10-12-10 25

+----------------------------------------------------+| || || **** || ** || ------------- || //// / \\\\ || /// / \\\ || | / | || ** | +----+ / +----+ | ** || ***| | |//-------| | |*** || ** | +----+ /+----+ | ** || | / | || \\\ / /// || \\\\ // //// || ------------- || ** || **** || || Welcome to LTPDA Toolbox || || Version: 2.3.1 || Release: (R2010b) || Date: 09-12-10 || |+----------------------------------------------------+

Go get it!


ltpda - max planck institute for gravitational · pdf fileltpda accountable and ... conj...

Documents