DPC definition status DPC latest news Support to sims and DA activities
LISA Data Processing Center status
Maude Le Jeune (APC / CNRS, Paris)on behalf of DPC team
1 / 21
DPC definition status DPC latest news Support to sims and DA activities
1. DPC definition status
2. DPC latest news
3. Support to sims and DA activities
2 / 21
DPC definition status DPC latest news Support to sims and DA activities
3 / 21
DPC definition status DPC latest news Support to sims and DA activities
Overview
Definition
The DPC is a set of tools and means provided to
perform the daily processing of LISA data
and ease the data analysis (new investigations)
→ only the daily processing has been roughly estimated so far (in 2012).
Daily/Weekly processing
CPU: 30 millions CPUhour for 5 years≈ 1500 cores at a time
disk: 250Tbincluding tmp data
Data analysis
CPU: ?improve the pipelineby testing new scenarii
disk: ?
Simulations
CPU: ?include all knowninstrumental systematicsdisk: ?
→ This remains small numbers as compared to other missions.→ Need to refine the cost estimate wrt DA and sims preparation activities
4 / 21
DPC definition status DPC latest news Support to sims and DA activities
DPC schedule
DPC activities today and in the coming years
Phase 0 → B1 : definition, design.In 2017-2018
I A DPC definition document.I A first draft of a work package (WP) breakdown structure
Ramp-up:support to the scientific activities during the preparation phases.
5 / 21
DPC definition status DPC latest news Support to sims and DA activities
DPC content
1 Development environment
ease the collaborative work
guarantee reproducibility of a rapidly evolving and composite data processing pipeline
keep control of performance, precision, readibility, etc
2 Data base / data model
sharing data among people and computing centers
tracability wrt code, parameters, input, etc
3 Execution environment
CPU load balanced between several DCC (Data Computing Center)
Pooling of CPU resources with a single scheduler for all DCCs (using virtualization)
Other collaborative tools
wiki, mailing list, file hosting a la dropbox, chatting program (like slack), etc, etc
6 / 21
DPC definition status DPC latest news Support to sims and DA activities
Organisation for development phase
7 / 21
DPC definition status DPC latest news Support to sims and DA activities
In the meantime: http://elisadpc.in2p3/fr/home
8 / 21
DPC definition status DPC latest news Support to sims and DA activities
1. DPC definition status
2. DPC latest news
3. Support to sims and DA activities
9 / 21
DPC definition status DPC latest news Support to sims and DA activities
Agenda
10 / 21
DPC definition status DPC latest news Support to sims and DA activities
DPC kickoff meeting in April 2017
Context
Context: France has the leadership on the Consortium DPC
Several agencies have expressed their interest to contribute
The meeting has been set to see what kind of contribution could be envisaged
8 countries have participated: Germany, Spain, Portugal, Italy, Sweden, Switzerland,Nederland, US, UK.
Some questions addressed during the meeting
Both contributions to sofware development and hardware supplying (DCC) proposed
The DPC is the unique executive agent of the board data release policy.
The DPC should be able to redo/refine L0→L1 pipeline, need a commonenvironment with SOC.
Actions
Refine the perimeter of the DPC → DPC definition document
WP should be defined asap in order to negociate source of funding/responsabilitiesfor each of them.
11 / 21
DPC definition status DPC latest news Support to sims and DA activities
DPC France kickoff meeting in July 2017
Context
In preparation of the coming negociation with other agencies, need to define the keyWPs which France wants to lead.
What are the interests and expertises of French institutes which are planning to joinLISA in the coming years.
8 institutes have participated: LAL, LPC, Artemis, LAM, Onera, CPPM, CCIN2P3,IAP (CEA ?)
Some questions addressed during the meeting
A significant expertise in space missions shared by many institutes.
IN2P3 + CC will support the project.
Request for a two sided contribution : science + engineering
Actions
Set up a mailing list to keep the people involved in this meeting updated→ [email protected]
Coordinate the french DPC effort: this meeting !!!
12 / 21
DPC definition status DPC latest news Support to sims and DA activities
Data Analysis WG meeting, Edimburgh, September 10-13th
Context
Following on the DPC kickoff meeting, draft a list of scientific WP
and the science products the consortium will deliver to the community
1 Waveform modeling
2 Data analysis tool (common framework)
3 Instrument response modeling
4 Low latency pipeline (alert, flags)
5 Individual and global source identification codes
6 Sources catalogues
7 Multi messenger, multi band
8 Interpretation , key-science projects
Actions
Circulate the draft list
Present and discuss the list at the consortium meeting
Some WP are due before phase A (modeling & assessing design choices)
13 / 21
DPC definition status DPC latest news Support to sims and DA activities
ESA Ground Segment meeting, ESTEC September 26th
Recall the list of documents and reviews for phase A and B
Coordinate the MOC / SOC / DPC plans to improve the mission definition.
SOC is the hub, will host a consortium project office (DPC team + IOT)
A consortium team will support L1 data processing at SOC
A common infrastructure between DPC and SOC DPC
14 / 21
DPC definition status DPC latest news Support to sims and DA activities
DPC support to mission definition activities
The protoDPC provides
a document management system to host the collaborationdocuments (through IN2P3/Atrium system and team).
95 external users (non IN2P3)including ∼ 10 ESA, ∼ 10 NASA, ∼ 10 CNES
in a short term: authentication + authorization systemprovided by AEI.
in the meantime, define a procedure to join thecollaboration (PI)
Short term perspectives
Connect the centralized authentication system to all DPCservices
Improve the document mamagement system (languageand other issues)
Calendar tool ? Wiki ? ... ?
15 / 21
DPC definition status DPC latest news Support to sims and DA activities
1. DPC definition status
2. DPC latest news
3. Support to sims and DA activities
16 / 21
DPC definition status DPC latest news Support to sims and DA activities
Simulation WG meeting, Zurich, September 4th
Context
Simulations required for phase A
Schedule a working plan for the coming months and years.
Some questions addressed during the meeting
Status of the current simulator / is it realistic ?
LPF heritage
Task list with priorities, people involved, duedate
Interaction/support to DA WG (LDC)
Simulation framework
TDI:ClockLaser
OB longTM tiltSC tilt
XYZAET
Telemetry
(4) GW signal extraction
Masterlaser
RINF-noise
OB PD
TM dynamicsSC dynamicsF_het
Delta phi, RIN
Electronicnoise
(1)GW source model
clocknoise
PMOutput
SciRedTMranging
PMOutput
SciRedTMranging
PMOutput
SciRedTMranging
Source location,orbits
slavelaser
RINF-noise (carrier)F-noise,RIN (side)PLC
noise
Delta-t
PM
Gaps
I_photo
Delta-S x 3Arm response, frequency plan,
antenna switching
(2) Constellation
Simulator
Filters, truncation
Preprocessing,Timestamp,armlength
slavelaser
RINF-noise (carrier)F-noise,RIN (side)PLC
noise
Delta-t
slavelaser
RINF-noise (carrier)F-noise,RIN (side)PLC
noise
Delta-t
slavelaser
RINF-noise (carrier)F-noise,RIN (side)PLC
noise
Delta-t
slavelaser
RINF-noise (carrier)F-noise,RIN (side)PLC
noise
Delta-t
(3) On Ground Processing
cavity
OBC, downsampling
System Simulator Drawing from the LISA Sim Meeting, Zurich
Sep 6, 2017
Actions
Luigi Ferraioli WG coordinator
weekly telecons (tuesday 16h CEST), mailing-list [email protected] / 21
DPC definition status DPC latest news Support to sims and DA activities
Support to simulation activities
The protoDPC provides
the version management system (Git),
continuous integration (Jenkins/Gitlab CI)
support to construction and deployment tool(docker)
Perspectives
LISACode → LISANode: a graph approach to build a model of the instrument
short term: support to the development, such that the framework could be used forthe entire mission.
long term: simulation to become DPC products (in the LISA DB + meta datarequired for traceability).
18 / 21
DPC definition status DPC latest news Support to sims and DA activities
LISA Data Challenges kickoff telecon May 5th
A Twofold plan
simple sources & realistic instrumental effects: needed for phase A
simplified instrument & a lot of sources: needed for DA preparation
Philosophy
going towards reproducible results
targeted challenges running on their own timescale
anyone can register, but LISA people have to participate to the consortiumcoordinated effort (telecon, DPC, etc).
Actions and schedule
Stas Babak LDC coordinator
biweekly telecons (friday 16h CEST),
mailing-list [email protected], gitlab projecthttps://gitlab.in2p3.fr/stas/MLDC
first data set ready for next month
19 / 21
DPC definition status DPC latest news Support to sims and DA activities
Support to LDC activities
The protoDPC provides
a web frontend + its postgresql data base toshare LDC data.
for now: data = Id - Format - Author - Tag -Date - File - Description - Thumbnail
some CPU resources to run the simulator
Perspectives
Define the LDC data model (could be used to help defining the LISA data model)
Develoment of tools to analyse the results, I/O optimized library, ...
Work on the reproducibility of the results: gather some of the analysis codes withinthe DPC with their parameters and execution environment.
20 / 21
DPC definition status DPC latest news Support to sims and DA activities
Conclusion: we need your help !
On the scientific side
LISA data analysis is challenging
Need to start to work on sims and pipeline now
Upcoming ESA reviews will assess our capability to analysis the data
LDC will restart this year
The simulation WG is very active, urgent need for scientific analysis of the designchoices
On the system side
Main task is to support all of the scientific activities listed above
Some R&D also needed to refine the DPC definitionI on the computing model (a single job queue for all DCC)I on the data model (following on LPF lessons learned)I ...
21 / 21