ligo-g040107-00-zlsc meeting march 2004 search pipelines for binary inspiral duncan brown inspiral...

LIGO-G040107-00-Z LSC Meeting March 2004

Search Pipelines for Binary Inspiral

Duncan BrownInspiral Working Group

University of Wisconsin-Milwaukee

LIGO-G040107-00-Z

LIGO-G040107-00-Z LSC Meeting March 2004

S2 Inspiral Pipeline

lalapps_tmpltbank

LIGO-G040107-00-Z LSC Meeting March 2004

S2 Inspiral Pipeline

lalapps_inspiral

LIGO-G040107-00-Z LSC Meeting March 2004

S2 Inspiral Pipeline

lalapps_inca

LIGO-G040107-00-Z LSC Meeting March 2004

S2 Inspiral Pipeline

lalapps_inca

LIGO-G040107-00-Z LSC Meeting March 2004

S2 Inspiral Pipeline

lalapps_inspiral

LIGO-G040107-00-Z LSC Meeting March 2004

S2 Inspiral Pipeline

lalapps_inca

LIGO-G040107-00-Z LSC Meeting March 2004

S2 Inspiral Pipeline

lalapps_inca

LIGO-G040107-00-Z LSC Meeting March 2004

S2 Inspiral Pipeline

LIGO_LW XML file

LIGO-G040107-00-Z LSC Meeting March 2004

Pipeline Infrastructure Requirements

• Ensure that all data is analyzed

• Automate pipeline as much as possible

• Provide flexible pipeline for testing and tuning

• Allow easy construction of complex workflows

• Simple reusable infrastructure

• Easy to debug

LIGO-G040107-00-Z LSC Meeting March 2004

Pipeline Implementation

• Condor to manage job submission to cluster

• lalapps code to execute components of pipeline» Use LAL functions for GW analysis

• Condor DAGman to manage execution of pipeline

• Standard file types for I/O» Read AS_Q and calibration from frame data» Writes triggers as LIGO_LW XML» Can write r(t), x2(t), PSD, filter data as frames

LIGO-G040107-00-Z LSC Meeting March 2004

Creation of the DAG

• Simple Python modules in lalapps to build scripts that write pipeline

• lalapps/src/lalapps/pipeline.py» Read segwizard files» Manipulate science segments (union, intersection, inverse)» Create Condor Jobs and DAGs

• lalapps/src/inspiral/inspiral.py» Construction of DAG nodes specific to inspiral

• lalapps/src/inspiral/inspiral_pipe.in» Use building blocks to construct pipeline

LIGO-G040107-00-Z LSC Meeting March 2004

Putting It All Together

data = pipeline.ScienceData()data.read(‘segwizard.txt’,2048)data.make_chunks(length,overlap,isplay)dag = pipeline.CondorDAG(‘mydag.dag’)datafind_job = pipeline.LSCDataFindJob()inspiral_job = inspiral.InspiralJob()

for seg in data: df = pipeline.LSCDataFindNode() df.set_start(seg.start()) df.set_end(seg.end()) for chunk in seg: insp = inspiral.InspiralNode() insp.set_start(chunk.start()) insp.set_end(chunk.end()) insp.add_parent(df)

dag.write()

LIGO-G040107-00-Z LSC Meeting March 2004

Putting It All Together

data = pipeline.ScienceData()data.read(‘segwizard.txt’,2048)data.make_chunks(length,overlap,isplay)dag = pipeline.CondorDAG(‘mydag.dag’)datafind_job = pipeline.LSCDataFindJob()inspiral_job = inspiral.InspiralJob()

for seg in data: df = pipeline.LSCDataFindNode() df.set_start(seg.start()) df.set_end(seg.end()) for chunk in seg: insp = inspiral.InspiralNode() insp.set_start(chunk.start()) insp.set_end(chunk.end()) insp.add_parent(df)

dag.write()

LIGO-G040107-00-Z LSC Meeting March 2004

Putting It All Together

data = pipeline.ScienceData()data.read(‘segwizard.txt’,2048)data.make_chunks(length,overlap,isplay)dag = pipeline.CondorDAG(‘mydag.dag’)datafind_job = pipeline.LSCDataFindJob()inspiral_job = inspiral.InspiralJob()

for seg in data: df = pipeline.LSCDataFindNode() df.set_start(seg.start()) df.set_end(seg.end()) for chunk in seg: insp = inspiral.InspiralNode() insp.set_start(chunk.start()) insp.set_end(chunk.end()) insp.add_parent(df)

dag.write()

LIGO-G040107-00-Z LSC Meeting March 2004

Putting It All Together

data = pipeline.ScienceData()data.read(‘segwizard.txt’,2048)data.make_chunks(length,overlap,isplay)dag = pipeline.CondorDAG(‘mydag.dag’)datafind_job = pipeline.LSCDataFindJob()inspiral_job = inspiral.InspiralJob()

for seg in data: df = pipeline.LSCDataFindNode() df.set_start(seg.start()) df.set_end(seg.end()) for chunk in seg: insp = inspiral.InspiralNode() insp.set_start(chunk.start()) insp.set_end(chunk.end()) insp.add_parent(df)

dag.write()

LIGO-G040107-00-Z LSC Meeting March 2004

Putting It All Together

data = pipeline.ScienceData()data.read(‘segwizard.txt’,2048)data.make_chunks(length,overlap,isplay)dag = pipeline.CondorDAG(‘mydag.dag’)datafind_job = pipeline.LSCDataFindJob()inspiral_job = inspiral.InspiralJob()

for seg in data: df = pipeline.LSCDataFindNode() df.set_start(seg.start()) df.set_end(seg.end()) for chunk in seg: insp = inspiral.InspiralNode() insp.set_start(chunk.start()) insp.set_end(chunk.end()) insp.add_parent(df)

dag.write()

LIGO-G040107-00-Z LSC Meeting March 2004

Putting It All Together

data = pipeline.ScienceData()data.read(‘segwizard.txt’,2048)data.make_chunks(length,overlap,isplay)dag = pipeline.CondorDAG(‘mydag.dag’)datafind_job = pipeline.LSCDataFindJob()inspiral_job = inspiral.InspiralJob()

for seg in data: df = pipeline.LSCDataFindNode() df.set_start(seg.start()) df.set_end(seg.end()) for chunk in seg: insp = inspiral.InspiralNode() insp.set_start(chunk.start()) insp.set_end(chunk.end()) insp.add_parent(df)

dag.write()

LIGO-G040107-00-Z LSC Meeting March 2004

Putting It All Together

data = pipeline.ScienceData()data.read(‘segwizard.txt’,2048)data.make_chunks(length,overlap,isplay)dag = pipeline.CondorDAG(‘mydag.dag’)datafind_job = pipeline.LSCDataFindJob()inspiral_job = inspiral.InspiralJob()

for seg in data: df = pipeline.LSCDataFindNode() df.set_start(seg.start()) df.set_end(seg.end()) for chunk in seg: insp = inspiral.InspiralNode() insp.set_start(chunk.start()) insp.set_end(chunk.end()) insp.add_parent(df)

dag.write()

LIGO-G040107-00-Z LSC Meeting March 2004

S2 Inspiral DAG

LIGO-G040107-00-Z LSC Meeting March 2004

Conclusions

• Use of Condor DAGman has been very successful» Simplifies management of analysis workflow» More time to concentrate on scientific questions

• Infrastructure written in lalapps is simple to use» Python modules are documented in lalapps documentation

• Reusable code» LIGO/TAMA inspiral analysis (Steve Fairhurst)» Stochastic lalapps pipeline (Adam Mercer)

• Fast, simple, efficient!