Advanced LIGO: Science aims, timescales and computing challenges

Stephen FairhurstCardiff University

Gravitational Waves• Gravitational waves are a

feature of Einstein’s general theory of relativity

• Accelerating mass emits gravitational waves

• “Ripples in space-time” propagate away from source at the speed of light

• Typical magnitude ~10-23

2

A global network

LIGO Hanford, USA

LIGO Livingston,

USA GEO 600,Germany

Virgo,Italy

KAGRA,Japan

LIGOIndia

3

The Advanced LIGO Detectors

18

Quadruple Suspension for Advanced LIGO

C Torrie, M Perreur-Lloyd, E Elliffe, R Jones

From Harry et al, CQG (2010) 4

Detector sensitivity

From Harry et al, CQG (2010)

Sensitivity evolution

101

102

103

10−24

10−23

10−22

10−21

frequency (Hz)

stra

in n

ois

e a

mp

litu

de

(H

z−1

/2)

Advanced Virgo

Early (2016−17, 20 − 60 Mpc)Mid (2017−18, 60 − 85 Mpc)Late (2018−20, 65 − 115 Mpc)Design (2021, 130 Mpc)BNS−optimized (145 Mpc)

101

102

103

10−24

10−23

10−22

10−21

frequency (Hz)

stra

in n

ois

e a

mp

litu

de

(H

z−1

/2)

Advanced LIGO

Early (2015, 40 − 80 Mpc)Mid (2016−17, 80 − 120 Mpc)Late (2017−18, 120 − 170 Mpc)Design (2019, 200 Mpc)BNS−optimized (215 Mpc)

From Aasi et al, arXiv:1304.0670

Advanced LIGO today

101 102 10310−24

10−23

10−22

10−21

10−20

10−19

Frequency [Hz], (BW, ENBW) = (0.125, 0.1875) [Hz]

Stra

in [(

m/m

)/rtH

z]

H1 Strain Sensivity, Oct 01 2015 01:30:43 UTCInput Power [W], (DHorizon, DSenseMon) = (157, 70) [Mpc]

crea

ted

by p

rodu

ceof

ficia

lstra

inas

ds_O

1 on

01−

Oct−2

015

, J. K

isse

l

Data rates

• Sample rates determined by detector’s sensitive band (Hz to kHz)

• Calibrated gravitational wave strain is few TB per detector per year

8

LIGO-T1500118–v6

aLIGO Instrument Storage Requirements

Data Type File Rate Data Rate(After Compression)

Raw 492,750 files/yr 10.00 MB/sSecond-Trend 52,560 files/yr 1.10 MB/sMinute-Trend 8,760 files/yr 0.08 MB/sRDS 123,188 files/yr 2.00 MB/sCalibrated Strain 123,188 files/yr 0.12 MB/sSFTs 17,520 files/yr 0.02 MB/s

Single Interferometer Total: 818000 files/yr 13.32 MB/sBoth aLIGO Interferometers: 1.6M files/yr 800 TB/yr

Table 4: aLIGO data rates per copy of data for di↵erent data types.

aLIGO Data Analysis Storage Requirements

Search Group TB/yr

Burst 254Compact Binary Coalescences (CBC) 1167Continuous Waves (CW) 493Stochastic GW background (SGWB) 25Detector Characterization (Detchar) 223Total 2162

Table 5: Storage requirements for each working groups.

3.2 Data Requirements

Table 4 shows the expected file and data generation rates per aLIGO interferometer for varioustypes of data. Raw data, which contains all the channels designated for permanent archivingdominates the data rates and the number of files generated. For dual-copy of the data from thetwo aLIGO detectors, the Laboratory will archive around 800TB/yr.

In addition, the search group and data quality activities are expected to generate about⇠ 2200TB/yras shown in Table 5. These analysis results will be archived in the central data archive at Caltechalong with the primary instrument data. The resulting total data rate per copy is ⇠ 3000TB/yr.

Table 6 shows the estimated data collected in the observing runs between 2015 and 2018. The LSC

Advanced Detector Era Observing Runs

Year Estimated Duration Detectors2015 3 months H1, L1

2016-17 6 months H1, L1, V12017-18 9 months H1, L1, V1

Table 6: Schedule for Advanced Detector Era observing runs where the detectors are: H1, LIGO Hanford;L1, LIGO Livingston; V1, Virgo.

14

Sources of Gravitational WavesContinuous Waves (CW):Spinning Neutron Stars

Short Bursts:Supernovae,Gamma Ray Bursts

Coalescing Binaries (CBC):Merging black holes and neutron stars

Stochastic Sources (SGWB): GW from the Big Bang

9

CPU requirements

10

Millions of service units. 1 SU equals 1 core-hour of execution time on a reference Intel Xeon E5-2670 CPU

Available computing

MSU commitment in Q4 2015:

• 11 - AEI Hannover

• 8 - LIGO Lab

• 6 - Syracuse

• 3 - Cardiff

• 3 - Virgo

• 3 - UW-Milwaukee

11

UK Contribution, hosted on Cardiff’s Raven cluster:• 1,440 Haswell cores• 660 Westmere cores• 200TB Storage• Dedicated LIGO head node

Spinning Neutron Stars• Small “mountain” on a

spinning neutron star gives a continuous gravitational wave signal

• Motion of earth modulates signal frequency

• Fully coherent search requires more computing than available

• Search is run under the BOINC crowd-sourced distributed computing through Einstein@Home, delivering 2 petaflops of compute power.

12

Searching for binary mergers

Time of merger

SNR

X /

=

Data Waveform Sensitivity

13

Search Costs• The cost of the search scales with the required number of template waveforms

• Current searches use 300,000 templates

• In 2017-18, will require 2 million templates

• Search is “pleasingly parallel”.

• Cost also scales with number of detectors

• Currently 2 advanced LIGO detectors

• Increasing to 3 with addition of advanced Virgo in 2016

• Then 4 and 5 as KAGRA and (hopefully) LIGO India join, around 2020.

• Processing speed

• 5,000 templates per E5-2670 core

• Consumer grade GPUs give best throughput per £.14

Parameter estimation

• Detailed parameter estimation performed on event candidates

• Dominant computational cost is waveform generation

15

20

FIG. 7: Comparison of probability density functions for the BNS signal (table II) as determined by each sampler. Shown areselected 2D posterior density functions in greyscale, with red cross-hairs indicating the true parameter values, and contoursindicating the 90% credible region as estimated by each sampler. On the axes are superimposed the one-dimensional marginaldistributions for each parameter, as estimated by each sampler, and the true value indicated by a vertical red line. The colourscorrespond to blue: Bambi, magenta: Nest, green: MCMC. (left) The mass posterior distribution parametrized by chirp massand symmetric mass ratio. (centre) The location of the source on the sky. (right) The distance dL and inclination ✓JN of thesource showing the characteristic V-shaped degeneracy.

FIG. 8: Comparison of probability density functions for the NSBH signal (table II), with same color scheme as fig 7. (first rowleft) The mass posterior distribution parametrized by chirp mass and symmetric mass ratio. (first row centre) The location ofthe source on the sky. (first row right) The distance dL and inclination ✓JN of the source. In this case the V-shaped degeneracyis broken, but the large correlation between dL and ✓JN remains. (second row left) The spin magnitudes posterior distribution.(second row centre) The spin and mass of the most massive member of the binary illustrating the degeneracy between massand spin. (second row right) The spin and symmetric mass ratio.

21

M (M�) ⌘ m1 (M�) m2 (M�) d (Mpc) ↵ (rad) � (rad)

Nest 1.12531.12551.1251 0.24870.25

0.2447 1.41.51.3 1.21.3

1.1 197251115 3.193.24

3.14 �0.997�0.956�1.02

MCMC 1.12531.12551.1251 0.24870.25

0.2447 1.41.51.3 1.21.3

1.1 195250113 3.193.24

3.14 �0.998�0.958�1.02

BAMBI 1.12531.12551.1251 0.24870.25

0.2449 1.41.51.3 1.21.3

1.1 196251114 3.193.24

3.14 �0.998�0.958�1.02

Injected 1.1253 0.2497 1/3382 1.249 134.8 3.17 -0.97

TABLE III: BNS recovered parameters. Median values and 5%� 95% credible interval for a selection of parameters for eachof the sampling algorithms.

M(M�) ⌘ m1 (M�) m2 (M�) d (Mpc) a1 a2 ↵ (rad) � (rad)

Nest 3.423.483.36 0.110.23

0.076 11155.3 1.72.9

1.4 612767383 0.360.75

0.041 0.490.950.046 0.8430.874

0.811 0.4590.4950.422

MCMC 3.423.483.36 0.120.23

0.077 11155.3 1.72.9

1.4 601763369 0.350.73

0.038 0.480.940.045 0.8430.874

0.812 0.4590.4960.422

BAMBI 3.423.483.37 0.110.22

0.075 11155.8 1.62.7

1.3 609767378 0.360.72

0.042 0.490.950.044 0.8430.874

0.811 0.4590.4950.422

Injected 3.477 0.076 15 1.35 397 0.63 0.0 0.82 0.44

TABLE IV: NSBH recovered parameters, defined as above.

M (M�) ⌘ m1 (M�) m2 (M�) d (Mpc) a1 a2 ↵ (rad) � (rad)

Nest 9.59.89.3 0.150.217

0.12 24.329.316.3 5.57.7

4.7 647866424 0.340.66

0.082 0.480.950.049 0.210.29

0.14 �0.612�0.564�0.659

MCMC 9.59.89.3 0.150.23

0.12 23.829.114.8 5.58.2

4.7 630847404 0.360.78

0.092 0.510.950.05 0.210.3

0.14 �0.612�0.563�0.658

BAMBI 9.59.89.3 0.1490.216

0.12 24.529.216.3 5.47.5

4.7 638859428 0.350.69

0.087 0.490.940.049 0.210.29

0.14 �0.612�0.565�0.659

Injected 9.44 0.227 15 8 500 0.79 0.77 0.230 -0.617

TABLE V: BBH recovered parameters, defined as above.

The computationally easiest waveform to generate isTaylorF2, where an analytic expression for the waveformin the frequency domain is available. For the BNS signalsimulated here, around 50 waveforms can be generatedper second at our chosen configuration (32 s of data sam-pled at 4096 Hz). On the other hand, more sophisticatedwaveforms, like SpinTaylorT4 with precessing spins, re-quire solving di↵erential equations in the time domain,and a subsequent FFT (the likelihood is always calcu-lated in the frequency domain), which raises the CPUtime required to generate a single waveform by an orderof magnitude.

The structure of the parameter space a↵ects the lengthof a run in several ways. The first, and most obvious, isthrough the number of dimensions: when waveforms withprecessing spins are considered a 15-dimension parame-ter space must be explored, while in the simpler case ofnon-spinning signals the number of dimensions is 9. Theduration of a run will also depend on the correlationspresent in the parameter space, e.g. between the distanceand inclination parameters [38]. Generally speaking runswhere correlations are stronger will take longer to com-plete as the codes will need more template calculationsto e↵ectively sample the parameter space and find theregion of maximum likelihood.

Table VI shows a comparison of the e�ciency of eachcode running on each of the simulated signals in terms ofthe cost in CPU time, wall time, and the CPU/wall timetaken to generate each sample which ended up in theposterior distribution. These numbers were computedusing the same hardware, Intel Xeon E5-2670 2.6 GHzprocessors.

BNS Bambi Nest MCMC

posterior samples 6890 19879 8363

CPU time (s.) 3317486 1532692 725367

wall time (s.) 219549 338175 23927

CPU seconds/sample 481.5 77.1 86.7

wall seconds/sample 31.9 17.0 2.9

NSBH Bambi Nest MCMC

posterior samples 7847 20344 10049

CPU time (s.) 2823097 9463805 4854653

wall time (s.) 178432 2018936 171992

CPU seconds/sample 359.8 465.2 483.1

wall seconds/sample 22.7 99.2 17.1

BBH Bambi Nest MCMC

posterior samples 10920 34397 10115

CPU time (s.) 2518763 7216335 5436715

wall time (s.) 158681 1740435 200452

CPU seconds/sample 230.7 209.8 537.5

wall seconds/sample 14.5 50.6 19.8

TABLE VI: Preformance of all three sampling methods onthe three signals from table II. The time quoted in the “CPUtime” line is the cumulative CPU-time across multiple cores,while the time quoted in the “wall time” line is the actualtime taken to complete the sampling. The di↵erence is an in-dication of the varying degrees of parallelism in the methods.

We note that at the time of writing the three samplershave di↵erent level of parallelization, which explains thedi↵erences between codes of the ratio CPU time to walltime.

From Veitch et al 2014

Einstein Telescope• Proposed next generation

gravitational wave observatory

• factor of 10 improvement in sensitivity

• low frequency sensitivity to 1 Hz

• Computational challenge:

• waveforms 100x as long

• 100x as many templates16