LHC-CMS Tier2 facility at TIFR

Kajari Mazumdar Department of High Energy Physics

Tata Institute of Fundamental Research Mumbai, India.

http://indiacms.res.inT2-IN-TIFR

NKN Workshop, IIT, Bombay November 1, 2012

e-Science and e-Research

Large collaborative research is made possible by sharing resources across the

globe via internet (data, computation, people’s expertise...).

– Crosses organisational , national and international boundaries.

CERN-LHC project is an excellent example of large scale colalboration dealing

with huge amount of data.

• WWW was born in CERN to satisfy the needs of previous generation HEP

experiments.

• LHC has been a driving force for GRID computing technology which evolved

naturally from internet .

High Energy Physics(HEP) is often compute as well as data intensive.

•Experiments performed by several thousand scientists over several years.

The GRID Computing Goal • Science without borders. • Provide Resources and Services to store/serve O(10) PB data/year

• Provide access to all interesting physics events to O(4000) collaborators

• Minimize constraints due to user localisation and resource variety • Decentralize control and costs of computing infrastructure Solution through LHC Computing GRID Much faster delivery of physics

To begin at the end:

• The operations of the LHC machine and the experiments have been a great success.

• Fantastic output, often only days after the data is taken about 200 scientific publications per experiment in 3 years.

LHC Computing Grid is the backbone of the success story

Amount of data written by LHC experiments during last 6 months: 19 Petabyte. Expected volume of data by end of 1st phase of LHC operation: 30 PB. Total transfer rate across globe: ~ 10 Gbps

• Today >140 sites • ~250k CPU cores • ~100 PB disk

Internal speed of computers have become comparable to the speed of large distance network.

Large Hadron Collider (LHC)

• 27 km circumference • at 1.9 K • at 10-13 Torr • at 50-175 m below surface • > 10K magnets 4 big experiments: >10K scientists, students, engineers. Operational since 2009, Q4.

Largest ever scientific project 20 years to plan, build 20 years to work with

Big Bang

Today

~300‘000 years

Experiments in Astrophysics & Cosmology

WMAP (2001) COBE(1989)

LHC: ~ 10-12 seconds (p-p) ~ 10-6 seconds (Pb-Pb)

What happens in LHC experiment

Proton-Proton 1380 bunch/beam Protons/bunch 2. 1011

Beam energy 4 TeV / proton Luminosity 7.6*1033 /cm 2/s Crossing rate 20 MHz Total event rate 108 Hz

2012

2 major multipurpose experiments at LHC • A Toroidal LHC Apparatus (ATLAS) • Compact Muon Solenoid (CMS)

• 10 Million electronic channels per experiment . • Proton bunches collide every 50 nano sec. • Higgs production <1 per sec.

Example of a modern detector 3170 scientists and engineers including ~800 students 169 institutes in 39 countries

India

Data rates @ CMS as foreseen for design parameters

Today’s data collection rate ~ 850 Hz

Tier 0

Tier 1 National centres

Tier 2 Regional groups in a continent/nation

Different Universities, Institutes in a country

Individual scientist’s PC,laptop, ..

Experimental site

CERN computer centre, Geneva

ASIA (Taiwan)

India China Korea Taiwan

France Italy Germany USA

TIFR BARC Panjab

Univ.

Indiacms T2_IN_TIFR

Layered Structure of CMS GRID connecting computers across globe

Delhi Univ.

Pakistan

Online data recording

Several Petabytes/sec.

10 Gbps

1-2.5 Gbps

CERN

CMS in Total: 1 Tier-0 at CERN 7 Tier-1s on 3 continents 50 Tier-2s on 4 continents

CMS T2 in India : one of the 5 in Asia-Pacific region Today : 7 collaborating institutes in CMS: ~ 30 scientists + 40 students 2.1% of signing authors in publication, Contributing to computing resource of CMS ~ 3%

Distributed Analysis is not a wish, it is a necessity tools have to be reliable!

•First job of the offline system is to process and monitor data at T0. •T0: 1 M jobs/day + test-jobs. traffic : 4Gbps input, > 13 Gbps served •CERN Tier 0 moves ~ 1 PB data per day, automated subscription • T1 processes data further several times a year, coordinates with T2s • T2s are the real workhorses of the system with growing roles Hosts specific data streams for analysis Gets main data from T1s, recently more communications among T2s Typically 100k analysis jobs/day/experiment Site readiness is at high level 24X7 availability lot of invisible effort! Storage in T2 partitioned into central and local space.

Quick description of LHC grid tiers

Middleware: - Storage Elements - Computing Elements - Workload Management - Local File Catalog - Information System - Virtual Organisation management - Inter-operability among different GRIDs

Resources - CPU : 3000 (7000) - Disk : 700 TB (950 TB) - Nework (WAN) : 1.5 - 2 Gbit/s

T2-IN-TIFR, Mumbai • Contributing to the computing efforts of CMS experiment credits earned against mandatory service jobs.

End-user does not have to bother about the resources

A large amount of invisible human effort is essential

1.5/2 Gbps to CERN peered to GEANT

2.5 Gbps NKN +TEIN3

TIFR-INDIACMS T2

1 Gbps to VECC RRCAT, IPR

Network connections

VECC/SINP: INDIA

Monitoring the performance of a CMS T2 site

• Availability and reliability : fraction of time all functional tests in a site are successful

• Data transfers number of active links with sustained rates • Job Robot (simulates small analysis jobs/user jobs) for failures, the logfile describes the nature. the investigations carried out locally + central experts.

~1.5× 109 events produced in 2011

Global status

Data Transfers from/to TIFR

•Total data volume at present ~ 250 TB

•Total transfers during last 6 months ~ 70 TB

• TIFR hosting i) centrally managed data (simulated, custodial) ii) collision data skims

• Current CMS total CPU pledge at T2s : 18k jobs slots • Nominal Analysis pledge : 50% • Slot utilization during Summer/Fall 09 was reasonable but need to go into sustained analysis mode

upload

download

August 15-18, 2011 Maximum: 1.5 Gbps Avg. : 1Gbps

Low latency transfers

Physics datasets for analysis

•Distribution of data to participating centres all over the world •Huge datasets (~ few TB) cannot be transferred to each user the analysis jobs go to “where the data is” Results come to user!

The KEY of GRID Computing: NETWORK

Need a service for monitoring networks.

• CMS Tier2 center at TIFR is in working condition and satisfies all CMS guidelines and requirements.

• Making what we have today more sustainable is a challenge

Conclusion

Sincere thanks to ERNET, EU-India grid and NKN for the support towards our effort in puttng India in CMS-Grid map.

Need to invest MORE in networks across the country and make full use of the distributed system for a remote participation to be satisfactory!

Backup

LHC is meant to resolve some of the most puzzling issues in physics: • Nature of elementary particles and interactions shortly after Big Bang how many interactions when the universe was much hotter which elementary particles existed with what properties? • we recreate conditions of very early universe at LHC. • Origin of the mass mass patterns among different particles in today’s universe why photon is massless, while carriers of weak interaction are massive? if the symmetry is broken spontaneously, what is the signature? existence of the God Particle? • The Higgs boson yet to be discovered, but coming soon, stay tuned! LHC is at the threshold of discovery likely to change the way we are used to think of Nature!

Motivation of LHC experiments

• Nature of dark matter: we know only 4% of the constituent of the universe • A good 25% of the rest is massive enough to dictate the motion of galaxies non-luminous, and hence “dark” LHC can tell us the nature of this dark matter! LHC will also shed light on: • why there is only matter and no antimatter today. • properties of the 4th state of matter: Quark-Gluon-Plasma which existed 1 pico sec. after the big bang, before formation of neutrons and protons. ….

Grand menu from LHC

All this is possible because LHC is essentially a microscope AND a telescope as well!

observed

Expected from visible Distribution of matter

•Versatile experiments, equipped with very specialized detectors. ~107 electronic channels per experiment, ready every 25 ns to collect information of debris from violent collisions. Reconstruct 20K charged tracks in a single event (lead-lead collisions at LHC)

Challenges

10 vertices in a single proton-proton collision, to be discriminated from the interesting process

• Event size related to flux/intensity • 1.5 Billion events recorded in 2010 • > 2B events, much more complicated, to be recorded during 2011 resource utilization to be prioritized by carefully throwing not-so-interesting collisions.

Charged tracks from heavy ion collision vertex

Enter a New Era in Fundamental Science

Exploration of a new energy frontier in p-p and Pb-Pb collisions

LHC ring: 27 km circumference

CMS

ALICE

LHCb

ATLAS

Discovered Higgs boson. Stay tuned!