lofar information system on grid a.n.belikov. lofar long term archive prototypes: egee astro-wise...

Lofar Information System on GRID

A.N.Belikov

Lofar Long Term Archive

Prototypes: EGEE Astro-WISE

Requirements to data storage

Tiers

Astro-WISE adaptation

Problem: optimal data placement

Lofar: Science

Lofar: Network

10 Gbps optical

single data processing center for raw data (CIT, Groningen)

Lofar: Data Processing

GbE

sw

itch

GbE

switc

hG

bEsw

itch

GbE

switc

hG

bEsw

itch

GbE

switc

h

Auxiliary processingsection

Stor

age

syst

em

Off-line processing/user software

Off-lineStorageOn-Line

Long term Archiving& Data ExportG

bEsw

itch

On-Line pipeline processingBlue Gene/L

WAN

Blue Gene/L

linux cluster

centralized data processing for raw data

long-term data storage

Lofar: LTA requirements

Lofar: LTA requirements

20 PB for 5 yearsdecentralized data storagedecentralized data reprocessinginternational partners

LHC & EGEE

15 PB data per year grid computing

EGEE: ~250 sites, >45000 CPU

www.eu-egee.org EGEE-III

EGEE: Tiers approach

Astro-Wise: grid

Astro-WISE: dataservers (1)

Astro-WISE: dataservers (2)

Astro-WiseLoWise

dataserverdata retrieved by URI0.5 TB 7 sec

3.5 TB 20 sec 1 PB 8 h

LoWISE: URI storage

LoWISE: Tiers

Astro-WISE-like interfaces user access (from PM to anonymous) Tier 1 as core element

LoWISE: data storage

distributed file systems: GPFS (IBM) Lustre (Sun)

dCAcheCASTOR

Lofar-WISE: Data Storage

initial configuration

distributed FS & EGEE

Oracle for metadata

Problem: users

users: observer, project managerprivileged users: operators

Problem: CE & SE geographical location file transfer user access research group driven

Problem: Access policy

Problem: Optimal Data Placement

priorities groups pool of queries TCF monitor system

Task

create and implement a system for an optimal data placement