agent based parallel processing tool for big scientific...
TRANSCRIPT
![Page 1: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/1.jpg)
Agent based parallel processing tool for Big Scientific Data
A. Tsaregorodtsev, CPPM-IN2P3-CNRS, Marseille
IHEP, Beijing, 5 May 2015
![Page 2: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/2.jpg)
Plan
2
} The problem of the HEP data processing
} DIRAC Project } Agent based Workload Management System } Accessible computing resources } Data Management } Interfaces
} DIRAC as a service
} Conclusions
![Page 3: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/3.jpg)
Big Data
3
} Data that exceeds the boundaries and sizes of normal processing capabilities, forcing you to take a non-traditional approach for the treatment
} Google trends:
Clouds
Grids Big Data
![Page 4: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/4.jpg)
Big Data
http://www.wired.com/magazine/2013/04/bigdata
4
LHC RAW data per year
§ Already in 2010 the LHC experiments produced 13 PB of data
– That rate outstripped any other scientific effort going on
![Page 5: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/5.jpg)
Big Data
http://www.wired.com/magazine/2013/04/bigdata
5
LHC RAW data per year
WLCG data on the Grid
§ LHC RAW data volumes are inflated by storing derived data products, replication for safety and efficient access, and by the need for storing even more simulated data than the RAW data:
§ ~130 PB overall
![Page 6: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/6.jpg)
6
200-400 MB/sec
Data flow to permanent storage: 4-6 GB/sec
~ 4 GB/sec
1-2 GB/sec
1-2 GB/sec
![Page 7: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/7.jpg)
Massive computations: HEP
7
} LHC experiments pioneered the massive use of
computational grids as a solution to the High Energy Physics Big Data problem } 10s of PBytes of data per year } 100s of thousands CPUs in 100s of centers } 10s of Gbyte/s data transfers } 100s of users from 100s of institutions
} CERN Director General Rolf Heuer about the Higgs discovery: "It was a global effort and it is a global success. The results today are only possible because of the extraordinary performance of the accelerators, including the infrastructure, the experiments, and the Grid computing.”
} Other domains are catching up quickly with the HEP experiments } Life sciences, earth sciences, astrophysics, social sciences, etc
![Page 8: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/8.jpg)
Worldwide LHC Computing Grid Collaboration (WLCG)
• Distributed infrastructure of 150 computing centers in 40 countries • 300+ k CPU cores (~ 2M HEP-SPEC-06) • The biggest site with ~50k CPU cores, 12 T2 with 2-30k CPU cores • Distributed data, services and operation infrastructure 8
![Page 9: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/9.jpg)
Computing grids
9
} Grids are providing common infrastructure and rules to integrate multiple computing centers } Common computing element access protocols } Common user payload scheduling system } Common policies and security rules
} Users are signing up once and have access to multiple computing centers
} Common monitoring of the user jobs } Common accounting of computing resources
![Page 10: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/10.jpg)
Standard Grid Architecture
10
} Centralized Workload Management System } Computing sites publish information on their availability } Users submit jobs to the central repository } The central Resource Broker (RB) pushes user jobs to sites
after matching the job/resource compatibility
![Page 11: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/11.jpg)
Standard grid problems
11
} Heavy reliance on the site status information flow } Any delays result in suboptimal scheduling decisions
} Insufficient RB power for large infrastructures } Necessity to deploy multiple central (!) RB’s
} Heavy infrastructures required on the sites – resources providers } Providing timely status, accounting and monitoring information } Ensuring per user authentication/authorization } Managing community policies via local batch queues
parameterization
![Page 12: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/12.jpg)
Virtualized Computing Resources
12
} Since recently computing centers started to provide their resources using cloud technologies } Virtualized computing infrastructures } Very flexible – provides resources adapted to the user needs
} Operating systems, memory, CPU power, etc.
} However, large scientific collaborations can have access to multiple computational clouds } Dealing with independent clouds separately requires a huge
management effort } Federating multiple clouds into a single coherent system is
necessary to provide a transparent access for users.
![Page 13: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/13.jpg)
DIRAC Grid Solution
} DIRAC is developed originally for the LHCb experiment at with the goals:} Integrate all the heterogeneous computing resources
available to the community} Provide solution for both WMS and DMS tasks} Minimize human intervention at sites providers of resources } Make the grid convenient for the users:
} Simpler intuitive interfaces} Fault tolerance, quicker turnaround of user jobs} Enabling Community policies
} DIRAC has introduced a notion of Pilot Jobs} A concept of distributed agents bringing together various
disjoint computing resources
13
![Page 14: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/14.jpg)
DIRAC Workload Management
14
![Page 15: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/15.jpg)
PhysicistUser
EGIPilotDirector
EGI/WLCGGrid
NDGPilotDirector
NDGGrid
AmazonPilotDirector
Amazon EC2 Cloud
CREAMPilotDirector
CREAMCE
MatcherService
ProductionManager
15
![Page 16: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/16.jpg)
WMS: using heterogeneous resources
} Including resources in different gridsand standalone clusters is simple with Pilot Jobs} Needs a specialized Pilot
Director per resource type} Users just see new sites
appearing in the job monitoring
16
![Page 17: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/17.jpg)
WMS: applying VO policies
17
u In DIRAC both User and Production jobs are treated by the same WMS } Same Task Queue
u This allows to apply efficiently policies for the whole VOª Assigning Job Priorities for different
groups and activitiesª Static group priorities are used currentlyª More powerful scheduler can be plugged in
● demonstrated with MAUI scheduler
● Users perceive the DIRAC WMS as a single large batch system
![Page 18: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/18.jpg)
DIRAC as a resource manager
18
![Page 19: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/19.jpg)
Computing Grids
19
} DIRAC was initially developed with the focus on accessing conventional Grid computing resources } WLCG grid resources for the LHCb Collaboration
} It fully supports gLite middleware based grids } European Grid Infrastructure (EGI), Latin America GISELA, etc
} Using gLite/EMI middleware } Northern American Open Science Grid (OSG)
} Using VDT middleware } Northern European Grid (NDGF)
} Using ARC middleware
} Other types of grids can be supported } As long we have customers needing that
![Page 20: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/20.jpg)
Clouds
20
} VM scheduler developed for Belle MC production system} Dynamic VM spawning taking
Amazon EC2 spot prices and Task Queue state into account
} Discarding VMs automatically when no more needed
} The DIRAC VM scheduler by means of dedicated VM Directors is interfaced to } OCCI compliant clouds:
} OpenStack, OpenNebula } CloudStack} Amazon EC2
![Page 21: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/21.jpg)
Standalone computing clusters
21
} Off-site Pilot Director} Site delegates control to the central
service} Site must only define a dedicated
local user account} The payload submission through the
SSH tunnel
} The site can be a single computer or a cluster with a batch system} LSF, BQS, SGE, PBS/Torque,
Condor, OAR} More to come:
} SLURM, LoadLeveler. etc
} The user payload is executed with the owner credentials} No security compromises with respect
to external services
![Page 22: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/22.jpg)
Standalone computing clusters
22
Examples: } DIRAC.Yandex.ru
} >2000 cores} Torque batch system, no grid
middleware, access by SSH} Second largest LHCb MC production
site
} LRZ Computing Center, Munich} SLURM batch system, GRAM5 CE
service} Gateway access by GSISSH} Considerable resources for
biomed community (work in progress)
} Mesocentre Aix-Marseille University} OAR batch system, no grid
middleware, access by SSH} Open to multiple communities (work
in progress)
![Page 23: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/23.jpg)
BOINC Desktop Grids
23
} On the client PC the third party components are installed: } VirtualBox hypervisor } Standard BOINC client
} A special BOINC application } Starts a requested VM within
the VirtualBox } Passes the Pilot Job to the VM
and starts it
} Once the Pilot Job starts in the VM, the user PC becomes a normal DIRAC Worker Node
![Page 24: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/24.jpg)
Data Storage and Catalogs
24
} Managing data is more complicated than workload management } Already existing data in various storage resources must be
connected } Contents of already existing File Catalogs must be either
connected or copied to the DIRAC File Catalog
} Need for common interfaces to various types of storage and catalog services
![Page 25: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/25.jpg)
Data Management components
} For DIRAC users the use of any Storage Element or File Catalog is transparent} Up to a user community to
choose components to use} Different SE types can be
mixed together} Several File Catalogs can be
used in parallel} Complementary functionality} Redundancy
} Users see depending on the DIRAC Configuration} Logical Storage Elements
} e.g. DIRAC-USER, M3PEC-disk} Logical File Catalog
25
![Page 26: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/26.jpg)
Interware: DMS
26
} Data Management } Multiple types of Storage Elements ( SRM, XRootD,
iRods, DAV, DIP, … ) seen as logical entities for the users
} SE Proxy service for protocol independence } Data replication:
} FTS3 ¨ File Transfer Service
} Third party } Local pipe
![Page 27: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/27.jpg)
Interware: DMS
27
} DIRAC File Catalog } Replica Catalog } User Metadata Catalog } Support for data provenance information } Efficient storage usage reports
} Allow for user quota policies
} Support for user defined datasets
} LCG File Catalog
} Specialized File Catalogs
![Page 28: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/28.jpg)
Resource provisioning service
28
} DIRAC attempts to provide access to all kinds of existing resources and services useful for its users
} At the same time it provides its own solutions, e.g. catalogs, storage and computing element, transfer service, etc.
} By design services from different providers can be used together in parallel not necessarily replacing one another } Example: use of DFC and LFC together, see below
} The final choice of the services to use is left to the user
![Page 29: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/29.jpg)
Interware
29
} DIRAC has all the necessary components to build ad-hoc grid infrastructures interconnecting computing resources of different types. This allows to speak about the DIRAC interware.
![Page 30: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/30.jpg)
Interfaces
30
![Page 31: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/31.jpg)
DIRAC: Secure Web Portal
31
} Focus on the Web Portal as the main user tool for interactions with the grid
} Intuitive desktop application like interface} Ajax, Tornado, ExtJS Javascript library
} Monitoring and control of all activities} User registration, proxy upload} User job monitoring and manipulation, downloading results} Data manipulation and downloads} DIRAC Systems configuration and management
} Secure access } Standard grid certificates} Fine grained authorization rules
![Page 32: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/32.jpg)
Web Portal examples
32
![Page 33: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/33.jpg)
Application Web Portals
} Specific application portals can be built in the DIRAC Web Portal framework } Community Application Servers
} DIRAC RESTful interface} Language neutral} Suitable to use with portals written in Java, PHP, etc
} Other interfaces include} Extensive Python API} A rich set of command line tools ( >200 commands )
} Available on UNIX-like systems
33
![Page 34: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/34.jpg)
DIRAC Users: large communities
34
![Page 35: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/35.jpg)
LHCb Collaboration
35
} Up to 100K concurrent jobs in ~120 distinct sites} Equivalent to running a virtual computing center with a power of
100K CPU cores} Further optimizations to increase the capacity are possible● Hardware, database optimizations, service load balancing, etc
![Page 36: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/36.jpg)
Experiments: Belle II
36
} Combination of the non-grid, grid sites and (commercial) clouds is a requirement
} 2 GB/s, 40 PB of data in 2019} Belle II grid resources
} WLCG, OSG grids} KEK Computing Center} Amazon EC2 cloud
} First production run is done Thomas Kuhr, Belle II
![Page 37: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/37.jpg)
Belle II
37
} DIRAC Scalability tests
Hideki Miyake, KEK
![Page 38: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/38.jpg)
Community installations
38
} ILC/CLIC detector Collaboration, Calice VO } Dedicated installation at CERN, 10 servers, DB-OD MySQL server } MC simulations } DIRAC File Catalog was developed to meet the ILC/CLIC requirements
} BES III, IHEP, China } Using DIRAC DMS: File Replica and Metadata Catalog, Transfer services } Dataset management developed for the needs of BES III
} CTA } CTA started as France-Grilles DIRAC service customer } Now is using a dedicated installation at PIC, Barcelona } Using complex workflows
} Geant4 } Dedicated installation at CERN } Validation of MC simulation software releases
} Alpes Laser } Company using dedicated DIRAC setup for simulation of the laser hardware } Mostly using commercial Amazon resources
} DIRAC evaluations by other experiments } LSST, Auger, TREND, Daya Bay, Juno … } Evaluations can be done with general purpose DIRAC services
![Page 39: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/39.jpg)
DIRAC as a Service
39
![Page 40: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/40.jpg)
DIRAC as a service
40
} DIRAC client is easy to install} Part of a usual tutorial
} DIRAC services are easy to install but} Needs dedicated hardware for hosting} Configuration, maintenance needs expert manpower} Monitoring computing resources is a tedious every-day task
} Small user communities can not afford maintaining dedicated DIRAC services} Still need easy access to computing resources
} Large grid infrastructures can provide DIRAC services for their users.
![Page 41: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/41.jpg)
France-Grid DIRAC service
41
} Several regional and university campus installations in France} Complex maintenance
} Joint effort to provide France-Grid DIRAC service} Hosted by the CC/IN2P3,
Lyon, T1 center} Distributed team of service
administrators} 15 VOs, ~100 registered users
} Mostly biomed applictions
http://dirac.france-grilles.fr
![Page 42: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/42.jpg)
Multi-VO DIRAC service in IHEP
42
} IHEP Computing Center is the main resource provider for the BES experiment
} The BES Collaboration decided to use DIRAC to build its distributed computing project } Developed a good expertize level in developing and managing DIRAC
services
} The BES dedicated DIRAC service is being extended to support other user communities
} CEPC, Juno, Daya Bay, … } Non-HEP communities ?
} Reuse of acquired experience } Single Administration team } Minimization of the user support and maintenance effort
![Page 43: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/43.jpg)
Conclusions
43
![Page 44: Agent based parallel processing tool for Big Scientific Dataindico.ihep.ac.cn/event/4964/contribution/2/material/slides/0.pdf · Grid Collaboration (WLCG) • Distributed infrastructure](https://reader033.vdocument.in/reader033/viewer/2022060420/5f16dd1b75dc477bac3a7711/html5/thumbnails/44.jpg)
Conclusions
44
} The computational grids are no more something exotic, they are used in a daily work for various applications
} Agent based workload management architecture allows to seamlessly integrate different kinds of grids, clouds and other computing resources
} DIRAC is providing a framework for building distributed computing systems and a rich set of ready to use services. This is used now in a number of DIRAC service projects on a regional and national levels
} Services based on DIRAC technologies can help users to get started in the world of distributed computations and reveal its full potential
http://diracgrid.org