infso-ri-031688 enabling grids for e-science egee-iii input from the peb erwin laure egee technical...

INFSO-RI-031688

Enabling Grids for E-sciencE

www.eu-egee.org

EGEE-III Input from the PEB

Erwin Laure

EGEE Technical Director

CERN, Switzerland

EGEE-III Planning 2


INFSO-RI-508833

Process

• After the first EGEE-III workshop the PEB refined the proposal for technical content– Vision and mission document (with issues)

https://edms.cern.ch/document/791303/1 – Refined tasks and produced first manpower estimates– Editorial team: Anna Cook, Frank Harris, Bob Jones, Erwin Laure

• This is main content of this presentation

• No assumptions on EGEE-III organization (activities) made, yet

• Effort levels based on what would technically make sense– Cuts will be necessary to meet the budget cuts and changes in FP7 cost models– These cuts should be made based on tasks not flat reduction

• Need to decide which tasks to keep inside EGEE, how to staff them, and how to best organize the tasks into activities– And how to organize cross-activity coherence– Have project wide deliverables and milestones?

EGEE-III Planning 3


INFSO-RI-508833

EGEE-III vision and mission

• The main vision of EGEE-III is to make a strong move towards a sustainable world-wide production quality Grid infrastructure by appropriate technical and organisational decisions.

• This must be capable of providing services to a rapidly increasing number of application areas, and make Grid technology easily accessible and usable for these communities.

• Issue: how much support is EGEE expected to give to 3rd parties wrt middleware, distribution, operational support and related documentation

EGEE-III Planning 4


INFSO-RI-508833

EGEE-III Goals

• The provision of continuous operational and deployment support – with world-wide coverage which can cope with the predicted growth in

sites and utilisation; • To establish and encourage best practices regarding security

– across all aspects of the project; applications, middleware, deployment and operations.

• The provision of reliable and performative Grid middleware foundation services – to users by development and maintenance services together with

appropriate integration and deployment;• The provision of selected high level middleware support services

– for selected, priority application areas, with an associated integration and deployment process;

• Collaboration with mature applications domains, – in particular via selected centres of excellence, as primary source of

innovation and collaboration with user communities;

EGEE-III Planning 5


INFSO-RI-508833

EGEE-III Goals

• The provision of high quality support for VOs – ranging from the mature ‘production’ VOs to the large number of new

VOs entering the Grid. – This would include the whole range of user support encompassing

training, access to documentation and information, applications support and day to day Grid user support;

• Make significant advances in the transfer of Grid technology to a wide range of business communities;

• Enhance the distribution of information – to a wide range of prospective and current users to aid the expansion

of the use of Grid technology world-wide; 2 aspects: marketing and technical information and support

• Ensure the overall world-wide coherence of the developments by ongoing work with other Grid projects, European and worldwide networking organizations, and world-wide standards organisations.

EGEE-III Planning 6


INFSO-RI-508833

Overall Principles

• Clusters of competence– Avoid thinly spread manpower; shown to be ineffective– Co-locate user, operations, and middleware support, training and dissemination

where appropriate to increase synergies– Invest in quality rather than full geographic coverage (quantity)– Minimum commitment of 50% for activity personnel

• More flexibility in tasks assignments by activity mgrs– Often issues arise that were not foreseen in the DoW– Need more flexibility to assign people to these issues; encourage volunteers– Partners should give more support to activity mgrs– Could we change the mgmt structure to better reflect the partner reality?

• Resource assignment processes need to involve resource owners– ROCs often can’t take appropriate decisions

• Need to agree on overall project policies from which decisions can be derived

EGEE-III Planning 7


INFSO-RI-508833

Operations

• Grid Management– OCC - CERN - overall coord of ROC & ops tasks– ROC management - All regions– Run services for production and pre-production - All regions– Accounting - UK+Spain– OAG (or better!) + regional resource coordination – 1320 PMs (5 FTE per region + 5 coord)

• Operations and support– Middleware deployment and support - All regions– Grid Operator on Duty - All regions (no exceptions this time)– Operations support - All regions– Regional cert --> could move into Integration & Cert

– 1560 PMs (6.5 FTE for each region - minimal)

EGEE-III Planning 8


INFSO-RI-508833

Operations Cont’d• Interoperation Coordination

– Interoperation with OSG, Naregi, etc, wLCG - Coord at CERN– Coordination with related infrastructure projects - Coord at CERN– Coordination with national grid projects - Regions with national grid efforts– 48 PMs

• Grid Security– EGEE security officer - CERN– Operational Security Coordination Team - All regions; coord CERN– Security Vulnerability group - All regions; coord UK– Joint Security Policy Group - All regions; coord UK– CA management and EUGridPMA - Regions with CA's; NL for GridPMA– Security Assessment - All regions– 288 PMs (1 FTE per region plus coordination)

• User support– GGUS coordination and GGUS tools - FZK? – 2 FTE– Support personnel for helpdesk (TPM, supporters) - All regions - no exceptions – 3 FTE– VO specific support discussed under applications– 768 PMs (3 FTE per region + 2 FTE coord and tools)

EGEE-III Planning 9


INFSO-RI-508833

Operations Cont’d

• Monitoring tools– SFT/SAM, Gstat, COD tools, dashboards, etc - Coord CERN– NPM etc – UK– 288 PMs

• Coordination w/other activities– TCG, JRA1, SA3, NA4 via various bodies/boards - All regions; but fractional

effort– 120 PMs

• Overhead tasks– Deliverables, reviews, etc. - All regions - 1FTE each– 240 PMs

• Activity management part of overhead and Grid mgmt task

• Total 4632 PMs (vs. 5475 PMs in EGEE-II)

EGEE-III Planning 10


INFSO-RI-508833

Application Support

• Effort within this activity must be clustered into teams. – This clustering should co-locate team members at the same

institute (or a small number of geographically-close institutes with a proven history of collaboration);

– the clusters should reflect existing expertise to maximise the benefits to the EGEE-III user community and to the EGEE-III project

– Aim at a reduction in the number of partners – People assigned to work on application support need to work on

application support.

• “community building” activity that will financially support travel and conference fees for non funding, yet important application people



INFSO-RI-508833

Application Support

• Task 1: Support– 3 teams to provide managerial support, technical consulting, and

operational support, respectively – 1.1: Virtual Organization support

VO mgrs group, following progress of VOs 72 PMs

– 1.2 Application porting support consult about EGEE services (and external services commonly used with

the EGEE infrastructure) 144 PMs

– 1.3: Direct user support Deal with application related issues in GGUS Provide application related documentation – link to UIG 192 PMs

– This task will get additional regional support through task 4.3.– Issue: policies need to be defined on how the effort of these “generic”

teams is allocated to user communities– Total 408 PMs



INFSO-RI-508833

Application Support

• Task 2: Community Building– 2.1: Overall Community Building Coordination (24 PMs)– 2.2: Bringing Communities Together

2 Users’ Forum events per year (one with EGEE conference) = 4 total

1 General NA4 meeting per year = 2 total 2 Topical meetings per theme x 8 themes = 16 total

– 2.3: Coordinating Community Effort 1 discipline meeting per year x 10 disciplines = 20 total 1 external scientific meeting per year x 5 disciplines = 10 total 48 PMs (split over disciplines) for coordination + significant travel

money (O(100K))

– Issue: policies for spending the money need to be defined– Total 72 PMs + O(100k) travel money



INFSO-RI-508833

Application Support

• Task 3: Application Specific Support and Advanced Application Services– teams coming from specific scientific disciplines to provide advanced services to

their community HEP, life science, comp chemistry, grid observatory (see later)

• fusion and earth science will put in related projects! • Need ~1FTE earmarked for coordination with EGEE

– Task: Provide application specific support – link to GGUS and UIG Enable applications to interface with EGEE infrastructure Provide application specific services that interface with gLite Drive middleware evolution

– Issue: how to enable communication and tool sharing among these communities? What level of overall integration should be done?

– Total Effort for task assuming that four teams are supported. Division within subtasks subject to discussions between the users and the project and

should not be specifically defined before the start of the project. Each team should have ~8 FTEs

– 792 PMs (assuming 4 teams + 1 FTE coordination with RPs)



INFSO-RI-508833

Application Support

• Task 4: Coordination– 4.1: Overall activity coordination

48 PMs: Activity manager and deputy 24 PMs: Administrative assistant

– 4.2: Activity communication 24 PMs: Web maintenance, dissemination of information within activity

– 4.3: Regional coordination (assuming 7 regions): 3 FTEs per region; approximately 0.5 FTE should be spent on purely

administrative concerns; the rest should contribute to Tasks 1 and 2); Depending on overlap with Operations user support task this could go down

to ~1 FTE minimum Assume 1.5 FTEs for now: 252 PMs

– Total: 348 PMs

• Total: 1620 PMs + Travel money (O(100k)); was 1861in EGEE-II



INFSO-RI-508833

Middleware

• Two main tasks– Middleware engineering

maintain and gradually improve the reliability, performance, and usability of the existing services ¾ of overall effort

– Middleware research and standardization looking into possible future developments of services, standards, interfaces, and concepts. produce demonstration prototypes that would be exposed to users for feedback operate a preview testbed ¼ of overall effort

• Performed by two vertical teams:– Security, CE, Job Management, L&B, JP

INFN, CESNET, Northern Federation ~850 pm

– Data management, Information System, Monitoring and Accounting CERN, CCLRC ~550 pm

– Same partner responsible for certification of their components in the integration and cert. tasks

• Total 1400 PMs (was 1272 in EGEE-II)– Data mgmt increased wrt EGEE-II



INFSO-RI-508833

Integration, Certification and Testing• Integration, configuration and testing

– validates software and maintains the distribution– co-located with a strong ETICS team– Infrastructure (build system, test infrastructure etc):

252PMs with ETICS, 360PMs without ETICS (out of which 144PMs external to core team for testbed)

– Integrating new components, upgrades, patches, security upgrades, writing tests: 546PMs with ETICS, 558PMs without ETICS needs small teams where different teams can be hosted at different locations, but the team itself

needs to be co-located. Assuming low workload: 4 new components, 1 platform change, 210 patches including security per

year• Issues:

– focus on components originating from the service task (see middleware slide) only? – How to deal with components developed by 3rd parties or even inside the project like by

applications?– Numbers need to be scaled based on numbers of components!

• Production components maintenance and improvements– This is the engineering task under middleware (see slide before)– Issue: must cover all of the current gLite foundation stack!



INFSO-RI-508833

Integration, Certification and Testing II

• Interoperability– 54 PMs (assuming 72 PMs are put in from peer system to complement

the effort) – Issue: should we continue to fund partners from peer software stack or

reduce to coordination• Porting

– collect, test and distribute ports for multiple platforms– Integrated with build system– 90 PMs (can be external to core team)

• Localisation – links with sites to interface them to Grid services– similar to the porting activity– Difficult to estimate efforts; mostly one-off efforts jointly with sites– No effort earmarked for that at the moment

• Activity management – included in tasks• Overall effort:

– 942 PMs (with ETICS); 1062 PMs (without ETICS); (was 725 in EGEE-II)



INFSO-RI-508833

Training• Clusters of competence

– not smaller than 3FTEs in the regions (O(27)FTEs), – core team (O(6) FTESs) @ Edinburgh, – complemented by “outposts” (O(10)FTEs)

• Course content creation and delivery– Induction, Appl.dev., train the trainers, site admin

Depends on input from other activities– 355 PMs

• Training course scheduling and organisation– 89 PMs– Issue: Should that be funded or we expect host institutions to deal with

that? • eLearning development and maintenance

– Repository, course module creation, course administration, eModeration– 115 PMs

• Translation – of training material– of eLearning course modules– 26 PMs



INFSO-RI-508833

Training Cont’d

• Activity management and partner coordination, general project tasks– 1 coordinator plus fractions of people in the regions– 142 PMs

• User Information Group– content generation and maintenance– Issue: this is a cross-activity task– 72 PMs (3 FTEs integrated across the project) – 24 PMs for

coordination• t-infrastructure

– development and maintenance – see t-infrastructure slide!

• Issues– Scope of training: wider than core EGEE technologies? Overlap

with ICEAGE

• Total 799 PMs (vs. 738 PMs in EGEE-II)



INFSO-RI-508833

Training and Inception Infrastructure

• Goal is to operate a reliable t-Infrastructure that is also used to expose new communities to Grids. The migration of these communities to the production infrastructure will be actively managed

• This objective will be pursued with the creation of a small number (3 or 4) of “clusters of competence” closely following the ROC model, or even co-located with ROCS. – Each “cluster of competence” will have a small team of people with

substantial experience in managing existing e-Infrastructures.



INFSO-RI-508833

Training and Inception Infrastructure

• Task 1: Infrastructure operations and reliability – Core team: coordination, core services mgmt and support, security

144PMs– Centers of excellence: regional operations

144PMs– Total: 288 PMs

• Task 2: Dissemination Support and Training Activities– relationship between organizers of dissemination events and t-Infrastructure operators – 48 PMs

• Task 3: New communities and new application support– Support new communities in getting started, actively manage the move towards the production

infrastructure– 48 PMs

• Task 4: Activity Mgmt– 48 PMs

• Issues: – Support for business applications– Relations with ROCs, need follow standard operational procedures

• Total: 432 PMs



INFSO-RI-508833

User Support

• User support is touched in different flavors in the discussion before, including– Applications support:

Virtual Organization Support, Application Porting Support, Direct User Support;

– Operational support;– Availability of a wide range of validated, reviewed information

presented via simple user portals;– Training.

• All this needs to be coordinated project wide across activities

• Need a “champion”



INFSO-RI-508833

Dissemination

• Overall vision:– Major reduction in regional effort, concentration of effort at

minimum number of sites– Business and Industry in a new activity and related project– In all areas specified below, about half the manpower should

come from people experienced in that specific area (in EGEE-II most NA2 people had no communication training)

– Relatively large non-personnel budgets requested, around 400,000 Euro, equivalent to 48PMs (2 FTEs), to avoid need for common fund in EGEE-III

– All partner institutes are obliged by Consortium Agreement to get their press offices to work with NA2, provide contacts and distribute press releases etc, lessening need for extensive local network



INFSO-RI-508833

Dissemination-II

• Task 1: Management, Admin & Coordination– General oversight, paperwork, administration, reporting, metrics– General mgmt should be close to PO; coordination with local press

offices/partners in the regions– 120 (general management + 3 FTEs coordination)

• Task 2: Web– Public website, technical pages oversight, sites for other groups (UIG etc)– One person with technical skills, one with more design skills– 24 PMs @ one site assuming public website can be outsorced

• Task 3: Materials and Publications– Including info sheets, flyers, brochures, newsletters, iSGTW, papers, printing &

distribution– 72PMs @ one site (should be the same as task 2)

• Task 4: Media and Public relations– Forging relationships with publications and journalists, ensuring coverage of

EGEE events and activities. Ensuring EGEE can explain itself to very general audiences.

– 48PMs @ one site



INFSO-RI-508833

Dissemination-II

• Task 5: Marketing to Scientists, Users (& Industry?)– Attending user and scientific events, visiting these groups on their own ground to find

their needs and expectations. Using full commercial marketing techniques to achieve this.

– 48PMs @ one site (could be the same as task 4)

• Task 6: Events (should be combined with Task 5) – Assuming that the EGEE main conference is outsourced. Staff to organise User Forum

and liase with conference organisation. – 24PMs @ one site

• Task 7: Regional effort– Making local contacts, tailoring content to local area, some very limited translation of

key material e.g. press releases.– Effort including in Task 1 but we may need some funding for translations and potential

interns.

• Total: 336 PMs (was 532 PMs in EGEE-II)– 400000 Euros materials budget– Translation and intern money

• Issue: we have now NOT foreseen any effort for maintaining the local websites. Maybe that could be carried out by the 3 people in management with outside translations.



INFSO-RI-508833

Networking

• Work focus on ENOC, relations to Geant/NRENs, and standardization/interoperability (IPv6, …)

• Task 1: ENOC– daily work of the ENOC: user support through GGUS, interface

with Géant2/NRENs NOC, support for the LHC-OPN – 48 PMs at a single partner

• Task 2: Management, standards/interop, internat. Relations, TNLC, operational tools– 48 PMs– This could involve the NRENs adding on the order of 48PMs

• Overall effort: ~144 PMs (160 PM in EGEE-II)



INFSO-RI-508833

Quality Assurance

• Overall Quality Assurance Coordination – Initiate Quality Plans and Measurement Plans– Coordinate the Quality Group– Deliverables and Milestones monitoring– Follow and gather Metrics (Overall; Activity; Partners metrics)– The partner metrics program should be completed with federation

reviews based on the experience gained within SA1. This review could be organized for one main activity or of all activities carried out in a given region. The QA coordinator will help in organising such as review

– Promote industry-standard tools, such as ITIL, helping to start implementation in some areas, for example the ENOC

– Maintain QA website

• Issue: coordination only, activities have to allocate effort for their internal QA

• 24 PMs (was 72 PMs in EGEE-II)– not counting the security coordination that was in JRA2)



INFSO-RI-508833

International Relations and Concertation

• Work focus on policy issues, relationships with other projects, cooperation with other geographical areas

• We should foresee travel money for EGEE personnel to attend related events; effort from the other groups need to be allocated for the technical work

• Task 1: Policy support– Planning and editorial support for the e-IRG publications, assistance to the organisation and running of e-

IRG events (meetings, workshops), travel money– 48 PMs (GRNET, CERN, JKU)

• Task 4: Keeping track of standardisation efforts– Provision of travel money– 12 PMs (CCLRC)

• Task 2: Concertation activities with other projects and activities– Related projects office etc. – 48 PMs (GRNET, CERN)

• Task 3: Cooperation with other geographical areas on policy level– US, AP, etc. – 24 PMs (GARR)

• Task 4: General Project Tasks– 6 PMs

• Total 138 PMs (vs 138 PMs in EGEE-II)– But with much less partners



INFSO-RI-508833

Business

• Main goals:

– Foster relations between industry and EGEE based on targeted approach to business

– Set the way for commercial exploitation of EGEE technology– Create self-sustainable EGEE Grid\e-Infrastructure Competence

Centres “for Industry” in order to provide technical support to the business community



INFSO-RI-508833

Business tasks and effort (first assessment)

• T1: Activity Management - (1,5 FTE)– Activity management; Coordination with related projects; Project Office

• T2: EGEE commercial exploitation - (1,5 FTE)– Set the way for innovative solutions & policies & funding schemes & collaboration between industry and

Research in commercial Grid adoption– Business model; Accounting/billing; Commercial software license policy; etc?– Preparing future for commercial exploitation of EGEE

• T3: Business outreach - (3FTE)– Organisation of events; Mailing list; Phone marketing; Production of material; Website; Customers database

follow-up. Etc– Team centralized in one place. Organizing events worldwide

• T4: Business uptake & follow-up (Business Line Manager: 0,5 * 5 regions = 2,5 FTE) – Responsible for the strategy for a segment(s) of market for all the region; Responsible for a region(s): on the field

for all sectors in the region(s). Physically going to workplaces (CIO, Federation, …). Recruit and follow Business applications; Gather requirements. Establish a network of multipliers.

• T5: User support - (0,5 FTE)– User support (Business VO manager = 0,5 FTE)– Help desk (GGUS-SA1); Training (NA3); Gilda (NA6)– Dedicated consultancy, training, support for middleware etc. could be proposed by EGEE industrial partners as a

payment model (fixed price offer to get businesses started etc.)• T6: Business requirements and studies on technical issues - (1,5 FTE)

– Provide a “second line” of technical support for EGEE Industry Users for problems not solved by “Business Line Managers” or Business VO manager

– Studies on technical issues: Security policy; SLA; Data Management; Standardisation, etc.• T7: Prototyping I-services - (1,5FTE)

– Implementing prototype services which will not be a top priority for EGEE-III without interfering with gLite mainstream (Tesbed & development)

• TOTAL = 12 FTE = 288 PM – without Applications funding– probably done in a RP



INFSO-RI-508833

Business Issues

• A business activity will need some pilot business applications from the beginning of the project– Not yet identified– The possibility to make them funded should be analysed

• Large non-manpower budget should be foreseen for the necessary material, events, travels, mailing lists, dedicated business studies, etc.– Depending of the FP7 funding schemes, the possibility to fund the

business activity or part of the activity at 100% should be analysed.

• Even if for funding reasons, the Business activity will be a separate related project, it should be considered as EGEE-III activity– with the same synergy as other EGEE-III activities (PEB, AAM, relying

on EGEE partners, etc)– Probably 1 FTE as link person from EGEE-III– Level of support EGEE-III will give to the project needs to be defined



INFSO-RI-508833

Grid Observatory

• The observatory integrates data collection and monitoring tools (bottom-up approach) with the development of an ontology for the domain knowledge (top-down approach).

• Task 1: Data collection and publication– The overall goals of this task are to gather reliable traces of complex

applications and very large grid behaviour, to index these data to facilitate research, and to and to furnish these data to the scientific public. This is achieved via

Data repository ontology and publication service Network-level monitoring

• Task 2: Models and evaluations– Based on the grid observatory, made of the high level representation

(ontology) and the low-level data describing the grid activity, the objectives of this task are to model the behaviour and usage of the grid. The enabling technologies are statistical machine learning and data mining on one hand, and signal processing and control on the other hand.



INFSO-RI-508833

Grid Observatory Cont’d• Task 3: Autonomic dependability

– While fault-tolerance must be embedded into the middleware and the applications, on-line failure detection should be an independent component relying on its own set of discovery techniques. More ambitious systems would include on-line explanation of these failures, and in some cases being able to anticipate them by on-line discovery of hazardous situations that are likely to drive to failures.

• Task 4: Coordination and dissemination

• Issues: – Data collection is an important goal for EGEE, not sure whether the other

tasks should be in EGEE or a related project Data collection done in other tasks as well and this needs to be integrated

– Task 2 and 3 could provide a valuable link to research activities outside EGEE

• Could be done as separate activity (~264 PMs) or embed task 2 and 3 in an application cluster (~8 FTE) plus 1 FTE in middleware/operations as additional support for task 1. The latter is assumed in the planning



INFSO-RI-508833

Project Management

• Project and Technical Director– 48 PMs @ CERN

• DPD and DTD – 48 PMs @ CERN

• Security Coordination– 24 PMs (was JRA2) @ KTH

• Project Office– Project Secretary, Director PA, Finance Officer, user registration

account service, PPT & indico support– 120 PMs @ CERN

• Total 240 PMs (was 168 PMs in EGEE-II)



INFSO-RI-508833

Budget Constraints

• EU contribution capped at 35 Million max• Assume a 5% increase in personnel cost (inflation,

salary rises)• New cost models allow to charge up to 75 % to EU• PO did a very rough simulation:

– Current EGEE-II manpower according to DoW: 11.165 PMs– Current EGEE-III plans according to PEB: 10.731 PMs

– Effort simulation assuming 50% EU funding: 9.461 PMs– Effort simulation assuming 75% EU funding: 6.308 PMs



INFSO-RI-508833

Overall Effort Estimation

• Total: 10.731 PMs (PEB request) vs 11165 PMs in EGEE-II (DoW)• Aim between 6.300 and 9.400 PMs (according to very rough EGEE-III

budget simulation)

EGEE-III Effort Distribution

Operations44%

Middleware13%

Integration9%

Applications16%

Mgmt2%

Observatory0%

Business0%

tInfrastructure4%

Training7%

Dissemination3%

Internat.Relations

1%

Networking1%

QA0%

EGEE-II Effort Distribution

NA12%

NA25% NA3

7%

NA417%

NA51%

SA149%

SA21%

SA36%

JRA111%

JRA21%

infso-ri-031688 enabling grids for e-science egee-iii input from the peb erwin laure egee technical...

Documents