enabling grids for e-science cod 19 meeting, bologna nordic rod experiences michaela lechner cod-19,...

Enabling Grids for E-sciencE

COD 19 meeting, Bologna

Nordic ROD experiences

Michaela Lechner

COD-19, Bologna


COD 19 meeting, Bologna 2/10

Overview

• NE ROC split into two regions (Federations)– BeNeLux (Belgium and the Netherlands) and Nordics (Denmark,

Finland, Norway, Sweden, and the Baltic states[MoU]).

• Nordic duty shifts– 6 teams, 3 from SNIC and 3 from NDGF.– http://www.egee-ne.org/internal/opdocs/NORDICroster2009 – 8 hours a day, 5 days a week coverage. Normal working time.– one team (usually one person) on duty for a week.– NDGF-T1 and EGEE operations overlap and are combined: ROD and Operator on Duty (OoD) at the same time!



ROD on-duty tasks

• On-duty tasks (for BeNeLux and Nordic ROD)– Perform all 1st line support and ROD tasks as defined in the 1st

line support and ROD operations manual:https://twiki.cern.ch/twiki/bin/view/EGEE/OperationalProceduresforRegionalCODAndInteractionsWithCCOD

– Watch other monitoring tools for T1 (Nagios, ganglia). (=OoD)– Use ROD dashboard tools– Assist sites where appropriate.

• Weekly duties (for Nordic OoD)– Handover– Fill out NDGF-T1 production report– Be present at WLCG weekly operation meeting next Mon 16:00– Prepare notes to weekly meeting on Friday 10:00



Nordic ROD communication

• Communication– All on-duty support via chat(s) and mailing list(s).– Mailing list for all NE ROC operators. The ROC mailing list:

[email protected] is archived under https://www.hpc2n.umu.se/mailman/private/roc/

– 1st line support over [email protected] – Shift handover via [email protected] mailing list.– XMPP (Jabber) chat: [email protected] – Contacts for sites and site admins are available on:

https://portal.nordu.net/display/ndgfwiki/Operation-Procedures – Communication through C-COD (Vera, Luuk) if necessary.– #cic-on-duty IRC chat @testbed011.cnaf.infn.it – Regular weekly status meetings, one each for EGEE (Thursday)

and NDGF (Friday) operations.



Nordic ROD knowledge sharing

• Short term– Mailing lists– Handover procedure– NDGF Chat– “None of the duties should be done acting alone”

• Longer term– Dashboard tools and notes therein. Notes: time and determination– GGUS: solutions are public and available. – NDGF wiki for OoD work– 1st line support over [email protected] (archived)



Daily Duties: Preliminaries

Global Grid User Support (GGUS) Overview:https://gus.fzk.de/ws/ticket_search.php?&supportunit=ROC_North&status=open&timeframe=no

CIC Portal:https://cic.gridops.org/index.php?section=roc&page=dashboard 1. You have to have a regional role in the GOCDB 2. Don't forget to set your preferences to one of these

regional roles, not site Administrator.

Log into the COD IRC chat

Change your nick in the NDGF chat eg. /nick caela(ROD, OoD)



Duties in detail: The Dashboard

• Home tab– overview fresh alarms, open tickets

• Ticket tab– React on colored tickets

• Dashboard tab– Actual work is done here– Set off alarms, when problems solved– read/write notes– Creating tickets or extract mail adresses used for 1st line support– Write informal mails

• Experience:– Nice with glide over and links, but slow in many ways– Not all tickets shown in ticket tab?– Creating tickets easier, administration fine



Bugs



When it burns (ASGC-DC)



Going for a cup of coffee



Nordic helpdesk solutions

• Regional helpdesk– BeNeLux have own ticketing system, but nothing similar

implemented for our Nordic sites, besides [email protected]

• Avoiding to talk with yourself– Being 1st line support, ROD and helpdesk all at the same time?– Too much responsibility on one person. – Furthermore one persons knowledge is not infinite: waiting one

week for new ideas?– Solution: Drawing more knowledge in (NDGF chat, mailinglists)



Nordic ROD workload

• Takes 2-3 days to get friendly with system– Hard to get an overview of new tools, new sites (Baltics)– Doing many things simultaneously– Too little time dedicated to real problem solving– Dashboard hideously slow (a feature, not a bug!)• Should definitely not be done at the same time as

TPM!



Conclusion

• Found our own good practice procedures :-)• Dashboard

– On the way to become a great administrative tool, but beware not to forget real support

– Preferred sites option is of great help

enabling grids for e-science cod 19 meeting, bologna nordic rod experiences michaela lechner cod-19,...

Documents