TF-NOC flash presentation DFN

TF-NOC Feb 2012 Dubrovnik

Thomas Schmid, schmid@dfn.de

DFN (1) • Head office in Berlin

– Management, Project Management, Planning, Ordering, Services (Lambda Service, Roaming, PKI, Federations, LIR, IP ressources, VoIP, secondary DNS, netnews, PoP facility management, …) …

– ca. 35 people

• Branch Office in Stuttgart: 2 Teams – NOC/Ops/Engineering/PERT (5 FTE, 2 PTE)

– IP, Mcast, VPNs, MPLS, Troubleshooting, Consulting, Testing, Planning, Projects, Migraeions, staging, …

– Normal office hours, 24x7 for escalation. 1 of 4 people on-call duty.

– Service Operation (VC, Mail, Spam-marking, Web, DSL, Serverhosting, PKI, AAI, Federations, Domain, …) (8 FTE)

DFN (2)

• DFN CERT GmbH – Hamburg, 30+ FTE/PTE

• WiN-Labor Erlangen – Statistics, Accounting, Hades, Perfsonar, Quality control, Geant

Projects ... (ca. 8 FTE/PTE, DFN, Univ. of Erlangen)

• Ixus GmbH – Spin-off in Berlin, 2 FTE

• CNM-Team – Located at LRZ Munich, working also for Geant Projects, 2 FTE.

IP Topology mix of 10G and parallel 2 x 10G links Routerplatform Cisco

4 x CRS-1 55 x 7609 major topology change this year tender for 100G routers ongoing

managed by DFN local loops to member institutes ordered and managed by DFN

ca. 770 institutes redundant connection to 2 PoPs

in sum 1000+ links PoPs at universities (except Frankfurt) Serving Universities, Research Institutes,

polytechnical Universities, government agencies, commercial research labs, medical institutes, … primary/secondary schools via DSL and webhosting (few)

DWDM Huawei BWS 1600G system

static lambdas, lead time 8 weeks for new lambdas managed by Huawei NOC has access to T2000 management client

tender ongoing Equipment will be replaced until end of year ROADM + OTN switching? new platform: for 18 months managed by supplier. Depending on experiences then perhaps managed in-house?

Fiber

country-wide leased fiber managed by fiber providers leased lambdas in few places

Service Desk Outsourced to Dimension Data

24x7x365 1st level

active monitoring of routers, core and access links

core: monitor interface status access: monitor BGP session to member or active link monitoring with ping

coordinates troubleshooting in case of link failures between e.g. Huawei/fiber carriers or local loop suppliers Coordinates emergency hardware replacements dedicated users can open tickets communication by phone, mail, web-interface escalation 24x7 to NOC proprietary ticket system Dimension Data also takes care of PoP

maintenance and is our Cisco contractor.

Tools: CNM

Customer view on network weathermap individual usage statistics developed at LRZ Munich and part of PerfSonar project

Tools: PoP hardware documentation database

home-made racks cabling power climate routers switches patches room plans PoP access ISDN/phone numbers … Used by NOC and Service Desk

Tools: GIS The most important Database/Tool in DFN! Topology database customer/user database IP addressing database (core, users) hardware database services database (DSL, VC, VoIP, …) statistics database billing contracts ressource-service mapping ordering procedures maintenance management process tracking data export to other tools/projects …

Developed and maintained by Ixus GmbH, spin-off from former DFN interns

Tools: HPOV

Mainly used during business hours. Probably replaced soon by e.g. Icinga.

Tools: MRTG frontend

home-made basic monitoring and alarming function integrated Top-N List generation for numerous parameters

Tools: Accounting

Traffic Matrix and user Accounting developed by Win-Labor Erlangen Operated by NOC used for planning

Tool: Mail

instead of internal ticketing system home-made

Tools, misc

• Tons of scripts that have been developed over the last 17 years – Yes, we still use e.g. one monitoring script that

was written in 1993! – Monitoring, network planning, logging, router

management, config management, alarming, process tracking, ...

– pre-nagios/cacti/RT/… origin

– Nagios, OTRS, … for services

Inter-NOC communication

• Internal communication via VC, phone, mail, Jabber – Weekly VCs with Berlin – periodic phone conferences/VCs for ongoing

projects – 2-day meetings outside Berlin for special topics

• NOC-NOC communication via phone, mail(lists), IRC (commercial ISPs)

Misc

• no written SLAs with customers • additional documentation in internal Wiki

• MediaWiki, will be replaced by FosWiki • better document integration

• “Betriebshandbuch” • documents operational procedures with

Dimension Data, Carriers