plnog 13: b. van der sloot, s. abdel-hafez: running a 2 tbps global ip network using open source...
DESCRIPTION
Bart van der Sloot joined FiberRing as Managing Director in April 2014, with the objectives to further grow and enhance the footprint, quality and business of FiberRing’s 2 Tbps global IP network, covering over 50 locations on 3 continents. From 1999 to 2013 Bart worked at Global Crossing (acquired by Level 3 in 2011), where he developed staffing, systems and business processes for Global Crossing’s European brand new sales team, built and coached a Wholesale Sales team to sign new telecom customers and grow revenues in various countries across Europe, led Global Crossing’s expansion into Central and Eastern Europe and established Level 3’s position in the Benelux broadcast market. Samer Abdel-Hafez joined the FiberRing network team in December 2013 as Network Design Engineer. Samer’s responsibilities within the team include planning capacity for the large traffic volume of FiberRing, arranging interconnections in new locations and markets, designing advanced ad-hoc solutions for the FiberRing network and customers and advise the Network support team on day to day issues. Abstract: FiberRing operates one of the largest content networks in the world, peaking at over 2 Tb/s. In order to facilitate troubleshooting, detect attacks and saving important data as router configurations, we implement a series of tools mostly implemented in house or open source. The key point of this presentation is to describe how FiberRing is using these tools for: monitoring: FiberRing makes extensive use of Opsview (Nagios) and NMIS. We utilise Opsview for alerts and reporting and NMIS for detailed traffic analysis. capacity planning: FiberRing choose PMACCT as netflow collector software and implemented an in house front-end solution that helps us locate strategic peering partners and explore ways to reduce the costs to deliver our content. DDOS attacks detection: As every large hosting provider, we are regularly target of DDoS attacks. We implement a set of linux boxes running running nfcapd to collect traffic flows with 1 minute/per host granularity. This gives us great flexibility and incredibly valuable data to quick detect attacks and take corrective actions. routers’ configuration backups: FiberRing is actively involved in the development of Oxidized, an innovative configuration backup tool which poses itself as rancid replacement.TRANSCRIPT
b
Running a 2 Tbps global IP network
using Open Source tools
Bart van der Sloot (Managing Director)
Samer Abdel-Hafez (Network Design Engineer)
Agenda
1. Introduction to FiberRing and AS16265:
a 2Tbps Global IP Network
2. Network management and operations challenges
3. Common Requirements: why Open Source?
4. Network Monitoring
5. Capacity Planning
6. Attack Detection
7. Device back-ups
8. Considerations for the future
9. Conclusions
1. Ocom: 1 group, 4 operating companies
Infrastructure
as a Service (IaaS)
Bare Metal Servers
CDN
Cloud
Colocation
Data Centers
Amsterdam
US East Coast
Data Center
Construction
Energy Efficient
Modular
Network Services
IP Transit
Internet Access
Longhaul MPLS
Remote IX
Amsterdam metro
services (dark fiber,
waves, MPLS)
Locations
Amsterdam, Frankfurt
US East Coast, US West Coast
Singapore
More to follow
Team
> 350 People
> 60 Software Developers
>20 Nationalities
1. FiberRing Global Network
1. FiberRing Global Network
Total traffic: 2.5Tbps (>1Tbps growth in last 12 months, >4 Tbps total capacity)
• AS16265 – LeaseWeb Global
Backbone
• AS38930 – FiberRing
• Various customer networks
Traffic distribution
> 500 peers
> 30 private peers
> 30 Internet Exchanges
> 10 Transit providers
3 100GE ports (Ams-IX)
> 480 10GE ports peering/Transit
> 120 10G waves in backbone
….a challenge to manage!
2. Challenges to address
• Network monitoring
• Capacity planning
• Attack detection
• Network devices configurations backup
3. Common requirements
1. Easy to use and maintain
2. Scalability up to 1000’s “objects”
3. Trivial integration with internal systems
4. Long life span expectation
5. Easy access to updates
6. Extensive documentation provided
3. Why open source tools
• Increased control over development
• Largely tested and documented
• Cost reduction
• Easy to extend to support own code
4. Network monitoring
• OpsView (nagios)
• NMIS
• Custom RRDs
• Custom code for specific monitoring
(e.g. BGP state)
5. Capacity planning
• PMACCT combined with UI developed
in-house
• Focus on traffic trends per destination AS
• Immediate results are measured by
monthly cost reduction
• Long term results are measured in
customer response
• Overall quality has greatly improved
6. Attack detection
• Nfdump
• Small dumps covering 60 seconds spans
• Immediate view of the attack sources and
destinations
• NOC response is almost instantaneous
• Very little training needed
• Huge reduction in sourced attacks
• Customer experience improved
• Requires resources & expertise to maintain
7. Network devices config backup
• Oxidized and GitLab
• Oxidized is a RANCID replacement
• Design team contributes to Oxidized
development
• Oxidized is easy to integrate with internal
systems
• Oxidized uses GitLab to store configuration
updates
• GitLab provides extensive overview over HTTP
and ability to store internal repository for free
8. Future considerations
• Replace RRDs with time series database
(influxdb)
• Extend oxidized-script tools to our needs
• Logs visualization tools
• Hire developers to enhance the existing
tools
• Early discussions on the development of
a new NMS
• Automation
8. Conclusions
• We can run a global IP Network with ~ 15
people in Network Design and Support • Including network expansions, new PoPs, peering & transit
negotiations
• Including managing internal networks of our customers, e.g.
LeaseWeb
• There are good Open Source tools available to support
this, but they require staff that understand both
“networking”, and “programming”
If you want to hop onto our network and try our services: come to our booth!
- Special offer for Remote Ams-IX connectivity –
Interested in joining a global fast growing company?
We are looking for Design & Support Engineers in Amsterdam….
b
Questions?
Bart van der Sloot [email protected]
Samer Abdel-Hafez [email protected]
Colin Boekhout [email protected]