addressing a globally distributed network the future of testing at akamai polish network operators...
TRANSCRIPT
Addressing a Globally Distributed NetworkThe Future of Testing at Akamai
Polish Network Operators Group 2013
Thomas Higdon – Akamai Technologies
©2013 AKAMAI | FASTER FORWARDTM
A little about me
• Software engineer at Akamai in the platform infrastructure team
• We develop the software that goes on every Akamai server out there (well over 100,000).
©2013 AKAMAI | FASTER FORWARDTM
Some statistics
Accelerating Daily Traffic of:• 10+ Tbps • 25+ million hits per second• 1.5+ trillion deliveries/day• 50+ petabytes/day• 10+ million concurrent streams
15-30%+ of Web Traffic
Delivering 250,000+ Domains• 5,000+ Customers • 9 of the top 10 Banks
• 9 of the top 10 Social
Media Sites• All top 30 M&E Companies
• All top 60 eCommerce Sites
A Global Platform:• 132,000+ Servers • 86 Countries
• 650+ Cities
• 1,150+ Networks
• 2,200+ Locations
©2013 AKAMAI | FASTER FORWARDTM
A little about me
• I’m addressing the problem of how to get new platform software out there faster, and with a smaller likelihood of disruption
• We’re using…
©2013 AKAMAI | FASTER FORWARDTM
Akamai in a nutshell
• Akamai is a globally-distributed cloud of physical servers
• Machines at Akamai are grouped into functional units called networks.
• Edge networks - CDN services, object/file storage, analytics, etc.
• Infrastructure – distributed file transfer, messaging, monitoring, reporting, etc.
©2013 AKAMAI | FASTER FORWARDTM
Akamai in a nutshell
• Each network runs a different set of software that depends on the role of the network in the ecosystem.
• Each also runs a common set of platform software• Generally OS software and Akamai-specific
File transferMessaging Monitoring
Network-specific software
OS
©2013 AKAMAI | FASTER FORWARDTM
Akamai in a nutshell
Infra network File transferMonitoringMessaging
Operating system/kernel
Edge network
Network-specific software
©2013 AKAMAI | FASTER FORWARDTM
Akamai in a nutshell
Infra network File transferMonitoringMessaging
Operating system/kernel
Edge network
Network-specific software
File transferMonitoringMessaging
Operating system/kernel
Network-specific softwareNetwork-specific software
Network-specific software
File transferMonitoringMessaging
Operating system/kernelFile transferMonitoringMessaging
Operating system/kernelFile transferMonitoringMessaging
Operating system/kernel
©2013 AKAMAI | FASTER FORWARDTM
Akamai in a nutshell
Infra network File transferMonitoringMessaging
Operating system/kernel
Edge network
Network-specific software
Edge network
Network-specific software
Edge network
Network-specific software
©2013 AKAMAI | FASTER FORWARDTM
Akamai in a nutshell
• State of a network• Installed software and configuration• Dynamic configuration/metadata• Persistent (across installation) filesystem state• Applied manual changes
• Each network tends to have built-up state necessary for proper functionality
• not well-documented or defined
©2013 AKAMAI | FASTER FORWARDTM
An Akamai “instance”
• A given unique set of Akamai networks that function together
• Self-contained with limited external dependencies
©2013 AKAMAI | FASTER FORWARDTM
Akamai “instances”
Production – 105 SQA – 102
SQA – 102
SQA – 102
Dev – 102
Dev – 102
Dev – 102
©2013 AKAMAI | FASTER FORWARDTM
Commoditizing an Akamai instance
• Instances are labor-intensive to manage
• We’d like the ability to create these as a commodity
• Automatically bootstrapping from scratch is possible, but difficult.
©2013 AKAMAI | FASTER FORWARDTM
Commoditizing an Akamai instance
• Solution: virtualization• Represent the state of an
instance using virtual machines.
• Save the virtual machines and clone them. SQA – 102 SQA copy– 102
SQA – 102
©2013 AKAMAI | FASTER FORWARDTM
Cloning Akamai instances
• Static IP configuration• Isolation• traditional NAT • “reverse” NAT• SOCKS proxy• Akamai “authgate”
©2013 AKAMAI | FASTER FORWARDTM
Accessing an instance
NATSOCKSauthgate
192.168.0.0/16
192.168.0.1
NATSOCKSauthgate
192.168.0.0/16
192.168.0.1Client
172.26.238.10
172.26.238.20
©2013 AKAMAI | FASTER FORWARDTM
Managing the Akamai side
• Private cloud infrastructure is great for managing VMs and their resources
• We needed another level of management• “Akamai” networks• Akamai instances• Expose specific Akamai machine types and services
©2013 AKAMAI | FASTER FORWARDTM
An Akamai instance service
• Service-oriented• Language/test harness agnostic• Share common resources
©2013 AKAMAI | FASTER FORWARDTM
An Akamai instance service
Instance service
Private Cloud Orchestrator
2. “Clone these machines”
3. Creates resources
4. “Here’s your instance!”
5. “Ok, let’s use it.”
Gateway
User1. “Give me an instance”
©2013 AKAMAI | FASTER FORWARDTM
A well-managed instance
• Leverage expertise from around the company to create an internal “master” instance
• Each additional network gets it closer to a “real” instance.
• Forking?
©2013 AKAMAI | FASTER FORWARDTM
Addressing existing instances
• Can also address existing instances of physical machines
• Production• Lab instances
Akamai ProductionInstance
Lab test instance
Lab test instance
Lab test instance
User
Instance service
©2013 AKAMAI | FASTER FORWARDTM
Unify these stages
Dev QA
SQA
Checklisting
• Use the same code to write tests by using a common interface to refer to an Akamai instance.
©2013 AKAMAI | FASTER FORWARDTM
Unify these stages
Dev QA
SQA
Checklisting
• Use the same code to write tests by using a common interface to refer to an Akamai instance.
©2013 AKAMAI | FASTER FORWARDTM
Takeaways
• We’re virtualizing Akamai and providing this as an internal service
• We’re simplifying our testing using the power of the cloud.