building cloud tools for netflix
DESCRIPTION
Using the Amazon cloud requires a lot of moving parts like AMIs, ASGs, and ELBs. See how a small Netflix team developed web-based tools to abstract and clarify these cloudy components for use by hundreds of engineers. Presented at "Talk Cloudy to Me II" hosted by the Silicon Valley Cloud Computing Group in 2011. Apple Keynote version with animations is on Google Docs at http://bit.ly/netflixcloudtoolsTRANSCRIPT
Building Cloud Tools for Netflix
@joesondow #netflixcloudSunday, September 25, 2011
Note takers, take notehttp://www.slideshare.net/joesondow
@joesondow #netflixcloudSunday, September 25, 2011
Agenda
@joesondow #netflixcloudSunday, September 25, 2011
Agenda
Who am I?
@joesondow #netflixcloudSunday, September 25, 2011
Agenda
Who am I?
Why am I here?
@joesondow #netflixcloudSunday, September 25, 2011
Agenda
Who am I?
Why am I here?
Cloud deployment model
@joesondow #netflixcloudSunday, September 25, 2011
Agenda
Who am I?
Why am I here?
Cloud deployment model
Netflix Application Console
@joesondow #netflixcloudSunday, September 25, 2011
Agenda
Who am I?
Why am I here?
Cloud deployment model
Netflix Application Console
Why not the AWS Console?
@joesondow #netflixcloudSunday, September 25, 2011
Who am I?
@joesondowSunday, September 25, 2011
Who am I?
@joesondowSunday, September 25, 2011
Who am I?
Joe Sondow
@joesondowSunday, September 25, 2011
Who am I?
Joe Sondow
New York City
@joesondowSunday, September 25, 2011
Who am I?
Joe Sondow
New York City
Netflix since 2010
@joesondowSunday, September 25, 2011
Who am I?
Joe Sondow
New York City
Netflix since 2010
@joesondowSunday, September 25, 2011
Who am I?
Joe Sondow
New York City
Netflix since 2010
Cloud Tools
@joesondowSunday, September 25, 2011
Who am I?
Joe Sondow
New York City
Netflix since 2010
Cloud Tools
Grails
@joesondowSunday, September 25, 2011
Who am I?
Joe Sondow
New York City
Netflix since 2010
Cloud Tools
Grails
jQuery
@joesondowSunday, September 25, 2011
Why am I here?
Sunday, September 25, 2011
Why am I here?
Sunday, September 25, 2011
Sell you something
Why am I here?
Sunday, September 25, 2011
Sell you something
Discuss business plans
Why am I here?
Sunday, September 25, 2011
Sell you something
Discuss business plans
Answer technical questions
Why am I here?
Sunday, September 25, 2011
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS
Why am I here?
Sunday, September 25, 2011
Sell you something
Discuss business plans
Answer technical questions
Be a smaller fish in AWS
Steal your engineers
Why am I here?
Sunday, September 25, 2011
Cloud deployment model
Sunday, September 25, 2011
Cloud deployment modelApplications
Clusters
Sunday, September 25, 2011
Cloud deployment model
Sunday, September 25, 2011
Cloud deployment model
Auto ScalingGroup
Sunday, September 25, 2011
Cloud deployment model
Auto ScalingGroup
LaunchConfiguration
Sunday, September 25, 2011
Cloud deployment model
Auto ScalingGroup
LaunchConfiguration
Elastic LoadBalancer
Sunday, September 25, 2011
Cloud deployment model
Auto ScalingGroup
LaunchConfiguration
Amazon MachineImage
Elastic LoadBalancer
Sunday, September 25, 2011
Cloud deployment model
Auto ScalingGroup
LaunchConfiguration
SecurityGroup
Amazon MachineImage
Elastic LoadBalancer
Sunday, September 25, 2011
Cloud deployment model
Auto ScalingGroup
LaunchConfiguration
SecurityGroup
Amazon MachineImage
Instances
Elastic LoadBalancer
Sunday, September 25, 2011
Cloud deployment model
Auto ScalingGroup
LaunchConfiguration
SecurityGroup
Amazon MachineImage
Instances
Elastic LoadBalancer
Sunday, September 25, 2011
Cloud deployment model
Auto ScalingGroup
LaunchConfiguration
SecurityGroup
Amazon MachineImage
Instances
Elastic LoadBalancer
Sunday, September 25, 2011
Cloud deployment model
Auto ScalingGroup
LaunchConfiguration
SecurityGroup
Amazon MachineImage
Instances
Elastic LoadBalancer
Sunday, September 25, 2011
Cloud deployment model
Sunday, September 25, 2011
Cloud deployment model
Search
Sunday, September 25, 2011
Cloud deployment model
API Search
Sunday, September 25, 2011
Cloud deployment model
API
Ratings
Search
Sunday, September 25, 2011
Cloud deployment modelStreaming Starts
API
Ratings
Search
Sunday, September 25, 2011
Cloud deployment modelStreaming Starts
Autocomplete API
Ratings
Search
Sunday, September 25, 2011
Cloud deployment modelStreaming Starts
Autocomplete API
Sign Up
Ratings
Search
Sunday, September 25, 2011
Cloud deployment modelStreaming Starts
Autocomplete API
Sign Up
Ratings
Search
ApplicationApplication
Application
Application
Application
Application
Sunday, September 25, 2011
Inventing the Application
Sunday, September 25, 2011
Inventing the Application
Problem:
Application is not an Amazon concept
Sunday, September 25, 2011
Inventing the Application
Problem:
Application is not an Amazon concept
Solution:
Create an Application domain in SimpleDB
Enforce naming conventions on Amazon objects
Sunday, September 25, 2011
Fast Rollback
Sunday, September 25, 2011
Fast Rollback
Optimism causes outages
Sunday, September 25, 2011
Fast Rollback
Optimism causes outages
Production traffic is unique
Sunday, September 25, 2011
Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Sunday, September 25, 2011
Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
Sunday, September 25, 2011
Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
Monitor results
Sunday, September 25, 2011
Fast Rollback
Optimism causes outages
Production traffic is unique
Keep old version running
Switch traffic to new version
Monitor results
Revert traffic quickly
Sunday, September 25, 2011
Fast Rollback
Sunday, September 25, 2011
Fast Rollback
api-usprod-v007
api-frontend
Sunday, September 25, 2011
Fast Rollback
api-usprod-v007
api-frontend
api-usprod-v008
Sunday, September 25, 2011
Fast Rollback
api-usprod-v007
api-frontend
api-usprod-v008
Sunday, September 25, 2011
Fast Rollback
api-usprod-v007
api-frontend
api-usprod-v008
Sunday, September 25, 2011
Fast Rollback
api-usprod-v007
api-frontend
api-usprod-v008
Sunday, September 25, 2011
Fast Rollback
api-frontend
api-usprod-v008
Sunday, September 25, 2011
Fast Rollback
Sunday, September 25, 2011
Fast Rollback
api-usprod-v007
api-frontend
Sunday, September 25, 2011
Fast Rollback
api-usprod-v007
api-frontend
api-usprod-v008
Sunday, September 25, 2011
Fast Rollback
api-usprod-v007
api-frontend
api-usprod-v008
Sunday, September 25, 2011
Fast Rollback
api-usprod-v007
api-frontend
api-usprod-v008
Sunday, September 25, 2011
Fast Rollback
api-usprod-v007
api-frontend
Sunday, September 25, 2011
Inventing the Cluster
Sunday, September 25, 2011
Inventing the Cluster
Problem:
Two ASGs with one function but different names
Sunday, September 25, 2011
Inventing the Cluster
Problem:
Two ASGs with one function but different names
Solution:
Append version number in reserved format
Parse ASG name to determine long-term “cluster”
Sunday, September 25, 2011
Netflix Application Console
Sunday, September 25, 2011
Netflix Application ConsoleCloud interface
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
Sunday, September 25, 2011
REST API is easy in Grails
Sunday, September 25, 2011
REST API is easy in Grails
Sunday, September 25, 2011
REST API is easy in Grails
Sunday, September 25, 2011
REST API is easy in GrailsApplicationController.groovy
Sunday, September 25, 2011
Why not the AWS console?
Sunday, September 25, 2011
Why not the AWS console?No, not to keep me busy
Sunday, September 25, 2011
Why not the AWS console?
Sunday, September 25, 2011
Why not the AWS console?
Sunday, September 25, 2011
Why not the AWS console?
Hide keys
Sunday, September 25, 2011
Why not the AWS console?
Hide keys
Customize model
Sunday, September 25, 2011
Why not the AWS console?
Hide keys
Customize model
Automate workflow
Sunday, September 25, 2011
Why not the AWS console?
Hide keys
Customize model
Automate workflow
Enforce conventions
Sunday, September 25, 2011
Why not the AWS console?
Hide keys
Customize model
Automate workflow
Enforce conventions
Log changes
Sunday, September 25, 2011
Why not the AWS console?
Hide keys
Customize model
Automate workflow
Enforce conventions
Log changes
Integrate systems
Sunday, September 25, 2011
Why not the AWS console?
Hide keys
Customize model
Automate workflow
Enforce conventions
Log changes
Integrate systems
Create REST API
Sunday, September 25, 2011
Thank you
@joesondowhttp://www.slideshare.net/joesondow
Sunday, September 25, 2011
Thank youQuestions?
@joesondowhttp://www.slideshare.net/joesondow
Sunday, September 25, 2011