wayin devops-2013
TRANSCRIPT
DevOps at WayinOrchestration with AWS Cloud Formation
David M. JohnsonSoftware Developer - Wayin, Inc.
Friday, June 21, 13
Agenda
•What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
“Orchestration describes the automated arrangement, coordination, and management of complex computer systems, middleware, and services.
It is often discussed as having an inherent intelligence or even implicitly autonomic control, but those are largely aspirations or analogies rather than technical descriptions.”
http://en.wikipedia.org/wiki/Orchestration_(computing)
Friday, June 21, 13
“cloud orchestration = automation
+ integration + best practices”
https://www.ibm.com/developerworks/mydeveloperworks/blogs/9e696bfa-94af-4f5a-ab50-c955cca76fd0/entry/orchestrating_the_cloud_to_simplify_and_accelerate_service_delivery1
Friday, June 21, 13
Cloud Orchestration is...
Automating the infrastructure needed to provide a computing service so that resources are automatically allocated and de-allocated to meet changing demand
Friday, June 21, 13
The Orchestra Analogy
Friday, June 21, 13
Benefits
•Saves money
•Saves time
•Repeatable, version-able process
•Less human intervention
•Fewer stupid mistakes
Friday, June 21, 13
Agenda
• What is Cloud Orchestration
•Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
Friday, June 21, 13
Friday, June 21, 13
Friday, June 21, 13
Friday, June 21, 13
Friday, June 21, 13
Friday, June 21, 13
Friday, June 21, 13
Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
•Wayin Hub deployment & app architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
Glossary• EC2 - server virtualization
• EBS - mountable block storage
• ELB - load balancer
• R53 - domain name services (DNS)
• S3 - file / bucket storage service
• CloudFront - content distribution network (CDN)
• Autoscale - automatically stop & start servers
• CloudWatch / Cloud Alarms - monitoring & triggering
• Security Group - restrict port access on group of instances
• SQS - Simple Queuing Service
Friday, June 21, 13
Application Architecture
Browser
Front end
HTML5jQuery
Backbone
Ubuntu
Back end
Java, JettyJerseyGuice
RESTHTTP
Ubuntu
Cassandra
Redis
Memcache
Queue Service
Twitter Feeds Other Soc. Nets
Friday, June 21, 13
Ubuntu
Jenkins
API Security Group
S3
R53CloudFront
ELB
EC2Browser
Front end
DB Security Group
EBS
SQS
TwitterFeeds
Tracker Security Group
Deployment Architecture
Cloud Formation
// CFN template { "Parameters": {
}, "Mappings": {
}, "Resources": {
}}
Friday, June 21, 13
Cloud Formation
“An easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.
You can use AWS CloudFormation’s sample templates or create your own. Once deployed, you can modify and update the AWS resources in a controlled and predictable way allowing you to version control your AWS infrastructure in the same way as you version control your software.
You can deploy and update a template and its associated collection of resources (called a stack) via the AWS Management Console, CloudFormation command line tools or APIs. CloudFormation is available at no additional charge, and you pay only for the AWS resources needed to run your applications.”
Friday, June 21, 13
Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
•How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
Separate auto-scale groups
Ubuntu
API server
WAR
Jetty
Ubuntu
API server
WAR
Jetty
Ubuntu
API server
WAR
Jetty
Min: 3Max: 9
Scale up (quickly):CPU > X% for Y minutes
Scale down (slowly):CPU < A% for B minutes
Ubuntu
Cassandra
Redis
Memcache
DB serverUbuntu
Cassandra
Redis
Memcache
DB server
Min: 2Max: 2
We decide when to scaleup and scale down our DB server instances
Ubuntu
Tracker
WAR
Jetty
Ubuntu
Tracker
WAR
Jetty
Ubuntu
Tracker
WAR
Jetty
Min: 2Max: 20
Scale up (quickly):Queue has > X messages for Y minutes
Scale down (slowly):Queue has < X message for Y minutes
Friday, June 21, 13
CFN Launches new API server instances
1
Self-configuring API servers
DB Security Group
S3 ELBCloudInit script downloads the
deployment bundle2
CloudInit scripts find DB servers by Security Group, starts Jetty
3
CloudInit scripts put server into ELB rotation
4
API Security Group
CFNCloud Watch
Friday, June 21, 13
Self-configuring DB servers
DB Security Group
S3
CloudInit scripts finds other DB servers by Server Group3
CloudInit scripts download latest
deployment bundle
2Human runs the command to increase
DB servers by one
1
Ubuntu
New DB server
New Cassandra node joins cluster when ready7
CFN
CloudInit scripts write Memcache config, start it4
MemcacheCloudInit scripts write Redis config, assume first server is master, start it5
RedisCloudInit scripts write Cassandra config, start it6
Cassandra
Friday, June 21, 13
Automated deployment
API Security Group
GitDeveloper merges code into deployment branch 1
Developer kicks off Jenkins build 2
S3
Jenkins copies bundle to S3 bucket 3
Developer kicks off deployment build 4
Jenkins runs deploy command
via SSH
5
On each server the script runs locally, downloads and installs the bundle
6
Ubuntu
Jenkins
Tracker Security Group
Friday, June 21, 13
Incremental backups copied to S3 bucket5
Automated Backups
S3
Jenkins runs daily backup job
1
Backup script creates EBS snapshot 3
DB Security Group
Cronjob kicks off incremental backups4
Cassandra snapshot backups happen simultaneously
2
Ubuntu
Jenkins
EBS
Friday, June 21, 13
Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
•About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
CFN Template skeleton{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "Example stack",
"Parameters": {
},
"Mappings": {
},
"Resources": {
}}
Friday, June 21, 13
CFN Parameters{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "WayinHub Three Server-type Prod Stack",
"Parameters": {
"DnsName" : { "Description" : "DNS name to be created", "Type" : "String", "Default": "stack-test" },
"DnsDomain" : { "Description" : "DNS domain to be used", "Type" : "String", "Default": "twt.pl" },
"KeyPair" : { "Description" : "EC2 Key Pair to allow SSH...", "Type" : "String", "Default": "wayin" },
Friday, June 21, 13
CFN Parameters "ReleaseBucket" : { "Description" : "S3 Bucket of release bundle", "Type" : "String", "Default": "hub-deployment-release" }, "ApiMinServers" : { "Description" : "Minimum number of API servers", "Type" : "Number", "Default": "2", "MinValue": "2" },
"ApiMaxServers" : { "Description" : "Maximum number API servers", "Type" : "Number", "Default": "5", "MinValue": "2" },
"ApiInstanceType": { "Description": "EC2 instance type for API servers", "Type": "String", "Default": "m1.large", "AllowedValues": ["m1.xlarge", "m3.xlarge"], "ConstraintDescription": "must be a valid EC2 type." },
Friday, June 21, 13
CFN Mappings
"Mappings": { "AWSRegionArch2AMI": { "us-east-1": { "64": "ami-e864da81" }, "us-west-2": { "64": "ami-50f97060" } }, "TwoAZs": { "us-east-1": { "AZ1" : "us-east-1a", "AZ2" : "us-east-1b" }, "us-west-2": { "AZ1" : "us-west-2a", "AZ2" : "us-west-2b" } } },
Friday, June 21, 13
CFN Resources "Resources": { "DbAutoScalingLaunchConfiguration": { "Type": "AWS::AutoScaling::LaunchConfiguration", }, "DbAutoScalingGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", }, "ApiAutoScalingLaunchConfiguration": { "Type": "AWS::AutoScaling:: LaunchConfiguration", }, "ApiAutoScalingGroup": { "Type": "AWS::AutoScaling::AutoScalingGroup", }, "ApiServerScaleUpPolicy" : { "Type" : "AWS::AutoScaling::ScalingPolicy", }, "ApiServerScaleDownPolicy" : { "Type" : "AWS::AutoScaling::ScalingPolicy", }, "CPUAlarmHigh": { }, "CPUAlarmLow": { },
Friday, June 21, 13
CFN Resources"ApiAutoScalingLaunchConfiguration": { "Type": "AWS::AutoScaling::LaunchConfiguration", "Properties": { "UserData": { "Fn::Base64" : { "Fn::Join": [ "", [ "#!/bin/bash -ex\n",
"# Log all output from this script\n", "exec >/var/log/wayin-bootstrap.log 2>&1\n",
"# Expose various environmental variables, etc.\n", "cat >/etc/profile.d/wayin-env.sh <<EOF\n", "export ELB_NAME=", { "Ref" : "ApiElasticLoadBalancer" }, "\n", "export EC2_REGION=", { "Ref" : "AWS::Region" }, "\n", "export DNS_NAME=", { "Ref" : "DnsName" }, "\n", "export DNS_DOMAIN=", { "Ref" : "DnsDomain" }, "\n", "EOF\n", "# Install various packages\n", "apt-get update\n", "apt-get -y install s3cmd\n", "\n", "# Download our application files\n", "cd /usr/share/wayinhub\n", "s3cmd --config=/etc/s3cfg get s3://",{"Ref":"ReleaseBucket"},"/twithub.tgz\n", "tar xf twithub.tgz\n", "rm -fr twithub.tgz\n", "cd ./init_instance\n", "sh ./init_api_server.sh\n" ]]} },
Friday, June 21, 13
CFN Console
Friday, June 21, 13
Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
•About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
Lock-in in automation
• Cloud Formation itself is lock-in
• Plenty of AWS dependencies in our:
• Instance init scripts
• Deployment scripts
• Backup scripts
Friday, June 21, 13
No lock-in in app code
• Bulk of our code is in our application
• AWS dependencies are pluggable
Friday, June 21, 13
Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
•Designing for orchestration
Friday, June 21, 13
Choose right components
• Good
• Peer-to-peer
• Linear scalability
• Bad
• Master-slave
• “Special” servers
• Single point-of-failure
Friday, June 21, 13
Bootstrap everything
• Every instance must self-configure
• Make services easy to discover
Friday, June 21, 13
Keep it simple
• Minimize number of different...
• Server types
• External services
• Learning curves
Friday, June 21, 13
Questions?
Friday, June 21, 13