deep dive: infrastructure as code

©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved

Infrastructure as Code

with AWS CloudFormationLee Kemp, Software Development Manager, AWS

You are on board …

continuous delivery

• services and applications

DevOps

• culture, automation, measurement, sharing

cloud

• infrastructure-as-code

Business needs to experiment, innovate, reduce risk

AWS CloudFormation

AWS CloudFormation

• Create templates of the infrastructure and applications you want to run on AWS

• Have the CloudFormation service automatically provision the required AWS resources and their relationships from the templates

• Easily version control, replicate, or update the infrastructure and applications using the templates

• Integrate with other development, CI/CD, and management tools

Basic workflow

design

create infrastructure

templates

write application

code

create stacks

iterate

depends on

Design: Building a food-ordering service

food catalog website

ordering website

customer DB service

inventory service

recommendations service

analytics servicefulfillment

service

payment

service

Create template for the food catalog website

security group

Auto Scaling group

EC2

instance

Elastic Load

Balancing

customer DB service

inventory service

recommendations service

software pkgs,

config, and data

CloudWatch

alarms

ElastiCache

Create template: Resources

security group

Auto Scaling group

EC2

instance

Elastic Load

Balancing

Software pkgs,

config, and data

CloudWatch

alarms

"Resources" : {"SecurityGroup" : {},"WebServerGroup" : {

"Type" : "AWS::AutoScaling::AutoScalingGroup","Properties" : {

"MinSize" : "1","MaxSize" : "3","LoadBalancerNames" : [ { "Ref" :

"LoadBalancer" } ],...

}},"LoadBalancer" : {},"CacheCluster" : {},"Alarm" : {}

},

CloudFormation template

ElastiCache

Create template: Parameters

Auto Scaling group

EC2

instance

recommendations Service

inventory service

customer DB service

info to customize stack at creation

Examples: instance type, app package version

"Parameters" : {"CustomerDBServiceEndPoint" : {

"Description" : "URL of the Customer DB Service","Type" : "String"

},"CustomerDBServiceKey" : {

"Description" : "API key for the Customer DB Service",

"Type" : "String","NoEcho" : "true"

},"InstanceType" : {

"Description" : "WebServer EC2 instance type","Type" : "String","Default" : "m3.medium","AllowedValues" :

["m3.medium","m3.large","m3.xlarge"],"ConstraintDescription" : "Must be a valid

instance type"


Create template: Outputs

Elastic Load

Balancing

"Resources" : {

"LoadBalancer" : {},

...

},

"Outputs" : {

"WebsiteDNSName" : {

"Description" : "The DNS name of the website",

"Value" : {

"Fn::GetAtt" : [ "LoadBalancer", "DNSName" ]

}

}

}


Create template: Deploy and configure software

Auto Scaling group

EC2

instance

Software pkgs,

config, and data

"AWS::CloudFormation::Init": {

"webapp-config": {

"packages" : {}, "sources" : {}, "files" : {},

"groups" : {}, "users" : {},

"commands" : {}, "services" : {}

},

"chef-config" : {}

}


declarative

debuggable

updatable

highly secure

BIOT™ (Bring in

Other Tools)

Create template: Language features

Create stack

Operate stack

Use a wide range of AWS services

Auto Scaling

Amazon CloudFront

AWS CloudTrail

Amazon CloudWatch

AWS Data Pipeline (new)

Amazon DynamoDB

Amazon EC2

Amazon ECS (new)

Amazon ElastiCache

AWS Elastic Beanstalk

Elastic Load Balancing

AWS managed policies in IAM (new)

Amazon Kinesis

AWS Lambda (new)

AWS OpsWorks

Amazon RDS

Amazon Redshift

Amazon Route 53

Amazon S3

Amazon SimpleDB

Amazon SNS

Amazon SQS

Amazon VPC

and more …

Basic workflow

design

create infrastructure

templates

write application

code

create stacks

iterate

Infrastructure-as-code workflow

codeversion control

code review

integrate

“It’s all software”

“It’s all software” – Organize like it’s software

front-end services

• consumer website, seller website, mobile back end

back-end services

• search, payments, reviews, recommendations

shared services

• CRM DBs, common monitoring /alarms, subnets, security groups

base network

• VPCs, Internet gateways, VPNs, NATs

identity • IAM users, groups, roles

“It’s all software” – Build and operate like it’s

software

application software

source code

package

loader/interpreter

desired application state in memory

infrastructure software

JSON templates/JSON template generators

JSON templates

AWS CloudFormation

desired infrastructure in the cloud

Iterate on infrastructure

Update stack

in place blue-green

faster

cost-efficient

simpler state and

data migration

working stack

not touched

Extending AWS CloudFormation

Extend with custom resources

security group

Auto Scaling group

EC2

instance

Elastic Load

Balancing

software pkgs,

config, and data

CloudWatch

alarms

Web Analytics

ServiceAWS

CloudFormation

provision

AWS resources

"Resources" : {

"WebAnalyticsTrackingID" : {

"Type" : "Custom::WebAnalyticsService::TrackingID",

"Properties" : {

"ServiceToken" : "arn:aws:sns:...",

"Target" : {"Fn::GetAtt" : ["LoadBalancer", "DNSName"]},

"Plan" : "Gold"

}

},

...

“success” + metadata

“create, update, roll back, or delete”

+ metadata

ElastiCache

Lambda-powered custom resources

security group

Auto Scaling group

EC2

instance

Elastic Load

Balancing

software pkgs,

config, and data

CloudWatch

alarms

your AWS CloudFormation stack

// implement custom logic here

look up an AMI ID

your AWS Lambda functions

look up VPC ID and subnet ID

reverse an IP address

Lambda-powered

custom resources

ElastiCache

Application deployment as code

infrastructure provisioning

EC2

SQS, SNS, Amazon Kinesis, etc.

databases

VPC

IAM

application deployment

download packages, install software, configure

apps, bootstrap apps, update software, restart

apps, etc.

CloudFormation

• templatize

• replicate

• automate

Application deployment as code

inside a CloudFormation template

Amazon Machine Images

CloudFormation::Init

Chef, Puppet, CodeDeploy

OpsWorks

Chef

Metadata

AWS::CloudFormation::Init


declarative

reusable

grouping and ordering

debuggable

updatable

highly secure

BIOT™ (Bring in Other Tools)

ow.ly/DiNCm

http://ow.ly/DiNCm


"AWS::CloudFormation::Init": {

"webapp-config": {

"packages" : {}, "sources" : {}, "files" : {},

"groups" : {}, "users" : {},

"commands" : {}, "services" : {}

Declarative


Debuggable


Supports updates

"packages" : {}, "sources" : {}, "files" : {}, "groups" : {}, "users" : {},"commands" : {}, "services" : {}


"install_chef" : {},

"install_wordpress" : {

"commands" : {

"01_get_cookbook" : {}, ...,

"05_configure_node_run_list" : {

"command" : "knife node run_list add -z `knife node list -z` recipe[wordpress]",

"cwd" : "/var/chef/chef-repo",

"env" : { "HOME" : "/var/chef" }

Flexibility to bring in other tools, such as AWS CodeDeploy and Chef

ow.ly/DiNkz

http://ow.ly/DiNkz


"YourInstance": {

"Metadata": {

"AWS::CloudFormation::Authentication": {

"S3AccessCreds": {

"type": "S3",

"roleName": { "Ref" : "InstanceRole"},

"buckets" : ["your-bucket"]

}

},

"AWS::CloudFormation::Init": {}

Supports role-based authentication

securely download

choose auth type

IAM role is

recommended

ow.ly/DqkrB

http://ow.ly/DqkrB

Use AWS::CloudFormation::Init

"UserData": {

"# Get the latest CloudFormation helper scripts package\n","yum update -y aws-cfn-bootstrap\n",

"# Trigger CloudFormation::Init configuration \n","/opt/aws/bin/cfn-init --stack ", {"Ref": "AWS::StackId"},

" --resource WebServerInstance ", " --region ", {"Ref": "AWS::Region"}, "\n",

"# Signal completion\n","/opt/aws/bin/cfn-signal –e $? --stack ", {"Ref": "AWS::StackId"},

" --resource WebServerInstance ", " --region ", {"Ref": "AWS::Region"}, "\n"

Use CloudWatch Logs for debugging

"install_logs": {

"packages" : { ... "awslogs" ... },

"services" : { ... "awslogs" ... }

"files": {

"/tmp/cwlogs/cfn-logs.conf": {}

file = /var/log/cfn-init.loglog_stream_name = {instance_id}/cfn-init.log

file = /var/log/cfn-hup.loglog_stream_name = {instance_id}/cfn-hup.log

ow.ly/E0zO3

http://ow.ly/E0zO3

Use CloudWatch Logs for debugging

ow.ly/E0zO3

http://ow.ly/E0zO3

Bake AMIs for faster booting and for maintaining

golden images

dev/test stacks bake AMI staging/prod stacks

tracking

Using CloudFormation and OpsWorks

together


EC2


databases

VPC

IAM



apps, bootstrap apps, update software, restart apps, etc.

CloudFormation

• templatize

• replicate

• automate

OpsWorks

• built-in application lifecycle

• interactive application console

OpsWorks and CloudFormation side by side

OpsWorks

• built-in application lifecycle

• interactive application console


EC2


databases

VPC

IAM



apps, bootstrap apps, update software, restart apps, etc.

CloudFormation

• templatize

• replicate

• automate

OpsWorks “inside” CloudFormation

Infrastructure-as-code in a CI/CD pipeline

CloudFormation in a CI/CD pipeline

AWS CloudFormationIssue Tracker

app developers

DevOps engineers,infrastructure developers,

systems engineers

dev env code repo

app pkgs, CloudFormationtemplates, etc.

CI server

test

staging

prodcode review

“infrastructure as code"

app code and templates

Templatize existing resources

CloudFormer: Templatize existing resources

1. Launch a CloudFormer

application stack

2. Walk through the

CloudFormer UI and select

resources to templatize

4. Customize

Example: parameterize

resource properties

5. Create a new stack

Practitioners of infrastructure-as-code

• Developers/DevOps teams value CloudFormation for its ability to treat infrastructure as code, allowing them to apply software engineering principles, such as SOA, revision control, code reviews, integration testing to infrastructure.

• IT admins and MSPs value CloudFormation as a platform to enable standardization, managed consumption, and role specialization.

• ISVs value CloudFormation for its ability to support scaling out of multi-tenant SaaS products by quickly replicating or updating stacks. ISVs also value CloudFormation as a way to package and deploy their software in their customer accounts on AWS.

CHICAGO

deep dive: infrastructure as code

Technology