aws infrastructure as code - september 2016 webinar series

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Chris Munns

Business Development Manager – DevOps

Amazon Web Services

September 2016

Infrastructure as Code: Best

Practices with AWS CloudFormation

CloudFormation concepts and technology

JSON/YAML formatted file

Parameter definition

Resource creation

Configuration actions

Framework

Stack creation

Stack updates

Error detection and rollback

Configured AWS resources

Comprehensive service support

Service event aware

Customizable

Template CloudFormation Stack

Infrastructure as Code workflow

codeversion control

code review

integrate deploy


codeversion control

code review

integrate deploy

Text EditorGit/SVN/

Perforce

Review Tools

Syntax Validation

Tools

AWS Services


codeversion control

code review

integrate deploy

“It’s all software”

Text EditorGit/SVN/

Perforce

Review Tools

Syntax Validation

Tools

AWS Services

In-place Blue-Green

Traffic

• Quicker and more cost efficient

• Updates or replaces resources

• Simpler state and data

migration

• Working stack stays intact for

easy roll backs

• Potentially need to sync data

• Shift traffic to new stack

Templates

Stacks

Update like software

Traffic

{

"AWSTemplateFormatVersion" : "2010-09-09",

"Description" : "AWS CloudFormation Sample Template EC2InstanceSample: **WARNING** This template an Amazon EC2 instances. You will be billed for the AWS resources used if

you create a stack from this template.",

"Parameters" : {

"KeyName" : {

"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance",

"Type" : "String"

},

"Environment": {

"Type" : "String",

"Default" : ”Dev",

"AllowedValues" : [”Mgmt", "Dev", ”Staging", "Prod"],

"Description" : "Environment that the instances will run in.”

}

},

"Mappings" : {

"RegionMap" : {

"us-east-1" : { "AMI" : "ami-7f418316" },

"us-west-2" : { "AMI" : "ami-16fd7026" }

}

},

"Conditions" : {

”EnableEBSOptimized" : {"Fn::Equals" : [{"Ref" : " Environment "}, ”Prod"]},

},

"Resources" : {

"Ec2Instance" : {

"Type" : "AWS::EC2::Instance",

"Properties" : {

"KeyName" : { "Ref" : "KeyName" },

"EbsOptimized " : {"Fn::If": [ " EnableEBSOptimized ", {“true”}, {“false”}]},

"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]},

"UserData" : { "Fn::Base64" : "80" }

}

}

},

"Outputs" : {

"InstanceId" : {

"Description" : "InstanceId of the newly created EC2 instance",

"Value" : { "Ref" : "Ec2Instance" }

},

"PublicDNS" : {

"Description" : "Public DNSName of the newly created EC2 instance",

"Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicDnsName" ] }

}

}

}

{




"Parameters" : {

"KeyName" : {


"Type" : "String"

},

"Environment": {

"Type" : "String",




}

},

"Mappings" : {

"RegionMap" : {

"us-east-1" : { "AMI" : "ami-7f418316" },

"us-west-2" : { "AMI" : "ami-16fd7026" }

}

},

"Conditions" : {


},

"Resources" : {

"Ec2Instance" : {


"Properties" : {




"UserData" : { "Fn::Base64" : "80" }

}

}

},

"Outputs" : {

"InstanceId" : {



},

"PublicDNS" : {



}

}

}

HEADERS

PARAMETERS

MAPPINGS

RESOURCES

OUTPUTS

CONDITIONALS

{




"Parameters" : {

"KeyName" : {


"Type" : "String"

},

"Environment": {

"Type" : "String",




}

},

"Mappings" : {

"RegionMap" : {

"us-east-1" : { "AMI" : "ami-7f418316" },

"us-west-2" : { "AMI" : "ami-16fd7026" }

}

},

"Conditions" : {


},

"Resources" : {

"Ec2Instance" : {


"Properties" : {




"UserData" : { "Fn::Base64" : "80" }

}

}

},

"Outputs" : {

"InstanceId" : {



},

"PublicDNS" : {



}

}

}

HEADERS

PARAMETERS

MAPPINGS

RESOURCES

OUTPUTS

CONDITIONALS

Description of what your stack does, contains, etc

Provision time values that add structured flexibility and

customization

Pre-defined conditional case statements

Conditional values set via evaluations of passed references

AWS resource definitions

Resulting attributes of stack resource creation

{ "Description" : "Create an EC2 instance.”,"Parameters": {

"KeyName": { "Description" : "Name of an existing EC2 KeyPair to enable SSH

access into the WordPress web server", "Type": "AWS::EC2::KeyPair::KeyName"

},"EC2InstanceType" : {

"Description" : "EC2 instance type", "Type" : "String","Default" : "t2.micro", "AllowedValues" : [ "t2.micro", "t2.small", "t2.medium" ], "ConstraintDescription" : "Must be t2.micro, t2.small, t2.medium"

},},

Template Anatomy - Parameters

{ "Description" : "Create an EC2 instance.”,"Resources" : {

"Ec2Instance" : { "Type" : "AWS::EC2::Instance", "Properties" : {

"KeyName" : “my-key-pair”, "ImageId" : "ami-6869aa05”,“InstanceType” : “m3.medium”

} }

}}

Template Anatomy - Resources

"Outputs" : { "WebsiteURL" : {

"Description" : ”DNS name of the website", "Value" : {

"Fn::GetAtt" : [ “LoadBalancer”, “DNSName” ] }

} }

Template Anatomy - Outputs

CloudFormation Best Practices

Visualize template resources

Modify template with drag-drop gestures

Customize sample templates

CloudFormation Designer

Avoid manual resource modifications

Avoid making quick-fixes out of band

Update your stacks with CloudFormation

Do not manually change resources

Consider using resource based permissions to

limit ability to make changes directly

Preview updates with Change Sets

Learn the intrinsic functions

"Mappings" : {"RegionMap" : {

"us-east-1" : { "32" : "ami-6411e20d", "64" : "ami-7a11e213" },"us-west-1" : { "32" : "ami-c9c7978c", "64" : "ami-cfc7978a" },"eu-west-1" : { "32" : "ami-37c2f643", "64" : "ami-31c2f645" },"ap-southeast-1" : { "32" : "ami-66f28c34", "64" : "ami-60f28c32" },"ap-northeast-1" : { "32" : "ami-9c03a89d", "64" : "ami-a003a8a1" }

}},

Fn::FindInMap

Fn::FindInMap

"Resources" : {"myEC2Instance" : {

"Type" : "AWS::EC2::Instance","Properties" : {

"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "32"]},

"InstanceType" : "m1.small"}

}}

"Resources" : {

"Ec2Instance" : {


"Properties" : {


"SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],


"UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[

"#!/bin/bash -ex","\n",

"yum -y install gcc-c++ make","\n",

"yum -y install mysql-devel sqlite-devel","\n",

"yum -y install ruby-rdoc rubygems ruby-mysql ruby-devel","\n",

"gem install --no-ri --no-rdoc rails","\n",

"gem install --no-ri --no-rdoc mysql","\n",

"gem install --no-ri --no-rdoc sqlite3","\n",

"rails new myapp","\n",

"cd myapp","\n",

"rails server -d","\n"]]}}

}

}

Use EC2 UserData, which is available as a property of AWS::EC2::Instance resources

Bootstrap your applications using EC2 UserData

AWS CloudFormation provides helper scripts

for deployment within your EC2 instances

Metadata Key —AWS::CloudFormation::Init

Cfn-init reads this metadata key and installs

the packages listed in this key (e.g., httpd,

mysql, and php). Cfn-init also retrieves and

expands files listed as sources.

Amazon EC2

AWS CloudFormation

cfn-init

cfn-hup

cfn-signal

cfn-get-

metadata

Bootstrap your applications using helper scripts

"Metadata": {

"AWS::CloudFormation::Init" : {

"config" : {

"packages" : {

},

"sources" : {

},

"commands" : {

},

"files" : {

},

"services" : {

},

"users" : {

},

"groups" : {

}

}

}

“WebAppHost" : {


"Metadata" : {

"AWS:CloudFormation::Init" : {

"config" : {

"packages" : {

"yum" : {

"gcc" : [],

"gcc-c++" : [],

"make" : [],

"automake" : [],

Use AWS::CloudFormation::Init with cfn-init to help bootstrap instances:

Bootstrapping example

{

"Statement" : [

{

"Effect" : "Allow",

"Action" : "Update:*",

"Principal": "*",

"Resource" : "*"

},

{

"Effect" : "Deny",

"Action" : "Update:*",

"Principal": "*",

"Resource" : "LogicalResourceId/ProductionDatabase"

}

]

}

Prevent stack updates to protected resources using Stack policies

Protect your resources using Stack policies

Ownership based template design

Limit one template to a single service

Aim for reusability across environments (development, testing, production)

Use nested stacks and cross-stack reference to break up large templates

Organize templates according to team structure/job function/line of business

Template File

Defining Stack

The entire infrastructure can be represented in an AWS

CloudFormation template.

Many Stacks & Environments from One Template

Template File

Defining Stack



Use the version control system of your choice to store and track changes to this template


Git

Perforce

SVN

…

Template File

Defining Stack

Git

Perforce

SVN

…

Dev

Test

Prod



Use the version control system of your choice to store and track changes to this template

Build out multiple environments, such as for Development, Test, Production and even DR using the same template


Ownership – nested stacks

Web-SG

Ownership – cross-stack references

App-SG

App-SG

DB-SG

Re-usable Templates – across AWS Regions

Consider environmental or regional differences

Amazon EC2 image Ids

VPC environment or “classic” environment

Available instance types

IAM policy principals

Endpoint names

Amazon Resource Names (arns)

"LogsBucketPolicy": {"Type": "AWS::S3::BucketPolicy","Properties": {

"Bucket": {"Ref": "LogsBucket”},"PolicyDocument": {

"Version": "2008-10-17","Statement": [{"Sid": "ELBAccessLogs","Effect": "Allow","Resource": {

"Fn::Join": [ "", [ “arn:aws:s3:::", { "Ref": "LogsBucket" }, "/", "Logs",

"/AWSLogs/", { "Ref": "AWS::AccountId" }, "/*” ]]},"Principal": …,"Action": [ "s3:PutObject" ]

}]}

}},

Use “pseudo-parameters” to retrieve environmental data

Account Id

Region

Stack Name and Id

Re-usable Templates – “Pseudo-Parameters”

Re-usable Templates - Using mappings

"LogsBucketPolicy": {"Type": "AWS::S3::BucketPolicy","Properties": {

"Bucket": {"Ref": "LogsBucket”},"PolicyDocument": {"Version": "2008-10-17","Statement": [{

"Sid": "ELBAccessLogs","Effect": "Allow","Resource": {"Fn::Join": [ "", [

{ "Fn::FindInMap" : ["RegionalConfig", {"Ref" : "AWS::Region"},"ArnPrefix”]},

"s3:::”, { "Ref": "LogsBucket" }, "/", "Logs", "/AWSLogs/”,{ "Ref": "AWS::AccountId" }, "/*" ] ]

},}

“Mappings” : {

“RegionalConfig” : {“us-east-1” : {

“AMI” : “ami-12345678”,

”ELBAccountId": "127311923021”,

“ArnPrefix” : “arn:aws:”

},“us-west-1” : {

“AMI” : “ami-98765432”

”ELBAccountId": “027434742980"

“ArnPrefix” :“arn:aws:”

},:

}}

Use mappings to define variables

Single place for configuration

Re-usable within the template

Use conditionals to customize

resources and parameters

"DBEC2SG": {"Type": "AWS::EC2::SecurityGroup","Condition" : "Is-EC2-VPC","Properties" : {…}

},

"DBSG": {"Type": "AWS::RDS::DBSecurityGroup","Condition" : "Is-EC2-Classic","Properties": {…}

},

"MySQLDatabase": {"Type": "AWS::RDS::DBInstance","Properties": {

:

"VPCSecurityGroups": { "Fn::If" : [ "Is-EC2-VPC", [ { "Fn::GetAtt": [ "DBEC2SG", "GroupId" ] } ], { "Ref" : "AWS::NoValue"}]},

"DBSecurityGroups": { "Fn::If" : [ "Is-EC2-Classic", [ { "Ref": "DBSG" } ],{ "Ref" : "AWS::NoValue"}]}

}}

}

"Conditions" : {

"Is-EC2-VPC” : { "Fn::Or" : [ {"Fn::Equals" : [

{"Ref” : "AWS::Region"},"eu-central-1" ]},

{"Fn::Equals" : [{"Ref" : "AWS::Region"}, "cn-north-1" ]}]},

"Is-EC2-Classic" : { "Fn::Not" : [{ "Condition" : "Is-EC2-VPC"}]}

},

Re-usable Templates – Using conditionals

Best Practices Summary

CloudFormation Designer

Avoid manual resource modifications

Preview updates with Change Sets

Learn the intrinsic functions

Bootstrap your applications using UserData and helper scripts

Protect critical resources using stack policies


Plan for environment and multi-region

Use Pseudo-Parameters

Use Mappings

Use Conditionals

YAML formatted templates

Overview of template structure / basics

New function formatting (!Ref / !GetAZs / !FindInMap)

New Intrinsic Function ( Fn::Sub )

Cross Stack References

New function Fn::ImportValue

Allows use of outputs from unrelated stacks with custom resource

New

Key new features

New

CloudFormation - YAML

Why YAML?

Better authoring and readability of templates

Native comment support

Simplification as templates get more and more complex

New

Cloudformation - YAML

Resources:

VPC1:

Type: "AWS::EC2::VPC"

Properties:

CidrBlock: !Ref VPC1Cidr

Tags:

-

Key: "Name"

Value: "TroubleShooting"

Structure is shown through indentation (one or more spaces).

Sequence items are denoted by a dash

Key value pairs within a map are separated by a colon.

Tips: Use a monospace font, don’t use Tab, save using UTF-8

All sections are the same as in a JSON template

---

AWSTemplateFormatVersion: "version date"

Description:

String

Metadata:

template metadata

Parameters:

set of parameters

Mappings:

set of mappings

Conditions:

set of conditions

Resources:

set of resources

Outputs:

set of outputs

CloudFormation – YAML Template Structure

Two ways to declare Intrinsic functions: Long and Short

Short Form:

!FindInMap [ MapName, TopLevelKey, SecondLevelKey ]

Long Form:

"Fn::FindInMap" : [ "MapName", "TopLevelKey", "SecondLevelKey"]

Tag = ! (Its not Negation operator)

Few things to note with Tags

You cannot use one tag immediately after another

!Base64 !Sub…

Instead, you can do this

"Fn::Base64": !Sub...

!Select [ !Ref Value, [1,2,3]]

CloudFormation – YAML Function Declaration

Fn::Base64 Fn::AndShort !Base64 valueToEncode Short !And [condition]

Long "Fn::Base64": valueToEncode Long "Fn::And": [condition]

Fn::Equals Fn::IfShort !Equals [value_1, value_2] Short !If [condition_name, value_if_true, value_if_false]

Long "Fn::Equals": [value_1, value_2] Long "Fn::If": [condition_name, value_if_true, value_if_false]

Fn::Not Fn::OrShort !Not [condition] Short !Or [condition, ...]

Long "Fn::Not": [condition] Long "Fn::Or": [condition, ...]

CloudFormation – Intrinsic Functions

NewCloudFormation – Fn::Sub

Substitute variables in an input string with values

Function accepts a string or a map as a parameter.

Usage

VarName: ${MyVariableValue}

Literal: ${!LiteralValue}

Use | if you are spanning multiple lines

Available in JSON as well

/tmp/create-wp-config:

content: !Sub |

#!/bin/bash -xe

cp /var/www/html/wordpress/wp-config-sample.php /var/www/html/wordpress/wp-config.php

sed -i "s/'database_name_here'/'${DBName}'/g" wp-config.php

sed -i "s/'username_here'/'${DBUser}'/g" wp-config.php

sed -i "s/'password_here'/'${DBPassword}'/g" wp-config.php

mode: '000500'

owner: root

group: root

configure_wordpress:

commands:

01_set_mysql_root_password:

command: !Sub |

mysqladmin -u root password '${DBRootPassword}'

test: !Sub |

$(mysql ${DBName} -u root --password='${DBRootPassword}' >/dev/null 2>&1 </dev/null); (( $? != 0 ))

02_create_database:

command: !Sub |

mysql -u root --password='${DBRootPassword}' < /tmp/setup.mysql

test: !Sub |

$(mysql ${DBName} -u root --password='${DBRootPassword}' >/dev/null 2>&1 </dev/null); (( $? !=0))

03_configure_wordpress:

command: /tmp/create-wp-config

cwd: /var/www/html/wordpress

CloudFormation – Fn::Sub Examples

CloudFormation – Cross Stack References

Sharing resources made easy

IAM roles, VPC, Security groups

Add an explicit “Export” declaration to stack output

Use the resource in another stack using a new intrinsic function,

Fn::ImportValue`

Few guidelines:

Export names must be unique within an account and region

Cannot create references across regions

Cannot delete a stack that is referenced by another stack (Dependencies

are communicated in errors)

Outputs cannot be modified or removed as long as it is referenced by a

current stack

New

The new intrinsic function for accessing exported outputs.

JSON{ "Fn::ImportValue" : sharedValueToImport }

YAML"Fn::ImportValue": sharedValueToImport

!ImportValue sharedValueToImport

CloudFormation – Fn::ImportValue

Stack A

Stack B

"Outputs": {

"WebServerSecurityGroup": {

"Description": "TheIDofthesecuritygroup",

"Value": {"Fn: : GetAtt": ["WebServerSecurityGroup", "GroupId"]},

"Export": { "Name": "AccountSecGroup"}}

}

"Resources" : {

"WebServerInstance" : {


"Properties" : {

"InstanceType" : "ts.micro",

"ImageId" : "ami-a1b23456",

"NetworkInterfaces" : [{

"GroupSet" : [{ "Fn::ImportValue" : "AccountSecGroup" ]} ]}

}

}

}

CloudFormation – Cross Stack Examples

Questions?

Thank you!

aws infrastructure as code - september 2016 webinar series

Technology