devops on aws: deep dive on infrastructure as code

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Alex Corley – Solutions ArchitectPublic Sector - State and Local Government

June 2016

Infrastructure as Code

Best Practices on AWS

Learning objectives

• Understand Infrastructure as Code

• Understand the AWS services that help you manage

your infrastructure as code

• Discover best practices for managing your AWS

infrastructure, host configuration, and applications

Background

Moving to the cloud and AWS allows you to provision and

manage infrastructure in new ways:

• Infrastructure can be provisioned in seconds

• Scale can be achieved without complicated capacity

planning

• APIs let you interact with infrastructure using languages

typically used in applications

What is Infrastructure as Code?

A practice in which traditional infrastructure management

techniques are supplemented by or replaced with code-

based tools and software development techniques

Infrastructure as Code workflow

CodeVersion

Control

Code

ReviewIntegrate Deploy


CodeVersion

Control

Code


Text EditorGit/SVN/

Perforce

Review

Tools

Syntax

Validation

Tools

AWS

Services


“It’s all software”

CodeVersion

Control

Code


Text EditorGit/SVN/

Perforce

Review

Tools

Syntax

Validation

Tools

AWS

Services

Application Configuration

AWS Resources


Operating System and Host Configuration

AWS ResourcesOperating System and

Host ConfigurationApplication Configuration



Infrastructure Resource

Management




Management

Host Configuration

Management




Management

Host Configuration

ManagementApplication Deployment



AWS CloudFormation

AWS OpsWorks

AWS CodeDeploy



AWS CloudFormation

AWS OpsWorks

AWS CodeDeploy

Amazon Virtual Private

Cloud (Amazon VPC)

Amazon Elastic Compute

Cloud (Amazon EC2)

AWS Identity and Access

Management (IAM)

Amazon Relational Database

Service (Amazon RDS)

Amazon Simple Storage

Service (Amazon S3)

AWS CodePipeline

…

Microsoft Windows Registry

Linux networking

OpenSSH

LDAP

Active Directory domain

registration

Centralized logging

System metrics

Deployment agents

Host monitoring

…

Application dependencies

Application configuration

Service registration

Management scripts

Database credentials

…

allOfThis == $Code

AWS CloudFormation

• Create templates that describe

and model AWS infrastructure

• CloudFormation then provisions

AWS resources based on

dependency needs

• Perform version control on,

replicate, and update the

templates like app code

• Integrates with development,

CI/CD, management tools

• No additional charge to use

Benefits

Templated resource

provisioning

Infrastructure

as code

Declarative

and flexible

Easy to use

CloudFormation concepts and technology

JSON formatted file

Parameter definition

Resource creation

Configuration actions

Framework

Stack creation

Stack updates

Error detection and rollback

Configured AWS resources

Comprehensive service support

Service event aware

Customizable

Template CloudFormation Stack

Anatomy of a CloudFormation template: JSON

Plain text

Perfect for version control

Can be validated

Anatomy of a CloudFormation template: JSON{

"AWSTemplateFormatVersion" : "2010-09-09",

"Description" : "AWS CloudFormation Sample Template EC2InstanceSample: **WARNING** This template an Amazon EC2 instances. You will be billed for the AWS resources used if you create a stack from this template.",

"Parameters" : {

"KeyName" : {

"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance",

"Type" : "String"

},

"Environment": {

"Type" : "String",

"Default" : ”Dev",

"AllowedValues" : [”Mgmt", "Dev", ”Staging", "Prod"],

"Description" : "Environment that the instances will run in.”

}

},

"Mappings" : {

"RegionMap" : {

"us-east-1" : { "AMI" : "ami-7f418316" },

"us-west-2" : { "AMI" : "ami-16fd7026" }

}

},

"Conditions" : {

”EnableEBSOptimized" : {"Fn::Equals" : [{"Ref" : " Environment

"}, ”Prod"]},

},

"Resources" : {

"Ec2Instance" : {

"Type" : "AWS::EC2::Instance",

"Properties" : {

"KeyName" : { "Ref" : "KeyName" },

"EbsOptimized " : {"Fn::If": [ " EnableEBSOptimized ", {“true”}, {“false”}]},

"ImageId" : { "Fn::FindInMap" : [ "RegionMap", { "Ref" : "AWS::Region" }, "AMI" ]},

"UserData" : { "Fn::Base64" : "80" }

}

}

},

"Outputs" : {

"InstanceId" : {

"Description" : "InstanceId of the newly created EC2 instance",

"Value" : { "Ref" : "Ec2Instance" }

},

"PublicDNS" : {

"Description" : "Public DNSName of the newly created EC2 instance",

"Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicDnsName" ] }

}

}

}

Anatomy of a CloudFormation template: JSON

Parameters

"Parameters" : {

"KeyName" : {

"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instance",

"Type" : "String"

},

"Environment": {

"Type" : "String",

"Default" : ”Dev",

"AllowedValues" : [”Mgmt", "Dev", ”Staging", "Prod"],

"Description" : "Environment that the instances will run in.”

}

},

Mappings

"Mappings" : {

"RegionMap" : {

"us-east-1" : { "AMI" : "ami-7f418316" },

"us-west-2" : { "AMI" : "ami-16fd7026" }

}

},

Conditionals

"Conditions" : {

”EnableEBSOptimized" : {"Fn::Equals" : [{"Ref" : " Environment "}, ”Prod"]},

},

Resources

"Resources" : {

"Ec2Instance" : {


"Properties" : {


"EbsOptimized " : {"Fn::If": [ " EnableEBSOptimized ", {“true”}, {“false”}]},


"UserData" : { "Fn::Base64" : "80" }

}

}

},

Outputs

Outputs" : {

"InstanceId" : {

"Description" : "InstanceId of the newly created EC2 instance",

"Value" : { "Ref" : "Ec2Instance" }

},

"PublicDNS" : {

"Description" : "Public DNSName of the newly created EC2 instance",

"Value" : { "Fn::GetAtt" : [ "Ec2Instance", "PublicDnsName" ] }

}

}

}

Headers

{

"AWSTemplateFormatVersion" : "2010-09-09",

"Description" : "AWS CloudFormation Sample Template EC2InstanceSample: **WARNING** This template an Amazon EC2 instances. You will be billed for the AWS resources used if you create a stack from this template.",

Description of what your stack does, contains, and so on

Provision time values that add structured flexibility and customization

Predefined conditional case statements

Conditional values set through evaluations of passed references

AWS resource definitions

Resulting attributes of stack resource creation

Headers

Parameters

Mappings

Conditionals

Resources

Outputs

Template components

Template example

Templates (in action):

"ImageId" : { "Fn::FindInMap" : [ "AWSRegionVirt2AMI", { "Ref" : "AWS::Region" }, {"Fn::FindInMap": ["AWSInstanceType2Virt", { "Ref" : "myInstanceType" }, "Virt"]} ]},

Template example


"ImageId" : { "Fn::FindInMap" : [ "AWSRegionVirt2AMI", { "Ref" : "AWS::Region" }, {"Fn::FindInMap": ["AWSInstanceType2Virt", { "Ref" : "myInstanceType" }, "Virt"]} ]},

“AWSRegionVirt2AMI” map

Template example


"ImageId" : { "Fn::FindInMap" : [ "AWSRegionVirt2AMI", { "Ref" : "AWS::Region" },


{"Fn::FindInMap": ["AWSInstanceType2Virt", { "Ref" : "myInstanceType" }, "Virt"]} ]},

“AWSInstanceType2Virt” map

Template example


"ImageId" : { "Fn::FindInMap" : [ "AWSRegionVirt2AMI", { "Ref" : "AWS::Region" },


{"Fn::FindInMap": ["AWSInstanceType2Virt",


{ "Ref" : "myInstanceType" }, "Virt"]} ]},

“myInstanceType” parameter

Template example


"ImageId" : { "Fn::FindInMap" : [ "AWSRegionVirt2AMI",


{"Fn::FindInMap": ["AWSInstanceType2Virt",


{ "Ref" : "myInstanceType" }, "Virt"]} ]},

“myInstanceType” parameter

{ "Ref" : "AWS::Region" },

AWS::Region pseudo parameter

Template example

"myInstanceType" : {

"Type" : "String",

"Default" : "t2.large",

"AllowedValues" : ["t2.micro", "t2.small", "t2.medium", "t2.large"],

"Description" : "Instance type for instances created, must be in the t2 family."

}

"AWSInstanceType2Virt": {

"t2.micro": {"Virt": "HVM"},

"t2.small": {"Virt": "HVM"},

"t2.medium": {"Virt": "HVM"},

"t2.large": {"Virt": "HVM"},

}

"AWSRegionVirt2AMI": {

"us-east-1": {

"PVM": "ami-50842d38",

"HVM": "ami-08842d60"

},

"us-west-2": {

"PVM": "ami-af86c69f",

"HVM": "ami-8786c6b7"

},

"us-west-1": {

"PVM": "ami-c7a8a182",

"HVM": "ami-cfa8a18a"

}

}

Parameters: Mappings: Mappings:

Bootstrapping applications and handling updates

"Resources" : {

"Ec2Instance" : {


"Properties" : {


"SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],


"UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[

"#!/bin/bash -ex","\n",

"yum -y install gcc-c++ make","\n",

"yum -y install mysql-devel sqlite-devel","\n",

"yum -y install ruby-rdoc rubygems ruby-mysql ruby-devel","\n",

"gem install --no-ri --no-rdoc rails","\n",

"gem install --no-ri --no-rdoc mysql","\n",

"gem install --no-ri --no-rdoc sqlite3","\n",

"rails new myapp","\n",

"cd myapp","\n",

"rails server -d","\n"]]}}

}

}

Option 1: Use Amazon EC2 UserData, which is available as a property of

AWS::EC2::Instance resources

cfn-init

cfn-hup

Bootstrapping applications and handling updates

Option 2: CloudFormation provides

helper scripts for deployment within

your EC2 instances

Metadata key—

AWS::CloudFormation::Init

The cfn-init helper script reads this

metadata key and installs the

packages listed in this key (for

example, httpd, mysql, and php); cfn-

init also retrieves and expands files

listed as sourcesEC2

CloudFormationcfn-signal

cfn-get-

metadata

Bootstrapping example

"Metadata": {

"AWS::CloudFormation::Init" : {

"config" : {

"packages" : {

},

"sources" : {

},

"commands" : {

},

"files" : {

},

"services" : {

},

"users" : {

},

"groups" : {

}

}

}

Use AWS::CloudFormation::Init with cfn-init to help bootstrap instances:

Bootstrapping example

“WebAppHost" : {


"Metadata" : {

"AWS:CloudFormation::Init" : {

"config" : {

"packages" : {

"yum" : {

"gcc" : [],

"gcc-c++" : [],

"make" : [],

"automake" : [],

Install packages with the native package management tool:

Manage a wide range of AWS services and resources

• Amazon EC2

• Amazon EC2 Container Service

• Amazon EC2 Container Registry

• Amazon EC2 Simple Systems Manager

• AWS Lambda (including event sources)

• AWS Elastic Beanstalk

• Auto Scaling (including Spot fleet)

• Amazon VPC and Managed NAT Gateway

• Elastic Load Balancing

• Amazon Route 53

• Amazon CloudFront

• AWS WAF

• Amazon S3

• Amazon RDS

• Amazon Redshift

• Amazon DynamoDB

• Amazon ElastiCache

• Amazon RDS (including Amazon Aurora)

• Amazon Elastic MapReduce

• Amazon Elasticsearch Service

• AWS Data Pipeline

• AWS Identity and Access Management (including

managed policies)

• AWS Directory Service (Amazon Simple AD) / Microsoft

Active Directory

• Amazon Kinesis

• Amazon SNS

• Amazon SQS

• AWS CloudTrail

• Amazon CloudWatch

• AWS Config

• AWS Key Management Service

• AWS OpsWorks

• AWS CodeDeploy

• AWS CodePipeline

• Amazon WorkSpaces

• Amazon GameLift

AWS resource support is always growing. See the most up-to-date list here.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-supported-resources.html

Template file

defining stack

• The entire infrastructure can

be represented in a

CloudFormation template

Many stacks and environments from one template

Template file

defining stack


be represented in a


• Use the version control

system of your choice to

store and track changes to

this template

Git

Perforce

SVN

…


Template file

defining stack


be represented in a


• Use the version control

system of your choice to

store and track changes to

this template

• Build out multiple

environments, such as for

development, test,

production, and even disaster

recovery, using the same

template

Git

Perforce

SVN

…

Dev

Test

Prod


Infrastructure as Code with CloudFormation

Versioning

You track changes within your code

Do it with your infrastructure:

• What is changing?

• Who made that change?

• When was it made?

• Why was it made?(Is it tied to a ticket or bug or project system?)

Testing your CloudFormation templates

Testing your template:

• Validate by using API or AWS Command Line Interface (CLI)

• $ aws cloudformation validate-template—confirm

CloudFormation syntax

• Use something like JSONLint (http://jsonlint.com/) to find

JSON issues like missing commas or brackets

• Throw this into your testing and/or continuous integration

pipelines

Visualizing your CloudFormation templates

• AWS

CloudFormation

Designer

• Visualize template

resources

• Modify template

with drag and drop

gestures

• Customize sample

templates

Deploying your CloudFormation templates

Deploy and update by using console, API, or CLI

OR

• aws cloudformation create-stack --stack-name

myteststack --template-body

file:////home//local//test//sampletemplate.json --

parameters

ParameterKey=string,ParameterValue=string

But what do we do once your

resources are provisioned and

running?

Your infrastructure needs ongoing management

• Updates or patches?

• New software?

• New configurations?

• New code deployments?

• Pool-specific changes?

• Environment-specific changes?

• Run commands across all hosts?

• Be on top of all running resources?

Ongoing management requires proper tooling

Some common challenges:

• Changing a vhost configuration on every web server across

multiple environments (development, staging, production)

• Installing a package on certain hosts to test out newer versions

• Changing the LDAP configuration on every running Amazon EC2

Linux host when the hosts exist across 25 different CloudFormation

templates

We need a tool to interact with

each host that we manage and

that makes it easier to

configure these hosts

AWS OpsWorks

• Configuration management service

for automating operational tasks

using Chef

• Model, control, and automate

applications of nearly any scale and

complexity

• Manage Linux and Microsoft

Windows environments

• Supports both AWS and on-

premises servers

• Launched in 2013

AWS OpsWorks concepts

A stack represents

the cloud

infrastructure and

applications that

you want to manage

together

A layer defines how

to set up and

configure a set of

instances and

related resources

You decide how to

scale: manually,

with 24/7 instances,

or automatically,

with load-based or

time-based

instances

Then deploy your

app to specific

instances and

customize the

deployment with

Chef recipes

AWS OpsWorks concepts: instance lifecycle

Set up Configure Deploy Undeploy Shut down

An agent on each instance understands a

set of commands that are triggered by

OpsWorks. The agent when triggered runs

Chef.

OpsWorks agent communication

1. The EC2 instance connects with the

OpsWorks service to send keepalive/

heartbeat and receive lifecycle events

2. OpsWorks sends a lifecycle event with a

pointer to the configuration JSON

(metadata, recipes) in an S3 bucket

3. The agent downloads configuration

JSON

4. The agent pulls cookbooks and other

build assets from your repository

5. The agent executes the recipe

6. The agent uploads the Chef log

7. The agent reports Chef run status

EC2

instanceOpsWorks

service

“Deploy App”

Your

repository,

for example

GitHub

How OpsWorks bootstraps EC2 instances

The EC2 instance is started by using an IAM role

• UserData passed with instance private key, OpsWorks public key

• The instance downloads and installs the OpsWorks agent

The agent connects to the instance service, gets run info

• Authenticates the instance using the instance’s IAM role

• Picks up configuration JSON from the OpsWorks instance queue

• Decrypts and verifies the message, runs Chef recipes

• Uploads Chef log, returns Chef run status

The agent then polls the instance service for more messages

AWS OpsWorks + Chef

OpsWorks uses Chef to configure the software on the

instance

OpsWorks provides many Chef Server functions to users

• Associates cookbooks with instances

• Dynamic metadata describes each registered node in the

infrastructure

Supports "push" command and control client runs

Supports community cookbooks

Working with Chef and OpsWorks

Similar to CloudFormation templates and application code:

• Mixture of JSON and a Ruby DSL

• Tools exist to do linting and syntax checking

• Versioning

• Built in cookbook versioning

• Some manual/processes scripted abilities

• But still can use source control for versioning

• Use with continuous integration systems like

CloudFormation templates and the rest of your code

Working with Chef and OpsWorks

Basics:

• Nodes

• Roles

• Cookbooks

• Recipes

• Attributes

• Data bags

• Environments

Host configuration management with Chef

package "ntp" do

action :install

end

service "ntpd" do

supports :status => true, :restart => true, :reload => true

action [ :enable, :start ]

end

cookbook_file "/etc/ntp.conf" do

source "ntp.conf"

owner "root"

group "root"

mode 00644

# Restart ntp.conf if /etc/ntp.conf changes

notifies :restart, resources(:service => "ntpd")

Endgroup "ganglia" do

gid 499end

user "ganglia" dohome "/var/lib/ganglia"shell "/sbin/nologin"uid 499gid "ganglia"

end

directory "/etc/ganglia" doaction :create

end

Examples:


template "/etc/ganglia/gmond.conf" do

source "gmond.conf.erb"

owner "root"

group "root"

mode 00644

notifies :restart, resources(:service => "gmond")

variables(

:gmetad_name => node[:ganglia][:gmetad_name],

:cluster_name => node[:ganglia][:cluster_name]

)

end

cron "all-gmetrics" do

command "for FILE in `ls /opt/bin/gmetric-*`; do command $FILE; done >/dev/null 2>&1"

end

Examples:

Custom JSON{

"opsworks": {

"data_bags": {

"myapp": {

"mysql": {

"username": "default-user",

"password": "default-pass"

}

}

}

}

}


Recipemything = data_bag_item("myapp", "mysql")

Chef::Log.info("username:

#{mything['username']}")

AWS OpsWorks

Deploying applications

Automates code deployments to any instance

Handles the complexity of updating your

applications

Use it to avoid downtime during application

deployment

Deploy to Amazon EC2 or on-premise servers,

in any language and on any operating system

Integrates with third-party tools and AWS

services

AWS CodeDeploy

AWS CodeDeploy concepts

Application

Revision #1

Revision #2

Revision #3

What to deploy?

Revision #1

How to deploy?

Instance

Instance

Instance

Deployment group

Auto Scaling group

Where to deploy?

How it works: package app with appspec.yml

version: 0.0

os: linux

files:

- source: /

destination: /var/www/html

permissions:

- object: /var/www/html

pattern: “*.html”

owner: root

group: root

mode: 755

hooks:

ApplicationStop:

- location: scripts/deregister_from_elb.sh

BeforeInstall:

- location: scripts/install_dependencies.sh

ApplicationStart:

- location: scripts/start_httpd.sh

ValidateService:

- location: scripts/test_site.sh

- location: scripts/register_with_elb.sh

How it works: package app with appspec.yml

version: 0.0

os: linux

files:

- source: /

destination: /var/www/html

• Send application files to one

directory and configuration files to

another

• Set specific permissions on specific

directories and files

• Remove or add instance to Elastic

Load Balancing

• Install dependency packages

• Start Apache

• Confirm successful deploy

• More!

permissions:- object: /var/www/html

pattern: “*.html”owner: rootgroup: rootmode: 755

hooks:ApplicationStop:- location: scripts/deregister_from_elb.sh

BeforeInstall:- location: scripts/install_dependencies.sh

ApplicationStart:- location: scripts/start_httpd.sh

ValidateService:- location: scripts/test_site.sh- location: scripts/register_with_elb.sh

How it works: Specify targets

Group instances by:

• Auto Scaling group

• Amazon EC2 tag

• On-premises tag

Development deployment group

AgentAgent Agent

Production deployment group

AgentAgent Agent

AgentAgent Agent

How it works: Deploy

• AWS CLI and SDKs

• AWS Management Console

• AWS CodePipeline and CI/CD partners

• Amazon S3, GitHub

aws deploy create-deployment \

--application-name MyApp \

--deployment-group-name TargetGroup \

--s3-location bucket=MyBucket,key=MyApp.zip

v2 v1 v1 v1 v1 v1 v1 v1

v2 v2 v1 v1 v1 v1 v1 v1

v2 v2 v2 v2 v1 v1 v1 v1

v2 v2 v2 v2 v2 v2 v2 v2

One at a timeMinimum healthy hosts = 99%

[Custom]Minimum healthy hosts = 75%

Half at a timeMinimum healthy hosts = 50%

All at onceMinimum healthy hosts = 0

Choose your deployment configuration

Summary

Summary

• Create, update, and manage AWS resources and their configuration

and properties with CloudFormation

• You can configure OpsWorks and CodeDeploy by using

CloudFormation

• Use OpsWorks for ongoing tweaks to software and configuration of

host-based applications and the operating system

• You can configure and deploy CodeDeploy’s agent with

OpsWorks

• Use CodeDeploy to deploy your applications and their configurations

Best practices

• Your CloudFormation templates and Chef cookbooks should

go in separate repositories

• Include the appspec.yml file and related scripts in your

application’s code repositories

• Every commit should cause an execution of your continuous

delivery pipeline to lint, validate, and/or test

• Use each related service’s CLI, console, and APIs to update or

deploy as necessary



AWS CloudFormation

AWS OpsWorks

AWS CodeDeploy

Amazon Virtual Private

Cloud (Amazon VPC)

Amazon Elastic Compute

Cloud (Amazon EC2)

AWS Identity and Access

Management (IAM)

Amazon Relational Database

Service (Amazon RDS)

Amazon Simple Storage

Service (Amazon S3)

AWS CodePipeline

…

Microsoft Windows Registry

Linux networking

OpenSSH

LDAP

Active Directory domain

registration

Centralized logging

System metrics

Deployment agents

Host monitoring

…

Application dependencies

Application configuration

Service registration

Management scripts

Database credentials

…

allOfThis == $Code

Learn more

• AWS CloudFormation

• https://aws.amazon.com/cloudformation/

• https://aws.amazon.com/documentation/cloudformation/

• https://aws.amazon.com/cloudformation/aws-cloudformation-templates/

• AWS OpsWorks

• https://aws.amazon.com/opsworks/

• https://aws.amazon.com/documentation/opsworks/

• https://github.com/aws/opsworks-cookbooks

• AWS CodeDeploy

• https://aws.amazon.com/codedeploy/

• https://aws.amazon.com/documentation/codedeploy/

• https://github.com/awslabs/aws-codedeploy-samples

Thank you!

devops on aws: deep dive on infrastructure as code

Engineering