aws re:invent 2016: infrastructure continuous delivery using aws cloudformation (dev313)
TRANSCRIPT
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dominic Divakaruni, Anil Kumar - AWS CloudFormation
11/30/2016
Infrastructure Continuous Delivery using
AWS CloudFormation
What to expect from this session
• We’ll show you how to:
• Architect your infrastructure using AWS CloudFormation
• Use AWS CloudFormation to set up AWS CodePipeline
pipelines
• Continuously deliver changes to stacks as you make
changes to your templates
• Demo
Lets look at release
processes
https://www.flickr.com/photos/jurvetson/5201796697/
• Integration
tests with
other systems
• Load testing
• UI tests
• Penetration
testing
Release processes have four major phases
Source Build Test Production
• Check-in
source code
such as .java
files.
• Peer review
new code
• Compile code
• Unit tests
• Style checkers
• Code metrics
• Create
container
images
• Deployment
to production
environments
Release processes levels
Source Build Test Production
Continuous integration
Continuous delivery
Continuous deployment
Continuous deployment
Release Processes levels
Source Build Test Production
Continuous integration
Our focus today
Continuous delivery
What about continuous delivery tools and
processes for infrastructure?
What do we need for infrastructure continuous
delivery?
• A way to treat infrastructure as code.
• Tools to manage the workflow that creates and updates
infrastructure resources.
• Tools to properly test and inspect your changes for
defects and potential issues
What do we need for infrastructure continuous
delivery?
Infrastructure as code
A practice in which infrastructure
is provisioned and managed using
code and software development
techniques, such as version
control and continuous
integration.
Workflow
Build, test, and deploy your code
every time there is a code
change, based on the release
process models you define,
enabling you to rapidly and
reliably deliver changes.
Infrastructure as Code
We need
Workflow
AWS CodePipelineAWS CloudFormation
AWS CloudFormation
• Create templates of your infrastructure
• Version control /code review /update
templates like code
• CloudFormation provisions AWS resources
based on dependency needs
• Integrates with development, CI/CD,
management tools
• No additional charge to use
Author templates in JSON or YAML
Use Change Sets to preview your changes
Continuous delivery workflows for stacks
Support for AWS Serverless App Model
Enable cross-stack references with Exports
Key new features
YAML
Author CloudFormation templates in JSON or YAML.
Syntax enhancements
JSON
YAML with enhancements
! function short form
Fn::Sub function
substitutes variables
Preview the set of actions CloudFormation will take on your
behalf before you create or update stacks.
CloudFormation Change Sets
Change Sets show you what resources will be created, updated
or replaced. This ensures that only expected operations are
executed.
Cross Stack References (Exports)
Network Stack
Outputs:VPCDescription: reference VPCValue: !Ref VPCExport:
Name: ProdVPC
App Stack
Resources:myTargetGroup:Type: AWS::ELBV2::TargetGroupProperties:
VpcId: Fn::ImportValue: ProdVPC
• Allows you to share information between independent stacks.
• Export a stacks output values. Other stacks in the same account and region
can import the exported values.
Nested stacks
Application
Resources:NetworkResources:
Type: AWS::CloudFormation::Stack
Network Resources
Resources: MyVPC
Type: AWS::EC2::VPC
ECS Service
Resources:MyService:
Type: AWS::ECS::Service
• Create a stack composed of multiple
templates.
• Compose and re-use templates with
frequently used resources
Considerations for Exports and Nested Stacks
Nested Stacks Cross Stack References
Recommended
uses cases
Advantages
Considerations
• Template reuse
• Use multiple templates but
manage as single stack
• Sharing common resources
• Allows for independent stacks based
on resource lifecycle or ownership.
• Convenient management.
One stack manages all
resources and nested stacks.
• Creation order and
dependencies are managed
• Separation of concern
• Share databases and VPCs
• Lets you limit blast radius with
safeguards
• Updates and rollbacks have
a wide surface area
• Reusing templates that have
custom resource names
• Replacing updates requires
changes to the importing stacks to
execute.
• Does not manage creation order
Use case
Let’s examine a sample application
Deconstruct the application into the necessary AWS resources
Create CloudFormation templates based your management needs
Model your continuous delivery pipeline
Continuously deliver infrastrucure changes as you iterate on your architecture
Use CloudFormation to model, provision and manage changes to your pipeline
Microservices application based on Amazon
ECS
Two interconnecting microservices deployed as ECS
services (website-service and product-service).
The application runs on a highly available ECS cluster
deployed across multiple availability zones with auto
scaling
Available at github.com/awslabs/ecs-refarch-cloudformation
Reference architecture
Public Subnet
Private Subnet
Availability Zone
Internet
Gateway
Public Subnet
Private Subnet
Availability Zone
Application
Load Balancer
NAT GatewayNAT Gateway
ECS Cluster
CloudWatch Logs (Container Logs)
ECS Host ECS Host ECS Host ECS HostAuto Scaling Group
github.com/awslabs/ecs-refarch-cloudformation
Decompose into AWS resource types
NAT Gateway
Elastic IP
Default Public Route
Public Subnet 1Private Subnet 1
Default PrivateRoute
PrivateRouteTable
NAT Gateway
Elastic IP
Public Subnet 2Private Subnet 2
Default PrivateRoute
PrivateRouteTable
Ava
ilab
ilit
y Zo
ne
1A
vaila
bil
ity
Zon
e 2
VPCInternetGateway
Public Route Table
Load BalancerSecurity Group
ECS Host Security Group
Application Load Balancer
Load BalancerListener
Load BalancerDefault TargetGroup
ECS Cluster
Auto ScalingGroup
Auto ScalingLaunch Configuration
ECS (IAM) Role
IAM Instance Profile
ECS Service
ECS Task Definition
CloudWatchLog Group
TargetGroup
Listener Rule
Service Role
ECS Service
ECS Task Definition
CloudWatchLog Group
TargetGroup
Listener Rule
Service Role
Network SecurityLoad
BalancingFront End
ServiceECS
ClusterBack EndService
Build CloudFormation templates based on this
logical grouping
Template Description
Network VPC, AZs, subnets, routing, NAT and internet gateways
Security groups Security groups for the application
Load balancers ALBs that are deployed to the public subnets
ECS cluster ECS cluster deployed to private subnets
Back end service ECS service and task definition for the back end app
Front end service ECS service and task definition for the webpage
Setup your template to flow configuration to
each other
NetworkTemplate
SecurityTemplate
Load BalancingTemplate
Front End svcTemplate
ECS ClusterTemplate
Back End SvcTemplate
Outputs
Load BalancerListener
Load BalancerDNS Name
ECS Cluster
Load BalancerSecurity Group
ECS Host Security Group
VPC
Public Subnets
PrivateSubnets
Network
Security
Load Balancing
ECS Cluster
Front End
Back End
with nested stacks
Use these templates to build your stacks
with cross-stack references
Network
Security
Load
Balancing
ECS Cluster
Front End
Back End
Parent Template Microservices
Stack
Nested
templates
Templates Individual Stacks
Applying continuous delivery
Applying continuous delivery for your
infrastructure
Continuous delivery service for fast and reliable
application and infrastructure updates
Builds, tests and deploys your code each time there is a
code change.
Built in actions for AWS CloudFormationAWS
CodePipeline
How does this align with release phases?
Source Test Deploy
Source stage for
CloudFormation
templates can be
AWS CodeCommit,
S3 or GitHub
Use CloudFormation
Change Sets to verify
deployments prior to
execution
Create, update or
delete Stacks or
Change Sets.
Model your pipelines
Iterate more often on your application
and infrastructure code
Launch new versions in Dev and
promote to prod
Manage your network resources
separately per its own cadence.
Maintain separate, mirror sandbox and
production network environments.
Production
VPC, Security Groups,
Load Balancing
Sandbox
VPC, Security Groups,
Load Balancing
Production
ECS Cluster, Application
Front & Back Ends
Dev
ECS Cluster, Application
Front & Back Ends
Application PipelineNetwork Resources Pipeline
Pipeline for network resources
Source repo
Networking resources for
Sanbox/Dev environments
Individual stacks. Ordered to
account for dependencies
Change Sets to preview changes to prod
Manual approval before you
changes are applied to prod
Apply Changes to Prod
1
2
3
4
5
Pipeline for your application
Pipeline triggered as soon as new
versions are posted
Run your tests and clean up
your dev environment when
done, so you aren’t charged
for the instances you don’t
use.
Review to ensure resource modification
or replacement is what you expect
Continuously deliver changes to Prod
1
2
3
4
Create and manage your pipeline using
CloudFormation
Pipeline Artifact Store
S3 BucketPipeline Notifications
SNS Email NotificationsPipeline IAM Roles
CloudFormation template to setup your pipeline
Could be provisioned
in a separate stack
with IAM resources –
with cross-stack refs
Create and manage your pipeline using
CloudFormation
Choose ‘deploy’ action with CloudFormation
as the provider
CloudFormation has enabled several action modes
– REPLACE_ON_FAILURE creates a new stack if one
doesn’t exist, updates it if it does or replaces it if its
in a failed state
You can use template configuration files or
specify parameter overrides within the template
that defines your pipeline
Stage
Action
Action
configuration
Name of your CloudFormation template
Template walkthrough – model your pipeline
DEMO!
FIN, ACK
We’ve seen how to compose and continuously deliver your
infrastructure as code on our software release process:
• Different ways to decompose your infrastructure into
templates and stacks
• Create and provision your continuous delivery pipeline
for your infrastructure
• Deliver changes to your environments with speed and
quality.
re:Invent 2016 sessions on Continuous Delivery:
• DEV201 - DevOps on AWS: Accelerating Software Delivery with the AWS Developer
Tools
• CON302 - Development Workflow with Docker and Amazon ECS
• DEV403 - DevOps on AWS: Advanced Continuous Delivery Techniques
Resources to learn more:
• Continuous delivery: https://aws.amazon.com/devops/continuous-delivery/
• Continuous delivery for CloudFormation stacks -
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-
delivery-codepipeline.html
• CodePipeline - https://aws.amazon.com/documentation/codepipeline/
But wait, there’s more!
Thank you!
Remember to complete
your evaluations!