automating management of amazon ec2 instances with auto scaling - march 2017 aws online tech talks

©2017,AmazonWebServices,Inc.oritsAffiliates.Allrightsreserved.

Michael Hanisch, AWS Solutions Architecture

3/23/2017

Automating Management of Amazon EC2 Instances with Auto Scaling

AutoScalinggroupAutoScalinggroup

AutoScaling

Dynamic Scaling

ELB

EC2Instances

ELB

CPUUtilization

EC2Instances

Fleet Management

Myth Fact

MyApplicationDoesn’tNeedScaling,SoIDon’tBenefitFromAutoScaling

It’sHardToUse

MyInstancesareStateful orUnique;Ican’tuseAutoScaling

ItWorksWellwithStateful Instances

YouCanGetStartedinMinutes

ItMonitorsandHealsInstances

Is Fleet Management For You?

“I’vegotinstancesservingabusiness-impactingapplication”

“Ifmyinstancesbecomeunhealthy,I’dlikethemreplacedautomatically”

“Iwouldlikemyinstancesdistributedtomaximizeresilience”

Auto Scaling – The Basics

instanceAvailability Zone a

instance instanceAvailability Zone bAvailability Zone a

instance instanceAvailability Zone bAvailability Zone a

Auto Scaling group

Minimum#=2 Maximum#=2

Desired#ofinstances=2

instance instanceAvailability Zone bAvailability Zone a

Elastic LoadBalancing

instance instanceAvailability Zone bAvailability Zone a

Elastic LoadBalancing

Auto Scaling group

Minimum#=2 Maximum#=2

Desired#ofinstances=2

Auto Scaling Groups

- Always keep minimum number of instances running- Launch or terminate instances to meet desired capacity- Never start more than maximum number of instances- Keeps capacity balanced across AZs

Launch Configurations

Determine what is going to be launched:- EC2 instance type & size- Amazon Machine Image (AMI)- Security groups, SSH keys, IAM instance profile- User data…

Bootstrapping

Installation & setup needs to be fully automated:- Use Amazon Machine Image (AMI) with all required

configuration & software (“golden image”)- Base AMI + install code & configuration as needed

- Via Userdata + scripts- Via Chef/Puppet/Ansible/…- Using AWS CodeDeploy- Using Amazon EC2 Systems Manager

…

Bootstrapping

#!/bin/bash# Install updatessudo yum update -y;# Install Amazon EC2 Systems Manager Agentcd /tmp;curl https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm -o amazon-ssm-agent.rpm && yum install -y amazon-ssm-agent.rpm;

Bootstrapping

#!/bin/bash# Install updatessudo yum update -y;# Install AWS CodeDeploy agentcd /home/ec2-user; curl https://aws-codedeploy-us-east-1.s3.amazonaws.com/latest/install \ -o install && chmod +x ./install && sudo ./install auto && sudo service codedeploy-agent start;

Monitoring

Auto Scaling gives you access to new metrics in Amazon CloudWatch:

- group-level metrics like number of running instances- aggregate metrics like average CPU utilization for all

instances in the group

Monitoring

Auto Scaling Concepts

LaunchConfiguration• AutoScalinggroupsusea

launchconfiguration tolaunchEC2instances.

• ProvidesinformationabouttheAMIandEC2instancetypes/size

ScalingPlan• AscalingplantellsAuto

Scalingwhenandhowtoscale.

• Createascalingplanbasedontheoccurrenceofspecifiedconditions(dynamicscaling)orcreateaplanbasedonaspecificschedule.

AutoScalingGroups• EC2instancesare

managedbyAutoScalinggroups.

• CreateAutoScalinggroupsbydefiningtheminimum,maximum,and,optionally,thedesirednumberofrunningEC2instances.

Termination Policies

Determine which instances are terminated first:- Longest running- Oldest launch configuration- Closest to full billing hour

But: rebalancing of capacity across AZs takes precedence!

Scaling Plans

Determine when the Auto Scaling group will scale in or out:

desired capacity > current capacity: launch instancesdesired capacity < current capacity: terminate instances

Scaling Plans

- Default: ensure current capacity of healthy instances remains within boundaries (never less than minimum)

- ‘Manual scaling’: modify desired capacity (via API, console, CLI) to trigger a scaling event

- Scheduled: scale in / out based on timed events- Dynamic scaling: scale based on CloudWatch metrics

Getting Started

What happens next?

Availability & Reliability

Auto Scaling Groups

- Always keep minimum number of instances running- Launch or terminate instances to meet desired capacity- Never start more than maximum number of instances- Keeps capacity balanced across AZs

Auto Scaling Groups

- Always keep minimum number of instances running- Launch or terminate instances to meet desired capacity- Never start more than maximum number of instances- Keeps capacity balanced across AZs- Replace unhealthy instances

Auto Scaling Groups

- Always keep minimum number of instances running- Launch or terminate instances to meet desired capacity- Never start more than maximum number of instances- Keeps capacity balanced across AZs- Replace unhealthy instances

Health Checks

- Performed periodically- Instances are marked as “Unhealthy” when checks fail- Unhealthy instances are terminated and replaced

(if new number of instances < minimum or < desired capacity)

Different Kinds of Health Checks

- EC2 instance status: Instance is unhealthy when instance state != ‘running’ or system health check == ‘impaired’

- ELB health checks:instance is unhealthy when ELB health check results in “OutOfService” (or EC2 health check failed)

- Manual: mark individual instances as ‘unhealthy’Instance unhealthy when marked as such or EC2 health check failed. Use to integrate with external monitoring systems.

instance instanceinstance instance

Auto Scaling group

Minimum=2 Maximum=6

Desired#ofinstances=6

instanceinstanceAvailability Zone bAvailability Zone a

Elastic LoadBalancing

Unhealthy Instances Get Replaced…

instance instanceinstance instance

Auto Scaling group

Minimum=2 Maximum=6

Desired#ofinstances=6

instanceinstanceAvailability Zone bAvailability Zone a

Elastic LoadBalancing

Unhealthy Instances Get Replaced…

instance instanceinstance instance

Auto Scaling group

Minimum=2 Maximum=6

Desired#ofinstances=6

instanceinstanceAvailability Zone bAvailability Zone a

Elastic LoadBalancing

Unhealthy Instances Get Replaced…

instance instanceinstance instance

Auto Scaling group

Minimum=2 Maximum=6

Desired#ofinstances=6

instanceinstanceAvailability Zone bAvailability Zone a

Elastic LoadBalancing

…In a Different AZ if Necessary

instanceinstance instanceinstance

Auto Scaling group

Minimum=2 Maximum=6

Desired#ofinstances=6

instanceAvailability Zone bAvailability Zone a

instance

Elastic LoadBalancing

Rebalancing Capacity

instanceinstance instanceinstance

Auto Scaling group

Minimum=2 Maximum=6

Desired#ofinstances=6

instanceAvailability Zone bAvailability Zone a

instance

Elastic LoadBalancing

instance instance instance

What happens when an instance is terminated?

The Instance Lifecycle

Instance Lifecycle

Scale OutEvent

Instancelaunching:Pending

InService

TerminatingTerminated

Scale InEvent

Health checkfailed

Instance Lifecycle

AddanInstance

Instancelaunching:Pending

InService

TerminatingTerminated

RemoveanInstance

Health checkfailed

Instance Lifecycle

Instancelaunching:Pending

InService

TerminatingTerminated

Entering Standby…

Standby

How can we influence the instance lifecycle?

Why? – Common Use Cases

• Assign Elastic IP address or ENI on launch• Register new instances with DNS, external monitoring

systems, firewalls, load balancers, …• Load existing state from S3 or other system• Pull down log files before instance is terminated• Investigate issues with an instance before terminating it• Persist instance state to external system• …

Lifecycle Hooks & Notifications

Instance Lifecycle Notifications

AddanInstance

Instancelaunching:Pending

InService

TerminatingTerminated

RemoveanInstance

Health checkfailed

EC2InstanceTerminateSuccessful

EC2InstanceLaunchSuccessful

EC2InstanceLaunchUnsuccessful

Instance Lifecycle Notifications

• Notifications get sent after a state transition.• Rely on notifications to react to changes that happened.• Available via Amazon Simple Notification Service and

Amazon CloudWatch Events.• Prefer CloudWatch Events due to ease of use and

extended feature set!

Sample Notification

Service:AutoScalingTime:2017-03-23T21:53:43.989ZRequestId:52e21eba-718a-43a7-81a8-3b379054cba6LifecycleActionToken:979c0f97-80c5-44bd-a2b6-5a8aae339f35AccountId:XXXXXXXXXAutoScalingGroupName:demo-asgLifecycleHookName:do-something-on-launchEC2InstanceId:i-XXXXXLifecycleTransition:autoscaling:EC2_INSTANCE_LAUNCHINGNotificationMetadata:null

Lifecycle Hooks

Instance Lifecycle Hooks

• When Lifecycle Hooks are defined, instances enter special “WAIT” states during state transitions.

• Allows you to react to lifecycle events & impact the state• WAIT states bring their own notifications, too.

Instance Lifecycle Hooks

Instancelaunching:Pending

InService

TerminatingTerminated

Pending:WAIT

Terminating:WAIT

InvokeHook(s)

Lifecycle Hooks- Executed before taking a new

instance into service / terminating it

- Put instances into a WAIT state while work can happen

Auto Scaling Notifications- Notifications get sent after an

instance has entered “InService” or “Terminated” state, respectively

- Cannot influence or stop a transition

Let’s see this in action…

TODO:Tasks? Secondpartofdemo:showhowtosetup

CWEevent+ruletoinvokeEC2SystemsManagerRunCommand directlytosavewebserverlogfiles.(Pointoutthatthisisacommonexamplebutthatwe’drecommendtouseaproperloggingsolution,e.g.CloudWatch Logs,instead)

TODO:Tasks?

Addarchitecturediagramofthesolutionwejustsetup– ordothisbefore?

instance instanceAvailability Zone bAvailability Zone a

Auto Scaling group

AutoScaling

Lambdafunction

CloudWatch Events AmazonEC2SystemsManager

1. Eventfires,triggersRule

2.RuleinvokesLambda 3.AsksEC2SSM

ToRunCommand

4.InvokeRunCommandonterminatinginstance

5.CommanduploadslogstoS3

AutoScaling6.CompleteHook

How Do I Write a Lifecycle Hook?

1. Code the lifecycle hook’s action2. Create new Rule in CloudWatch Events3. Associate the lifecycle hook with the Auto Scaling group

Writing a Lifecycle Hook

1. Code the lifecycle hook’s action1. Extract instanceID, auto scaling group, other params.2. Do stuff…

• Beware of timeouts!• Send “heartbeats” if you need more time

3. Call CompleteLifecycleAction to signal that you’re done!

Writing a Lifecycle Hook

1. Code the lifecycle hook’s action• AWS Lambda function• Amazon EC2 Systems Manager RunCommand• Any Code that Consumes Kinesis Streams/ SQS/ SNS

Writing a Lifecycle Hook

{ "schemaVersion": "1.2",

"description": "Backup logs to S3", "parameters": {},

"runtimeConfig": {

"aws:runShellScript": {

"properties": [ {

"id": "0.aws:runShellScript",

"runCommand": [ "",

"ASGNAME='demo-asg'",

"LIFECYCLEHOOKNAME='demo-asg-backup-hook'",

"INSTANCEID=$(curl http://169.254.169.254/latest/meta-data/instance-id)",

"REGION=$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone)",

"REGION=${REGION::-1}",

"HOOKRESULT='CONTINUE’”,

[…]

Writing a Lifecycle Hook

[…]

aws s3 cp /tmp/${INSTANCEID}.tar s3://${S3BUCKET}/${INSTANCEID}/ &> /tmp/backup",

" MESSAGE=$(cat /tmp/backup)",

"fi",

"",

"aws autoscaling complete-lifecycle-action

--lifecycle-hook-name ${LIFECYCLEHOOKNAME}

--auto-scaling-group-name ${ASGNAME}

--lifecycle-action-result ${HOOKRESULT}

--instance-id ${INSTANCEID} --region ${REGION}”

]

} }

}

}

Writing a Lifecycle Hook

2. Create new Rule in CloudWatch Events

Writing a Lifecycle Hook2. Create new Rule in CloudWatch Events

Writing a Lifecycle Hook

aws autoscaling put-lifecycle-hook \

--auto-scaling-group-name demo-asg \

--lifecycle-hook-name demo-hook-terminate \

--lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING

3. Associate the lifecycle hook with the Auto Scaling group

Dealing with StatefulApplications

Dealing With Stateful Applications

- While ”InService”:- Persist state to EBS volume on a regular basis- Tag with InstanceId, application name

- On “Instance-terminating Lifecycle Action”:- Detach EBS volume with state information- Remove InstanceId tag, keep application name tag

- On “Instance-launch Lifecycle Action” event:- Find & Attach EBS volume tagged w/ application name- Tag with InstanceId, Resume Application

Conclusion

Fact

ItWorksWellwithStateful Instances

YouCanGetStartedinMinutes

ItMonitorsandHealsInstances

• Direct Integration with CloudWatch• Instance Replacement• AZ Rebalancing

• Options for Easy Bootstrapping• Start off with Existing Instances

• Lots of Control via Lifecycle Hooks• Keep Track with Notifications

Questions?

Questions?

https://aws.amazon.com/blogs/compute/fleet-management-made-easy-with-auto-scaling/

http://docs.aws.amazon.com/autoscaling/latest/userguide/WhatIsAutoScaling.html

https://aws.amazon.com/autoscaling/getting-started/

Thankyou!