real life aws_user_data_auto_scale
DESCRIPTION
Real Life AWS usages of user-data and AutoScale.TRANSCRIPT
![Page 1: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/1.jpg)
Real Life AWS: user-data & AutoScale
Adam NeumannSolutions Architect at IndexMedia
&
![Page 2: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/2.jpg)
Part 1: user-data
1.What is user-data?2.A problem3.user-data in the solution4.Further work
![Page 3: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/3.jpg)
What is user-data?
• user specified instance attribute• accessible via HTTP and curl/wget• usage wrapped by cloud-init• 'runs' the user data once on boot• great for configuring instances
![Page 4: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/4.jpg)
The concept
• any script, or init-cloud format• #! /bin/bash etc etc• cmd line arg or via UI:
![Page 5: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/5.jpg)
A Problem
• too hard to spin up new instances• complex, undocumented, manual• fab, bash, and misc scripts• slow: 30 min per instance• first: consolidate, then: user-data
![Page 6: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/6.jpg)
IndexMedia example 1
• small user-data script, bootstrap• load big script from S3• setup apache, hadoop, virtualenv,...• 16 mins, vanilla ubuntu to serving• automated: yes, optimal: no• could be improved...
![Page 7: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/7.jpg)
IndexMedia example 2
• split script into 2 scripts :)• 1: heavy, general, static = 15 min• runs on vanilla ubuntu, creates AMI• 2: light, instance specific = 90 secs• runs on AMI• from 30 min manual to 90 sec auto
![Page 8: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/8.jpg)
Further work
• investigate cloud-init format• include Puppet/Chef (first, learn
Puppet/Chef...)• integrate into CI process
(environment becomes versioned, deployable, etc)
![Page 9: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/9.jpg)
user-data summary
• great way to configure instances• UI and cmd line accessible• enables automated+repeatable
deployment• reduced deployment time, 30 mins to
90 secs
![Page 10: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/10.jpg)
Part 2: AutoScale
1.What is AutoScale2.Our (common) problem3.Show me the examples!4.Learnings
![Page 11: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/11.jpg)
Part 2: AutoScale
• cmd line only = no UI = hidden = unknown = scary
• basically:1.launching & terminating instances–within some bounds–based upon metrics• a second order system :)
![Page 12: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/12.jpg)
AutoScale concepts
1. Launch Configuration
individual: instance type, AMI, user-data, etc
2. AutoScale Group
collective: ideal/max/min, AZs
3. Policy
how: a reaction, increase, decrease
4. Alerts
when: an event, high/low load, CW metrics
![Page 13: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/13.jpg)
Our (common) problem
• maintain consistent experience for users, ie. Pip load time
• turn on servers when load is high• bootstrapped, must turn off servers
when load is low!• enter AutoScale...
![Page 14: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/14.jpg)
Solution = AutoScale
• servers start/stop based upon current system state
• spending on IT matches user demand exactly!
• 4 commands to set it up...
![Page 15: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/15.jpg)
1/4: Launch Configuration
as-create-launch-config gorilla_config_7d2 \--image-id ami-XXXXXXX \ --instance-type t1.micro \--group "gorilla-live" --key indexmedia-live \--user-data-file /path/to/bootstrap.sh
![Page 16: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/16.jpg)
2/4: Group
as-create-auto-scaling-group as_group \--launch-configuration gorilla_config_7d2 \--availability-zones us-west-1a us-west-1b \--min-size 10 --max-size 20 \--load-balancers my_balancer \--health-check-type EC2 \ --grace-period 120
![Page 17: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/17.jpg)
3/4: Policy
as-put-scaling-policy gorilla_scaleup_policy \--auto-scaling-group as_group \ --type ChangeInCapacity \--adjustment=1
![Page 18: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/18.jpg)
4/4: Alarm
mon-put-metric-alarm "gorilla high load" \ --dimensions "AutoScalingGroupName=as_group" \ --namespace "AWS/EC2" \ --statistic Average --metric-name CPUUtilization \ --comparison-operator GreaterThanThreshold \ --threshold 80 \ --evaluation-periods 5 --period 60 \ --alarm-actions ${POLICY-ARN}
![Page 19: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/19.jpg)
... and done! Learnings
• 4 steps (well 6...) = elastic system• feedback = second order system• avoid bad harmonics!• shorter instance boot time == quicker
system response time• validate alarms & behaviour w/ load
tests on AWS
![Page 20: Real life aws_user_data_auto_scale](https://reader036.vdocument.in/reader036/viewer/2022081413/54849744b47959ce0c8b4c21/html5/thumbnails/20.jpg)
AutoScale summary
• cmd line only; just jump in!• 4 distinct concepts, one per cmd• quick boot = shorter grace period =
quicker scaling up (enter user-data...)• key to our multi-AZ, HA elastic
cluster, problem solved!