creating a culture of cost management in your organization
TRANSCRIPT
![Page 1: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/1.jpg)
Creating A Culture Of Cost
Management In Your Organization
Toban Zolman
VP of Product Development
Cloudability
![Page 2: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/2.jpg)
Cloud Infrastructure Analytics
![Page 3: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/3.jpg)
Why should I care about Cost
Management?
![Page 4: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/4.jpg)
![Page 5: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/5.jpg)
NEW WAY: Many little decisions every day by many different
people
![Page 6: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/6.jpg)
Why does it need to be cultural?
![Page 7: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/7.jpg)
The cloud is no longer one person’s
responsibility…
Engineers Finance Operations Capacity Execs
![Page 8: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/8.jpg)
![Page 9: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/9.jpg)
How do you build the culture?
• Put data in the hands of the people
• Enact policies and evangelize best practices
• Incentivize good behavior
![Page 10: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/10.jpg)
What happens when it works?
• There are no surprises
• Everyone is on the same page
• Make budgeting easier
• Better decisions are made from engineer to CEO
• Save money
![Page 11: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/11.jpg)
0
45
90
135
180
225
Dev/T
est
PO
C
App #
1
App #
3
App #
2
Exec team starts
really caringL
oa
d t
est
A story of growth
![Page 12: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/12.jpg)
Visibility
Allocation Efficiency
Savings
Unit Cost
![Page 13: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/13.jpg)
Visibility
Allocation Efficiency
Savings
Unit Cost
![Page 14: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/14.jpg)
Visibility
Allocation Efficiency
Savings
Unit Cost
I. Cost visibility
![Page 15: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/15.jpg)
Two types of people
Incur costs Watch costs
![Page 16: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/16.jpg)
Two types of people
Incur costs Watch costs
![Page 17: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/17.jpg)
Two types of people
Incur costs Watch costs
![Page 18: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/18.jpg)
Two types of people
Incur costs Watch costs
![Page 19: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/19.jpg)
Two types of people
Manage costs Watch costs
![Page 20: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/20.jpg)
Tips for cost visibility
• Get each stakeholder the spending
fundamentals daily
• Let each team see other teams’ spending habits
• Create broadly available dashboards
![Page 21: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/21.jpg)
![Page 22: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/22.jpg)
Visibility
Allocation Efficiency
Savings
Unit Cost
II. Allocation
![Page 23: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/23.jpg)
First, get everyone involved
๏ Multiple major business units
๏ Lots of products or cost centers
๏ Thousands of shared AWS
resources
๏ Distributed teams using AWS
![Page 24: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/24.jpg)
Tools for splitting up resources
• Tags are highly flexible, but 100% coverage is difficult due to compliance
• Linked accounts offer clean chargeback but limit reporting options
![Page 25: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/25.jpg)
![Page 26: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/26.jpg)
Pro tips: Allocating Costs
• Get consensus on the taxonomy (but let Finance drive)
• Define 2-3 mandatory tags like “project” or “environment”
• Consider a “tag or terminate” rule to enforce compliance
![Page 27: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/27.jpg)
Visibility
Allocation Efficiency
Savings
Unit Cost
III. Efficiency
![Page 28: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/28.jpg)
168hours in a
week
![Page 29: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/29.jpg)
168hours in a
week
nights &
weekend
s
108
![Page 30: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/30.jpg)
Don’t run the cloud like a data center:
65% of the hours in a month are
nights and weekends
![Page 31: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/31.jpg)
Three easy wins for Amazon EC2 efficiency…
![Page 32: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/32.jpg)
Turn off underutilized instances
Low CPU, low bandwidth, low disk I/O, >1 day old | This example: Save $1,682 per week
![Page 33: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/33.jpg)
Schedule Dev/Test downtime
![Page 34: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/34.jpg)
Find legacy instances that could be upgraded
1. Newer instance families are faster/cheaper but have smaller SSDs
2. Look for instances that have attached Amazon Elastic Block Store
(EBS) and low ephemeral disk access
3. Move M1 to M3 to save $0.03-$0.10 per hour
Source: http://aws.amazon.com/ec2/pricing/
![Page 35: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/35.jpg)
Get alerts when thresholds are crossed
![Page 36: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/36.jpg)
Tips for encouraging efficient behavior
1. Automate weekly waste reporting for each team
2. Gamify cleanup by creating a visible leaderboard
1. Do a monthly, company-wide waste review
![Page 37: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/37.jpg)
Visibility
Allocation Efficiency
Savings
Unit Cost
IV. Savings
![Page 38: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/38.jpg)
Spot vs Reserved Instances?
Reservations require no engineering,
enabling finance to autonomously
reduce costs
![Page 39: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/39.jpg)
Buy Reserved Instances based on current hourly data
Don’t wait to buy reservations until you’ve completed the perfect analysis— you’ll waste
more money than you save
![Page 40: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/40.jpg)
![Page 41: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/41.jpg)
Aligning RIs to infrastructure is more important
than ever because RIs are all sunk cost
![Page 42: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/42.jpg)
![Page 43: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/43.jpg)
![Page 44: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/44.jpg)
Tips for reservation efficiency
1. Appoint a person to own reservation purchases– often
a tech-minded finance person
2. Start with an immediate small and uncontroversial buy
1. Buy iteratively on a schedule that you hold sacred
![Page 45: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/45.jpg)
Visibility
Allocation Efficiency
Savings
Unit Cost
V. Unit Cost
![Page 46: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/46.jpg)
Why does unit cost matter?
0
45
90
135
180
225
De
v/T
est
PO
C
App #
1
App #
3
App #
2
Exec team starts
really caring
Hic
cup
![Page 47: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/47.jpg)
An increasing bill isn’t necessarily a bad
thing
0
45
90
135
180
225
The bill still
goes up!
Sta
ge III
Efficiency
Sta
ge IV
RIs
![Page 48: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/48.jpg)
Unit cost is
cost per X
![Page 49: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/49.jpg)
cost per subscriber
cost per pageview
cost per customer
cost per API call
![Page 50: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/50.jpg)
Focus on reducing unit cost, even at total cost grows
0
30
60
90
120
150
Unit cost Total cost
![Page 51: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/51.jpg)
Unit cost =
total cost / business metric
![Page 52: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/52.jpg)
$1000 / 1000 customers =
$1 unit cost
![Page 53: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/53.jpg)
Visibility
Allocation Efficiency
Savings
Unit Cost
![Page 54: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/54.jpg)
What does it look like in the real world?
![Page 55: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/55.jpg)
Case study: Enterprise with 100s of apps
• Split the detailed billing reports into 750 different views broken down by
BUs/applications/environments/etc.
• Daily spending updates
800 users get daily emails with just their spending
• Oversight for reporting and control
Execs/finance see the total spending in a simple weekly digest
"Our goal is to put control of AWS resources in each product team’s hands. We want
to get out of the way but at the same time we still need oversight to stop
overspending.”
![Page 56: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/56.jpg)
Case study: Scaled web business pushing unit cost
• Product owners track unit cost daily tying spending to revenue
• Operations examines infrastructure changes in terms of unit cost impact
• Finance reports monthly & quarterly on margin impacts based on unit cost
• VPs set goals on unit cost and worry less about setting arbitrary total cost
goals
Shortcut: Track your AWS bill as a percentage of revenue
![Page 57: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/57.jpg)
realestate.com.au: Increasing efficient use among
ops/engineers
• Lots of developers using dev/test
instances and forgetting to turn them
off
• Used Cloudability API data to
normalize dev/test environment
“health”
• Developed an algorithm to determine
health/sickness of any dev/test
environment
avg hourly node cost ($)
x
avg node uptime (%)
x
sq rt inverse of CPU util (%)
x
avg node running life (hrs)
x
avg daily unique nodes^1.3
Midge’s law
![Page 58: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/58.jpg)
Adobe: Optimizing thousands of reserved instances
• One team purchases Reserved
Instances across Adobe
• Established a consistent
monthly purchase to account
for change
25th of the month:
Central team drives decisions with
minimal input from teams
26th of the month:
Modifications made each month
ahead of purchase
28th of the month:
Purchase based on the current
month’s utilization data
![Page 59: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/59.jpg)
Getting started on your culture
![Page 60: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/60.jpg)
Things to do right after this Summit
I. Visibility: Set up a daily spending report for stakeholders
II. Allocation: Meet with your finance team to determine the taxonomy for tags & linked
accounts
III. Efficiency: Look for underutilized instances (low CPU+BW+disk) and M3 candidates
IV. Savings: Make an immediate small and uncontroversial Reserved Instance buy
V. Unit cost: Determine a top-line business metric, divide last months’ costs by it
![Page 61: Creating a Culture of Cost Management in Your Organization](https://reader031.vdocument.in/reader031/viewer/2022032506/55cd5da8bb61eb32618b45a4/html5/thumbnails/61.jpg)
NEW YORK