© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

November 12, 2014 | Las Vegas

ENT207

Creating a Culture of Cost Management

in Your Organization

J.R. Storment, Cloudability

0

45

90

135

180

225

De

v/T

est

PO

C

Ap

p #

1

Ap

p #

3

Ap

p #

2

Exec team starts

really caringL

oa

d te

st

A story of growth

Why does this need to be cultural?

The cloud is no longer one person’s responsibility…

Engineers Finance Operations Capacity Execs

How do you build the culture?• Put data in the hands of the people

• Enact policies and evangelize best practices

• Incentivize good behavior

Rolling out a cost management program

• Stage I: Cost visibility for all (emails, alerts, dashboards)

• Stage II: Allocating costs to departments (tags, linked accounts)

• Stage III: Efficiently using only what you need (underutilized, rightsizing)

• Stage IV: Saving by lowering hourly costs (Reserved and Spot Instances)

• Stage V: Tying spending to the bottom line (unit cost)

Visibility

Allocation Efficiency

Savings

Unit Cost

I. Cost visibilityVisibility

Two types of people

Incur costs Watch costs

Two types of people

Incur costs Watch costs

Two types of people

Incur costs Watch costs

Two types of people

Incur costs Watch costs

Tips for cost visibility

• Get each stakeholder the spending fundamentals daily

• Let each teams see other teams’ spending habits

• Create broadly available dashboards

Case study: Enterprise with 100s of apps

"Our goal is to put control of AWS resources in each product

team’s hands. We want to get out of the way but at the same

time we still need oversight to stop overspending.”

- Split the detailed billing reports into 750 different views broken

down by BUs/applications/environments/etc.

- Daily spending updates

800 users get daily emails with just their spending

- Oversight for reporting and control

Execs/finance see the total spending in a simple weekly digest

II. Allocation

Allocation

First, get everyone involved

๏ Multiple major business units

๏ Lots of products or cost

centers

๏ Thousands of shared AWS

resources

๏ Distributed teams using AWS

Tools for splitting up resources

• Tags are highly flexible, but 100% coverage is difficult due to compliance

• Linked accounts offer clean chargeback but limit reporting options

Pro-tips: Allocating costs

• Get consensus on the taxonomy (but let Finance drive)

• Define 2–3 mandatory tags like “project” or

“environment”

• Consider a “tag or terminate” rule to enforce compliance

- Split the accounts based on product and environment

- SAP data as source of truth for grouping metadata:

Cost center, product group, business unit, etc

- Chargeback aligned to finance’s view of the business,

not just the operational linked account structure

Case study: The enterprise with 500 linked accounts,

no oversight, hundreds of influencers

III. Efficiency

Efficiency

168hours in a

week

nights &

weekends

108

Don’t run the cloud like a data center:

65% of the hours in a month are nights and

weekends

Three easy wins for Amazon EC2 efficiency…

Low CPU, low bandwidth, low disk I/O, >1 day old | This example: Save $1,682 per week

Turn off underutilized instances

Schedule dev/test downtime

Find M1s that could be upgraded

1. Newer instance families are faster & cheaper

but have smaller SSDs

2. Look for instances that have Amazon Elastic Block Store

(EBS) devices attached and low ephemeral disk access

3. Move M1 to M3 to save $0.03-$0.10 an hour

Source: http://aws.amazon.com/ec2/pricing/

Get alerts when thresholds are crossed

Tips for encouraging efficient behavior

• Automate weekly waste reporting to each team

• Gamify clean-up by creating a visible leaderboard

• Do a monthly whole-company waste review

For more check out:

“MegaRun: Behind the 156,000 Core HPC Run”

on Friday at 1pm

Case study: High traffic site with decentralized ops

avg hourly node cost ($)

x

avg node uptime (%)

x

sq rt inverse of CPU util (%)

x

avg node running life (hrs)

x

avg daily unique nodes^1.3

Lots of developers using dev/test instances

and forgetting to turn them off

Used Cloudability API data to normalize

dev/test environment “health”

Developed an algorithm to determine

health/sickness of any dev/test

environment

Midge’s law

IV. SavingsSavings

Spot vs. Reserved Instances?

Reservations require no engineering,

enabling finance to autonomously reduce costs

Don’t fear the reservation

• Prices drop, but not fast enough to erode savings

• With usage over 50% three-year RIs always wonSource: http://blog.cloudability.com/cloud-cost-war-shouldnt-stop-buying-reserved-instances/

Buy based on current hourly data

Don’t wait to buy reservations until you’ve completed the

perfect analysis—you’ll waste more money than you save

Tips for reservation efficiency

•Appoint a person to own reservation purchases—

often a tech-minded finance person

•Start with an immediate small and uncontroversial buy

•Buy iteratively on a schedule that you hold sacred

Case study: Buying for 12,500+ daily instances

25th of the month:

Central team drives decisions with minimal input from teams

26th of the month:

Modifications made each month ahead of purchase

28th of the month:

Purchase based on the current month’s utilization data

For more, check out:

Iterating Your Way to 95% Reserved Instances

(ENT 203)

Today at 4:30pm

V. Unit cost

Unit cost

Why does unit cost matter?

0

45

90

135

180

225

De

v/T

est

PO

C

Ap

p #

1

Ap

p #

3

Ap

p #

2

Exec team starts

really caring

Hic

cu

p

An increasing bill isn’t necessarily a bad thing

0

45

90

135

180

225

0

45

90

135

180

225

The bill still

goes up!Exec team starts

really caring

Sta

ge

III

Efficiency

Sta

ge

IV

RIs

@cloudabilit

y

Unit cost is

cost per X

@cloudabilit

y

cost per subscriber

cost per pageview

cost per customer

cost per api call

0

30

60

90

120

150

Focus on reducing unit cost, even as total cost grows

Unit cost Total cost

@cloudabilit

y

Unit cost =

total cost / business

metric

@cloudabilit

y

$1000 / 1000 customers

=

$1 unit cost

Case study: Scaled web business pushing unit cost

• Product owners track unit cost daily tying spending to revenue

• Operations examines infrastructure changes in terms of unit cost impact

• Finance reports monthly & quarterly on margin impacts based on unit cost

• VPs set goals on unit cost and worry less about setting arbitrary total cost

goals

Shortcut: Track your AWS bill as a percentage of revenue

Getting started on your culture

Things to do right after re:InventI. Visibility: Give stakeholders a daily view into spending

II. Allocation: Put together a taxonomy with your finance team and start

splitting linked accounts and tagging

III.Efficiency: Look at underutilized instances (low CPU + BW + disk) and M1

to M3 candidates (disk access + attached storage)

IV.Hourly savings: Make an immediate small and uncontroversial RI buy

V.Unit cost: Determine your top-line business metric, divide costs by them

Start a cost management group

to share best practices within your company

and encourage accountability

http://bit.ly/awsevals