Cutting the Electric Bill for Internet-Scale SystemsAndreas AndreouCambridge University, [email protected]
What’s this all about?
• Energy expenses are an increasingly important fraction of data center operating costs
• Electricity prices show both temporal and geographical variation
• Exploit variations in electricity prices for economic gain
Key observations• Electricity prices vary• Prices vary on an hourly basis• Often not well correlated at different locations• Substantial variations
• Large distributed systems already incorporate request routing and replication• Dynamic request routing to map clients to servers• Mechanisms to replicate data necessary to process requests at
multiples sites
Problem Specification• Large system composed of server clusters spread out
geographically• Map client requests to clusters such that the total electricity
cost is minimized• Assumptions• System fully replicated• Optimize for cost every hour• No knowledge of the future• Rate of change slow enough to be compatible with existing
routing mechanisms• Fast enough to respond to electricity market fluctuations• Incorporate bandwidth and performance goals as constraints
Terminology• Energy Elasticity• Degree to which energy consumed by a cluster depends on the
load placed on it• Ideally: no load, no power• Worst case: no difference between peak and idle power• State-of-the-art: idle power around 60% of peak
• Differential Duration• Number of hours one location is favored over another by more
than $5/MWh• PUE• Power usage effectiveness (measure of data center energy
efficiency)
Background
Wholesale Electricity Markets (1)• Generation• Government and independent power producers• Coal (~50%), natural gas (~20%), nuclear power (~20%),
hydroelectric generation (~6%)• Different regions, different power generation profiles
• Transmission• Producers and consumers are connected to an electric grid• 8 reliability regions
Wholesale Electricity Markets (2)• Market Structure• Each region managed by Regional Transmission Organization
(RTO)• RTO administer wholesale electricity markets• Auctioning mechanism:
• Producers present supply offers• Consumers present demand bids• Coordinating body determines flow and sets prices
• Market Types• Day-ahead markets• Real-time markets
Wholesale Electricity Markets (3)• Market Structure• Assumptions
• Real-time prices are known and vary hourly• Electric bill is proportional to consumption and indexed to wholesale
prices• Request routing behavior induced by our method doesn’t significantly
alter prices and market behavior
Daily Variation
Different Market Types• Hourly real-time (RT) market is more volatile than day-ahead
market
Hour-to-Hour Volatility
Geographic Correlation
Price Differentials
Differential Distributions
Time-of-Day
Differential Duration
Akamai: Traffic and Bandwidth• Over 2000 content provider customers in the US• 9-region traffic with electricity price data• Data covering 24 days worth of traffic• Traffic data of 5-minute intervals from public clusters
• Bandwidth costs are significant• Aggressively optimized to reduce bandwidth costs• 95/5 billing model
• Client-Server Distances• Use geographic distance as a coarse proxy for network
performance
Cluster Energy Consumption (1)• Roughly linear to its utilization
• Pidle : average idle power draw of single server
• Ppeak : average peak power draw of single server
• r: empirical derived constant• ut : average CPU utilization at time t
• • what is important in determining savings
Routing Energy• Increased path lengths will not alter energy consumption
significantly• Average energy for a packet to pas through is on the order of
2mJ• Incremental energy dissipated by each packet passing
through a core router would be as low as 50μJ per medium size packet
• New routes may overload existing routers• Additional bandwidth could lead to upgrade• Can ignore by incorporating 95/5 bandwidth constraints
Simulation Strategy• Real-time market prices for 29 different locations• Traffic data for Akamai public clusters in 9 of those• Data set spanning Jan 2006 through Mar 2009• Workload data set contains 5-minute samples in 25 cities• Period of 24 days and some hours• Discarded 7 and grouped remaining 18 cities to 9 clusters
• Akamai’s geographic server distribution• Two routing schemes• Akamai’s original allocation• Distance constrained electricity price optimizer
• Energy model as shown before
24 Days of Traffic (1)• Energy Elasticity
• Bandwidth Costs
24 Days of Traffic (2)• Distance and savings
39 Months of Prices• Derived from 24-day Akamai workload (US traffic only)
• Dynamic beats static
Results• Existing systems can reduce energy costs be at least 2%
without any increase in bandwidth costs or significant reduction in client performance• Google-like energy elasticity• Akamai-like server distribution• 95/5 bandwidth constraints
• Savings increase with energy elasticity• Fully elastic system with relaxed bandwidth constraints can
reduce energy cost be 30% (13% with bandwidth constraints)
• Allowing increase of client-server distances leads to increased savings
Considerations (1)• Not reacting immediately to price changes noticebly reduces
overall savings
Considerations (2)• Server operators should be able to negotiate contractual
arrangements
• Distributed systems with energy elastic clusters can be more flexible than traditional consumers
• Triggered demand response programs
Future Work• Implementing Joint Optimization
• RTO Interaction
• Weather Differentials
• Environmental Cost