new howtomakeproﬁt: … · 2018. 6. 14. · mqm@`2m2r #h2 2h2+i`b+biv bqm`+2b non-renewable plus...

How to Make Profit:Exploiting Fluctuating Electricity Prices with Albatross,A Runtime System for Heterogeneous HPC Clusters

Timo Hönig, Christopher Eibel, Adam Wagenhäuser, Maximilian Wagner,

and Wolfgang Schröder-Preikschat

8th International Workshop on Runtime and Operating Systems forSupercomputers (ROSS 2018), Tempe, AZ, USA.

June 12, 2018

Gode Wind Farm: 582 MW North Sea, Europe

Photo: DONG Energy

Solana Generating Station: 280 MW

Arizona, USA

Photo: Abengoa Energy

Solana Generating Station: 280 MW Arizona, USA

Photo: Abengoa Energy, Google

Arizona State University: 24 MW Arizona, USA

Photo: Arizona State University

Building a Runtime System for Heterogeneous HPC Clusters

☼ Reenewable electricity sources havea great impact on the grid.

Availability of renewables leads tofluctuating electricity prices.

How can we design and build an HPC runtimesystem to exploit dynamic electricity pricing?

Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 5

Energy Mix (Germany, 2017)

Others1 %

Renewables38 %

Natural Gas9 %

Non-renewables62 %

Nuclear Power13 %


■ non-renewables: 339.93 TWh, renewables: 209.97 TWh

■ dynamic availability of renewables fluctuation electricity prices

Energy Mix (Germany, 2017)

Others1 %

Renewables38 %

Natural Gas9 %

Non-renewables62 %

Nuclear Power13 %


■ non-renewables: 339.93 TWh, renewables: 209.97 TWh■ dynamic availability of renewables fluctuation electricity prices

Fluctuation of Electricity Prices

Mo Tu We Th Fr Sa Su

20

40

60

80

Time [day of week]

Powe

r[GW

]

-100-80-60-40-20020406080100

Price[EUR/M

Wh]

0

non-renewable electricity sourcesnon-renewable plus renewable electricity sourceselectricity price

[week 43/2017]


Others1 %

Renewables38 %

Natural Gas9 %

Non-Renewables

Nuclear Power13 %

■ non-renewables can be planned…

■ …but renewables are often unpredictable■ consequence: strong fluctuation of the electricity price■ extreme imbalance of supply & demand negative prices



20

40

60

80

Time [day of week]

Powe

r[GW

]

-100-80-60-40-20020406080100

Price[EUR/M

Wh]

0


[week 43/2017]


Others1 %

Renewables

Natural Gas9 %

Non-Renewables

Nuclear Power13 %

■ non-renewables can be planned…■ …but renewables are often unpredictable

■ consequence: strong fluctuation of the electricity price■ extreme imbalance of supply & demand negative prices



20

40

60

80

Time [day of week]

Powe

r[GW

]

-100-80-60-40-20020406080100

Price[EUR/M

Wh]

negative price

0


[week 43/2017]


■ non-renewables can be planned…■ …but renewables are often unpredictable

■ consequence: strong fluctuation of the electricity price■ extreme imbalance of supply & demand negative prices

Building a System that Benefits from Dynamic Electricity Prices

10:00 11:00 12:00 13:00 14:00 15:00 16:00

0

2

4

6

8

Time [hours]

Powe

rD

eman

d[kW

]

HPC ClusterPower Demand

Excess Demand / Under SupplyPenalty zone I

Target zone

Under Demand / Excess Supply Penalty zone II


■ challenge: take advantage of low and negative prices■ consider contracts (i.e., penalties enforced by grid operators)■ integrate with existing HPC cluster infrastructure


10:00 11:00 12:00 13:00 14:00 15:00 16:00

0

2

4

6

8

Time [hours]

Powe

rD

eman

d[kW

]

HPC ClusterPower Demand


Target zone



■ challenge: take advantage of low and negative prices■ consider contracts (i.e., penalties enforced by grid operators)■ integrate with existing HPC cluster infrastructure

build a flexible runtime system that integrates data onelectricity pricing into its operating decisions

reduce energy demand when prices are high increase energy demand when prices are low (or negative)


10:00 11:00 12:00 13:00 14:00 15:00 16:00

0

2

4

6

8

Time [hours]

Powe

rD

eman

d[kW

]

With Albatross

Without Albatross


Target zone



Our proposed runtime system Albatross:

■ implements low- and high-power operation modes■ considers heterogeneity aspects of the cluster hardware■ respects QoS and non-functional requirements of workloads

Albatross: Design and Implementation


■ foobar

Job1

Job2

Jobm

QoSconstraints ...

Electricity & priceconstraints

Master node

Clustermanager

Jobscheduler

Resource group1

Resource groupl

...

Exec1 Execn1 Probe1 Computenode1…


...

Exec1 Execnk Probek Computenodek…

Powermeters

■ generic system model of an HPC cluster without Albatross

Job1

Job2

Jobm

...

Master node

Clustermanager

Jobscheduler

Execution unit1Compute

node1


node2...

Execution unitnCompute

noden



■ extended system model of an HPC cluster with Albatross

Job1

Job2

Jobm

QoSconstraints ...


Master node

Clustermanager

Jobscheduler

Resource group1

Resource groupl

...



...


Powermeters

■ generic system model of an HPC cluster without Albatross

Job1

Job2

Jobm

...

Master node

Clustermanager

Jobscheduler


node1


node2...

Execution unitnCompute

noden




Job1

Job2

Jobm

QoSconstraints ...


Master node

Resourcegovernor

Clustermanager

Jobscheduler

Resource group1

Resource groupl

...



...


Powermeters

■ input interfaces for job submission and control■ submission interface for electricity-pricing data■ real-time data as provisioned by grid operator■ detailed job description, added QoS constraints




Job1

Job2

Jobm

QoSconstraints ...


Master node

Resourcegovernor

Clustermanager

Jobscheduler

Resource group1

Resource groupl

...



...


Powermeters

■ power monitoring and control interfaces■ probes operate power-management features (e.g., power capping)■ hardware-specific metering (e.g., RAPL, measuring devices)■ delivery of base data for feedback loop to the resource governor




Job1

Job2

Jobm

QoSconstraints ...


Master node

Resourcegovernor

Clustermanager

Jobscheduler

Resource group1

Resource groupl

...



...


Powermeters

■ runtime control of the HPC system by the resource governor■ job submission based on QoS data and current electricity prices■ control and measurement of power demand for individual jobs■ consider heterogeneity aspects (system, component, and configuration)




Job1

Job2

Jobm

QoSconstraints ...


Master node

Resourcegovernor

Clustermanager

Jobscheduler

Resource group1

Resource groupl

...



...


Powermeters

■ runtime control of the HPC system by the resource governor■ job submission based on QoS data and current electricity prices■ control and measurement of power demand for individual jobs■ consider heterogeneity aspects (system, component, and configuration)

Het

erog

eneit

ySc

ope

System Component

Albatross: Prototype Implementation

Feature Moab Tivoli Univa Slurm

Job pinning ○␣ ○␣ CPU allocation ○␣ Quality of service ○␣ Generic resources ○␣ Cluster status Heterogeneity aware ○␣ ○␣ Power/price aware ○␣ ○␣ ○␣ ○␣

Table: Feature comparison of Albatross with other workload managers.


■ core of Albatross implementation is based on Slurm■ constraint-aware job-to-node–assignment strategy

Evaluation Overview■ evaluation scenario NAS Parallel Benchmarks (NPB) suite,

Albatross runtime system

heterogeneous cluster with Intel & ARMCPUs, Nvidia & Intel GPUs, and variouspower meters

■ evaluation experiments and goals1. impact of system configuration on

energy demand and execution times2. influence of heterogeneity on the

energy–delay product and power demand3. combination of power and price awareness

to exploit dynamic electricity pricing


1 MCP39F511

1

3 Current probe

3

2 PCIe riser card

2

4 Red Pitaya

4

C Intel Xeon

C

A / B ARM ODROID-C1+/C2

A B

D Intel HD GPU

D

E Nvidia GPU

E

Evaluation Overview

■ evaluation scenario

NAS Parallel Benchmarks (NPB) suite,Albatross runtime system

heterogeneous cluster with Intel & ARMCPUs, Nvidia & Intel GPUs, and variouspower meters

■ evaluation experiments and goals

1. impact of system configuration onenergy demand and execution times

2. influence of heterogeneity on theenergy–delay product and power demand

3. combination of power and price awarenessto exploit dynamic electricity pricing


1 MCP39F511

1

3 Current probe

3

2 PCIe riser card

2

4 Red Pitaya

4

C Intel Xeon

C

A / B ARM ODROID-C1+/C2

A B

D Intel HD GPU

D

E Nvidia GPU

E

Experiment 1: Energy Demand and Execution Times


■ impact of system-level and configuration-level heterogeneity

CG EP FT IS MG0

100

200

Ener

gyD

eman

d[J

]

ODROID-C1+ ODROID-C2 Xeon Xeoncap

ExecutionTime 9.1

s8.0

s0.6

s0.6

s27

.9s

12.6

s1.9

s2.6

s55

.1s

22.8

s1.2

s1.4

s3.4

s1.5

s0.3

s0.3

s14

.8s

7.8s

1.5s

1.5s

■ Xeon and Xeoncap dominate with performance■ energy demand in some cases lower for ARM platforms■ occasional relationship between process energy demand

and execution time

Experiment 1: Energy Demand and Execution Times


■ impact of system-level and configuration-level heterogeneity

CG EP FT IS MG0

100

200

Ener

gyD

eman

d[J

]

ODROID-C1+ ODROID-C2 Xeon Xeoncap

ExecutionTime 9.1

s8.0

s0.6

s0.6

s27

.9s

12.6

s1.9

s2.6

s55

.1s

22.8

s1.2

s1.4

s3.4

s1.5

s0.3

s0.3

s14

.8s

7.8s

1.5s

1.5s

■ Xeon and Xeoncap dominate with performance■ energy demand in some cases lower for ARM platforms■ occasional relationship between process energy demand

and execution time

Key take-away

race to sleep does not always reduce energy demand knowledge on systems’ impact is absolutely necessary for

operation, requires measurements

Experiment 2: Energy–Delay Product and Power Demand


■ impact of component-level heterogeneity

0

0.2

0.4

0.6

0.8

1

Norm

alized

EDP

CG EP FT IS MG0

20

40

60

Power

Dem

and

[W]

Xeon Intel HD Nvidia

■ EDP is best for Nvidia for all but one benchmark■ Intel HD has the worst EDP results, but lowest average power

Experiment 2: Energy–Delay Product and Power Demand


■ impact of component-level heterogeneity

0

0.2

0.4

0.6

0.8

1

Norm

alized

EDP

CG EP FT IS MG0

20

40

60

Power

Dem

and

[W]

Xeon Intel HD Nvidia

■ EDP is best for Nvidia for all but one benchmark■ Intel HD has the worst EDP results, but lowest average power

Key take-away

both, EDP and average power must be considered to balance the cluster’s power drain Albatross trades

average power for EDP (and vice versa)

Experiment 3: Combining Power and Price Awareness


0 10 20 30 40 50 600

40

80

120

160

Time [min]

Power

Drain

[W]

AlbatrossSLURM

upper limitlower limit

-0.1

0

0.1

Time [day of week]

Cost

[EUR]

-100

-50

0

50

100

Price

[EUR/MW

h]

Th Fr Sa Su

negative price

Albatross

SLURM

price

[week 43/2017]

Conclusion

Albatross, an HPC runtime system thatsuccessfully exploits dynamic electricity pricing.

Albatross…■ considers QoS of jobs and electricity prices for

deploying workloads

■ exploits heterogeneity at system-level,component-level, and configuration-level

■ prepares HPC clusters for future settings whereenergy is just another operating resource


Discussion

HPDC Poster Session(Wednesday Evening)

Others1 %

Renewables

Natural Gas9 %

Non-Renewables

Nuclear Power13 %


interface Albatross_Cluster_Control {/* Get cluster configuration */NodeList get_nodes();/* Get all running jobs and their allocation */JobList get_jobs();/* Submit job with attached job−resource info */JobResult submit_job(Job j);

}


Job1

Job2

Jobm

QoSconstraints ...


Master node

Resourcegovernor

Clustermanager

Jobscheduler

Resource group1

Resource groupl

...



...


Powermeters


interface Albatross_Power_Control {/* Triggers power measuring for an exec. unit */void start_measurement(Node n, ExecUnit u);PowerValues stop_measurement(Node n, ExecUnit u);

/* Sets the value of a power−management feature */void set_pm(Node n, ExecUnit u, PowerConfig c);

}


Job1

Job2

Jobm

QoSconstraints ...


Master node

Resourcegovernor

Clustermanager

Jobscheduler

Resource group1

Resource groupl

...



...


Powermeters

Timo Hönig, Christopher Eibel, Adam Wagenhäuser, Maximilian Wagner, andWolfgang Schröder-Preikschat:How to Make Profit: Exploiting Fluctuating Electricity Prices with Albatross,A Runtime System for Heterogeneous HPC ClustersProceedings of the 8th International Workshop on Runtime and OperatingSystems for Supercomputers (ROSS’18), 12 June 2018, Tempe, AZ, USAACM, New York, NY, USA, ISBN: 978-1-4503-5864-4DOI: 10.1145/3217189.3217193

Timo Hönig, Christopher Eibel, Adam Wagenhäuser, Maximilian Wagner, andWolfgang Schröder-Preikschat:Making Profit with ALBATROSS: A Runtime System for HeterogeneousHigh-Performance–Computing ClustersProceedings of the 27th International Symposium on High-Performance Paralleland Distributed Computing (HPDC’18), 11 - 15 June 2018, Tempe, AZ, USAACM, New York, NY, USA, ISBN: 978-1-4503-5899-6DOI: 10.1145/3220192.3220457
https://dl.acm.org/citation.cfm?doid=3217189.3217193https://dl.acm.org/citation.cfm?doid=3217189.3217193http://www.mcs.anl.gov/events/workshops/ross/2018/https://doi.org/10.1145/3217189.3217193https://dl.acm.org/citation.cfm?id=3220457https://dl.acm.org/citation.cfm?id=3220457http://www.hpdc.org/2018/https://doi.org/10.1145/3220192.3220457

new howtomakeproﬁt: … · 2018. 6. 14. · mqm@`2m2r #h2 2h2+i`b+biv bqm`+2b non-renewable plus...

Documents