networking israeli day 2013 - hecatonchire: transparent memory aggregation

Aidan Shribman; Sr. Researcher; SAP Research Israel With the contribution of Benoit Hudzia, Roei Tell, Steve Walsh, Peter Izsak

Hecatonchire: Transparent Resource Aggregation Israeli Networking Day 2013

© 2012 SAP AG. All rights reserved. 2

In the talk today

Motivation: the SAP HANA in-

memory database.

Intro: the Hecatonchire project

Capabilities: mainly memory

aggregation.

Numbers: quicksort & SAP

HANA TCP-H benchmarks.


Order Country Product Sales

Back story: SAP HANA In-Memory Real-Time Analytics

456 France corn 1000

457 Italy wheat 900

458 Italy corn 600

459 Spain rice 800

456

457

458

459

France

Italy

Italy

Spain

corn

wheat

corn

rice

1000

900

600

800

Row order organization Column order organization

Order Country Product Sales

SAP HANA In-Memory Database (2010)

Big multi-core large-memory systems; high

memory/core ratio: 128 GB; 256 GB; … 1024 GB.

OLAP (OnLine Analytical Processing): on-the-fly

analytics on real-time data

Single-scan aggregation: SELECT Country, SUM(Sales) FROM Sales_Orders

WHERE Product=‘corn’ GROUP BY Country

SAP ERP R/3 (1980s)

Moderate hardware requirements; Is optimized for slow

storage devices.

OLTP (OnLine Transactioanl Processing) : mainly

storing and retrieving transactions

Single-record access: SELECT * FROM Sales_Orders

WHERE Order = ‘457’

G


A “gap” in Current Cloud Offerings

Guest are constrained by host sizes

Limited by physical constraints (AWS high-

memory is 244 GB guest atop a 256 GB host).

Host resources are pooled together (future?)

No limitation: Better scalability; Better economics;

Better performance.

H/W OS App VM

H/W

OS

App

VM

H/W

OS

App

VM

Guests

CPUs

Memory

I/O

CPUs

Memory

I/O

CPUs

Memory

I/O

Server #1 Server #2 Server #n

Fast RDMA Communication

H/W OS App VM

H/W

OS

App

VM

H/W

OS

App

VM

H/W

OS

App

VM

Guests

CPUs

Memory

I/O

CPUs

Memory

I/O

CPUs

Memory

I/O

Server #1 Server #2 Server #n

Fast RDMA Communication


The Hecatonchire Project

Hecatonchires in Greek mythology means “Hundred Handed One”. The project addresses the

transparent aggregation of compute resources such as memory, CPU and storage

Hecatonchire aims at extending (not replacing) existing cloud and virtualization stacks

(Linux/KVM, QEMU, OpenStack) while using of commodity hardware and modern

interconnects (10 GbE or InfiniBand)

Hecatonchire is an open source project hosted on github developed by SAP Research


Faster to Scale-Out …

Author: Chaim Bendalac


Challenges of Resources Aggregation

CPU (future)

State is MBs in size (can be migrated sub-second)

Can be virtualized using hypervisor vCPU abstraction

Difficult challenge: Implementing efficient fine-grained cache-coherency (difficult)

Memory (in-progress)

GBs to TBs is size (accessed in page granularity 4KB – 4MB)

Can be virtualized by extending the MMU (Main Memory Unit) system

Challenge: achieving near-native performance while not degrading availability

Storage (mostly-done)

TBs to PBs in size (but accessed in blocks MBs in size)

Already virtualized using VFS; BLK; SCSI; iSCSI; virtio/vhost ;iSER; LIO.

Continued challenge: high bandwidth and high IOPS (partially achieved)


VM Memory Pooling

Fast Remote Page Access

Low latency RDMA (Remote Direct Memory Access) page transfer protocol

Supports best price/performance hardware: InfiniBand, iWARP, RoCE

Demand pre-paging (pre-fetching) mechanism

Built-in RAID-1 style Memory Mirroring

No extra single points for failure

Minimal overhead : 1st slave to answer synchronous; 2nd - asynchronous

Complimentary to VM-based High-Availability (e.g. Kemari)

Transparent for guest VMs

Integration by adding static hooks into Linux MMU (page fault, swap-out, etc.).

QEMU Binds Remote Memory to Guest Linux/KVM Virtual Machine.


Multithreaded 2 GB Quicksort Benchmark

94.00%

96.00%

98.00%

100.00%

102.00%

104.00%

106.00%

108.00%

110.00%

75% 50% 33% 25% 20%

Co

mp

leti

on

Tim

e /

Nati

ve T

ime

Local Memory Percentage

Native

Remote

Mirrored


SAP HANA TPC-H like OLAP Benchmark (128 GB / 40 vCPU)

99.00%

99.50%

100.00%

100.50%

101.00%

101.50%

102.00%

50% 33%

Co

mp

leti

on

Tim

e /

Nati

ve T

ime

Local Memory Percentage

Native

Mirrored


Application Memory Pooling

Required if virtualizaiton can„t be used due to

performance of usage of exotic hardware.

When using virtualization we „remotify“ the entire

guest VM address space.

Here we need to „remofity“ all processes

associated with the application.


Hecatonchire key takeaways

Already provides transparent VM / application

memory aggregation

Extends existing cloud & virtualization

solutions

Released as open source / hosted on GitHub

Developed by SAP Research Technology

Infrastructure (TI) Practice

Thank you!

Aidan Shribman

SAP Research Israel

[email protected]

The Hecatonchire Project

http://www.hecatonchire.com

[email protected]

mailto:[email protected]

http://www.hecatonchire.com/

mailto:[email protected]


Legal Disclaimer

The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of

SAP. This presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP

has no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or

release any functionality mentioned therein. This document, or any related presentation and SAP's strategy and possible future

developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at

any time for any reason without notice. The information on this document is not a commitment, promise or legal obligation to

deliver any material, code or functionality. This document is provided without a warranty of any kind, either express or implied,

including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This

document is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or

omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent.

All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially

from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as

of their dates, and they should not be relied upon in making purchasing decisions.

networking israeli day 2013 - hecatonchire: transparent memory aggregation

Documents