bocx_hana_2011-04-13_v1.0

The Road to HANA: SAP In-

memory Appliance

SAP HANA 1.0:

Deep Dive into Architecture

The Road to HANA: SAP In-memory

Appliance (SAP HANA 1.0)

Deep Dive into Architecture

Marc Bernard

SAP Technology Regional Implementation Group

April 13, 2011

2011 SAP AG. All rights reserved. / Page 3

Disclaimer

This presentation outlines our general product direction and should not be relied on in

making a purchase decision. This presentation is not subject to your license

agreement or any other agreement with SAP.

SAP has no obligation to pursue any course of business outlined in this presentation

or to develop or release any functionality mentioned in this presentation. This

presentation and SAP's strategy and possible future developments are subject to

change and may be changed by SAP at any time for any reason without notice.

This document is provided without a warranty of any kind, either express or implied,

including but not limited to, the implied warranties of merchantability, fitness for a

particular purpose, or non-infringement. SAP assumes no responsibility for errors or

omissions in this document, except if such damages were caused by SAP

intentionally or grossly negligent.


Vision: In-Memory Computing

Background and Context

Technology that allows the

processing of

massive quantities of real

time data

in the main memory of the

server

to provide immediate results

from

analyses and transactions


EXPAND PARTNER ECOSYSTEM

Partner-built applications, Hardware partners

CUSTOMER CO-INNOVATION

Design with customers

TECHNOLOGY INNOVATION BUSINESS VALUE

Real-Time Analytics, Process Innovation, Lower TCO

GU

ID

IN

G P

RIN

CIP

LE

S

INNOVATION WITHOUT DISRUPTION

New Capabilities For Current Landscape

HEART OF FUTURE APPLICATIONS

Packaged Business Solutions for Industry and Line of Business

SAP Strategy for In-Memory Computing


In-Memory Computing The Time is NOWOrchestrating Technology Innovations

HW Technology Innovations

64bit address space 2TB in current servers

100GB/s data throughput

Dramatic decline in

price/performance

Multi-Core Architecture (8 x 8core CPU

per blade)

Massive parallel scaling with many

blades

One blade ~$50.000 = 1 Enterprise

Class Server

Row and Column Store

Compression

Partitioning

No Aggregate Tables

Insert Only on Delta

The elements of in-memory computing are not new. However, dramatically improved hardware economics and technology innovations

in software have now made it possible for SAP to deliver on its vision of the Real-Time Enterprise with in-memory business applications

SAP SW Technology Innovations


In-Memory Computing Naming

SAP In-Memory Appliance

(SAP HANA)SAP In-Memory Database

Application Name, Advanced by SAP In-

Memory Computing

Example: SAP BusinessObjects Strategic Workforce Planning,

Advanced by SAP In-Memory Computing

SAP In-Memory Computing

Technology

Appliance Database

Applications

Formerly known as

SAP High-Performance Analytic Appliance (SAP HANA)

Formerly known as

SAP In-Memory Computing Engine

Formerly known as

in-memory computing

SAP In-Memory Computing studio

Studio

Name remains the same


Preconfigured Analytical Appliance

In-Memory software + hardware(HP, IBM, Fujitsu, Cisco, Dell)

In-Memory Computing Engine Software

Data Modeling and Data Management

Real-time Data Replication Data Services for SAP Business Suite, SAP BW and 3rd Party Systems

Capabilities Enabled

Analyze information in real-time at unprecedented speeds on large volumes of non-

aggregated data

Create flexible analytic models based on real-time and historic business data

Foundation for new category of applications (e.g., planning, simulation) to significantly

outperform current applications in category

Minimizes data duplication

SAP In-Memory Appliance (SAP HANA)

Architecture

BICS SQL MDXSQL

Modeling

Studio

RealTime Replication

Services

Data

Services

SAP HANA

SAP BusinessObjects Other Applications

SAP NetWeaver

BW

SAP Business

Suite3rd Party

In-Memory Computing Engine

Calculation and

Planning Engine


Agenda

1. Architecture Overview

2. Row Store

3. Column Store

4. Persistency Layer

5. Modeling

6. Q&A


ERP

Architecture Overview

In-Memory Computing Engine and Surroundings

ERP DB


Clients (planned, e.g.) BI4 Explorer

Dashboard

DesignSAP BI4 universes

(WebI,...)

Request Processing / Execution Control

MS Excel

BI4 Analysis

SQL Parser MDX

SQL Script Calc Engine

Transaction

Manager

Session Management

Relational Engines

Row Store Column Store

Persistence LayerPage Management Logger

Disk StorageLog VolumesData Volumes

Authorization

Manager

Metadata

Manager

In-Memory Computing Studio

Administration Modeling

Load

ControllerReplication

Agent

Replication

Server

SAP Business Objects BI4

Data

Services

Designer

SBO BI4

servers

( program

for client)

SBO BI4

Information

Design Tool

Other Source Systems

SAP

NetWeaver

BW3rd Party

Data

Services


ERP


The Engine

LogERP DB

Clients (planned, e.g.) SBOP Explorer 4.0

Xcelsius SAP BI universes (WebI,...)

MS Excel

SBOP Analysis

IMC Studio


Load


Agent

Business Objects Enterprise

Data

Services

Designer

SBO server

programs

for clients

SBO

Information

Design Tool


SAP

NetWeaver

BW3rd Party

Data

Services



SQL Parser MDX


Transaction

Manager

Session Management

Relational Engines




Authorization

Manager

Metadata

Manager

Replication

Server


ERP


Loading Data into SAP HANA

ERP DB



SQL Parser MDX


Transaction

Manager

Session Management

Relational Engines




Authorization

Manager

Metadata

Manager



Load


Agent

Replication

Server


Data

Services

Designer

SBO BI4

servers

( program

for client)

SBO

Information

Design Tool


SAP

NetWeaver

BW3rd Party

Data

Services


Dashboard


(WebI,...)

MS Excel

BI4 Analysis


ERP


Data Modeling

ERP DB



SQL Parser MDX


Transaction

Manager

Session Management

Relational Engines




Authorization

Manager

Metadata

Manager



Load


Agent

Replication

Server


Data

Services

Designer

SBO BI4

servers

( program

for client)

SBO

Information

Design Tool


SAP

NetWeaver

BW3rd Party

Data

Services


Dashboard


(WebI,...)

MS Excel

BI4 Analysis


Clients (planned, e.g.)

ERP


Reporting

ERP DB



SQL Parser MDX


Transaction

Manager

Session Management

Relational Engines




Authorization

Manager

Metadata

Manager



Load


Agent

Replication

Server


Data

Services

Designer

SBO BI4

servers

( program

for client)

SBO

Information

Design Tool


SAP

NetWeaver

BW3rd Party

Data

Services

BI4 Explorer

Dashboard


(WebI,...)

MS Excel

BI4 Analysis


ERP


Administration

ERP DB



SQL Parser MDX


Transaction

Manager

Session Management

Relational Engines




Authorization

Manager

Metadata

Manager



Load


Agent

Replication

Server


Data

Services

Designer

SBO BI4

servers

( program

for client)

SBO

Information

Design Tool


SAP

NetWeaver

BW3rd Party

Data

Services


Dashboard


(WebI,...)

MS Excel

BI4 Analysis


DB Server

SAP High-Performance Analytic Appliance 1.0

SAP HANA

JDBC ODBC ODBOSQL

DBC

SAP In-Memory

Computing Engine

Replication

Server

SAP In-Memory Computing Studio

SAP Business

Application

Replication

Agent

SAP BusinessObjects

Data Services 4.0

Any

source

SAP

BusinessObjects

BI 4.0

Repository

SAP BusinessObjects BI clients

SQ

L

MD

X

BIC

S

Auth

entication

Conte

nt m

gm

t

sync

Adm

in &

model

load (optional)

(optional)

(optional)

(existing)


Request Processing and Execution Control

Conceptual View

Standard SQL

Processed directly by DB engine

SQL Script, MDX and planning engine

interface

Domain-specific programming

languages or models

Converted into calculation models

Calc Engine

Create logical execution plan for

calculation models

Execute user defined functions

Relational Engine

DB optimizer produces physical

executing plan

Access to row and column store


Calc Engine for Dummies

The easiest way to think of Calculation Models is to see them as dataflow graphs,

where the modeler can define data sources as inputs and different operations (join,

aggregation, projection,) on top of them for data manipulations.

The Calculation Engine will break up a model, for example some SQL Script, into

operations that can be processed in parallel (rule based model optimizer). Then these

operations will be passed to the database optimizer which will determine the best

plan for accessing row or column stores (algebraic transformations and cost based

optimizations based on database statistics).


Calc Engine for Dummies

Example


Agenda


2. Row Store

3. Column Store


5. Modeling

6. Q&A



High Level Architecture

Row Store

One of the

relational engines

Interfaced from

calculation /

execution layer

Pure in-memory

store

Persistence

managed in

persistence

layer

SAP in-memory

computing engine

HANA


Row Store Architecture

Row Store Block Diagram

Row Store Block Diagram

Transactional Version Memory

Contains temporary versions

Needed for Multi-Version

Concurrency Control (MVCC)

Segments

Contain the actual data (content of

row-store tables) in pages

Page Manager

Memory allocation

Keeping track of free/used pages

Version Memory Consolidation

Think garbage collector for MVCC

Persistence Layer

Invoked in write operations (log)

And in performing savepoints checkpoint writer


Row Store Architecture

Highlights

Write Operations

Mainly go into Transactional Version Memory

INSERT also writes to Persisted Segment

Read Operations

Write Operations

Transactional

Version

Memory

Main Memory

Persisted

Segment

Data that

may be

seen by all

active

transactions

Recent

versions of

changed

records

Version Memory

Consolidation

Version Consolidation

Moves visible version from Transaction Version

Memory into Persisted

Segment (based on

Commit ID)

Clears outdated record versions from Transactional

Version Memory

Memory Handling

Row store tables are

linked list of memory

pages

Pages are grouped in

segments

Page size: 16 KB

Persisted Segment

Contains data that may be seen by any

ongoing transaction

Data that has been committed before

any active transaction was started)


Indexes for Row Store Tables

Primary Index / Row ID / Index Persistence

Each row-store table has a primary index

Primary index maps ROW ID primary key of table

ROW ID: a number specifying for each record its memory segment and page

How to find the memory page for a table record?

A structure called ROW ID contains the segment and the page for the record

The page can then be searched for the records based on primary key

ROW ID is part of the primary index of the table

Secondary indexes can be created if needed

Persistence of indexes in row store

Indexes in row store only exist in memory

No persistence of index data

Index definition stored with table metadata

Indexes filled on-the-fly when system loads tables into memory on system start-up


Agenda


2. Row Store

3. Column Store


5. Modeling

6. Q&A



High Level Architecture

Column Store

One of the relational

engines

Interfaced from

calculation / execution

layer

Pure in-memory store

Persistence

managed in

persistence layer

Optimized for high

performance of read

operation

Good performance of

write operations

Efficient data

compression

SAP in-memory

computing engine

HANA


Column Store Architecture

Column Store Block Diagram

Column Store Block Diagram

Optimizer and Executor

Handles queries and

execution plan

Main and Delta Storage

Compressed data for fast read

Delta data for fast write

Asynchronous delta merge

Consistent View Manager

Transaction Manager

Persistence Layer


Column Store

Highlights

Storage Separation (Main & Delta)

Enables high compression and high write

performance at the same time

Delta Merge Operation

See next slide

Read Operations

Write

Operations

Main

Main Memory

Delta

Write

optimized

Compressed

and

Read

optimized

Read Operations

Always have to read from both

main & delta storages and merge

the results.

Engine uses multi version

concurrency control (MVCC) to

ensure consistent read operations.

Data Compression in Main

Storage

Compression by creating

dictionary and applying further

compression methods

Speed up

Data load into CPU cache

Equality check Search

The compression is computed

during delta merge operation.

Write Operations

Only in delta storage because write optimized.

The update is performed by inserting a new

entry into the delta storage.


Column Store

Delta Management

Delta Merge Operation

Purpose

To move changes in delta storage into the compressed and read optimized main storage

Characteristics

Happens asynchronously

Even during merge operation the columnar table will be still available for read and write

operations

To fulfil this requirement, a second delta and main storage are used internally

Read Operations

Write

Operations

Main

Before Merge

Delta

Read Operations

Write

Operations

Main

New

After Merge

Delta

New

Read Operations

Write

Operations

Main

During Merge

Main

New

Delta

NewDelta

Merge Operations


Agenda


2. Row Store

3. Column Store


5. Modeling

6. Q&A


Persistence Layer

Purpose and Scope

Why Does An In-memory Database Need A Persistence Layer?

Main Memory is volatile. What happens upon

Database restart?

Power outage?

...

Data needs to be stored in a non-volatile way

Backup and restore

SAP in-memory computing engine offers one persistence layer which is used by row store and

column store

Regular savepoints full persisted image of DB at time of savepoint

Logs capturing all DB transactions since last savepoint (redo logs and undo logs written)

restore DB from latest savepoint onwards

Ability to create "snapshots"

used for backups


Persistence Layer

System Restart and Population of In-memory Stores

Actions During System Restart

Last savepoint must be restored plus

Undo logs must be read for uncommitted transactions saved with last savepoint

Redo logs for committed transactions since last savepoint

Complete content of row store is loaded into memory

Column store tables may be marked for preload or not

Only tables marked for preload

are loaded into memory during

startup

If table is marked for loading

on demand, the restore

procedure is invoked on first

access


Agenda


2. Row Store

3. Column Store


5. Modeling

6. Q&A


Row Store vs. Column Store

When to Use Which Store

Modeling Only Possible For Column Tables

This answers the frequently asked question:

"Where should I put a table row store or column store?"

Information Modeler only works with column tables

Replication server creates tables in column store per default

Data Services creates tables in column store per default

SQL to create column table: "CREATE COLUMN TABLE ..."

Store can be changed with "ALTER TABLE "

System Tables Are Created Where They Fit Best

Administrative tables in row store:

Schema SYS caches, administrative tables of engine

Tables from statistics server

Administrative tables in column store:

Schema _SYS_BI metadata of created views + master data for MDX

Schema _SYS_BIC some generated tables for MDX

Schema _SYS_REPO e.g. lists of active/modified versions of models



Look and Feel

Navigator

View

Quick Launch

View

Properties

View



Features

Information Modeler Features

Modeling

No materialized aggregates

Database views

Choice to publish and consume at 4 levels of modeling

Attribute View, Analytic View, Analytic View enhanced with Attribute View, Calculation View

Data Preview

Physical tables

Information Models

Import/Export

Models

Data Source schemas (metadata) mass and selective load

Landscapes

Data Provisioning for SAP Business Applications (both initial load and replication)

Analytic Privileges / Security


Modeling Process Flow

Import Source System metadata

Physical tables are created dynamically (1:1 schema definition of source system tables)

Provision Data

Physical tables are loaded with content.

Create Information Models

Database Views are created

Attribute Views Analytic Views Calculation

Views

Deploy

Column views are created and activated

Consume

Consume with choice of client tools

BICS, SQL, MDX



Terminology

Information Modeler Terminology

Data

Attributes descriptive data (known as Characteristics SAP BW terminology)

Measures data that can be quantified and calculated (known as key figures in SAP BW)

Views

Attribute Views i.e. dimensions

Analytic Views i.e. cubes

Calculation Views similar to virtual provider with services concept in BW

Hierarchies

Leveled based on multiple attributes

Parent-child hierarchy

Analytic Privilege security object



Navigator View - Default Catalog

HANA Instance ()

HANA Server Name

and Instance Number

User Database schema

Schema Content:

Column Views,

Functions, Tables,

Views



Navigator View - Information Models

Information Models organized

in Packages

Attribute Views, Analytic Views,

Calculation Views, Analytic Privileges

organised in folders


Attribute Views

Attribute View

What is an Attribute View?

Attributes add context to data.

Attributes are modeled using Attribute Views.

Can be regarded as Master Data tables

Can be linked to fact tables in Analytical Views

A measure e.g. weight can be defined as an attribute.

Table Joins and Properties

Join Types

leftOuter, rightOuter,

fullOuter, textTable

Cardinality

1:1

N:1

1:N

Language Column


Analytical View

Analytical View

An Analytical View can be regarded as a cube.

Analytical Views does not store any data. The data is stored in column store or table view

based on the Analytical View Structure.

Attribute and Measures

Can create Attribute Filters

Must have at least one Attribute

Must have at least one Measure

Can create Restricted Measures

Can create Calculated Measures

Can rename Attribute and

Measures on the property tab


Analytical View

Analytical View: Data Preview

There are three main views one can select from when previewing data.

Raw Data table format of data

Distinct Values graphical and text format identifying unique values

Analysis select fields (attributes and measures) to display in graphical format.


Calculation View (Scripting)

Calculation View

Define Table Output Structure

Write SQL Statement.

Ensure that the selected fields corresponds to previously defined Output table structure of the function.

Example :

SQL_A = SELECT MATNR, KUNNR, . FROM

SQL_P = SELECT MATTNR_KUNNR, FROM

TABLE_OUTPUT_STRUCTURE =

SELECT * FROM UNION

SELECT * FROM ;



Pre-Delivered Administration Console

Navigator

View

Properties

View

Administration

View


Agenda


2. Row Store

3. Column Store


5. Modeling

6. Q&A


Thank you!


Further Information on

SAP HANA and In-Memory Technologies

In-Memory Computing

http://www.sap.com/platform/in-memory-computing

Real-Real Time Business with HANA

http://www.youtube.com/watch?v=uUqtUw-m7mQ

SAP Community Network Topic Page

http://www.sdn.sap.com/irj/sdn/in-memory

SAP Community Forum

http://forums.sdn.sap.com/forum.jspa?forumID=491

The SAP NetWeaver BW SAP HANA Relationship

http://www.sdn.sap.com/irj/scn/weblogs?blog=/pub/wlg/21575

SAP HANA Ramp-Up Knowledge Transfer (login required)

http://service.sap.com/rkt-hana

SAP HANA Documentation (login required during ramp-up)

https://cw.sdn.sap.com/cw/community/docupedia/hana


No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation.

IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries, eServer, z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation.

Linux is the registered trademark of Linus Torvalds in the U.S. and other countries.

Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries.

Oracle is a registered trademark of Oracle Corporation.

UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group.

Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc.

HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C, World Wide Web Consortium, Massachusetts Institute of Technology.

Java is a registered trademark of Sun Microsystems, Inc.

JavaScript is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape.

SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, Clear Enterprise, SAP BusinessObjects Explorer and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries.

Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP France in the United States and in other countries.

All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.

The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.

This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. Please note that this document is subject to change and may be changed by SAP at any time without notice.

SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement.

SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence.

The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages.

2011 SAP AG. All Rights Reserved

bocx_hana_2011-04-13_v1.0

Documents

sap ag

memory appliance sap

memory computingsap

memory computingexample

memory databaseapplication

main memory

memory computingbackground

memory computing namingsap