effizientes speichern von cold-datakonferenz-nz.dlr.de/pages/storage2016/present/1... · title: how...

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |

Effizientes Speichern von Cold-Data

Dr. Dirk Gebh Storage Sales Consultant Oracle Deutschland


Program Agenda

Cold-Data

OHSM Introduction

Use Case – Removing Cold Data from Primary Storage

OHSM and Cloud

OHSM new Features

1

2

3

3

4

5


Program Agenda

Cold-Data

OHSM Introduction


OHSM and Cloud

OHSM new Features

1

2

3

4

4

5

Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | 5

Infrequent access but significant efford

Cold-Data

Svalbard Global Seed Vault

Example:

=> Log Term Preservation !

„forgotten“ vegetables


80% of Data Never Accessed After 90 Days

6


It depends...

When Is Long Term Preservation Needed?

1. Delete „trash data“ as soon as possible! Don‘t store, don‘t backup, don‘t archive That is most efficient for „trash data“ Deduplication will not solve this problem !

2. Select „worth data“ – It can be hot or cold Put Right Data on Right Storage at Right Time


Example: Aircraft construction data

Long Time Preserving Data up to 80 Years required

Which storage medium and architecture will meet this requirement?


Seagate Constellation ES.2 StorageTek T10000D

9

Bit Error Rate of Tape is 10,000 Times Lower Than Disk

How long is data transferred before an uncorrectable error?


Seagate Constellation ES.2 StorageTek T10000D

10

Bit Error Rate of Tape is 10,000 Times Lower Than Disk

9 days Equivalent to 40 RAID rebuilds per year per disk!

Higher disk capacities = Longer rebuild times = Increasing data loss exposure

165 years


“You can have all the tapes in the world, but that won't do you any good without hardware to read them.” – Jory Prum, LucasArts veteran

11

http://www.polygon.com/2015/1/27/7921837/grim-fandango-remastered-interview-double-fine-disney-lucasfilm-sony


Don’t forget the Cables, Interface and Drivers

12

Old Device Cable Interface Driver Computer

Do Not Ignore It … Do Validate it! Do Migrate it!

Keep up with Technology Changes

?


All Data must be Accessable during Life Cycle!

13


Long Term Preservation

Make sure data is unchanged

Make sure data is valid

?

Checksum

protection

Re-archive

if needed


Program Agenda

Cold-Data

OHSM Introduction


OHSM and Cloud

OHSM new Features

1

2

3

15

4

5


Oracle Hierarchical Storage Management

Right Data

Right Time

Lower Costs, Improve Efficiencies, Ensure Access

Right Storage


Archiving with Oracle HSM

• End to end data integrity validation (DIV)

• Fixity

• Open formats

• Product integrations

• High Availability

• Remote Archiving

• OKM support

Ensure Access

• Policy based movement

• Limitless scalability

• Horizontal scaling

• Automated media migration

• Backup and Archiving

• Tiered Storage

• Private Storage Clouds

• Object Cloud Storage

• Oracle Archive Cloud

Lower Costs Improve Efficiencies


Save 77% by Archiving Rarely Used Data

$1,600,000

$363,000

Backup Only Backup and Archive

Archive Software

Archive Storage

Backup Software

Backup Disk

Primary Disk

18

Save 20x on Backup Software – This savings alone pays for entire archive solution!


Manage Tiered Storage with Oracle HSM

Ap

plic

atio

ns

Op

en

Stac

k Sw

ift

AP

I St

and

ard

PO

SIX

File

Sy

stem

AP

I

Sto

rage

Ab

stra

ctio

n L

aye

r (O

racl

e H

SM)

Primary Storage

Archive Storage

19


Improve Efficiency with Policy-based Data Movement

20

Hierarchical Storage Management

and Archive

Management

Types of Policies:

• Archiving, staging, releasing

• Number of copies • Preferred media type • Recycling tape volumes • Media migration • DIV • Fixity

Policy Characteristics:

• Name pattern • Owner • Group • Size • Access age


Application

How Oracle HSM Simplifies Management

Application writes file to StorageTek QFS file system

Archive copies created based on policy

Application requests file and the file is retrieved from Oracle HSM

1

2

3

Disk Cache Archive Devices

21


Application

Oracle HSM provides transparency for applications

Application only knows that it wrote a file

And brought it back

1

2

Disk Cache Archive Devices

22


• >1 billion files per file system

• Users with 7 billion+ files

• Scale to 1EB+

• Multiple users with 40+PBs

• Achieve 10s of GB/s by scaling horizontally

• Ingest at 5+ GB/s

Achieve Massive Scale with Oracle HSM

Volume Capacity Throughput

23


Increase Data Ingest Rates by 10X

• Ingest >1 PB of data per day

• Use 10 servers to archive data instead of 1

• Perform faster migration of data to new tape drive technology

• Enable the use of commodity servers

Tape I/O Horizontal Scalability

24


License Cost Effective Scale Out Architecture

Enterprise Scalability: Horizontal Tape I/O

QFS Clients

Metadata

Server

(MDS)

Primary

Disk

Tape Storage

Tier

Yellow = metadata / control path

Black = disk data path

Red = tape data path

•Metadata controlled by MDS

•MDS and QFS clients read/write

to disk

•I/O for tape read/write comes

from MDS and QFS clients

•No bottleneck – I/O load spread

evenly across all servers


Oracle HSM Automated Media Migration

26

Leverage higher density formats with the efficiency of automatic migration

Migrates multiple data streams without impacting production

Seamless file system management

Policy-based for automatic, behind-the-scenes migration


Oracle HSM Automates Data Integrity Validation

27

Set policy-based

DIV audits

Verify integrity on each read,

write, or any time

Dedicate tape drives for

DIV audits

Self-heals Automatically

Supports StorageTek T10000C/

T10000D and LTO 5/6/7


Oracle HSM Ensures Long Term Access with Fixity Support

Supports industry standard checksums: MD5, SHA-1, SHA-256, SHA-384, SHA-512

Verifies checksum on each read, write, or copy of a file

Validates file

remains unchanged


Support movement of data between different systems

Provide protection for data recall separate from the archive system

Simplify migration from one storage medium to another

The Need for Open Formats – Ensure Access

29


• Primary Use case: File interchange

• Describes layout of data structures stored on tape

• Introduced in 2010

• ISO standard established in 2016

• Primary Use case: Long term storage

• Collects many files into one archive data set

• Standard established in 1988 and 2001

Oracle HSM supports Open Formats

30

Linear Tape File System (LTFS) Tape Archive (TAR)


Ensure Future Access with LTFS Format Support

FILE

FILE INDEX

LTFS Format

Oracle HSM Archive

FILE

FILE INDEX

LTFS Format

Export Import

LTFS Access

Import LTFS formatted tapes

Export archive data onto LTFS formatted tapes

31


Oracle HSM Integrates with WebCenter Content to Ensure Access Throughout Data Lifecycle

WebCenter Content

• Ingest • Access • Collaboration • Search • Reuse

Tiered Storage • Cost Effective • Dynamic • Policy Based • Storage

Virtualization

Storage Experts

Storage Abstraction

Content Experts


Program Agenda

Cold-Data

OHSM Introduction


OHSM and Cloud

OHSM new Features

1

2

3

33

4

5


Data on Primary Storage Large installed NAS infrastructure – Problem: 80% Cold Data!

Slow NDMP Backup (full / incremental)

all data: hot and cold

Cold data Hot data


Removing Cold Data from existing Primary Storage Use Case

Existing In

frastructu

re

Oracle Partner Offering

Oracle Offering


Program Agenda

Cold-Data

OHSM Introduction


OHSM and Cloud

OHSM new Features

1

2

3

36

4

5


Ap

plic

atio

ns

Stan

dar

d P

OSI

X F

ile S

yste

m A

PI

Primary Disk

Capacity Disk

Digital Tape

2. After 30 days of inactivity make copies to tape and/or cloud

3. Make additional copies to tape

Without automation everything stays on expensive primary disk…

With automation via policy…

4. After 1 year remove from disk archive

Automation Is Key Enabler Simplifying Administration & Improving Storage Efficiency

Cloud

1. After 5 minutes create a copy to tape for Data Protection

37


OpenStack Swift Object Store Ingest & Access

Private Cloud Infrastructure Solution

Tape Disk Flash

Oracle Hierarchical Storage Manager

Swift

Object store with support for OpenStack Swift

RESTful API

Web interface for tape storage

Make low cost tape easily accessible to web

users

Enable object storage

Custom metadata, no namespace limitation,

near infinite scalability

Eliminate backup software

Deliver a cold storage service directly to end

users

38

http://medianetwork.oracle.com/video/player/3653848713001


Create On-Premise Storage Clouds with Oracle HSM and OpenStack Swift

39

Tape Archive

Primary Disk

Ap

plic

atio

ns

Op

en

Stac

k Sw

ift

RES

T A

PI


Cloud Object Storage – Storing on Disk Storage Nodes !

• OpenStack Swift architecture

• Proxy: authenticates Users and routes requests to the appropriate storage nodes

• Account: tracks the containers associated with an account

• Container: tracks the objects associated with a container

• Object: stores objects in the file system

40

3 copies to different storage nodes

GET / PUT / DELETE


Cloud Object Storage with OHSM Backend

single data copy

up to 4 data copies handeled by OHSM system

External Swift Clients

GET / PUT / DELETE


Putting It All Together

Cloud Tier

QFS Clients

Metadata Server

On Premise Primary Disk

On Premise Tape Storage

(control path)

Data path

NFS Gateway

Remote Site / Private Cloud

Store data on the optimal storage device in the optimal location

Applications / Data Source

Archive

Application Interfaces (NFS, CIFS, HTTP, FTP, WebDav, OpenStack Swift)

42

Oracle Cloud Software Appliance


Oracle HSM Featured at Oracle OpenWorld! Announced: Oracle HSM integrates with the cloud for huge cost savings

43


Oracle HSM Support for Oracle Storage Cloud - Archive Service – Coming Soon! Breakthrough cost for off-site data protection

Archive

Disk Storage

Archive Storage Amazon Glacier

$0.007- 0.013* $/GB/Mo.

* Region dependent

Google Nearline

$0.011 $/GB/Mo.

$0.001 $/GB/Mo.

$12,000/PB/Yr.

Oracle Storage Cloud Service – Archive

Service

44

Oracle HSM


Program Agenda

Cold-Data

OHSM Introduction

OHSM and Cloud

OHSM new Features

1

2

3

45

4


Strong Investment in Oracle HSM

Data integrity validation (DIV)

WORM enhancements

Stager enhancements

Horizontal scaling

Cloud

Automated DIV

Linear tape file system (LTFS) import/export

Media migration

Fixity

Expanded LUN size

Extended attribute metadata

5.3 6.0 6.1 - New!

46

Up to 5.2

Archive

Virtualization

Automation


Media Migration

• Accelerates migration

• Frees CPU cycles and personnel

Fixity Enhancement

• Supports standard checksum algorithms

• Verifies file integrity throughout entire lifecycle

Extended Attribute Metadata

• Improves file system performance

Expanded LUN Size

• Supports hardware advancements

• Increases scalability

New Advancements in Oracle HSM 6.1

47


Solve Migration Challenges with Oracle HSM Automated Media Migration

48

Migration Challenges

Oracle HSM Automated Media Migration

1. Policy-based migration

2. Eliminates data staging

3. Enables parallel migration

4. Reduces resource burden

5. Fastest path to single format

• Time: months or years to complete

• Cost: hardware, network, and personnel

• Complexity: manage multiple formats


Improved Media Migration with Oracle HSM

49

Migration Challenges

Oracle HSM Automated Media MigrationTM

1. Memory-Assisted Copy

2. StorageTek Direct Copy

• Time: months or years to complete

• Cost: CPU hungry with little business value

• Complexity: manage multiple formats


Accelerate Migration with Oracle HSM Automated Media MigrationTM

50 50

• Copy to disk and then to tape • Slow, CPU-intensive migration

Current migration method

• Copy directly from tape to tape • Fast/parallel migrate unlimited streams

T10000A/B/C/D LTO3/4/5/6 TS1130/40/50

T10000D

HSM ctrl

dat

a

With Oracle HSM and StorageTek Direct Copy

With Oracle HSM Memory-Assisted CopyTM

• Copy to memory and then to tape • Fast migrate up to four streams

HSM Disk ctrl

data

ctrl

d

ata

ctrl

d

ata


Migrate All Data or Repack a Portion of Data

51

• Repack only active source files sequentially to a single cartridge

• Migrate entire contents of multiple cartridges to a single cartridge

Migrate Repack

scsi

/sta

tus

dat

a

Active

Expired

Active

Expired

Active

Expired

Active Active Active

scsi

/sta

tus

dat

a


Migrate All Data or Repack a Portion of Data Automated Media Migration Feature at Scale

52

• Repack only active source files sequentially to a single cartridge

• Migrate entire contents of multiple cartridges to a single cartridge

Migrate Repack

Active

Expired Active

Expired Active

Expired


scsi

/sta

tus

dat

a

Active

Expired Active

Expired Active

Expired

Active

Expired Active

Expired Active

Expired




Fixity Enhancement

End-to-end fixity Verify checksum

for each read /write /copy

Verify checksum Ingest file and

checksum

53

Standard checksums: MD5, SHA-1, SHA-256, SHA-384, SHA-512

Solaris library utilized for hardware acceleration

Checksum stored in extended attribute


Dataflow of Setting and Using Fixity

• Ingest Data

•Write to disk cache

•Run ssum or API

•Calculate the hash and store

•OR calculate and compare with given hash

•Update attributes

•Access file for archive

•Validate

•Access file for stage

•Validate

2 1 4 3 5 6

Archive to tape

•Stage to disk cache


Improved Performance for Extended Attribute Metadata

55

Metadata Tier

Disk Cache Tier

Archive Tiers

Inodes / POSIX Metadata

Extended Metadata Fast Access Data

Archive Data

Oracle HSM 6.0

Metadata Tier

Disk Cache Tier

Archive Tiers

Inodes / POSIX Metadata

Extended Metadata

Fast Access Data

Archive Data

Oracle HSM 6.1


Max LUN Size Increased 8x

56

16 TB LUN 4 PB

128 TB LUN 32 PB

Prior to 6.1

Oracle HSM 6.1


Oracle HSM 6.1 HW / SW Support Updates

Solaris 11.3 LTO7

Tape Drives ACSLS 8.4


Lower Costs

Improve Efficiencies

Ensure Access

Archiving With Oracle HSM

58

Put Right Data on Right Storage at Right Time

Cold - Data


Safe Harbor Statement

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

effizientes speichern von cold-datakonferenz-nz.dlr.de/pages/storage2016/present/1... · title: how...

Documents