data & analytics - session 4 - understanding storage options

61
James Brown, Business Development Manager Choosing the right data storage options with AWS

Upload: amazon-web-services

Post on 14-Jul-2015

406 views

Category:

Technology


1 download

TRANSCRIPT

Page 1: Data & Analytics - Session 4 - Understanding Storage Options

James Brown, Business Development Manager

Choosing the right data

storage options with AWS

Page 2: Data & Analytics - Session 4 - Understanding Storage Options

Object Storage

Block Storage

Connecting AWS Storage to On-premise

What are we going to cover?

Page 3: Data & Analytics - Session 4 - Understanding Storage Options

We are sincerely eager to hear your feedback on this

presentation and on re:Invent.

Please fill out an evaluation form when you have a chance. We are constantly producing more data

Page 4: Data & Analytics - Session 4 - Understanding Storage Options

We are sincerely eager to hear your feedback on this

presentation and on re:Invent.

Please fill out an evaluation form when you have a chance.

From all types of industries

Page 5: Data & Analytics - Session 4 - Understanding Storage Options

#1

Object Storage

●○○

Page 6: Data & Analytics - Session 4 - Understanding Storage Options

Amazon S3 Simple Storage Service

Page 7: Data & Analytics - Session 4 - Understanding Storage Options

99.999999999%

Durability

Page 8: Data & Analytics - Session 4 - Understanding Storage Options

Over 2 Trillion

Unique Customer Objects

Page 9: Data & Analytics - Session 4 - Understanding Storage Options

Over 1.1 million Peak Transactions Per Second

Page 10: Data & Analytics - Session 4 - Understanding Storage Options

Spotify adds over

20,000 tracks a

day.

“Amazon S3 gives

us confidence in

our ability to

expand storage

quickly while also

providing high

data durability.”

Emil Fredriksson,

operations director

Page 11: Data & Analytics - Session 4 - Understanding Storage Options

Amazon S3 Store objects from 1 byte to 5 terabytes

Uses standards based REST and SOAP protocols

Highly durable, fully managed

Designed to make web-scale computing easier

Amazon S3 - Unlimited Storage

Page 12: Data & Analytics - Session 4 - Understanding Storage Options

Amazon S3 Websites Easily handle peak loads at low cost

Pay as you go pricing

Inexpensive and Highly Scalable web hosting

Static Website Hosting

Page 13: Data & Analytics - Session 4 - Understanding Storage Options

Real Clear Politics

served up to 20x

normal demand

with S3 Websites

Page 14: Data & Analytics - Session 4 - Understanding Storage Options

CloudFront Pay as you go pricing with no long term

commitments

Caches content closer to your users to

lower latency and better performance

Integrated low latency content distribution network

Deliver content with lower latency with CloudFront

Page 15: Data & Analytics - Session 4 - Understanding Storage Options

Storage costs as low as 5½ cents per GB

Page 16: Data & Analytics - Session 4 - Understanding Storage Options

USE AMAZON S3

WHEN YOU NEED:

High Durability

Unlimited Storage Capacity

High Scale

High Volume Static Websites

Delivery via CloudFront

Page 17: Data & Analytics - Session 4 - Understanding Storage Options

Amazon Glacier Low-Cost Archiving Service

Page 18: Data & Analytics - Session 4 - Understanding Storage Options

Per GB / Month

Page 19: Data & Analytics - Session 4 - Understanding Storage Options

$125

Per TB / Year

Page 20: Data & Analytics - Session 4 - Understanding Storage Options

99.999999999%

Durability

Page 21: Data & Analytics - Session 4 - Understanding Storage Options

3-5 Hours

Data Retrieval

Page 22: Data & Analytics - Session 4 - Understanding Storage Options

Storage Costs

vs

Retrieval Costs

Page 23: Data & Analytics - Session 4 - Understanding Storage Options

Lifecycle Policies Create rules on object prefix

Create rules to also delete expired object

Automatically tier S3 data in Glacier

Using Lifecycle policies to move data

Page 24: Data & Analytics - Session 4 - Understanding Storage Options

Riverbed Whitewater Eliminates the burden of tape

Improve DR

Additional layer of security

Reduce storage requirements by 10 to 30x

A physical device can manage integration with Glacier

3rd Party Solution integration with Glacier

Page 25: Data & Analytics - Session 4 - Understanding Storage Options

USE AMAZON GLACIER

WHEN YOU NEED:

Long term storage of infrequently accessed data

Reduce costs of existing storage

High durability

Page 26: Data & Analytics - Session 4 - Understanding Storage Options

From theory into

practice with

Stuart Wright –

Director IT

Francis Hart –

Systems Architect

Page 27: Data & Analytics - Session 4 - Understanding Storage Options

SEGA

AWS Summit 2013

Stuart Wright - Director IT & Networks

Francis Hart – Systems Architect

Page 28: Data & Analytics - Session 4 - Understanding Storage Options

• SEGA overview

• Publishing & Development

• Games & Data

• Infrastructure past and present

• What we use

• Infrastructure close up

• S3

• Glacier

• CloudFront

Agenda

Page 29: Data & Analytics - Session 4 - Understanding Storage Options

SEGA overview

• Who – Games development and publishing

• What - Games – Platforms

• Where - Publishing headquarters

Page 30: Data & Analytics - Session 4 - Understanding Storage Options

Publishing & Development

• Game publishing, Game hosting

• QA, Logistics, Marketing, Sales, PR, etc

• Game development

• SEGA Europe

FM , Total War, Company of Heroes, Sonic mobile

Page 31: Data & Analytics - Session 4 - Understanding Storage Options

Games & Data

• Changing games

• Online components

• Analytics

• Game evolution

• Live systems

Page 32: Data & Analytics - Session 4 - Understanding Storage Options

Infrastructure Past and Present

• SEGA Datacentres

• Physical Hardware

• Flexibility and reliability

• Contracts

• Cloud IAAS

Page 33: Data & Analytics - Session 4 - Understanding Storage Options

What we use

• AWS storage options

• Application development and delivery

• Backup, Archive, Disaster recovery

Page 34: Data & Analytics - Session 4 - Understanding Storage Options

Infrastructure close up

• Shift to Virtualization and Cloud Services

• Consumable computing resource

• Consumption and production of data

• Retention of data

Page 35: Data & Analytics - Session 4 - Understanding Storage Options

SEGA S3

• Shared Storage for EC2

• Static content

• Configuration

• Interface and Integration with other

storage options

Page 36: Data & Analytics - Session 4 - Understanding Storage Options

SEGA S3

• Example:

EC2 Cloud Formation Cluster

Page 37: Data & Analytics - Session 4 - Understanding Storage Options

SEGA Glacier

• Backups

• Data Retention

• Legal Retention

• RAW Analytic Data Storage

Page 38: Data & Analytics - Session 4 - Understanding Storage Options

SEGA Glacier

• Example:

Log Retention (S3 Glacier)

Page 39: Data & Analytics - Session 4 - Understanding Storage Options

SEGA CloudFront

• Simple

• Cost Effective

• Reliable

Page 40: Data & Analytics - Session 4 - Understanding Storage Options

SEGA CloudFront

• Static Web Content

• Game Patching

• Media Content

Page 41: Data & Analytics - Session 4 - Understanding Storage Options

#2

Block Storage

○●○

Page 42: Data & Analytics - Session 4 - Understanding Storage Options

Amazon EBS Elastic Block Store

Page 43: Data & Analytics - Session 4 - Understanding Storage Options

Ephemeral storage is on every EC2 instance

Page 44: Data & Analytics - Session 4 - Understanding Storage Options

Amazon EBS Particularly suited for applications that require

a database, file system, or access to raw block

level storage.

Can be attached to a running instance

Network attached persistent block level storage volumes

EBS Drives can be moved between instances

Page 45: Data & Analytics - Session 4 - Understanding Storage Options

How do our customers use EBS?

Enterprises

Enterprise workloads are built

on block storage

Oracle, SAP, Microsoft

Gaming customers

Very high performance

databases.

500,000 IOPS for a new game launch

Social Network / Mobile Apps

Very high performance and consistent I/O for

NoSQL and relational DBs

Marketing / Analytics

High performance I/O databases

Page 46: Data & Analytics - Session 4 - Understanding Storage Options

Multiple drives on one Amazon EC2 instance

Page 47: Data & Analytics - Session 4 - Understanding Storage Options

10GB 1TB

Page 48: Data & Analytics - Session 4 - Understanding Storage Options

Provisioned IOPS Amazon EBS

Page 49: Data & Analytics - Session 4 - Understanding Storage Options

Provisioned IOPS P-IOPS is designed to run transactional

applications that require high and

consistent I/O

Workloads on Provisioned IOPs

Use Cases

Relational Databases

NoSQL Databases (eg MongoDB)

High Performance File Systems

Enterprise Workloads (eg CRM, MS Exchange, ERP)

Page 50: Data & Analytics - Session 4 - Understanding Storage Options

Provisioned IOPS - For steady I/O use Provisioned IOPS

- For bursty I/O, run the numbers

Optimized for price/performance

Page 51: Data & Analytics - Session 4 - Understanding Storage Options

EBS Snapshots Amazon EBS

Page 52: Data & Analytics - Session 4 - Understanding Storage Options

EBS Snapshots - Snapshots from EBS are stored in Amazon S3

- Rollback to a previous version

- Can be copied between regions

EBS Snapshots – Stored in Amazon S3

Page 53: Data & Analytics - Session 4 - Understanding Storage Options

USE AMAZON EBS

WHEN YOU NEED:

Filesystem for an instance NTFS, ExtFS, RAID, LVM…

Long-term persistent storage

Data changes frequently

Access to raw, unformatted

block-level storage

Page 54: Data & Analytics - Session 4 - Understanding Storage Options

#3 Connecting AWS Storage to

On-Premise Environments ○○●

Page 55: Data & Analytics - Session 4 - Understanding Storage Options

AWS Storage Gateway Connecting with on-premise

Page 56: Data & Analytics - Session 4 - Understanding Storage Options

AWS Storage Gateway - Low latency access to frequently accessed data

- Minimise the need to scale out on-premise

storage

- Snapshots allow access to old data

Up to 150TBs supported from a single Storage Gateway

Extend your on-premise storage

Page 57: Data & Analytics - Session 4 - Understanding Storage Options

USE AWS STORAGE GATEWAY

WHEN YOU NEED:

Synchronize data for disaster

recovery

Departmental fileshare

Backup your data

Page 58: Data & Analytics - Session 4 - Understanding Storage Options

Oracle Secure Backup Module

Page 59: Data & Analytics - Session 4 - Understanding Storage Options

Oracle Secure Backup (OSB) Cloud

Module allows customers to backup

Oracle Databases directly to Amazon S3

using the Oracle Recovery Manager

(RMAN)

Backup Oracle into Amazon S3

Page 60: Data & Analytics - Session 4 - Understanding Storage Options

Restore times

reduced from 15

to 2½ hours

Took less that 1

hour to configure

per database

Offered significant

improvements in

durability of

backups

Page 61: Data & Analytics - Session 4 - Understanding Storage Options

IT’S ALL ABOUT

CHOICE PERFORMANCE-ORIENTED

COST-ORIENTED