integrating on-premises enterprise storage workloads with aws (ent301) | aws re:invent 2013
DESCRIPTION
AWS gives designers of enterprise storage systems a completely new set of options. Aimed at enterprise storage specialists and managers of cloud-integration teams, this session gives you the tools and perspective to confidently integrate your storage workloads with AWS. We show working use cases, a thorough TCO model, and detailed customer blueprints. Throughout we analyze how data-tiering options measure up to the design criteria that matter most: performance, efficiency, cost, security, and integration.TRANSCRIPT
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
ENT301: Integrating On-Premises Enterprise
Storage Workloads with AWS
Harry Dewedoff, NASDAQ OMX
Yinal Ozkan, Amazon Web Services
November 14, 2013
What this session is not?
• Vendor feature , technology comparison
• Vendor / product discussion
• Cloud-only workloads
• Individual / retail storage options
Agenda
• Section 1: What is new with enterprise storage?
• Section 2: On-premises storage cloud integration
• Section 3: NASDAQ OMX and cloud storage – History
– Options provided to NASDAQ OMX teams
– PoC
– NASDAQ OMX technology selection
– Managing operations
– Security
• Section 4: Evaluating a sample storage workload
WHAT IS NEW WITH ENTERPRISE
STORAGE
Section 1
Storage Services
Scalable and durable
high performance cloud storage
Compute Storage
AWS Global Infrastructure
Database
App Services
Deployment & Administration
Networking
Amazon Glacier Low-cost Archive Storage in the Cloud
Amazon Elastic Block Store
Persistent Block Storage for EC2
AWS Storage Gateway
Corporate File Sharing and Seamless Backup
of Enterprise Data to Amazon S3
Amazon S3 Redundant, High-Scale Object Store
Amazon S3 Standard Storage Is…
Designed to provide 99.999999999% durability and
99.99% availability of objects over a given year.
If you put 10,000 objects in S3 you can expect to
lose 1 object every 10,000,000 years
Common Data Storage Challenges
Primary
Block
Storage
Primary File
Storage
Archival
Storage
Disk Based
Backup
Storage
Tape
Infrastructure &
Management
Replicated
Storage for
Disaster
Recovery
Offsite
Locations
Geo-
Resilience
Traditional On-Premises Solutions
Internet Web
Services API
HTTP(S)
Block File
AWS
Cloud
Customer
Data
Center
Storage
Use
Cases Archive Backup Disaster Recovery
Next Generation Enterprise Storage
AWS Direct
Connect
Next Generation Enterprise Storage Benefits
Why Next Generation Enterprise Storage
with AWS?
Next Generation Enterprise Storage Benefits
Amazon Storage Tiers (S3 RRS Glacier)
ON-PREMISES STORAGE CLOUD
INTEGRATION OPTIONS
Section 2
Cloud Data Tiering Options
• Option 1: Software Integration
• Option 2: Plain file transfer
• Option 3: AWS Storage Gateway
• Option 4: Enterprise storage gateways
Option 1: Software Integration
1. Configure on-premises backup software to use S3
2. Backup and restore directly from software
3. Backup server communicates with cloud (S3) over Internet links
4. Use software-based encryption, compression, dedupe, backup management tools
5. Check security / integrity / functionality / performance / operations / speed
Tapeless Cloud Backup
Virtual Physical Backup Server
EU West
Region
(Ireland)
US West
Region (N.
California)
S. America
Region
(Sao Paulo)
US West
Region
(Oregon)
APAC
Region
(Singapore)
AWS
GovCloud
Region (US)
Japan
Region
(Tokyo)
US East
Region (N.
Virginia)
Option 2: Plain File Transfer
1. Store target file(s) on a file share.
2. Configure policies on target Amazon S3 buckets
3. Encrypt / compress data sets on premises
4. Transfer files via regular file transfer (Amazon S3, SFTP, SCP, FTP etc). Or use massively parallel file-transfer options
5. Retrieve encrypted file from Amazon S3 using using the same options
6. Test integrity / security / operations / performance
7. Add parallelization for performance optimization
Plain File Transfer Diagram
Encrypt Output File
Encrypted Data Is Written to on
FileShare
Customer Data Center AWS Region
Transfer / Retrieve
Encrypted File to
Amazon S3 Using
Regular File Transfer
Internet
or Direct
Connection
Create Output
1.Store Backup File on FileShare
Option 3 : Use AWS Gateway
• Integrates on-premises IT environments with cloud storage
for remote office backup and DR
• Utilizes a virtual appliance that sits in customer datacenter
• Exposes compatible iSCSI interface on front end
• Provides low-latency on-premises performance
• Asynchronously uploads data to AWS where it is stored in
Amazon S3 as Amazon EBS snapshots
Option 3 : Use AWS Gateway
• Support for Windows and RedHat iSCSI initiators
• Point-in-time snapshots accessible locally and from Amazon EBS
• Encryption via SSL and Amazon S3 server side encryption
• Snapshot scheduling
• WAN compression
• Supported in all public regions
• Bandwidth throttling
• CACHED VOLUMES / VTL SUPPORT
Backup
Corporate File
Sharing
Cacheable data like departmental file
shares, home directories
Store files in Amazon S3, while
keeping recently accessed data on
premises
iSCSI interface compatible with traditional backup applications (Netbackup, Tivoli, Backup Exec, etc.)
Store backups in Amazon S3, keep recent backups on premises
Gateway-Cached Volumes – Key Use
Cases
Gateway-Cached Volumes Architecture
VTL Gateway – Archive Your Data to Glacier
Amazon
S3 Corporate Data Center
Amazon
Glacier AWS Cloud
App/DB/SAN/NAS
Backup
Software
VTL Gateway
Corporate Data Center
Archive to AWS
versus
Traditional
Approach
SAN Disk
Backup
Offsite Tape
Storage Tier 2
Storage
Architecture of VTL Gateway
NetBackup /
CommVault /
[Backup
Software] On-premises
Host
AWS Storage
Gateway VM
Direct Attached or
Storage Area Network Disks
(for internal cache & buffer storage)
iSCSI SSL
AWS Storage
Gateway
Service
Amazon S3
Production
Systems
AWS Storage Gateway for
Virtual Tape Library
Customer Data Center
Media Changer
Tape Drive 1
Tape Drive 2
Tape Drive N
…
Amazon
Glacier
NetBackup /
CommVault /
[Backup Software]
on EC2
AWS Storage
Gateway on
EC2 AMI
AWS Storage Gateway on EC2
for Disaster Recovery or Data Mirroring
EC2
Application
VTL (1500 tapes) VTS (unlimited
tapes)
Tape Ingestion into Glacier
VTL Gateway Characteristics • Single point virtual appliance for archive use case and for customer in need for a
simple VTL Interface
• Each virtual appliance can manage up to 140TB in VTL (Virtual Tape Library) but unlimited in VTS (Virtual Tape Shelf)
• Cost of each appliance could be around $125
• Ease of mgmt. when data grows in multi PBs per year
• Current ingest rate is about 3-5 TBs per day per gateway (option to use multiple GW in a cluster environment)
• Data passed through VTL gateway is not deduped (ease of restore and reuse) – suited for long-term archive
Bottom line: Archive, fixed content, entertainment, scientific, social networks, compliance and
unstructured data requirements generate much of today’s tier 3 storage demand and have become the
primary drivers for tape storage demand. With Amazon Glacier and VTL gateway, AWS is very well
poised to help customers leverage the benefits of cloud storage!
Option 4
Enterprise Storage Options on AWS
How Does It Work?
• Enterprise storage gateway presents itself as – CIFS/NFS file share
– iSCSI endpoint
– File archive via file tiering policies from filers
– Policy-based routing from FC switch
• Gateway cache data locally, tiers data back to Amazon S3-based on policies after dedupe, encryption, compression
• Data is accessible to all other gateways
Design Considerations
• Ingest / restore / access rates
• Deduplication / compression rates
• Throughput rates
• High availability / integrity
• Restore in the cloud option
• Data transfer costs
• Security integration
Option 4: Enterprise Storage Gateway
Corporate Data Center AWS Region
Enterprise Storage
Gateway Dedupes,
Compresses and
Encrypts Data and
Then Moves Data
to AWS
AWS Direct
Connect
Enterprise Storage Gateway
Block File
AWS
Cloud
Customer
Data
Center
Storage
Use Cases Archive Backup
Disaster
Recovery
Internet Web
Services API
HTTP(S)
AWS Direct
Connect
Amazon Glacier
Gateway Appliance/ AWS Storage Gateway
Amazon S3
• Block storage: – Data organized as an array of unrelated blocks
– Host file system places data on disk: Microsoft NTFS or Unix ZFS
– Structured data is predicted to grow at 18.7% CAGR until 2018
• File storage: – Unrelated data blocks managed by a file (serving) system
– Native file system places data on disk: EMC UxFS or NTAP WAFL
– Unstructured data is predicted to grow at 47.3% CAGR until 2018
• What is object storage?: – A new data access, data storage, and data management model
• API access to data vs. traditional block or file system access
• Metadata driven, policy-based, self-managing storage
• No host overhead for storage functions
– A system that stores virtual containers that encapsulate the data, data attributes, metadata, and Object IDs
Block vs. File vs. Object Storage
Internet Web
Services API
HTTP(S)
S3
SDK: Application developers can
leverage the Amazon S3 SDK for
custom application integration
Plug & Play: IT can bridge on-
premises environments with familiar
storage interfaces and methodologies
via cloud storage gateways
Cloud Storage: SDK or Plug & Play?
S3
AWS Storage Gateway
Example Deployments for
Enterprise Storage Gateways
NFS / CIFS Archive
• Offload stale data to low-cost cloud storage
• Scale instantly as needed
• Integrate seamlessly with standard archiving, tiering solutions
• “Cloud drive” just another disk target, accessible anywhere
Global Deduplication
Encryption
Multiple Gbps
Case Data
Analysis Data
Administrative Data
Global Online Access
Cloud Storage
NFS / CIFS Backup
• 100s TB raw local cache
• Eliminate tape from infrastructure
• Slash time and manpower for data protection
• Global deduplication
• Military-grade encryption
• Seamless integration with major backup tools
• Restore anywhere, virtual or physical
Global Deduplication
Encryption
Multiple Gbps
Global Online Access
Cloud Storage
Panzura + Amazon S3
• Eleven 9s durability
• Four 9s vailability
• Highly-secure sites
• Unlimited scale
• Commodity pricing
• Glacier option
• Multiple-geos
• Largest public cloud
▪ Global file system
▪ Military-grade encryption
▪ Global deduplication
▪ CIFS/NFS
▪ Global file locking
▪ Local caching/pinning
▪ AD integration/ACLs
▪ Snapshots
NextGen Enterprise Storage
S3 Glacier
Local file system interface to the SOAP / REST
API used by the Amazon storage cloud platform
Virtual namespace to seamlessly integrate local
and Amazon cloud storage cloud for users
1
SOAP / REST
CIFS / NFS
2
Automatically identify inactive and other
appropriate files to store in the Amazon storage
cloud
3 4
Migrate files to Amazon storage cloud platform
without disrupting user access or causing
downtime 5
Encrypt every file stored in the Amazon storage
cloud for data security
File Archive with S3
Local file system interface to the SOAP / REST
API used by the Amazon storage cloud platform
Virtual namespace to seamlessly integrate local
and Amazon cloud storage cloud for users
Automatically identify inactive and other
appropriate files to store in the Amazon storage
cloud
Migrate files to Amazon storage cloud platform
without disrupting user access or causing
downtime
Local Storage
Unified storage
• Files, databases, & VMs
• NAS & SAN
Unified Storage with Amazon S3
• Thin snapshots
• File & Bare Metal
CTERA Appliance
Mobile
Devices
LOCAL
BACKUP
NAS
• NFS, CIFS,
AFP, FTP, rsync
• AD Integration
CLOUD
STORAGE
• Pay as you go
SECURE
REPLICATION
• AES-256 + SSL
REMOTE
MANAGEMENT
• De-duplicated
• Bandwidth controlled
• RAID 0/1/5/6 • Administration
• Central logging • Automated
• Firmware updates
• Secure & redundant
Customer Location AWS Cloud
• Exchange, SQL,
AD Recovery
• Incremental • Incremental
• Thin snapshots
Roaming Laptops
Workstations
Servers
• Flexible backend
options
• Partner dashboard
• Compression
NextGen Enterprise Storage
Fast File Transfer into AWS
Supported Access
• NFS
• CIFS
• WebDav
• FTP
• Eliminate the need for a cloud storage gateway
Maintain all ECM capabilities
• Automatic version control
• Rules & workflow
• Full-text search
• Policy enforcement
NextGen Enterprise Storage
NASDAQ OMX AND CLOUD
STORAGE
Section 3
NASDAQ OMX and AWS
• History
• Options provided to NASDAQ OMX teams
• Evaluation of architectural options and
NASDAQ OMX technology selection
• Managing operations for cloud backup
• Security
History of Relationship and FinQloud
Sample FinQloud Workflow – How Does It Work?
Customer data sets are ingested at Nasdaq hosted inbox via
secure file transfer
Nasdaq preprocesses data (e.g., trade data) at Nasdaq
Facilities
Split trade data records into chunks (about 1M records per
chunk)
Each file is encrypted with AES-256 / FIPS complaint system
Custom encryption is applied (e.g. per client/per day, random
initialization)
Sample FinQloud Workflow – How Does It Work?
A custom metadata header file is attached to encrypted chunk.
Metadata is signed via SHA-256
File chunks are uploaded to Amazon S3/R3 – each FinQloud
customer gets a new AWS account, a new bucket
WORM or regular Amazon S3 buckets can be utilized
Search and retrieve functionality is performed by Amazon Elastic
MapReduce (AWS-managed Hadoop) for performance
Each customer gets an assigned Amazon Virtual Private Cloud (VPC)
Amazon EMR request key files from Nasdaq hosted host security files
Sample FinQloud Workflow – How Does It Work?
For cloud-based processing (e.g. reporting) trade data chunks,
and files are decrypted in the memory with Amazon EMR
Data is never in clear-text in transit or at rest
Once the jobs are completed, data sets are re-encrypted again
and either written to Amazon S3/R3 or shipped back to
Nasdaq
Results data-sets can be decrypted at Nasdaq facilities via
HSM hosted keys or customer can integrate their PGP keys
for asymmetric encryption and download the results
Selection of Enterprise Storage Workload
Type for First Cloud Project
• Cloud first strategy
• Tier 3 storage vs backup workloads
• Selection of backup as the first use case
– Greater control of the implementation / outcome
– Less risk as it was backup data vs production data
• Backup technology options
Selection of Architecture and Technology
• Why Riverbed Whitewater was chosen – Ease of deployment
– Strong vendor support model
– Good integration and compatibility
– On-premises cache repository for backup platform
– Inline dedupe, compression
– Data encryption at appliance before data transfer
– Listed company
NASDAQ OMX Cloud Backup Architecture
How NASDAQ OMX Runs Storage
Operations for the Cloud
• No major changes since Netbackup is integrated
with Riverbed Whitewater
• Riverbed appliance looks like a standard disk
pool to the media server
• NBU policies altered to make us of RB disk pool,
which in turn sends data into Amazon S3
How Does Security and Isolation Work?
AWS standard security +
– Data is always encrypted at transit and at rest
– Keys are stored at Nasdaq facilities
– Nasdaq InfoSec department performed security
review and provided sign off of security
measures.
Integration Points
• All AWS services+
– End-to-end isolation
– End-to-end encryption
– Separation of duties by hybrid security
– Patented worm: R3
– Dynamic key management
PROOF OF CONCEPT EXAMPLE
WITH TIER 3 STORAGE
Section 4
• NASDAQ OMX is planning to leverage scale, durability
and the cost advantages of cloud-based storage solutions
• In addition to backup and archive workloads, testing Tier-
3 storage on the cloud makes sense for NASDAQ OMX
due to the ratio of spend on Tier-3 storage (compared
with backup/archive workloads)
• There is a management initiative to leverage cloud
technologies at NASDAQ OMX
Why Tier-3 Cloud Storage Solution?
AWS Storage Gateway is a service that connects an on-
premises software appliance with cloud-based storage to
provide seamless and secure integration between on-premises
IT environment and AWS' storage. AWS Storage Gateway is:
– Native AWS offering
– Scalable
– Cost effective
– Controllable from AWS Management Console
– Promising roadmap
Why AWS Storage Gateway?
• The objective of this Proof-of-Concept (PoC) is to provide a
high-level analysis and checklist of all elements and attributes
necessary to successfully implement a Tier-3 Cloud Storage
Gateway.
• The PoC is the initial step prior to undertaking a detailed system
design and implementation and is intended to function as a
prototype system. It is meant to demonstrate key technologies,
as well as provide an environment for experimentation and
evaluation. The design and implementation of a POC, while very
detailed and organized, does not serve as a replacement for a
complete system analysis and design.
Objective of Proof-of-Concept
AS-IS Architecture
PoC Architecture
VMware:
• VMware ESXi Hypervisor (v 4.1 or v 5)
• 4 virtual processors assigned to the VM
• 7.5 GB of RAM assigned to the VM
• 75 GB of disk space for .ova installation and system data
External Connectivity:
• Ports 80 and 443 are used by the vSphere client to communicate to the ESXi host.
• Port 80 is used when you activate your gateway from the AWS Storage Gateway console.
• Port 3260 is the default port that your application server uses to connect to iSCSI targets.
PoC AWS Storage Gateway Requirements
VMware:
• VMware ESXi Hypervisor server x 2 (existing servers can be used)
Ethernet NICs:
• Existing NICs can be leveraged; dual NIC tests are recommended
Ethernet Switch:
• Existing network switches can be leveraged; isolation and
bandwidth allocation recommended
Connectivity:
• Existing AWS connectivity or Internet connections can be leveraged
PoC On Premises Hardware Requirements
Provision 2 VMware physical servers (hosts)
Download the AWS Storage Gateway software at
http://console.aws.amazon.com/storagegateway
Allocate on-premises storage for active data
Activate gateway and select an AWS region
Create and mount iSCSI volumes
Provision Ethernet Cards and Network Infrastructure
Test Primary Storage Access over iSCSI on new volumes
Configure volume management to copy data sets from
existing volumes to new volumes
Estimate the approximate data plan to write on a daily basis.
It is recommended to allocate at least 150 GBs.
Sizing On-Prem Storage Upload Buffer
Backup use cases: at least the size of upload buffer
File share use cases: 20% of current storage
Cache Storage Sizing On-Prem Storage
• Solaris iSCSI initiators
• Cache storage is durable store
• Allocate separate disks for cache storage and
upload buffer
• Quick format vs. full format of drives
• Virus scanning
Additional Configuration Considerations
• By January 31 this PoC should be completed.
• This POC is not an open-ended project. It is a limited
implementation for a fixed period of time. The project
duration will be a direct result of the project objectives.
• This duration includes project time necessary to plan,
design, and implement the POC system.
• Project management teams should actively work to
control scope.
PoC Schedule
Amazon Web Services
• High-level design
• AWS Storage Gateway installation and configuration
• AWS cloud components configuration
• Assistance in iSCSI and VMware configurations
• Provide test plans
• Assistance is test execution
• Delivery of final analysis
NASDAQ OMX
• On-premises hardware installation (NICs, hosts)
• VMware installation and configuration
• On-premises network configuration (Switches, VLANs etc.)
• Providing test targets, assistance in test plans
• Execute tests
• Assistance in final analysis
PoC Resources and Responsibilities
• Virtualization hardware (VMware physical host)
and network systems are not for production
• Hardware and network limitations that are not
critical for success will not be addressed
• Compromises will be made to accommodate the
smaller scale of the implementation
Limitations of PoC
• Access tier-3 volumes in cloud over iSCSI links
• Failover / failback / redundancy tests
• Reliability tests
• Performance tests
• Security controls
• Manageability tests
• Cost analysis
PoC Test Plans
• Meet test targets 80% and over
PoC Success Criteria
• Nasdaq interviews
• Issue review
• Implementation review
• Reliability review
• Scalability review
• Performance review
• Security review
• Manageability review
• Cost review
• Final project analysis
Results Analysis
Please give us your feedback on this
presentation
As a thank you, we will select prize
winners daily for completed surveys!
ENT301