(stg401) amazon s3 deep dive & best practices

© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Omair Gillani, Sr. Product Manager

Hisham Baz, Solutions Architect

October 2015

Amazon S3: Deep Dive and

Best Practices

STG 401

New for 2015…

Cross-region

replicationAmazon S3 Standard-IA

AWS CloudTrail support

for Amazon S3

Amazon CloudWatch

metrics for Amazon S3

VPC endpoint

for Amazon S3

Amazon S3 bucket

limit increase

Event notifications

Read-after-write

consistency in all regions

Video sharing service

VidShare


VidShare

Launch

V2 Optimize on cost

V3 Expand globally

V4 Enterprise enablement


VidShare

- Thumbnail

- Update Index

- Update WebApp


Event

MetadataThumbnail

logs

VidShare

Amazon S3 event notifications

Events

SNS topic

SQS

queue

Lambda

function

• Notification when objects are

created via PUT, POST, Copy, or

Multipart Upload, DELETE

• Filtering on prefixes and suffixes

for all types of notifications

Fast IntegratedSimple

Request specific notifications

Request notifications on specific

PUT APIs

Request notifications on specific

DELETE APIs

s3:ObjectCreated:*

s3:ObjectCreated:Put

s3:ObjectCreated:Post

s3:ObjectCreated:Copy

s3:ObjectCreated:CompleteMultipartUpload

s3:ObjectRemoved:*

s3:ObjectRemoved:Delete

s3:ObjectRemoved:DeleteMarkerCreated


Hisham Baz

Solutions Architect,

Amazon

DEMO

VidShare

Empty bucket in Oregon

Create uploadVideo handler

VidShare v1.0

Share a video

Put fires, metadata parsed, thumbnail generated

Amazon DynamoDB metadata index updated

Setup delete notification – AWS CLI

Setup delete notifications – config and execute

Delete files directly from S3 bucket

Delete notification fires, app updated

Thumb deleted and metadata index updated

Launch

V2 Optimize on cost

V3 Expand globally


Optimizing VidShare

VidShare

T T+3 days T+5 days T+ 15 days T + 25 days T + 30 days T + 60 days T + 90 days T + 150 days T + 250 days T + 365 days

Acce

ss

Fre

quency

Choice of storage classes on Amazon S3

Standard

Active data Archive dataInfrequently accessed data

Standard - Infrequent Access Amazon Glacier

11 9’s of Durability

Standard-Infrequent Access storage

Infrequently accessed data

Designed for

99.9% availability

Durable AvailableSame throughput as

Amazon S3 Standard storage

High performance

• Server-side encryption

• Use your encryption keys

• KMS managed encryption keys

Secure

• Lifecycle management

• Versioning

• Event notifications

• Metrics

Integrated

• No impact on user

experience

• Simple REST API

• Single bucket

Easy to use

- Transition Standard to Standard-IA

- Transition Standard-IA to Amazon Glacier

storage

- Expiration lifecycle policy

- Versioning support

Standard-Infrequent Access storage

Integrated with lifecycle

Integrated: Lifecycle management

Save money on VidShare

Lifecycle policy

Standard Storage -> Standard-IA

<LifecycleConfiguration>

<Rule>

<ID>sample-rule</ID>

<Prefix>documents/</Prefix>

<Status>Enabled</Status>

<Transition>

<Days>30</Days>

<StorageClass>STANDARD-IA</StorageClass>

</Transition>

<Transition>

<Days>365</Days>

<StorageClass>GLACIER</StorageClass>

</Transition>

</Rule>

</LifecycleConfiguration>


Lifecycle Policy

Standard Storage -> Standard-IA

<LifecycleConfiguration>

<Rule>

<ID>sample-rule</ID>

<Prefix>documents/</Prefix>


<Transition>

<Days>30</Days>

<StorageClass>STANDARD-IA</StorageClass>

</Transition>

<Transition>

<Days>365</Days>

<StorageClass>GLACIER</StorageClass>

</Transition>

</Rule>

</LifecycleConfiguration>

Standard-IA Storage -> Amazon Glacier


Hisham Baz


Amazon

DEMO

VidShare

Setup lifecycle policy

Transition older videos to Standard-IA


39%

* Assumes the highest public pricing tier

VidShare is global!

VidShare

Launch

V2 Optimize on cost

V3 Expand globally


Remote replicas managed

by separate AWS accounts

Secure

Distribute data to regional

customers

Lower Latency

Store hundreds of

miles apart

Compliance

Amazon S3 cross-region replicationAutomated, fast, and reliable asynchronous replication of data across AWS regions

• Usual charges for

storage, requests, and

inter-region data transfer

for the replicated copy of

data

• Replicate into Standard-IA

or Amazon Glacier

Cost

HEAD operation on a source

object to determine replication

status

• Replicated objects will not be

re-replicated

• Use Amazon S3 COPY to

replicate existing objects

Replication status

DELETE without object

version ID• Marker replicated

DELETE specific object

version ID• Marker NOT replicated

Delete operation

Cross-region replication: Details

Object ACL updates are

replicated

• Objects with Amazon

managed encryption key

replicated

• KMS encryption not

replicated

Access control

Versioning with cross-region replication

A

B

Vid1- v2

Vid1- v1

Key: A/vid1 Key: B/vid1

Vid1- v2

Vid1- v1

Vid1- v4Vid1- v3

Versioning with cross-region replication

A

B

Vid1- v2

Vid1- v1

Key: A/vid1 Key: B/vid1

Vid1- v2

Vid1- v1

Vid1- v4Vid1- v3

Vid1- v3Vid1- v4

Replicate VidShare videos to Japan

PUT /?replication HTTP/1.1

Host: examplebucket.s3.amazonaws.com

x-amz-date: Wed, 11 Feb 2015 02:11:21 GMT

Content-MD5: q6yJDlIkcBaGGfb3QLY69A== Authorization:

authorization string

Content-Length: 406

<ReplicationConfiguration>

<Role>arn:aws:iam::35667example:role/CrossRegionReplicationRoleFo

rS3</Role>

<Rule>

<ID>rule1</ID>

<Prefix>vid/</Prefix>


<Destination>

<Bucket>arn:aws:s3:::vidsharebucketjapan</Bucket>

</Destination>

</Rule>

</ReplicationConfiguration>

Setting up cross-region replication policy (same AWS account)

1. Enable versioning on both

buckets

2. Add Replication Configuration

3. Validate replication

Replicate VidShare videos to Japan

1. Add bucket policy on the destination bucket to allow the source bucket

owner permission for replication actions

2. Create an IAM role in AWS Account A.

3. Specify IAM role when adding replication configuration on the source

bucket

4. Enable versioning on both buckets

5. Add Replication Configuration on source bucket AWS Account A

Setting up cross-region replication policy (different AWS accounts)


Hisham Baz


Amazon

DEMO

VidShare

VidShare v3.0 – Global expansion

Setup replication using AWS CLI

Setup replication – Execute via AWS CLI

Share video clip

Put fires, replication begins

File in bucket in Oregon

View replication status

View video replicated

Delete the video, and watch replication

Fully replicated delete

VidShare for enterprises!

Virtual

Private Cloud (Amazon VPC)

VidShare

Launch

V2 Optimize on cost

V3 Expand globally


Using Amazon S3 with VPC endpoints – Previously…

mybucket

InternetPUT S3

PU

T S

3NAT\

Internet

Gateway

• Public IP on EC2 instances

and IGW

• Private IP on EC2

instances and NAT

Amazon S3 VPC endpointsAccess Amazon S3 from your Amazon VPC using VPC endpoints

VP

C

En

dp

oin

tPUT S3

VPC Policy

mybucket

Internet


Using Amazon S3 VPC endpoints

Improved throughput

from VPC resources to

Amazon S3

High availability

High performance

High availability

Reduce cost by

avoiding expensive

NAT, Internet

gateways

Lower cost

Simple to setup, no

need to manage NATs

and Internet gateways

Simple

Improved security, no

need to route traffic

through the internet

Secure



Get VPC

PolicyVP

C

En

dp

oin

t

VPC Policy

mybucket


PUT S3

VP

C

En

dp

oin

t

VPC Policy

mybucket

Evaluate

VPC Policy



VP

C

En

dp

oin

t

VPC Policy

mybucket

Evaluate S3

Bucket\ACL

Policy



VP

C

En

dp

oin

t

VPC Policy

mybucket

PUT S3

PU

T S

3



Amazon S3 VPC endpoints


• Control access to buckets from specific Amazon

VPC endpoints, or specific VPCs

• Control which VPCs or VPC endpoints have

access to your S3 buckets by using S3 bucket

policies


Using Amazon S3

VPC endpoints

Amazon VPC policy to restricts access to a

specific bucket

{

"Statement": [

{

"Sid": "Access-to-specific-bucket-only",

"Principal": "*",

"Action": [

"s3:GetObject",

"s3:PutObject" ],

"Effect": "Allow",

"Resource": ["arn:aws:s3:::my_secure_bucket",

"arn:aws:s3:::my_secure_bucket/*"]

}

]

}

Amazon S3 bucket policy to allow a specific

VPC endpoint access to my S3 bucket

{

"Version": "2012-10-17",

"Id": "Policy1415115909152",

"Statement": [

{

"Sid": "Access-to-specific-VPCE-only",

"Principal": "*",

"Action": "s3:*",

"Effect": "Deny",

"Resource": ["arn:aws:s3:::my_secure_bucket",

"arn:aws:s3:::my_secure_bucket/*"],

"Condition":

{

"StringNotEquals": {

"aws:sourceVpce": "vpce-1a2b3c4d" }

}

} ] }


Using Amazon S3

VPC endpoints


Hisham Baz


Amazon

DEMO

VidShare

Start in Amazon VPC console

Create a new VPC endpoint

Configure route tables

View routes

View bucket policy

Apply bucket policy

Play recorded video in Amazon WorkSpaces

Launch Amazon WorkSpaces

Watch video from Amazon WorkSpaces

Audit logs Amazon S3Demonstrate compliance, improve security

Log Amazon S3 API

using AWS CloudTrail

Track bucket-level operations• Creation and deletion of buckets

• Changes to access control, lifecycle policy, cross

region replications policy etc.

Integrated with Amazon CloudWatch• Alarm if a specific API called

Configure once per AWS Account• Track multiple services with AWS CloudTrail

Amazon S3 storage metricsUnderstand your Amazon S3 buckets

Amazon CloudWatch metrics for Amazon S3

Bucket-level metrics include:

• Total bytes for Standard storage

• Total bytes for Standard-IA storage

• Total bytes for Reduced-Redundancy storage

• Total number of objects for a given S3 bucket

Alarm on S3 metrics

• Set thresholds for alarms

Daily metrics

• Metrics emitted daily, after midnight GMT

Thank you!

Remember to complete

your evaluations!

(stg401) amazon s3 deep dive & best practices

Technology