aws re:invent 2016: how netflix achieves email delivery at global scale with amazon ses (mbl204)
TRANSCRIPT
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
MBL204
November 29, 2016
How Netflix Achieves Email Delivery at Global Scale
with Amazon SESBest Practices for Building Scalable and Reliable Email Applications using Amazon SES
Devika Chawla, Netflix
Kadir Rathnavelu, Amazon SES
Challenges with Email Sending
• Delivery of email to your recipient’s inbox is hard
• Managing the complaint feedback loop is complex
• The undifferentiated heavy lifting of setting up an email
infrastructure
Netflix Email Delivery at Global Scale with
Amazon SES
Agenda
• Netflix Email Overview
• History, Challenges & Motivation for Amazon SES
• Move to Amazon SES
Netflix Email Overview
Netflix Ecosystem
Sign up flow
Sign up flow
Sign up flow
Sign up flow
Non Member – Remarketing Series
Sign up flow
Members - Welcome Series
Members – Personalized Messages
Cancel Membership
Former Member - Rejoin Series
Emails across the customer lifecycle
Acquisition
Engagement Transactional
Rejoin
Message Classification
Member - Transactional
Most Important
Very Low Risk
Member - Other
Very Important
Low Risk
Non/Former Member
Important
Medium Risk
History, Challenges &
Motivation for Amazon SES
Messaging Platform in AWS & Send from Data Center
us-west-2 us-east-1 eu-west-1
Challenges
• Keep email delivery software running in Data Center
• Optimize settings for ISPs
• Process bounce logs
• Work with 3rd party for complaint feedback loop
• Process complaint feedback
Challenges
Requirements for Netflix Email
• Great Deliverability for 100 million plus messages/day
• Linear scale with global subscriber growth, at cost
• Multiregion delivery of transactional messages
• Risk Isolation for different types of messages
• Operational simplification
Move to Amazon SES
Dedicated Pools of IPs
Messages
Message
Processor
Message
Metadata
Service
SES
Transactional
Pool
SES
Member
Pool
SES
Non/Former
Member
Pool
Password
Reset
New
Season
Re-
marketing
Multi Region Presence
us-west-2 us-east-1 eu-west-1
SESSES SES
Streamlined Feedback for Delivery, Bounces, Complaints
Delivered/sec Bounces/sec
Complaints/sec Latency – 95th Percentile
Gradual Ramp up to Amazon SES (IP Warming)
us-west-2 us-east-1 eu-west-1
SESSES SES
IP Reputation (from 3rd Party vendor)
Inbox Deliverability (from 3rd Party vendor)
Migration
Monitor
Bounces
IP Reputation
Inbox Deliverability
Increase Traffic to SES
100% Emails on Amazon SES
us-west-2 us-east-1 eu-west-1
SESSES SES
Building Scalable and Reliable Email
Applications
Reiterating the challenges with sending emails
• The reputation of the email sender
• Managing the complaints feedback loop is complex
• The undifferentiated heavy lifting of setting up an email
infrastructure
Amazon SES eliminates these challenges
• Delivering emails to an inbox is a shared responsibility
• Amazon SES provides…
• Higher deliverability via proactive reputation management
• Maintains complaint feedback loops with major ISPs
• Elasticity
• High availability
• The breadth of features to build reliable sending programs
Fundamentals of Setting up an
Email Program
Setting up email authentication
• Verify an email address or a domain
Setting up email authentication
• Verify an email address or a domain
• Set up DomainKeys Identified Mail (DKIM)
DomainKeys Identified Mail (DKIM)
• An authentication standard that allows you to sign your email
cryptographically
• ISPs decode the cryptographic signature using a public key to
ensure that your emails are not modified in transit
• Set up DKIM for your domain with DKIM from Amazon SES Console
Setting up email authentication
• Verify an email address or a domain
• Set up DomainKeys Identified Mail (DKIM)
• Add an Sender Policy Framework (SPF) record
Sender Policy Framework (SPF)
• SPF indicates which mail servers are authorized to send email for
their domain
• Receiving mail server compares MAIL FROM domain’s DNS record
for authorization
• Two ways to set up SPF on Amazon SES
• Use default MAIL FROM domain of Amazon SES
• Publish an SPF record
Setting up email authentication
• Verify an email address or a domain
• Set up DomainKeys Identified Mail (DKIM)
• Add an Sender Policy Framework (SPF) record
More on set up
• Use Mailbox Stimulator for scalability testing
• Use HTTP instead of SMTP
• Send in Parallel
Feedback Processing
Why do you want to process feedback?
• Listen to feedback about the emails you send
• Delivered
• Bounce
• Complaints
• Acting on bounces and complaints improves the deliverability of
your emails
• Improves customer experience
Monitoring your sending activity
Monitoring Method How to Access Data Level of Detail Granularity
Feedback Notifications Amazon SNS, or Email Details on each
event
Across entire AWS Account
Event Publish Amazon CloudWatch, or Amazon
Kinesis Firehose
Details on each
event
Fine grained (based on
user-defined email
characteristics)
Amazon SES Event Publishing
• Fine-grained metrics based on a definable email characteristics like
marketing vs. transactional, product, etc.
• If you are selling music, you could define and track by ‘genre’
• Publish to Amazon CloudWatch or Amazon Kinesis Firehose
Amazon SES Event Publishing
• Create a configuration set
• Provide the name of the configuration
set, and one or more message tags
Amazon SES Event Publishing
• Amazon CloudWatch is best suited for aggregated metrics
• Metrics are available under Amazon SES namespace in Amazon
CloudWatch
• Event are aggregated by type and combination of specified dimensions
• Amazon Kinesis Firehose enables you to see individual events
• Stream data in to S3, Amazon Kinesis Analytics, Amazon Redshift, or
Elasticsearch Service Cluster
Dedicated IPs
Shared IPs
• Amazon SES customers send their emails from a single set of IP
addresses
• Everyone is responsible for the reputation of the IP addresses
What are dedicated IPs?
• Dedicated IPs are Amazon SES IP addresses that are reserved
exclusively for your email sending
• Take more control and responsibility over the deliverability
• Scenarios:
• Segmenting mail streams for separate programs, or
engagement metrics
• Sending transactional mail, such as order confirmations, in
isolation from your marketing mail
• Send diagnostic or operational email from a certain set of IPs,
and reject all other mails
Dedicated IP addresses
• Your IP reputation will ultimately be driven by your own
sending practices
• Gradual warm up will reduce throttling and potential
rejects from the ISPs
• Start small
• Measure results
• Make corrections
• Ramp up the sending
Dedicated IP addresses
• Failure to keep the IPs warmed up will result in worse
deliverability
• You are responsible for estimating any potential volume
increases
Thank you!
Remember to complete
your evaluations!