aws webcast - database in the cloud series - scalable games and analytics with aws

AWS Gaming Solutions

Scalable Games and Analytics with AWS

Nate Wiger @nateware Principal Gaming Solutions Architect April 2014

Artist: Abel Oroz Vicente

What's In It For Me?

•  Why AWS for Games? •  Core Game Backend •  Scaling with DynamoDB •  Low-Latency Multiplayer with C3 •  Game Analytics with Redshift and Kinesis

Gratuitous Logo Slide

Handle Viral Spikes

10 Regions

51 CloudFront POPs

Continuous Expansion

Global is Good

Big picture slide

Elastic Beanstalk •  Managed Container •  6 Languages •  Git Push or Zip Upload •  ELB, EC2, DB, DNS •  Web Dashboard •  Same Performance •  Full Access •  So Yeah, Use It

Hill Of Beans

•  Beanstalk Manages •  ELB •  EC2 •  Auto Scaling •  Monitoring •  RDS

•  Add Other Services •  S3 •  CloudFront •  SNS Mobile Push

Elastic Beanstalk Container

CloudFront CDN

Push It

cd MyGame eb init vi app.rb require 'sinatra' get '/hello.json' do {message: "Hello World!"} end git commit -‐m "app updates" -‐a git aws.push

•  Initialize Beanstalk •  Write Code •  Git Commit •  Git Push to Beanstalk •  Coffee / Beer •  Rinse, Repeat

http://bit.ly/ubisoft-aws

Region

Writing is Painful

Availability Zone A

Availability Zone B

•  Games are Write Heavy •  Constant State Updates •  Key Value Key Value •  Binary Data •  Caching of Limited Use •  Database = Bottleneck

CloudFront CDN

DynamoDB

Availability Zone A

Availability Zone B

•  NoSQL datastore •  Fully-managed •  Provisioned throughput •  PUT/GET keys •  Secondary indexes •  Handles any SANE

Gaming use case EC2 EC2

CloudFront CDN

Elastic Beanstalk Container

DynamoDB: Low Latency at Scale

Docs, SDK's: http://aws.amazon.com/dynamodb/developer-resources/ Auto Scaling: https://github.com/sebdah/dynamic-dynamodb

DynamoDB: Durable at Scale

WRITES Replicated to 2 AZ’s before OK Persisted to disk (SSD)

READS Strong or eventual consistency

No trade-off in latency

Leaderboard in DynamoDB

•  Hash key = Primary key •  Range key = Sub key •  Others attributes are

unstructured •  How to sort based on

High Score per Game? •  Secondary Index on

Game + Score

Python DynamoDB Leaderboard

table = Table('scores', schema=[ HashKey('user'), RangeKey('level') ], throughput={ 'read': 5, 'write': 15 }, global_indexes=[ GlobalAllIndex('highscore', parts=[ HashKey('level'), RangeKey('score', data_type=NUMBER) ], throughput={ 'read': 5, 'write': 15 } ) ])

new_score = Item(table, data={ 'user': user, 'level': level, 'score': score }) new_score.save() topscores = table.query(index='highscore') for ts in topscores: print(ts['user'], ts['score'])

EC2: New C3 Instance

•  High packets per second •  Low latency, no jitter •  Intel Ivy Bridge CPU •  SSD's •  Built for games •  10.5 cents / hour

Enhanced Networking (SR-IOV)

Before: Hypervisor

After: Hardware

Step 1: HVM Machine Image

Step 2: Choose C3 Instance

Step 3: Launch in VPC

Recap: Enabling SR-IOV

①  Use HVM image (Linux or Windows) ②  Select C3 instance ③  Must launch in VPC ④  Verify with CLI

aws ec2 describe-‐instance-‐attribute \ -‐-‐attribute sriovNetSupport -‐-‐instance-‐id i-‐12345678

⑤  Full docs: http://bit.ly/ec2-enhanced-net

EC2: New G2 Instance

•  NVIDIA Kepler GPU •  Game Streaming •  Rendering Assets •  Build Servers •  AppStream! EVE Online:

http://bit.ly/appstream-eve-online

http://bit.ly/red5-aws

http://bit.ly/rovio-aws

Analyze What?

Emotions •  Enjoying game •  Engaged •  Like/dislike new content •  Stuck on a level •  Bored •  Abandonment

Behaviors •  Hours played day/week •  Number of sessions/day •  Level progression •  Friend invites/referrals •  Response to mobile push •  Money spent/week

Analytics Flow

Ingest Store Process Analyze Profit!

Off The Shelf Analytics

•  Easy To Integrate •  Pre-Baked Reports •  Rate Limits •  Retention Windows •  Data Lock-In

•  Write Events File on Device •  Periodically Upload to S3 •  Process into Redshift •  Point GUI Tool to Redshift

Start Simple

2014-‐01-‐24,nateware,e4df,login 2014-‐01-‐24,nateware,e4df,gamestart 2014-‐01-‐24,nateware,e4df,gameend 2014-‐01-‐25,nateware,a88c,login 2014-‐01-‐25,nateware,a88c,friendlist 2014-‐01-‐25,nateware,a88c,gamestart

Profit!

Tableau + Redshift

http://bit.ly/tableau-redshift

•  Also Collect Server Logs •  Periodically Upload to S3 •  Stuff into Redshift •  External Analytics Data Too

More Data Sources

External Analytics

•  Integrate Game DB •  Load Directly into Redshift •  Redshift does Intelligent Merge •  Tracks DynamoDB Columns

Direct From DynamoDB

Real-Time Event Stream

2014-‐03-‐17T09:52:08-‐07:00,nateware,e4b5,login 2014-‐03-‐17T09:52:54-‐07:00,nateware,e4b5,gamestart 2014-‐03-‐17T09:53:15-‐07:00,nateware,e4b5,levelup 2014-‐03-‐17T09:54:06-‐07:00,nateware,e4b5,gameend 2014-‐03-‐17T09:54:23-‐07:00,nateware,30a4,gamestart 2014-‐03-‐17T09:55:14-‐07:00,nateware,30a4,gameend 2014-‐03-‐17T09:55:41-‐07:00,nateware,30a4,gamestart 2014-‐03-‐17T09:57:12-‐07:00,nateware,6ebd,levelup 2014-‐03-‐17T09:58:50-‐07:00,nateware,6ebd,levelup 2014-‐03-‐17T09:59:52-‐07:00,nateware,6ebd,gameend

•  Stream Data With Kinesis •  Multiple Writers and Readers •  Still Output to Redshift

Speeding Towards Real-Time

•  Stream Data With Kinesis •  Multiple Writers and Readers •  Still Output to Redshift •  Stream to Spark on EMR •  Storm via Kinesis Spout •  Custom EC2 Workers

Speeding Towards Real-Time

Wrap It Up Already

•  Start Simple With Beanstalk •  Go Directly to DynamoDB, Do Not Pass Go •  Leverage the EC2 C3 Instance •  Analytics 1-2-3: CSV => S3 => Redshift •  Kinesis + Spark/EMR for Real-Time Analytics

http://bit.ly/minecraft-aws

aws webcast - database in the cloud series - scalable games and analytics with aws

kinesis multiple

availability

low latency

aws

instance

gamestart

score

hashkey

Technology

aws webcast - build mobile apps with a secure, scalable back...

aws webcast - introducing amazon redshift

aws webcast - active directory on aws

aws webcast - library systems on the aws cloud

aws webcast - business continuity in the aws cloud

aws webcast - highly available sql server on aws

aws webcast - introduction to ebs

aws webcast - aws 101 - journey to the aws cloud:...

aws webcast - scaling on aws 13 08-20

aws webcast - understanding the aws security model

aws webcast - aws opsworks continuous integration demo

aws webcast - dynamo db

aws webcast - sumo logic

aws webcast - what is cloud computing with aws

aws webcast - discover cloud computing

aws webcast - library storage webinar

aws webcast - continuous integration with aws and ravello

aws webcast - understanding database options

aws webcast - explore the aws cloud

aws webcast - disaster recovery