aws re:invent 2016: how telltale games migrated its story analytics from apache couchdb to amazon...
TRANSCRIPT
![Page 1: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/1.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Zac Litton
VP of Engineering
Telltale Games
DAT316
How Telltale Games Migrated Its Story Analytics
from Apache CouchDB to Amazon DynamoDB
Greg McConnel
Solutions Architect
AWS
![Page 2: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/2.jpg)
What to Expect from the Session
● Intro to DynamoDB
● Telltale story analytics
● Early infrastructure (SQL and
Apache CouchDB)
● Migration to DynamoDB
● Better data, better stories
![Page 3: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/3.jpg)
Telltale Games
Telltale Games history
![Page 4: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/4.jpg)
Why did AWS Build Amazon DynamoDB?
It’s hard to engineer for the
performance you need.
Traditional NoSQL databases run
into challenges as they scale.
Managing non-relational databases is hard.
![Page 5: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/5.jpg)
Quick Intro on Amazon DynamoDB
Document or Key-Value Scales to Any WorkloadFully Managed NoSQL
Access Control Event Driven
Programming
Fast and Consistent
![Page 6: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/6.jpg)
Fast, Consistent Performance
Single-digit millisecond latency
• At any scale
Data stored on Solid State Drives (SSDs)
Automatic partitioning means no need for
hotspot management
![Page 7: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/7.jpg)
Highly Scalable
Simply specify each table’s read and write
throughput capacity
Increase and decrease capacity as needed
• No upper limit
DynamoDB manages all the scaling behind
the scenes
![Page 8: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/8.jpg)
Flexible
Key-value store model
Each item in a DynamoDB
table is a list of attributes (fields)
and values
No need for every item to have the
same attributes
Add attributes at will
Document store
Place JSON-formatted data
into DynamoDB items for robust,
nested data structures
![Page 9: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/9.jpg)
Amazon DynamoDB is a schemaless database
table items
Attributes (name/value
pairs or JSON
documents)
![Page 10: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/10.jpg)
Each item includes a key
Partition key
(DynamoDB maintains
an unordered index)
![Page 11: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/11.jpg)
Each item includes a key
Partition Key
Sort Key
(DynamoDB maintains a
sorted index)
![Page 12: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/12.jpg)
Integration capabilities
DynamoDB Triggers
❑ Implemented as AWS
Lambda functions
❑ Your code scales
automatically
❑ Java, Node.js, and Python
DynamoDB Streams
❑ Stream of table updates
❑ Asynchronous
❑ Exactly once
❑ Strictly ordered
❑ 24-hr lifetime per item
![Page 13: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/13.jpg)
Telltale Story Analytics
Telltale Games will remember that
![Page 14: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/14.jpg)
Telltale Games
![Page 15: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/15.jpg)
Telltale Games
![Page 16: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/16.jpg)
Telltale Games
![Page 17: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/17.jpg)
Telltale Games
![Page 18: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/18.jpg)
Telltale Games
![Page 19: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/19.jpg)
Telltale Games
![Page 20: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/20.jpg)
Choice data
![Page 21: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/21.jpg)
Choice data
Everything is unified JSON event logs
![Page 22: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/22.jpg)
Choice data
Large amounts of diverse data
● Episodes contain over 2000 nodes plus additional data
● Millions of worldwide users
● 21TB of events stored
● >1 million parsed ‘sessions’ daily
● 10x spikes for release, free episodes, & advertising
![Page 23: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/23.jpg)
How we use the data
Aggregated back to players
![Page 24: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/24.jpg)
How we use the data
Player heat map and evaluations
![Page 25: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/25.jpg)
How we use the data
Personalize the stories
![Page 26: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/26.jpg)
Early Infrastructure
![Page 27: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/27.jpg)
SQL, right?
● Can’t handle size
● No real time scaling
● Non starter
![Page 28: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/28.jpg)
Apache CouchDB (Scaling/Maintenance)
● Manual process of spinning servers
● Frequent time consuming node failures
● Document limits
● Full time maintenance for 2 people
![Page 29: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/29.jpg)
Apache CouchDB (Processing)
● Processing nearly impossible with data size
● Limited to more aggregation than analysis
● Couldn’t scale up easily for ‘speedy’ processing
● New queries impractical
![Page 30: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/30.jpg)
Migration to DynamoDB
![Page 31: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/31.jpg)
Managed & Scale
● Immediately ended our maintenance
● No storage limitations(200 Billion event peak)
● 21TB of events, 10GB/day.
● 1M session uploads per day with 900ms response
● Automation scripts to adjust to spikes
● Start at 50 r/w per second, up to 20K write spikes
● Autoscaling using Dynamic DynamoDB
![Page 32: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/32.jpg)
Amazon
ElastiCache
Amazon
RDS
Amazon
DynamoDBAmazon S3
Amazon
CloudFront
Game Clients
Amazon
Route 53
Load
Balancers
Web Servers
Amazon
EC2
Patches &
DLC
![Page 33: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/33.jpg)
Processing
● Separate tables per game for independent processing
● Reading only the data needed
● Export entire tables to S3 in 24 hours with no loss
● Capable of adjusting to new queries
![Page 34: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/34.jpg)
Costs
● 1 server handles what 12 did before
● Costs roughly equivalent, but load is 10x
● Reading only the data needed
● No longer paying for static usage with pay per usage
![Page 35: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/35.jpg)
Challenges / Improvements
● Too much old data still in
couchDB
● Integrated read/write
provisioning
![Page 36: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/36.jpg)
Better data, better stories
![Page 37: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/37.jpg)
Understanding player behavior
● Internal tools read aggregated data as player head
maps
● Know the characters and lines working
● Episodes personalize to audience in a way no
entertainment medium can
But we wanted to take this a step further. . .
![Page 38: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/38.jpg)
User Playstyle Clustering
● Used K-means clustering was based on 2,200 Walking Dead story choice nodes
○ Algorithm determines number of means and initial seed value
○ 8 clusters that represented 88% of our players.
● Analyzed each cluster on two metrics:
○ Which story choices a single cluster endorsed at high rates
○ Which story choices are effective at splitting apart two or more clusters
● Developed player personas
○ Highlighted general preferences of a cluster (e.g., protecting resources over
helping people)
○ Identified minor characters that were influential to play style
![Page 39: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/39.jpg)
Customer Playstyle ClusteringModel OutputModel Selection Personas
![Page 40: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/40.jpg)
“Amoral and Ambivalent”
The second biggest cluster, representing 22% of our players, seems to value independence.
They offer peace first to Russians but lie about Jane's whereabouts, potentially because they think they are bluffing for survival's sake. They don’t start with force, but likely to follow-up with force if not complied with. Highly favors pointing out they have a baby to the Russians during the showdown, perhaps appealing to their humanity, but while this cluster helps others or offers peace when convenient, they don’t hesitate to react with violence once pushed.
![Page 41: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/41.jpg)
“My Best Self”This cluster is reasonable and logical; they may be even tempered individuals, or Players who feel comfortable being a little distanced from the content of the game they are playing. Their responses in conversation generally seem to pick out the most responsive/reactive threads. This cluster is conventionally compassionate and frequently chose offers of condolences and sympathy when appropriate.
Their most common two endings can perhaps be read as either: disillusionment at Howe’s when Clem turns away the family or loyalty as when Clem keeps on with Kenny to Wellington then leaves with him.
![Page 42: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/42.jpg)
Thank you!
![Page 43: AWS re:Invent 2016: How Telltale Games migrated its story analytics from Apache CouchDB to Amazon DynamoDB (DAT316)](https://reader031.vdocument.in/reader031/viewer/2022030306/586f90651a28ab54768b7999/html5/thumbnails/43.jpg)
Remember to complete
your evaluations!