how rovio teaches angry birds to fly in the cloud using ml
TRANSCRIPT
![Page 1: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/1.jpg)
![Page 2: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/2.jpg)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How Rovio teaches Angry Birds to fly in the cloud using ML
G A M 3 0 1
David Mason
SVP Technology
Rovio Entertainment Corporation
Asko Relas
Senior Data Engineer
Rovio Entertainment Corporation
![Page 3: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/3.jpg)
Agenda
• Happy Birdday!
• Beacon: Our cloud services
• Our machine learning (ML) story
• Use case: Reinforcement learning (RL) for level difficulty prediction
• Takeaways
![Page 4: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/4.jpg)
![Page 5: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/5.jpg)
![Page 6: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/6.jpg)
![Page 7: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/7.jpg)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
![Page 8: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/8.jpg)
Beacon: Our cloud services
![Page 9: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/9.jpg)
Beacon
![Page 10: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/10.jpg)
AWS is our preferred cloud provider
# Player accounts > 1Bn
# Requests per second > 10k
# Analytics events per day > 2Bn
# Amount of data per day > 1TB
# EC2 instances > 1300
![Page 11: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/11.jpg)
Analytics
A/B testing Personalization
Payments Ads + XP
Push
Surveys
Game server infrastructure
LiveOps calendar
Audience builderUA
BuildPrivacy
AI assisted dev
Singular
Player identity
Amazon Elasticsearch
Service (Amazon ES)
![Page 12: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/12.jpg)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
![Page 13: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/13.jpg)
Analytics pipeline
Lake of wisdom
Real-time Flink
streams
Amazon EMR
ORC format
with schemas
Amazon S3Secor, Spark, Hive
KPIs, player profiles, ad-hoc queries
Advanced analytics, ML
Spark, R, Python
• Redash• Grafana• Presto
Dashboards, app personalization
Event
gateway Kafka
Mobile client
Services
Amazon
EC2
Services
Amazon
EC2
Services Amazon
EC2
Amazon
EC2
Amazon
EC2
Amazon
EC2
Amazon
EC2
Amazon
Athena
Amazon
Redshift
Amazon
Athena
Amazon
EC2
• Cassandra• Presto• Druid
Amazon
SageMaker
Amazon
EMR
Amazon EMR
Amazon EMR
![Page 14: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/14.jpg)
Game server infrastructure: Reusable templatesPlatform services
Deployment
Network
Management
Amazon
RDS
Amazon
DynamoDB
Amazon
ElastiCache
Amazon Elasticsearch
Service (Amazon ES)
Amazon
CloudWatchAmazon S3
AWS Certificate
Manager (ACM)
AWS
CodeDeployAmazon ECS
Elastic Load
Balancing
Amazon Route
53
AWS Shield
Advanced
Jenkins TerraformAWS Management
Console
Network Load
Balancer
Application Load
Balancer
Service group 1 Service group 2Load test
generators
Logs Monitoring Discovery
AlertsConfig
VPN Gateway
![Page 15: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/15.jpg)
Game server microservices: Chirp
Game serverGame
clients
Consul
Amazon ES
Amazon RDS
Index messages
Store messages
Profanity
check
Community
sift
Discover nodes
Chirp server
Notifications
Firebase cloud
messaging
Push
notifications
Translate
Translate
message
Request
response
![Page 16: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/16.jpg)
Our machine learning (ML) story
![Page 17: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/17.jpg)
ML/AI in gaming
AI creates art,
music, &
narratives
Creative
personalization
Game ideaGame
development Distribution Marketing Live operation Monetization
Market
analysis
Modeling of fun
Bots for QA
& level
balancing
Automatic level
& game object
generation
LTV
modeling
Player
onboarding
Campaign
optimization
(paid/xpromo)
Personalized
events
Moderation
Player in-game
support &
automation
Segmentation
& targeting
Monitoring
& alerting,
anomaly
detection
Pricing
optimization
Ads optimization
Creative
analysis &
optimization
![Page 18: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/18.jpg)
Our goal: Games built for you
MICRO
2011
MACRO
13 15 17 19 21
INDIVIDUAL
Angry Birds Match AI
heuristic bot to play
levels & estimate pass
rate
ML driven market
analysis tooling
Churn prediction used
for cross-promo
LTV prediction in profiles
User-specific app install
propensity prediction for
cross-promotion
Anomaly detection
Fruit Nibblers
using ML for churn
prediction to improve
level funnel
Angry Birds Dream
Blast ML level
balancing
First session spender
prediction
Rovio starts
collecting data
from games
Churn &
conversion
prediction
![Page 19: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/19.jpg)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
![Page 20: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/20.jpg)
Project overview
Designing the level funnel of a puzzle game is a balancing act• Too difficult levels give a feeling of pay-to-win and players churn
• Too easy levels make the game boring
Balancing new levels requires repetitive testing from level designers
Being able to predict the difficulty of unreleased levels• Consistent experience for the most advanced players
• Less time spent on manual testing
![Page 21: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/21.jpg)
Measuring difficulty
![Page 22: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/22.jpg)
Overall concept
Level designer
Submits a pack of new levels for evaluation
Bot
Plays the levels and records statistics of them
Prediction
Real players’ pass rates are predicted from the
bot’s statistics
![Page 23: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/23.jpg)
From end users’ point of view
![Page 24: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/24.jpg)
From end users’ point of view
![Page 25: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/25.jpg)
From end users’ point of view
![Page 26: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/26.jpg)
From end users’ point of view
![Page 27: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/27.jpg)
Solution architecture
Amazon API Gateway
Bots API Bots
1. Store levels
2. Run bot
NotificationSubmit levels
HTTP/REST
PredictionsLevels
Regression models
Amazon S3 Amazon RDS
![Page 28: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/28.jpg)
Different bot approaches
Concept Pros Cons
HeuristicScore every available action
based on predefined rules
The simplest to
implement
One policy for the whole
game
Tree searchBuild a game state tree to
find best actions, e.g. MCTS
No understanding of
game mechanics needed
Slow and computationally
heavy
Imitation learning
Collect data on how people
play the game, train a bot to
imitate
Possibly most human-like
Only one policy
More game
instrumentation needed
Reinforcement
learning
The bot learns how to play
the game through
experimentation
Can learn a different
policy for each level
Slower than imitation, but
faster than tree search
![Page 29: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/29.jpg)
Reinforcement learning (RL)
Agent
Environment
ActionReward
![Page 30: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/30.jpg)
![Page 31: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/31.jpg)
![Page 32: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/32.jpg)
Reinforcement learning (RL)
Our adaptation of deep q-learning:
• Feature engineered game state instead of raw graphics
• Feed-forward neural net instead of a deep convolutional net
• Dueling double q-learning with prioritized experience replay
PyTorch as deep learning framework
![Page 33: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/33.jpg)
Data collection
Level 1
![Page 34: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/34.jpg)
Data collection
Level 1
Pass rate: 100%
Goals cleared: 100%
Moves left: 8.9
Extra moves: 0
![Page 35: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/35.jpg)
Data collection
Level 1 Level 2
Pass rate: 100%
Goals cleared: 100%
Moves left: 8.9
Extra moves: 0
Pass rate: 99%
Goals cleared: 99%
Moves left: 7.4
Extra moves: 0
![Page 36: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/36.jpg)
Data collection
Level 1 Level 2 Level 3
Pass rate: 100%
Goals cleared: 100%
Moves left: 8.9
Extra moves: 0
Pass rate: 99%
Goals cleared: 99%
Moves left: 7.4
Extra moves: 0
Pass rate: 97%
Goals cleared: 98%
Moves left: 4.9
Extra moves: 0
![Page 37: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/37.jpg)
Data collection
Level 1 Level 2 Level 3 Level 982
Pass rate: 100%
Goals cleared: 100%
Moves left: 8.9
Extra moves: 0
Pass rate: 99%
Goals cleared: 99%
Moves left: 7.4
Extra moves: 0
Pass rate: 97%
Goals cleared: 98%
Moves left: 4.9
Extra moves: 0
Pass rate: 2%
Goals cleared: 54%
Moves left: 0.02
Extra moves: 7.3
![Page 38: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/38.jpg)
Data collection
Collecting the data set takes time
• For the most accurate statistics, the bot first learns a level, then plays it 1,000 times
• One game round takes 5-10 seconds even with sped-up physics
• Collecting the data set by playing the levels sequentially would take over a year
Solution
• Bundle the game binary and bot Python source code to a Docker container
• Run one container per level using AWS Batch
→ Data collection takes two days for any number of levels
• Learn from multiple parallel instances of the game within one container
→ Data collection takes four hours for any number of levels
![Page 39: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/39.jpg)
Parallelized RL
Agent
Game
![Page 40: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/40.jpg)
Parallelized RL
Agent
Game
Agent
Game
![Page 41: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/41.jpg)
Parallelized RL
Agent
Game
Buffer
Agent
Game
Experience
![Page 42: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/42.jpg)
Parallelized RL
Agent
Game
Learner
Buffer
Agent
Game
ExperienceLearned
NN model
![Page 43: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/43.jpg)
Parallelized RL
Agent
Game
Learner
Buffer
Agent
Game
ExperienceLearned
NN model
![Page 44: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/44.jpg)
Parallelized RL
Agent
Game
Learner
Buffer
Agent
Game
ExperienceLearned
NN model
![Page 45: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/45.jpg)
Parallelized RL
Amazon EC2 C5 instance
Agent
Game
Learner
Buffer
Agent
Game
ExperienceLearned
NN model
![Page 46: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/46.jpg)
Parallelized RL
Eight-fold parallelism with a single c5.2xlarge
If we increase the number of agents, the learner becomes a bottleneck
Can scale further through:
● Using a GPU, eg. p3.8xlarge with 32 vCPU
● Scaling out across multiple machines
Amazon EC2 C5 instance
Agent
Game
Learner
Buffer
Agent
Game
ExperienceLearned
NN model
![Page 47: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/47.jpg)
Predicting difficulty
Bot stats
Player pass rates
Learn a regression model from bot stats and player pass rates
![Page 48: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/48.jpg)
Predicting difficulty
New level bot stats
Get prediction from learned regression
model
Player pass rate prediction
![Page 49: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/49.jpg)
Regression modeling
Statistics collected from the bot
• Pass rate
• Moves left
• Level objectives cleared
• Extra moves given
scikit-learn as ML toolkit
Best prediction accuracy with XGBoost
• Tree-based algorithms can’t extrapolate
→ Additional classification between impossible/passable levels is necessary
![Page 50: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/50.jpg)
Prediction accuracy
MAE 0.15 in validation data set
![Page 51: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/51.jpg)
Then what?
Difficulty
Churn Revenue
![Page 52: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/52.jpg)
Then what?
f(x) y(x)
Difficulty
Churn Revenue
![Page 53: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/53.jpg)
Difficulty versus churn
Pass rate
Churn
![Page 54: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/54.jpg)
Difficulty versus churn
Level 300
Churn
Pass rate
![Page 55: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/55.jpg)
Difficulty versus churn
Level 800
Level 300
Churn
Pass rate
![Page 56: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/56.jpg)
Difficulty versus churn
Churn
Pass rate
![Page 57: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/57.jpg)
Difficulty versus revenue
Level 300
Level 1,000
Revenue
Pass rate
![Page 58: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/58.jpg)
Case summary
• AWS Batch is great for parallel workloads, such as having bots play 1,000 levels concurrently
• Parallelization is key to implementing fast-enough RL for business use
• Next step: Procedural content creation?
![Page 59: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/59.jpg)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
![Page 60: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/60.jpg)
Building an unfair advantage
• Put the player at the center
• Analytics is foundational
• Have vision, experiment, learn, productize
• ML will change the game
![Page 61: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/61.jpg)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
![Page 62: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/62.jpg)
Thank you!
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Contact information
David [email protected]
Asko [email protected]
![Page 63: How Rovio teaches Angry Birds to fly in the cloud using ML](https://reader033.vdocument.in/reader033/viewer/2022060603/62969b9210ab0a3b7b710b99/html5/thumbnails/63.jpg)
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.