![Page 1: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/1.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cloud Based Game Development
David Pippenger, Director DevOps, GREE
NET308
How GREE Launched New Games Faster and
More Securely with AWS Marketplace and
Amazon VPC
![Page 2: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/2.jpg)
Company Overview
GREE International
Entertainment, Inc.
San Francisco, CA
~200 People, 2011
GREE Inc.
Tokyo, Japan
~2000 People, 2004
Started using AWS 2012
Evolved toward
immutable infrastructure
to run games
Now continuing to
evolve toward
decentralized model
![Page 3: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/3.jpg)
Business Model: Publishing Engine
Buy a studio,
operate it as a
programming
division
First Party
Commission a
game to GREE’s
specifications, pay
development costs
Second Party
Buy an existing
game
Third Party
![Page 4: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/4.jpg)
Game Publishing Engine
Marketing Analytics LiveOps
in-game
sales events
TechOps
AWS Infrastructure
![Page 5: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/5.jpg)
Games have a lifecycle. Nothing lasts forever.
New
Game
Creation
Long tail games with loyal users
BUY GAME SELL GAME
TOP 10
REVENUE
GAMES
![Page 6: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/6.jpg)
Gree started in
colos until the
Funzio acquistion
in 2012
Funzio was a
small mobile
gaming company
with 3 successful
games
![Page 7: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/7.jpg)
Dedicated jobs and cron hosts
Deploying tarballs
Hadoop
EC2 Based Relays
EC2 Based Vertica
Ma
nagem
ent
Amazon EC2 memcache clusters
EC2 MySQL
EC2 Zeus load balancers
Provisioning hosts using Runbooks
Amazon ElastiCache
Amazon RDS
Elastic Load Balancing
Puppet
Jenkins for code deployments
Deploying packages (RPM, Docker)
Amazon Elastic Map Reduce (EMR)
Amazon Kinesis
Amazon Redshift
AW
S
![Page 8: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/8.jpg)
SSH Jump hosts and OpenSwan
Netw
ork
ing
ELK logging on EC2
Nagios on EC2
SumoLogic
DataDogSaaS
![Page 9: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/9.jpg)
Technical
Stack
EC2 AutoscalingCloudformation S3 ELB
Kinesis RDS Elasticache DynamoDB
![Page 10: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/10.jpg)
1 Leverage SaaS
2 Avoid building things you can buy
3 Infrastructure as code
1 Leverage AWS services
2 Automate everything
3 Immutable infrastucture
4 Servers are cattle not pets
DevOps
Principles
![Page 11: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/11.jpg)
GAME STACK
DevOps at GREE – Architecture High Level Design
Amazon
S3
Amazon
Route 53
AWS
Directory
Service
AWS
Code
Commit
Active
Directory
GitHub
Enterprise
AWS
VPC
![Page 12: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/12.jpg)
DevOps at GREE – Architecture Low Level Design
Service 1 Service 2Amazon
DynamoDB
ILB
ELB
GAME
Puppet
Jenkins
Aviatrix
GatewayAuto Scaling
Auto Scaling Auto Scaling
VPC
![Page 13: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/13.jpg)
Secure Remote User Access
GREE AWS Account
Region 1 Region 2
IPSEC
Aviatrix
GatewayAviatrix
Gateway
AWSAWS
Dragon Soul AWS Account
VPC VPC
Remote Users
(dev, devops, admins)
SSL VPN
![Page 14: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/14.jpg)
Patterns of Enterprise Application Architecture
inspired by Martin Fowler’s Books
![Page 15: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/15.jpg)
Codebase
• One codebase tracked in
revision control, many deploys
• Use Git
I
![Page 16: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/16.jpg)
Dependencies
• Explicitly declare and isolate
dependencies
• Use Puppet
II
![Page 17: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/17.jpg)
Config
• Store config in the environment
• Use Puppet with HieradataIII
![Page 18: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/18.jpg)
Backing Services
• Treat backing services as
attached resources
• Use Puppet with Hieradata to
capture endpoints
IV
![Page 19: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/19.jpg)
Build,
release, run
• Strictly separate build and run
stages
• Use Jenkins to produce
package artifacts
• Use Jenkins pipeline to
orchestrate deployment
V
![Page 20: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/20.jpg)
Processes
• Execute the app as one or
more stateless processes
• Use ELB
• STONITH
• https://en.wikipedia.org/wiki/ST
ONITH
VI
![Page 21: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/21.jpg)
Port Binding
• Export services via port binding
• Use ILBVII
![Page 22: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/22.jpg)
Concurrency
• Scale out via the process
model
• Use ELB
VIII
![Page 23: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/23.jpg)
Disposability
• Maximize robustness with fast
startup and graceful shutdown
• Use Auto Scaling
• Use custom AMIs
• Use ElastiCache or DynamoDB
• STONITH
IX
![Page 24: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/24.jpg)
Dev/Prod
Parity
• Keep development, staging
and production as
similar as possible
• Use AWS CloudFormation
• Use Puppet
• Use Docker
X
![Page 25: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/25.jpg)
Logs
• Treat logs as event streams
• Use Sumo logicXI
![Page 26: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/26.jpg)
Admin Processes
• Run admin/management tasks
as one-off processes
• Use Jenkins
XII
![Page 27: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/27.jpg)
14 DevOps engineers
Takeaways
Hundreds of monitoring pages
6
10
![Page 28: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/28.jpg)
VPC History at GREE
VPC I
• AWS Classic
• EU-west-1 VPC only
• 6 week deadline,
things were rushed
• Poor choices and
bad shortcuts
VPC II
• Second attempt at
VPC: hire an expert
• Company mandate to
move from AWS Classic
to VPC
• 6 months of design, lots
of over engineering
VPC III
• Initial setup is less
than an hour
• Clean and simple
model
![Page 29: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/29.jpg)
Demo video: Bringing a cloud controller online
![Page 30: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/30.jpg)
Aviatrix - New VPC with AGW
![Page 31: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/31.jpg)
GREE purchased the
DragonSoul game from PerBlue
During the due diligence phase,
Aviatrix was used to link
GREE to the VPC
Use Case
![Page 32: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/32.jpg)
![Page 33: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/33.jpg)
W1 W2 W3 W4 W10
RDS
Aviatrix
Gateway
World
Game
Backend
Systems
Game
Admin
VPC
Payment ADS
Original Deployment
![Page 34: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/34.jpg)
Dragon Soul Deployment 1
Region 1 Region 2
IPSEC
Cross Account / Region Peering
Aviatrix Gateway Aviatrix Gateway
AWSAWSDragon Soul Deployment 2
VPC VPC
![Page 35: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/35.jpg)
W11Wn
RDS
11
ELB11Aviatrix
Gateway VPC ELB12 ELBn
W12
Auto Scaling
Group
Auto Scaling
Group
Auto Scaling
Group
RDS
12
RDS
n
Second Deployment
![Page 36: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/36.jpg)
Challenges
Single points of failure
RDS db.r3.8xlarge at capacity
World servers are stateful and unwinding the Java ORM
will take time.
Goals
Grow the game fast: double the
number of players
Don’t rock the boat: maintain continuity in existing
game service
![Page 37: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/37.jpg)
In Summary:
DevOps is not just technology, it’s people
using technology.
Docker is great
but it has some
sharp edges
Games have a
lifecycle,
nothing lasts
forever
CloudFormation
is great but it
is hard to get
started
Use dedicated
AWS sub-
accounts
Use Aviatrix
cloud
networking
People leave,
institutional
knowledge is lost,
so capture
everything in
automation
People take
shortcuts, magic
happens, so
enforce
automation
![Page 38: AWS re:Invent 2016: How Gree Launched New Games Faster and More Securely with AWS Marketplace and Amazon VPC (NET308)](https://reader031.vdocument.in/reader031/viewer/2022022203/586f7b301a28ab10258b770d/html5/thumbnails/38.jpg)
Thank you!