deploying the graph
DESCRIPTION
A talk given in November 2013 to the infrastructure coders group in Melbourne on how PlayUp deploys the backend for it's latest generation of applications, using packer and amazon web services.TRANSCRIPT
Infrastructure Coders - Melbourne - November 2013Kevin O’Neill - CTO PlayUp - @kevinoneill
Kevin O’Neill CTO PlayUp
Deploying the Graph
@kevinoneill
November 2013
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
A little History
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Engage & Entertain Around Live Sport
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Mobile Games
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
SMS Based
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Number of Runs
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Complex
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Carrier Integration
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Smart Phone Revolution
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Move from a Content Creator to a Content
Enabler
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Top 10 in Engagement
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Over 3 Million Likes
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
30% - 40% Active
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Manually Curated
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
We Know there's an Opportunity
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
The Graph
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Scala
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Scala
• A Scalable language
• Object-Oriented
• Functional
• Seamless Java Interoperability
• Functions are Objects
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
AKKA
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
AKKA
• Actors
• Fault Tolerance
• Location Transparency
• Transactors
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Neo4j
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Neo4j
• Graph Database
• Vertices and Edges
• Properties
• Full Text Index
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
WhiteboardOutlining the bits
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Runtime
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
AWS
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
AWS
• us-west
• m2.2xlarge
• ubuntu 13.04
• jre 7
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
JVM
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
JVM
• Graph - 8G of Heap
• Others 128M - 1G
• Parallel GC
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
The remaining memory is reserved for file system cache and
memory mapped buffers
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Building the Components
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
SBT
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Builds the jars
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Debian Packages
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Debian Packages
• Places binaries in opt
• Adds an upstart file for the
• Permissions on data and log directories
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
graph-server.upstart
setuid ubuntu setgid ubuntu !console log !chdir /var/lib/playup/spot/graph-‐server !exec java -‐server -‐Xms4096M -‐Xmx8192M -‐Xss1M \ -‐XX:MaxPermSize=256M -‐XX:+UseParallelGC \ -‐jar /opt/playup/spot/graph-‐server/run.jar \ com.playup.spot.graph.service.NeoKernel
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Base configuration is for production
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Baking
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Packer
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Create machine images for multiple platforms from a single config
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Images have lots of benefits
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Fast Deployment
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Stability
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Testability
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Packer makes creating images a breeze
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Example
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Builders
"builders": [ { "type": "amazon-‐ebs", "instance_type": "t1.micro", "region": "ap-‐southeast-‐2", "ami_name": "spot-‐graph-‐{{user `version`}}-‐{{timestamp}}", "source_ami": "ami-‐4bf26f71", "ssh_username": "ubuntu", "tags": { "Name": "spot-‐graph-‐{{user `version`}} ({{isotime}})", "OS": "Ubuntu", "Version": "{{user `version`}}-‐{{timestamp}}" }, "ami_regions":["us-‐west-‐2"] } …
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Builders
"builders": [ { "type": "amazon-‐ebs", "instance_type": "t1.micro", "region": "ap-‐southeast-‐2", "ami_name": "spot-‐graph-‐{{user `version`}}-‐{{timestamp}}", "source_ami": "ami-‐4bf26f71", "ssh_username": "ubuntu", "tags": { "Name": "spot-‐graph-‐{{user `version`}} ({{isotime}})", "OS": "Ubuntu", "Version": "{{user `version`}}-‐{{timestamp}}" }, "ami_regions":["us-‐west-‐2"] } …
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Builders
"builders": [ { "type": "amazon-‐ebs", "instance_type": "t1.micro", "region": "ap-‐southeast-‐2", "ami_name": "spot-‐graph-‐{{user `version`}}-‐{{timestamp}}", "source_ami": "ami-‐4bf26f71", "ssh_username": "ubuntu", "tags": { "Name": "spot-‐graph-‐{{user `version`}} ({{isotime}})", "OS": "Ubuntu", "Version": "{{user `version`}}-‐{{timestamp}}" }, "ami_regions":["us-‐west-‐2"] } …
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Builders
"builders": [ { "type": "amazon-‐ebs", "instance_type": "t1.micro", "region": "ap-‐southeast-‐2", "ami_name": "spot-‐graph-‐{{user `version`}}-‐{{timestamp}}", "source_ami": "ami-‐4bf26f71", "ssh_username": "ubuntu", "tags": { "Name": "spot-‐graph-‐{{user `version`}} ({{isotime}})", "OS": "Ubuntu", "Version": "{{user `version`}}-‐{{timestamp}}" }, "ami_regions":["us-‐west-‐2"] } …
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Builders
"builders": [ { "type": "amazon-‐ebs", "instance_type": "t1.micro", "region": "ap-‐southeast-‐2", "ami_name": "spot-‐graph-‐{{user `version`}}-‐{{timestamp}}", "source_ami": "ami-‐4bf26f71", "ssh_username": "ubuntu", "tags": { "Name": "spot-‐graph-‐{{user `version`}} ({{isotime}})", "OS": "Ubuntu", "Version": "{{user `version`}}-‐{{timestamp}}" }, "ami_regions":["us-‐west-‐2"] } …
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Variables
"variables": { "version": "development" }
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Provisioners "provisioners": [ { "script": "scripts/update-‐packages.sh", "type": "shell" }, { "script": "scripts/install-‐puppet.sh", "type": "shell" }, { "type": "file", "source": "packages", "destination": "/tmp" }, { "script": "scripts/install-‐spot-‐packages.sh", "type": "shell" }, { "type": "puppet-‐masterless", "manifest_file": "puppet/spot-‐graph.pp", "module_paths": ["puppet/local", "puppet/common"] }, { "script": "scripts/build-‐date.sh", "type": "shell" }
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Provisioners "provisioners": [ { "script": "scripts/update-‐packages.sh", "type": "shell" }, { "script": "scripts/install-‐puppet.sh", "type": "shell" }, { "type": "file", "source": "packages", "destination": "/tmp" }, { "script": "scripts/install-‐spot-‐packages.sh", "type": "shell" }, { "type": "puppet-‐masterless", "manifest_file": "puppet/spot-‐graph.pp", "module_paths": ["puppet/local", "puppet/common"] }, { "script": "scripts/build-‐date.sh", "type": "shell" }
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Provisioners "provisioners": [ { "script": "scripts/update-‐packages.sh", "type": "shell" }, { "script": "scripts/install-‐puppet.sh", "type": "shell" }, { "type": "file", "source": "packages", "destination": "/tmp" }, { "script": "scripts/install-‐spot-‐packages.sh", "type": "shell" }, { "type": "puppet-‐masterless", "manifest_file": "puppet/spot-‐graph.pp", "module_paths": ["puppet/local", "puppet/common"] }, { "script": "scripts/build-‐date.sh", "type": "shell" }
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Provisioners "provisioners": [ { "script": "scripts/update-‐packages.sh", "type": "shell" }, { "script": "scripts/install-‐puppet.sh", "type": "shell" }, { "type": "file", "source": "packages", "destination": "/tmp" }, { "script": "scripts/install-‐spot-‐packages.sh", "type": "shell" }, { "type": "puppet-‐masterless", "manifest_file": "puppet/spot-‐graph.pp", "module_paths": ["puppet/local", "puppet/common"] }, { "script": "scripts/build-‐date.sh", "type": "shell" }
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Builder Provisioners
"provisioners": [ ! … ! { "type": "shell", "script": "scripts/vagrant.sh", "only": ["virtualbox"] }, { "type": "shell", "script": "scripts/guest-‐additions.sh", "only": ["virtualbox"] }, ! …
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Build
$ packer build -‐-‐only "amazon-‐ebs" template.json
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
In a few minutes you have a shiny new image
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
CI
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
CI
• Variables
• source ami
• region
• default build is for local development
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Starting an instance
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Remember
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Backed config is for Production*
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
We have staging plus multiple developer environments
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
CloudInit
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Configure values in /etc/playup/…
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Configure the puppet master*
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Cloud Formation
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Which images
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
What CloudInit values
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Challenges
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Too many to list …
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Persistant Data
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Service Registration
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Cluster Membership
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Investigating
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Docker
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Apache Mesos
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Come Join Us
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Disrupting
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Hiring Selectively
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Talk to Me or Andrea
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
You miss 100 percent of the the shots you don't take
– Wayne Gretzky
Kevin O’Neill - CTO PlayUp - @kevinoneill Infrastructure Coders - Melbourne - November 2013
Kevin O’NeillCTO - PlayUp
[email protected]@kevinoneill
Questions?