utopia kingdoms scaling case. from 4 users to 50.000+
Post on 19-May-2015
378 Views
Preview:
DESCRIPTION
TRANSCRIPT
● scaling case. From 4 users to 90k+●
● Jaime Buelta● Soft. Developer at
●
The Game
Get image from game
Utopia Kingdoms
● Fantasy strategy game● Build your own Kingdom● Create armies and attack other Kingdoms● Join other Kingdoms in an Alliance● Manage resources● Available in Facebook and Kongregate
http://www.facebook.com/UtopiaKigdomsGamehttp://www.kongregate.com/games/JoltOnline/utopia-kingdoms
Technology stack
Technology Stack - Backend
PythonCherrypy framework
Amazon SimpleDBLinux in Amazon EC2
Stack of technologies - Frontend
HTML( generated by Genshi templates)
jQuery
Stack of technologies - Frontend
HTML( generated by Genshi templates)
jQuery
Some points of interest(will discuss them later)
● Your resources (population, gold, food, etc) grows with time
● You actions (build something, attack a player) typically takes some time
● Players are ranked against the rest● You can add friends and enemies
Do not guess
Measure
Measurement tools
● OS tools● Task manager (top)● IO Monitor (iostat)
● Monitoring tools (Munin, Nagios)
● Logs● Needs to find a good compromise detailed/relevance
● Profiling
You've got to love profiling
● Generate profiles with cProfile module
Profile whole application with
python -m cProfile -o file.prof my_app.py
(not very useful in a web app)
● If you're using a framework, profile only your functions to reduce noise
Profile decorator (example)
def profile_this(func):
import cProfile
prof = cProfile.Profile()
retval = prof.runcall(func)
filename = 'profile-{ts}.prof'.format(time.time())
prof.dumpstats(filename)
return retval
Analyzing profile
● gprof2dot● Using dot, convert to graph
gprof2dot -f pstats file.prof | dot -Tpng -o file.png
● Good for workflows
● RunSnakeRun● Good for cumulative times
Example of RunSnakeRunRAZR
Example of gprof2dot
The power of cache
All static should be out of python
● Use a good web server to serve all static content (Static HTML, CSS, JavaScript code)
● Some options● Apache● Nginx● Cherokee● Amazon S3
Use memcached(and share the cache between your servers)
Example
● Asking for friends/enemies to DB● Costly request in SimpleDB (using SQL statement)
● On each request● Cache the friends on memcache for 1 hour● Invalidate the cache if adding/removing
friends or enemies
Caching caveats
● Cache only after knowing there is a problem● Do not trust in cache for storage● Take a look on size of cached data● Choosing a good cache time can be difcult /
Invalidate cache can be complex● Some data is too dynamic to be cached
Caching is not just memcached
● More options available:● Get on memory on start● File cache● Cache client side
Parse templates just once
● The template rendering modules have options to parse the templates just once
● Be sure to activate it in production● In development, you'll most likely want to
parse them each time
● Same apply to regex, specially complex ones
More problems
Rankings
● Sort players on the DB is slow when you grow the number of players
● Solution:● Independent ranking server (operates just in
memory)● Works using binary trees● Small Django project, communicates using xmlrpc
● Inconvenient:● Data is not persistent, if the rankings server goes
down, needs time to reconstruct the rankings
Database pulling - Resources
● There was a process just taking care of the growth of resources.● It goes element by element, and increasing the
values● It pulls the DB constantly, even when the user has
their values to maximum● Increment the resources of a user just the next
time is accessed (by himself or by others)● No usage of DB when the user is not in use● The request already reads from DB the user
Database pulling - Actions
● Lots of actions are delayed. Recruit a unit, buildings, raids...
● A process check each user if an action has to be done NOW.● Tons of reads just to check “not now”● Great delay in some actions, as they are not
executed in time
Database pulling - Actions
● Implement a queue to execute the actions at the proper time:● Beanstalk (allows deferred extraction)● A process listen to this queue and performs the
action, independently from request servers.● The process can be launched in a diferent
machine.● Multiple process can extract actions faster.
DataBase Issues
Amazon SimpleDB
● Key – Value storage● Capable of SQL queries● Store a dictionary (schemaless, multiple
columns)● All the values are strings● Access through boto module● Pay per use
Problems with SimpleDB
● Lack of control● Can't use local copy
– In development, you must access Amazon servers (slow and costly)
● Can't backup except manually● Can't analyze or change DB (e.g. can't define
indexes)● Can't monitor DB
Problems with SimpleDB
● Bad tool support● Slow and high variability (especially on SQL
queries)● Sometime, the queries just timeout and had to be
repeated.
Migrate to MongoDB
MongoDB
● NoSQL● Schemaless● Fast● Allow complex queries● Retain control (backups, measure queries, etc)● Previous experience using it from ChampMan
Requisites of the migration
● Low-level approach● Objects are basically dictionaries● Be able to save dirty fields (avoid saving
unchanged values)● Log queries to measure performance
MongoSpell● Thin wrap over pymongo● Objects are just dictionary-like elements● Minimal schema● Fast!● Able to log queries● It will probably be released soon as Open
Source
Definition of collections
class Spell(Document):
collection_name = 'spells'
needed_fields = ['name',
'cost',
'duration']
optional_fields = [
'elemental',
]
activate_dirty_fields = True
indexes = ['name__unique', 'cost']
Querying from DB
Spell.get_from_db(name='fireball')
Spell.filter()
Spell.filter(sort='name')
Spell.filter(name__in=['fireball', 'magic missile'])
Spell.filter(elemental__fire__gt=2)
Spell.filter(duration__gt=2,
cost=3, hint='cost')
Spell.filter(name='fireball', only='cost')
Some features
● Dirty fields● No type checks● Query logs● 10x faster than SimpleDB!!!
Query logs
[07:46:06]- 2.6 ms – get_from_db - Reinforcement - Reinforcements.py(31)[07:46:06]- 4.3 ms - get_from_db - Player - Player.py(876)[07:46:10]- 0.1 ms - filter - Membership- AllianceMembership.py(110) [07:46:10]- 1.3 ms - get_from_db - Reinforcement -Reinforcements.py(31)[07:46:10]- 1.4 ms - get_from_db - Notifications - Notifications.py (56)
Scalability vs Efciency
Scalable vs Efcient
Scalable● Can support more
users adding more elements
Efficient● Can support more
users with the same elements
Work on both to achieve your goals
Keep measuring and improving!(and monitor production to be proactive)
Thank you for your interest!
Questions?
jaime.buelta@gmail.comhttp://WrongSideOfMemphis.wordpress.com
http://www.joltonline.com
top related