evgeniy karelin. mongo db integration example solving performance and high load problems. drupalcamp...
DESCRIPTION
This presentation is about real life example of using MongoDB on our not specific project for Drupal which supports more than 25m pageviews per day, more than 500k registered users with page load time less than 1sec. It will give an understanding how MongoDB can be easily used to increase performance of web-site. My presentation will be as easy as possible with simple examples and schemas but it will require at least intermediate level of developers. - project and tasks overview. http://freerice.com/ - quiz game site. There are a lot of dynamic info, users (registered and anonymous), groups and their different game statistis, user statuses etc. - problems while using MySQL - server optimization attempts. Memcache+Varnish. MySQL replication. Using game as separate script and AJAX blocks with "light" bootstrap. - MongoDB overview and it's benefits on current project. - PHP and MongoDB - project's MongoDB architecture overview - nodes and MongoDB - users/groups, their statistics and MongoDB - switching MySQL to MongoDB in Views. - indexing problems and statistic calculations. - multilingual support - scalability and using MongoDB replica set. - totals Этот доклад о применении MongoDB в одном из наших реальных проектов, который на данный момент обслуживает более 25млн показов страниц в день, более 500тыс зарегистрированных пользователей с скоростью загрузки страниц менее 1сек. Он позволит понять каким образом можно использовать MongoDB для увеличения производительности сайта. Мой доклад будет на столько простым на сколько это возможно с несложными схемами и примерами, но он требует как минимум среднего уровня разработчиков для полного понимания. - краткое описание проекта и поставленных задач. http://freerice.com/ - игра-викторина. много динамических данных, группы, игроки (зарегистрированные и анонимусы) и их статистики по разным параметрам, статусы игроков и т.д. - возникшие проблемы с работой MySQL. - попытки серверной оптимизации. Memcache+Varnish. Репликация MySQL. Перенос игры в отдельный скрипт и AJAX блоки с использованием "легкого" бутстрапа. - краткое описание MongoDB и приимущества его применения в текущем проекте. - PHP и MongoDB - общее описание архитектуры MongoDB на проекте. - работа с нодами в MongoDB. - работа с юзерами/группами и их статистиками. - переход с MySQL на MongoDB в Views. - проблемы с индексами, пересчеты статистик. - поддержка многоязычности. - масштабируемость MongoDB. Использование реплики. - итогиTRANSCRIPT
![Page 1: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/1.jpg)
MongoDB Integration Example Solving Performance And High
Load Problems
![Page 2: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/2.jpg)
Gold Sponsor ofDrupalCamp Kyiv 2011
![Page 3: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/3.jpg)
Silver Sponsors ofDrupalCamp Kyiv 2011
![Page 4: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/4.jpg)
![Page 5: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/5.jpg)
Freerice v2
Easy questions and categories management Users registration Facebook/Twitter integration Friendship system Groups Detailed game statistics per user/group/total,
rating system, user statuses
![Page 6: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/6.jpg)
Round 1 Questions = Nodes. Data imported.
Categories = Nodes + Taxonomy
User profiles = Content profile module
Groups = Organic groups module
Friendship system = Flag module
Game logic, statistics = custom
Blog, comments… = Core modules
![Page 7: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/7.jpg)
Beta test – OK
Live test - Fail
![Page 8: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/8.jpg)
MySQL is slow
Views
node_load()
user_load()
Organic group
Game data processing
…
= SLOW QUERIES
![Page 9: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/9.jpg)
What can help? Pressflow
Memcache
Varnish + AJAX Block
Game as separate script
MySQL replication
Code optimization
![Page 10: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/10.jpg)
![Page 11: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/11.jpg)
AJAX Blocks
AJAX blocks request -> index.php
New module for managing AJAX blocks
Set of blocks on page with needed bootstrap level
![Page 12: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/12.jpg)
Game as separate script
![Page 13: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/13.jpg)
MySQL is slow
![Page 14: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/14.jpg)
What is MongoDB?
Fast INSERTs
Fast SELECTs
Pretty fast UPDATEs
No JOINs
![Page 15: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/15.jpg)
PHP and MongoDB
PECL extension for MongoDB
Very easy to use
Redundancy of documentation and examples on both MongoDB and PHP.net sites
![Page 16: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/16.jpg)
Code examples
![Page 17: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/17.jpg)
Code example
![Page 18: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/18.jpg)
What is 1 right answer?
10 rice
1 update into user tables
1 update into group tables
1 update into totals tables
1 update in user status tables
…
![Page 19: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/19.jpg)
1.5-4.5 millions right answers per day
![Page 20: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/20.jpg)
statistics.grains collection
Every right answer = 1 insert
Statistics recalculation every night
"uid": 1, "cid": 17555, "gid": 0, "amt": 10, "dat": 1305496800, "lvl": 1
![Page 21: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/21.jpg)
Nodes collections
node.question, node.caterogy collection
hook_nodeapi() for insert, update, delete
Full Question and Category nodes into MongoDB
Questions and Categories SELECTs millions time often than INSERTs/UPDATEs
Nothing changed for site administration
![Page 22: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/22.jpg)
statistics.user_totals collection
![Page 23: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/23.jpg)
Other collections
statistic.group_totals
statistic.totals
statistic.useractivity
![Page 24: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/24.jpg)
Views
Bunch of views used on site
Views produce a lot of SELECTs into MySQL
It is not possible to cache them
It is not possible to replace them
![Page 25: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/25.jpg)
sview module
![Page 26: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/26.jpg)
![Page 27: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/27.jpg)
Sview disadvantages
Hard to support
No page/block view “from a box”, custom code needed
![Page 28: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/28.jpg)
Multilingual support
MySQL data is translated as usual
Nothing to translate from users, groups collections
Questions and Categories nodes are fully saved in MongoDB -> easy to translate
![Page 29: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/29.jpg)
MongoDB replication
![Page 30: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/30.jpg)
Why ReplicaSet
Data redundancy
Automated failover
Distribute read load
Simplify maintenance
Recommended for sharding
![Page 31: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/31.jpg)
How to use ReplicaSet
![Page 32: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/32.jpg)
Freerice today
560k registered users
17k groups
25m page views per day
9.3b rice gained = 1m meals
3 DB servers and 2 Front end servers
![Page 33: Evgeniy Karelin. Mongo DB integration example solving performance and high load problems. DrupalCamp Kyiv 2011](https://reader033.vdocument.in/reader033/viewer/2022052505/5553aa8db4c905d9448b4805/html5/thumbnails/33.jpg)
If you have any questions
http://www.mongodb.org/ http:/php.net/manual/en/book.mongo.php mail: [email protected]
skype: smart_here