manetodb: key/value storage, bigdata in open stack_Сергей Ковалев, Илья...
DESCRIPTION
Конференция "AI&BigData Lab", 12 апреля 2014TRANSCRIPT
![Page 1: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/1.jpg)
MagnetoDB Key-Value Storageservice for OpenStack
Mirantis, 2014
![Page 2: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/2.jpg)
Serge.Kovaleffgmail.comlinkedin.comfacebook.com
@
Odessa, GeeksLab 4/12/2014@
isviridov at FreeNode
![Page 3: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/3.jpg)
WII FM
![Page 4: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/4.jpg)
WII F
![Page 5: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/5.jpg)
WII FM
![Page 6: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/6.jpg)
![Page 7: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/7.jpg)
![Page 8: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/8.jpg)
![Page 9: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/9.jpg)
What the heck is OpenStack?
![Page 10: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/10.jpg)
What the heck is OpenStack?
![Page 11: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/11.jpg)
to be continued ...
![Page 12: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/12.jpg)
![Page 13: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/13.jpg)
Is it Big?
Community
1227 active contributors in IceHousemore than 166 organizations
Communication
44 active official IRC channels2887 e-mails in March
![Page 14: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/14.jpg)
Is it Big?
![Page 15: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/15.jpg)
Is it about data?
CERN- 3 PB per custer- 50000 + 35000 core
PayPal 80000 VMs (according to Forbes)
![Page 16: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/16.jpg)
OpenStack Big Data
OpenStack Sahara - Elastic Hadoop clusters provisioning and management on OpenStack and elastic data processing (on-demand Hadoop job workflow)
OpenStack MagentoDB - key-value storage service
![Page 17: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/17.jpg)
What is MagnetoDB?
MagnetoDB - OpenSource Amazon DynamoDB API implementation for OpenStack, the key-value database service for storing any amount of data with seamless scalability and predictable performance.
![Page 18: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/18.jpg)
MagnetoDB key features ● Easy-to-integrate REST-like API (AWS SDK, boto
clients)● OpenStack standards following REST API● Schemaless, non-relational table-based model● Put/get/query/scan item operations● Eventual and strong consistency reads● Local Secondary indexes● Batch read/write operations● Designed to handle any amount of data and any level
of request traffic● Seamless throughput and storage scaling● Fault tolerance
![Page 19: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/19.jpg)
MagnetoDB use cases
● High request traffic level applications
● Time series data
● Queries are distributed across the data
● Any other applications
● Migration of workloads between AWS and
OpenStack
![Page 20: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/20.jpg)
Architecture highlights
● Layered architecture
● Horizontally scalable
● Pluggable database backend
● Uses native OpenStack infrastructure
○ OpenStack HEAT for provisioning
○ OpenStack Neutron LoadBalancer
○ Integration with OpenStack Keystone planned
![Page 21: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/21.jpg)
Overall architecture
![Page 22: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/22.jpg)
Integration with OpenStack services
![Page 23: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/23.jpg)
Current status
○ 2.0.2 version released
○ Open source, Apache 2 license
○ Launchpad homepage: https://launchpad.net/magnetodb
○ Code published on github: https://github.
com/stackforge/magnetodb
○ Deployable to Vanilla OpenStack
○ Compatible with existing DynamoDB clients
![Page 24: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/24.jpg)
MagnetoDB Pilot Features
● Table CRUD API○ ListTables○ DescribeTable○ CreateTable○ DeleteTable
● Item CRUD API○ PutItem○ UpdateItem○ DeleteItem○ GetItem
● Data querying API○ Query○ Scan
● Error Handling API
![Page 25: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/25.jpg)
MagentoDB use-case: dynamodb-geo
“Geo Library for Amazon DynamoDB allows to easily create and query geospatial data”
![Page 26: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/26.jpg)
Next steps● General
○ Move the development to OpenStack Infra - done○ Integration with OpenStack Keystone - done
● MagnetoDB API, following OpenStack tenets - in progress● DynamoDB API
○ Atomic counters○ Asynchronous table creation○ Atomic conditional write operations○ Batch operations - in progress
● Other features○ Management API○ Quota management
![Page 27: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/27.jpg)
Summary
● The 2.0.2 implementation is available on GitHub in public
repository
● MagnetoDB is compatible with AWS SDK and boto library
● It is now possible to run applications which use DynamoDB as
a storage on top of OpenStack using MagnetoDB
![Page 28: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/28.jpg)
Links and Resources
○ Wiki: https://wiki.openstack.org/wiki/MagnetoDB
○ Launchpad: https://launchpad.net/magnetodb/
○ Source code: https://github.com/stackforge/magnetodb
○ IRC: #magnetodb @ FreeNode
○ AWS DynamoDB http://aws.amazon.com/dynamodb/
○ dynamodb-geo https://github.com/awslabs/dynamodb-geo
![Page 29: ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья Свиридов](https://reader031.vdocument.in/reader031/viewer/2022020207/54c66e684a795953538b4626/html5/thumbnails/29.jpg)
Q&A
Got Questions?