Конференция "AI&BigData Lab", 12 апреля 2014

1. MagnetoDB
Key-Value Storage
service for
OpenStack
Mirantis, 2014

2. Serge.Kovaleff
gmail.com
linkedin.com
facebook.com
@
sviridov.ilya@gmail.com
Odessa, GeeksLab 4/12/2014@
isviridov at FreeNode

3. WII FM

4. WII F

5. WII FM

9. What the heck is OpenStack?

10. What the heck is OpenStack?

11. to be continued ...

13. Is it Big?
Community
1227 active contributors in IceHouse
more than 166 organizations
Communication
44 active official IRC channels
2887 e-mails in March

14. Is it Big?

15. Is it about data?
CERN
- 3 PB per custer
- 50000 + 35000 core
PayPal 80000 VMs (according to Forbes)

16. OpenStack Big Data
OpenStack Sahara - Elastic Hadoop clusters
provisioning and management on OpenStack and elastic data
processing (on-demand Hadoop job workflow)
OpenStack MagentoDB - key-value storage service

17. What is MagnetoDB?
MagnetoDB - OpenSource Amazon DynamoDB API
implementation for OpenStack, the key-value database service
for storing any amount of data with seamless scalability and
predictable performance.

18. MagnetoDB key features
● Easy-to-integrate REST-like API (AWS SDK, boto
clients)
● OpenStack standards following REST API
● Schemaless, non-relational table-based model
● Put/get/query/scan item operations
● Eventual and strong consistency reads
● Local Secondary indexes
● Batch read/write operations
● Designed to handle any amount of data and any level
of request traffic
● Seamless throughput and storage scaling
● Fault tolerance

19. MagnetoDB use cases
● High request traffic level applications
● Time series data
● Queries are distributed across the data
● Any other applications
● Migration of workloads between AWS and
OpenStack

20. Architecture highlights
● Layered architecture
● Horizontally scalable
● Pluggable database backend
● Uses native OpenStack infrastructure
○ OpenStack HEAT for provisioning
○ OpenStack Neutron LoadBalancer
○ Integration with OpenStack Keystone planned

21. Overall architecture

22. Integration with OpenStack services

23. Current status
○ 2.0.2 version released
○ Open source, Apache 2 license
○ Launchpad homepage: https://launchpad.net/magnetodb
○ Code published on github: https://github.
com/stackforge/magnetodb
○ Deployable to Vanilla OpenStack
○ Compatible with existing DynamoDB clients

24. MagnetoDB Pilot Features
● Table CRUD API
○ ListTables
○ DescribeTable
○ CreateTable
○ DeleteTable
● Item CRUD API
○ PutItem
○ UpdateItem
○ DeleteItem
○ GetItem
● Data querying API
○ Query
○ Scan
● Error Handling API

25. MagentoDB use-case: dynamodb-geo
“Geo Library for Amazon DynamoDB
allows to easily create and query
geospatial data”

26. Next steps
● General
○ Move the development to OpenStack Infra - done
○ Integration with OpenStack Keystone - done
● MagnetoDB API, following OpenStack tenets - in progress
● DynamoDB API
○ Atomic counters
○ Asynchronous table creation
○ Atomic conditional write operations
○ Batch operations - in progress
● Other features
○ Management API
○ Quota management

27. Summary
● The 2.0.2 implementation is available on GitHub in public
repository
● MagnetoDB is compatible with AWS SDK and boto library
● It is now possible to run applications which use DynamoDB as
a storage on top of OpenStack using MagnetoDB

28. Links and Resources
○ Wiki: https://wiki.openstack.org/wiki/MagnetoDB
○ Launchpad: https://launchpad.net/magnetodb/
○ Source code: https://github.com/stackforge/magnetodb
○ IRC: #magnetodb @ FreeNode
○ AWS DynamoDB http://aws.amazon.com/dynamodb/
○ dynamodb-geo https://github.com/awslabs/dynamodb-geo

29. Q&A
Got Questions?