Riak Core--an open-source Erlang library created by Basho Technologies that powers Riak KV and Riak Search--allows developers to build distributed, scalable, failure-tolerant applications based on a generalized version of Amazon's Dynamo architecture. In this talk, Rusty explains why Riak Core was built, discuss what problems it solves and how it works, and walk through the steps to using Riak Core in an Erlang application.
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
Riak Core: Building Distributed Applications Without Shared State
1. Riak Core:
Building Distributed
Applications Without
Shared State
Commercial Users of Functional Programming
Baltimore, MD · October 2010
Rusty Klophaus (@rklophaus)
Basho Technologies
Wednesday, October 6, 2010
2. You suddenly feel
an uncontrollable desire to
learn Erlang.
http://www.flickr.com/photos/procsilas/18014203
2
Wednesday, October 6, 2010
3. What is Riak Core?
How does it work?
How can you use it?
3
Wednesday, October 6, 2010
13. “We Generalized the
Dynamo Architecture and
Open-Sourced the Bits.”
13
Wednesday, October 6, 2010
14. Wait, doesn’t *Erlang* let you build
distributed, scalable, failure-tolerant
applications?
14
Wednesday, October 6, 2010
15. Erlang makes it easy to connect the
components of your application.
Client
Service A Service B
Service C
Queue E
Resource D
Wednesday, October 6, 2010
16. Riak Core helps you build a service that
harnesses the power of many nodes.
Node A Node B Node C Node D
Node E Node F Node G Node H
Service
Node I Node J Node K Node L
Node M Node N Node O ...
Wednesday, October 6, 2010
17. “People use languages
other than Erlang?!?
I find that hilarious.”
-Al Gore actually said this to me.
http://www.flickr.com/photos/scobleizer/2216445692
Wednesday, October 6, 2010
18. How does Riak Core work?
18
Wednesday, October 6, 2010
19. Command ObjectName, Payload
Wednesday, October 6, 2010
21. Hash the Object Name
Command ObjectName, Payload
SHA1(ObjName), Payload
0 to 2^160
Wednesday, October 6, 2010
22. A Naive Approach
Command ObjectName, Payload
SHA1(ObjName), Payload
Node A Node B Node C Node D
Wednesday, October 6, 2010
23. A Naive Approach
Command ObjectName, Payload
SHA1(ObjName), Payload
Node A Node B Node C Node D Node E
Wednesday, October 6, 2010
24. "All problems in computer
science can be solved by
another level of indirection."
- David Wheeler
24
Wednesday, October 6, 2010
25. Add VNodes
What
Virtual Node. Logical subdivision of the cluster.
Handles incoming commands, does work, replies.
For Parallelism
# of VNodes = maximum concurrent requests
For Rebalancing the Cluster
Smallest block that can be shifted to a new node.
For Resiliance
The system restarts failed VNodes.
25
Wednesday, October 6, 2010
26. Routing with Consistent Hash
Command ObjectName, Payload
SHA1(ObjName), Payload
VNode 0 VNode 1 VNode 2 VNode 3 VNode 4 VNode 5 VNode 6 VNode 7
Node A Node B Node C Node D
Wednesday, October 6, 2010
27. Adding a Node
Command ObjectName, Payload
SHA1(ObjName), Payload
VNode 0 VNode 1 VNode 2 VNode 3 VNode 4 VNode 5 VNode 6 VNode 7
Node A Node B Node C Node D Node E
Wednesday, October 6, 2010
28. Removing a Node
Command ObjectName, Payload
SHA1(ObjName), Payload
VNode 0 VNode 1 VNode 2 VNode 3 VNode 4 VNode 5 VNode 6 VNode 7
Node A Node B Node C Node D Node E
Wednesday, October 6, 2010
29. The Ring
Hash Location
Wednesday, October 6, 2010
34. Router in the Middle
Client Client Client
Router
VNode VNode VNode VNode VNode VNode VNode VNode
0 1 3 4 2 5 6 7
Node A Node B Node C Node D Node E
Wednesday, October 6, 2010
35. Riak Core - Router on Each Node
Client Client Client
Router Router Router Router Router
VNode VNode VNode VNode VNode VNode VNode VNode
0 1 3 4 2 5 6 7
Node A Node B Node C Node D Node E
Wednesday, October 6, 2010
36. Eventually - Router on the Client
Client Client Client
Router Router Router
VNode VNode VNode VNode VNode VNode VNode VNode
0 1 3 4 2 5 6 7
Node A Node B Node C Node D Node E
Wednesday, October 6, 2010
37. No Shared State
Router Router Router Router Router
VNode VNode VNode VNode VNode VNode VNode VNode
0 1 3 4 2 5 6 7
Node A Node B Node C Node D Node E
Wednesday, October 6, 2010
38. Gossip
Local Incoming
Ring State Ring State
Are rings equivalent?
Strictly descendent?
Or different?
Wednesday, October 6, 2010
39. Handoff
When
Nodes is added to the system.
Node is removed from the system.
Node has temporarily failed.
What
Ship the data backing a VNode from one node to
another.
39
Wednesday, October 6, 2010
40. Not Mentioned
Vector Clocks
Merkle Trees
Bloom Filters
40
Wednesday, October 6, 2010
41. Distinguished
gentlemen
prefer
Erlang.
http://www.flickr.com/photos/rebcal/3987226359
Wednesday, October 6, 2010
42. How do you use Riak Core?
42
Wednesday, October 6, 2010
43. Two Things to Think About
Command
Command = ObjectName, Payload
The commands/requests/operations that you will send
through the system.
VNode Module
The callback module that will receive the commands.
43
Wednesday, October 6, 2010
50. Start the riak_core application
riak_core
riak_core_vnode_sup riak_core_handoff_* riak_core_node_*
X_vnode X_vnode
riak_core_ring_* riak_core_gossip_*
X_vnode ...
Monitor node liveness,
broadcast to registered modules.
50
Wednesday, October 6, 2010
51. Start the riak_core application
riak_core
riak_core_vnode_sup riak_core_handoff_* riak_core_node_*
X_vnode X_vnode
riak_core_ring_* riak_core_gossip_*
X_vnode ...
Send ring information to other nodes.
Reconcile different views of the cluster.
Rebalance cluster when nodes join or leave.
51
Wednesday, October 6, 2010
52. In your application...
riak_core
riak_core_vnode_sup riak_core_handoff_* riak_core_node_*
X_vnode X_vnode
riak_core_ring_* riak_core_gossip_*
X_vnode ...
Start the vnodes for your application.
Master = {
riak_X_vnode_master, {
riak_core_vnode_master, start_link, [riak_X_vnode]
},
permanent, 5000, worker, [riak_core_vnode_master]
},
{ok, { {one_for_one, 5, 10}, [Master]} }.
52
Wednesday, October 6, 2010
53. In your application...
riak_core
riak_core_vnode_sup riak_core_handoff_* riak_core_node_*
X_vnode X_vnode
riak_core_ring_* riak_core_gossip_*
X_vnode ...
Tell riak_core that your application
is ready to receive requests.
riak_core:register_vnode_module(riak_X_vnode),
riak_core_node_watcher:service_up(riak_X,
self())
53
Wednesday, October 6, 2010
54. In your application...
riak_core
riak_core_vnode_sup riak_core_handoff_* riak_core_node_*
X_vnode X_vnode
riak_core_ring_* riak_core_gossip_*
X_vnode ...
riak_core
riak_core_vnode_sup riak_core_handoff_* riak_core_node_*
X_vnode X_vnode
riak_core_ring_* riak_core_gossip_*
X_vnode ...
Join to an existing node in the cluster.
riak_core_gossip:send_ring(ClusterNode,
node())
54
Wednesday, October 6, 2010
55. Start Sending Commands
# Figure out the preflist...
{_Verb, ObjName, _Payload} = Command,
PrefList = riak_core_apl:get_apl(ObjName,
NVal,
riak_X),
# Send the command...
riak_core_vnode_master:command(PrefList,
Command,
riak_X_vnode_master)
55
Wednesday, October 6, 2010
56. Review
Riak Core
Open source Erlang library for building distributed, scalable,
failure tolerant applications.
Continual improvement in the coming months.
Riak KV
Key/Value datastore with map/reduce based on Riak Core.
Riak Search
Full-text, near real-time search engine based on Riak Core.
56
Wednesday, October 6, 2010
57. At the center of the universe,
beneath the bottom-most turtle,
is a cluster of Erlang nodes.
http://www.flickr.com/photos/jurvetson/469492885
Wednesday, October 6, 2010
58. At the center of the universe,
beneath the bottom-most turtle,
is a cluster of Erlang nodes.
With 100% uptime.
http://www.flickr.com/photos/jurvetson/469492885
Wednesday, October 6, 2010
59. Thanks! Questions?
Learn More
More Information: http://wiki.basho.com
Amazon’s Dynamo Paper
Get the Code
http://hg.basho.com/riak_core
Get in Touch
Twitter: @rklophaus, @basho/team
Email: rusty@basho.com
59
Wednesday, October 6, 2010