Slides from my Lambda Jam 2013 talk: Distributed Programming with Erlang, Riak Core, and Riak Pipe

1. DISTRIBUTED SYSTEMS:
ERLANG, RIAK CORE,
RIAK PIPE
A few clues, a few rants, and a call to action
1Tuesday, July 9, 13

2. ABOUT ME
• Technical Evangelist for Basho
• @macintux
• jdaily@basho.com
• http://tinyurl.com/jrdlj13
2Tuesday, July 9, 13

3. ZERG.ERLANGONXEN.COM
3Tuesday, July 9, 13

4. THOU SHALT LOOK AT
ERLANG
John’s first commandment of infrastructure software
4Tuesday, July 9, 13

5. 5Tuesday, July 9, 13

6. 6Tuesday, July 9, 13

7. The Erlang virtual machine provides fault-tolerance
features and a highly scalable architecture while the Erlang
programming language keeps the required source code
small and easy to follow.
It’s important to understand that few programming
languages can provide real fault-tolerance with scalability.
In fact, I’d say Erlang is roughly alone in this regard.
MICHAELTRUOG
7Tuesday, July 9, 13

8. WHY DISTRIBUTED
PROGRAMMING IS HARD
• Clocks
8Tuesday, July 9, 13

9. WHY DISTRIBUTED
PROGRAMMING IS HARD
• Clocks
• Latency
9Tuesday, July 9, 13

10. WHY DISTRIBUTED
PROGRAMMING IS HARD
• Clocks
• Latency
• Lost messages
10Tuesday, July 9, 13

11. WHY DISTRIBUTED
PROGRAMMING IS HARD
• Clocks
• Latency
• Lost messages
• Servers break
11Tuesday, July 9, 13

12. 12Tuesday, July 9, 13

13. CAP LIGHTNINGTALK
13Tuesday, July 9, 13

14. 14Tuesday, July 9, 13

15. 15Tuesday, July 9, 13

16. 16Tuesday, July 9, 13

17. 17Tuesday, July 9, 13

18. 18Tuesday, July 9, 13

19. /CAP LIGHTNINGTALK
/memes
19Tuesday, July 9, 13

20. • Key/value store
DYNAMO & RIAK
20Tuesday, July 9, 13

21. • Key/value store
• Consistent hashing
DYNAMO & RIAK
21Tuesday, July 9, 13

22. • Key/value store
• Consistent hashing
• Granular compute engines
DYNAMO & RIAK
22Tuesday, July 9, 13

23. • Key/value store
• Consistent hashing
• Granular compute engines
• Replication
DYNAMO & RIAK
23Tuesday, July 9, 13

24. 24Tuesday, July 9, 13

25. 25Tuesday, July 9, 13

26. The spirit of the cloud is ‘every node can die’, writing the
administration software for a cloud expecting the
administration nodes to stay alive is hypocritical.
Yet just saying ‘hey the API will be down if you lose the
controlling node’ kind of sucks so it comes down having a
distributed system either with stateless nodes and a
distributed database or with distributed nodes and a local
database.
HEINZ GIES, PROJECT FIFO
26Tuesday, July 9, 13

27. 27Tuesday, July 9, 13

28. 28Tuesday, July 9, 13

29. RIAK PIPE
Like UNIX pipes, but not!
29Tuesday, July 9, 13

30. RIAK PIPE COMPONENTS
• Pipeline (client, fittings, sink)
• Fitting (specification of behavior, partitioner)
• Coordinator (one/fitting/cluster)
• Worker process (one/fitting/vnode, implements a behavior)
30Tuesday, July 9, 13

31. RIAK PIPE PROCESSING
1. Client submits a pipeline specification
2. Coordinators are spawned from last to first
3. When input arrives at a vnode, if no worker for that fitting
exists yet the appropriate coordinator will be asked for the
fitting specification
4. The last fitting sends its output to the sink, which is typically
the client
31Tuesday, July 9, 13

32. RIAK PIPE DEBUGGING
• Log and trace messages are optionally sent to the sink
• riak_pipe:status will return a list of fittings; each item is a list of
worker details: node ID, state, queue length, idle time, etc
• Fittings can define a validator function to be invoked on inputs
before processing
• Intermediate outputs can be forwarded directly to sink
32Tuesday, July 9, 13

33. THE END IS NIGH
• Riak 1.4
• RICON
• Meetups
33Tuesday, July 9, 13