A discussion of Erlang/OTP, based on the "A History of Erlang" paper, with the addition of covering some of the methods used in the paper that enable Riak to be a highly reliable, distributed datastore, while leveraging the work of giants.

1. Erlang/OTP in Riak
Twitter: @Sargun
Chief Solutions Architect, Basho

2. Erlang/OTP

3. History

4. “To provide a better way of programming
telephony applications”
The Mission

5. Birthplace:
Ericsson Computer
Science Lab

6. Requirements
• Handling a very large number of concurrent activities
• Actions to be performed at a certain point of time or within a certain time
• Systems distributed over several computers
• Interaction with hardware must be abstracted
• Very large software systems
• Complex functionality such as feature interaction
• Continuous operation over several years
• Software maintenance (reconfiguration, etc.) without stopping the system
• Stringent quality and reliability requirements
• Fault tolerance both to hardware failures and software errors

7. Relation of Erlang to existing languages.
"A History of Erlang”

8. COP: Concurrency Oriented
Programming
• Systems are built from processes.
• Process share nothing.
• Processes interact by asynchronous message
passing.
• Processes are isolated.

9. AXD301: Success
• Came out in 1998
• ATM switch, control-plane written in Erlang
• 2.6 Million lines of Erlang code
• Nine-nines of uptime: 99.99999999%
• Built highly reliable systems from unreliable
components
• Scale-out

10. Erlang
• Only accessible to the community since December
1998
• Designed for telephony control planes
• Actor Model
• Functional

11. Universal Server

12. Universal Server
1>
UniversalServer
=
1>
fun
US()
-­‐>
1>
receive
{become,
F}
-­‐>
1>
F()
1>
end
1>
end.
#Fun<erl_eval.44.90072148>

13. Named Funs (Lambda)
1>
UniversalServer
=
1>
fun
US()
-­‐>
1>
receive
{become,
F}
-­‐>
1>
F()
1>
end
1>
end.
#Fun<erl_eval.44.90072148>

14. Message Passing
1>
UniversalServer
=
1>
fun
US()
-­‐>
1>
receive
{become,
F}
-­‐>
1>
F()
1>
end
1>
end.
#Fun<erl_eval.44.90072148>

15. Lightweight Processes
2>
Server
=
spawn(UniversalServer).
<0.45.0>

16. Detour:
Calculating Factorials

17. Recurrence Relation

18. Calculating Factorials
5>
FC
=
fun
5>
Factorial(0)
-­‐>
1;
5>
Factorial(N)
-­‐>
N
*
Factorial(N-­‐1)
5>
end.
#Fun<erl_eval.30.90072148>

19. It Works!
8>
FC(10).
3628800

20. Turning It Into A Server
9>
FServer
=
9>
fun
FactorialServer()
-­‐>
9>
receive
9>
{From,
N}
-­‐>
9>
From
!
FC(N),
9>
FactorialServer()
9>
end
9>
end.
#Fun<erl_eval.44.90072148>

21. Converting the Server
10>
Server
!
{become,
FServer}.
{become,#Fun<erl_eval.44.90072148>}

22. Testing It Out
11>
Server
!
{self(),
10}.
{<0.42.0>,10}
12>
flush().
Shell
got
3628800

23. As A Module

24. As A Module
-­‐module(factorial_server).
-­‐export([start_link/0,
calc/2,
server/0]).
start_link()
-­‐>
spawn_link(?MODULE,
server,
[]).
calc(X,
Pid)
-­‐>
Pid
!
{self(),
X},
receive
N
-­‐>
N
end.
server()
-­‐>
receive
{From,
N}
-­‐>
From
!
factorial(N),
server()
end.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).

25. Module name
-­‐module(factorial_server).
-­‐export([start_link/0,
calc/2,
server/0]).
start_link()
-­‐>
spawn_link(?MODULE,
server,
[]).
calc(X,
Pid)
-­‐>
Pid
!
{self(),
X},
receive
N
-­‐>
N
end.
server()
-­‐>
receive
{From,
N}
-­‐>
From
!
factorial(N),
server()
end.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).

26. Exports
-­‐module(factorial_server).
-­‐export([start_link/0,
calc/2,
server/0]).
start_link()
-­‐>
spawn_link(?MODULE,
server,
[]).
calc(X,
Pid)
-­‐>
Pid
!
{self(),
X},
receive
N
-­‐>
N
end.
server()
-­‐>
receive
{From,
N}
-­‐>
From
!
factorial(N),
server()
end.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).

27. Start Module
-­‐module(factorial_server).
-­‐export([start_link/0,
calc/2,
server/0]).
start_link()
-­‐>
spawn_link(?MODULE,
server,
[]).
calc(X,
Pid)
-­‐>
Pid
!
{self(),
X},
receive
N
-­‐>
N
end.
server()
-­‐>
receive
{From,
N}
-­‐>
From
!
factorial(N),
server()
end.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).

28. Client Interface
-­‐module(factorial_server).
-­‐export([start_link/0,
calc/2,
server/0]).
start_link()
-­‐>
spawn_link(?MODULE,
server,
[]).
calc(X,
Pid)
-­‐>
Pid
!
{self(),
X},
receive
N
-­‐>
N
end.
server()
-­‐>
receive
{From,
N}
-­‐>
From
!
factorial(N),
server()
end.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).

29. Server Loop
-­‐module(factorial_server).
-­‐export([start_link/0,
calc/2,
server/0]).
start_link()
-­‐>
spawn_link(?MODULE,
server,
[]).
calc(X,
Pid)
-­‐>
Pid
!
{self(),
X},
receive
N
-­‐>
N
end.
server()
-­‐>
receive
{From,
N}
-­‐>
From
!
factorial(N),
server()
end.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).

30. Calculator
-­‐module(factorial_server).
-­‐export([start_link/0,
calc/2,
server/0]).
start_link()
-­‐>
spawn_link(?MODULE,
server,
[]).
calc(X,
Pid)
-­‐>
Pid
!
{self(),
X},
receive
N
-­‐>
N
end.
server()
-­‐>
receive
{From,
N}
-­‐>
From
!
factorial(N),
server()
end.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).

31. Using The Module
8>
Pid
=
factorial_server:start_link().
<0.72.0>
9>
factorial_server:calc(10,
Pid).
3628800

32. Error Handling

33. Introduce A Bug
factorial(0)
-­‐>
1;
factorial(10)
-­‐>
1=2;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).

34. Trigger The Bug
9>
factorial_server:calc(10,
Pid).
=ERROR
REPORT====
5-­‐Apr-­‐2015::06:04:40
===
Error
in
process
<0.59.0>
with
exit
value:
{{badmatch,2},
[{factorial_server,factorial,1,[{file,"factorial_server.erl"},{line,
15}]},{factorial_server,server,0,[{file,"factorial_server.erl"},{line,
11}]}]}
**
exception
exit:
{badmatch,2}
in
function
factorial_server:factorial/1
(factorial_server.erl,
line
15)
in
call
from
factorial_server:server/0
(factorial_server.erl,
line
11)

35. Ping The Server Again?
10>
10>
factorial_server:calc(10,
Pid).
...Hangs
Forever

36. OTP

37. Behaviours

38. Behaviours are formalizations of common
patterns. The idea is to divide the code for
a process in a generic part (a behaviour
module) and a specific part (a callback
module).

39. Gen_Server

40. A behaviour module for
implementing the server
of a client-server relation.

41. As An OTP Module
-­‐module(factorial_server_otp).
-­‐behaviour(gen_server).
-­‐export([start_link/0,
init/1,
handle_call/3,
terminate/2,
code_change/3]).
start_link()
-­‐>
gen_server:start_link({local,
?MODULE},
?MODULE,
[],
[]).
init([])
-­‐>
{ok,
state}.
handle_call(X,
_From,
state)
-­‐>
{reply,
factorial(X),
state}.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).
terminate(shutdown,
state)
-­‐>
ok.
code_change(_OldVsn,
state,
_Extra)
-­‐>
{ok,
state}.

42. Behaviour Definition
-­‐module(factorial_server_otp).
-­‐behaviour(gen_server).
-­‐export([start_link/0,
init/1,
handle_call/3,
terminate/2,
code_change/3]).
start_link()
-­‐>
gen_server:start_link({local,
?MODULE},
?MODULE,
[],
[]).
init([])
-­‐>
{ok,
state}.
handle_call(X,
_From,
state)
-­‐>
{reply,
factorial(X),
state}.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).
terminate(shutdown,
state)
-­‐>
ok.
code_change(_OldVsn,
state,
_Extra)
-­‐>
{ok,
state}.

43. Start Gen_Server
-­‐module(factorial_server_otp).
-­‐behaviour(gen_server).
-­‐export([start_link/0,
init/1,
handle_call/3,
terminate/2,
code_change/3]).
start_link()
-­‐>
gen_server:start_link({local,
?MODULE},
?MODULE,
[],
[]).
init([])
-­‐>
{ok,
state}.
handle_call(X,
_From,
state)
-­‐>
{reply,
factorial(X),
state}.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).
terminate(shutdown,
state)
-­‐>
ok.
code_change(_OldVsn,
state,
_Extra)
-­‐>
{ok,
state}.

44. Setup Server
-­‐module(factorial_server_otp).
-­‐behaviour(gen_server).
-­‐export([start_link/0,
init/1,
handle_call/3,
terminate/2,
code_change/3]).
start_link()
-­‐>
gen_server:start_link({local,
?MODULE},
?MODULE,
[],
[]).
init([])
-­‐>
{ok,
state}.
handle_call(X,
_From,
state)
-­‐>
{reply,
factorial(X),
state}.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).
terminate(shutdown,
state)
-­‐>
ok.
code_change(_OldVsn,
state,
_Extra)
-­‐>
{ok,
state}.

45. Server Reply Handler
-­‐module(factorial_server_otp).
-­‐behaviour(gen_server).
-­‐export([start_link/0,
init/1,
handle_call/3,
terminate/2,
code_change/3]).
start_link()
-­‐>
gen_server:start_link({local,
?MODULE},
?MODULE,
[],
[]).
init([])
-­‐>
{ok,
state}.
handle_call(X,
_From,
state)
-­‐>
{reply,
factorial(X),
state}.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).
terminate(shutdown,
state)
-­‐>
ok.
code_change(_OldVsn,
state,
_Extra)
-­‐>
{ok,
state}.

46. Internal Callbacks
-­‐module(factorial_server_otp).
-­‐behaviour(gen_server).
-­‐export([start_link/0,
init/1,
handle_call/3,
terminate/2,
code_change/3]).
start_link()
-­‐>
gen_server:start_link({local,
?MODULE},
?MODULE,
[],
[]).
init([])
-­‐>
{ok,
state}.
handle_call(X,
_From,
state)
-­‐>
{reply,
factorial(X),
state}.
factorial(0)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).
terminate(shutdown,
state)
-­‐>
ok.
code_change(_OldVsn,
state,
_Extra)
-­‐>
{ok,
state}.

47. Using The OTP Module
1>
factorial_server_otp:start_link().
{ok,<0.34.0>}
2>
gen_server:call(factorial_server_otp,
9).
362880

48. Triggering The Bug
5>
gen_server:call(factorial_server_otp,
10).
=ERROR
REPORT====
5-­‐Apr-­‐2015::06:03:43
===
**
Generic
server
factorial_server_otp
terminating
**
Last
message
in
was
10
**
When
Server
state
==
state
**
Reason
for
termination
==
…

49. Subsequent Calls
6>
gen_server:call(factorial_server_otp,
10).
**
exception
exit:
{noproc,{gen_server,call,
[factorial_server_otp,10]}}
in
function
gen_server:call/2
(gen_server.erl,
line
180)

50. Detour:
Links,
Supervisors,
&
Applications

51. Links

52. Begin

53. spawn_link()

54. Bidirectional Link Forms

55. Process Dies

56. Cohort is Terminated

57. Supervisors

58. A behaviour module for
implementing a supervisor, a
process which supervises other
processes called child processes.

59. Factorial Supervisor
-­‐module(factorial_server_sup).
-­‐behaviour(supervisor).
-­‐export([start_link/0]).
-­‐export([init/1]).
start_link()
-­‐>
supervisor:start_link({local,
factorial_server_sup},
?MODULE,
[]).
init(_Args)
-­‐>
{ok,
{{one_for_one,
1,
60},
[{factorial_server_otp,
{factorial_server_otp,
start_link,
[]},
permanent,
brutal_kill,
worker,
dynamic}]}}.

60. Child Specs
-­‐module(factorial_server_sup).
-­‐behaviour(supervisor).
-­‐export([start_link/0]).
-­‐export([init/1]).
start_link()
-­‐>
supervisor:start_link({local,
factorial_server_sup},
?MODULE,
[]).
init(_Args)
-­‐>
{ok,
{{one_for_one,
1,
60},
[{factorial_server_otp,
{factorial_server_otp,
start_link,
[]},
permanent,
brutal_kill,
worker,
dynamic}]}}.

61. Applications

62. In OTP, application denotes a component
implementing some specific functionality,
that can be started and stopped as a unit,
and which can be re-used in other systems
as well.

63. Application Resource File
{application,
factorial_server,
[{description,
"Factorial
Calculator"},
{vsn,
"1"},
{modules,
[factorial_server_otp,
factorial_server_app,
factorial_server_sup]},
{registered,
[]},
{applications,
[kernel,
stdlib,
sasl]},
{mod,
{factorial_server_app,[]}}
]}.

64. Dependencies
{application,
factorial_server,
[{description,
"Factorial
Calculator"},
{vsn,
"1"},
{modules,
[factorial_server_otp,
factorial_server_app,
factorial_server_sup]},
{registered,
[]},
{applications,
[kernel,
stdlib,
sasl]},
{mod,
{factorial_server_app,[]}}
]}.

65. Modules
{application,
factorial_server,
[{description,
"Factorial
Calculator"},
{vsn,
"1"},
{modules,
[factorial_server_otp,
factorial_server_app,
factorial_server_sup]},
{registered,
[]},
{applications,
[kernel,
stdlib,
sasl]},
{mod,
{factorial_server_app,[]}}
]}.

66. Callback Module
{application,
factorial_server,
[{description,
"Factorial
Calculator"},
{vsn,
"1"},
{modules,
[factorial_server_otp,
factorial_server_app,
factorial_server_sup]},
{registered,
[]},
{applications,
[kernel,
stdlib,
sasl]},
{mod,
{factorial_server_app,[]}}
]}.

67. Application Callback
Module
-­‐module(factorial_server_app).
-­‐behaviour(application).
-­‐export([start/2,
stop/1]).
start(_Type,
_Args)
-­‐>
factorial_server_sup:start_link().
stop(_State)
-­‐>
ok.

68. Using It
3>
application:ensure_all_started(factorial_server).
{ok,[]}

69. Observer

70. Observer, is a graphical tool for observing the
characteristics of erlang systems. Observer displays
system information, application supervisor trees, process
information, ets or mnesia tables and contains a frontend
for erlang tracing.

71. Observer: Factorial Server

72. Introspecting Program State

73. Review
• Factorial Server is now OTP Application
• gen_server
• supervisor
• application
• Factorial Server has a bug where it crashes at
numbers >10.
• Gen_server crashes handled by supervision tree

74. Try Again
8>
gen_server:call(factorial_server_otp,
10).
=ERROR
REPORT====
5-­‐Apr-­‐2015::07:45:00
===
**
Generic
server
factorial_server_otp
terminating
**
Last
message
in
was
10
**
When
Server
state
==
state
**
Reason
for
termination
==
...

75. New Behaviour
9>
gen_server:call(factorial_server_otp,
9).
362880

76. Live Code Patch
17>
c(factorial_server_otp).
{ok,factorial_server_otp}

77. Now
18>
gen_server:call(factorial_server_otp,
12).
479001600

78. Erlang/OTP
• OTP provides framework for simplifying actor
model
• Error handling semantics through crashing
• Provides model for DRY code

79. Distributed

80. “You cannot build a fault-
tolerant system if you
only have one computer”
A History of Erlang

81. EPMD

82. This daemon acts as a name
server on all hosts involved in
distributed Erlang computations.

83. Intra-System

84. Distributed Erlang

85. EPMD Acts as Broker

86. EPMD Responds

87. BEAM: Initial Contact

88. Bidirectional Communication

89. Inter-System

90. Inter-System

91. Initial Contact

92. Responds With Host List

93. BEAM: Initial Contact

94. Bidirectional Communication

95. In Practice

96. Node 1: Foo
3c075477e55e:talk
sdhillon
erl
-­‐sname
foo
Erlang
R16B03-­‐1
(erts-­‐5.10.4)
[source]
[64-­‐bit]
[smp:8:8]
[async-­‐threads:
10]
[hipe]
[kernel-­‐poll:false]
Eshell
V5.10.4
(abort
with
^G)
(foo@3c075477e55e)1>
application:ensure_all_started(factorial_server).
{ok,[sasl,factorial_server]}
(foo@3c075477e55e)2>

97. Node Name
3c075477e55e:talk
sdhillon
erl
-­‐sname
foo
Erlang
R16B03-­‐1
(erts-­‐5.10.4)
[source]
[64-­‐bit]
[smp:8:8]
[async-­‐threads:
10]
[hipe]
[kernel-­‐poll:false]
Eshell
V5.10.4
(abort
with
^G)
(foo@3c075477e55e)1>
application:ensure_all_started(factorial_server).
{ok,[sasl,factorial_server]}
(foo@3c075477e55e)2>

98. Node 1: Bar
3c075477e55e:talk
sdhillon
erl
-­‐sname
bar
Erlang
R16B03-­‐1
(erts-­‐5.10.4)
[source]
[64-­‐bit]
[smp:8:8]
[async-­‐threads:
10]
[hipe]
[kernel-­‐poll:false]
Eshell
V5.10.4
(abort
with
^G)
(bar@3c075477e55e)1>
net_adm:ping('foo@3c075477e55e').
pong
(bar@3c075477e55e)2>
gen_server:call({factorial_server_otp,
'foo@3c075477e55e'},
10).
3628800

99. Contact
3c075477e55e:talk
sdhillon
erl
-­‐sname
bar
Erlang
R16B03-­‐1
(erts-­‐5.10.4)
[source]
[64-­‐bit]
[smp:8:8]
[async-­‐threads:
10]
[hipe]
[kernel-­‐poll:false]
Eshell
V5.10.4
(abort
with
^G)
(bar@3c075477e55e)1>
net_adm:ping('foo@3c075477e55e').
pong
(bar@3c075477e55e)2>
gen_server:call({factorial_server_otp,
'foo@3c075477e55e'},
10).
3628800

100. Remote Execution
3c075477e55e:talk
sdhillon
erl
-­‐sname
bar
Erlang
R16B03-­‐1
(erts-­‐5.10.4)
[source]
[64-­‐bit]
[smp:8:8]
[async-­‐threads:
10]
[hipe]
[kernel-­‐poll:false]
Eshell
V5.10.4
(abort
with
^G)
(bar@3c075477e55e)1>
net_adm:ping('foo@3c075477e55e').
pong
(bar@3c075477e55e)2>
gen_server:call({factorial_server_otp,
'foo@3c075477e55e'},
10).
3628800

101. Monitoring Nodes
(bar@3c075477e55e)1>
net_adm:ping('foo@3c075477e55e').
pong
(bar@3c075477e55e)2>
net_kernel:monitor_nodes(true).
ok
(bar@3c075477e55e)3>
flush().
Shell
got
{nodedown,foo@3c075477e55e}
ok

102. Distributed Erlang
• Useful for building highly reliable applications
• Useful for building applications that scale out
• The underlying system of nearly all Riak requests
• Authentication through cookies

103. Dialyzer

104. Types

105. Erlang:
Dynamically Typed

106. Typespec

107. Typespec
-­‐spec
handle_call(non_neg_integer(),
{pid(),
any()},
state)
-­‐
>
{reply,
non_neg_integer(),
state}.
handle_call(X,
_From,
state)
-­‐>
{reply,
factorial(X),
state}.
-­‐spec
factorial(non_neg_integer())
-­‐>
non_neg_integer().
factorial(1)
-­‐>
-­‐1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).

108. Tying it Together
Checking whether the PLT out.plt is up-to-date... yes
Proceeding with analysis...
factorial_server_otp.erl:6: Invalid type specification for function
factorial_server_otp:handle_call/3. The success typing is
(pos_integer(),_,'state') -> {'reply',neg_integer(),'state'}
factorial_server_otp.erl:9: Invalid type specification for function
factorial_server_otp:factorial/1. The success typing is
(pos_integer()) -> neg_integer()
done in 0m0.79s
done (warnings were emitted)

109. Fixed
-­‐spec
handle_call(non_neg_integer(),
{pid(),
any()},
state)
-­‐
>
{reply,
non_neg_integer(),
state}.
handle_call(X,
_From,
state)
-­‐>
{reply,
factorial(X),
state}.
-­‐spec
factorial(non_neg_integer())
-­‐>
non_neg_integer().
factorial(1)
-­‐>
1;
factorial(N)
-­‐>
N
*
factorial(N-­‐1).

110. Tying it Together
3c075477e55e:talk sdhillon$ dialyzer --plt out.plt
factorial_server_otp.erl
Checking whether the PLT out.plt is up-to-date... yes
Proceeding with analysis... done in 0m0.89s
done (passed successfully)

111. Dialyzer
• Utilizes typespec
• Static, offline analysis
• Utilizes PLTs (Persistent Lookup Table)

112. Garbage Collection

113. Memory
• Process Heaps
• ETS Tables
• Atom Table
• Large Binary Space

114. Memory
• Process Heaps
• ETS Tables
• Atom Table
• Large Binary Space

115. Per-Process Heap

116. Only for items <64B

117. GC:
Copying Collector

118. Large Binary Space

119. GC:
Generational Collector

120. GC:
Synchronization Free

121. Benefit:
Fewer Unpredictable
Pauses

122. Memory Model
• Split memory model
• Useful for building high performance systems
• Predictability is key
• Garbage collected
• Shared heap only GC’d under memory pressure

123. Tying it back to Riak

124. Applications

125. Core Applications
• Riak_KV
• Riak_Core
• Riak_API

126. Riak_API

127. This OTP application encapsulates
services for presenting Riak's
public-facing interfaces.

128. ~1.5KLoC

129. 2 Supervisors

130. 5 Servers

131. Riak_Core

132. Riak Core is the distributed systems framework
that forms the basis of how Riak distributes
data and scales. More generally, it can be
thought of as a toolkit for building distributed,
scalable, fault-tolerant applications.

133. ~25KLoC

134. 11 Supervisors

135. 30 Servers

136. Riak_KV

137. ~30KLoC

138. 9 Supervisors

139. 12 Servers

140. Ancillary Applications
• Lager: Logging
• Storage Layer:
• Bitcask
• Leveldb
• Mochiweb: HTTP Interface
• Riak_pipe: Distributed map-reduce
• Riak_Ensemble: Consensus layer

141. It Works

142. Demos

143. Troubles In Paradise
• Distributed Erlang can be problematic due to head-
of-line blocking
• NIFs (external code) can be problematic due to
performance and debugging
• BEAM needs a JIT compiler
• Work being done here!

144. Erlang/OTP & Riak
@Sargun