Two years ago Rackspace had a problem: how do we backup 20K network devices, in 8 datacenters, across 3 continents, with less than a 1% failure rate -- every single day? Many solutions were tried and found wanting: a pure Perl solution, a vendor solution and then one in Ruby, none worked well enough. They not fast enough or they were not reliable enough, or they were not transparent enough when things went wrong. Now we all love Ruby but good Rubyists know that it is not always the best tool for the job. After re-examining the problem we decided to rewrite the application in a mixture of Erlang and Ruby. By exploiting the strengths of both -- Erlang's astonishing support for parallelism and Ruby's strengths in web development -- the problem was solved. In this talk we'll get down and dirty with the details: the problems we faced and how we solved them. We'll cover the application architecture, how Ruby and Erlang work together, and the Erlang approach to asynchronous operations (hint: it does not involve callbacks). So come on by and find out how you can get these two great languages to work together.

1. Polyglot Parallelism
A Case Study in Using Erlang
and Ruby at Rackspace

2. The Problem
Part 1

3. 20,000
network devices

4. 9 Datacenters,
3 Continents

5. devices not designed for
high-throughput
management

6. we need a high
throughput solution

7. the time spent in I/O
is the primary
bottleneck

8. if you want to speed
things up you have to
talk to more devices
in parallel

9. The Problem
Part II

10. huge blobs of data

11. lots of backups
equals big database

12. ad-hoc searching
is difficult but
important

13. customer SLA means
need to restore from
backup quickly

14. an event must be
generated for each
device interaction

15. migrations are
problematic with
that much data

16. rigid schema made
adapting to new
devices difficult

17. each device type has
different properties

18. “backup” means
different things for
each device type

19. need to grow with
the business

20. Previous
Solution

21. multiple Ruby apps

22. difficult to scale

23. vendor device
managers

24. New Solution

26. the simplest thing
that could possibly
work

27. most db writes
come from
scheduled jobs

28. Other Clients
Rails ReST API
Erlang
MongoDB Network
Devices

30. Joe Armstrong

31. AXD301
ATM Switch

32. 99.9999999%

33. Functional

34. Dynamically Typed

35. Single Assignment

36. A = 1. %=> 1
A = 2. %=> badmatch

37. [B, 2, C] = [1, 2, 3].
B = 1. %=> 1
C = 3. %=> 3

38. Immutable
Data
Structures

39. D = dict:new().
D1 = dict:store(foo, 1, D).
D2 = dict:store(bar, 2, D1).

40. Concurrency
Oriented

41. -module(fact).
-export([fac/1]).
fac(0) -> 1;
fac(N) -> N * fac(N-1).

42. -module(quicksort).
-export([quicksort/1]).
quicksort([]) -> [];
quicksort([Pivot|Rest]) ->
quicksort([Front || Front <- Rest, Front < Pivot])
++ [Pivot] ++
quicksort([Back || Back <- Rest, Back >= Pivot]).

43. Details

44. jobs framework

45. Runner
Callback
Workers Module

46. Callback
Runner Worker Module
start
ready
item
process
process
ready
.
.
.
stop

47. “behaviour” is interface

48. behaviour_info(callbacks) -> [
{init, 1},
{process_item, 3},
{worker_died, 5},
{job_stopping, 1},
{job_complete, 2}
].

49. running({worker_ready, WorkerPid, ok}, S) ->
case queue:out(S#state.items) of
{empty, I2} ->
stop_worker(WorkerPid, S),
{next_state, complete, S#state{items = I2}};
{{value, Item}, I2} ->
job_worker:process(WorkerPid, Item, now(),
S#state.job_state),
{next_state, running, S#state{items = I2}}
end;

50. handle_info({'DOWN', _, process, WorkerPid, Info},
StateName, S) ->
{Item, StartTime} = clear_worker(WorkerPid, S),
Callback = S#state.callback,
spawn(Callback, worker_died,
[Item, WorkerPid, StartTime,
Info, S#state.job_state]),
%% Start a replacement worker
start_workers(1, Callback),
{next_state, StateName, S};

51. handle_cast({process, Item, StartTime, JS}, S) ->
Callback = S#state.callback,
Continue =
try
Callback:process_item(Item, StartTime, JS)
catch
throw: Error ->
error_logger:error_report(Error),
ok
end,
job_runner:worker_ready(S#state.runner, self(),
Continue),
{noreply, S}.

52. story time

53. ReSTful API
with Webmachine
The Convention Over Configuration Webserver

54. http://webmachine.basho.com
HTTP Request Lifecycle Diagram

55. If you know HTTP
Webmachine Is Simple
As Proven by the “Number of Types of Things”
Measurement of Complexity

56. The 3 Most Important Types of
Things In Webmachine
1. Dispatch Rules (pure data--barely a thing!)
2. Resources (composed of simple functions!)
3. Requests (simple get/set interface!)

57. Dispatch Rules
{ ["devices", server], device_resource, [] }
GET /devices/12345
Webmachine inspects the device_resource module for
defined callbacks, and sets the Request record’s “server”
value to 12345.

58. Resources
• POEM (Plain Old Erlang Module)
• Composed of referentially transparent functions*
• Functions are callbacks into the request lifecycle
• Approximately 30 possible callback functions, e.g.:
• resource_exists → 404 Not Found
• is_authorized → 401 Not Authorized
* mostly

59. Resource Functions
Perma-404
resource_exists(Request, Context) ->
{false, Request, Context}.
Lucky Auth
is_authorized(Request, Context) ->
S = calendar:time_to_seconds(now()),
case S rem 2 of
0 -> {true, Request, Context};
1 -> {“Basic realm=lucky”, Request, Context}
end.

60. Requests
• The first argument to each resource function
• Set and read request & response data
RemoteIP = wrq:peer(Request).
wrq:set_resp_header(“X-Answer”, “42”, Request).

61. Retrieving a JSON Firewall Representation
content_types_provided(Request, Context) ->
Types = [{"application/json", to_json}],
{Types, Request, Context}.
to_json(Request, Context) ->
Device = proplists:get_value(device, Context),
UserId = get_user_id(Request),
case fe_api_firewall:get_config(Device, UserId) of
{ok, Config} ->
success_response(Config, Request, Context);
{error, Reason} ->
error_response(502, Reason, Request, Context)
end.

62. Gotchas

63. primitive obsession

64. string-ish
“hi how are you”
<<“hello there”>>
[<<"easy as ">>, [$a, $b, $c], "
☺n"].

65. hashes vs records

66. to loop is human,
to recur divine

67. Erlang conditionals
always return a value

68. design for testability

69. don’t
spawn,
use OTP

70. Downsides

71. Erlang changes very
slowly

72. 3rd party libraries

73. standard library
can be inconsistent

74. package
management

75. Questions

76. http://spkr8.com/t/7806
Phil: @philtoland
http://github.com/toland
http://philtoland.com
Mike: @lifeinzembla
http://github.com/msassak