1.
Asynchronous Service Servers<br />Based on AMQP for Ruby<br />

2.
Versapay<br />Point of sales<br />eCommerce payment processing<br />Partner with JPChase<br />No.1 Growth Company in Canada (Profit50)<br />

3.
Prelude: ConcurrencyOr, ouch, it hurts…<br />

4.
Nietzsche on Concurrency<br />…if you gaze for long into an abyss, the abyss gazes also into you.<br />

5.
Spectrum of Grainedness<br />Parallelism<br />The Multicore Future™<br />Code structuring (e.g. block/wait)<br />Concurrency<br />Simultaneous tasks<br />Distributed Services<br />Internet/REST<br />REST (all its constraints: stateless, layered, cacheable, uniform interace, …)<br />

6.
Lazy Scalability<br />Worry Later?<br />Very hard to fix a wrong architecture<br />Get it right, right now!<br />Architecture (constraints) makes scaling possible<br />Actual scaling when you need it<br />Path from concurrency to SOA<br />Actor Model (e.g. Erlang)<br />Key-value store (no join)<br />

7.
Actor Model<br /> The Actor model is characterized by inherent concurrency of computation within and among actors, dynamic creation of actors, inclusion of actor addresses in messages, and interaction only through direct asynchronous message passing with no restriction on message arrival order.<br />N.B. From wikipedia<br />

8.
Actor Model<br />Identified by address<br />Mobile (doesn’t matter which machine)<br />Transient (doesn’t matter which process)<br />Decouples the “who” of computation<br />Message passing<br />Ask, don’t tell<br />Decouples the “what” of computation<br />Asynchrony<br />Event-based<br />Decouples the “when” of computation<br />

9.
Erlang-Style Concurrency<br />Creating and destroying processes is very fast.<br />Sending messages between processes is very fast.<br />Processess behave the same way on all operating systems.<br />We can have very large numbers of processes.<br />Processes share no memory and are completely independent.<br />The only way for processes to interact is through message passing.<br />Joe Armstrong<br />

10.
More on Erlang<br />Message passing is asynchronous.<br />Processes can monitor each other.<br />It is possible to selectively receive messages.<br />Remote processes appear largely the same as local processes.<br />Ulf Wiger<br />

11.
ASS kicking with RabbitMQ<br />Stealing Ideas from Erlang<br />

12.
AMQP<br />Just a protocol specification<br />Open, interoperable<br />Binary<br />Extensible<br />Dynamic creation of entities<br />For enterprise message middleware<br />RabbitMQ is a free implementation<br />Qpid is an Apache project<br />

13.
AMQP(&lt;1.0) Details<br />AMQP Entities<br />Exchange (publish messages here)<br />Queue (route messages here)<br />Binding (establishes routing between exchange and queue)<br />AMQP Error Model<br />The sound of success is silence<br />An error just closes connection<br />QOS, prefetch window<br />

14.
ASS<br />Inspired by Erlanggen_server<br />Hide concurrency behind generic server<br />Parameterize generic server with modules<br />Familiar sequential programming<br />Pass around Ruby objects by marshalling<br />Switch to JSON?<br />Full access to AMQP if you need it<br />

15.
A Simple ASS Actor<br />require &apos;rubygems&apos;<br />require &apos;ass&apos;<br />ASS.start do<br />ASS.actor(&quot;demo&quot;) {<br /> def echo(o)<br /> o<br /> end<br /> }<br />end<br />

16.
Demo<br />Server/Client<br />Load distribution<br />Store and forward<br />Zero downtime<br />Future & RPC<br />Wait in batches<br />Failure recovery with :ack =&gt; true<br />

17.
RPC is an Illusion<br />ASS is fundamentally asynchronous<br />Async/event-based programming is unfamiliar<br />RPC client does blocking wait to simulate sequential programming<br />“Future” gives some asynchrony back<br />Timeout as generic exception<br />Usually just give up and die<br />

18.
RPC is EVIL when…<br />Hides too much<br />Random failures<br />Doesn’t hide enough<br />Exposing the implementation<br />Leaky abstraction<br />See:<br />http://steve.vinoski.net/blog/2008/07/01/convenience-over-correctness/<br />

19.
Experiences So Far<br />

20.
Programming Murphy’s Computer<br />Anything can fail at any time…<br />Think in critical sections<br />Retry is the bedrock of reliability<br />:ack =&gt; true<br />FSM is a good mental aid<br />Mark progress with a database (ACID state)<br />It’s ok to die… process is transient, data is forever<br />

21.
API<br />Message verification<br />Pattern Matching<br />http://github.com/hayeah/matchmaker<br />Fail Fast and Hard at the API<br />Avoid error spreading through the services<br />Share the API spec among services<br />Weak API<br />

22.
Processes, Not Threads<br />Avoid threads, avoid horrors<br />The stack is opaque, you don’t know if anything you use might cause blocking<br />Let OS and DB do the hard work<br />Process isolation<br />Anything wrong? Easy, kill it!<br />IPC cost, but you don’t care (most of the time)<br />

23.
Testing Concurrent App<br />No established practice…<br />Test the real thing, not mocks<br />Pretty easy with ASS, hard with web-services<br />Test in a single process, deploy to multiple processes, then hosts, then networks<br />Invariants are super important<br />Some kind of property testing<br />Must test with enough load<br />

24.
Future Work<br />Like, TODO, right now!<br />

25.
System Health<br />Monitoring<br />Event based<br />Metrics<br />Logging<br />Self-management<br />Supervisor hierarchy<br />Zookeeper?<br />

26.
Better Testing<br />Stress Test<br />Testing with generated message sequences<br />Some kind of proxy to force message order<br />Some kind of code injection technique<br />System invariant checker<br />Test failures (connection dying, db dying, etc)<br />Continuous integration<br />

27.
What Datastore?<br />We are using ActiveRecord at the moment<br />Because it’s there…<br />Some kind of schemaless key-value store<br />Like Friendsfeed<br />

28.
Conclusion<br />

29.
Three Legged Stool<br />Scalability gives,<br />Distributability gives,<br />Reliability gives scalability<br />http://github.com/hayeah/ASS<br />ZERO documentation<br />Quirks<br />It’s not Rails<br />