93. Image Credits
Complexity (pipes): http://www.flickr.com/photos/bhophoto/384574407/
Simple Pipeline: http://tierracos.com/our-companies/tierra-pipeline/
Highway: http://farnorthdallas.advocatemag.com/wp-content/uploads/2013/04/882349_500402383328407_539732406_o.jpg
Homer & doughnuts: http://thechurchillreview.blogspot.com/2012/10/feeling-terror-too-young-aka-kiddie.html
Tom Brady: http://stamperdad.wordpress.com/category/tom-brady/
Telephone Exchange Operator: http://fineartamerica.com/featured/telephone-exchange-1920s-granger.html
People in Queue: http://www.guardian.co.uk/money/2010/mar/23/dole-queue-jobseekers-online
Cookie Monster: http://muppet.wikia.com/wiki/Cookie_Monster_Through_the_Years
Scrooge McDuck: http://smallbusiness.uprinting.com/how-pennies-are-hurting-small-business/
Too Many Cooks: http://www.ecommercesystems.com/featured-articles/cooks-kitchen-driving-ecommerce-business/
Clustered Macs: http://www.uiowa.edu/mihpclab/micg.html
Resuscitate:
http://www.dailymail.co.uk/health/article-2034160/Do-resuscitate-Theyre-fateful-words-meaning-doctors-wont-try-save-you-collapse-hospital.h
Iron Man: http://images.wikia.com/marvelmovies
Mail Sorting: http://riversidechamber.files.wordpress.com
Twitter Logo & app chart: https://blog.twitter.com/engineering
Modern Server Farm: http://bookriot.com/2013/03/26/book-less-libraries-and-other-contemporary-realities/
Grey World: http://upload.wikimedia.org/wikipedia/commons/d/d0/BlankMap-World-1ce.png
Notes de l'éditeur
-I won’t try to make anyone guess what this is
-I should explain it a bit, as I’ll be following a similar format for other diagrams.
-Blue symbol represents some feature - browsing, managing, payments, cart
-black box is a single web node or application
-cloud is internet
Let’s talk about complexity and scalability
-A few questions, before we begin.
-I’d like for you to keep the answers to these in your head as we go along
-or rather, is everything in trunk / master?
-Have you ever had to find the needle in that giant haystack?
-do you miss TDD?
whether it be a total overhaul or just a few features.
And if so, did it take you longer than coding that new feature?
These are all symptoms of a Monolithic application, If you answered “YES”, then you probably have a Monolithic application
“convienent”
The Complexity will become enormous. Any gains you think you may have made at the outset will not continue, particularly once you start to attract users.
..And grow your team.
- A monothlic app will not scale!
You may say to yourself, “Wait, Steve!” what do you mean it won’t scale?
I can load balance my nodes...
Add Swimlanes, master-slave dbs in case that doesn’t work
-or add caching in between my nodes
Can you really look at a class or schema diagram like this and really think, “Yeah, that’s awesome. Nailed it.”?
-this was the largest I could find on Google images. I’m sure we’ve all seen the massive versions of this that cover multiple pages, taped to the wall.
- If you ever have to tape to the wall, done something wrong
The ability to add new features, fix bugs, or resolve technical debt cannot scale with the size of the application. Throwing more developers at it also will not help
- One of the best passive-aggressive statements I’ve seen
- Passive aggressively scrawled on the wall: You can’t make a baby in 1 month with 9 women.
-And trying to refactor anything?
As the size of your codebase increases, the computational complexity will exponentially increase and will have adverse effects on maintenance. This has been known for some time; there’s been papers written on the subject. Although it’s difficult to measure this.
This is such a misguided architecture, it makes you wonder how exactly we collectively ended up here.
I blame it largely at the feet of the large, popular web frameworks, e.g. Rails, Django, Grails.
Don’t get me wrong, they have their uses
But they’re touted as if they’re the magic cure for building your app or product; that your company should be based entirely within a framework. You hear folks say, “Oh, what are you using in your startup?” “We’re a Rails shop, obviously!”.
I argue that your framework choice is largely irrelevant. When people ask “what technology are you using?” the answer should be something like “Ugh, well, that’s difficult”.
There have been attempts to mitigate
-Let’s take our anecdotal design, an e-commerce store.
With SOA, one would break up each individual component, and setup communication between each service over some direct service calls. Eg SOAP or REST.
-Note that these calls are traditionally Synchronous over http, so it’s imperative that each node return as quickly as possible (that whole ‘responsive’ notion from the Manifesto)
Having disparate services keeps the individual components easier to manage and keep track of, which reduces overall complexity dramatically
… as each feature set will be in its own simpler repository
With distributed applications, each node becomes its own concentrated ecosystem. New features can be added with relatively little pain.
And the code is easier to maintain. I highlight this point for a reason. Bugs are vastly easier to discover. Refactoring is highly concentrated and (shouldn’t) disrupt the work of the other nodes.
and other numbers that managers love
-organizing the event based communication and see things flow and respond, it’s magical
possibly delete
However! (There’s always a downside, eh?)
You’ll need someone to oversee the team and catch those engineers which are straying from the distributed vision
-find out how each will communicate
- converting to a distributed, services oriented app is an investment: a decent amount of upfront cost, but the dividends are fantastic.
HTTP calls are relatively slow
* Having a web of interconnected services can also grow out of control. Having to configure service nodes so that each knows where the others are located in the network can be cumbersome to manage.
In other words, SOA (along with the other architecture patterns that have attempted to fix monolithic) do a decent job...
... we can do better
So what does this have to do with the Reactive pattern?
being event driven increases decoupling of components
-Couldn’t think of proper images to describe these
-Adhering to these 4 tenants is all well and good
Furthermore: being Reactive is a development ethos and there’s not one right answer
So, going back to the original e-commerce example…
- how to go reactive?
First of course, is to break up our application into individual components, but this is not good enough.
Could still adhere to be resilient and responsive, and it’s certainly more scalable than it was.
Simply breaking them apart won’t make each service node resilient or responsive, that’s up to internal dev architecture (more from Dave)
In other words, extend the idea of having an event driven node or service into the notion of having event-driven communication between services.
-And how do we facilitate event based communication between our services?
One method: Message Brokers!
Your basic approach. A single Producer, a Queue of Messages, and a Consumer. In this case, quite the prodigious one.
Note the queue backup. He should probably share.
The key to a message oriented approach is when certain events occur to encapsulate data into individual messages and drop them on some queue or exchange
TODO: Image of Postman / message.
Can spin up or down nodes for targeted scaling of your services as demand dictates
-Example from before, an e-commerce site
-we’ve broken the features into individual applications and spun up one node per each
-connected the users’ cart to a queue, and our order processing into another queue
-suppose the cart and order processing start experiencing heavy load
- programatically spin up new instances to handle it
- Going back into the refactoring discussion -> Can pit different implementations of a service against each other via AB testing
TIME: Asynchronous
LOCATION: Nodes do not care where the other nodes are, only worry about the broker
CARDINALITY: Each service instance does not care about the number of others out there
There are quite a few to choose from. I point out 0mq, activemq, open amq, and my favorite…
Why?
uses the raw AMQP protocol (as do some of the others I mentioned)
the actual broker itself is a lightweight Erlang application
Offers two key security features: message persistance and error recovery
And it’s fast.
Although not nearly as fast as some other Brokers, specifically zeroMQ
If you’re struggling to process 4k+ messages/ second, that sounds like an awesome engineering problem and you contact us.
The previous description of message queues glosses over what is happening a bit.
-Messages are routed through an EXCHANGE in to one or many queues.
-Messages are then plucked off the queue by an attached, waiting Consumer. Or, the first available if multiple are attached.
-Exchanges can have different delivery techniques.
If a message is mishandled or the consumer crashes / has an error during message processing, message is returned to the queue for another consumer to pick it up or try again
Furthermore, Queues and exchanges can be made “durable”, meaning they’ll survive a restart or server crash with messages intact
RabbitMQ nodes can be clustered
-federated-> shared / copied
-shoveled -> pushed
There are several message delivery strategies one can set up for their exchanges
With the whole purpose of scaring you, here’s the Topic Queue, the most complicated
-gets complicated
-allows for all sorts of wild routing schemes
-all messages go to the BACK of the queue
-each queue could equate to namespaced events, like “order.fail” or “order.success”. A service could emit an ‘order.fail’ event for subscriber queues to receive
- A simpler concept is the direct exchange, where queues register on an exact String key, and messages state which key they should be ‘addressed’ to
I feel these are the easiest to use and the best way to get started, but consider the topic exchanges for true power
Here, too, a practical usage could be equate a queue key registration as an event name.
This is a depiction of a Work Queue, the most common configuration.
-An exchange routes messages via some method (topic, direct) to a queue with multiple consumers
-Work queues operate on a first-come-first serve approach
-I think this is the coolest, one can monitor the state of the exchange and programmatically spin up or down the number of consumer nodes
-All of these consumers are peers...
-Finally, there’s ‘fanout’, which distributes a single message to all attached queues. Good for a general broadcast notification / pub-sub to all your nodes.
-there needs to be a one-to-one queue to consumer relation, though, I believe.
-A special use, case, really.
- should say type=fanout under that X
…Twitter
They had one of the largest Rails deployments in the world.
Epitome of Monolithic.
Engineers needed to pull in Global Experts to get anything done on the codebase
Spent more time on “Archeological Digs” or “Whale Hunting expeditions” to track down bugs or learn how things worked than on developing new features
Turning point came during 2010 World Cup. Flurry of tweets brought twitter to its knees.
Service was unavailable, engineers worked through the night
Debated putting a vuvuzella sound effect here.
They shattered their monolithic application into multiple individual applications..
The size of the circles represent the number of deployments of that particular application
-I know I just spoke about Rabbit, but Twitter developed their own tool called Finagle which handles asynchronous communication between nodes
Each with their own small, determined, dedicated teams.
they switched from the Ruby Virtual Machine to the JVM, were no strangers to the JVM
-Search written in Java
Embraced Event-based programming from the bottom up
Went with Netty (as opposed to something like NodeJS), which is an NIO technology for the JVM
Netty is NIO
When Twitter was Monolithic, they were able to achieve about 200 to 300 requests or tweets per second per application node.
After going reactive, they’re up to 10 to 20 thousand. 2 orders or magnitude?
So, as of August 2013, Twitter handles an average of 5700 Tweets per minutes, and under load, can dynamically ramp that number up to 144k without any direct action by staff
-> Incidentally, JVM advocates and Rails detractors love to talk about how Twitter moved away from Rails to Java. And while that’s true that Java would give better performance, the Architecture choice was vastly more important
We didn’t want to go down that same old route.
-Second thing we did was take a step back and look at their current setup.