In these days of web scale and microservices, gracefully handling faulty or crashing backend web services becomes increasingly important. As long as your application uses just a handful of such backends, there’s no problem. But what if you have hundreds of them? How will you prevent your application from the negative effects of inevitable incidental crashes or timeouts when contacting other web services?
In this talk, we’ll take a real-world example from a web scale system. This system is targeted at 10 million users and has around 400 heterogeneous backends. Akka and Scala power a system that is capable of running smoothly even when some backends might not be so stable.
1. HOW TO STAYHOW TO STAY
RESPONSIVE WITH 400RESPONSIVE WITH 400
BACKENDSBACKENDS
@mthmulders
2.
3.
4.
5. So, we need something that is...
Mobile First
Scalable
Secure
... but ...
doesn't store insurance data
retrieves that in real-time
from ~ 400 different companies!
9. All problems in computer science can be
solved by another abstraction layer.
-- not David Wheeler (1927 - 2004)
10. JUST MOVE THE PROBLEM AWAYJUST MOVE THE PROBLEM AWAY
Main Systems
Routing Engine
Insurer 01 Insurer 02 Insurer 03 .... Insurer nn
11. FIRE UP YOUR IDE & START CODINGFIRE UP YOUR IDE & START CODING
Standardized interface for retrieving insurance data.
One endpoint address for each insurance company.
Retrieve data using link code that includes insurer ID.
Shouldn't be too hard!
15. NOT RELATED, BUTNOT RELATED, BUT
ANALYSISANALYSIS
Blocking network calls
No time-outs
The whole JVM is waiting!
If an insurer is down, it will still be contacted.
This will slow down the system even further.
18. MESSAGE DRIVENMESSAGE DRIVEN
Reactive Systems rely on asynchronous
message-passing to establish a boundary
between components that ensures loose
coupling, isolation and location
transparency.
19. ELASTICELASTIC
The system stays responsive under varying
workload. Reactive Systems can react to
changes in the input rate by increasing or
decreasing the resources allocated to
service these inputs.
20. RESILIENTRESILIENT
The system stays responsive in the face of
failure. Failures are contained within each
component, [...] thereby ensuring that
parts of the system can fail and recover
without compromising the system as a
whole.
21. RESPONSIVERESPONSIVE
The system responds in a timely manner if
at all possible. Responsiveness is the
cornerstone of usability and utility [...].
Responsive systems focus on providing
rapid and consistent response times [...].
31. INITIATE RETRIEVAL WITH TIME-OUTINITIATE RETRIEVAL WITH TIME-OUT
val actor = context.actorOf(
ConsultInsurersActor.props(), s"consult-$berrfnr" )
actor ! RetrieveAllPolicyData(linkingCode)
context.system.scheduler.scheduleOnce(2 seconds) {
self ! RetrievalTimeout(berrfnr, linkingCodes)
}
32. CONSULT ONE INSURERCONSULT ONE INSURER
// Send the request
val request = HttpRequest(POST, Uri(url), Nil, entity)
http ? request
// Receive the response
response.entity.as[ResponseData] match {
case Left(failure) => client ! Failure(failure)
case Right(data) => client ! PolicyDataRetrieved(data)
}
33. CONSULT INSURER (WITH CIRCUIT BREAKER)CONSULT INSURER (WITH CIRCUIT BREAKER)
val cb = new CircuitBreaker(system.scheduler,
maxFailures = 5,
callTimeout = 5 seconds,
resetTimeout = 1 minute) // or obtain from config
// Send the request
val request = HttpRequest(POST, Uri(url), Nil, entity)
cb.withCircuitBreaker(http ? request) pipeTo self
34.
35.
36.
37.
38. KEY TAKEAWAYSKEY TAKEAWAYS
Actor model enables building resilient solutions
Suitable for large portions of concurrent work
Think outside the box
Log unexpected messages
Use a whiteboard
39. THANKS!THANKS!
QUESTIONS?QUESTIONS?
IMAGESIMAGES
Insurance Policy by .
"Success Kid" a.k.a Sammy Griner by .
Blendtec founder/owner Tom Dickson by .
Reactive Manifesto by .
Actor Model by .
Pictures of Money (Flickr)
Laney Griner (Flickr)
Tim Provost
Wiljag Denekamp
Wiljag Denekamp
