As presented at ArchConf 2016 in San Diego, CA.

1. REACTIVE FAULT TOLERANT PROGRAMMING
with Hystrix and RxJava
Matt Stine (@mstine)

2. Matt StineSenior Product Manager - Pivotal Software, Inc.
Author of:
http://bit.ly/cloud-native-book

3. © Copyright 2015 Pivotal. All rights reserved. 3
MICROSERVICES!!!!!

4. Typical Microservices Architecture
ÂAPI
ÂAPI
!
!
!
µS
µS
µS
µS
µS

5. Trace a Request
ÂAPI
ÂAPI
!
!
!
µS
µS
µS
µS
µS

6. Consider this subgraph…
ÂAPI
µS
µS

7. public ResponseEntity<Foo> handleIncomingRequest() {
Foo foo = new Foo();
foo.addPart(serviceOne.getContributionToFoo());
foo.addPart(serviceTwo.getContributionToFoo());
return new ResponseEntity<>(foo, HttpStatus.OK);
}
Block and wait!

8. Meanwhile in Service Two…
!
!
µS µS
µS

9. public ResponseEntity<FooPart> handleIncomingRequest() {
FooPart fooPart = new FooPart();
fooPart.addSubPart(serviceThree.getContributionToFooPart());
fooPart.addSubPart(serviceFour.getContributionToFooPart());
return new ResponseEntity<>(fooPart, HttpStatus.OK);
}
Block and wait!

10. BUT
DAT
LATENCY
THO

11. Futures!
ExecutorService executor = createExecutorService();
public ResponseEntity<Foo> handleIncomingRequest() {
Foo foo = new Foo();
Future<FooPart> partOne = executor.submit(new CallToServiceOne());
Future<FooPart> partTwo = executor.submit(new CallToServiceTwo());
foo.addPart(partOne.get());
foo.addPart(partTwo.get());
return new ResponseEntity<>(foo, HttpStatus.OK);
}

12. The service graph changes…
public ResponseEntity<Foo> handleIncomingRequest() {
Foo foo = new Foo();
Future<FooPart> partOne = executor.submit(new CallToServiceOne());
Future<FooPart> partTwo = executor.submit(
new CallToServiceTwo(partOne.get()));
Future<FooPart> partThree = executor.submit(new CallToServiceThree())
foo.addPart(partTwo.get());
foo.addPart(partThree.get());
return new ResponseEntity<>(foo, HttpStatus.OK);
}
Block and wait!
Blocked until
two completes!

13. CompletableFutures
public ResponseEntity<Foo> handleIncomingRequest() {
Foo foo = new Foo();
CompletableFuture<FooPart> partTwo = CompletableFuture.supplyAsync(
new CallToServiceOne())
.thenApplyAsync(new CallToServiceTwo());
CompletableFuture<FooPart> partThree = CompletableFuture.supplyAsync(
new CallToServiceThree());
foo.addPart(partTwo.get());
foo.addPart(partThree.get());
return new ResponseEntity<>(foo, HttpStatus.OK);
}

14. As composition becomes more complex,
CompletableFuture API is lacking…

15. An API like this would be nice…
List<Integer> transactionsIds =
transactions.stream()
.filter(t -> t.getType() == Transaction.GROCERY)
.sorted(comparing(Transaction::getValue).reversed())
.map(Transaction::getId)
.collect(toList());

16. Hold that thought…

18. RESPONSIVE

19. RESILIENT

20. ELASTIC

21. MESSAGE-DRIVEN

23. RxJava
http://reactivex.io
https://github.com/ReactiveX/RxJava

24. Observer Pattern

25. Hello Observable
Observable<Integer> observableString = Observable.create(
new Observable.OnSubscribe<Integer>() {
public void call(Subscriber<? super Integer> subscriber) {
for (int i = 0; i < 5; i++) {
subscriber.onNext(i);
}
subscriber.onCompleted();
}
});
Emit an item!
I’m done emitting items!

26. Observable<Integer> observableString = Observable.create(
subscriber -> {
for (int i = 0; i < 5; i++) {
subscriber.onNext(i);
}
subscriber.onCompleted();
});
Hello Observable (JDK 8)

27. Hello Subscription
Subscription subscription = observableString.subscribe(
new Observer<Integer>() {
public void onCompleted() {
System.out.println("Observable completed");
}
public void onError(Throwable throwable) {
System.out.println("Oh noes! Something wrong happened!");
}
public void onNext(Integer integer) {
System.out.println("Item is " + integer);
}
});

28. Hello Subscription (JDK 8)
Subscription subscription = observableString.subscribe(
item -> System.out.println("Lambda says: " + item),
throwable -> System.out.println("Lambda says: Oh noes! Something wrong happened!"),
() -> System.out.println("Lambda says: Observable completed"));

29. All together now…
Observable.create(
subscriber -> {
for (int i = 0; i < 5; i++) {
subscriber.onNext(i);
}
subscriber.onCompleted();
}).subscribe(
item -> System.out.println("Lambda says: " + item),
throwable -> System.out.println("Lambda says: Oh noes! Something wrong happened!"),
() -> System.out.println("Lambda says: Observable completed"));

30. Back to Our Service Graph
ÂAPI
ÂAPI
!
!
!
µS
µS
µS
µS
µS

31. With Callables
Observable.fromCallable(new CallToServiceOne())
.flatMap(serviceOneResult -> Observable.fromCallable(
new CallToServiceTwo(serviceOneResult)))
.zipWith(Observable.fromCallable(
new CallToServiceThree()),
(FooPart resultFromServiceTwo, FooPart resultFromServiceThree) -> {
Foo foo = new Foo();
foo.addPart(resultFromServiceTwo);
foo.addPart(resultFromServiceThree);
return foo;
}).subscribe(System.out::println);

32. With Lambdas
Observable.<FooPart>create(serviceOneSubscriber -> {
serviceOneSubscriber.onNext(new FooPart("one"));
serviceOneSubscriber.onCompleted();
}).flatMap(serviceOneResult -> Observable.<FooPart>create(serviceTwoSubscriber -> {
serviceTwoSubscriber.onNext(new FooPart(serviceOneResult + " + two"));
serviceTwoSubscriber.onCompleted();
})).zipWith(Observable.<FooPart>create(serviceThreeSubscriber -> {
serviceThreeSubscriber.onNext(new FooPart("three"));
serviceThreeSubscriber.onCompleted();
}), (resultFromServiceTwo, resultFromServiceThree) -> {
Foo foo = new Foo();
foo.addPart(resultFromServiceTwo);
foo.addPart(resultFromServiceThree);
return foo;
}).subscribe(System.out::println);

33. RxJava Operator Tour

34. Combining Observables
• merge() - combine multiple Observables so they act like a single Observable
• zip() - combine sets of items from multiple Observables via a Function, and emit the
results
https://github.com/ReactiveX/RxJava/wiki/Combining-Observables

35. Conditionals
• doWhile() - emit Observable sequence, then repeat the sequence as long as the
condition remains true
• ifThen() - only emit Observable sequence if a condition is true, otherwise emit empty or
default sequence
• skipUntil() - discard emitted items from Observable until a second Observable emits an
item, then emit the remainder
• takeUntil() - emit items from Observable until a second Observable emits or notifies
https://github.com/ReactiveX/RxJava/wiki/Conditional-and-Boolean-Operators

36. Boolean Operators
• all() - do all the emitted items meet some criteria?
• contains() - does the Observable emit a particular item?
• exists() / isEmpty() - does an Observable emit items or not?
• sequenceEqual() - do two Observables emit equal sequences?
https://github.com/ReactiveX/RxJava/wiki/Conditional-and-Boolean-Operators

37. Filtering
• filter() - filter Observable with a predicate
• take(n) - emit only the first n items
• skip(n) - ignore the first n items emitted
• sample() - sample items according to a periodic interval
https://github.com/ReactiveX/RxJava/wiki/Filtering-Observables

38. Transforming
• map() - apply a function to each emitted item
• flatMap() - transform items emitted by Observable into Observables, then flatten
• scan() - apply a function to each emitted item, then feedback result and repeat
• buffer() - gather emitted items into bundles and emit the bundles
https://github.com/ReactiveX/RxJava/wiki/Transforming-Observables

39. Backpressure!

40. – https://github.com/ReactiveX/RxJava/wiki/Backpressure
“…a quickly-producing Observable meets a slow-consuming observer.”

41. Backpressure Observable
Observable.create(subscriber -> IntStream.iterate(0, i -> i + 2)
.forEach(value -> subscriber.onNext(value)))
.onBackpressureBuffer()
.subscribe(new BackpressureSubscriber());

42. Backpressure Subscriber
public class BackpressureSubscriber extends Subscriber<Object> {
private int counter = 0;
@Override
public void onStart() {
request(10);
}
@Override
public void onNext(Object o) {
if (counter < 9) {
processItem(o);
} else {
processItem(o);
resetCounter();
request(10);
}
}
Please only give me this many!
OK, I can handle more now.

43. Backpressure Subscriber
private void processItem(Object o) {
try {
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
counter++;
System.out.println(counter + " : " + o);
}
private void resetCounter() {
counter = 0;
System.out.println("FETCH MORE");
}
Simulate Processing Latency
}

44. WHAT ABOUT FAILURES?

45. What happens if we cut here?
ÂAPI
ÂAPI
!
!
!
µS
µS
µS
µS
µS

46. Or here?
ÂAPI
ÂAPI
!
!
!
µS
µS
µS
µS
µS

47. Or here?
ÂAPI
ÂAPI
!
!
!
µS
µS
µS
µS
µS

48. 500 INTERNAL SERVER ERROR

49. Fault Tolerance at Netflix
http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html

50. Without taking steps to ensure fault tolerance, 30 dependencies
each with 99.99% uptime would result in 2+ hours downtime/
month (99.99%30
= 99.7% uptime = 2.16 hours/month).
http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html

51. The Circuit Breaker Pattern

52. Circuit Breaker State Machine
Closed
on call / pass through
call succeeds / reset count
call fails / count failure
threshold reached / trip breaker
Open
on call / fail
on timeout / attempt reset
Half-Open
on call / pass through
call succeeds / reset
call fails / trip breaker
trip
breaker
trip
breaker
attempt
reset
reset

53. https://github.com/Netflix/Hystrix

54. Hello Hystrix World
public class CommandHelloWorld extends HystrixCommand<String> {
private final String name;
public CommandHelloWorld(String name) {
super(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));
this.name = name;
}
@Override
protected String run() {
// Do something that might fail...
return "Hello " + name + "!";
}
}

55. Synchronous
String s = new CommandHelloWorld("Bob").execute();
Block and wait!

56. Asynchronous
Future<String> s = new CommandHelloWorld("Bob").queue();

57. Reactive!
Observable<String> s = new CommandHelloWorld("Bob").observe();
s.subscribe(val -> {
// value emitted here
});

58. Fail Fast
public class CommandThatFailsFast extends HystrixCommand<String> {
private final boolean throwException;
public CommandThatFailsFast(boolean throwException) {
super(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));
this.throwException = throwException;
}
@Override
protected String run() {
if (throwException) {
throw new RuntimeException("failure from CommandThatFailsFast");
} else {
return "success";
}
}
}

59. Fail Silently
public class CommandThatFailsSilently extends HystrixCommand<String> {
private final boolean throwException;
public CommandThatFailsSilently(boolean throwException) {
super(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"));
this.throwException = throwException;
}
@Override
protected String run() {
if (throwException) {
throw new RuntimeException("failure from CommandThatFailsSilently");
} else {
return "success";
}
}
@Override
protected String getFallback() {
return null;
}
}
Fallback behavior!
}

60. Static Fallback
@Override
protected String run() {
if (throwException) {
throw new RuntimeException("failure from CommandWithStaticFallback");
} else {
return "success";
}
}
@Override
protected String getFallback() {
return "fallback";
}
Fallback behavior!
}

61. Stubbed Fallback
@Override
protected UserAccount run() {
// fetch UserAccount from remote service
// return UserAccountClient.getAccount(customerId);
throw new RuntimeException("forcing failure for example");
}
@Override
protected UserAccount getFallback() {
/**
* Return stubbed fallback with some static defaults, placeholders,
* and an injected value 'countryCodeFromGeoLookup' that we'll use
* instead of what we would have retrieved from the remote service.
*/
return new UserAccount(customerId, "Unknown Name",
countryCodeFromGeoLookup, true, true, false);
}
Fallback behavior!
}

62. Fallback: Cache via Network

63. Primary + Secondary with Fallback

64. How Hystrix Works!

65. Hystrix Dashboard

66. A Failing Circuit

67. RxJava + Hystrix

68. With Callables
Observable.fromCallable(new CallToServiceOne())
.flatMap(serviceOneResult -> Observable.fromCallable(
new CallToServiceTwo(serviceOneResult)))
.zipWith(Observable.fromCallable(
new CallToServiceThree()),
(FooPart resultFromServiceTwo, FooPart resultFromServiceThree) -> {
Foo foo = new Foo();
foo.addPart(resultFromServiceTwo);
foo.addPart(resultFromServiceThree);
return foo;
}).subscribe(System.out::println);

69. With HystrixObservableCommands
new CallToServiceOneCommand("callToServiceOne").observe()
.flatMap(serviceOneResult ->
new CallToServiceTwoCommand("callToServiceTwo", serviceOneResult).observe())
.zipWith(new CallToServiceThreeCommand("callToServiceThree").observe(),
(FooPart resultFromServiceTwo, FooPart resultFromServiceThree) -> {
Foo foo = new Foo();
foo.addPart(resultFromServiceTwo);
foo.addPart(resultFromServiceThree);
return foo;
}).subscribe(System.out::println);

70. Call to Service One
public class CallToServiceOneCommand extends HystrixObservableCommand<FooPart> {
public CallToServiceOneCommand(String name) {
super(HystrixCommandGroupKey.Factory.asKey(name));
}
@Override
protected Observable<FooPart> construct() {
return Observable.create(new Observable.OnSubscribe<FooPart>() {
@Override
public void call(Subscriber<? super FooPart> subscriber) {
subscriber.onNext(new FooPart("one"));
subscriber.onCompleted();
}
});
}
}

71. public class CallToServiceTwoCommand extends HystrixObservableCommand<FooPart> {
private final FooPart dependencyFromServiceOne;
private CallToServiceTwoCommand(String name, FooPart dependencyFromServiceOne) {
super(HystrixCommandGroupKey.Factory.asKey(name));
this.dependencyFromServiceOne = dependencyFromServiceOne;
}
@Override
protected Observable<FooPart> construct() {
return Observable.create(new Observable.OnSubscribe<FooPart>() {
@Override
public void call(Subscriber<? super FooPart> subscriber) {
subscriber.onNext(new FooPart(dependencyFromServiceOne + " + two"));
subscriber.onCompleted();
}
});
}
}
Call to Service Two

72. Spring Cloud
http://cloud.spring.io

73. Hystrix with Spring Cloud
@SpringBootApplication
@EnableDiscoveryClient
@EnableCircuitBreaker
public class Application {
public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}

74. Hystrix with Spring Cloud
@Service
@EnableConfigurationProperties(FortuneProperties.class)
public class FortuneService {
@Autowired
FortuneProperties fortuneProperties;
@Autowired
RestTemplate restTemplate;
@HystrixCommand(fallbackMethod = "fallbackFortune")
public Fortune randomFortune() {
return restTemplate.getForObject("http://fortunes/random", Fortune.class);
}
private Fortune fallbackFortune() {
return new Fortune(42L, fortuneProperties.getFallbackFortune());
}
}
}

75. RxJava + Hystrix + Spring Cloud
@HystrixCommand(fallbackMethod = "stubMovie")
public Observable<Movie> getMovie(final String mlId) {
return new ObservableResult<Movie>() {
@Override
public Movie invoke() {
return restTemplate.getForObject(
"http://springbox-catalog/movies/{mlId}", Movie.class, mlId);
}
};
}
private Movie stubMovie(final String mlId) {
Movie stub = new Movie();
stub.setMlId(mlId);
stub.setTitle("Interesting...the wrong title. Sssshhhh!");
return stub;
}

76. RxJava + Hystrix + Spring Cloud
@RequestMapping("/movie/{mlId}")
public Observable<MovieDetails> movieDetails(@PathVariable String mlId) {
return Observable.zip(
catalogIntegrationService.getMovie(mlId),
reviewsIntegrationService.reviewsFor(mlId),
recommendationsIntegrationService.getRecommendations(mlId),
(movie, reviews, recommendations) -> {
MovieDetails movieDetails = new MovieDetails();
movieDetails.setMlId(movie.getMlId());
movieDetails.setTitle(movie.getTitle());
movieDetails.setReviews(reviews);
movieDetails.setRecommendations(recommendations);
return movieDetails;
}
);
}

77. Reactive Landscape

78. Reactive Streams
http://www.reactive-streams.org/

79. Project Reactor
https://projectreactor.io/

80. JDK 9 Flow API
http://download.java.net/jdk9/docs/api/java/util/concurrent/Flow.html

81. REACTIVE FAULT TOLERANT PROGRAMMING
with Hystrix and RxJava
Get your FREE eBook!
http://bit.ly/cloud-native-book
Matt StineSenior Product Manager - Pivotal Software, Inc.
@mstine
matt.stine@gmail.com
http://mattstine.com