Successfully reported this slideshow.
ReactiveProgramming @ Scale
Paul Downey
● Director of UI & APIs for demand advertising
● Based in Dublin, Ireland
● Email: paul.downey@teamaol.com
Joh...
A
A
A
P
P
P
Exchange
RTB Ad Cycle
100 ms
20,000,000,000+
● Responsive
● Scale
● Resilient
● Capacity to process lots of data
System Requirements
Moore’s Law
7
Race Conditions
Deadlocks
Contention
Non-Deterministic State
Thread Starvation
Livelocks
Amadahl’s Law
9
Concurrency in
Practice.
Java Concurrency In Theory
11
Java Concurrency In Practice
12
Why is it like this?
13
Why is it like this?
14
Can we fix this?
15
What is Reactive?
16
What is Reactive?
17
Functional Reactive Programming?
Actor based concurrency?
Functional Reactive
Programming.
Is it something new?
19
1997
Big in 1997?
20
Is it something new?
21
Well established on the front end
22
What is it?
Functional programming?
24
Functional programming
25
Less buggy code
Easier to parallelise
Easier to optimise
Reactive programming
26
Reactive pipeline manages flow
What not How
Reactive programming
27
What to do
Change each element by multiplying by 100
Remove all elements over 550
Print out each r...
Reactive programming
28
Composition
Change each element by multiplying by 100
Remove all elements over 550
Print out each ...
Reactive programming
29
.map(e -> e*100)
.forEach(System.out::println)
.filter(e->e<551)
Example
ReactiveSeq.of(6,5,1,2)
A Reactive Pipeline
30
first stage second stage terminal stage
A Reactive Pipeline
31
first stage
(map)
second stage
(filter)
terminal stage
(forEach)
A Reactive Pipeline
32
first stage
(map)
second stage
(filter)
terminal stage
(forEach)
Data is pushed through the pipelin...
Forecasting
Forecasting
34
Campaign Planning and Optimization
Query data across petabytes data
Accuracy & Responsiveness
Price / Volume Curve
Every Day We Process
36
Bid Request
Records
Impression Records Viewability Records
20B 2B 2B
Indexing Data
37
Reactive
Indexing
What we need to do
A Reactive Pipeline
40
first stage
(map)
second stage
(filter)
terminal stage
(forEach)
Data is pushed through the pipelin...
Scaling Up
41
first stage
(map)
second stage
(filter)
terminal stage
(forEach)
core 1
first stage
(map)
second stage
(filt...
Scaling Up for I/O
42
first stage
(map)
second stage
(filter)
terminal stage
(forEach)
future 1
future 2
future 3
future 4...
Performance Differences
43
What would the
code look like?
Sequential Stream
45
.map(e -> e*100)
.forEach(System.out::println)
.filter(e->e<551)
Example
ReactiveSeq.of(6,5,1,2)
.fut...
Async I/O
46
.map(e -> e*100)
.filter(e->e<551)
Example
FutureW.of(()->loadData(6))
//non-blocking, FutureW executes async...
Parallel Async Stream
47
.map(this::loadData)
.forEach(System.out::println)
.filter(e->e<551)
Example
new LazyReact(100).o...
What we need to do
Reactive Indexing Architecture
49
What have we
learnt?
Conclusions
51
Conclusions
52
Conclusions
53
Thank You.
Reactive programming at scale
Prochain SlideShare
Chargement dans…5
×

Reactive programming at scale

1 442 vues

Publié le

Overview of why (and how) we have adopted a functional reactive approach to solve problems of scale @ AOL, in particular within AOL Advertising's forecasting platform.

Publié dans : Technologie

Reactive programming at scale

  1. 1. ReactiveProgramming @ Scale
  2. 2. Paul Downey ● Director of UI & APIs for demand advertising ● Based in Dublin, Ireland ● Email: paul.downey@teamaol.com John McClean ● Systems Architect Demand Side Forecasting ● Based in Dublin, Ireland. ● Twitter: cyclops_aol ● Medium: https://medium.com/@johnmcclean ● Email: john.mcclean@teamaol.com About Speakers
  3. 3. A A A P P P Exchange
  4. 4. RTB Ad Cycle 100 ms 20,000,000,000+
  5. 5. ● Responsive ● Scale ● Resilient ● Capacity to process lots of data System Requirements
  6. 6. Moore’s Law 7
  7. 7. Race Conditions Deadlocks Contention Non-Deterministic State Thread Starvation Livelocks
  8. 8. Amadahl’s Law 9
  9. 9. Concurrency in Practice.
  10. 10. Java Concurrency In Theory 11
  11. 11. Java Concurrency In Practice 12
  12. 12. Why is it like this? 13
  13. 13. Why is it like this? 14
  14. 14. Can we fix this? 15
  15. 15. What is Reactive? 16
  16. 16. What is Reactive? 17 Functional Reactive Programming? Actor based concurrency?
  17. 17. Functional Reactive Programming.
  18. 18. Is it something new? 19 1997
  19. 19. Big in 1997? 20
  20. 20. Is it something new? 21
  21. 21. Well established on the front end 22
  22. 22. What is it?
  23. 23. Functional programming? 24
  24. 24. Functional programming 25 Less buggy code Easier to parallelise Easier to optimise
  25. 25. Reactive programming 26 Reactive pipeline manages flow What not How
  26. 26. Reactive programming 27 What to do Change each element by multiplying by 100 Remove all elements over 550 Print out each remaining element
  27. 27. Reactive programming 28 Composition Change each element by multiplying by 100 Remove all elements over 550 Print out each remaining element map(e -> e*100) forEach(System.out::println) filter(e->e<551)
  28. 28. Reactive programming 29 .map(e -> e*100) .forEach(System.out::println) .filter(e->e<551) Example ReactiveSeq.of(6,5,1,2)
  29. 29. A Reactive Pipeline 30 first stage second stage terminal stage
  30. 30. A Reactive Pipeline 31 first stage (map) second stage (filter) terminal stage (forEach)
  31. 31. A Reactive Pipeline 32 first stage (map) second stage (filter) terminal stage (forEach) Data is pushed through the pipeline 500 2 1 5 600
  32. 32. Forecasting
  33. 33. Forecasting 34 Campaign Planning and Optimization Query data across petabytes data Accuracy & Responsiveness
  34. 34. Price / Volume Curve
  35. 35. Every Day We Process 36 Bid Request Records Impression Records Viewability Records 20B 2B 2B
  36. 36. Indexing Data 37
  37. 37. Reactive Indexing
  38. 38. What we need to do
  39. 39. A Reactive Pipeline 40 first stage (map) second stage (filter) terminal stage (forEach) Data is pushed through the pipeline 500 2 1 5 600
  40. 40. Scaling Up 41 first stage (map) second stage (filter) terminal stage (forEach) core 1 first stage (map) second stage (filter) terminal stage (forEach) core 2 first stage (map) second stage (filter) terminal stage (forEach) core 3 first stage (map) second stage (filter) terminal stage (forEach) core 4
  41. 41. Scaling Up for I/O 42 first stage (map) second stage (filter) terminal stage (forEach) future 1 future 2 future 3 future 4 future 5 future 6 future 7 future 8 future 9 future 10 future 11
  42. 42. Performance Differences 43
  43. 43. What would the code look like?
  44. 44. Sequential Stream 45 .map(e -> e*100) .forEach(System.out::println) .filter(e->e<551) Example ReactiveSeq.of(6,5,1,2) .futureOperations(executor)
  45. 45. Async I/O 46 .map(e -> e*100) .filter(e->e<551) Example FutureW.of(()->loadData(6)) //non-blocking, FutureW executes asynchronously .forEach(System.out::println)
  46. 46. Parallel Async Stream 47 .map(this::loadData) .forEach(System.out::println) .filter(e->e<551) Example new LazyReact(100).of(6,5,1,2)
  47. 47. What we need to do
  48. 48. Reactive Indexing Architecture 49
  49. 49. What have we learnt?
  50. 50. Conclusions 51
  51. 51. Conclusions 52
  52. 52. Conclusions 53
  53. 53. Thank You.

×