SlideShare une entreprise Scribd logo
1  sur  64
Télécharger pour lire hors ligne
at
“Top microservices and
fast data experts in the world.”
How Events Are Reshaping Modern Systems
Jonas Bonér

Lightbend CTO
The future is a function of the past - A. P. Robertson
The (local) present is a merge function of
multiple concurrent pasts - Jonas Bonér
val newLocalPresent = observedPasts
foldLeft(oldLocalPresent) { _ merge _ }
Stream All the Things!!
Dean Wampler

VP Fast Data Engineering

Lightbend
Hadoop classic batch architecture
Streaming, Fast Data Architecture
Latency Use case Suggested engine
Picoseconds to a few microseconds
(real-time)
Space X rocket
Custom hardware (FPGA), no JVM or
anything on top
Less than 100 microseconds Financial services Actors on the JVM
Less than 10 milliseconds Authorising credit cards
Fast data streaming like Flink, Akka
Streams, Kafka Streams
Less than 100 milliseconds Dashboards, Machine learning
We can think of micro-batches, so to
process records in bulk. Spark /
Streaming SQL
Less than 1 second to a minute ETL (Extract, Transform, Load) Mini-batches (spark streaming)
More than a few minutes
Check for fake reviews on a travel
website
Batch jobs
Latency cases to choose a streaming engine
Other choosing criteria
• Volume of data?

• Process records individually or in bulk?

• How do I want to deploy my app?
Overview of streaming engines and relation
with micro services
“Dean Wampler

StreamAllTheThings.pdf”
http://bit.ly/lightbend-fast-data
Search for:
Using Akka, Spark and Cassandra
To Scale Real-Time Auto Loan
Decisioning At Capital One
Fredrick Crable

Director Software Development

Capital One
• Capital One is the #1 bank in the US for loans

• Every loan request needs to be validated

• Legacy depending on 3rd party DSL

• Chose Akka after investigation
End result: a decision performance which runs
faster on smaller hardware.
Very happy with Akka (but I suspect they regret
being “too early” in the game)
Islands in the Stream: Integrating
Akka Streams and Akka Actors
Colin Breck

Tesla
Gather measurements from
a set of wind turbines
Attempt with actors, one actor per turbine
• Each actor sends a measure to the DB.

• Actor gets an internal counter to 1000.

• Actor gets a period flush with a scheduler.

• Actor gets some state to see if we just flushed.

• Actor gets some rate limiting mechanism.
Actor has become quite wordy and bug-prone.
Akka Streams to the rescue!
getMeasurementSource
.groupWithin(1000, 1.second)
.throttle(50, 1.second, 50, ThrottleMode.shaping)
.mapAsync(1)(seq => dbActor ? newMeasures(seq) …)
Source.queue
Source.actorRef
mapAsync + ask
Sink.actorRefWithAck
Bringing streams & actors together
• Streams can be encapsulated within actors

• Actor - Streams communication:
Simulating A LOT OF wind turbines
with actor sharding
Final demo
http://blog.colinbreck.com
Speeding up innovation at Verizon

(part of IBM’s talk)
Keyur Shah 

Associate Fellow

Verizon
• Two years ago: massive monolithic app - 2+
months to rollout new features

• Set up a small team and investigated the
possible platforms.

• They picked up the Reactive platform,
completely microservices driven.
They liked it so much that they want to have
all groups in Verizon to adpot it.
Next on the roadmap, try to use all this data
better. Looking at IBM Watson.
8 Akka Anti-Patterns you better be aware of
Manuel Bernhardt

Consultant
1. Using mutable state
• use immutable messages

• use queries like ask for state inquiries

• use pipeTo pattern -- including pipeTo self
instead…
2. Flat actor hierarchies.
If we don't have a hierarchy, we're missing the
point: the most important thing is the failure
tolerance part. No hierarchy == no failure
handling.
Behaviour of catch blocks
in some random study
3. Too many ActorSystems
1. string concatenation
2. non-async logging
3. not turning debug logging off in prod system
4. logging to files (we should stop doing that)
4. Wrong logging
a. use Akka's logging facility
b. carefully consider the logback appenders, with async variants
c. use placeholders: log.debug("{} hell", callback)
d. use a logging aggregation like logstash, don’t log to files
4. Lose touch with your hardware
We use a lot of abstractions. 

VM, then docker, then JVM, then Akka.
a. know your hardware (and configure
accordingly)

b. beware of virtualization 

c. run load tests on the same hardware of
the production system (!!!)
6. Blocking.
a. use a dedicated dispatcher if you really
must block.

b. if you need to wait, instead of
Thread.sleep, use a context scheduler.
7. Using Akka Remoting directly.
Use Akka cluster instead,
with membership service.
8. Using Java serialization.
Bad performance!
Use a proper binary format from the start (avro,
thrift, kryo), because once you are in production
it's already too late.
Reactive party
Thanks!

Contenu connexe

Tendances

Aleksandr Khokhlov - Do not chase the hype train. Just steal ideas
Aleksandr Khokhlov - Do not chase the hype train. Just steal ideasAleksandr Khokhlov - Do not chase the hype train. Just steal ideas
Aleksandr Khokhlov - Do not chase the hype train. Just steal ideas
OdessaJS Conf
 
Infrastructure as code with Terraform
Infrastructure as code with TerraformInfrastructure as code with Terraform
Infrastructure as code with Terraform
Sam Bashton
 

Tendances (20)

(APP310) Scheduling Using Apache Mesos in the Cloud | AWS re:Invent 2014
(APP310) Scheduling Using Apache Mesos in the Cloud | AWS re:Invent 2014(APP310) Scheduling Using Apache Mesos in the Cloud | AWS re:Invent 2014
(APP310) Scheduling Using Apache Mesos in the Cloud | AWS re:Invent 2014
 
Using eBPF to Measure the k8s Cluster Health
Using eBPF to Measure the k8s Cluster HealthUsing eBPF to Measure the k8s Cluster Health
Using eBPF to Measure the k8s Cluster Health
 
RxJS Animations Talk - 2017
RxJS Animations Talk - 2017RxJS Animations Talk - 2017
RxJS Animations Talk - 2017
 
Keeping Latency Low and Throughput High with Application-level Priority Manag...
Keeping Latency Low and Throughput High with Application-level Priority Manag...Keeping Latency Low and Throughput High with Application-level Priority Manag...
Keeping Latency Low and Throughput High with Application-level Priority Manag...
 
Aleksandr Khokhlov - Do not chase the hype train. Just steal ideas
Aleksandr Khokhlov - Do not chase the hype train. Just steal ideasAleksandr Khokhlov - Do not chase the hype train. Just steal ideas
Aleksandr Khokhlov - Do not chase the hype train. Just steal ideas
 
Software Exploitation Techniques by Amit Malik
Software Exploitation Techniques by Amit MalikSoftware Exploitation Techniques by Amit Malik
Software Exploitation Techniques by Amit Malik
 
ILM - Pipeline in the cloud
ILM - Pipeline in the cloudILM - Pipeline in the cloud
ILM - Pipeline in the cloud
 
Cassandra at Glogster
Cassandra at GlogsterCassandra at Glogster
Cassandra at Glogster
 
Sync with async
Sync with  asyncSync with  async
Sync with async
 
From APIs to Electrons: A JS on Hardware Journey
From APIs to Electrons: A JS on Hardware JourneyFrom APIs to Electrons: A JS on Hardware Journey
From APIs to Electrons: A JS on Hardware Journey
 
Big data Lambda Architecture - Batch Layer Hands On
Big data Lambda Architecture - Batch Layer Hands OnBig data Lambda Architecture - Batch Layer Hands On
Big data Lambda Architecture - Batch Layer Hands On
 
London Hug 19/5 - Terraform in Production
London Hug 19/5 - Terraform in ProductionLondon Hug 19/5 - Terraform in Production
London Hug 19/5 - Terraform in Production
 
Scalable Event Tracking
Scalable Event TrackingScalable Event Tracking
Scalable Event Tracking
 
Processing Big Data in Real-Time - Yanai Franchi, Tikal
Processing Big Data in Real-Time - Yanai Franchi, TikalProcessing Big Data in Real-Time - Yanai Franchi, Tikal
Processing Big Data in Real-Time - Yanai Franchi, Tikal
 
Scaling Ethereum using Zero-Knowledge Proofs
Scaling Ethereum using Zero-Knowledge ProofsScaling Ethereum using Zero-Knowledge Proofs
Scaling Ethereum using Zero-Knowledge Proofs
 
Apache Storm
Apache StormApache Storm
Apache Storm
 
Veritas NetBackup versus Veeam Benchmark Infographic
Veritas NetBackup versus Veeam Benchmark InfographicVeritas NetBackup versus Veeam Benchmark Infographic
Veritas NetBackup versus Veeam Benchmark Infographic
 
Infrastructure as code with Terraform
Infrastructure as code with TerraformInfrastructure as code with Terraform
Infrastructure as code with Terraform
 
Comparison nodejs frameworks using Polls API
Comparison nodejs frameworks using Polls APIComparison nodejs frameworks using Polls API
Comparison nodejs frameworks using Polls API
 
From Gust To Tempest: Scaling Storm
From Gust To Tempest: Scaling StormFrom Gust To Tempest: Scaling Storm
From Gust To Tempest: Scaling Storm
 

Similaire à Reactive Summit 2017 Highlights!

Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011
Andy Parsons
 
Capacity Management from Flickr
Capacity Management from FlickrCapacity Management from Flickr
Capacity Management from Flickr
xlight
 
Reactive Stream Processing with Akka Streams
Reactive Stream Processing with Akka StreamsReactive Stream Processing with Akka Streams
Reactive Stream Processing with Akka Streams
Konrad Malawski
 
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
xlight
 

Similaire à Reactive Summit 2017 Highlights! (20)

Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015Akka in Production - ScalaDays 2015
Akka in Production - ScalaDays 2015
 
Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011Andy Parsons Pivotal June 2011
Andy Parsons Pivotal June 2011
 
3.2 Streaming and Messaging
3.2 Streaming and Messaging3.2 Streaming and Messaging
3.2 Streaming and Messaging
 
Springone2gx 2014 Reactive Streams and Reactor
Springone2gx 2014 Reactive Streams and ReactorSpringone2gx 2014 Reactive Streams and Reactor
Springone2gx 2014 Reactive Streams and Reactor
 
Ruby Proxies for Scale, Performance, and Monitoring - GoGaRuCo - igvita.com
Ruby Proxies for Scale, Performance, and Monitoring - GoGaRuCo - igvita.comRuby Proxies for Scale, Performance, and Monitoring - GoGaRuCo - igvita.com
Ruby Proxies for Scale, Performance, and Monitoring - GoGaRuCo - igvita.com
 
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopProject Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
 
Streaming data analytics (Kinesis, EMR/Spark) - Pop-up Loft Tel Aviv
Streaming data analytics (Kinesis, EMR/Spark) - Pop-up Loft Tel Aviv Streaming data analytics (Kinesis, EMR/Spark) - Pop-up Loft Tel Aviv
Streaming data analytics (Kinesis, EMR/Spark) - Pop-up Loft Tel Aviv
 
Reactive Streams - László van den Hoek
Reactive Streams - László van den HoekReactive Streams - László van den Hoek
Reactive Streams - László van den Hoek
 
Capacity Management from Flickr
Capacity Management from FlickrCapacity Management from Flickr
Capacity Management from Flickr
 
John adams talk cloudy
John adams   talk cloudyJohn adams   talk cloudy
John adams talk cloudy
 
Reactive Stream Processing with Akka Streams
Reactive Stream Processing with Akka StreamsReactive Stream Processing with Akka Streams
Reactive Stream Processing with Akka Streams
 
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...
 
Fixing twitter
Fixing twitterFixing twitter
Fixing twitter
 
Fixing_Twitter
Fixing_TwitterFixing_Twitter
Fixing_Twitter
 
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
 
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...Fixing Twitter  Improving The Performance And Scalability Of The Worlds Most ...
Fixing Twitter Improving The Performance And Scalability Of The Worlds Most ...
 
Golang Performance : microbenchmarks, profilers, and a war story
Golang Performance : microbenchmarks, profilers, and a war storyGolang Performance : microbenchmarks, profilers, and a war story
Golang Performance : microbenchmarks, profilers, and a war story
 
Performance and predictability (1)
Performance and predictability (1)Performance and predictability (1)
Performance and predictability (1)
 
Performance and Predictability - Richard Warburton
Performance and Predictability - Richard WarburtonPerformance and Predictability - Richard Warburton
Performance and Predictability - Richard Warburton
 
DotNext 2020 - When and How to Use the Actor Model and Akka.NET
DotNext 2020 - When and How to Use the Actor Model and Akka.NETDotNext 2020 - When and How to Use the Actor Model and Akka.NET
DotNext 2020 - When and How to Use the Actor Model and Akka.NET
 

Plus de Fabio Tiriticco

Plus de Fabio Tiriticco (13)

Intro slides - Global Reactive Meetup - Move over JDBC!
Intro slides - Global Reactive Meetup - Move over JDBC!Intro slides - Global Reactive Meetup - Move over JDBC!
Intro slides - Global Reactive Meetup - Move over JDBC!
 
Planespotting - From Zero To Deep Learning
Planespotting - From Zero To Deep Learning Planespotting - From Zero To Deep Learning
Planespotting - From Zero To Deep Learning
 
From Zero To Deep Learning With Scala
From Zero To Deep Learning With ScalaFrom Zero To Deep Learning With Scala
From Zero To Deep Learning With Scala
 
Reactive Amsterdam - Maxim Burgerhout - Quarkus Intro
Reactive Amsterdam - Maxim Burgerhout - Quarkus IntroReactive Amsterdam - Maxim Burgerhout - Quarkus Intro
Reactive Amsterdam - Maxim Burgerhout - Quarkus Intro
 
Ten Frustrations From The Community Trenches (And How To Deal With Them)
Ten Frustrations From The Community Trenches (And How To Deal With Them)Ten Frustrations From The Community Trenches (And How To Deal With Them)
Ten Frustrations From The Community Trenches (And How To Deal With Them)
 
We all need friends and Akka just found Kubernetes
We all need friends and Akka just found KubernetesWe all need friends and Akka just found Kubernetes
We all need friends and Akka just found Kubernetes
 
Cloud native akka and kubernetes holy grail to elasticity
Cloud native akka and kubernetes   holy grail to elasticityCloud native akka and kubernetes   holy grail to elasticity
Cloud native akka and kubernetes holy grail to elasticity
 
Reactive Programming or Reactive Systems? (spoiler: both)
Reactive Programming or Reactive Systems? (spoiler: both)Reactive Programming or Reactive Systems? (spoiler: both)
Reactive Programming or Reactive Systems? (spoiler: both)
 
Beyond Fault Tolerance with Actor Programming
Beyond Fault Tolerance with Actor ProgrammingBeyond Fault Tolerance with Actor Programming
Beyond Fault Tolerance with Actor Programming
 
Akka Streams at Weeronline
Akka Streams at WeeronlineAkka Streams at Weeronline
Akka Streams at Weeronline
 
Reactive in Android and Beyond Rx
Reactive in Android and Beyond RxReactive in Android and Beyond Rx
Reactive in Android and Beyond Rx
 
Reactive Android: RxJava and beyond
Reactive Android: RxJava and beyondReactive Android: RxJava and beyond
Reactive Android: RxJava and beyond
 
WebSockets wiith Scala and Play! Framework
WebSockets wiith Scala and Play! FrameworkWebSockets wiith Scala and Play! Framework
WebSockets wiith Scala and Play! Framework
 

Dernier

Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 

Dernier (20)

Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024Finology Group – Insurtech Innovation Award 2024
Finology Group – Insurtech Innovation Award 2024
 
What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?What Are The Drone Anti-jamming Systems Technology?
What Are The Drone Anti-jamming Systems Technology?
 

Reactive Summit 2017 Highlights!

  • 1. at
  • 2. “Top microservices and fast data experts in the world.”
  • 3. How Events Are Reshaping Modern Systems Jonas Bonér Lightbend CTO
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14. The future is a function of the past - A. P. Robertson The (local) present is a merge function of multiple concurrent pasts - Jonas Bonér val newLocalPresent = observedPasts foldLeft(oldLocalPresent) { _ merge _ }
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21. Stream All the Things!! Dean Wampler VP Fast Data Engineering Lightbend
  • 22. Hadoop classic batch architecture
  • 23. Streaming, Fast Data Architecture
  • 24.
  • 25. Latency Use case Suggested engine Picoseconds to a few microseconds (real-time) Space X rocket Custom hardware (FPGA), no JVM or anything on top Less than 100 microseconds Financial services Actors on the JVM Less than 10 milliseconds Authorising credit cards Fast data streaming like Flink, Akka Streams, Kafka Streams Less than 100 milliseconds Dashboards, Machine learning We can think of micro-batches, so to process records in bulk. Spark / Streaming SQL Less than 1 second to a minute ETL (Extract, Transform, Load) Mini-batches (spark streaming) More than a few minutes Check for fake reviews on a travel website Batch jobs Latency cases to choose a streaming engine
  • 26. Other choosing criteria • Volume of data? • Process records individually or in bulk? • How do I want to deploy my app? Overview of streaming engines and relation with micro services
  • 28.
  • 29. Using Akka, Spark and Cassandra To Scale Real-Time Auto Loan Decisioning At Capital One Fredrick Crable Director Software Development Capital One
  • 30. • Capital One is the #1 bank in the US for loans • Every loan request needs to be validated • Legacy depending on 3rd party DSL • Chose Akka after investigation
  • 31.
  • 32.
  • 33. End result: a decision performance which runs faster on smaller hardware. Very happy with Akka (but I suspect they regret being “too early” in the game)
  • 34. Islands in the Stream: Integrating Akka Streams and Akka Actors Colin Breck Tesla
  • 35. Gather measurements from a set of wind turbines
  • 36. Attempt with actors, one actor per turbine • Each actor sends a measure to the DB. • Actor gets an internal counter to 1000. • Actor gets a period flush with a scheduler. • Actor gets some state to see if we just flushed. • Actor gets some rate limiting mechanism. Actor has become quite wordy and bug-prone.
  • 37. Akka Streams to the rescue! getMeasurementSource .groupWithin(1000, 1.second) .throttle(50, 1.second, 50, ThrottleMode.shaping) .mapAsync(1)(seq => dbActor ? newMeasures(seq) …)
  • 38. Source.queue Source.actorRef mapAsync + ask Sink.actorRefWithAck Bringing streams & actors together • Streams can be encapsulated within actors • Actor - Streams communication:
  • 39. Simulating A LOT OF wind turbines with actor sharding
  • 42. Speeding up innovation at Verizon (part of IBM’s talk) Keyur Shah Associate Fellow Verizon
  • 43. • Two years ago: massive monolithic app - 2+ months to rollout new features • Set up a small team and investigated the possible platforms. • They picked up the Reactive platform, completely microservices driven.
  • 44.
  • 45.
  • 46.
  • 47. They liked it so much that they want to have all groups in Verizon to adpot it. Next on the roadmap, try to use all this data better. Looking at IBM Watson.
  • 48. 8 Akka Anti-Patterns you better be aware of Manuel Bernhardt Consultant
  • 49. 1. Using mutable state • use immutable messages • use queries like ask for state inquiries • use pipeTo pattern -- including pipeTo self instead…
  • 50. 2. Flat actor hierarchies. If we don't have a hierarchy, we're missing the point: the most important thing is the failure tolerance part. No hierarchy == no failure handling.
  • 51. Behaviour of catch blocks in some random study
  • 52. 3. Too many ActorSystems
  • 53. 1. string concatenation 2. non-async logging 3. not turning debug logging off in prod system 4. logging to files (we should stop doing that) 4. Wrong logging a. use Akka's logging facility b. carefully consider the logback appenders, with async variants c. use placeholders: log.debug("{} hell", callback) d. use a logging aggregation like logstash, don’t log to files
  • 54. 4. Lose touch with your hardware
  • 55. We use a lot of abstractions. VM, then docker, then JVM, then Akka.
  • 56. a. know your hardware (and configure accordingly) b. beware of virtualization c. run load tests on the same hardware of the production system (!!!)
  • 57. 6. Blocking. a. use a dedicated dispatcher if you really must block. b. if you need to wait, instead of Thread.sleep, use a context scheduler.
  • 58. 7. Using Akka Remoting directly.
  • 59. Use Akka cluster instead, with membership service.
  • 60. 8. Using Java serialization. Bad performance! Use a proper binary format from the start (avro, thrift, kryo), because once you are in production it's already too late.
  • 62.
  • 63.