SlideShare une entreprise Scribd logo
1  sur  32
Télécharger pour lire hors ligne
Assorted Topics on Scalable
Distributed Services
- Sharma Podila
Senior Software Engineer
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
Prezo tooracleteam (2)
PS: Lots of things can, and will, fail at that scale
Topics
Managing Service Dependencies
Asynchronous Event Processing (RxJava)
Async IO
Mesos - DataCenter OS
Deployment and Build Automation
Managing Service Dependencies
Distributed architectures can have dozens of dependencies
Each can fail independently
Even 0.01% downtime on each of dozens of services
equates to potentially hours a month of downtime if not
engineered for resilience
Service A
Service B
Service C
request
Dependency
Dependency
A Healthy Request Flow
When One Backend Service Becomes
Latent
Threads and other resources can exhaust
with high volume
Hystrix to the Rescue
Wrap calls to external systems in a
dependency command object, run in
separate thread
Timeout calls after time ~ >99.5th of all
latencies
Control #threads w/ a pool or semaphore
Measure success, trip circuit if needed
Perform fallback logic
Monitor metrics and change in real time
Hystrix Wiki
Taming Tail Latencies of Service Calls
Real time metrics show problems as they occur
Trends help configure timeouts
Set timeouts based on histogram data
99.5th + buffer is a good start
Tier timeouts for retrials on other servers (e.g., 5, 15, 30)
Reactive Data/Event Stream
Processing
Processing a Data/Event Stream
Iterator<T> iterator = dataStream.iterator();
while(iterator.hasNext()) { process(iterator.next()); }
What if dataStream represents an unbounded stream?
What if data comes over the network? With latencies, failures.
What if data comes from multiple sources?
How would you manage concurrency? Threads? Semaphores?
RxJava implementation of reactive extensions addresses these questions
“...provides a collection of operators with which you can filter, select,
transform, combine, and compose Observables. This allows for efficient
execution and composition…”
RxJava
Java impl for Reactive Extensions
A library for composing asynchronous and event-based programs by using
observable sequences
Extends the observer pattern to support sequences of data/events and adds
operators that allow you to compose sequences together declaratively while
abstracting away concerns about things like low-level threading,
synchronization, thread-safety, concurrent data structures, and non-blocking
I/O
Event Iterable (pull) Observable (push)
retrieve data T next() onNext(T)
discover error throws Exception onError(Exception)
complete returns onComplete()
RxJava Operator Examples
Example Code: Iterable and Observable
getDataFromLocalMemory()
.skip(10)
.take(5)
.map({ s -> return s + "
transformed" })
.forEach({ println "next
=> " + it })
getDataFromNetwork()
.skip(10)
.take(5)
.map({ s -> return s + "
transformed" })
.subscribe({ println
"onNext => " + it })
Data can be pushed from multiple sources
No need to block for result availability
RxJava is a tool to react to push data
Java Futures as an alternative are non-trivial with nested async execution
Async IO
Async IO with Netty, RxNetty
Netty is an NIO client server framework
(see Java IO Vs. NIO)
Supports non-blocking IO
High throughput, low latency, less resource consumption
RxNetty is Reactive Extensions adaptor for Netty
When using something like Netty,
Total #threads in app = Total #cores in the system
RxNetty Server Example
public static void main(final String[] args) {
final int port = 8080;
RxNetty.createHttpServer(port, new RequestHandler<ByteBuf, ByteBuf>() {
@Override
public Observable<Void> handle(HttpServerRequest<ByteBuf> request, final HttpServerResponse<ByteBuf> response) {
System.out.println("New request recieved");
System.out.println(request.getHttpMethod() + " " + request.getUri() + ' ' + request.getHttpVersion());
for (Map.Entry<String, String> header : request.getHeaders().entries()) {
System.out.println(header.getKey() + ": " + header.getValue());
}
<continued…>
RxNetty Server Example (Cntd.)
return request.getContent().materialize()
.flatMap(new Func1<Notification<ByteBuf>, Observable<Void>>() {
@Override
public Observable<Void> call(Notification<ByteBuf> notification) {
if (notification.isOnCompleted()) {
return response.writeStringAndFlush("Welcome!!!");
} else if (notification.isOnError()) {
return Observable.error(notification.getThrowable());
} else {
ByteBuf next = notification.getValue();
System.out.println(next.toString(Charset.defaultCharset()));
return Observable.empty();
}
}
});
}
}).startAndWait();
}
Mesos
Mesos Cluster Manager
Resource allocation across distributed applications (aka
Frameworks) on shared pool of nodes.
Akin to Google Borg
Plugable isolation for CPU, I/O, etc. via Linux CGroups,
Docker, etc.
Fault tolerant leader election via ZooKeeper
Used at Twitter, AirBnB, etc.
Mesos Architecture
Mesos Resource Offers
Mesos Framework Development
Implement Framework Scheduler and Executor
Scheduler:
resourceOffers(SchedulerDriver driver, java.util.List<Offer> offers)
executorLost(SchedulerDriver driver, ExecutorID executorId, SlaveID slaveId, int status)
statusUpdate(SchedulerDriver driver, TaskStatus status)
… and more
Executor:
launchTask(ExecutorDriver driver, TaskInfo task)
killTask(ExecutorDriver driver, TaskID taskId)
… and more
Mesos Framework Fault Tolerance
Mesos task reconciliation
Periodic heartbeats from executors
State engine
taskStatesStream
.groupBy(JobId)
.flatmap(groupedObservable) {
groupedObservable.takeWhile(state != TerminalState)
.debounce(2000)
.doOnNext(state) { schedule(taskStuckInState(state), stateTimeout); }
}
RxJava Style
Prezo tooracleteam (2)

Contenu connexe

Tendances

Clean Architecture on Android
Clean Architecture on AndroidClean Architecture on Android
Clean Architecture on AndroidTianming Xu
 
Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Brian Brazil
 
Efficient monitoring and alerting
Efficient monitoring and alertingEfficient monitoring and alerting
Efficient monitoring and alertingTobias Schmidt
 
A Deep Learning use case for water end use detection by Roberto Díaz and José...
A Deep Learning use case for water end use detection by Roberto Díaz and José...A Deep Learning use case for water end use detection by Roberto Díaz and José...
A Deep Learning use case for water end use detection by Roberto Díaz and José...Big Data Spain
 
Monitoring microservices with Prometheus
Monitoring microservices with PrometheusMonitoring microservices with Prometheus
Monitoring microservices with PrometheusTobias Schmidt
 
Complex Event Processing - A brief overview
Complex Event Processing - A brief overviewComplex Event Processing - A brief overview
Complex Event Processing - A brief overviewIstván Dávid
 
Structured streaming for machine learning
Structured streaming for machine learningStructured streaming for machine learning
Structured streaming for machine learningSeth Hendrickson
 
Reactive programming on Android
Reactive programming on AndroidReactive programming on Android
Reactive programming on AndroidTomáš Kypta
 
Building Stateful Microservices With Akka
Building Stateful Microservices With AkkaBuilding Stateful Microservices With Akka
Building Stateful Microservices With AkkaYaroslav Tkachenko
 
Batch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache FlinkBatch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache FlinkVasia Kalavri
 
Introduction to Structured Streaming
Introduction to Structured StreamingIntroduction to Structured Streaming
Introduction to Structured StreamingKnoldus Inc.
 
A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...
A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...
A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...Databricks
 
Mantis: Netflix's Event Stream Processing System
Mantis: Netflix's Event Stream Processing SystemMantis: Netflix's Event Stream Processing System
Mantis: Netflix's Event Stream Processing SystemC4Media
 
Universal metrics with Apache Beam
Universal metrics with Apache BeamUniversal metrics with Apache Beam
Universal metrics with Apache BeamEtienne Chauchot
 
CloudBrew 2016 - Building IoT solution with Service Fabric
CloudBrew 2016 - Building IoT solution with Service FabricCloudBrew 2016 - Building IoT solution with Service Fabric
CloudBrew 2016 - Building IoT solution with Service FabricTeemu Tapanila
 
Complex Event Processing with Esper
Complex Event Processing with EsperComplex Event Processing with Esper
Complex Event Processing with EsperAntónio Alegria
 
Predictive Datacenter Analytics with Strymon
Predictive Datacenter Analytics with StrymonPredictive Datacenter Analytics with Strymon
Predictive Datacenter Analytics with StrymonVasia Kalavri
 
Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...Databricks
 

Tendances (20)

Clean Architecture on Android
Clean Architecture on AndroidClean Architecture on Android
Clean Architecture on Android
 
Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)Prometheus for Monitoring Metrics (Fermilab 2018)
Prometheus for Monitoring Metrics (Fermilab 2018)
 
Efficient monitoring and alerting
Efficient monitoring and alertingEfficient monitoring and alerting
Efficient monitoring and alerting
 
A Deep Learning use case for water end use detection by Roberto Díaz and José...
A Deep Learning use case for water end use detection by Roberto Díaz and José...A Deep Learning use case for water end use detection by Roberto Díaz and José...
A Deep Learning use case for water end use detection by Roberto Díaz and José...
 
Monitoring microservices with Prometheus
Monitoring microservices with PrometheusMonitoring microservices with Prometheus
Monitoring microservices with Prometheus
 
Complex Event Processing - A brief overview
Complex Event Processing - A brief overviewComplex Event Processing - A brief overview
Complex Event Processing - A brief overview
 
Structured streaming in Spark
Structured streaming in SparkStructured streaming in Spark
Structured streaming in Spark
 
Structured streaming for machine learning
Structured streaming for machine learningStructured streaming for machine learning
Structured streaming for machine learning
 
Reactive programming on Android
Reactive programming on AndroidReactive programming on Android
Reactive programming on Android
 
Building Stateful Microservices With Akka
Building Stateful Microservices With AkkaBuilding Stateful Microservices With Akka
Building Stateful Microservices With Akka
 
Batch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache FlinkBatch and Stream Graph Processing with Apache Flink
Batch and Stream Graph Processing with Apache Flink
 
Introduction to Structured Streaming
Introduction to Structured StreamingIntroduction to Structured Streaming
Introduction to Structured Streaming
 
A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...
A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...
A Practical Approach to Building a Streaming Processing Pipeline for an Onlin...
 
Mantis: Netflix's Event Stream Processing System
Mantis: Netflix's Event Stream Processing SystemMantis: Netflix's Event Stream Processing System
Mantis: Netflix's Event Stream Processing System
 
Universal metrics with Apache Beam
Universal metrics with Apache BeamUniversal metrics with Apache Beam
Universal metrics with Apache Beam
 
CloudBrew 2016 - Building IoT solution with Service Fabric
CloudBrew 2016 - Building IoT solution with Service FabricCloudBrew 2016 - Building IoT solution with Service Fabric
CloudBrew 2016 - Building IoT solution with Service Fabric
 
Complex Event Processing with Esper
Complex Event Processing with EsperComplex Event Processing with Esper
Complex Event Processing with Esper
 
Predictive Datacenter Analytics with Strymon
Predictive Datacenter Analytics with StrymonPredictive Datacenter Analytics with Strymon
Predictive Datacenter Analytics with Strymon
 
So you think you can stream.pptx
So you think you can stream.pptxSo you think you can stream.pptx
So you think you can stream.pptx
 
Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...Building Continuous Application with Structured Streaming and Real-Time Data ...
Building Continuous Application with Structured Streaming and Real-Time Data ...
 

En vedette

QCon NY 2014 - Evolving REST for an IoT World
QCon NY 2014 - Evolving REST for an IoT WorldQCon NY 2014 - Evolving REST for an IoT World
QCon NY 2014 - Evolving REST for an IoT WorldTodd Montgomery
 
Cassandra drivers
Cassandra driversCassandra drivers
Cassandra driversTyler Hobbs
 
Scala the good and bad parts
Scala the good and bad partsScala the good and bad parts
Scala the good and bad partsbenewu
 
Lightblue project
Lightblue project Lightblue project
Lightblue project Luan Cestari
 
The responsible-developer-thomas-sundberg-jdd-2014
The responsible-developer-thomas-sundberg-jdd-2014The responsible-developer-thomas-sundberg-jdd-2014
The responsible-developer-thomas-sundberg-jdd-2014TSundberg
 
JDD 2013 JavaFX
JDD 2013 JavaFXJDD 2013 JavaFX
JDD 2013 JavaFX益裕 張
 
Intro to Functional Programming with RxJava
Intro to Functional Programming with RxJavaIntro to Functional Programming with RxJava
Intro to Functional Programming with RxJavaMike Nakhimovich
 

En vedette (10)

#Hackmtl
#Hackmtl#Hackmtl
#Hackmtl
 
QCon NY 2014 - Evolving REST for an IoT World
QCon NY 2014 - Evolving REST for an IoT WorldQCon NY 2014 - Evolving REST for an IoT World
QCon NY 2014 - Evolving REST for an IoT World
 
Cassandra drivers
Cassandra driversCassandra drivers
Cassandra drivers
 
Scala the good and bad parts
Scala the good and bad partsScala the good and bad parts
Scala the good and bad parts
 
Lightblue project
Lightblue project Lightblue project
Lightblue project
 
The responsible-developer-thomas-sundberg-jdd-2014
The responsible-developer-thomas-sundberg-jdd-2014The responsible-developer-thomas-sundberg-jdd-2014
The responsible-developer-thomas-sundberg-jdd-2014
 
JDD 2013 JavaFX
JDD 2013 JavaFXJDD 2013 JavaFX
JDD 2013 JavaFX
 
Reactive Web Applications
Reactive Web ApplicationsReactive Web Applications
Reactive Web Applications
 
Cuidar a las personas y la inteligencia colectiva
Cuidar a las personas y la inteligencia colectivaCuidar a las personas y la inteligencia colectiva
Cuidar a las personas y la inteligencia colectiva
 
Intro to Functional Programming with RxJava
Intro to Functional Programming with RxJavaIntro to Functional Programming with RxJava
Intro to Functional Programming with RxJava
 

Similaire à Prezo tooracleteam (2)

Apache Flink Overview at SF Spark and Friends
Apache Flink Overview at SF Spark and FriendsApache Flink Overview at SF Spark and Friends
Apache Flink Overview at SF Spark and FriendsStephan Ewen
 
Mario Fusco - Reactive programming in Java - Codemotion Milan 2017
Mario Fusco - Reactive programming in Java - Codemotion Milan 2017Mario Fusco - Reactive programming in Java - Codemotion Milan 2017
Mario Fusco - Reactive programming in Java - Codemotion Milan 2017Codemotion
 
High-speed, Reactive Microservices 2017
High-speed, Reactive Microservices 2017High-speed, Reactive Microservices 2017
High-speed, Reactive Microservices 2017Rick Hightower
 
Apache Samza 1.0 - What's New, What's Next
Apache Samza 1.0 - What's New, What's NextApache Samza 1.0 - What's New, What's Next
Apache Samza 1.0 - What's New, What's NextPrateek Maheshwari
 
Web Oriented Architecture at Oracle
Web Oriented Architecture at OracleWeb Oriented Architecture at Oracle
Web Oriented Architecture at OracleEmiliano Pecis
 
Observability foundations in dynamically evolving architectures
Observability foundations in dynamically evolving architecturesObservability foundations in dynamically evolving architectures
Observability foundations in dynamically evolving architecturesBoyan Dimitrov
 
Building Scalable Stateless Applications with RxJava
Building Scalable Stateless Applications with RxJavaBuilding Scalable Stateless Applications with RxJava
Building Scalable Stateless Applications with RxJavaRick Warren
 
Reactive programming with examples
Reactive programming with examplesReactive programming with examples
Reactive programming with examplesPeter Lawrey
 
Intelligent Monitoring
Intelligent MonitoringIntelligent Monitoring
Intelligent MonitoringIntelie
 
Flink Streaming Hadoop Summit San Jose
Flink Streaming Hadoop Summit San JoseFlink Streaming Hadoop Summit San Jose
Flink Streaming Hadoop Summit San JoseKostas Tzoumas
 
Server side JavaScript: going all the way
Server side JavaScript: going all the wayServer side JavaScript: going all the way
Server side JavaScript: going all the wayOleg Podsechin
 
Azure Service Fabric and the Actor Model: when did we forget Object Orientation?
Azure Service Fabric and the Actor Model: when did we forget Object Orientation?Azure Service Fabric and the Actor Model: when did we forget Object Orientation?
Azure Service Fabric and the Actor Model: when did we forget Object Orientation?João Pedro Martins
 
20160609 nike techtalks reactive applications tools of the trade
20160609 nike techtalks reactive applications   tools of the trade20160609 nike techtalks reactive applications   tools of the trade
20160609 nike techtalks reactive applications tools of the tradeshinolajla
 
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...MSDEVMTL
 
Introduction tomcat7 servlet3
Introduction tomcat7 servlet3Introduction tomcat7 servlet3
Introduction tomcat7 servlet3JavaEE Trainers
 
Visualizing Big Data in Realtime
Visualizing Big Data in RealtimeVisualizing Big Data in Realtime
Visualizing Big Data in RealtimeDataWorks Summit
 
High-Speed Reactive Microservices - trials and tribulations
High-Speed Reactive Microservices - trials and tribulationsHigh-Speed Reactive Microservices - trials and tribulations
High-Speed Reactive Microservices - trials and tribulationsRick Hightower
 
Akka streams - Umeå java usergroup
Akka streams - Umeå java usergroupAkka streams - Umeå java usergroup
Akka streams - Umeå java usergroupJohan Andrén
 

Similaire à Prezo tooracleteam (2) (20)

Apache Flink Overview at SF Spark and Friends
Apache Flink Overview at SF Spark and FriendsApache Flink Overview at SF Spark and Friends
Apache Flink Overview at SF Spark and Friends
 
Mario Fusco - Reactive programming in Java - Codemotion Milan 2017
Mario Fusco - Reactive programming in Java - Codemotion Milan 2017Mario Fusco - Reactive programming in Java - Codemotion Milan 2017
Mario Fusco - Reactive programming in Java - Codemotion Milan 2017
 
High-speed, Reactive Microservices 2017
High-speed, Reactive Microservices 2017High-speed, Reactive Microservices 2017
High-speed, Reactive Microservices 2017
 
Apache Samza 1.0 - What's New, What's Next
Apache Samza 1.0 - What's New, What's NextApache Samza 1.0 - What's New, What's Next
Apache Samza 1.0 - What's New, What's Next
 
Advancedservletsjsp
AdvancedservletsjspAdvancedservletsjsp
Advancedservletsjsp
 
Web Oriented Architecture at Oracle
Web Oriented Architecture at OracleWeb Oriented Architecture at Oracle
Web Oriented Architecture at Oracle
 
Observability foundations in dynamically evolving architectures
Observability foundations in dynamically evolving architecturesObservability foundations in dynamically evolving architectures
Observability foundations in dynamically evolving architectures
 
iiwas2009
iiwas2009iiwas2009
iiwas2009
 
Building Scalable Stateless Applications with RxJava
Building Scalable Stateless Applications with RxJavaBuilding Scalable Stateless Applications with RxJava
Building Scalable Stateless Applications with RxJava
 
Reactive programming with examples
Reactive programming with examplesReactive programming with examples
Reactive programming with examples
 
Intelligent Monitoring
Intelligent MonitoringIntelligent Monitoring
Intelligent Monitoring
 
Flink Streaming Hadoop Summit San Jose
Flink Streaming Hadoop Summit San JoseFlink Streaming Hadoop Summit San Jose
Flink Streaming Hadoop Summit San Jose
 
Server side JavaScript: going all the way
Server side JavaScript: going all the wayServer side JavaScript: going all the way
Server side JavaScript: going all the way
 
Azure Service Fabric and the Actor Model: when did we forget Object Orientation?
Azure Service Fabric and the Actor Model: when did we forget Object Orientation?Azure Service Fabric and the Actor Model: when did we forget Object Orientation?
Azure Service Fabric and the Actor Model: when did we forget Object Orientation?
 
20160609 nike techtalks reactive applications tools of the trade
20160609 nike techtalks reactive applications   tools of the trade20160609 nike techtalks reactive applications   tools of the trade
20160609 nike techtalks reactive applications tools of the trade
 
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...
Stephane Lapointe & Alexandre Brisebois: Développer des microservices avec Se...
 
Introduction tomcat7 servlet3
Introduction tomcat7 servlet3Introduction tomcat7 servlet3
Introduction tomcat7 servlet3
 
Visualizing Big Data in Realtime
Visualizing Big Data in RealtimeVisualizing Big Data in Realtime
Visualizing Big Data in Realtime
 
High-Speed Reactive Microservices - trials and tribulations
High-Speed Reactive Microservices - trials and tribulationsHigh-Speed Reactive Microservices - trials and tribulations
High-Speed Reactive Microservices - trials and tribulations
 
Akka streams - Umeå java usergroup
Akka streams - Umeå java usergroupAkka streams - Umeå java usergroup
Akka streams - Umeå java usergroup
 

Plus de Sharma Podila

Podila mesos con-northamerica_sep2017
Podila mesos con-northamerica_sep2017Podila mesos con-northamerica_sep2017
Podila mesos con-northamerica_sep2017Sharma Podila
 
Netflix container scheduling talk at stanford final
Netflix container scheduling talk at stanford   finalNetflix container scheduling talk at stanford   final
Netflix container scheduling talk at stanford finalSharma Podila
 
Podila mesos con europe keynote aug sep 2016
Podila mesos con europe keynote aug sep 2016Podila mesos con europe keynote aug sep 2016
Podila mesos con europe keynote aug sep 2016Sharma Podila
 
Scheduling a fuller house - Talk at QCon NY 2016
Scheduling a fuller house - Talk at QCon NY 2016Scheduling a fuller house - Talk at QCon NY 2016
Scheduling a fuller house - Talk at QCon NY 2016Sharma Podila
 
Lessons from Netflix Mesos Clusters
Lessons from Netflix Mesos ClustersLessons from Netflix Mesos Clusters
Lessons from Netflix Mesos ClustersSharma Podila
 
Prezo at-mesos con2015-final
Prezo at-mesos con2015-finalPrezo at-mesos con2015-final
Prezo at-mesos con2015-finalSharma Podila
 
Resource Scheduling using Apache Mesos in Cloud Native Environments
Resource Scheduling using Apache Mesos in Cloud Native EnvironmentsResource Scheduling using Apache Mesos in Cloud Native Environments
Resource Scheduling using Apache Mesos in Cloud Native EnvironmentsSharma Podila
 
AWS re:Invent 2014 talk: Scheduling using Apache Mesos in the Cloud
AWS re:Invent 2014 talk: Scheduling using Apache Mesos in the CloudAWS re:Invent 2014 talk: Scheduling using Apache Mesos in the Cloud
AWS re:Invent 2014 talk: Scheduling using Apache Mesos in the CloudSharma Podila
 

Plus de Sharma Podila (9)

Podila mesos con-northamerica_sep2017
Podila mesos con-northamerica_sep2017Podila mesos con-northamerica_sep2017
Podila mesos con-northamerica_sep2017
 
Netflix container scheduling talk at stanford final
Netflix container scheduling talk at stanford   finalNetflix container scheduling talk at stanford   final
Netflix container scheduling talk at stanford final
 
Podila QCon SF 2016
Podila QCon SF 2016Podila QCon SF 2016
Podila QCon SF 2016
 
Podila mesos con europe keynote aug sep 2016
Podila mesos con europe keynote aug sep 2016Podila mesos con europe keynote aug sep 2016
Podila mesos con europe keynote aug sep 2016
 
Scheduling a fuller house - Talk at QCon NY 2016
Scheduling a fuller house - Talk at QCon NY 2016Scheduling a fuller house - Talk at QCon NY 2016
Scheduling a fuller house - Talk at QCon NY 2016
 
Lessons from Netflix Mesos Clusters
Lessons from Netflix Mesos ClustersLessons from Netflix Mesos Clusters
Lessons from Netflix Mesos Clusters
 
Prezo at-mesos con2015-final
Prezo at-mesos con2015-finalPrezo at-mesos con2015-final
Prezo at-mesos con2015-final
 
Resource Scheduling using Apache Mesos in Cloud Native Environments
Resource Scheduling using Apache Mesos in Cloud Native EnvironmentsResource Scheduling using Apache Mesos in Cloud Native Environments
Resource Scheduling using Apache Mesos in Cloud Native Environments
 
AWS re:Invent 2014 talk: Scheduling using Apache Mesos in the Cloud
AWS re:Invent 2014 talk: Scheduling using Apache Mesos in the CloudAWS re:Invent 2014 talk: Scheduling using Apache Mesos in the Cloud
AWS re:Invent 2014 talk: Scheduling using Apache Mesos in the Cloud
 

Prezo tooracleteam (2)

  • 1. Assorted Topics on Scalable Distributed Services - Sharma Podila Senior Software Engineer
  • 9. PS: Lots of things can, and will, fail at that scale
  • 10. Topics Managing Service Dependencies Asynchronous Event Processing (RxJava) Async IO Mesos - DataCenter OS Deployment and Build Automation
  • 11. Managing Service Dependencies Distributed architectures can have dozens of dependencies Each can fail independently Even 0.01% downtime on each of dozens of services equates to potentially hours a month of downtime if not engineered for resilience Service A Service B Service C request Dependency Dependency
  • 13. When One Backend Service Becomes Latent
  • 14. Threads and other resources can exhaust with high volume
  • 15. Hystrix to the Rescue Wrap calls to external systems in a dependency command object, run in separate thread Timeout calls after time ~ >99.5th of all latencies Control #threads w/ a pool or semaphore Measure success, trip circuit if needed Perform fallback logic Monitor metrics and change in real time Hystrix Wiki
  • 16. Taming Tail Latencies of Service Calls Real time metrics show problems as they occur Trends help configure timeouts Set timeouts based on histogram data 99.5th + buffer is a good start Tier timeouts for retrials on other servers (e.g., 5, 15, 30)
  • 18. Processing a Data/Event Stream Iterator<T> iterator = dataStream.iterator(); while(iterator.hasNext()) { process(iterator.next()); } What if dataStream represents an unbounded stream? What if data comes over the network? With latencies, failures. What if data comes from multiple sources? How would you manage concurrency? Threads? Semaphores? RxJava implementation of reactive extensions addresses these questions “...provides a collection of operators with which you can filter, select, transform, combine, and compose Observables. This allows for efficient execution and composition…”
  • 19. RxJava Java impl for Reactive Extensions A library for composing asynchronous and event-based programs by using observable sequences Extends the observer pattern to support sequences of data/events and adds operators that allow you to compose sequences together declaratively while abstracting away concerns about things like low-level threading, synchronization, thread-safety, concurrent data structures, and non-blocking I/O Event Iterable (pull) Observable (push) retrieve data T next() onNext(T) discover error throws Exception onError(Exception) complete returns onComplete()
  • 21. Example Code: Iterable and Observable getDataFromLocalMemory() .skip(10) .take(5) .map({ s -> return s + " transformed" }) .forEach({ println "next => " + it }) getDataFromNetwork() .skip(10) .take(5) .map({ s -> return s + " transformed" }) .subscribe({ println "onNext => " + it }) Data can be pushed from multiple sources No need to block for result availability RxJava is a tool to react to push data Java Futures as an alternative are non-trivial with nested async execution
  • 23. Async IO with Netty, RxNetty Netty is an NIO client server framework (see Java IO Vs. NIO) Supports non-blocking IO High throughput, low latency, less resource consumption RxNetty is Reactive Extensions adaptor for Netty When using something like Netty, Total #threads in app = Total #cores in the system
  • 24. RxNetty Server Example public static void main(final String[] args) { final int port = 8080; RxNetty.createHttpServer(port, new RequestHandler<ByteBuf, ByteBuf>() { @Override public Observable<Void> handle(HttpServerRequest<ByteBuf> request, final HttpServerResponse<ByteBuf> response) { System.out.println("New request recieved"); System.out.println(request.getHttpMethod() + " " + request.getUri() + ' ' + request.getHttpVersion()); for (Map.Entry<String, String> header : request.getHeaders().entries()) { System.out.println(header.getKey() + ": " + header.getValue()); } <continued…>
  • 25. RxNetty Server Example (Cntd.) return request.getContent().materialize() .flatMap(new Func1<Notification<ByteBuf>, Observable<Void>>() { @Override public Observable<Void> call(Notification<ByteBuf> notification) { if (notification.isOnCompleted()) { return response.writeStringAndFlush("Welcome!!!"); } else if (notification.isOnError()) { return Observable.error(notification.getThrowable()); } else { ByteBuf next = notification.getValue(); System.out.println(next.toString(Charset.defaultCharset())); return Observable.empty(); } } }); } }).startAndWait(); }
  • 26. Mesos
  • 27. Mesos Cluster Manager Resource allocation across distributed applications (aka Frameworks) on shared pool of nodes. Akin to Google Borg Plugable isolation for CPU, I/O, etc. via Linux CGroups, Docker, etc. Fault tolerant leader election via ZooKeeper Used at Twitter, AirBnB, etc.
  • 30. Mesos Framework Development Implement Framework Scheduler and Executor Scheduler: resourceOffers(SchedulerDriver driver, java.util.List<Offer> offers) executorLost(SchedulerDriver driver, ExecutorID executorId, SlaveID slaveId, int status) statusUpdate(SchedulerDriver driver, TaskStatus status) … and more Executor: launchTask(ExecutorDriver driver, TaskInfo task) killTask(ExecutorDriver driver, TaskID taskId) … and more
  • 31. Mesos Framework Fault Tolerance Mesos task reconciliation Periodic heartbeats from executors State engine taskStatesStream .groupBy(JobId) .flatmap(groupedObservable) { groupedObservable.takeWhile(state != TerminalState) .debounce(2000) .doOnNext(state) { schedule(taskStuckInState(state), stateTimeout); } } RxJava Style