Why Actor-Based Systems Are The Best For Microservices

•

2 j'aime•945 vues

The document discusses why actor-based systems are well-suited for microservices architectures. Actor models use message passing for communication between loosely coupled components, avoiding tight coupling issues of HTTP/RPC APIs. Message queues provide lightweight pub/sub messaging that is scalable, available, and promotes eventual consistency over tight synchronization. Actors represent message-driven processes that handle incoming messages asynchronously, supporting natural distributed and concurrent processing. Their built-in failure handling, routing, and back-pressure capabilities make actors a simple and high-level abstraction well-aligned with distributed, message-driven microservices.

Logiciels

Technical Overview
[Client]
[Date]
Why Actor-Based Systems Are
The Best For Microservices
Yaroslav Tkachenko
Senior Software Engineer, Lead at Mobify
@sap1ens

“Easy” way – HTTP / RPC API
POST /foo
service-x.example.com

§ Destination – where to send request?
§ Service discovery
§ Tight coupling
§ Time – expect reply right away?
§ Failure – always expect success?
§ Retries and failure handling
§ Back-pressure / circuit breakers

But wait…
How about Netflix, Twitter, etc.?

Yes, it’s possible to build reliable systems
with HTTP/RPC APIs! But:
§ Are you ready to invest a lot of resources
§ Do you really need it?

Reactive Revealed: Resiliency, Failures vs Errors, Isolation,
Delegation and Replication by Jonas Boner

Surviving Micro-services by Richard Rodger

So, let’s try to:
§ Use message queues for communication
§ Use actors as publishers and subscribers

- Send …
- Receive …
Another way - Messaging
- Send …
- Receive …
queue.events

So, why message queues now?
§ Not ESBs!
§ Lightweight pub/sub instead
§ Low coupling
§ Promotes eventual consistency
§ Highly scalable and available
§ Built-in discovery, retries, tracing, etc.
§ Serverless!

By the way, I don’t think that ALL your communication should
happen through the message queues. But it’s something that
you should strive for!

External request
(HTTP)Write
Always
asynchronous
using messages
Read
Alternatives to
blocking call:
- Caching
- Denormalization
- Different routing

“Macroservices” External APIs
New user!
User created

EIP was published in 2003 and it contains 65 patterns

Patterns from the example:
§ Publish – Subscribe Channel (“Broadcast”)
§ Durable Subscriber
§ Idempotent Receiver

So, have I convinced you that messaging patterns are great?

But why actors? I can just use a message queue and a bunch
of consumers/producers

An actor is a computational entity that, in response to a
message it receives, can concurrently:
• send a finite number of messages to other actors
• create a finite number of new actors
• designate the behavior to be used for the next message it
receives.

Actors are known for concurrency, but let’s focus on message
passing instead

ActiveMQ message listener in Java (just kidding)

ActiveMQ message listener in Java (zoom-in)

Actor-based (Akka) message listener in Scala + Camel

Receiver Handler
onMessage(Message)
???
Receiver ActorSender Actor
Mailbox
Message
VS

I thought of objects being like biological cells and/or individual
computers on a network, only able to communicate with
messages...
OOP to me means only messaging, local retention and
protection and hiding of state-process, and extreme late-
binding of all things.
Alan Kay, one of the fathers of the idea
of object-oriented programming

Actors over message handlers:
• Semantically natural message processing that ”feels right”
• No difference between local and remote communication
• Straightforward state management
• Mailbox, routing and back-pressure
• Built-in failure handling (supervisors)

Actors in general:
• Simple and high-level abstraction for distribution,
concurrency and parallelism
• Asynchronous, non-blocking and highly performant
message-driven programming model
• Very lightweight message-driven processes

Instead of a summary:
Everything is a trade-off!

Contenu connexe

Tendances

Things were easier when all our data used to be offline, analyzed overnight in batches. Now our data is online, in motion, and generated constantly. For architects, developers and their businesses, this means that there is an urgent need for tools and applications that can deliver real-time (or near real-time) streaming ETL capabilities. In this session by Konrad Malawski, author, speaker and Senior Akka Engineer at Lightbend, you will learn how to build these streaming ETL pipelines with Akka Streams, Alpakka and Apache Kafka, and why they matter to enterprises that are increasingly turning to streaming Fast Data applications.

Build Real-Time Streaming ETL Pipelines With Akka Streams, Alpakka And Apache...

Lightbend

Since its stable release in 2016, Akka Streams is quickly becoming the de facto standard integration layer between various Streaming systems and products. Enterprises like PayPal, Intel, Samsung and Norwegian Cruise Lines see this is a game changer in terms of designing Reactive streaming applications by connecting pipelines of back-pressured asynchronous processing stages. This comes from the Reactive Streams initiative in part, which has been long led by Lightbend and others, allowing multiple streaming libraries to inter-operate between each other in a performant and resilient fashion, providing back-pressure all the way. But perhaps even more so thanks to the various integration drivers that have sprung up in the community and the Akka team—including drivers for Apache Kafka, Apache Cassandra, Streaming HTTP, Websockets and much more. In this webinar for JVM Architects, Konrad Malawski explores the what and why of Reactive integrations, with examples featuring technologies like Akka Streams, Apache Kafka, and Alpakka, a new community project for building Streaming connectors that seeks to “back-pressurize” traditional Apache Camel endpoints. * An overview of Reactive Streams and what it will look like in JDK 9, and the Akka Streams API implementation for Java and Scala. * Introduction to Alpakka, a modern, Reactive version of Apache Camel, and its growing community of Streams connectors (e.g. Akka Streams Kafka, MQTT, AMQP, Streaming HTTP/TCP/FileIO and more). * How Akka Streams and Akka HTTP work with Websockets, HTTP and TCP, with examples in both in Java and Scala.

Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Lightbend

Reactive java - Reactive Programming + RxJava

NexThoughts Technologies

Full Stack Reactive In Practice

Lightbend

Lightbend Fast Data Platform

Lightbend

Multi-threading in the modern era: Vertx Akka and Quasar

Gal Marder

What’s expected in Spring 5

Gal Marder

Organizations like Starbucks, HPE, and PayPal (see our customers) have selected the Akka toolkit for their enterprise scale distributed applications; and when it comes to squeezing out the best possible performance, the secret is using two particular modules in tandem: Akka Cluster and Akka Streams. In this webinar by Nolan Grace, Senior Solution Architect at Lightbend, we look at these two Akka modules and discuss the features that will push your application architecture to the next tier of performance. For the full blog post, including the video, visit: https://www.lightbend.com/blog/akka-at-enterprise-scale-performance-tuning-distributed-applications

Akka at Enterprise Scale: Performance Tuning Distributed Applications

Lightbend

Spring cloud Service-Discovery

Nikhil Hiremath

Akka for big data developers

Taras Fedorov

Microservices. Streaming data. Event Sourcing and CQRS. Concurrency, routing, self-healing, persistence, clustering… You get the picture. The Akka toolkit makes all of this simple for Java and Scala developers at Amazon, LinkedIn, Starbucks, Verizon and others. So how does Akka provide all these features out of the box? Join Hugh McKee, Akka expert and Developer Advocate at Lightbend, on an illustrated journey that goes deep into how Akka works–from individual Akka actors to fully distributed clusters across multiple datacenters.

Akka A to Z: A Guide To The Industry’s Best Toolkit for Fast Data and Microse...

Lightbend

In this special guest webinar with Mirco Dotta, co-founder of Triplequote LLC (the creators of Hydra), we take a deeper look into what affects Scala compilation speed, why a combination of language features, external libraries, and type annotations make compilation times generally unpredictable, and what you can do to speed it up by orders of magnitude. We’ll go through: * Understanding some of the most common bottlenecks in Scala builds. * Effective use of type class auto-derivation for cutting compilation times. * What are some average compilation speeds, and how to know if you have a productivity blocker.

Making Scala Faster: 3 Expert Tips For Busy Development Teams

Lightbend

Do's and don'ts when deploying akka in production

jglobal

What’s expected in Java 9

Gal Marder

We recently learned about “Fault Tree Analysis” and decided to apply the technique to bulletproof our Apache Kafka deployments. In this talk, learn about fault tree analysis and what you should focus on to make your Apache Kafka clusters resilient. This talk should provide a framework for answers the following common questions a Kafka operator or user might have: What guarantees can I promise my users? What should my replication factor? What should the ISR setting be? Should I use RAID or not? Should I use external storage such as EBS or local disks?

Bulletproof Kafka with Fault Tree Analysis (Andrey Falko, Lyft) Kafka Summit ...

confluent

One of the most frequent questions that we get asked at Lightbend is “what’s the difference between Akka Streams and Kafka Streams?” After all, there is only a 1 letter difference between these two technologies, so how different could they be? Well, as we see in this presentation, they are actually quite different. Both tools are part of the streaming Fast Data stack, but were created with entirely different technological approaches in mind. For example, While Akka Streams emerged as a dataflow-centric abstraction for the Akka Actor model, designed for general-purpose microservices, very low-latency event processing, and supports a wider class of application problems and third-party integrations via Alpakka, Kafka Streams is purpose-built for reading data from Kafka topics, processing it, and writing the results to new topics in a Kafka-centric way. In this webinar by Dr. Dean Wampler, VP of Fast Data Engineering at Lightbend, we will: * Discuss the strengths and weaknesses of Kafka Streams and Akka Streams for particular design needs in data-centric microservices * Contrast them with Spark Streaming and Flink, which provide richer analytics over potentially huge data sets * Help you map these streaming engines to your specific use cases, so you confidently pick the right ones for your jobs

Streaming Microservices With Akka Streams And Kafka Streams

Lightbend

Apache Kafka–part of Lightbend Fast Data Platform–is a distributed streaming platform that is best suited to run close to the metal on dedicated machines in statically defined clusters. For most enterprises, however, these fixed clusters are quickly becoming extinct in favor of mixed-use clusters that take advantage of all infrastructure resources available. In this webinar by Sean Glover, Fast Data Engineer at Lightbend, we will review leading Kafka implementations on DC/OS and Kubernetes to see how they reliably run Kafka in container orchestrated clusters and reduce the overhead for a number of common operational tasks with standard cluster resource manager features. You will learn specifically about concerns like: * The need for greater operational knowhow to do common tasks with Kafka in static clusters, such as applying broker configuration updates, upgrading to a new version, and adding or decommissioning brokers. * The best way to provide resources to stateful technologies while in a mixed-use cluster, noting the importance of disk space as one of Kafka’s most important resource requirements. * How to address the particular needs of stateful services in a model that natively favors stateless, transient services.

Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OS

Lightbend

Compared to a traditional analysis-centric data hub, today's data platforms need to fulfill many different use cases. The need for real-time, transport-agnostic data protocols have become a crucial feature shared across many different use cases. During this talk, we will discuss our approach to bootstrapping and bounded context subscription, leveraging a mix of open source technologies and home-grown services aimed at providing a full end-to-end solution. We will demonstrate and discuss our use of Kafka, Spark Streaming and Akka to orchestrate a unified data transfer protocol that frees developers from having to listen to and process events within their bounded contexts. More specifically: - Leveraging Kafka as the source of truth - Topic serialization formats, Avro, and retention rules - Using Kafka's distributed commit logs to produce durable datasets - Log compaction and its role in service bootstrapping - Ingestion at scale: consuming data in different formats across different teams at different latencies - Using Spark Streaming and Akka to perform near real-time replication protocols

Bootstrapping Microservices with Kafka, Akka and Spark

Alex Silva

Akka and Kubernetes: Reactive From Code To Cloud

Lightbend

Have you ever imagined what it would be like to build a massively scalable streaming application on Kafka, the challenges, the patterns and the thought process involved? How much of the application can be reused? What patterns will you discover? How does it all fit together? Depending upon your use case and business, this can mean many things. Starting out with a data pipeline is one thing, but evolving into a company-wide real-time application that is business critical and entirely dependent upon a streaming platform is a giant leap. Large-scale streaming applications are also called event streaming applications. They are classically different from other data systems; event streaming applications are viewed as a series of interconnected streams that are topologically defined using stream processors; they hold state that models your use case as events. Almost like a deconstructed realtime database. In this talk, I step through the origins of event streaming systems, understanding how they are developed from raw events to evolve into something that can be adopted at an organizational scale. I start with event-first thinking, Domain Driven Design to build data models that work with the fundamentals of Streams, Kafka Streams, KSQL and Serverless (FaaS). Building upon this, I explain how to build common business functionality by stepping through patterns for Scalable payment processing Run it on rails: Instrumentation and monitoring Control flow patterns (start, stop, pause) Finally, all of these concepts are combined in a solution architecture that can be used at enterprise scale. I will introduce enterprise patterns such as events-as-a-backbone, events as APIs and methods for governance and self-service. You will leave talk with an understanding of how to model events with event-first thinking, how to work towards reusable streaming patterns and most importantly, how it all fits together at scale.

Kafka summit SF 2019 - the art of the event-streaming app

Neil Avery

Tendances (20)

Build Real-Time Streaming ETL Pipelines With Akka Streams, Alpakka And Apache...

Exploring Reactive Integrations With Akka Streams, Alpakka And Apache Kafka

Reactive java - Reactive Programming + RxJava

Full Stack Reactive In Practice

Lightbend Fast Data Platform

Multi-threading in the modern era: Vertx Akka and Quasar

What’s expected in Spring 5

Akka at Enterprise Scale: Performance Tuning Distributed Applications

Spring cloud Service-Discovery

Akka for big data developers

Akka A to Z: A Guide To The Industry’s Best Toolkit for Fast Data and Microse...

Making Scala Faster: 3 Expert Tips For Busy Development Teams

Do's and don'ts when deploying akka in production

What’s expected in Java 9

Bulletproof Kafka with Fault Tree Analysis (Andrey Falko, Lyft) Kafka Summit ...

Streaming Microservices With Akka Streams And Kafka Streams

Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OS

Bootstrapping Microservices with Kafka, Akka and Spark

Akka and Kubernetes: Reactive From Code To Cloud

Kafka summit SF 2019 - the art of the event-streaming app

Similaire à Why Actor-Based Systems Are The Best For Microservices

Spring Boot Microservices vs Akka Actor Cluster

OpenCredo

The Actor model has been around for a while, but only the Reactive revolution is bringing it to trend. Find out how your application can benefit from Actors to achieve Resilience - the ability to spring back into shape from a failure state. Akka is a toolkit that brings Actors to the JVM - think Java or Scala - and that leverages on them to help you build concurrent, distributed and resilient applications.

Beyond fault tolerance with actor programming - Fabio Tiriticco - Codemotion ...

Codemotion

Actor Programming is a software building approach that lets you can go beyond fault tolerance and achieve Resilience, which is the capacity of a system to self-heal and spring back into a fresh shape. First I'll introduce the difference between Reactive Programming and Reactive Systems, and then we'll go over a couple of implementation examples using Scala and Akka. The coupled GitHub repository with the code is here: https://github.com/ticofab/ActorDemo

Beyond Fault Tolerance with Actor Programming

Fabio Tiriticco

Building Asynchronous Applications

Johan Edstrom

RestMS Introduction

pieterh

Asynchronous Service Server

guestbba9241

Орхан Гасимов: "Reactive Applications in Java with Akka"

Anna Shymchenko

Erlang, Elixir, Scala con Akka, solo per fare gli esempi piu' famosi, sono implementazioni di un modello matematico, formulato nel 1973, ma del tutto attuale: l'Actor Model. Utilizzato per sviluppare soluzioni "concorrenti e distribuite", il concetto di "attore" si sposa alla perfezione con il mondo del "cloud". In questa sessione, dopo una breve introduzione teorica sull'Actor Model, analizzeremo le possibilità messe a disposizione dell'ecosistema .net

Un actor (model) per amico - Alessandro Melchiori - Codemotion Milan 2016

Codemotion

High powered messaging with RabbitMQ

James Carr

The Art of Message Queues - TEKX

Mike Willbanks

Developing Actors in Azure with .net

Marco Parenzan

AKKA and Scala @ Inneractive

Gal Aviv

Microservices in a Streaming World

Hans Jespersen

The API-Application Semantic Gap

3scale

Concurrent processing and distributed computing

Rahul Ramteke

NoSQL afternoon in Japan Kumofs & MessagePack

Sadayuki Furuhashi

NoSQL afternoon in Japan kumofs & MessagePack

Sadayuki Furuhashi

"This is a tale of two streams when the pandemic hit and how we changed with the times and built a revolutionary recruitment platform for “going into work”. We engaged employers, recruiters and job seekers from industrial, healthcare, retail, hospital and facilities management sectors by building a unique platform where the job seeker has full control to pick their schedule, pay rate and what meets their preferences. Our goals are to give job seekers and employers a platform that thrives on simplicity, transparency and low costs. The Flexer stands today with full control of their time at the edge of opportunities to thrive on. This presentation will go into the details of how we are tearing down a monolithic platform piece by piece and building a robust architecture, - Routing events between two platforms - Many sources and, - Consumed by downstream several applications We will discuss the caveats and bugs we learned when we worked with schema registry and evolution of schemas. We will highlight improvements we gained from automation and observability with Datadog integration for Confluent Cloud. If you’re in discussions surrounding event driven systems at your organization then this talk is for you. Join Ronak and me for this talk and let’s have a discussion."

Indeed Flex: The Story of a Revolutionary Recruitment Platform

HostedbyConfluent

Clustersoftware

Roger Johansson

In this meetup, Kobi Salant - Data Platform Technical Lead & Vladi Feigin - Data System Architect, both from Liveperson will talk about : Making scale a non-issue for real-time Data apps. Have you ever tried to build a system processing in real-time hundreds of thousands events per second and servicing more than 1M concurrent visitors? We're going to talk about the LivePerson real-time stream processing solution doing exactly that. Learn how we empower digital call centers with insights for their critical decision making processes and never-ending efficiency goals.

Liveperson DLD 2015

LivePerson

Similaire à Why Actor-Based Systems Are The Best For Microservices (20)

Spring Boot Microservices vs Akka Actor Cluster

Beyond fault tolerance with actor programming - Fabio Tiriticco - Codemotion ...

Beyond Fault Tolerance with Actor Programming

Building Asynchronous Applications

RestMS Introduction

Asynchronous Service Server

Орхан Гасимов: "Reactive Applications in Java with Akka"

Un actor (model) per amico - Alessandro Melchiori - Codemotion Milan 2016

High powered messaging with RabbitMQ

The Art of Message Queues - TEKX

Developing Actors in Azure with .net

AKKA and Scala @ Inneractive

Microservices in a Streaming World

The API-Application Semantic Gap

Concurrent processing and distributed computing

NoSQL afternoon in Japan Kumofs & MessagePack

NoSQL afternoon in Japan kumofs & MessagePack

Indeed Flex: The Story of a Revolutionary Recruitment Platform

Clustersoftware

Liveperson DLD 2015

Plus de Yaroslav Tkachenko

Change Data Capture (CDC) is a popular technique for extracting data from databases in realtime. However, many CDC deployments are static: e.g. a single connector is configured to ingest data for one or several tables. At Goldsky, we needed a way to configure CDC for a large Postgres database dynamically: the list of tables to ingest is driven by customer-facing features and is constantly changing. We started using Flink CDC connectors built on top of the Debezium project, but we immediately faced many challenges caused mainly by the lack of incremental snapshotting. But even after implementing incremental snapshotting ourselves, we still faced an issue around using replication slots in Postgres: we couldn't use a single connector to ingest all tables (it's just too much data), and we couldn't create a new connector for every new set of tables (we'd quickly run out of replication slots). So we needed to find a way to maintain a fixed number of replication slots for a dynamic list of tables. In the end, we chose a consistent hashing algorithm to distribute the list of tables across multiple Flink jobs. The jobs also required some customizations to support the incremental snapshotting semantics from Flink CDC. We learned a lot about Debezium, Flink CDC and Postgres replication, and we're excited to share our learnings with the community!

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Yaroslav Tkachenko

SQL is the lingua franca of data analysis, but should we use it more as data engineers? Modern tools like dbt make it easier to express transformations in SQL, but streaming is more complicated than batch. Streaming pipelines usually require higher SLAs and many CI/CD and observability practices, so data engineers prefer to use familiar languages like Python, Java and Scala along with many useful frameworks and libraries. Can SQL replace that? I was very skeptical when I first heard the idea of using SQL for writing somewhat complex stream-processing data application a few years ago. How do you unit test it? How do you version it? Over the years, Spark SQL streaming, Flink SQL, ksqlDB and similar tools have matured, now they easily support complex stateful transformations. However, developer experience is still questionable: it's easy to write a SQL statement, but how do you maintain it over the years as a long-running application? In this presentation, I hope to share the discoveries I made over the years in this area, as well as working practices and patterns I've seen.

Streaming SQL for Data Engineers: The Next Big Thing?

Yaroslav Tkachenko

At the beginning of 2021, Shopify Data Platform decided to adopt Apache Flink to enable modern stateful stream-processing. Shopify had a lot of experience with other streaming technologies, but Flink was a great fit due to its state management primitives. After about six months, Shopify now has a flourishing ecosystem of tools, tens of prototypes from many teams across the company and a few large use-cases in production. Yaroslav will share a story about not just building a single data pipeline but building a sustainable ecosystem. You can learn about how they planned their platform roadmap, the tools and libraries Shopify built, the decision to fork Flink, and how Shopify partnered with other teams and drove the adoption of streaming at the company.

Apache Flink Adoption at Shopify

Yaroslav Tkachenko

State is an essential part of the modern streaming pipelines: it enables a variety of foundational capabilities like windowing, aggregation, enrichment, etc. But usually, the state is either transient, so we only keep it until the window is closed, or it's fairly small and doesn't grow much. But what if we treat the state differently? The keyed state in Flink can be scaled vertically and horizontally, it's reliable and fault-tolerant... so is scaling a stateful Flink application that different from scaling any data store like Kafka or MySQL? At Shopify, we've worked on a massive analytical data pipeline that's needed to support complex streaming joins and correctly handle arbitrarily late-arriving data. We came up with an idea to never clear state and support joins this way. We've made a successful proof of concept, ingested all historical transactional Shopify data and ended up storing more than 10 TB of Flink state. In the end, it allowed us to achieve 100% data correctness.

Storing State Forever: Why It Can Be Good For Your Analytics

Yaroslav Tkachenko

Lambda Architecture has been a common way to build data pipelines for a long time, despite difficulties in maintaining two complex systems. An alternative, Kappa Architecture, was proposed in 2014, but many companies are still reluctant to switch to Kappa. And there is a reason for that: even though Kappa generally provides a simpler design and similar or lower latency, there are a lot of practical challenges in areas like exactly-once delivery, late-arriving data, historical backfill and reprocessing. In this talk, I want to show how you can solve those challenges by embracing Apache Kafka as a foundation of your data pipeline and leveraging modern stream-processing frameworks like Apache Flink.

It's Time To Stop Using Lambda Architecture

Yaroslav Tkachenko

Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streaming

Yaroslav Tkachenko

In the last two years Apache Kafka rapidly introduced new versions, going from 0.10.x to 2.x. It can be hard to keep up with all the updates and a lot of companies still run 0.10.x clusters (or even older ones). Join this session to learn new exciting features in Kafka introduced in 0.11, 1.0, 1.1 and 2.0 versions including, but not limited to, the new protocol and message headers, transactional support and exactly-only delivery semantics, as well as controller changes that make it possible to shutdown even large clusters in seconds.

Apache Kafka: New Features That You Might Not Know About

Yaroslav Tkachenko

What can be easier than building a data pipeline nowadays? You add a few Apache Kafka clusters, some way to ingest data (probably over HTTP), design a way to route your data streams, add a few stream processors and consumers, integrate with a data warehouse... wait, it does start to look like A LOT of things, doesn't it? And you probably want to make it highly scalable and available in the end, correct? We've been developing a data pipeline in Demonware/Activision for a while. We learned how to scale it not only in terms of messages/sec it can handle, but also in terms of supporting more games and more use-cases. In this presentation you'll hear about the lessons we learned, including (but not limited to): - Message schemas - Apache Kafka organization and tuning - Topics naming conventions, structure and routing - Reliable and scalable producers and ingestion layer - Stream processing

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lesson...

Yaroslav Tkachenko

Designing Scalable and Extendable Data Pipeline for Call Of Duty Games

Yaroslav Tkachenko

10 tips for making Bash a sane programming language

Yaroslav Tkachenko

With more and more companies adopting microservices and service-oriented architectures, it becomes clear that the HTTP/RPC synchronous communication (while great) is not always the best option for every use case. In this presentation, I discuss two approaches to an asynchronous event-based architecture. The first is a "classic" style protocol (Python services driven by callbacks with decorators communicating using a messaging layer) that we've been implementing at Demonware (Activision) for Call of Duty back-end services. The second is an actor-based approach (Scala/Akka based microservices communicating using a messaging layer and a centralized router) in place at Bench Accounting. Both systems, while event based, take different approaches to building asynchronous, reactive applications. This talk explores the benefits, challenges, and lessons learned architecting both Actor and Non-Actor systems.

Actors or Not: Async Event Architectures

Yaroslav Tkachenko

Stream processing is getting more & more important in our data-centric systems. In the world of Big Data, batch processing is not enough anymore - everyone needs interactive, real-time analytics for making critical business decisions, as well as providing great features to the customers. There are many stream processing frameworks available nowadays, but the cost of provisioning infrastructure and maintaining distributed computations is usually very high. Sometimes you just have to satisfy some specific requirements, like using HDFS or YARN. Apache Kafka is de facto a standard for building data pipelines. Kafka Streams is a lightweight library (available since 0.10) that uses powerful Kafka abstractions internally and doesn't require any complex setup or special infrastructure - you just deploy it like any other regular application. In this session I want to talk about the goals behind stream processing, basic techniques and some best practices. Then I'm going to explain main fundamental concepts behind Kafka and explore Kafka Streams syntax and streaming features. By the end of the session you'll be able to write stream processing applications in your domain, especially if you already use Kafka as your data pipeline.

Kafka Streams: the easiest way to start with stream processing

Yaroslav Tkachenko

Data Pipeline team at Demonware (Activision) has to deal with routing large amounts of data from various sources to many destinations every day. Our team always wanted to be able to query processed data for debugging and analytical purposes, but creating large data warehouses was never our priority, since it usually happens downstream. AWS Athena is completely serverless query service that doesn't require any infrastructure setup or complex provisioning. We just needed to save some of our data streams to AWS S3 and define a schema. Just a few simple steps, but in the end we were able to write complex SQL queries against gigabytes of data and get results in seconds. In this presentation I want to show multiple ways to stream your data to AWS S3, explain some underlying tech, show how to define a schema and finally share some of the best practices we applied.

Querying Data Pipeline with AWS Athena

Yaroslav Tkachenko

Быстрая и безболезненная разработка клиентской части веб-приложений

Yaroslav Tkachenko

Plus de Yaroslav Tkachenko (14)

Dynamic Change Data Capture with Flink CDC and Consistent Hashing

Streaming SQL for Data Engineers: The Next Big Thing?

Apache Flink Adoption at Shopify

Storing State Forever: Why It Can Be Good For Your Analytics

It's Time To Stop Using Lambda Architecture

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streaming

Apache Kafka: New Features That You Might Not Know About

Building Scalable and Extendable Data Pipeline for Call of Duty Games: Lesson...

Designing Scalable and Extendable Data Pipeline for Call Of Duty Games

10 tips for making Bash a sane programming language

Actors or Not: Async Event Architectures

Kafka Streams: the easiest way to start with stream processing

Querying Data Pipeline with AWS Athena

Быстрая и безболезненная разработка клиентской части веб-приложений

Dernier

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal.

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

masabamasaba

%in Midrand+277-882-255-28 abortion pills for sale in midrand

masabamasaba

WSO2CON 2024 Slides - Unlocking Value with AI

WSO2

Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic near me by a powerful astrologer and spell caster in Atlanta. Bring back your lover with Asaf's voodoo-love witchcraft. Psychic Reading | Astrologer | Spell Caster | Obsession Spells | Black Magic | Witchcraft | Protection spell | wiccan spells. Black magic expert and voodoo love spells that work overnight to retrieve that love | lost love spell caster with powerful love psychic reading. Best astrologer & psychic in Sandy Springs, GA to renew your relationship & make your relationship stronger. love spells to bring back the feelings of love for ex-lovers. Increase the intimacy, affection & love between you and your lover using voodoo relationship obsession spells in USA. Money spells, easy love spells with just words, think of me spell, powerful love spell, spells of love, spells that work, love potion to attract a man, easy love spells with just words, pink candle prayer, white magic spells, call me spell, manifestation spell, gay love spells, Commitment spells, business spells and, how to bring back lost love in a relationship, Witchcraft love spells that work immediately to increase love & intimacy in your relationship. Attraction love spells to attract someone, stop a divorce, prevent a breakup & get your ex back. When using love binding spells, there are several things you should know, particularly how to protect yourself from negative energies and how to use the powers of incantations for the good of all people involved. When the focus is love, the results are truly magical. It’s important to remember love is not manipulative, it is not forceful, and it does not bend another to its will. Love is free-flowing, accepting, kind, and generous. For your love spell to work as intended, you must have good intentions in your heart. Below, we share the top five love spells you can use today to shift the energies in your life and create a future filled with love and fulfilment. Obsession spells to Get Your Ex-Lover Back. All women want one thing the most in life to be love and love in return – not much to ask. A lady wants a good man who loves you unconditionally and loyally. You want a man who is honest, hardworking, and loyal – not a complainer or a weakling or a self-centred man. You are a strong, independent, sensual, caring, loving woman. And you don’t think it’s asking too much to want to be with a loving, intelligent man with a good sense of humour and daring, an upright and confident man – who knows who he is. No one wants a chauvinistic and macho man, but you do want someone who will be willing to protect and care for you. Who loves you for you and not some kind of imaginary. You have no doubt that when the right guy appears on your doorstep, you’ll never let him go. But sometimes a man doesn’t realize he has that good woman.

Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...

chiefasafspells

Announcing Codolex 2.0 from GDK Software

Jim McKeeth

WSO2CON 2024 Slides - Open Source to SaaS

WSO2

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

Philip Schwarz

%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...

masabamasaba

WSO2CON2024 - It's time to go Platformless

WSO2

WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...

WSO2

WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...

WSO2

WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...

WSO2

WSO2CON 2024 - How to Run a Security Program

WSO2

%in Soweto+277-882-255-28 abortion pills for sale in soweto

masabamasaba

WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...

WSO2

VTU technical seminar 8Th Sem on Scikit-learn

AmarnathKambale

tonesoftg

lanshi9

%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...

masabamasaba

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

masabamasaba

WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source

WSO2

Dernier (20)

%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...

%in Midrand+277-882-255-28 abortion pills for sale in midrand

WSO2CON 2024 Slides - Unlocking Value with AI

Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...

Announcing Codolex 2.0 from GDK Software

WSO2CON 2024 Slides - Open Source to SaaS

Direct Style Effect Systems -The Print[A] Example- A Comprehension Aid

%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...

WSO2CON2024 - It's time to go Platformless

WSO2CON 2024 - Navigating API Complexity: REST, GraphQL, gRPC, Websocket, Web...

WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...

WSO2Con2024 - From Blueprint to Brilliance: WSO2's Guide to API-First Enginee...

WSO2CON 2024 - How to Run a Security Program

%in Soweto+277-882-255-28 abortion pills for sale in soweto

WSO2CON 2024 - Cloud Native Middleware: Domain-Driven Design, Cell-Based Arch...

VTU technical seminar 8Th Sem on Scikit-learn

tonesoftg

%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...

%in tembisa+277-882-255-28 abortion pills for sale in tembisa

WSO2CON 2024 - Freedom First—Unleashing Developer Potential with Open Source

Why Actor-Based Systems Are The Best For Microservices

1. Technical Overview [Client] [Date] Why Actor-Based Systems Are The Best For Microservices Yaroslav Tkachenko Senior Software Engineer, Lead at Mobify @sap1ens

3. Monolith Microservices

4. “Easy” way – HTTP / RPC API POST /foo service-x.example.com

5. § Destination – where to send request? § Service discovery § Tight coupling § Time – expect reply right away? § Failure – always expect success? § Retries and failure handling § Back-pressure / circuit breakers

6. But wait… How about Netflix, Twitter, etc.?

7. Yes, it’s possible to build reliable systems with HTTP/RPC APIs! But: § Are you ready to invest a lot of resources § Do you really need it?

8. Reactive Revealed: Resiliency, Failures vs Errors, Isolation, Delegation and Replication by Jonas Boner

9. Surviving Micro-services by Richard Rodger

10. So, let’s try to: § Use message queues for communication § Use actors as publishers and subscribers

11. - Send … - Receive … Another way - Messaging - Send … - Receive … queue.events

12. Enterprise Service Bus?

13. Kafka Connect

14. So, why message queues now? § Not ESBs! § Lightweight pub/sub instead § Low coupling § Promotes eventual consistency § Highly scalable and available § Built-in discovery, retries, tracing, etc. § Serverless!

15. By the way, I don’t think that ALL your communication should happen through the message queues. But it’s something that you should strive for!

16. External request (HTTP)

17. External request (HTTP)Write Always asynchronous using messages Read Alternatives to blocking call: - Caching - Denormalization - Different routing

18. Example: integrations

19. “Macroservices” External APIs

20. “Macroservices” External APIs New user!

21. “Macroservices” External APIs New user! User created

22. EIP was published in 2003 and it contains 65 patterns

23. Enterprise Integration Patterns

24. Patterns from the example: § Publish – Subscribe Channel (“Broadcast”) § Durable Subscriber § Idempotent Receiver

25. So, have I convinced you that messaging patterns are great?

26. But why actors? I can just use a message queue and a bunch of consumers/producers

27. It’s all about semantics

28. An actor is a computational entity that, in response to a message it receives, can concurrently: • send a finite number of messages to other actors • create a finite number of new actors • designate the behavior to be used for the next message it receives.

29. Actors are known for concurrency, but let’s focus on message passing instead

30. Actors in Scala/Akka and Erlang

31. Actor ActorMessage

32. Actor ActorMessage queue

33. ActiveMQ message listener in Java (just kidding)

34. ActiveMQ message listener in Java (zoom-in)

35. Actor-based (Akka) message listener in Scala + Camel

36. Receiver Handler onMessage(Message) ??? Receiver ActorSender Actor Mailbox Message VS

37. I thought of objects being like biological cells and/or individual computers on a network, only able to communicate with messages... OOP to me means only messaging, local retention and protection and hiding of state-process, and extreme late- binding of all things. Alan Kay, one of the fathers of the idea of object-oriented programming

38. Actors over message handlers: • Semantically natural message processing that ”feels right” • No difference between local and remote communication • Straightforward state management • Mailbox, routing and back-pressure • Built-in failure handling (supervisors)

39. Actors in general: • Simple and high-level abstraction for distribution, concurrency and parallelism • Asynchronous, non-blocking and highly performant message-driven programming model • Very lightweight message-driven processes

40.

41. Instead of a summary: Everything is a trade-off!

42. Questions? sap1ens.com @sap1ens

Why Actor-Based Systems Are The Best For Microservices

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Why Actor-Based Systems Are The Best For Microservices

Similaire à Why Actor-Based Systems Are The Best For Microservices (20)

Plus de Yaroslav Tkachenko

Plus de Yaroslav Tkachenko (14)

Dernier

Dernier (20)

Why Actor-Based Systems Are The Best For Microservices