Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021

•Télécharger en tant que PPTX, PDF•

0 j'aime•167 vues

Apache Pulsar is used for various streaming use cases. There is a strong requirement for storing checkpoints while processing the stream in Pulsar Functions, so that in case of any interruption stream processing engine could go back to the last checkpoint. Pulsar uses Zookeeper not only for leader elections or service discovery like critical use cases but also for storing various metadata which puts unnecessary load on zookeeper which hampers mission critical use of Zookeeper. A durable key value store based off of the Apache Pulsar ecosystem addresses the above mentioned use cases nicely. This talk focuses on taking existing Apache Bookkeeper Table Service/State store implementation and taking it to production. Furthermore this talk also touches upon contributing all the features, bug fixes, tools and other improvements back to open source.

Technologie

Pulsar Virtual Summit North America 2021
Apache BookKeeper State Store:
A Durable Key-Value Store

Pulsar Virtual Summit North America 2021
Prashant Kumar
Principal Software Engineer @ Splunk
● Principal Software Developer at Splunk.
● Ex Yahoo, Verizon Media.
● Prior Experience, Key member and
contributor to Sherpa, the geographically
replicated multi tenant key value store @
Yahoo.

Pulsar Virtual Summit North America 2021
Agenda
I. Introduction to Apache Bookkeeper Statestore.
II. Why yet another KV store?
III. How does it fit into Pulsar ecosystem?
IV. Intended use case.
V. Brief architecture.
VI. Current state and production worthiness.
VII. Product roadmap and future work.

Pulsar Virtual Summit North America 2021
Apache Bookkeeper Statestore
● It’s a Key-Value store.
● It’s durable.
● It’s locally replicated.
● It’s eventually consistent.
● It’s fault tolerant.
● It’s cloud native and k8s based deployment.

Pulsar Virtual Summit North America 2021
Argh!. Yet another KV store?

Pulsar Virtual Summit North America 2021
Integral part of Apache Pulsar ecosystem
● Uses same Zookeeper deployment that Pulsar uses.
● Uses same Bookkeeper deployment that Pulsar uses.
● Uses same infrastructure for metrics, dashboards etc as
Bookkeeper
● Part of bookkeeper/stream code base.
● Existing client side integration in Apache Pulsar function
service.

Pulsar Virtual Summit North America 2021
Primary use cases
● Store and access function state and checkpoints
● A secondary metadata store for Apache Pulsar, away from
Zookeeper
● Other various KV store use cases

Pulsar Virtual Summit North America 2021
Data model

Pulsar Virtual Summit North America 2021
High level serving architecture (Bird view)

Pulsar Virtual Summit North America 2021
High level Datastore architecture

Pulsar Virtual Summit North America 2021
Benchmarking
● Benchmarking with YCSB
● Setup
○ YCSB Thread count = 40
○ # k8s pods = 3
○ cpuRequest = 8
○ cpuLimit = 16
○ memoryRequest = 24Gi
○ memoryLimit = 24Gi
● Read output
○ Throughput = 22557.7 Ops/S
○ Average latency = 1.699 ms
○ 99%tile latency = 5.323 ms
● Write output
○ Throughput = 15256.16 Ops/S
○ Average latency = 8.820 ms
○ 99%tile = 27.071 ms

Pulsar Virtual Summit North America 2021
Production readiness
● It has already been in production for last few
months
● It’s a k8s based deployment
● Sustained production traffic
○ Read throughput 240 Ops/S
○ Write throughput 90 Ops/S

Pulsar Virtual Summit North America 2021
Product roadmap
● Contribute internal changes back to open source
● Storage hardening
○ Pick up the data litter actively and reactively
○ Clear obsolete transaction log
● Operability
○ Improved and granular monitoring and alerting.
● Availability
○ Implementation of replica.
○ Serving read traffic from a replica.
○ Elevation of replica to be primary when primary fails
● Scaling and load balancing
○ Splitting a shard
○ Moving shards across cluster

Pulsar Virtual Summit North America 2021
References
● Statestore Repo:
https://github.com/apache/bookkeeper/tree/master/stream
● Distributedlog Repo:
https://github.com/apache/bookkeeper/tree/master/stream/distri
butedlog
● Pulsar Function - Statestore integration:
https://github.com/apache/pulsar/blob/master/pulsar-
functions/worker/src/main/java/org/apache/pulsar/functions/wor
ker/PulsarWorkerService.java#L420

Contenu connexe

Tendances

In order to leverage the best performance characters of your data or stream backend, it is important to understand the nitty gritty details of how your backend store and compute works, how data is stored, how is it indexed and how the read path is. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand. With this underlying philosophy, in this slide deck, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code! The focus will be more on storage backend so as to not keep this tailored to pulsar specifically but to be able to apply it different data stores or streams.

Apache Con 2021 : Apache Bookkeeper Key Value Store and use cases

Shivji Kumar Jha

Kafka presentation

Mohammed Fazuluddin

Why is Kafka so fast? Why is Kafka so popular? Why Kafka? This slide deck is a tutorial for the Kafka streaming platform. This slide deck covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example to demonstrate failover of brokers as well as consumers. Then it goes through some simple Java client examples for a Kafka Producer and a Kafka Consumer. We have also expanded on the Kafka design section and added references. The tutorial covers Avro and the Schema Registry as well as advance Kafka Producers.

Kafka Tutorial - Introduction to Apache Kafka (Part 1)

Jean-Paul Azar

Introduction to Apache Kafka

AIMDek Technologies

Multi cluster, multitenant and hierarchical kafka messaging service slideshare

Allen (Xiaozhong) Wang

Apache pulsar - storage architecture

Matteo Merli

Reliability Guarantees for Apache Kafka

confluent

Fundamentals of Apache Kafka

Chhavi Parasher

Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream. In this talk, we will introduce Loki, its architecture and the design trade-offs in an approachable way. We’ll both cover Loki and Promtail, the agent used to scrape local logs to push to Loki, including the Prometheus-style service discovery used to dynamically discover logs and attach metadata from applications running in a Kubernetes cluster. Finally, we’ll show how to query logs with Grafana using LogQL - the Loki query language - and the latest Grafana features to easily build dashboards mixing metrics and logs.

Grafana Loki: like Prometheus, but for Logs

Marco Pracucci

Deep Dive into Building Streaming Applications with Apache Pulsar

Timothy Spann

Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications. The talk will cover a general overview of the project and technology, with some use cases, and a demo.

Introduction and Overview of Apache Kafka, TriHUG July 23, 2013

mumrah

Apache Kafka - Martin Podval

Martin Podval

High Availability and Disaster Recovery in PostgreSQL - EQUNIX

Julyanto SUTANDANG

Apache Kafka Introduction

Amita Mirajkar

Apache Pulsar Development 101 with Python

Timothy Spann

MySQL/MariaDB Proxy Software Test

I Goo Lee

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Autoscaling Flink with Reactive Mode

Flink Forward

Kafka Streams is a new stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a natural DSL for writing stream processing applications. As such it is the most convenient yet scalable option to analyze, transform, or otherwise process data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Spark Streaming or Storm, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka.

Introduction to Kafka Streams

Guozhang Wang

Introduction to Apache Kafka

Jeff Holoman

Jay Kreps is a Principal Staff Engineer at LinkedIn where he is the lead architect for online data infrastructure. He is among the original authors of several open source projects including a distributed key-value store called Project Voldemort, a messaging system called Kafka, and a stream processing system called Samza. This talk gives an introduction to Apache Kafka, a distributed messaging system. It will cover both how Kafka works, as well as how it is used at LinkedIn for log aggregation, messaging, ETL, and real-time stream processing.

Apache Kafka at LinkedIn

Discover Pinterest

Tendances (20)

Apache Con 2021 : Apache Bookkeeper Key Value Store and use cases

Kafka presentation

Kafka Tutorial - Introduction to Apache Kafka (Part 1)

Introduction to Apache Kafka

Multi cluster, multitenant and hierarchical kafka messaging service slideshare

Apache pulsar - storage architecture

Reliability Guarantees for Apache Kafka

Fundamentals of Apache Kafka

Grafana Loki: like Prometheus, but for Logs

Deep Dive into Building Streaming Applications with Apache Pulsar

Introduction and Overview of Apache Kafka, TriHUG July 23, 2013

Apache Kafka - Martin Podval

High Availability and Disaster Recovery in PostgreSQL - EQUNIX

Apache Kafka Introduction

Apache Pulsar Development 101 with Python

MySQL/MariaDB Proxy Software Test

Autoscaling Flink with Reactive Mode

Introduction to Kafka Streams

Introduction to Apache Kafka

Apache Kafka at LinkedIn

Similaire à Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021

Apache Pulsar is the next-generation cloud-native distributed messaging and streaming platform. It has high performance, extreme scalability, and lots of great features. However, only with expertise could administrators deploy production-grade Pulsar clusters and manage their lifecycles properly. StreamNative embedded this expertise into a set of operators to deploy and manage lifecycles of pulsar clusters. In this session, I will share how these operators work and how to use them to deploy and manage pulsar clusters.

Manage Pulsar Cluster Lifecycles with Kubernetes Operators - Pulsar Summit NA...

StreamNative

ABSTRACT OF THE TALK 6 months have passed since our last DoK webinar about benchmarking PostgreSQL workloads in a Kubernetes environment. In the meantime, many things have happened at EDB, and we’re happy to share what we’ve learned in this timeframe. We’ll use cnp-bench and cnp-sandbox to help us describe some of the challenges we might face when running PostgreSQL workloads, how to spot them, and what actions to take to make your databases healthier and more longeve. cnp-bench is a collection of Helm charts that help run storage and database benchmarks, using popular open source tools like fio, pgbench, and HammerDB. cnp-sandbox is a Helm chart that sets up a Prometheus/Grafana stack, including basic metrics and dashboards for Cloud Native PostgreSQL, the Kubernetes operator developed by EDB. Both cnp-sandbox and cnp-bench are open source and recommended for development, testing, and pre-production environments only. BIO A long time open-source programmer and entrepreneur, Gabriele has a degree in Statistics from the University of Florence. After having consistently contributed to the growth of 2ndQuadrant and its members through nurturing a lean and devops culture, he is now leading the Cloud Native initiative at EDB. Gabriele lives in Prato, a small but vibrant city located in the northern part of Tuscany, Italy - famous for having hosted the first European PostgreSQL conferences. His second home is Melbourne, Australia, where he studied at Monash University and worked in the ICT sector. He loves playing the Blues with his Fender Stratocaster, but his major passions are called Elisabeth and Charlotte! KEY TAKE-AWAYS FROM THE TALK - A methodology for benchmarking a PostgreSQL database in Kubernetes - Open source set of tools for benchmarking a PostgreSQL database in Kubernetes - Reasons why benchmarking both the storage and the database is important https://github.com/EnterpriseDB/cnp-sandbox https://github.com/EnterpriseDB/cnp-bench

Benchmarking for postgresql workloads in kubernetes

DoKC

In this talk AWS’ Ken Krupa, Head of Specialized Solutions Architecture, will describe the architecture and capabilities of two new AWS EC2 instance types perfect for data-intensive storage and IO-heavy workloads like ScyllaDB: the Intel-based I4i and the Graviton2-based I4g series. The Intel Xeon Ice Lake-based I4i series provides unparalleled raw horsepower for your most demanding workloads. Meanwhile, the Graviton2-powered I4g instances provide lower cost per storage on a power-efficient platform to deploy your cloud-native applications. Ken will also describe the AWS Nitro SSD, a new form of high-speed NVMe storage with a Flash Translation Layer built with Nitro controllers, which powers both of these instance families. ScyllaDB VP of Product Tzach Livyatan will then share benchmarking results showing how ScyllaDB behaves under load on these two instance types, providing maximum system utility and efficiency. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Scylla Summit 2022: New AWS Instances Perfect for ScyllaDB

ScyllaDB

Infinitic is a free-to-use, source-available event-driven workflow engine built on top of Pulsar. Implementing a "workflow as code" pattern, Infinitic lets you write your distributed applications as if you were coding for a single infallible machine. It proposes a fault-tolerant stateful programming model that removes most of the complexities of building and orchestrating a distributed system. Infinitic makes intensive use of Pulsar features: functions, transactions, schema registry, subscription types, sinks, connectors. During this talk, I will showcase Infinitic capabilities and explain its architecture based on Pulsar.

Infinitic: Building a Workflow Engine on Top of Pulsar - Pulsar Summit NA 2021

StreamNative

Ceph Community Talk on High-Performance Solid Sate Ceph

Ceph Community

In-memory No SQL- GIDS2014

Hazelcast

When low latency (P99) and high performance are core requirements, what NoSQL database attributes should you consider, and what tradeoffs are key? While we live in a world of multi-CPU, multi-core servers capable of storing tens of terabytes of data, if your database isn’t architected to take advantage of this, you’re being penalized on performance or cost. Join this webinar to learn about the critical elements for a high-performance, low-latency NoSQL database. ScyllaDB’s engineers will discuss how they addressed core database performance challenges, including the pros and cons of each, and provide a detailed explanation of the architectural principles they applied to achieve their performance objectives. We’ll take a deep dive into the strategies applied to: Achieve precise control over I/O and compute-intensive workloads Avoid locks and contention on the CPU level Bypass kernel bottlenecks Squeeze the most out of modern multi-core hardware Satisfy SLAs while maintaining system stability

Critical Attributes for a High-Performance, Low-Latency Database

ScyllaDB

The Netdata Agent is free, open source single-node monitoring software. Netdata Cloud is a free, closed source, software-as-a-service that brings together metadata from endpoints running the Netdata Agent, giving a complete view of the health and performance of an infrastructure. All the metrics remain on the Netdata Agent, making Netdata Cloud the focal point of a distributed, infinitely scalable, low cost solution.  The heart of Netdata Cloud is Pulsar. Almost every message coming from and going to the open source agents passes through Pulsar. Pulsar's infinite number of topics has given us the flexibility we needed and in some cases, every single Netdata Agent has its own unique Pulsar topic.  A single message from an agent or from a service that processes a front end request can trigger several other Pulsar messages, as we also use Pulsar for communication between microservices (using a CQRS pattern with shared subscriptions for scalability). The reliable persistence of messages has allowed us to replay old events to rebuild old and build new materialized views and debug specific production issues. It's also what will enable us to implement an event sourcing pattern, for a new set of features we want to introduce shortly.  We have had a few issues with a specific client and our shared subscriptions that we're working on resolving, but overall Pulsar has proven to be one of the most reliable parts of our infrastructure and we decided to proceed with a managed services agreement.

How Pulsar Enables Netdata to Offer Unlimited Infrastructure Monitoring for F...

StreamNative

Oracle big data appliance and solutions

solarisyougood

SkySQL implements a groundbreaking, state-of-the-art architecture based on Kubernetes and ServiceNow, and with a strong emphasis on cloud security – using compartmentalization and indirect access to secure and protect customer databases. In this session, we’ll walk through the architecture of SkySQL and discuss how MariaDB leverages an advanced Kubernetes operator and powerful ServiceNow configuration/workflow management to deploy and manage databases on cloud infrastructure.

The architecture of SkySQL

MariaDB plc

Architectural caching patterns for kubernetes

Rafał Leszko

Pulsar Function is a succinct computing abstraction Apache Pulsar provides users to express simple ETL and streaming tasks. The simplicity comes in two folds: Simple Interface and Simple Deployment. As it has been adopted, we realized that the native support of organizing multiple functions into integrity will be very beneficial. With such support, people can express and manage multi-stage jobs easily. In addition, this support also provides the possibility of higher-level abstraction DSL to further simplify the job composition. We call this new feature -- Function Mesh. This talk aims to provide a thorough walkthrough of this new Function Mesh Feature, including its design, implementation, use cases, and examples, to help people seeking simple streaming solutions understand this newly created powerful tool in Apache Pulsar.

Function Mesh: Complex Streaming Jobs Made Simple - Pulsar Summit NA 2021

StreamNative

Learn from HomeAway Hadoop Development and Operations Best Practices

Driven Inc.

[DevConf.US 2019]Quarkus Brings Serverless to Java Developers

Daniel Oh

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...

Nitin Kumar

Ohio Devfest - Visual Analysis with GCP

Wesley Workman

YARN

Alex Moundalexis

SVC / Storwize analysis cost effective storage planning (use case)

Michael Pirker

Aujourd’hui, un nombre important de nos clients ont franchi le cap et utilisent Oracle 12c pour des applications en production. Parmi tous ces clients, nous constatons de plus en plus un intérêt prononcé pour l’option Multitenant. Même si le passage à l’option Multitenant ne présente pas de difficultés en soi, il existe de nombreux pièges à éviter ainsi que quelques points à clarifier, notamment sur les aspects relatifs à la performance. En abordant un cas concret avec l’un de nos clients, cette présentation vous apportera des éléments de réponses afin de vous permettre d’envisager sereinement la mise en œuvre de cette option qui constitue un changement important et très intéressant d’un point vue architecture.

Retour d'expérience d'un environnement base de données multitenant

Swiss Data Forum Swiss Data Forum

Architectural caching patterns for kubernetes

Rafał Leszko

Similaire à Apache BookKeeper State Store: A Durable Key-Value Store - Pulsar Summit NA 2021 (20)

Manage Pulsar Cluster Lifecycles with Kubernetes Operators - Pulsar Summit NA...

Benchmarking for postgresql workloads in kubernetes

Scylla Summit 2022: New AWS Instances Perfect for ScyllaDB

Infinitic: Building a Workflow Engine on Top of Pulsar - Pulsar Summit NA 2021

Ceph Community Talk on High-Performance Solid Sate Ceph

In-memory No SQL- GIDS2014

Critical Attributes for a High-Performance, Low-Latency Database

How Pulsar Enables Netdata to Offer Unlimited Infrastructure Monitoring for F...

Oracle big data appliance and solutions

The architecture of SkySQL

Architectural caching patterns for kubernetes

Function Mesh: Complex Streaming Jobs Made Simple - Pulsar Summit NA 2021

Learn from HomeAway Hadoop Development and Operations Best Practices

[DevConf.US 2019]Quarkus Brings Serverless to Java Developers

Kafka meetup seattle 2019 mirus reliable, high performance replication for ap...

Ohio Devfest - Visual Analysis with GCP

YARN

SVC / Storwize analysis cost effective storage planning (use case)

Retour d'expérience d'un environnement base de données multitenant

Architectural caching patterns for kubernetes

Plus de StreamNative

So, you are a responsible software engineer building microservices for Apache Kafka, and life is good. Eventually, you hear the community talking about the outstanding experience they are having with Apache Pulsar features. They talk about infinite event stream retention, a rebalance-free architecture, native support for event processing, and multi-tenancy. Exciting, right? Most people would want to migrate their code to Pulsar. Especially when you know that Pulsar also supports Kafka clients natively via the protocol handler known as KoP — which enables the Kafka client APIs on Pulsar. But, as said before, you are responsible; and you don't believe in fairy tales, just like you don't believe that migrations like this happen effortlessly. This session will discuss the architecture behind protocol handlers, what it means having one enabled on Pulsar, and how the KoP works. It will detail the effort required to migrate a microservice written for Kafka to Pulsar, and whether the code need to change for this.

Is Using KoP (Kafka-on-Pulsar) a Good Idea? - Pulsar Summit SF 2022

StreamNative

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

In this talk, learn how Toast leverages our Envoy control-plane to manage blue-green deploys of Pulsar consumers, and how this has helped drive adoption across the engineering organization. Dive into the history of Pulsar at Toast, starting from its introduction in 2019 to provide event-driven architecture across a rapidly scaling restaurant software platform. We will detail some of the hurdles that we encountered gaining buy-in across a diverse set of teams, and dive deep into how we enforce best practices and integrate with our service control plane.

Blue-green deploys with Pulsar & Envoy in an event-driven microservice ecosys...

StreamNative

Event streaming architectures launched a reexamination of applications and systems architectures across the board. We live in a world where answers are needed now in a constant real-time flow. Yet beyond the event streaming system itself, what are the corequisites to ensure our large scale distributed database systems can keep pace with this always-on, always-current real time flow of data? What are the requirements and expectations for this next tech cycle?

Distributed Database Design Decisions to Support High Performance Event Strea...

StreamNative

Pulsar Functions is a succinct framework provided by Apache Pulsar to conduct real-time data processing. Its use cases include ETL pipeline, event-driven applications, and simple data analytics. While Pulsar Functions already provides an extremely simple programming interface, we want to further lower the barrier for users to access real-time data. Since SQL is one of the universal languages in the technology world and well accepted by the vast majority of data engineers, we decided to add a SQL expressing layer on top of Pulsar Functions runtime. In this talk, we will discuss the architecture and implementation of this new service. We will see how SQL syntax, Pulsar Functions, and Function Mesh can work together to deliver a unique user development experience for real-time data jobs in the cloud environment. We will also walk through use cases like filtering, routing, and projecting messages as well as integrating with the Pulsar IO Connectors framework.

Simplify Pulsar Functions Development with SQL - Pulsar Summit SF 2022

StreamNative

Starting with version 2.10, the Apache ZooKeeper dependency has been eliminated and replaced with a pluggable framework that enables you to reduce the infrastructure footprint of Apache Pulsar by leveraging alternative metadata and coordination systems based on your deployment environment. In this talk, walk through the steps required to utilize the existing etcd service running inside Kubernetes to act as Pulsar's metadata store, thereby eliminating the need to run ZooKeeper entirely, leaving you with a Zookeeper-less Pulsar.

Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022

StreamNative

Apache Pulsar is a highly available, distributed messaging system that provides guarantees of no message loss and strong message ordering with predictable read and write latency. In this talk, learn how this can be validated for Apache Pulsar Kubernetes deployments. Various failures are injected using Chaos Mesh to simulate network and other infrastructure failure conditions. There are many questions that are asked about failure scenarios, but it could be hard to find answers to these important questions. When a failure happens, how long does it take to recover? Does it cause unavailability? How does it impact throughput and latency? Are the guarantees of no message loss and strong message ordering kept, even when components fail? If a complete availability zone fails, is the system configured correctly to handle AZ failures? This talk will help you find answers to these questions and apply the tooling and practices to your own testing and validation.

Validating Apache Pulsar’s Behavior under Failure Conditions - Pulsar Summit ...

StreamNative

Despite what the Ghostbusters said, we’re going to go ahead and cross (or, join) the streams. This session covers getting started with streaming data pipelines, maximizing Pulsar’s messaging system alongside one of the most flexible streaming frameworks available, Apache Flink. Specifically, we’ll demonstrate the use of Flink SQL, which provides various abstractions and allows your pipeline to be language-agnostic. So, if you want to leverage the power of a high-speed, highly customizable stream processing engine without the usual overhead and learning curves of the technologies involved (and their interconnected relationships), then this talk is for you. Watch the step-by-step demo to build a unified batch and streaming pipeline from scratch with Pulsar, via the Flink SQL client. This means you don’t need to be familiar with Flink, (or even a specific programming language). The examples provided are built for highly complex systems, but the talk itself will be accessible to any experience level.

Cross the Streams! Creating Streaming Data Pipelines with Apache Flink + Apac...

StreamNative

Apache Pulsar depends upon message acknowledgments to provide at-least-once or exactly-once processing guarantees. With these guarantees, any transmission between the broker and its producers and consumers requires an acknowledgment. But what happens if an acknowledgment is not received? Resending the message introduces the potential of duplicate processing and increases the likelihood of out or order processing. Therefore, it is critical to understand the Pulsar message redelivery semantics in order to prevent either of these conditions. In this talk, we will walk you through the redelivery semantics of Apache Pulsar, and highlight some of the control mechanisms available to application developers to control this behavior. Finally, we will present best practices for configuring message redelivery to suit various use cases.

Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022

StreamNative

Lakehouses are quickly growing in popularity as a new approach to Data Platform Architecture bringing some of the long-established benefits from OLTP world to OLAP, including transactions, record-level updates/deletes, and changes streaming. In this talk, we will discuss Apache Hudi and how it unlocks possibilities of building your own fully open-source Lakehouse featuring a rich set of integrations with existing technologies, including Apache Pulsar. In this session, we will present: - What Lakehouses are, and why they are needed. - What Apache Hudi is and how it works. - Provide a use-case and demo that applies Apache Hudi’s DeltaStreamer tool to ingest data from Apache Pulsar.

Unlocking the Power of Lakehouse Architectures with Apache Pulsar and Apache ...

StreamNative

Pulsar is a horizontally scalable messaging system, so the traffic in a logical cluster must be balanced across all the available Pulsar brokers as evenly as possible, in order to ensure full utilization of the broker layer. You can use multiple settings and tools to control the traffic distribution which requires a bit of context to understand how the traffic is managed in Pulsar. In this talk, we will walk you through the load balancing capabilities of Apache Pulsar, and highlight some of the control mechanisms available to control the distribution of load across the Pulsar brokers. Finally, we will discuss the various loading shedding strategies that are available. At the end of the talk, you will have a better understanding of how Pulsar's broker level auto-balancing works, and how to properly configure it to meet your workload demands.

Understanding Broker Load Balancing - Pulsar Summit SF 2022

StreamNative

Building an Asynchronous Application Framework with Python and Pulsar - Pulsa...

StreamNative

In today’s world, we are seeing a big shift toward the Cloud. With this shift comes a big shift in the expectations we have for a messaging system, especially when the messaging system is presented as managed service in a large-scale, multi-tenant environment. For any large-scale enterprise, it’s very important to evaluate messaging system and be confident before expanding complex distributed data systems like Apache Pulsar from on-premise to elastically scalable, fully managed services on cloud services. We must consider aspects such as: migration from and integration with large-scale on-premise clusters, security, cost efficiency, and the cloud friendliness of the architecture, modeling cost and capacity, tenant isolation, deployment robustness, availability, monitoring, etc. Not every messaging system is built to be cloud-native and run as a managed service with cost efficiency. We have been running large-scale Apache Pulsar at Yahoo for the last 8 years on various platforms and hardware configurations while meeting application SLAs and serving more than 1M topics in a cluster. In this talk, we will talk about Pulsar’s journey in Yahoo! from an on-premise platform to a hybrid cloud and on-premise system. We will talk about Pulsar’s architecture and features that make Pulsar a good cloud-native messaging-system choice for any enterprise.

Pulsar's Journey in Yahoo!: On-prem, Cloud and Hybrid - Pulsar Summit SF 2022

StreamNative

Pulsar Summit San Francisco is the event dedicated to Apache Pulsar. This one-day, action-packed event will include 5 keynotes, 12 breakout sessions, and 1 amazing happy hour. Speakers are from top companies, including Google, AWS, Databricks, Onehouse, StarTree, Intel, ScyllaDB, and more! It’s the perfect opportunity to network with Pulsar thought leaders in person. Join developers, architects, data engineers, DevOps professionals, and anyone who wants to learn about messaging and event streaming for this one-day, in-person event. Pulsar Summit San Francisco brings the Apache Pulsar Community together to share best practices and discuss the future of streaming technologies.

Event-Driven Applications Done Right - Pulsar Summit SF 2022

StreamNative

Our services team creates, builds, and maintains the as a service offering for base platform services within our organization. Several thousand applications use these custom services daily generating more than 700 million requests per minute. One of these services was our publish / subscriber offering, BQ with custom SDK and custom metrics based on Apache Pulsar. BQ is the core communication service within our organization, having more 200M RPM. All the core processes of the organization depend on this service for operation: the CDC of any of our RDBMS or NoSQL offering, all the eventing efforts of the organization, async communication between apps, notification systems, etc. The backend of the solution was Apache Pulsar running on EC2 on AWS and on top of that we built several components as wrappers of the actual backend, creating our own SDKs and abstractions and in many ways extending the features provided by Pulsar. We had a multi-cluster setup 100% on AWS, with custom Pulsar Docker images running on large ASG setups, along with our own wrapping and admin APIs and DBs. All of this in turn transformed the solution into a volatile solution.

Pulsar @ Scale. 200M RPM and 1K instances - Pulsar Summit SF 2022

StreamNative

There is an increasing need to unleash analytical capabilities directly to the end-users to democratize decision-making. User-Facing Analytics is a new frontier that will shape the products of tomorrow and push the limits of existing technology. It demands a solution that will scale to millions of users to provide fast, real-time insights. In this session, Xiang will talk about his journey to build Apache Pinot to tackle the analytics problem space with the architectural changes and technology inventions made over the past decade. He will also talk about how other big data companies such as LinkedIn, Uber, and Stripe power their user-facing analytical applications.

Data Democracy: Journey to User-Facing Analytics - Pulsar Summit SF 2022

StreamNative

Beam + Pulsar: Powerful Stream Processing at Scale - Pulsar Summit SF 2022

StreamNative

Welcome and Opening Remarks - Pulsar Summit SF 2022

StreamNative

Milvus is an open-source vector database that leverages a novel data fabric to build and manage vector similarity search applications. As the world's most popular vector database, it has already been adopted in production by thousands of companies around the world, including Lucidworks, Shutterstock, and Cloudinary. With the launch of Milvus 2.0, the community aims to introduce a cloud-native, highly scalable and extendable vector similarity solution, and the key design concept is log as data. Milvus relies on Pulsar as the log pub/sub system. Pulsar helps Milvus to reduce system complexity by loosely decoupling each micro service, making the system stateless by disaggregating log storage and computation, which also makes the system further extendable. We will introduce the overview design, the implementation details of Milvus and its roadmap in this topic. Takeaways: 1) Get a general idea about what is a vector database and its real-world use cases. 2) Understand the major design principles of Milvus 2.0. 3) Learn how to build a complex system with the help of a modern log system like Pulsar.

Log System As Backbone – How We Built the World’s Most Advanced Vector Databa...

StreamNative

MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments. This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.

MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...

StreamNative

Plus de StreamNative (20)