Contenu connexe

Similaire à Serverless Event Streaming Applications as Functionson K8(20)


Serverless Event Streaming Applications as Functionson K8

  1. Timothy Spann Developer Advocate Serverless Event Streaming Applications as Functions on K8 16 May 2022 I Valencia, Spain
  2. Apache Pulsar is a Cloud-Native Messaging and Event-Streaming Platform.
  3. Key Pulsar Concepts: Messaging vs Streaming Message Queueing - Queueing systems are ideal for work queues that do not require tasks to be performed in a particular order. Streaming - Streaming works best in situations where the order of messages is important.
  4. Pulsar Cluster CLUSTER Global ZooKeeper Configuration Store ZK3 ZK2 ZK1 Local ZooKeeper Quorum Bookie 0 Bookie 1 Bookie 2 BookKeeper Bookie Ensemble Pulsar Broker 0 Pulsar Broker 1 Pulsar Broker 2
  5. ● Consume messages from one or more Pulsar topics. ● Apply user-supplied processing logic to each message. ● Publish the results of the computation to another topic. ● Support multiple programming languages (Java, Python, Go) ● Can leverage 3rd-party libraries Pulsar Functions
  6. import java.util.function.Function; public class MyFunction implements Function<String, String> { public String apply(String input) { return doBusinessLogic(input); } } Entire Function Why Pulsar Functions?
  7. from pulsar import Function from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer import json class Chat(Function): def __init__(self): pass def process(self, input, context): fields = json.loads(input) sid = SentimentIntensityAnalyzer() ss = sid.polarity_scores(fields["comment"]) row = { } row['id'] = str(msg_id) if ss['compound'] < 0.00: row['sentiment'] = 'Negative' else: row['sentiment'] = 'Positive' row['comment'] = str(fields["comment"]) json_string = json.dumps(row) return json_string Entire Function Pulsar Python NLP Function
  8. Function Mesh Pulsar Functions, along with Pulsar IO/Connectors, provide a powerful API for ingesting, transforming, and outputting data. Function Mesh, another StreamNative project, makes it easier for developers to create entire applications built from sources, functions, and sinks all through a declarative API.
  9. K8 Deploy
  10. Function Execution
  11. Apache Pulsar Kafka K8
  12. Webinar Series: Building Microservices with Pulsar Watch now Read more Learn how Pulsar Functions, can be leveraged to build a message bus for event-driven microservices
  13. Apache Pulsar Resources ● ● ● ● ar-cluster-in-kubernetes-808ecdfd87
  14. Visit My Booth #S91 Stop By Scan the QR Code to learn more about StreamNative and our exclusive booth offerings.
  15. StreamNative Solution Application Messaging Data Pipelines Real-time Contextual Analytics Tiered Storage APP Layer Computing Layer Storage Layer StreamNative Platform IaaS Layer Micro Service Notification Dashboard Risk Control Auditing Payment ETL
  16. Timothy Spann Developer Advocate FLiP(N) Stack = Flink, Pulsar and NiFi Stack Streaming Systems & Data Architecture Expert Experience: 15+ years of experience with streaming technologies including Pulsar, Flink, Spark, NiFi, Kafka, Big Data, Cloud, MXNet, IoT and more. Today, he helps to grow the Pulsar community sharing rich technical knowledge and experience at both global conferences and through individual conversations.
  17. FLiP Stack Weekly This week in Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark, Elasticsearch and open source friends.
  18. Accessing historical as well as real-time data Pub/sub model enables event streams to be sent from multiple producers, and consumed by multiple consumers To process large amounts of data in a highly scalable way When is Messaging and Streaming used?
  19. CONFIDENTIAL. DO NOT SHARE. Apache Pulsar adoption is being driven by organizations seeking cloud-native architectures and new uses cases. Microservices Apache Pulsar - Built for Containers / Modern Cloud Cloud Native Hybrid & Multi-Cloud Containers
  20. ● “Bookies” ● Stores messages and cursors ● Messages are grouped in segments/ledgers ● A group of bookies form an “ensemble” to store a ledger ● “Brokers” ● Handles message routing and connections ● Stateless, but with caches ● Automatic load-balancing ● Topics are composed of multiple segments ● ● Stores metadata for both Pulsar and BookKeeper ● Service discovery Store Messages Metadata & Service Discovery Metadata & Service Discovery Metadata Store (ZK, RocksDB, etcd, …) Pulsar Cluster
  21. Building Microservices Asynchronous Communication Building Real Time Applications Highly Resilient Tiered storage Pulsar Benefits
  22. Apache Pulsar Apache BookKeeper Offloader & Tiered Storage Offloading Broker 0 Producer Consumer Broker 1 Broker 2 Bookie 0 Bookie 1 Bookie 2 Bookie 3 Bookie 4 S S S S T 1 T 2 T 3 T 4 T 0 S