Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, Reddit

•

0 j'aime•543 vues

HostedbyConfluent

Convincing developers to write tests for new code is hard; convincing developers to write tests for new event data is even harder. At Reddit, engineers have often deployed new app versions, only to find out later that the event wasn’t firing at all, or it was missing critical fields. So this begs the question, “How can engineers at Reddit be confident that the events they instrument are accurate and complete?” In this session, we will learn about an internal tool developed at Reddit to QA events in real-time. This KSQL-powered web app streams events from our pipeline, allowing developers to filter events they care about using criteria like User ID, Device ID or the type of user interaction. With a backbone of KSQL and Kafka Streams, engineers can get real-time feedback on how accurate (or how erroneous) their event data is.

Live Event Debugging With
kSQLdb at Reddit
Hannah Hagen, Paul Kiernan
Replace with
circle-cropped
image (LINK) or
Snoovatarof
presenter

2023

Release new
feature
The Old Testing Process
QA
(fires events)
WAIT 2
HOURS
Query the data
warehouse

Drawback
of 2 hour
delay
➔ Testing was not coupled with
development

What we
really
wanted
(and
didn’t
have)
➔ Quick and efficient testing
procedure (no delays!)
➔ No prior knowledge required (e.g.
how to access our data
warehouse, etc)
➔ In summary, we wanted to
remove the barriers to test event
data.

Why
kSQLdb?
➔ Real-time event stream
➔ We already had Kafka-based
infrastructure.
➔ Easy set up. Just need to run a
few “create stream” and “create
table” statements and deploy a
kSQLdb cluster.
➔ SQL-like filters meant developers
can search for the event type or
the testing session they care
about.

2023
Live Event
Debugger

2023
Live Event
Debugger

2023
Live Event
Debugger

2023
Live Event
Debugger

2023
Live Event
Debugger

2023
Source: post_lightbox
Action: click
Noun: upvote

2023
Source: post_lightbox
Action: click
Noun: upvote

2023
Source: post_lightbox
Action: click
Noun: upvote

End Result
➔ Couple seconds delay between
firing event and seeing it in Web
UI
➔ UI-driven QA lows barriers for
testing event data

2023
Thank You
We’re hiring!
redditinc.com/careers

Contenu connexe

Tendances

In this session we share our experience of building a real-time data pipelines at Tencent PCG - one that handles 20 trillion daily messages with 700 clusters and 100Gb/s bursting traffic from a single app. We discuss our roadmap of enhancing Kafka to break its limits in terms of scalability, robustness and cost of operation. We first built a proxy layer that aggregates physical clusters in a way agnostic to the clients. While this architecture solves many operational problems, it requires significant development to stay future-proof. With retrospection with our customer and careful study of the ongoing work from the community, we then designed a region federation solution in the broker layer, which allows us to deploy clusters at a much larger scale than previously possible, while at the same time providing better failure recovery and operability. We discuss how we make this development compatible with KIP-500 and KIP-405, and the two KIP (693, 694) that we submitted for discussion.

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...

HostedbyConfluent

Activision Data team has been running a data pipeline for a variety of Activision games for many years. Historically we used a mix of micro-batch microservices coupled with classic Big Data tools like Hadoop and Hive for ETL. As a result, it could take up to 4-6 hours for data to be available to the end customers. In the last few years, the adoption of data in the organization skyrocketed. We needed to de-legacy our data pipeline and provide near-realtime access to data in order to improve reporting, gather insights faster, power web and mobile applications. I want to tell a story about heavily leveraging Kafka Streams and Kafka Connect to reduce the end latency to minutes, at the same time making the pipeline easier and cheaper to run. We were able to successfully validate the new data pipeline by launching two massive games just 4 weeks apart.

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

HostedbyConfluent

In 2019 we presented “Secure Kafka at scale in true multi-tenant environment” at SFO Kafka summit. Back then, kafka was mainly used for event driven architectures, high-throughput pub/sub use cases and as a data-plane for log aggregation and for transporting metadata & metrics. A lot has changed since then - Kafka plant has grown to handle 400B incoming events in a day just in production, introduced stretch cluster pattern in addition to Active-Active cluster replication pattern. Moreover, new use cases have emerged in using Kafka for near real time stream processing and data moving pipelines across the cloud environments. Moving data in near-real time across the system is a hard problem to solve. Kafka Connect is a framework to stream data in/out of kafka reliably and can be used to achieve near-real time data moving pipelines. In this talk, we will present how kafka adoption has evolved over the last couple of years in our space and deep dive into how we approached in providing Managed Kafka Connect, a newest addition to our service portfolio.

Changing landscapes in data integration - Kafka Connect for near real-time da...

Changing landscapes in data integration - Kafka Connect for near real-time da...

Changing landscapes in data integration - Kafka Connect for near real-time da...

HostedbyConfluent

Do your event streams use connected-data domains such as fraud detection, live logistics routing, or predicting network outages? How can you maintain the analysis and leverage those connections real-time? Graph databases differ from traditional, tabular ones in that they treat connections between data as first class citizens. This means they are optimized for detecting and understanding these relationships – providing insight at speed and at scale. By combining event streams from Kafka along with the power of the Neo4j graph database for interrogating and investigating connections, you make real-time, event-driven intelligent insight a reality. Neo4j Streams integrates Neo4j with Apache Kafka event streams, to serve as a source of data, for instance Change Data Capture or a sink to ingest any kind of Kafka event into your graph. In this session we’ll show you how to get up and running with Neo4j Streams to show you how to sink and source between graphs and streams.

Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connec...

Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connec...

Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connec...

HostedbyConfluent

Joins in Kafka Streams and ksqlDB are a killer-feature for data processing and basic join semantics are well understood. However, in a streaming world records are associated with timestamps that impact the semantics of joins: welcome to the fabulous world of _temporal_ join semantics. For joins, timestamps are as important as the actual data and it is important to understand how they impact the join result. In this talk we want to deep dive on the different types of joins, with a focus of their temporal aspect. Furthermore, we relate the individual join operators to the overall ""time engine"" of the Kafka Streams query runtime and explain its relationship to operator semantics. To allow developers to apply their knowledge on temporal join semantics, we provide best practices, tip and tricks to ""bend"" time, and configuration advice to get the desired join results. Last, we give an overview of recent, and an outlook to future, development that improves joins even further.

Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ

Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ

Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ

HostedbyConfluent

Without seeing what’s wrong with today’s messaging queues, it can be initially confusing to view Apache Kafka as more. By adding additional functionality, true storage, and guarantees it opens opportunities to take full advantage of a publish/subscribe model. Joined by Yelp’s Justin Cunningham we’ll see how their infrastructure has quickly evolved. Powered by Kafka, Yelp has made the leap to microservices and is seeing the benefits of efficiency and performance. Speakers: Justin Cunningham Technical Lead, Software Engineer, Yelp Gehrig Kunz Technical Product Marketing Manager, Confluent

How Yelp Leapt to Microservices with More than a Message Queue

How Yelp Leapt to Microservices with More than a Message Queue

How Yelp Leapt to Microservices with More than a Message Queue

In this talk, we'll discuss how VillageMD is able to use Kafka topic compaction for rapidly scaling our reprocessing pipelines to encompass hundreds of feeds. Within healthcare data ecosystems, privacy and data minimalism are key design priorities. Being able to handle data deletion in a reliable, timely manner within event-driven architectures is becoming more and more necessary with key governance frameworks like the GDPR and HIPAA. We'll be giving an overview of the building and governance of dead-letter queues for streaming data processing. We'll discuss: 1. How to architect a data sink for failed records. 2. How topic compaction can reduce duplicate data and enable idempotency. 3. Building a tombstoning system for removing successfully reprocessed records from the queues. 4. Considerations for monitoring a reprocessing system in production -- what metrics, dataops, and SLAs are useful?

Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...

Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...

Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...

HostedbyConfluent

Cloud is changing the world; Kubernetes is changing the world; real-time event streaming is changing the world. In this talk we explore some of best practices to synergistically combine the power of these paradigm shifts to achieve a much greater return on your Kafka investments. From declarative deployments, zero-downtime upgrades, elastic scaling to self-healing and automated governance, learn how you can bring the next level of speed, agility, resilience, and security to your Kafka implementations.

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

HostedbyConfluent

All Streams Ahead! ksqlDB Workshop ANZ

All Streams Ahead! ksqlDB Workshop ANZ

All Streams Ahead! ksqlDB Workshop ANZ

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

HostedbyConfluent

Watch this talk here: https://www.confluent.io/online-talks/scaling-security-on-100s-of-millions-of-mobile-devices-using-kafka-and-scylla-on-demand Join mobile cybersecurity leader Lookout as they talk through their data ingestion journey. Lookout enables enterprises to protect their data by evaluating threats and risks at post-perimeter endpoint devices and providing access to corporate data after conditional security scans. Their continuous assessment of device health creates a massive amount of telemetry data, forcing new approaches to data ingestion. Learn how Lookout changed its approach in order to grow from 1.5 million devices to 100 million devices and beyond, by implementing Confluent Platform and switching to Scylla.

Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...

Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...

Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...

In this workshop we will set up a streaming framework which will process realtime data of traffic sensors installed within the Belgian road system. Starting with the intake of the data, you will learn best practices and the recommended approach to split the information into events in a way that won't come back to haunt you. With some basic stream operations (count, filter, ... ) you will get to know the data and experience how easy it is to get things done with Spring Boot & Spring Cloud Stream. But since simple data processing is not enough to fulfill all your streaming needs, we will also let you experience the power of windows. After this workshop, tumbling, sliding and session windows hold no more mysteries and you will be a true streaming wizard.

Stream Processing Live Traffic Data with Kafka Streams

Stream Processing Live Traffic Data with Kafka Streams

Stream Processing Live Traffic Data with Kafka Streams

Tom Van den Bulck

Mark Teehan, Principal Solutions Engineer, Confluent Use the Debezium CDC connector to capture database changes from a Postgres database - or MySQL or Oracle; streaming into Kafka topics and onwards to an external data store. Examine how to setup this pipeline using Docker Compose and Confluent Cloud; and how to use various payload formats, such as avro, protobuf and json-schema. https://www.meetup.com/Singapore-Kafka-Meetup/events/276822852/

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

Speakers: Joe Beda, Co-founder and CTO, Heptio + Gwen Shapira, Principal Data Architect, Confluent With the rapid adoption of microservices, there is a growing need for solutions to manage deployment, resources and data for fleets of microservices. Kubernetes is a resource management framework for containers that is rapidly growing in popularity. Apache Kafka is a streaming platform that makes data accessible to the edges of an organization. It's no wonder the question of running Kafka on Kubernetes keeps coming up! In this online talk, Joe Beda, CTO of Heptio and co-creator of Kubernetes, and Gwen Shapira, principal data architect at Confluent and Kafka PMC member, will help you navigate through the hype, address frequently asked questions and deliver critical information to help you decide if running Kafka on Kubernetes is the right approach for your organization. You will: -Get an introduction to the basic concepts you need to know as you plan to deploy services on Kubernetes. -Learn which parts of the Kafka ecosystem fit Kubernetes like a glove, and which require special attention. -Pick up useful tips for getting started. -See why Confluent Platform for Kubernetes is the simplest solution to deploying and orchestrating Kafka on Kubernetes, using container images and a Kubernetes operator. Watch the recording: https://videos.confluent.io/watch/yoZcuazDjDDTcj1sRnaD3J?.

Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes

Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes

Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

HostedbyConfluent

Viktor Gamov, Confluent, Developer Advocate Apache Kafka is an open source distributed streaming platform that allows you to build applications and process events as they occur. Viktor Gamov (developer Advocate at Confluent) walks through how it works and important underlying concepts. As a real-time, scalable, and durable system, Kafka can be used for fault-tolerant storage as well as for other use cases, such as stream processing, centralized data management, metrics, log aggregation, event sourcing, and more. This talk will explain what a streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. https://www.meetup.com/Chennai-Kafka/events/269942117/

What is Apache Kafka®?

What is Apache Kafka®?

What is Apache Kafka®?

Westpac Bank Tech Talk 1: Dive into Apache Kafka

Westpac Bank Tech Talk 1: Dive into Apache Kafka

Westpac Bank Tech Talk 1: Dive into Apache Kafka

How to over-engineer things and have fun? | Oto Brglez, OPALAB

How to over-engineer things and have fun? | Oto Brglez, OPALAB

How to over-engineer things and have fun? | Oto Brglez, OPALAB

HostedbyConfluent

Logging ingestion infrastructure at Pinterest is built around Apache Kafka to support thousands of pipelines with over 1 trillion (1PB) new messages generated by hundreds of services (written in 5 different languages) and transported to data lake (AWS S3) every day. In the past, we have focused on scalability and auto operation of the infrastructure to help internal teams quickly onboard new pipelines (Kafka Summit 2018, 2020). However, we had constantly observed data loss and data corruption due to the design decisions we made to favor scalability and availability over durability and consistency. To tackle these problems, we designed and implemented logging auditing framework which consists of (1) audit client library integrated into every component of the infrastructure to detect data corruption for every message and send out audit events for randomly picked messages, (2) Kafka clusters receiving audit events, and (3) realtime and batch application processing audit events to generate insights for alerting and reporting. Focusing on zero negative impact to existing ingestion pipelines, scalability and cost efficiency led us to make various design decisions to eventually achieve auditing rollout to every pipeline with zero downtime and fundamentally improve the data ingestion quality at Pinterest in general by tracking data loss and removing data corruption which in the past can block downstream applications for hours and often lead to severe incidents.

Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...

Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...

Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...

HostedbyConfluent

To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

Tendances (20)

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...

Changing landscapes in data integration - Kafka Connect for near real-time da...

Changing landscapes in data integration - Kafka Connect for near real-time da...

Changing landscapes in data integration - Kafka Connect for near real-time da...

Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connec...

Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connec...

Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connec...

Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ

Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ

Cloud native Kafka | Sascha Holtbruegge and Margaretha Erber, HiveMQ

How Yelp Leapt to Microservices with More than a Message Queue

How Yelp Leapt to Microservices with More than a Message Queue

How Yelp Leapt to Microservices with More than a Message Queue

Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...

Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...

Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

All Streams Ahead! ksqlDB Workshop ANZ

All Streams Ahead! ksqlDB Workshop ANZ

All Streams Ahead! ksqlDB Workshop ANZ

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...

Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...

Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...

Scaling Security on 100s of Millions of Mobile Devices Using Apache Kafka® an...

Stream Processing Live Traffic Data with Kafka Streams

Stream Processing Live Traffic Data with Kafka Streams

Stream Processing Live Traffic Data with Kafka Streams

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes

Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes

Stateful, Stateless and Serverless - Running Apache Kafka® on Kubernetes

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

What is Apache Kafka®?

What is Apache Kafka®?

What is Apache Kafka®?

Westpac Bank Tech Talk 1: Dive into Apache Kafka

Westpac Bank Tech Talk 1: Dive into Apache Kafka

Westpac Bank Tech Talk 1: Dive into Apache Kafka

How to over-engineer things and have fun? | Oto Brglez, OPALAB

How to over-engineer things and have fun? | Oto Brglez, OPALAB

How to over-engineer things and have fun? | Oto Brglez, OPALAB

Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...

Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...

Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

Similaire à Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, Reddit

Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...

Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...

Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...

Riccardo Zamana

Big Data Tools in AWS

Big Data Tools in AWS

Big Data Tools in AWS

Exactly-once Data Processing with Kafka Streams - July 27, 2017

Exactly-once Data Processing with Kafka Streams - July 27, 2017

Exactly-once Data Processing with Kafka Streams - July 27, 2017

Several recent high-profile security incidents were due to compromised software supply chains. Software Supply Chain is a collective term used to describe the stages of software lifecycle from source to deployment through CI/CD pipelines, and all the static and dynamic analyses in between. In the world of microservices and cloud computing, trust in your company’s supply chain is critical, as most of the tooling and dependencies are from open source and vendor projects. When the code hits production, it’s essential to have enough observability to detect and investigate the problem and get to the root cause and mitigation as quickly as possible. With software supply chain attacks, not only is the newly deployed code under suspicion, but also all the tooling used to produce it becomes a potential attack vector, so an efficient and effective way to verify the integrity of the supply chain is paramount. This talk will discuss what information needs to be collected to allow DevOps to inspect and verify the integrity of the supply chain, the challenges of having the right level of detail to reduce mean-time-to-detection and mean-time-to-understanding, some of the existing solutions and open problems in this space.

Software Supply Chains for DevOps @ InfoQ Live 2021

Software Supply Chains for DevOps @ InfoQ Live 2021

Software Supply Chains for DevOps @ InfoQ Live 2021

Aysylu Greenberg

Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)

Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)

Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)

Building Event-Based Systems for the Real-Time Web

Building Event-Based Systems for the Real-Time Web

Building Event-Based Systems for the Real-Time Web

QuASD/PROFES 2018, Wolfsburg: Talk by Marcus Ciolkowski (@M_Ciolkowski, Principal IT Consultant at QAware) and Florian Lautenschlager (@flolaut, Senior Software Engineer) === Please download slides if blurred! === Abstract: Important and critical aspects of technical debt often surface at runtime only and are difficult to measure statically. This is a particular challenge for cloud applications because of their highly distributed nature. Fortunately, mature frameworks for collecting runtime data exist but need to be integrated. In this paper, we report an experience from a project that implements a cloud application within Kubernetes on Azure. To analyze the runtime data of this software system, we instrumented our services with Zipkin for distributed tracing; with Prometheus and Grafana for analyzing metrics; and with fluentd, Elasticsearch and Kibana for collecting, storing and exploring log files. However, project team members did not utilize these runtime data until we created a unified and simple access using a chat bot. We argue that even though your project collects runtime data, this is not sufficient to guarantee its usage: In order to be useful, a simple, unified access to different data sources is required that should be integrated into tools that are commonly used by team members. Get the research paper: http://bitly.com/2QmSNwl

Making Runtime Data Useful for Incident Diagnosis: An Experience Report

Making Runtime Data Useful for Incident Diagnosis: An Experience Report

Making Runtime Data Useful for Incident Diagnosis: An Experience Report

Presented at Data Science Meetup on 4/12/2018. In this talk, Grega Kespret (head of analytics group) will present Celtra’s data analytics pipeline and how it evolved through the years - sometimes forward, sometimes backward. On this journey, we became early adopter of different technologies: BigQuery, Vertica (pre-join projections), Spark (version 0.5), Databricks (beta users) and Snowflake (one of the first users). As the business grew and the product evolved, volume and complexity of data increased ten-fold, as has the number of users generating insights from this data. How come BigQuery did not scale? Why was choosing Vertica a mistake for our use case, and what have we learned from it? What requirements did we have for the analytics database, why did we have to abandon MySQL, and why we finally chose Snowflake? This talk will be heavily opinionated and will describe our experience and learnings - what worked for us and what didn't.

How we evolved data pipeline at Celtra and what we learned along the way

How we evolved data pipeline at Celtra and what we learned along the way

How we evolved data pipeline at Celtra and what we learned along the way

Why so continuous

Why so continuous

Why so continuous

Kafka Summit SF 2017 - Exactly-once Stream Processing with Kafka Streams

Kafka Summit SF 2017 - Exactly-once Stream Processing with Kafka Streams

Kafka Summit SF 2017 - Exactly-once Stream Processing with Kafka Streams

Real Time Stream Processing with KSQL and Kafka David Peterson, Confluent APAC APIdays Melbourne 2018 Unordered, unbounded and massive datasets are increasingly common in day-to-day business. Using this to your advantage is incredibly difficult with current system designs. We are stuck in a model where we can only take advantage of this *after* it has happened. Many times, this is too late to be useful in the enterprise. KSQL is a streaming SQL engine for Apache Kafka. KSQL lowers the entry bar to the world of stream processing, providing a simple and completely interactive SQL interface for processing data in Kafka. KSQL (like Kafka) is open-source, distributed, scalable, and reliable. A real time Kafka platform moves your data up the stack, closer to the heart of your business, allowing you to build scalable, mission-critical services by quickly deploying SQL-like queries in a severless pattern. This talk will highlight key use cases for real time data, and stream processing with KSQL: Real time analytics, security and anomaly detection, real time ETL / data integration, Internet of Things, application development, and deploying Machine Learning models with KSQ. Real time data and stream processing means that Kafka is just as important to the disrupted as it is to the disruptors.

Real-Time Stream Processing with KSQL and Apache Kafka

Real-Time Stream Processing with KSQL and Apache Kafka

Real-Time Stream Processing with KSQL and Apache Kafka

Ein moderner DevOps-Prozess wäre ohne Feature Branches und Pull-Request-Workflows nicht mehr denkbar. Wie wird nun diese Änderung verifiziert? Oft schafft nur ein Deployment in eine isolierte PR-Umgebung Klarheit. Somit sind Tests im Gesamtkontext mit Umsystemen und echter Datenpersistenz möglich, der Deployment- und Upgrade-Vorgang kann mitgetestet werden und nicht zuletzt kann auch ein Product Owner die Änderung an einem laufenden System anschauen. Dieser Vortrag erklärt, was für ein Pull Request Deployment alles nötig ist und zeigt auch Ansätze für Herausforderungen wie Datenbankschema-Deployment und Dateninitialisierung. Die Beispiele werden anhand von Azure DevOps und Azure Kubernetes Services vorgestellt.

BASTA Spring 2023 - SCHNELLES FEEDBACK MIT PULL REQUEST DEPLOYMENTS

BASTA Spring 2023 - SCHNELLES FEEDBACK MIT PULL REQUEST DEPLOYMENTS

BASTA Spring 2023 - SCHNELLES FEEDBACK MIT PULL REQUEST DEPLOYMENTS

DEVNET-1140 InterCloud Mapreduce and Spark Workload Migration and Sharing: Fi...

DEVNET-1140 InterCloud Mapreduce and Spark Workload Migration and Sharing: Fi...

DEVNET-1140 InterCloud Mapreduce and Spark Workload Migration and Sharing: Fi...

Exactly-once Stream Processing with Kafka Streams

Exactly-once Stream Processing with Kafka Streams

Exactly-once Stream Processing with Kafka Streams

LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack

LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack

LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack

OpenShift Origin

Speaker: Gabriel Schenker, Lead Curriculum Developer, Confluent Streaming platforms have emerged as a popular, new trend, but what exactly is a streaming platform? Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration. With Apache Kafka® at the core, event streaming platforms offer an entirely new perspective on managing the flow of data. This talk will explain what an event streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. New developments in this area such as KSQL will also be discussed.

What is Apache Kafka and What is an Event Streaming Platform?

What is Apache Kafka and What is an Event Streaming Platform?

What is Apache Kafka and What is an Event Streaming Platform?

From Kafka to BigQuery - Strata Singapore

From Kafka to BigQuery - Strata Singapore

From Kafka to BigQuery - Strata Singapore

Dsdt meetup 2017 11-21

Dsdt meetup 2017 11-21

Dsdt meetup 2017 11-21

DSDT Meetup Nov 2017

DSDT Meetup Nov 2017

DSDT Meetup Nov 2017

Yet another presentation about Event Sourcing? Yes and no. Event Sourcing is a really great concept. Some could say it’s a Holy Grail of the software architecture. I might agree with that, while remembering that everything comes with a price. This session is a summary of my experience with ES gathered while working on 3 different commercial products. Instead of theoretical aspects, I will focus on possible challenges with ES implementation. What could explode (very often with delayed ignition)? How and where to store events effectively? What are possible schema evolution solutions? How to achieve the highest level of scalability and live with eventual consistency? And many other interesting topics that you might face when experimenting with ES.

Event sourcing - what could possibly go wrong ? Devoxx PL 2021

Event sourcing - what could possibly go wrong ? Devoxx PL 2021

Event sourcing - what could possibly go wrong ? Devoxx PL 2021

Andrzej Ludwikowski

Similaire à Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, Reddit (20)

Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...

Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...

Time series Analytics - a deep dive into ADX Azure Data Explorer @Data Saturd...

Big Data Tools in AWS

Big Data Tools in AWS

Big Data Tools in AWS

Exactly-once Data Processing with Kafka Streams - July 27, 2017

Exactly-once Data Processing with Kafka Streams - July 27, 2017

Exactly-once Data Processing with Kafka Streams - July 27, 2017

Software Supply Chains for DevOps @ InfoQ Live 2021

Software Supply Chains for DevOps @ InfoQ Live 2021

Software Supply Chains for DevOps @ InfoQ Live 2021

Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)

Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)

Introduction to ksqlDB and stream processing (Vish Srinivasan - Confluent)

Building Event-Based Systems for the Real-Time Web

Building Event-Based Systems for the Real-Time Web

Building Event-Based Systems for the Real-Time Web

Making Runtime Data Useful for Incident Diagnosis: An Experience Report

Making Runtime Data Useful for Incident Diagnosis: An Experience Report

Making Runtime Data Useful for Incident Diagnosis: An Experience Report

How we evolved data pipeline at Celtra and what we learned along the way

How we evolved data pipeline at Celtra and what we learned along the way

How we evolved data pipeline at Celtra and what we learned along the way

Why so continuous

Why so continuous

Why so continuous

Kafka Summit SF 2017 - Exactly-once Stream Processing with Kafka Streams

Kafka Summit SF 2017 - Exactly-once Stream Processing with Kafka Streams

Kafka Summit SF 2017 - Exactly-once Stream Processing with Kafka Streams

Real-Time Stream Processing with KSQL and Apache Kafka

Real-Time Stream Processing with KSQL and Apache Kafka

Real-Time Stream Processing with KSQL and Apache Kafka

BASTA Spring 2023 - SCHNELLES FEEDBACK MIT PULL REQUEST DEPLOYMENTS

BASTA Spring 2023 - SCHNELLES FEEDBACK MIT PULL REQUEST DEPLOYMENTS

BASTA Spring 2023 - SCHNELLES FEEDBACK MIT PULL REQUEST DEPLOYMENTS

DEVNET-1140 InterCloud Mapreduce and Spark Workload Migration and Sharing: Fi...

DEVNET-1140 InterCloud Mapreduce and Spark Workload Migration and Sharing: Fi...

DEVNET-1140 InterCloud Mapreduce and Spark Workload Migration and Sharing: Fi...

Exactly-once Stream Processing with Kafka Streams

Exactly-once Stream Processing with Kafka Streams

Exactly-once Stream Processing with Kafka Streams

LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack

LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack

LinuxCon 2013 Steven Dake on Using Heat for autoscaling OpenShift on Openstack

What is Apache Kafka and What is an Event Streaming Platform?

What is Apache Kafka and What is an Event Streaming Platform?

What is Apache Kafka and What is an Event Streaming Platform?

From Kafka to BigQuery - Strata Singapore

From Kafka to BigQuery - Strata Singapore

From Kafka to BigQuery - Strata Singapore

Dsdt meetup 2017 11-21

Dsdt meetup 2017 11-21

Dsdt meetup 2017 11-21

DSDT Meetup Nov 2017

DSDT Meetup Nov 2017

DSDT Meetup Nov 2017

Event sourcing - what could possibly go wrong ? Devoxx PL 2021

Event sourcing - what could possibly go wrong ? Devoxx PL 2021

Event sourcing - what could possibly go wrong ? Devoxx PL 2021

Plus de HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Renaming a Kafka Topic | Kafka Summit London

Renaming a Kafka Topic | Kafka Summit London

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Evolution of NRT Data Ingestion Pipeline at Trendyol

Evolution of NRT Data Ingestion Pipeline at Trendyol

Evolution of NRT Data Ingestion Pipeline at Trendyol

HostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

HostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Exactly-once Stream Processing with Arroyo and Kafka

Exactly-once Stream Processing with Arroyo and Kafka

Exactly-once Stream Processing with Arroyo and Kafka

HostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Fish Plays Pokemon | Kafka Summit London

Fish Plays Pokemon | Kafka Summit London

Fish Plays Pokemon | Kafka Summit London

HostedbyConfluent

Tiered Storage 101 | Kafla Summit London

Tiered Storage 101 | Kafla Summit London

Tiered Storage 101 | Kafla Summit London

HostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

Building a Self-Service Stream Processing Portal: How And Why

Building a Self-Service Stream Processing Portal: How And Why

Building a Self-Service Stream Processing Portal: How And Why

HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

HostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Navigating Private Network Connectivity Options for Kafka Clusters

Navigating Private Network Connectivity Options for Kafka Clusters

Navigating Private Network Connectivity Options for Kafka Clusters

HostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

HostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

Explaining How Real-Time GenAI Works in a Noisy Pub

Explaining How Real-Time GenAI Works in a Noisy Pub

Explaining How Real-Time GenAI Works in a Noisy Pub

HostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

TL;DR Kafka Metrics | Kafka Summit London

TL;DR Kafka Metrics | Kafka Summit London

TL;DR Kafka Metrics | Kafka Summit London

HostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

A Window Into Your Kafka Streams Tasks | KSL

A Window Into Your Kafka Streams Tasks | KSL

A Window Into Your Kafka Streams Tasks | KSL

HostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

HostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Data Contracts Management: Schema Registry and Beyond

Data Contracts Management: Schema Registry and Beyond

Data Contracts Management: Schema Registry and Beyond

HostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Code-First Approach: Crafting Efficient Flink Apps

Code-First Approach: Crafting Efficient Flink Apps

Code-First Approach: Crafting Efficient Flink Apps

HostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Debezium vs. the World: An Overview of the CDC Ecosystem

Debezium vs. the World: An Overview of the CDC Ecosystem

Debezium vs. the World: An Overview of the CDC Ecosystem

HostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Beyond Tiered Storage: Serverless Kafka with No Local Disks

HostedbyConfluent

Plus de HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Renaming a Kafka Topic | Kafka Summit London

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Evolution of NRT Data Ingestion Pipeline at Trendyol

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Exactly-once Stream Processing with Arroyo and Kafka

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Fish Plays Pokemon | Kafka Summit London

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Tiered Storage 101 | Kafla Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

Building a Self-Service Stream Processing Portal: How And Why

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Navigating Private Network Connectivity Options for Kafka Clusters

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

Explaining How Real-Time GenAI Works in a Noisy Pub

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

TL;DR Kafka Metrics | Kafka Summit London

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

A Window Into Your Kafka Streams Tasks | KSL

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Data Contracts Management: Schema Registry and Beyond

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Code-First Approach: Crafting Efficient Flink Apps

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Debezium vs. the World: An Overview of the CDC Ecosystem

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Dernier

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Principled Technologies

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

A Principled Technologies deployment guide Conclusion Deploying VMware Cloud Foundation 5.1 on next gen Dell PowerEdge servers brings together critical virtualization capabilities and high-performing hardware infrastructure. Relying on our hands-on experience, this deployment guide offers a comprehensive roadmap that can guide your organization through the seamless integration of advanced VMware cloud solutions with the performance and reliability of Dell PowerEdge servers. In addition to the deployment efficiency, the Cloud Foundation 5.1 and PowerEdge solution delivered strong performance while running a MySQL database workload. By leveraging VMware Cloud Foundation 5.1 and PowerEdge servers, you could help your organization embrace cloud computing with confidence, potentially unlocking a new level of agility, scalability, and efficiency in your data center operations.

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Principled Technologies

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Manulife - Insurer Innovation Award 2024

Manulife - Insurer Innovation Award 2024

Manulife - Insurer Innovation Award 2024

The Digital Insurer

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Increase engagement and revenue with Muvi Live Paywall! In this presentation, we will explore the five key benefits of using Muvi Live Paywall to monetize your live streams. You'll learn how Muvi Live Paywall can help you: Monetize your live content easily: Set up pay-per-view access to your live streams and start generating revenue from your content. Increase audience engagement: Provide exclusive, premium content behind the paywall to keep your viewers engaged. Gain valuable viewer insights: Track viewer data and analytics to better understand your audience and tailor your content accordingly. Reduce content piracy: Muvi Live Paywall's security features help protect your content from unauthorized distribution. Streamline your workflow: The all-in-one platform simplifies the process of managing and monetizing your live streams. With Muvi Live Paywall, you can take control of your live stream monetization and create a sustainable business model for your content. Learn more about Muvi Live Paywall and start generating revenue from your live streams today!

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Dernier (20)

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

Boost PC performance: How more available memory can improve productivity

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Powerful Google developer tools for immediate impact! (2023-24 C)

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Exploring the Future Potential of AI-Enabled Smartphone Processors

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Data Cloud, More than a CDP by Matt Robison

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

How to Troubleshoot Apps for the Modern Connected Worker

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Strategies for Landing an Oracle DBA Job as a Fresher

Manulife - Insurer Innovation Award 2024

Manulife - Insurer Innovation Award 2024

Manulife - Insurer Innovation Award 2024

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

A Domino Admins Adventures (Engage 2024)

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Boost Fertility New Invention Ups Success Rates.pdf

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Live Event Debugging With ksqlDB at Reddit | Hannah Hagen and Paul Kiernan, Reddit

1. Live Event Debugging With kSQLdb at Reddit Hannah Hagen, Paul Kiernan Replace with circle-cropped image (LINK) or Snoovatarof presenter

3. Release new feature The Old Testing Process QA (fires events) WAIT 2 HOURS Query the data warehouse

4. Drawback of 2 hour delay ➔ Testing was not coupled with development

5. What we really wanted (and didn’t have) ➔ Quick and efficient testing procedure (no delays!) ➔ No prior knowledge required (e.g. how to access our data warehouse, etc) ➔ In summary, we wanted to remove the barriers to test event data.

6. Why kSQLdb? ➔ Real-time event stream ➔ We already had Kafka-based infrastructure. ➔ Easy set up. Just need to run a few “create stream” and “create table” statements and deploy a kSQLdb cluster. ➔ SQL-like filters meant developers can search for the event type or the testing session they care about.

7. 2023 Live Event Debugger

8. 2023 Live Event Debugger

9. 2023 Live Event Debugger

10. 2023 Live Event Debugger

11. 2023 Live Event Debugger

12. 2023 Source: post_lightbox Action: click Noun: upvote

13. 2023 Source: post_lightbox Action: click Noun: upvote

14. 2023 Source: post_lightbox Action: click Noun: upvote

15. End Result ➔ Couple seconds delay between firing event and seeing it in Web UI ➔ UI-driven QA lows barriers for testing event data

16. 2023 Thank You We’re hiring! redditinc.com/careers