Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Merlino, Imply

•

1 like•1,805 views

Apache Druid is a high-performance distributed analytics store for modern analytics applications. It supports ingesting millions of events per second and sub-second query processing. Druid supports various types of data sources for ingestion, including Apache Kafka. You can immediately query on stream events once they get ingested into Druid. Since Kafka provides scalable and robust data delivery while Druid supports advanced complex analysis on streams, Kafka and Druid are widely used together for BI and operational analytics use cases, which require interactivity, scalability, real-time, and performance. This talk is based on our real-world experiences building out streaming analytics stacks powering production use cases across many industries.

Technology

A complete data platform for
real-time analytics
Apr 2021
Druid + Kafka
Transform your data-in-motion to analytics-in-motion
July 2021
© 2021 Imply Data, Inc.

© 2021 Imply Data, Inc.
Who am I?
Gian Merlino
Committer & PMC chair at
Cofounder at (we’re hiring!)

© 2021 Imply Data, Inc.
Agenda
What does it mean to set analytics in motion?
Introduction to Apache Druid
Learn how Druid + Kafka work together
Try this at home

© 2021 Imply Data, Inc.
What does it mean to set analytics in motion?

© 2021 Imply Data, Inc.
Analytics-in-motion requirements
Interactive queries
Sub-second OLAP queries
Highly scalable
Massive data volumes, query
concurrency, ingestion throughput
Real-time
Ingest in true real-time from Kafka
Cost-eﬀective
Fastest queries at lowest cost

© 2021 Imply Data, Inc.
Analytics-in-motion applications

© 2021 Imply Data, Inc.
Intro to Apache Druid

© 2021 Imply Data, Inc.
Druid is a powerful real-time database
=

© 2021 Imply Data, Inc.
Druid is a fancy calculator
=

© 2021 Imply Data, Inc.
Druid in the wild

© 2021 Imply Data, Inc.
Some numbers from Imply customers
Petabytes of data under management
1000s of daily active users
Millions of rows/second ingested
90%ile query time < 1s
99%ile query time < 2s

© 2021 Imply Data, Inc.
Druid is designed for real-time performance
Data sourced from: Correia, José & Costa, Carlos & Santos, Maribel. (2019). Challenging SQL-on-Hadoop Performance with Apache Druid.

© 2021 Imply Data, Inc.
Druid is designed for real-time performance
Data layer
Deep storage
Query layer
Streaming data is
immediately queryable
Hot, already-ingested
data cached locally
Filtered and aggregated data
transferred to query layer
Ingested data
stored in indexed
columnar format

© 2021 Imply Data, Inc.
How Druid + Kafka work together

© 2021 Imply Data, Inc.
Stream-only pattern
Kafka
Data
sources
Real-time
processing
Real-time
delivery
Kafka Streams
ksqlDB
Real-time
database
Presentation

© 2021 Imply Data, Inc.
Kafka + data lake pattern
Data lake
Kafka
Data
sources
File-based
storage
Data lake
Data lake
Query
engine
Query
engine
Query
engine
🔥 Analytics in motion
Federated query engine
(Presto, Hive, Spark, …)
Real-time
delivery
Direct load from Kafka topic
Backﬁll if
needed

© 2021 Imply Data, Inc.
Try this at home

© 2021 Imply Data, Inc.
Stay in touch
19
@druidio
Join the community!
https://druid.apache.org/community/
Free online training by Imply!
https://imply.io/druid-university
Follow the Druid project on Twitter!

© 2021 Imply Data, Inc.
Run with Imply
20
Available at https://imply.io/
Point and click deployment
AWS, GCP, Kubernetes
Bring-your-own VM / bare metal
Batteries included: visualizations,
dashboarding, alerts

A complete data platform for
real-time analytics
Apr 2021
Thank you for attending
July 2021
© 2021 Imply Data, Inc.

What's hot

A Thorough Comparison of Delta Lake, Iceberg and Hudi

Databricks

Introduction to PySpark

Russell Jurney

Spark tuning

GMO-Z.com Vietnam Lab Center

How to Build a Data Lake in Amazon S3 & Amazon Glacier - AWS Online Tech Talks

Amazon Web Services

The Spark Listener interface provides a fast, simple and efficient route to monitoring and observing your Spark application - and you can start using it in minutes. In this talk, we'll introduce the Spark Listener interfaces available in core and streaming applications, and show a few ways in which they've changed our world for the better at SpotX. If you're looking for a "Eureka!" moment in monitoring or tracking of your Spark apps, look no further than Spark Listeners and this talk!

Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring

Databricks

Apache Spark Architecture

Alexey Grishchenko

Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark

Databricks

OSS NA 2019 - Demo Booth deck overview of Egeria

ODPi

HDInsight for Architects

Ashish Thapliyal

Spark SQL

Joud Khattab

Watch this talk here: https://www.confluent.io/online-talks/how-apache-kafka-works-on-demand Pick up best practices for developing applications that use Apache Kafka, beginning with a high level code overview for a basic producer and consumer. From there we’ll cover strategies for building powerful stream processing applications, including high availability through replication, data retention policies, producer design and producer guarantees. We’ll delve into the details of delivery guarantees, including exactly-once semantics, partition strategies and consumer group rebalances. The talk will finish with a discussion of compacted topics, troubleshooting strategies and a security overview. This session is part 3 of 4 in our Fundamentals for Apache Kafka series.

How Apache Kafka® Works

confluent

Introducing DataFrames in Spark for Large Scale Data Science

Databricks

Spark SQL Deep Dive @ Melbourne Spark Meetup

Databricks

Introduction to Spark with Python

Gokhan Atil

Interactive Analytics in Human Time

DataWorks Summit

Building data product requires having Lambda Architecture to bridge the batch and streaming processing. AirStream is a framework built on top of Apache Spark to allow users to easily build data products at Airbnb. It proved Spark is impactful and useful in the production for mission-critical data products. On the streaming side, hear how AirStream integrates multiple ecosystems with Spark Streaming, such as HBase, Elasticsearch, MySQL, DynamoDB, Memcache and Redis. On the batch side, learn how to apply the same computation logic in Spark over large data sets from Hive and S3. The speakers will also go through a few production use cases, and share several best practices on how to manage Spark jobs in production.

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...

Databricks

"Most data practitioners grapple with data quality issues and data pipeline complexities—it's the bane of their existence. Data engineers, in particular, strive to design and deploy robust data pipelines that serve reliable data in a performant manner so that their organizations can make the most of their valuable corporate data assets. Databricks Delta, part of Databricks Runtime, is a next-generation unified analytics engine built on top of Apache Spark. Built on open standards, Delta employs co-designed compute and storage and is compatible with Spark API’s. It powers high data reliability and query performance to support big data use cases, from batch and streaming ingests, fast interactive queries to machine learning. In this tutorial we will discuss the requirements of modern data pipelines, the challenges data engineers face when it comes to data reliability and performance and how Delta can help. Through presentation, code examples and notebooks, we will explain pipeline challenges and the use of Delta to address them. You will walk away with an understanding of how you can apply this innovation to your data architecture and the benefits you can gain. This tutorial will be both instructor-led and hands-on interactive session. Instructions in how to get tutorial materials will be covered in class. WHAT YOU’LL LEARN: – Understand the key data reliability and performance data pipelines challenges – How Databricks Delta helps build robust pipelines at scale – Understand how Delta fits within an Apache Spark™ environment – How to use Delta to realize data reliability improvements – How to deliver performance gains using Delta PREREQUISITES: – A fully-charged laptop (8-16GB memory) with Chrome or Firefox – Pre-register for Databricks Community Edition" Speakers: Steven Yu, Burak Yavuz

Building Robust Production Data Pipelines with Databricks Delta

Databricks

Real-time Analytics with Apache Flink and Druid

Jan Graßegger

Spark overview

Lisa Hua

Flink vs. Spark

Slim Baltagi

What's hot (20)

A Thorough Comparison of Delta Lake, Iceberg and Hudi

Introduction to PySpark

Spark tuning

How to Build a Data Lake in Amazon S3 & Amazon Glacier - AWS Online Tech Talks

Apache Spark Listeners: A Crash Course in Fast, Easy Monitoring

Apache Spark Architecture

Arbitrary Stateful Aggregations using Structured Streaming in Apache Spark

OSS NA 2019 - Demo Booth deck overview of Egeria

HDInsight for Architects

Spark SQL

How Apache Kafka® Works

Introducing DataFrames in Spark for Large Scale Data Science

Spark SQL Deep Dive @ Melbourne Spark Meetup

Introduction to Spark with Python

Interactive Analytics in Human Time

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...

Building Robust Production Data Pipelines with Databricks Delta

Real-time Analytics with Apache Flink and Druid

Spark overview

Flink vs. Spark

Similar to Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Merlino, Imply

The world’s largest enterprises run their infrastructure on Oracle, DB2 and SQL and their critical business operations on SAP applications. Organisations need this data to be available in real-time to conduct necessary analytics. However, delivering this heterogeneous data at the speed it’s required can be a huge challenge because of the complex underlying data models and structures and legacy manual processes which are prone to errors and delays. Unlock these silos of data and enable the new advanced analytics platforms by attending this session. Find out how to: • To overcome common challenges faced by enterprises trying to access their SAP data • You can integrate SAP data in real-time with change data capture (CDC) technology • Organisations are using Attunity Replicate for SAP to stream SAP data in to Kafka

Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

DataWorks Summit/Hadoop Summit

"Streaming systems are a powerful tool that unlock real-time analytics and applications within organizations. But, they come at a cost: it often takes a team of experts a considerable amount of time to set these systems up, operate them in production, and make meaningful changes to the underlying business logic. When Pluralsight noticed the high cost and slow time to value of their real-time data products, they knew they needed a new tool for the job. Enter Materialize, the first operational data warehouse. Materialize makes many of the benefits of popular streaming systems available through a simpler, higher-level interface: SQL. Using Materialize, complex and expensive data pipelines can be swapped out for incrementally-maintained materialized views. In this talk, we'll present how Pluralsight is replacing streaming systems with an operational data warehouse for their real-time use cases. Today, Materialize powers Pluralsights Plan Analytics and core data models for their content offerings. Looking forward, the Pluralsight team plans to use Materialize for all of their real-time, user-facing applications."

Operationalizing Pluralsight's Data with Materialize

HostedbyConfluent

Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

HostedbyConfluent

Your team is always under pressure to accelerate the adoption of the most modern and powerful technologies. Simultaneously, your existing investments, such as IBM i, your organization’s most critical data asset, remain in a silo. The only practical path forward is to connect the new and existing with a streaming technology like Apache Kafka to feed real-time applications that power use cases ranging from marketing and order replenishment to fraud detection. Join this Precisely webinar to learn how to unlock the potential of your IBM i data by creating data pipelines that integrate, transform, and deliver it to users when and where they need it. Additionally, hear how Stark Denmark, uses Precisely Connect CDC to provide data to their organization in real-time. Join this webinar to: - Understand the benefits and challenges of building data pipelines that access and integrate data from IBM i systems to modern data platforms - Learn how Precisely can help you build real-time data pipelines - Hear from Stark Denmark on how they are using Connect CDC from Precisely and the benefits they are getting

Streaming IBM i to Kafka for Next-Gen Use Cases

Precisely

From raw data to business insights. A modern data lake

javier ramirez

963

Annu Ahmed

Have you ever wanted a "process in the cloud" button on your laptop for your data scientists? In this workshop we will explore how to achieve fast processing speeds by making use of an open-source project called PyWren to massively parallelize data analytics jobs across hundreds or thousands of AWS Lambda functions. We will learn how to use PyWren's wrapper to execute your Python function in parallel, potentially achieving tenths of TFLOPS at peak with Amazon S3 as event coordinator. Lastly we will make use of PyWren to process large amount of data from the Amazon Public datasets, achieving high throughput on Amazon S3, triggered straight from your laptop.

Massively Parallel Data Processing with PyWren and AWS Lambda - SRV424 - re:I...

Amazon Web Services

Over 90% of today’s data was generated in the last 2 years, and the rate of data growth isn’t slowing down. In this session, we’ll step through the challenges and best practices on how to capture all the data that is being generated, understand what data you have, and start driving insights and even predict the future using purpose built AWS Services. We’ll frame the session and demonstrations around common pitfalls of building Data Lakes and how to successful drive analytics and insights from the data. This session will focus on the architecture patterns bringing together key AWS Services and rather than a deep dive on any single service. We’ll show how services such as Amazon S3, Amazon Glue, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, and Amazon Kinesis, and Amazon Machine Learning services are put together to build a successful data lake for various role including both data scientists and business users.

BDA305 Building Data Lakes and Analytics on AWS

Amazon Web Services

Introduction to Hadoop

POSSCON

963

Annu Ahmed

In this session, we show you how to understand what data you have, how to drive insights, and how to make predictions using purpose-built AWS services. Learn about the common pitfalls of building data lakes, and discover how to successfully drive analytics and insights from your data. Also learn how services such as Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, Amazon Kinesis, and Amazon ML services work together to build a successful data lake for various roles, including data scientists and business users.

Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...

Amazon Web Services

In this session, we show you how to understand what data you have, how to drive insights, and how to make predictions using purpose-built AWS services. Learn about the common pitfalls of building data lakes and discover how to successfully drive analytics and insights from your data. Also learn how services such as Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, Amazon Kinesis, and Amazon ML services work together to build a successful data lake for various roles, including data scientists and business users.

Building Data Lakes and Analytics on AWS

Amazon Web Services

In this session, we show you how to understand what data you have, how to drive insights, and how to make predictions using purpose-built AWS services. Learn about the common pitfalls of building data lakes and discover how to successfully drive analytics and insights from your data. Also learn how services such as Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, Amazon Kinesis, and Amazon ML services work together to build a successful data lake for various roles, including data scientists and business users.

Building Data Lakes and Analytics on AWS

Amazon Web Services

In this presentation, Daniel will walk through his journey through his experiences with Apache Hudi and the Hudi Streamer, exploring everything from their initial requirements and outputs, their past and current challenges, and their ideas for their future Hudi usage. He'll talk about their Hudi integrations and how it has impacted their data extraction and loading processes. Join us on Wednesday, February 28th at 9 am PT to hear how Hudi has revolutionized Funding Circle's data infrastructure. catch the live event https://youtu.be/P29dfaxUdTU

Feb 2024 Apache Hudi Community Sync with Daniel Ford

nadine39280

Big Data Goes Airborne. Propelling Your Big Data Initiative with Ironcluster ...

Precisely

Traditional databases and batch ETL operations have not been able to serve the growing data volumes and the need for fast and continuous data processing. How can modern enterprises provide their business users real-time access to the most up-to-date and complete data? In our upcoming webinar, our experts will talk about how real-time CDC improves data availability and fast data processing through incremental updates in the big data lake, without modifying or slowing down source systems. Join this session to learn: What is CDC and how it impacts business The various methods for CDC in the enterprise data warehouse The key factors to consider while building a next-gen CDC architecture: Batch vs. real-time approaches Moving from just capturing and storing, to capturing enriching, transforming, and storing Avoiding stopgap silos to state-through processing Implementation of CDC through a live demo and use-case You can view the webinar here - https://www.streamanalytix.com/webinar/planning-your-next-gen-change-data-capture-cdc-architecture-in-2019/ For more information visit - https://www.streamanalytix.com

Planning your Next-Gen Change Data Capture (CDC) Architecture in 2019 - Strea...

Impetus Technologies

The Presentation Talks about how Cloud Computing is Big Data's Best Friend and How AWS Cloud Components Fit in to complete your Big Data Life Cycle. Agenda: - How Big is Big Data Actually growing? - How Cloud has the potential to become Big Data's Best Friend - A tour on The Big Data Life Cycle - How AWS Cloud Components Fit in to this Life Cycle - A Case Study of Our Log Analytics Tool Cloudlytics, using Big Data Implementation on AWS Cloud.

Big Data Building Blocks with AWS Cloud

Blazeclan Technologies Private Limited

Opening Keynote - AWS Summit SG 2017

Amazon Web Services

Opening Keynote - AWS Summit SG 2017

Amazon Web Services

Paris Spark Meetup (Feb2015) ccarbone : SPARK Streaming vs Storm / MLLib / Ne...

Cedric CARBONE

Similar to Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Merlino, Imply (20)

Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud

Operationalizing Pluralsight's Data with Materialize

Hybrid Kafka, Taking Real-time Analytics to the Business (Cody Irwin, Google ...

Streaming IBM i to Kafka for Next-Gen Use Cases

From raw data to business insights. A modern data lake

963

Massively Parallel Data Processing with PyWren and AWS Lambda - SRV424 - re:I...

BDA305 Building Data Lakes and Analytics on AWS

Introduction to Hadoop

963

Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...

Building Data Lakes and Analytics on AWS

Feb 2024 Apache Hudi Community Sync with Daniel Ford

Big Data Goes Airborne. Propelling Your Big Data Initiative with Ironcluster ...

Planning your Next-Gen Change Data Capture (CDC) Architecture in 2019 - Strea...

Big Data Building Blocks with AWS Cloud

Opening Keynote - AWS Summit SG 2017

Paris Spark Meetup (Feb2015) ccarbone : SPARK Streaming vs Storm / MLLib / Ne...

More from HostedbyConfluent

"In this talk, attendees will be provided with an introduction to Kafka Connect and the basics of Single Message Transforms (SMTs) and how they can be used to transform data streams in a simple and efficient way. SMTs are a powerful feature of Kafka Connect that allow custom logic to be applied to individual messages as they pass through the data pipeline. The session will explain how SMTs work, the types of transformations they can be used for, and how they can be applied in a modular and composable way. Further, the session will discuss where SMTs fit in with Kafka Connect and when they should be used. Examples will be provided of how SMTs can be used to solve common data integration challenges, such as data enrichment, filtering, and restructuring. Attendees will also learn about the limitations of SMTs and when it might be more appropriate to use other tools or frameworks. Additionally, an overview of the alternatives to SMTs, such as Kafka Streams and KSQL, will be provided. This will help attendees make an informed decision about which approach is best for their specific use case. Whether attendees are developers, data engineers, or data scientists, this talk will provide valuable insights into how Kafka Connect and SMTs can help streamline data processing workflows. Attendees will come away with a better understanding of how these tools work and how they can be used to solve common data integration challenges."

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

HostedbyConfluent

"While Apache Kafka lacks native support for topic renaming, there are scenarios where renaming topics becomes necessary. This presentation will delve into the utilization of MirrorMaker 2.0 as a solution for renaming Kafka topics. It will illustrate how MirrorMaker 2.0 can efficiently facilitate the migration of messages from the old topic to the new one and how Kafka Connect Metrics can be employed to monitor the mirroring progress. The discussion will encompass the complexity of renaming Kafka topics, addressing certain limitations, and exploring potential workarounds when using MirrorMaker 2.0 for this purpose. Despite not being originally designed for topic renaming, MirrorMaker 2.0 has a suitable solution for renaming Kafka topics. Blog Post : https://engineering.hellofresh.com/renaming-a-kafka-topic-d6ff3aaf3f03"

Renaming a Kafka Topic | Kafka Summit London

HostedbyConfluent

"Trendyol, Turkey's leading e-commerce company, is committed to positively impacting the lives of millions of customers. Our decision-making processes are entirely driven by data. As a data warehouse team, our primary goal is to provide accurate and up-to-date data, enabling the extraction of valuable business insights. We utilize the benefits provided by Kafka and Kafka Connect to facilitate the transfer of data from the source to our analytical environment. We recently transitioned our Kafka Connect clusters from on-premise VMs to Kubernetes. This shift was driven by our desire to effectively manage rapid growth(marked by a growing number of producers, consumers, and daily messages), ensuring proper monitoring and consistency. Consistency is crucial, especially in instances where we employ Single Message Transforms to manipulate records like filtering based on their keys or converting a JSON Object into a JSON string. Monitoring our cluster's health is key and we achieve this through Grafana dashboards and alerts generated through kube-state-metrics. Additionally, Kafka Connect's JMX metrics, coupled with NewRelic, are employed for comprehensive monitoring. The session will aim to explain our approach to NRT data ingestion, outlining the role of Kafka and Kafka Connect, our transition journey to K8s, and methods employed to monitor the health of our clusters."

Evolution of NRT Data Ingestion Pipeline at Trendyol

HostedbyConfluent

"Join our lightning talk to delve into the strategies vital for maintaining a resilient Kafka service. While proactive monitoring is key for issue prevention, failures will still occur. Rapid detection tools will enable you to identify and resolve problems before they impact end-users. This session explores the techniques employed by Kafka cloud providers for this detection, many of which are also applicable if you are managing independent Kafka clusters or applications. The talk focuses on health-checking, a powerful tool that encompasses an application and its monitoring to validate Kafka environment availability. The session navigates through Kafka health-check methods, sharing best practices, identifying common pitfalls, and highlighting the monitoring of critical performance metrics like throughput and latency for early issue detection. Attendees will gain valuable insights into the art of health-checking their Kafka environment, equipping them with the tools to identify and address issues before they escalate into critical problems. We invite all Kafka enthusiasts to join us in this talk to foster a deeper understanding of Kafka health-checking and ensure the continued smooth operation of your Kafka environment."

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

HostedbyConfluent

"Stream processing systems traditionally gave their users the choice between at least once processing and at most once processing: accepting duplicate data or missing data. But ideally we would provide exactly-once processing, where every event in the input data is represented exactly once in the output. Kafka provides a transaction API that enables exactly-once when using Kafka as your source and sink. But this API has turned out to not be well suited for use by high level streaming systems, requiring various work arounds to still provide transactional processing. In this talk, I’ll cover how the transaction API works, and how systems like Arroyo and Flink have used it to build exactly-once support, and how improvements to the transactional API will enable better end-to-end support for consistent stream processing."

Exactly-once Stream Processing with Arroyo and Kafka

HostedbyConfluent

"In this talk, we will explore the exciting world of IoT and computer vision by presenting a unique project: Fish Plays Pokemon. Using an ESP Eye camera connected to an ESP32 and other IoT devices, to monitor fish's movements in an aquarium. This project showcases the power of IoT and computer vision, demonstrating how even a fish can play a popular video game. We will discuss the challenges we faced during development, including real-time processing, IoT device integration, and Kafka message consumption. By the end of the talk, attendees will have a better understanding of how to combine IoT, computer vision, and the usage of a serverless cloud to create innovative projects. They will also learn how to integrate IoT devices with Kafka to simulate keyboard behavior, opening up endless possibilities for real-time interactions between the physical and digital worlds."

Fish Plays Pokemon | Kafka Summit London

HostedbyConfluent

Tiered Storage 101 | Kafla Summit London

HostedbyConfluent

"Real-time 24/7 monitoring and verification of massive data is challenging – even more so for the world’s second largest manufacturer of memory chips and semiconductors. Tolerance levels are incredibly small, any small defect needs to be identified and dealt with immediately. The goal of semiconductor manufacturing is to improve yield and minimize unnecessary work. However, even with real-time data collection, the data was not easy to manipulate by users and it took many days to enable stream processing requests – limiting its usefulness and value to the business. You’ll hear why SK hynix switched to Confluent and how we developed a self-service stream process portal on top of it. Now users have an easy-to-use service to manipulate the data they want. Results have been impressive, stream processing requests are available the same day – previously taking 5 days! We were also able to drive down costs by 10% as stream processing requests no longer require additional hardware. What you’ll take away from our talk: - What were the pain points in the previous environment - How we transitioned to Confluent without service downtime - Creating a self-service stream processing portal built on top of Connect and ksqlDB - Use case of stream process portal"

Building a Self-Service Stream Processing Portal: How And Why

HostedbyConfluent

"Discover how default configurations might impact ingestion times, especially when dealing with large files. We'll explore a real-world scenario with a 20,000,000+ line file, assessing metrics and exploring the bottleneck in the default setup. Understand the intricacies of batch size calculations and how to optimize them based on your unique data characteristics. Walk away with actionable insights as we showcase a practical example, turning a 7-hour ingestion process into a mere 30 minutes for over 30,000,000 records in a Kafka topic. Uncover metrics, configurations, and best practices to elevate the performance of your Kafka Connect CSV source connectors. Don't miss this opportunity to optimize your data pipeline and ensure smooth, efficient data flow."

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

HostedbyConfluent

"In order to meet the current and ever-increasing demand for near-zero RPO/RTO systems, a focus on resiliency is critical. While Kafka offers built-in resiliency features, a perfect blend of client and cluster resiliency is necessary in order to achieve a highly resilient Kafka client application. At Fidelity Investments, Kafka is used for a variety of event streaming needs such as core brokerage trading platforms, log aggregation, communication platforms, and data migrations. In this lightening talk, we will discuss the governance framework that has enabled producers and consumers to achieve their SLAs during unprecedented failure scenarios. We will highlight how we automated resiliency tests through chaos engineering and tightly integrated observability dashboards for Kafka clients to analyze and optimize client configurations. And finally, we will summarize the chaos test suite and the ""test, test and test"" mantra that are helping Fidelity Investments reach its goal of a future with zero down-time."

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

HostedbyConfluent

"There are various strategies for securely connecting to Kafka clusters between different networks or over the public internet. Many cloud providers even offer endpoints that privately route traffic between networks and are not exposed to the internet. But, depending on your network setup and how you are running Kafka, these options ... might not be an option! In this session, we’ll discuss how you can use SSH bastions or a self managed PrivateLink endpoint to establish connectivity to your Kafka clusters without exposing brokers directly to the internet. We explain the required network configuration, and show how we at Materialize have contributed to librdkafka to simplify these scenarios and avoid fragile workarounds."

Navigating Private Network Connectivity Options for Kafka Clusters

HostedbyConfluent

"In my talk, we will examine all the stages of building our self-service Streaming Data Platform based on Apache Flink and Kafka Connect, from the selection of a solution for stateful streaming data processing, right up to the successful design of a robust self-service platform, covering the challenges that we’ve met. I will share our experience in providing non-Java developers with a company-wide self-service solution, which allows them to quickly and easily develop their streaming data pipelines. Additionally, I will highlight specific business use cases that would not have been implemented without our platform.0 characters0 characters"

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

HostedbyConfluent

"Almost everyone has heard about large language models, and tens of millions of people have tried out OpenAI ChatGPT and Google Bard. However, the intricate architecture and underlying mathematics driving these remarkable systems remain elusive to many. LLM's are fascinating - so let's grab a drink and find out how these systems are built and dive deep into their inner workings. In the length of time it to enjoy a round of drinks, you'll understand the inner workings of these models. We'll take our first sip of word vectors, enjoy the refreshing taste of the transformer, and drain a glass understanding how these models are trained on phenomenally large quantities of data. Large language models for your streaming application - explained with a little maths and a lot of pub stories"

Explaining How Real-Time GenAI Works in a Noisy Pub

HostedbyConfluent

"Monitoring is a fundamental operation when running Kafka and Kafka applications in production. There are numerous metrics available when using Kafka, however the sheer number is overwhelming, making it challenging to know where to start and how to properly utilise them. This session will introduce you to some of the key metrics that should be monitored and best practices in fine tuning your monitoring. We will delve into which metrics are the indicators for cluster’s availability and performance and are the most helpful when debugging client applications."

TL;DR Kafka Metrics | Kafka Summit London

HostedbyConfluent

Kafka Streams relies on state restoration for maintaining standby tasks as failure recovery mechanism as well as for restoring the state after rebalance scenarios. When you are scaling up or down your application instances, it is necessary to know the current state of the restoration process for each active and standby task in order to prevent a long restoration process as much as possible. During this presentation, you will get an understanding of how KIP-869 provides valuable information about the current active task restoration after a rebalance and KIP-988 opens a window to the continuous process of standby restoration. When you encounter a situation in which you need to choose whether or not to scale up or down your application instances, both KIPs will be an invaluable ally for you.

A Window Into Your Kafka Streams Tasks | KSL

HostedbyConfluent

"In this talk, we will dive into the world of Kafka producer configs and explore how to understand and optimize them for better performance. We will cover the different types of configs, their impact on performance, and how to tune them to achieve the best results. Whether you're new to Kafka or a seasoned pro, this session will provide valuable insights and practical tips for improving your Kafka producer performance. - Introduction to Kafka producer internal and workflow - Understanding the producer configs like linger.ms, batch.size, buffer.memory and their impact on performance - Learning about producer configs like max.block.ms, delivery.timeout.ms, request.timeout.ms and retries to make producer more resilient. - Discuss configs like enable.idempotence, max.in.flight.requests.per.connection and transaction related configs to achieve delivery guarantees. - Q&A session with attendees to address specific questions and concerns."

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

HostedbyConfluent

"Data contracts are one of the hottest topics in the data management community. A data contract is a formal agreement between a data producer and its consumers, aimed at reducing data downtime and improving data quality. Schemas are an important part of data contracts, but they are not the only relevant element. In this talk, we’ll: 1. see why data contracts are so important but also difficult to implement; 2. identify the characteristics of a well-designed data contract: discuss the anatomy of a data contract, its main elements and, how to formally describe them; 3. show how to manage the lifecycle of a data contract leveraging Confluent Platform's services."

Data Contracts Management: Schema Registry and Beyond

HostedbyConfluent

"In the realm of stateful stream processing, Apache Flink has emerged as a powerful and versatile platform. However, the conventional SQL-based approach often limits the full potential of Flink applications. We will delve into the benefits of adopting a code-first approach, which provides developers with greater control over application logic, facilitates complex transformations, and enables more efficient handling of state and time. We will also discuss how the code-first approach can lead to more maintainable and testable code, ultimately improving the overall quality of your Flink applications. Whether you're a seasoned Flink developer or just starting your journey, this talk will provide valuable insights into how a code-first approach can revolutionize your stream processing applications."

Code-First Approach: Crafting Efficient Flink Apps

HostedbyConfluent

"Change Data Capture (CDC) has become a commodity in data engineering, much in part due to the ever-rising success of Debezium [1]. But is that all there is? In this lightning talk, we’ll outline the current state of the CDC ecosystem, and understand why adopting a Debezium alternative is still a hard sell. If you’ve ever wondered what else is out there, but can’t keep up with the sprawling of new tools in the ecosystem; we’ll wrap it up for you! [1] https://debezium.io/"

Debezium vs. the World: An Overview of the CDC Ecosystem

HostedbyConfluent

"Separation of compute and storage has become the de-facto standard in the data industry for batch processing. The addition of tiered storage to open source Apache Kafka is the first step in bringing true separation of compute and storage to the streaming world. In this talk, we'll discuss in technical detail how to take the concept of tiered storage to its logical extreme by building an Apache Kafka protocol compatible system that has zero local disks. Eliminating all local disks in the system requires not only separating storage from compute, but also separating data from metadata. This is a monumental task that requires reimagining Kafka's architecture from the ground up, but the benefits are worth it. This approach enables a stateless, elastic, and serverless deployment model that minimizes operational overhead and also drives inter-zone networking costs to almost zero."

Beyond Tiered Storage: Serverless Kafka with No Local Disks

HostedbyConfluent

More from HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Recently uploaded

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

2024: Domino Containers - The Next Step. News from the Domino Container commu...

Martijn de Jong

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

💉💊+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHABI}}+971581248768 +971581248768 Mtp-Kit (500MG) Prices » Dubai [(+971581248768**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Maya Whatsapp +971581248768 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971581248768''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971581248768' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Cl

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@

Advantages of Hiring UIUX Design Service Providers for Your Business

Pixlogix Infotech

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

Finology Group – Insurtech Innovation Award 2024

The Digital Insurer

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Recently uploaded (20)

Driving Behavioral Change for Information Management through Data-Driven Gree...

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Automating Google Workspace (GWS) & more with Apps Script

A Domino Admins Adventures (Engage 2024)

2024: Domino Containers - The Next Step. News from the Domino Container commu...

🐬 The future of MySQL is Postgres 🐘

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...

Advantages of Hiring UIUX Design Service Providers for Your Business

Handwritten Text Recognition for manuscripts and early printed texts

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Data Cloud, More than a CDP by Matt Robison

Boost Fertility New Invention Ups Success Rates.pdf

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Axa Assurance Maroc - Insurer Innovation Award 2024

How to Troubleshoot Apps for the Modern Connected Worker

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Tech Trends Report 2024 Future Today Institute.pdf

Finology Group – Insurtech Innovation Award 2024

How to Troubleshoot Apps for the Modern Connected Worker

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Merlino, Imply

5. © 2021 Imply Data, Inc. Analytics-in-motion requirements Interactive queries Sub-second OLAP queries Highly scalable Massive data volumes, query concurrency, ingestion throughput Real-time Ingest in true real-time from Kafka Cost-eﬀective Fastest queries at lowest cost

11. © 2021 Imply Data, Inc. Some numbers from Imply customers Petabytes of data under management 1000s of daily active users Millions of rows/second ingested 90%ile query time < 1s 99%ile query time < 2s

12. © 2021 Imply Data, Inc. Druid is designed for real-time performance Data sourced from: Correia, José & Costa, Carlos & Santos, Maribel. (2019). Challenging SQL-on-Hadoop Performance with Apache Druid.

13. © 2021 Imply Data, Inc. Druid is designed for real-time performance Data layer Deep storage Query layer Streaming data is immediately queryable Hot, already-ingested data cached locally Filtered and aggregated data transferred to query layer Ingested data stored in indexed columnar format

16. © 2021 Imply Data, Inc. Kafka + data lake pattern Data lake Kafka Data sources File-based storage Data lake Data lake Query engine Query engine Query engine 🔥 Analytics in motion Federated query engine (Presto, Hive, Spark, …) Real-time delivery Direct load from Kafka topic Backﬁll if needed

19. © 2021 Imply Data, Inc. Stay in touch 19 @druidio Join the community! https://druid.apache.org/community/ Free online training by Imply! https://imply.io/druid-university Follow the Druid project on Twitter!

20. © 2021 Imply Data, Inc. Run with Imply 20 Available at https://imply.io/ Point and click deployment AWS, GCP, Kubernetes Bring-your-own VM / bare metal Batteries included: visualizations, dashboarding, alerts

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Merlino, Imply

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Merlino, Imply

Similar to Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Merlino, Imply (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Merlino, Imply