Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connector | Ljubica Lazarevic, Neo4j

•

1 j'aime•361 vues

Do your event streams use connected-data domains such as fraud detection, live logistics routing, or predicting network outages? How can you maintain the analysis and leverage those connections real-time? Graph databases differ from traditional, tabular ones in that they treat connections between data as first class citizens. This means they are optimized for detecting and understanding these relationships – providing insight at speed and at scale. By combining event streams from Kafka along with the power of the Neo4j graph database for interrogating and investigating connections, you make real-time, event-driven intelligent insight a reality. Neo4j Streams integrates Neo4j with Apache Kafka event streams, to serve as a source of data, for instance Change Data Capture or a sink to ingest any kind of Kafka event into your graph. In this session we’ll show you how to get up and running with Neo4j Streams to show you how to sink and source between graphs and streams.

Technologie

Neo4j, Inc. All rights reserved 2021
Neo4j, Inc. All rights reserved 2021
1
Supercharge your real-time event
processing with Neo4j's Streams
Kafka Connector
Lju Lazarevic
Developer Relations
@ellazal

Neo4j, Inc. All rights reserved 2021
An introduction to Neo4j
2
What is Neo4j?
• A natively-built, property graph DB
• ACID-compliant DBMS
• Analytical & transactional workloads
• Cypher query language
2
Why is different to relational databases?
• Joins on write - avoid SQL join pain!
• Performant for connected data queries
• Database size ≠ query speed

Neo4j, Inc. All rights reserved 2021
3
You may have come across...
• ICIJ Panama Papers
• NASA’s Lessons Learned DB
• CovidGraph
And you’ve probably been using:
• Recommendations engines
• IT & network management
• Fraud detection
• Social network graphs
actors movies
actor_movie
Actor Movie
ACTED_IN

Neo4j, Inc. All rights reserved 2021
Neo4j + Apache Kafka = Magic!
Imagine easier, performant, event driven…
• IT & network incident management
• Fraud detection
• Product recommendation

Neo4j, Inc. All rights reserved 2021
Connecting Apache Kafka to Neo4j

Neo4j, Inc. All rights reserved 2021
Connecting Apache Kafka to Neo4j
Ways to
connect
Neo4j Connector for Apache
Kafka (aka Neo4j Streams)
Kafka-Connect Plugin

Neo4j, Inc. All rights reserved 2021
Getting started with Neo4j Streams
• Installation
◦ Download the latest jar and copy into
the plugins folder
• Setup connection configuration, including:
◦ Kafka server details
◦ Buffer memory, batch sizes, records
to pull, group ID, etc.
◦ Enable sink/source/procedures
• For sink:
◦ Cypher template
◦ Change Data Capture
◦ Pattern extraction (JSON/AVRO)
◦ Create/Update/Delete
• For source: topic,nodes, rels

Neo4j, Inc. All rights reserved 2021
Getting started with Kafka Connect
• Installation
◦ Confluent Hub client
◦ Download and extract to plugins dir
• Set configuration for sink - options
◦ Cypher template
◦ Change Data Capture
◦ Pattern extraction (JSON/AVRO)
◦ Create/Update/Delete

Neo4j, Inc. All rights reserved 2021
Where do I find out more?
Guides and content
• Neo4j streams home page: neo4j.com/labs/kafka/
• Confluent Neo4j sink connector: confluent.io/hub/neo4j/kafka-connect-
neo4j
• Neo4j + Apache Kafka YouTube playlist: dev.neo4j.com/kafka-playlist
Other questions? Come say hello!
• Forum: dev.neo4j.com/forum
• Discord: dev.neo4j.com/chat

Neo4j, Inc. All rights reserved 2021
Neo4j, Inc. All rights reserved 2021
Ljubica Lazarevic
Developer Relations
@ElLazal
lju@neo4j.com
Join the conversation at dev.neo4j.com/forum
Come say hello on Discord at dev.neo4j.com/chat

Contenu connexe

Tendances

As cyber threats continuously grow in sophistication and frequency, companies need to quickly acclimate to effectively detect, respond, and protect their environments. At Intel, we’ve addressed this need by implementing a modern, scalable Cyber Intelligence Platform (CIP) based on Splunk and Apache Kafka. We believe that CIP positions us for the best defense against cyber threats well into the future. Our CIP ingests tens of terabytes of data each day and transforms it into actionable insights through streams processing, context-smart applications, and advanced analytics techniques. Kafka serves as a massive data pipeline within the platform. It provides us the ability to operate on data in-stream, enabling us to reduce Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR). Faster detection and response ultimately leads to better prevention. In our session, we’ll discuss the details described in the IT@Intel white paper that was published in Nov 2020 with same title.

Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka | J...

HostedbyConfluent

Legacy migration is a journey. Mainframes cannot be replaced in a single project. A big bang will fail. This has to be planned long-term. Mainframe offloading and replacement with Apache Kafka and its ecosystem can be used to keep a more modern data store in real-time sync with the mainframe, while at the same time persisting the event data on the bus to enable microservices, and deliver the data to other systems such as data warehouses and search indexes. This session walks through the different steps some companies are already gone through. Technical options like Change Data Capture (CDC), MQ, and third-party tools for mainframe integration, offloading and replacement are explored.

Mainframe Integration, Offloading and Replacement with Apache Kafka | Kai Wae...

HostedbyConfluent

Robinhood uses Kafka in every line of its business, from stock and crypto trading to clearing and data analytics. One interesting aspect of our architecture is that many of our microservices leveraging Kafka are written in Python. When you combine Python's relatively slow performance coupled, its reliance on process-based parallelism and Robinhood’s scale, the result is a massive fleet of application processes producing to and consuming from our Kafka clusters. This fleet generates an atypical workload on Kafka that warrants a deeper investment in scalability and reliability. This talk discusses our investments in Kafka infrastructure for a large-scale Python-based environment: kafkahood: our librdkafka-based client library wrapper that codifies best practices, sane defaults and deep client-side observability. kafkaproxy: a Rust-based sidecar proxy that reduces connection fan-in from Python gunicorn worker pools to our Kafka clusters. We'll also present challenges we encountered along the way and share our learnings with the audience.

Taming a massive fleet of Python-based Kafka apps at Robinhood | Chandra Kuch...

HostedbyConfluent

Speaker: Gabriel Schenker, Lead Curriculum Developer, Confluent Streaming platforms have emerged as a popular, new trend, but what exactly is a streaming platform? Part messaging system, part Hadoop made fast, part fast ETL and scalable data integration. With Apache Kafka® at the core, event streaming platforms offer an entirely new perspective on managing the flow of data. This talk will explain what an event streaming platform such as Apache Kafka is and some of the use cases and design patterns around its use—including several examples of where it is solving real business problems. New developments in this area such as KSQL will also be discussed.

What is Apache Kafka and What is an Event Streaming Platform?

confluent

How to over-engineer things and have fun? | Oto Brglez, OPALAB

HostedbyConfluent

The auto industry is midst a data revolution that is transforming how companies do business. Once a scarce resource, data has now become abundant and cheap. What are the new technologies that change the way we produce, collect, process, store, and analyze data. What new streams of data are being created with Industry 4.0 and the Internet of Things on the horizon, is there significant value to taking a strategic approach to Fast Data. How is Porsche building the next level Data Streaming Platform with open source technologies and how we are using CI/CD pipelines amongst others in order to serve our use cases.

0-330km/h: Porsche's Data Streaming Journey | Sridhar Mamella, Porsche

HostedbyConfluent

My team at Zalando fell in love with KStreams and their programming model straight out of the gate. However, as a small team of developers, building out and supporting our infrastructure while still trying to deliver solutions for our business has not always resulted in a smooth journey. Can a small team of a couple of developers run their own Kafka infrastructure confidently and still spend most of their time developing code? In this talk, we will dive into some of the problems we experienced while running Kafka brokers and Kafka streams applications, as well as the consultations we had with other teams around this matter. We will outline some of the pragmatic decisions we made regarding backups, monitoring and operations to minimize our time spent administering our Kafka brokers and various stream applications.

War Stories: DIY Kafka

confluent

When we set out to launch our Conversations Platform at Expedia Group our goals were simple. Enable millions of travelers to have natural language conversations with an automated agent via text, Facebook, or their channel of choice. Let them book trips, make changes or cancellations, and ask questions -- “How long is my layover?” “Does my hotel have a pool?” How much will I get charged if I want to bring my golf clubs?”. Then take all that we know about that customer across all of our brands and apply machine learning models to give customers what they are looking for immediately and automatically, whether it be a straightforward answer or a complex new itinerary. And the final goal: go from zero to production in four months. Such a platform is no place for batch jobs, back-end processing, or offline APIs. To quickly make decisions that incorporate contextual information, the platform needs data in near real-time and it needs it from a wide range of services and systems. Meeting these needs meant architecting the Conversations Platform around a central nervous system based on Confluent Cloud and Apache Kafka. Kafka made it possible to orchestrate data from loosely coupled systems, enrich data as it flows between them so that by the time it reaches its destination it is ready to be acted upon, and surface aggregated data for analytics and reporting. Confluent Cloud made it possible for us to meet our tight launch deadline with limited resources. With event streaming as a managed service, we had no costly new hires to maintain our clusters and no worries about 24x7 reliability. When we built the platform, we did not foresee the worldwide pandemic and the profound effect it would have on the travel industry. Companies were hit with a tidal wave of customer questions, cancellations, and rebookings. Throughout this once-in-a-lifetime event, the Conversations Platform proved up to the challenge, auto-scaling as necessary and taking much of the load off of live agents. In this session, we’ll share how we built and deployed the Conversations Platform in just four months, the lessons we learned along the way, key points to consider for anyone architecting a platform with similar requirements, and how it handled the unprecedented demands placed upon it by the pandemic. We’ll also show a demo of the platform that includes high-level insights obtained from analytics and a visualization of the low-level events that make up a conversation.

Launching the Expedia Conversations Platform: From Zero to Production in Four...

HostedbyConfluent

For many industries the need to group together related events based on a period of activity or inactivity is key. Advertising businesses, content producers are just a few examples of where session windows can be used to better understand user behavior. While such sessionization has been possible in Apache Kafka up to this point, implementing it has been rather complex and required leveraging low-level APIs. In the most recent release of Kafka, however, new capabilities have been added making session windows much easier to implement. In this online talk, we’ll introduce the concept of a session window, talk about common use cases, and walk through how Apache Kafka can be used for session-oriented use cases.

user Behavior Analysis with Session Windows and Apache Kafka's Streams API

confluent

Guru Sattanathan, Confluent, Senior Solutions Engineer Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats! Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts? If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today. In this session, we will go in this order: -Recap on Enterprise Integration technologies -What are the key flaws & What needs improvement? -What is Apache Kafka? -Rethinking Integration using Apache Kafka https://www.meetup.com/KafkaMelbourne/events/280590162/

Death of the dumb pipes: Using Apache Kafka® for Integration projects

HostedbyConfluent

You have learned about Kafka event sourcing with streams and using Kafka as a database, but you may be having a tough time wrapping your head around what that means and what challenges you will face. Kafka’s exactly once semantics, data retention rules, and stream DSL make it a great database for real-time transaction processing. This talk will focus on how to use Kafka events as a database. We will talk about using KTables vs GlobalKTables, and how to apply them to patterns we use with traditional databases. We will go over a real-world example of joining events against existing data and some issues to be aware of. We will finish covering some important things to remember about state stores, partitions, and streams to help you avoid problems when your data sets become large.

Using Kafka as a Database For Real-Time Transaction Processing | Chad Preisle...

HostedbyConfluent

Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations. When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce. In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.

Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik

HostedbyConfluent

Streaming architectures have been on the rise steadily and as a result, we have seen the adoption of Kafka go up too. With the diverse spread of use cases across multiple industries, we have seen a variety of Kafka deployments across our hundreds of Kafka customers. Along the way, we have learnt some best practices as well as what not to do in mission-critical architectures. Join Joe Niemiec, Sr. Product Manager at Cloudera, as he shares these insights in this session that covers topics such as - The many ways that Kafka has been deployed in the field Standalone clusters, multiple clusters in a single data center and multiple clusters geographically distributed performing replication Clusters of all sizes small and large, few messages to hundreds of thousands per second Discussion about architecture failure domains Configurations tuned and used in specific deployments

Lessons from the field: Catalog of Kafka Deployments | Joseph Niemiec, Cloudera

HostedbyConfluent

Scaling a backend for a big data and blockchain environment by Rafael Ríos at...

Big Data Spain

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

HostedbyConfluent

Time series data is everywhere -- connected IoT devices, application monitoring & observability platforms, and more. What makes time series datastreams challenging is that they often have orders of magnitude more data than other workloads, with millions of time series datapoints being quite common. Given its ability to ingest high volumes of data, Kafka is a natural part of any data architecture handling large volumes of time series telemetry, specifically as an intermediate buffer before that data is persisted in InfluxDB for processing, analysis, and use in other applications. In this session, we will show you how you can stream time series data to your IoT application using Kafka queues and InfluxDB, drawing upon deployments done at Hulu and Wayfair that allow both to ingest 1 million metrics per second. Once this session is complete, you’ll be able to connect a Kafka queue to an InfluxDB instance as the beginning of your own time series data pipeline.

Stream processing IoT time series data with Kafka & InfluxDB | Al Sargent, In...

HostedbyConfluent

A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff ...

HostedbyConfluent

As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

HostedbyConfluent

As an AWS shop, Zillow engineering teams have been using various messaging and streaming services for years. As Zillow 2.0 piled through, new requirements and pain points made us rethink our streaming stack. The need for high data quality, decoupling producers & consumers and real time homes data called for a new platform which would empower developers, enable data governance and reduce incidents caused by bad data. In this session, you will learn why Zillow decided to go with Kafka for that platform, what tools we built to meet developers where they are and what common challenges you could face as you migrate other streaming solutions to Kafka.

How Zillow Unlocked Kafka to 50 Teams in 8 months | Shahar Cizer Kobrinsky, Z...

HostedbyConfluent

APAC Kafka Summit - Best Of

confluent

Tendances (20)