Real-Time Dynamic Data Export Using the Kafka Ecosystem
0 j'aime
vues
Check these out next
Real-Time Dynamic Data Export Using the Kafka Ecosystem
0 j'aime
vues
(Preston Thompson, Braze) Kafka Summit SF 2018 If you collect billions of data points every day and create billions more sending and tracking messages, then you know you need to get your infrastructure right. Our clients use Braze to engage their users over their lifecycle via push notifications, emails, in-app messages and more. Using our Currents product, clients can enable multiple configurable integrations to export this event data in real time to a variety of third-party systems, allowing them to tightly integrate with the rest of their operations and understand the impacts of their engagement strategy. We use Kafka and the Kafka ecosystem to power this high volume real-time export. As you’d expect in a big data environment, we take data collected from a variety of sources—our SDKs, email partner APIs, our own systems—and produce it to Kafka, with topics for each type of event (about 30 types). Kafka Streams filters and transforms this data according to the configurations set by our clients. Clients can choose which types of events should be sent to which third-party systems. Kafka Connect helps to export the data to third-party systems in real time using custom developed connectors. We run a connector instance for each integration for each customer that consumes from the integration-specific topic. On top of it all, we built a service to manage the pipeline. The service provides configurations to the Streams application and also creates topics for new integrations and uses the Connect REST API to create and manage connectors. In this talk, I will discuss: -How we started our journey in designing this large-scale streaming architecture -Why streaming technologies were necessary to solve our technology and business issues -The lessons we learned along the way that can help you with your Kafka-based architecture
Recommandé
Real-Time Dynamic Data Export Using the Kafka Ecosystem
- Real-Time Dynamic Data Export Using the Kafka Ecosystem
- 2 Product Overview What were we trying to build? Architecture How did we build it? What did we learn building it? Today
- About Me • Preston Thompson • Senior Software Engineer • 4 years at Braze • Backend Application Developer • Data Infrastructure • Application Infrastructure
- More than 1 Billion MAU ON SIX CONTINENTS
- 1.5B+ MAU 30B+ EVENTS PER DAY Scale
- 8 TOC Product Overview
- Currents • Real time data export • Customers can create one or more exports to seven different partner destinations • Data Warehouse - AWS S3, Azure Blob, Google Cloud Storage • Behavioral Analytics - Amplitude, Mixpanel • Customer Data Platform - mParticle, Segment • 30 different event types • Message Engagement Events • Customer Behavior Events • 200+ active integrations • ~1B events exported per day
- 1 2 TOC Architecture + Lessons Learned
- Events • Ruby applications producing to Kafka using ruby-kafka • All events are actions related to a specific end-user • Push Notification Send • Push Notification Open • Campaign Conversion • Purchase • Custom Event • 30 different events types, one topic each • Events for all Braze customers are mixed together within a partition • Use user ID for key to guarantee balanced partitions
- Filter and Transform • Requirements • Events types are configurable per integration • 7 different destinations • REST API • Object storage • Solution • Kafka Streams application • Input topics = event topics • Output topics = integration-specific topics • Configuration file storing integration settings (e.g. which events to send, anonymous users)
- Denormalization • Events have lots of IDs to reference other items • Campaigns • Message Variations • Apps • Names can be nice in some of the destinations • Example: Amplitude dashboard when selecting campaign • Transform • New topic with database changes • Global State Store
- Connect • One topic per integration • Independent processing • Invalid credentials • Partner downtime • Rate limiting • Difficult because we must limit number of partitions per connector • Rebalance loops • Increase number of hosts • Future - maybe split integrations across many connect clusters • REST API • Automatically restart failed tasks • Manage active connectors - new and recently updated • Scale number of active tasks when needed
- Custom Connectors • HTTP Connectors • Try to batch the best we can • Retries are difficult • Retry immediately a few times • If that fails, throw RetriableException • Exponential retry not built in • Object storage connectors • S3, Azure Blob, GCS • Built on top of Confluent S3 Connector using pluggable classes • Inner Avro format • Different credentials per connector
- Volume Metrics • Counts the number of events exported • Kafka Streams application • Consumes integration-specific topics • Uses a simple aggregator • Requires state, so a bit more complicated • Interactive Queries
- Misato • Manages applications to be in the desired state of the system • Creates a configuration file for Streams • Restarts Streams to pick up changes to that file • Creates new topics for integrations • Creates new connectors to read from those topics • Updates configurations for connectors • Manages topics read by Volume Metrics app
- 2 7 TOC Quick Recap
- Thanks! • Team at Braze • We’re hiring! • braze.com/careers • Confluent • Arcadia Data