Apache Flink and what it is used for

•Télécharger en tant que PPTX, PDF•

0 j'aime•1,620 vues

Aljoscha Krettek is the PMC chair of Apache Flink and Apache Beam, and co-founder of data Artisans. Apache Flink is an open-source platform for distributed stream and batch data processing. It allows for stateful computations over data streams in real-time and historically. Flink supports batch and stream processing using APIs like DataSet and DataStream. Data Artisans originated Flink and provides an application platform powered by Flink and Kubernetes for building stateful stream processing applications.

Technologie

Aljoscha Krettek – PMC of Apache Flink and Apache Beam, Co-Founder at data Artisans
Apache Flink® and what it is used for

© 2018 data Artisans2
About Data Artisans
Original Creators of
Apache Flink®
RealTime Stream Processing
Enterprise Ready
2008 2009 2011
Studying and working at IBM
Development on Stratosphere begins
I joinTU Berlin research group with other co-founders
2014
data Artisans is founded

© 2018 data Artisans3
What is Apache Flink?
Queries
Applications
Devices
etc.
Database
Stream
File / Object
Storage
Historic
Data
Streams
Application
Stateful computations over streams
real-time and historic
fast, scalable, fault tolerant, in-memory,
event time, large state, exactly-once

© 2018 data Artisans4
Overview of Flink Use Cases
Event-driven
applications
Stream
Processing
Batch
processing

© 2018 data Artisans5
Batch Processing
a.k.a. collect now, figure
out later

© 2018 data Artisans6
Batch Processing Technologies Supported by Flink
DataSet
API

© 2018 data Artisans7
Observation 1
The origin of data are streams

© 2018 data Artisans8
Stream Processing
we can build applications directly
on data streams

© 2018 data Artisans9
Stream Processing Technologies Supported by Flink
DataStream
API

© 2018 data Artisans10
Observation 2
Stream Processing changes the database-centric
architecture

© 2018 data Artisans11
Changing the Two-Tier Architecture
reads/writes across
tier boundary
asynchronous writes
of large blobs
all modifications
are local
Classic tiered architecture Streaming architectureClassic tiered architecture
database
layer
compute
layer
application working state
+ historic state
compute
+
stream storage
and
snapshot storage
(backup)
application state
Streaming architecture

© 2018 data Artisans12
Keystone Routing Pipeline at Netflix
(as presented at Flink Forward San Francisco, 2018)
@
 Athena X
 SQL to define metrics
 Thresholds and actions to trigger
 Blends analytics and
actions
Streams from
Hadoop, Kafka, etc
SQL, thresholds,
actions
Analytics
Alerts
Derived streams

© 2018 data Artisans13
Observation 3
Stream Processing is about building applications,
not platforms

© 2018 data Artisans14
Internal streaming data platforms
built with Apache Flink

© 2018 data Artisans15
KubernetesResource Manager
Logging
Metrics
CI / CD
Application Platforms
deploying new
applications
scaling
applications
Kubernetes

© 2018 data Artisans16
Kubernetes
Database
Kubernetes
• Example: Scaling down a replicated database
• 3 replicas, 4 node scale down
need to move or
reorganize data
before container
shutdown
Kubernetes & stateful applications What about stateful containers?

© 2018 data Artisans17
• consistent stateful upgrades
‒application evolution and bug fixes
• migration of application state
‒cluster migration, A/B testing
• re-processing and reinstatement
‒fix corrupt results, bootstrap new applications
• state evolution (schema evolution)
A B
Stateful Questions
Container-based
Resource Orchestration
Stateful Stream
Processing & Snapshots
Kubernetes Apache Flink
Container-based
platform for stateful
data-driven applications
dA Platform
Code, Resource, Config, and
Snapshot Management
Application
Manager

© 2018 data Artisans18
Architecture
Apache Flink
Stateful stream processing
Kubernetes
Container platform
Logging
Metrics
dA Application
Manager
Application
lifecycle
management
Storage
In Action
App
Manager
Kubernetes
Resource
Allocation
Job Control
Snapshot
Management
CI/CDWeb
interface

© 2018 data Artisans19
Powered by Apache Flink

© 2018 data Artisans20
Free Trial Download of data Artisans Platform
data-artisans.com/download

THANK YOU! WE ARE HIRING
data-artisans.com/careers
@aljoscha
@dataArtisans
@ApacheFlink

Contenu connexe

Tendances

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Autoscaling Flink with Reactive Mode

Flink Forward

Flexible and Real-Time Stream Processing with Apache Flink

DataWorks Summit

A Thorough Comparison of Delta Lake, Iceberg and Hudi

Databricks

Introduction to Apache Flink

mxmxm

Running Apache NiFi with Apache Spark : Integration Options

Timothy Spann

Apache Flink is the most popular and widely adopted streaming processing framework, powering real time stream event computations at extremely large scale in companies like Uber, Lyft, AWS, Alibaba, Pinterest, Splunk, Yelp, etc. In this talk, we will go over use cases and basic (yet hard to achieve!) requirements of stream processing, and how Flink fills the gaps and stands out with some of its unique core building blocks, like pipelined execution, native event time support, state support, and fault tolerance. We will also take a look at how Flink is going beyond stream processing into areas like unified data processing, enterprise intergration, AI/machine learning (especially online ML), and serverless computation, and how Flink fits with its distinct value. SPEAKER: Bowen Li SPEAKER BIO: Bowen is a committer of Apache Flink, senior engineer at Alibaba, and host of Seattle Flink Meetup.

Apache Flink 101 - the rise of stream processing and beyond

Bowen Li

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Flink Forward

Apache Flink Stream Processing

Suneel Marthi

Fundamentals of Apache Kafka

Chhavi Parasher

Kafka Streams is a new stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a natural DSL for writing stream processing applications. As such it is the most convenient yet scalable option to analyze, transform, or otherwise process data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Spark Streaming or Storm, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka.

Introduction to Kafka Streams

Guozhang Wang

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

Introducing the Apache Flink Kubernetes Operator

Flink Forward

Real-time data beats slow data. That’s true for almost every use case. Nevertheless, enterprise architects build new infrastructures with the Lambda architecture that includes separate batch and real-time layers. This video explores why a single real-time pipeline, called Kappa architecture, is the better fit for many enterprise architectures. Real-world examples from companies such as Disney, Shopify, Uber, and Twitter explore the benefits of Kappa but also show how batch processing fits into this discussion positively without the need for a Lambda architecture. The main focus of the discussion is on Apache Kafka (and its ecosystem) as the de facto standard for event streaming to process data in motion (the key concept of Kappa), but the video also compares various technologies and vendors such as Confluent, Cloudera, IBM Red Hat, Apache Flink, Apache Pulsar, AWS Kinesis, Amazon MSK, Azure Event Hubs, Google Pub Sub, and more. Video recording of this presentation: https://youtu.be/j7D29eyysDw Further reading: https://www.kai-waehner.de/blog/2021/09/23/real-time-kappa-architecture-mainstream-replacing-batch-lambda/ https://www.kai-waehner.de/blog/2021/04/20/comparison-open-source-apache-kafka-vs-confluent-cloudera-red-hat-amazon-msk-cloud/ https://www.kai-waehner.de/blog/2021/05/09/kafka-api-de-facto-standard-event-streaming-like-amazon-s3-object-storage/

Kappa vs Lambda Architectures and Technology Comparison

Kai Wähner

Apache kafka

NexThoughts Technologies

Running Apache Spark on Kubernetes: Best Practices and Pitfalls

Databricks

Apache Kafka Security

DataWorks Summit/Hadoop Summit

Flink Streaming

Gyula Fóra

Flink Forward San Francisco 2022. In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you! by Olena Babenko

Tuning Apache Kafka Connectors for Flink.pptx

Flink Forward

Speaker: Matthias J. Sax, Software Engineer, Confluent ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries. Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems. https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/

ksqlDB: A Stream-Relational Database System

confluent

Apache kafka 관리와 모니터링

JANGWONSEO4

Securing Kafka

confluent

Tendances (20)

Autoscaling Flink with Reactive Mode

Flexible and Real-Time Stream Processing with Apache Flink

A Thorough Comparison of Delta Lake, Iceberg and Hudi

Introduction to Apache Flink

Running Apache NiFi with Apache Spark : Integration Options

Apache Flink 101 - the rise of stream processing and beyond

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...

Apache Flink Stream Processing

Fundamentals of Apache Kafka

Introduction to Kafka Streams

Introducing the Apache Flink Kubernetes Operator

Kappa vs Lambda Architectures and Technology Comparison

Apache kafka

Running Apache Spark on Kubernetes: Best Practices and Pitfalls

Apache Kafka Security

Flink Streaming

Tuning Apache Kafka Connectors for Flink.pptx

ksqlDB: A Stream-Relational Database System

Apache kafka 관리와 모니터링

Securing Kafka

Similaire à Apache Flink and what it is used for

Stream processing still evolves and changes at a speed that can make it hard to keep up with the developments. Being at the forefront of stream processing technology, the evolution of Apache Flink has mirrored many of these developments and continues to do so. We will take you on a journey through the major milestones of stream processing technology in past years, diving into the latest additions that Apache Flink and other communities introduced to the stream processing landscape, such as Streamng SQL, Time Versioned Tables, cluster-library-duality, language portability, etc. We will take a sneak peek into our crystal ball and present in what the Flink community is working on next.

The Past, Present, and Future of Apache Flink®

Aljoscha Krettek

(Past), Present, and Future of Apache Flink

Aljoscha Krettek

dA Platform is a production-ready platform for stream processing with Apache Flink®. The Platform includes open source Apache Flink, a stateful stream processing and event-driven application framework, and dA Application Manager, a central deployment and management component. dA Platform schedules clusters on Kubernetes, deploys stateful Flink applications, and controls these applications and their state.

dA Platform Overview

Robert Metzger

In this talk, we are going to present dA Platform, a production-ready platform for stream processing with Apache Flink® from data Artisans. The platform includes open source Apache Flink and Application Manager, a central deployment and management component. dA Platform schedules clusters on Kubernetes, deploys stateful Flink applications, and controls these applications and their state. We will look at how dA Platform makes stream processing easier for enterprises and explain how it works. The talk will also include a demonstration of dA Platform’s capabilities.

Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...

Flink Forward

The Past, Present, and Future of Apache Flink

Aljoscha Krettek

Flink Forward Berlin 2018: Aljoscha Krettek & Till Rohrmann - Keynote: "A Yea...

Flink Forward

“The mainframe is going away” is as true now as it was 10, 20 and 30 years ago. Mainframes are still crucial in handling critical business transactions, they were however built for an era where batch data movement was the norm and can be difficult to integrate into today’s data-driven, real-time, analytics-focused business processes as well as the environments that support them. Until now. Join experts from Confluent, Attunity, and Capgemini for a one-hour online talk session where you’ll learn how to: Unlock your mainframe data with unique change data capture (CDC) functionality without incurring the complexity and expense that come with sending ongoing queries into the mainframe database How using CDC benefits advanced analytics approaches such as deep machine learning and predictive analytics Deliver ongoing streams of data in real-time to the most demanding analytics environments Ensure that your analytics environment includes the broadest possible range of data sources and destinations while ensuring true enterprise-grade functionality Identify use cases that can help you get started delivering value to the business moving from POC to Pilot to Production

Leveraging Mainframe Data for Modern Analytics

confluent

Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...

VMware Tanzu

Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. In this session, we first present an end-to-end streaming data solution using Amazon Kinesis Data Streams for data ingestion, Amazon Kinesis Data Analytics for real-time processing, and Amazon Kinesis Data Firehose for persistence. We review in detail how to write SQL queries for operational monitoring using Kinesis Data Analytics. Learn how PNNL is building their ingestion flow into their Serverless Data Lake leveraging the Kinesis Platform. At times migrating existing NiFi Processes where applicable to various parts of the Kinesis Platform, replacing complex flows on Nifi to bundle and compress the data with Kinesis Firehose, leveraging Kinesis Streams for their enrichment and transformation pipelines, and using Kinesis Analytics to Filter, Aggregate, and detect anomalies.

Analyzing Data Streams in Real Time with Amazon Kinesis: PNNL's Serverless Da...

Amazon Web Services

Aljoscha Krettek offers an overview of the modern stream processing space, details the challenges posed by stateful and event-time-aware stream processing, and shares core archetypes ("application blueprints”) for stream processing drawn from real-world use cases with Apache Flink. Topics include: * Aggregating IoT event data, in which event-time-aware processing, handling of late data, and state are important * Data enrichment, in which a stream of real-time events is “enriched” with data from a slowly changing database of supplemental data points * Dynamic stream processing, in which a stream of control messages and dynamically updated user logic is used to process a stream of events for use cases such as alerting and fraud detection

Stream processing for the practitioner: Blueprints for common stream processi...

Aljoscha Krettek

Date: 14th November 2018 Location: Fast Data Theatre Time: 12:30 - 13:00 Speaker: Konstantin Knauf Organisation: Data Artisans About: In this session we will discuss how stream processing is evolving from a technology in the sidelines of Big Data to a technology that covers real-time computations, ACID transactions on event data, and also completely subsume batch processing through the work that data Artisans and other prominent members of the Flink community have been reporting on. In addition, we will discuss how stream processing is transforming businesses to adopt a more customer-oriented, real-time, and data-driven mindset resulting in realized business transformations

Big Data LDN 2018: STREAM PROCESSING TAKES ON EVERYTHING

Matt Stubbs

This talk is an application-driven walkthrough to modern stream processing, exemplified by Apache Flink, and how this enables new applications and makes old applications easier and more efficient. In this talk, we will walk through several real-world stream processing application scenarios of Apache Flink, highlighting unique features in Flink that make these applications possible. In particular, we will see (1) how support for handling out of order streams enables real-time monitoring of cloud infrastructure, (2) how the ability handle high-volume data streams with low latency SLAs enables real-time alerts in network equipment, (3) how the combination of high throughput and the ability to handle batch as a special case of streaming enables an architecture where the same exact program is used for real-time and historical data processing, and (4) how stateful stream processing can enable an architecture that eliminates the need for an external database store, leading to more than 100x performance speedup, among many other benefits.

Streaming in the Wild with Apache Flink

Kostas Tzoumas

Confluent kafka meetupseattle jan2017

Nitin Kumar

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

Hortonworks

Overview of Apache Flink: the 4G of Big Data Analytics Frameworks

DataWorks Summit/Hadoop Summit

Slides of my talk at the Hadoop Summit Europe in Dublin, Ireland on April 13th, 2016. The talk introduces Apache Flink as both a multi-purpose Big Data analytics framework and real-world streaming analytics framework. It is focusing on Flink's key differentiators and suitability for streaming analytics use cases. It also shows how Flink enables novel use cases such as distributed CEP (Complex Event Processing) and querying the state by behaving like a key value data store.

Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks

Slim Baltagi

Overview of Apache Fink: The 4G of Big Data Analytics Frameworks

Slim Baltagi

This face to face talk about Apache Flink in Sao Paulo, Brazil is the first event of its kind in Latin America! It explains how Apache Flink 1.0 announced on March 8th, 2016 by the Apache Software Foundation (link), marks a new era of Big Data analytics and in particular Real-Time streaming analytics. The talk maps Flink's capabilities to real-world use cases that span multiples verticals such as: Financial Services, Healthcare, Advertisement, Oil and Gas, Retail and Telecommunications. In this talk, you learn more about: 1. What is Apache Flink Stack? 2. Batch vs. Streaming Analytics 3. Key Differentiators of Apache Flink for Streaming Analytics 4. Real-World Use Cases with Flink for Streaming Analytics 5. Who is using Flink? 6. Where do you go from here?

Apache Flink: Real-World Use Cases for Streaming Analytics

Slim Baltagi

Are you interested in learning how to schedule batch jobs in container runtimes? Maybe you’re wondering how to apply continuous delivery in practice for data-intensive applications? Perhaps you’re looking for an orchestration tool for data pipelines? Questions like these are common, so rest assured that you’re not alone. In this webinar, we’ll cover the recent feature improvements in Spring Cloud Data Flow. More specifically, we’ll discuss data processing use cases and how they simplify the overall orchestration experience in cloud runtimes like Cloud Foundry and Kubernetes. Please join us and be part of the community discussion! Presenters : Sabby Anandan, Product Manager Mark Pollack, Software Engineer, Pivotal

Cloud-Native Patterns for Data-Intensive Applications

VMware Tanzu

xGem Data Stream Processing

Jorge Hirtz

Similaire à Apache Flink and what it is used for (20)

The Past, Present, and Future of Apache Flink®

(Past), Present, and Future of Apache Flink

dA Platform Overview

Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...

The Past, Present, and Future of Apache Flink

Flink Forward Berlin 2018: Aljoscha Krettek & Till Rohrmann - Keynote: "A Yea...

Leveraging Mainframe Data for Modern Analytics

Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...

Analyzing Data Streams in Real Time with Amazon Kinesis: PNNL's Serverless Da...

Stream processing for the practitioner: Blueprints for common stream processi...

Big Data LDN 2018: STREAM PROCESSING TAKES ON EVERYTHING

Streaming in the Wild with Apache Flink

Confluent kafka meetupseattle jan2017

Hortonworks DataFlow (HDF) 3.3 - Taking Stream Processing to the Next Level

Overview of Apache Flink: the 4G of Big Data Analytics Frameworks

Overview of Apache Fink: the 4 G of Big Data Analytics Frameworks

Overview of Apache Fink: The 4G of Big Data Analytics Frameworks

Apache Flink: Real-World Use Cases for Streaming Analytics

Cloud-Native Patterns for Data-Intensive Applications

xGem Data Stream Processing

Plus de Aljoscha Krettek

In diesem Vortrag wird es zunächst einen kurzen Überblick über den aktuellen Stand im Bereich der Streaming-Datenanalyse geben. Danach wird es mit einer kleinen Einführung in das Apache-Flink-System zur Echtzeit-Datenanalyse weitergehen, bevor wir tiefer in einige der interessanten Eigenschaften eintauchen werden, die Flink von den anderen Spielern in diesem Bereich unterscheidet. Dazu werden wir beispielhafte Anwendungsfälle betrachten, die entweder direkt von Nutzern stammen oder auf unserer Erfahrung mit Nutzern basieren. Spezielle Eigenschaften, die wir betrachten werden, sind beispielsweise die Unterstützung für die Zerlegung von Events in einzelnen Sessions basierend auf der Zeit, zu der ein Ereignis passierte (event-time), Bestimmung von Zeitpunkten zum jeweiligen Speichern des Zustands eines Streaming-Programms für spätere Neustarts, die effiziente Abwicklung bei sehr großen zustandsorientierten Streaming-Berechnungen und die Zugänglichkeit des Zustandes von außerhalb.

Apache Flink(tm) - A Next-Generation Stream Processor

Aljoscha Krettek

Flink is a great stream processor, Python is a great programming language, Apache Beam is a great programming model and portability layer. Using all three together is a great idea! We will demo and discuss writing Beam Python pipelines and running them on Flink. We will cover Beam's portability vision that led here, what you need to know about how Beam Python pipelines are executed on Flink, and where Beam's portability framework is headed next (hint: Python pipelines reading from non-Python connectors)

Talk Python To Me: Stream Processing in your favourite Language with Beam on ...

Aljoscha Krettek

Beginning with MapReduce and its first popular open-source implementation in Apache Hadoop the data processing landscape has evolved quite a bit. Since then we have seen several paradigm shifts and open-source systems evolved to support new types of applications and to attract new audiences. We will follow developments using the example of the open-source stream processing system Apache Flink and in the end we will see how expressive APIs, support for event-driven applications, Flink SQL for seamless batch and stream processing, and a powerful runtime enable a wide range of applications.

The Evolution of (Open Source) Data Processing

Aljoscha Krettek

Python Streaming Pipelines with Beam on Flink

Aljoscha Krettek

Aljoscha Krettek offers a very short introduction to stream processing before diving into writing code and demonstrating the features in Apache Flink that make truly robust stream processing possible, with a focus on correctness and robustness in stream processing. All of this will be done in the context of a real-time analytics application that we’ll be modifying on the fly based on the topics we’re working though, as Aljoscha exercises Flink’s unique features, demonstrates fault recovery, clearly explains why event time is such an important concept in robust, stateful stream processing, and covers the features you need in a stream processor to do robust, stateful stream processing in production. We’ll also use a real-time analytics dashboard to visualize the results we’re computing in real time, allowing us to easily see the effects of the code we’re developing as we go along. Topics include: * Apache Flink * Stateful stream processing * Event time versus processing time * Fault tolerance * State management in the face of faults * Savepoints * Data reprocessing

Robust stream processing with Apache Flink

Aljoscha Krettek

Apache Beam lets you process unbounded, out-of-order, global-scale data with portable high-level pipelines, but not all use cases are pipelines of simple “map” and “combine” operations. Aljoscha Krettek introduces Beam’s new State API, which brings scalability and consistency to fine-grained stateful processing while interoperating with Beam’s other features such as consistent event-time windowing and windowed side inputs—all while remaining portable to any Beam runner, including Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow. Aljoscha covers the new state and timer features in Beam and shows how to use them to express common real-world use cases in a backend-agnostic manner. Examples of new use cases unlocked by Beam’s new mutable state and timers include: * Microservice-like streaming applications such as new user account verification and digital ordering * Complex aggregations that cannot easily be expressed as an efficient associative combiner * Output based on customized conditions, such as limiting to only “significant” changes in a learned model (resulting in potentially large cost savings in subsequent processing) * Fine control over retrieval and storage of intermediate values during aggregation * Reading from and writing to external systems with detailed management of the nature and size of requests

Unified stateful big data processing in Apache Beam (incubating)

Aljoscha Krettek

Advanced Flink Training - Design patterns for streaming applications

Aljoscha Krettek

Apache Flink - A Stream Processing Engine

Aljoscha Krettek

If you are in the business of processing a stream of events you sooner or later come upon different notions of time. There is processing time, the current time of the machine your program is running on and event time, the local time at which an event occurred. In this talk we will look at why this distinction is relevant and also how Flink manages to work with these different ideas of time. We will look at how Flink tracks the progress of time and how you can employ windows to perform aggregating operations on an infinite stream of events.

Adventures in Timespace - How Apache Flink Handles Time and Windows

Aljoscha Krettek

Flink 0.10 - Upcoming Features

Aljoscha Krettek

Data Analysis with Apache Flink (Hadoop Summit, 2015)

Aljoscha Krettek

Apache Flink Hands-On

Aljoscha Krettek

Plus de Aljoscha Krettek (12)

Apache Flink(tm) - A Next-Generation Stream Processor

Talk Python To Me: Stream Processing in your favourite Language with Beam on ...

The Evolution of (Open Source) Data Processing

Python Streaming Pipelines with Beam on Flink

Robust stream processing with Apache Flink

Unified stateful big data processing in Apache Beam (incubating)

Advanced Flink Training - Design patterns for streaming applications

Apache Flink - A Stream Processing Engine

Adventures in Timespace - How Apache Flink Handles Time and Windows

Flink 0.10 - Upcoming Features

Data Analysis with Apache Flink (Hadoop Summit, 2015)

Apache Flink Hands-On

Dernier

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Bhuvaneswari Subramani

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

Discover the innovative features and strategic vision that keep WSO2 an industry leader. Explore the exciting 2024 roadmap of WSO2 API management, showcasing innovations, unified APIM/APK control plane, natural language API interaction, and cloud native agility. Discover how open source solutions, microservices architecture, and cloud native technologies unlock seamless API management in today's dynamic landscapes. Leave with a clear blueprint to revolutionize your API journey and achieve industry success!

WSO2's API Vision: Unifying Control, Empowering Developers

WSO2

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

FWD Group - Insurer Innovation Award 2024

The Digital Insurer

Accelerating FinTech Innovation: Unleashing API Economy and GenAI Vasa Krishnan, Chief Technology Officer - FinResults Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

apidays

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Vector Search -An Introduction in Oracle Database 23ai.pptx

Remote DBA Services

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Orbitshub

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

Following the popularity of “Cloud Revolution: Exploring the New Wave of Serverless Spatial Data,” we’re thrilled to announce this much-anticipated encore webinar. In this sequel, we’ll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you’re building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

The microservices honeymoon is over. When starting a new project or revamping a legacy monolith, teams started looking for alternatives to microservices. The Modular Monolith, or 'Modulith', is an architecture that reaps the benefits of (vertical) functional decoupling without the high costs associated with separate deployments. This talk will delve into the advantages and challenges of this progressive architecture, beginning with exploring the concept of a 'module', its internal structure, public API, and inter-module communication patterns. Supported by spring-modulith, the talk provides practical guidance on addressing the main challenges of a Modultith Architecture: finding and guarding module boundaries, data decoupling, and integration module-testing. You should not miss this talk if you are a software architect or tech lead seeking practical, scalable solutions. About the author With two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Victor Rentea

In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.

Platformless Horizons for Digital Adaptability

WSO2

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

DBX First Quarter 2024 Investor Presentation

Dropbox

Retrieval augmented generation (RAG) is the most popular style of large language model application to emerge from 2023. The most basic style of RAG works by vectorizing your data and injecting it into a vector database like Milvus for retrieval to augment the text output generated by an LLM. This is just the beginning. One of the ways that we can extend RAG, and extend AI, is through multilingual use cases. Typical RAG is done in English using embedding models that are trained in English. In this talk, we’ll explore how RAG could work in languages other than English. We’ll explore French, Chinese, and Polish.

Introduction to Multilingual Retrieval Augmented Generation (RAG)

Zilliz

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

Dernier (20)

Elevate Developer Efficiency & build GenAI Application with Amazon Q

Apidays New York 2024 - The value of a flexible API Management solution for O...

Strategies for Landing an Oracle DBA Job as a Fresher

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

WSO2's API Vision: Unifying Control, Empowering Developers

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

FWD Group - Insurer Innovation Award 2024

Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...

AWS Community Day CPH - Three problems of Terraform

Corporate and higher education May webinar.pptx

Vector Search -An Introduction in Oracle Database 23ai.pptx

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Artificial Intelligence Chap.5 : Uncertainty

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024

Platformless Horizons for Digital Adaptability

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

DBX First Quarter 2024 Investor Presentation

Introduction to Multilingual Retrieval Augmented Generation (RAG)

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

Apache Flink and what it is used for

1. Aljoscha Krettek – PMC of Apache Flink and Apache Beam, Co-Founder at data Artisans Apache Flink® and what it is used for

2. © 2018 data Artisans2 About Data Artisans Original Creators of Apache Flink® RealTime Stream Processing Enterprise Ready 2008 2009 2011 Studying and working at IBM Development on Stratosphere begins I joinTU Berlin research group with other co-founders 2014 data Artisans is founded

3. © 2018 data Artisans3 What is Apache Flink? Queries Applications Devices etc. Database Stream File / Object Storage Historic Data Streams Application Stateful computations over streams real-time and historic fast, scalable, fault tolerant, in-memory, event time, large state, exactly-once

11. © 2018 data Artisans11 Changing the Two-Tier Architecture reads/writes across tier boundary asynchronous writes of large blobs all modifications are local Classic tiered architecture Streaming architectureClassic tiered architecture database layer compute layer application working state + historic state compute + stream storage and snapshot storage (backup) application state Streaming architecture

12. © 2018 data Artisans12 Keystone Routing Pipeline at Netflix (as presented at Flink Forward San Francisco, 2018) @  Athena X  SQL to define metrics  Thresholds and actions to trigger  Blends analytics and actions Streams from Hadoop, Kafka, etc SQL, thresholds, actions Analytics Alerts Derived streams

16. © 2018 data Artisans16 Kubernetes Database Kubernetes • Example: Scaling down a replicated database • 3 replicas, 4 node scale down need to move or reorganize data before container shutdown Kubernetes & stateful applications What about stateful containers?

17. © 2018 data Artisans17 • consistent stateful upgrades ‒application evolution and bug fixes • migration of application state ‒cluster migration, A/B testing • re-processing and reinstatement ‒fix corrupt results, bootstrap new applications • state evolution (schema evolution) A B Stateful Questions Container-based Resource Orchestration Stateful Stream Processing & Snapshots Kubernetes Apache Flink Container-based platform for stateful data-driven applications dA Platform Code, Resource, Config, and Snapshot Management Application Manager

18. © 2018 data Artisans18 Architecture Apache Flink Stateful stream processing Kubernetes Container platform Logging Metrics dA Application Manager Application lifecycle management Storage In Action App Manager Kubernetes Resource Allocation Job Control Snapshot Management CI/CDWeb interface

21. THANK YOU! WE ARE HIRING data-artisans.com/careers @aljoscha @dataArtisans @ApacheFlink

Notes de l'éditeur

• data Artisans was founded by the original creators of Apache Flink • We provide dA Platform, a complete stream processing infrastructure with open-source Apache Flink
Alibaba 472 mio records / second at peak Largest job: thousands of subtasks, tens of TBs of state Thousands of jobs >5k nodes >500k CPU cores Netflix ~ 3 trillion events/day ~2000 routing jobs ~10k containers ~200k parallel operator instances Uber Athena X SQL to define metrics Thresholds and actions to trigger Blends analytics and actions All have in common: built internal platforms
• Also included is the Application Manager, which turns dA Platform into a self-service platform for stateful stream processing applications. • dA Platform is generally available, and you can download a free trial today!

Apache Flink and what it is used for

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Apache Flink and what it is used for

Similaire à Apache Flink and what it is used for (20)

Plus de Aljoscha Krettek

Plus de Aljoscha Krettek (12)

Dernier

Dernier (20)

Apache Flink and what it is used for

Notes de l'éditeur