Elastic Streams at Scale @ Flink Forward 2018 Berlin

•

1 j'aime•455 vues

This document discusses Elastic Streams at scale using Apache Flink and Mesos. It describes how Flink jobs can be deployed on Mesos clusters by having the Flink master process request resources from the Mesos resource manager. The resource manager then allocates Mesos containers for the Flink master and task managers, allowing the Flink processes to be deployed and tasks to run on the Mesos cluster resources. A Mesos dispatcher can be used to start and monitor the Flink master process.

Technologie

Elastic Streams at Scale
@stsffap @joerg_schad

time
Workload
Resources
time
Workload
Resources

partitionId: 1, offset: 42
partitionId: 3, offset: 10
partitionId: 6, offset: 27
partitionId: 1, offset: 42
partitionId: 3, offset: 10
partitionId: 6, offset: 27
• Store offset for each partition
• Individual entries are repartitionable

© 2018 Mesosphere, Inc. All Rights Reserved.
Flink Mesos Integration
Mesos Master
Mesos Cluster
Client
(2) HTTP POST
JobGraph/Jars
Flink Master Process
Flink Mesos
ResourceManager
JobManager
(4) Start Process
(and supervise)
(8) Deploy
Tasks
(7) Register
(5) Request slots
Flink Mesos
Dispatcher
(3) Allocate
container
for Flink master
(6) Allocate
containers
for TaskManagers
Marathon
(1) Start and
monitor
dispatcher
Mesos Task
TaskManager
Mesos Task
TaskManager

Mesos
Master
Mesos Cluster
Cluster Client
(1) Submit Mesos App.
(JobGraph / JARs)
Application Master
Flink-Mesos
ResourceManager
JobManager TaskManager
TaskManager
TaskManager
(2) Spawn
Application Master
(4) Start
TaskManagers
(6) Deploy
Tasks
(5) Register
(3) Request slots

THANK YOU!
@stsffap
@dataArtisans
@ApacheFlink
WE ARE HIRING
data-artisans.com/careers

Contenu connexe

Tendances

Streaming data is now the new trend, and for very good reasons. Most data is produced continuously, and it makes sense that it is processed and analyzed continuously. Whether it is the need for more real-time products, adopting micro-services, or building continuous applications, stream processing technology offers to simplify the data infrastructure stack and reduce the latency to decisions. Before Apache Flink, users of stream processing frameworks had to make hard choices and trade off either latency, throughput, or result accuracy. Flink was the first open source framework (and still the only one), that has been demonstrated to deliver (1) throughput in the order of tens of millions of events per second in moderate clusters, (2) sub-second latency that can be as low as few 10s of milliseconds, (3) guaranteed exactly once semantics for application state, as well as exactly once end-to-end delivery with supported sources and sinks (e.g., pipelines from Kafka to Flink to HDFS or Cassandra), and (4) accurate results in the presence of out of order data arrival through its support for event time. In this talk, I will cover the basics on Flink: why the project exists, where it came from, what gap does it fill, how it differs from all the other stream processing projects, and what is it being used for. I will also recent developments in the Flink community, what the community is working on currently, and touch upon a longer-term vision for Flink.

Kostas Tzoumas - Apache Flink®: State of the Union and What's Next

Ververica

data Artisans Product Announcement

Flink Forward

This talk shares experiences from deploying and tuning Flink steam processing applications for very large scale. We share lessons learned from users, contributors, and our own experiments about running demanding streaming jobs at scale. The talk will explain what aspects currently render a job as particularly demanding, show how to configure and tune a large scale Flink job, and outline what the Flink community is working on to make the out-of-the-box for experience as smooth as possible. We will, for example, dive into - analyzing and tuning checkpointing - selecting and configuring state backends - understanding common bottlenecks - understanding and configuring network parameters

Stephan Ewen - Experiences running Flink at Very Large Scale

Ververica

Modern stream processing engines not only have to process millions of events per second at sub-second latency but also have to cope with constantly changing workloads. Due to the dynamic nature of stream applications where the number of incoming events can strongly vary with time, systems cannot reliably predetermine the amount of required resources. In order to meet guaranteed SLAs as well as utilizing system resources as efficiently as possible, frameworks like Apache Flink have to adapt their resource consumption dynamically. In this talk, we will take a look under the hood and explain how Flink scales stateful application in and out. Starting with the concept of key groups and partionable state, we will cover ways to detect bottlenecks in streaming jobs and discuss efficient strategies how to scale out operators with minimal down-time.

Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...

Till Rohrmann

Stream Processing is a powerful paradigm, especially when backed by a system like Apache Flink. With each release and year, we see Flink being used for more challenging use case and applications. But beyond the individual application (though it may be grand and challenging in itself), stream processing is a much broader building block: the foundational piece of a platform that brings together the different parts of a data architecture. A platform that integrates data analytics, data ingestion, SQL, Machine Learning, data provenance, databases, and other aspects of a data-driven infrastructure in a meaningful way. In this keynote, we look at what goes into building a stream processing platform that is more than the sum of its parts.

Flink Forward San Francisco 2018 keynote: Stephan Ewen - "What turns stream p...

Flink Forward

As stream processing engines become more and more popular and are used in different environments, the demand to support different deployment scenarios increases. Depending on the user's infrastructure, a stream processor might be run on a bare metal cluster in standalone mode, deployed via Apache Yarn and Mesos, or run in a containerized environment. In order to fulfill the requirements of different deployment options and to provide enough flexibility for the future, the Flink community has recently started to redesign Flink's distributed architecture. This talk will explain the limitations of the old architecture and how they are solved with the new design. We will present the new building blocks of a Flink cluster and demonstrate, using the example of Flink's Mesos and Docker support, how they can be combined to run Flink nearly everywhere.

Flink Forward SF 2017: Till Rohrmann - Redesigning Apache Flink’s Distributed...

Flink Forward

We are HanSight, a leading security startup based in China. We provide solutions for enterprise cybersecurity with a main focus on User Behavior Analytics(UBA). Typical UBA deployment in large scale enterprise needs to handle 10k+ unique users over 10+ dimensions. Real-time analysis and detection on that scale of data has become a must have functionality yet a challenge for traditional security solutions. Most of the products on the market usually struggles with high throughput(100k TPS) and real-time analysis accuracy. With Flink’s streaming nature, we are able to present a next generation UBA system that tackles the large scale real-time data analysis challenge. Basically, Flink serves as a CEP engine processing data in a streaming fashion. And UBA engine (anomaly detection algorithms, rule engine) runs on top of Flink to achieve dynamic ETL rule configuration and hot deployment. Also we provide a stunning UI design for rule configuration, incident response and system monitoring.

Flink Forward Berlin 2017: Hao Wu - Large Scale User Behavior Analytics by Flink

Flink Forward

Apache Mesos allows operators to run distributed applications across an entire datacenter and is attracting ever increasing interest. As much as distributed applications see increased use enabled by Mesos, Mesos also sees increasing use due to a growing ecosystem of well integrated applications. One of the latest additions to the Mesos family is Apache Flink. Flink is one of the most popular open source systems for real-time high scale data processing and allows users to deal with low-latency streaming analytical workloads on Mesos. In this talk we explain the challenges solved while integrating Flink with Mesos, including how Flink’s distributed architecture can be modeled as a Mesos framework, and how Flink was integrated with Fenzo. Next, we describe how Flink was packaged to easily run on DC/OS.

Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Till Rohrmann

In our fast moving world it becomes more and more important for companies to gain near real-time insights from their data to make faster decisions. These insights do not only provide a competitve edge over ones rivals but also enable a company to create completely new services and products. Amongst others, predictive user interfaces and online recommendation can be implemented when being able to process large amounts of data in real-time. Apache Flink, one of the most advanced open source distributed stream processing platforms, allows you to extract business intelligence from your data in near real-time. With Apache Flink it is possible to process billions of messages with milliseconds latency. Moreover, its expressive APIs allow you to quickly solve your problems, ranging from classical analytical workloads to distributed event-driven applications. In this talk, I will introduce Apache Flink and explain how it enables users to develop distributed applications and process analytical workloads alike. Starting with Flink’s basic concepts of fault-tolerance, statefulness and event-time aware processing, we will take a look at the different APIs and what they allow us to do. The talk will be concluded by demonstrating how we can use Flink’s higher level abstractions such as FlinkCEP and StreamSQL to do declarative stream processing.

Modern Stream Processing With Apache Flink @ GOTO Berlin 2017

Till Rohrmann

Apache Flink Berlin Meetup May 2016

Stephan Ewen

Stream Processing has evolved quickly in a short time: a few years ago, stream processing was mostly simple real-time aggregations with limited throughput and consistency. Today, many stream processing applications have complex logic, strict correctness guarantees, high performance, low latency, and maintain large state without databases. Since then, Stream processing has become much more sophisticated because the stream processors – the systems that run the application code, coordinate the distributed execution, route the data streams, and ensure correctness in the face of failures and crashes – have become much more technologically advanced. In this talk, we walk through some of the techniques and innovations behind Apache Flink, one of the most powerful open source stream processors. In particular, we plan to discuss: The evolution of fault tolerance in stream processing, Flink’s approach of distributed asynchronous snapshots, and how that approach looks today after multiple years of collaborative work with users running large scale stream processing deployments. How Flink supports applications with terabytes of state and offers efficient snapshots, fast recovery, rescaling, and high throughput. How to build end-to-end consistency (exactly-once semantics) and transactional integration with other systems. How batch and streaming can both run on the same execution model with best-in-class performance.

Flink Forward San Francisco 2018: Stefan Richter - "How to build a modern str...

Flink Forward

http://flink-forward.org/kb_sessions/running-apache-flink-everywhere-standalone-yarn-mesos-docker-kubernetes-etc/ The world of cluster managers and deployment frameworks is getting complicated. There is zoo of tools to deploy and manage data processing jobs, all of which have different resource management and fault tolerance slightly different. Some tools have a only per-job processes (Yarn, Docker/Kubernetes), while others require some long running processes (Mesos, Standalone). In some frameworks, streaming jobs control their own resource allocation (Yarn, Mesos), while for other frameworks, resource management is handled by external tools (Kubernetes). To be broadly usable in a variety of setups, Flink needs to play well with all these frameworks and their paradigms. This talk describes Flink’s new proposed process and deployment model that will make it work together well with the above mentioned frameworks. The new abstraction is designed to cover a variety of use cases, like isolated single job deployments, sessions of multiple short jobs, and multi-tenant setups.

Stephan Ewen - Running Flink Everywhere

Flink Forward

Apache Flink, a powerful distributed stateful stream processing framework, is an especially good fit for deployment on a containerization platform: its storage requirement is primarily external (e.g. HDFS or S3), clusters often share the lifetime of the jobs that run on them, and the flexibility of allocating resources on such a platform allows for scaling jobs up and down as necessary. In this talk I will give a brief introduction to Apache Flink, then describe the journey to making it a first-class citizen of the container world. I will cover my experience preparing to publish the “official repository” of Flink images on Docker Hub, the challenges of fitting a Flink deployment in a Kubernetes-shaped box, and the rough edges of Flink itself that were exposed by this process.

Flink Forward Berlin 2017: Patrick Lucas - Flink in Containerland

Flink Forward

Last year we (TouK) introduced Flink in one of the biggest polish telcoms in the domain of real time marketing and fraud detection. One of the most significant problems in adoption was lack of programming skills at our client - the users were supposed to be analytics/business people. Therefore, we developed a custom platform - TouK Nussknacker - which allows users to design processes with GUI by drawing diagrams. Our project is going to be open-sourced soon - this will happen before Flink Forward. We believe it can make stream processing with Flink more accessible in many use cases, especially in companies that don't have their own development teams. During the talk I’m going to describe architecture of our platform, why we made certain design decisions and about our future plans. I’ll also describe our experiences - when being able to use GUI is great and when it’s better to develop jobs as normal code. If time permits I’ll also show a quick demo of our solution.

Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...

Flink Forward

Over 109 million subscribers are enjoying more than 125 million hours of TV shows and movies per day on Netflix. This leads to massive amount of data flowing through our data ingestion pipeline to improve service and user experience. They are powering various data analytic cases like personalization, operational insight, fraud detection. At the heart of this massive data ingestion pipeline is a self-serve stream processing platform that processes 3 trillion events and 12 PB of data every day. We have recently migrated this stream processing platform from Samza to Flink. In this talk, we will share the challenges and issues that we run into when running Flink at scale in cloud. We will dive deep into the troubleshooting techniques and lessons learned.

Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud"

Flink Forward

Data stream processing has redefined how many of us build data pipelines. Apache Flink is one of the systems at the forefront of that development: With its versatile APIs (event-time streaming, Stream SQL, events/state) and powerful execution model, Flink has been part of re-defining what stream processing can do. By now, Apache Flink powers some of the largest data stream processing pipelines in open source data stream processing. In this keynote, we will look at the evolution of Stream Processing and Apache Flink during the last year, and what we believe will be the next wave of stream processing applications. We show how the Flink community and users evolved, what use cases are coming up, and how new and upcoming features in Flink are making new types of applications possible. We will also discuss common challenges that companies are facing when adopting stream processing, and how we can help companies to rapidly adopt and roll out stream processing company-wide.

Flink Forward Berlin 2017: Stephan Ewen - The State of Flink and how to adopt...

Flink Forward

SQL is undoubtedly the most widely used language for data analytics. It is declarative, many database systems and query processors feature advanced query optimizers and highly efficient execution engines, and last but not least it is the standard that everybody knows and uses. With stream processing technology becoming mainstream a question arises: “Why isn’t SQL widely supported by open source stream processors?”. One answer is that SQL’s semantics and syntax have not been designed with the characteristics of streaming data in mind. Consequently, systems that want to provide support for SQL on data streams have to overcome a conceptual gap. Apache Flink is a distributed stream processing system. Due to its support for event-time processing, exactly-once state semantics, and its high throughput capabilities, Flink is very well suited for streaming analytics. Since about a year, the Flink community is working on two relational APIs for unified stream and batch processing, the Table API and SQL. The Table API is a language-integrated relational API and the SQL interface is compliant with standard SQL. Both APIs are semantically compatible and share the same optimization and execution path based on Apache Calcite. A core principle of both APIs is to provide the same semantics for batch and streaming data sources, meaning that a query should compute the same result regardless whether it was executed on a static data set, such as a file, or on a data stream, like a Kafka topic. In this talk we present the semantics of Apache Flink’s relational APIs for stream analytics. We discuss their conceptual model and showcase their usage. The central concept of these APIs are dynamic tables. We explain how streams are converted into dynamic tables and vice versa without losing information due to the stream-table duality. Relational queries on dynamic tables behave similar to materialized view definitions and produce new dynamic tables. We show how dynamic tables are converted back into changelog streams or are written as materialized views to external systems, such as Apache Kafka or Apache Cassandra, and are updated in place with low latency.

Fabian Hueske - Stream Analytics with SQL on Apache Flink

Ververica

A look at Flink 1.2

Stefan Richter

dA Platform is a production-ready platform for stream processing with Apache Flink®. The Platform includes open source Apache Flink, a stateful stream processing and event-driven application framework, and dA Application Manager, a central deployment and management component. dA Platform schedules clusters on Kubernetes, deploys stateful Flink applications, and controls these applications and their state.

dA Platform Overview

Robert Metzger

Developing and operating real-time applications with Oceanus The Tencent Data team (data.qq.com) is responsible to build a reliable and scalable infrastructure for various products at Tencent including Tencent Games, Tencent Video and WeChat. We have to process over 1.6 trillion events per day and the peak throughput reaches 210 million per second. Such a large scale exposes great challenges in the developing and operating of real-time applications. In this talk, we will give an introduction to Oceanus, a platform built at Tencent to facilitate the development of real-time applications. We will introduce typical user cases and how applications are developed, deployed and monitored with Oceanus. We will also introduce the management of Flink jobs in Oceanus, together with some experience gained in our operating practice. Finally, we will discuss some efforts we have done to enhance the reliability and efficiency, e.g. master failover and local keyed streams.

Flink Forward San Francisco 2019: Developing and operating real-time applicat...

Flink Forward

Tendances (20)

Kostas Tzoumas - Apache Flink®: State of the Union and What's Next

data Artisans Product Announcement

Stephan Ewen - Experiences running Flink at Very Large Scale

Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...

Flink Forward San Francisco 2018 keynote: Stephan Ewen - "What turns stream p...

Flink Forward SF 2017: Till Rohrmann - Redesigning Apache Flink’s Distributed...

Flink Forward Berlin 2017: Hao Wu - Large Scale User Behavior Analytics by Flink

Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Modern Stream Processing With Apache Flink @ GOTO Berlin 2017

Apache Flink Berlin Meetup May 2016

Flink Forward San Francisco 2018: Stefan Richter - "How to build a modern str...

Stephan Ewen - Running Flink Everywhere

Flink Forward Berlin 2017: Patrick Lucas - Flink in Containerland

Flink Forward Berlin 2017: Maciek Próchniak - TouK Nussknacker - creating Fli...

Flink Forward San Francisco 2018: Steven Wu - "Scaling Flink in Cloud"

Flink Forward Berlin 2017: Stephan Ewen - The State of Flink and how to adopt...

Fabian Hueske - Stream Analytics with SQL on Apache Flink

A look at Flink 1.2

dA Platform Overview

Flink Forward San Francisco 2019: Developing and operating real-time applicat...

Similaire à Elastic Streams at Scale @ Flink Forward 2018 Berlin

The current craze of Docker has everyone sticking their processes inside a container… but do you really understand cgroups and how they work? Do you understand the difference between CPU Sets and CPU Shares? Spark is a Scala application that lives inside a Java Runtime, do you understand the consequence of what impact the cgroup constraints have on the JRE? This talk starts with a deep understand of Java’s memory management and GC characteristics and how JRE characteristics change based on core count. We will continue the talk looking at containers and how resource isolation works.

Jörg Schad - NO ONE PUTS Java IN THE CONTAINER - Codemotion Milan 2017

Codemotion

Azure storage deep dive

Yves Goeleven

azure track -04- azure storage deep dive

ITProceed

Presto At Treasure Data

Taro L. Saito

Our applications are continuously sending log messages about their state. We usually have many different solutions/applications, which means, that we have many different logs. And that is never good. In short, your data is mutating, evolving and growing, but traditional analyses are no long in step with these changes. Business forte is all about sourcing, sculpting and analyzing this variety of data. The talk will focus on technology that makes this possible. URL: https://www.youtube.com/watch?v=Se2rVAg4EVs

Search and analyze data in real time

Rohit Kalsarpe

Organizations continue to adopt Solr because of its ability to scale to meet even the most demanding workflows. Recently, LucidWorks has been leading the effort to identify, measure, and expand the limits of Solr. As part of this effort, we've learned a few things along the way that should prove useful for any organization wanting to scale Solr. Attendees will come away with a better understanding of how sharding and replication impact performance. Also, no benchmark is useful without being repeatable; Tim will also cover how to perform similar tests using the Solr-Scale-Toolkit in Amazon EC2.

Benchmarking Solr Performance at Scale

thelabdude

Performance

Christophe Marchal

[投影片錯誤更正] p.43 中間32數字改成64。右上藍色小框64改成63 原本要整理Meltdown與Spectre，但這兩個所利用的硬體行為之後都跟cache side channel有關係，所以閱讀Meltdown與Spectre之餘，就整理了相關cache side channel攻擊與防禦。回饋問題: 一: 為什麼LLC要切割成LLC slice? "Modern Intel processors, starting with the Sandy Bridge microarchitecture, use a more complex architecture for the LLC, to improve its performance. The LLC is divided into per-core slices, which are connected by a ring bus. Slices can be accessed concurrently and are effectively separate caches, although the bus ensures that each core can access the full LLC (with higher latency for remote slices)." 二: flush+reload with shared memory pages，為什麼要 flush+reload? 不是可以直接存取到資料? 討論的是共用shared library，洩漏victim使用shared library的情形。三: RDTSCP ? 可量測執行指令的cycle數。四: side channel攻擊需要環境運作的程式不能太複雜? Kuon: 實際案例 embed運作環境並不複雜，e.g. trustzone上可能只運作openSSL。 AJ: 就算在複雜環境，可以找到觸發Victim的特定運算點，也是可以進行觀測。

[若渴]Study on Side Channel Attacks and Countermeasures

Aj MaChInE

K8s monitoring with elk

윤종원 윤종원

Apache Mesos apporte une abstraction des ressources CPU, mémoire et stockage, d'un ensemble de machines (physiques ou virtuelles). Un cluster Mesos forme un système distribué, tolérant à la panne et permettant de scaler les services déployés. Venez découvrir les fonctionnalités de Mesos, ses stratégies de déploiement, et l'écosystème Mesosphere qui apporte scalabilité et découverte de services. Nous visiterons un cluster hébergeant une application JS et un backend REST conteneurisés. Pour finir, nous jouerons au Chaos Monkey afin de prouver la résilience du système mis en place. Speaker : Pablo Lopez, Aurélien Maury, Jean-Baptiste Claramonte et Jean-Pascal Thiery à Devoxx France 2015

Mesos university - Devoxx France 2015 - part 2

Publicis Sapient Engineering

Ecosysteme mesos university - devoxx france - 8 avril2015

Jean-Baptiste Claramonte

Make Your Site SpEEdy

Ruthie BenDor

Ops Jumpstart: MongoDB Administration 101

MongoDB

keti companion classifier

JEE HYUN PARK

Proving out flash storage array performance using swingbench and slob

Kapil Goyal

Openstack meetup lyon_2017-09-28

Xavier Lucas

(Stephen Parente + Jeff Field, Blizzard) Kafka Summit SF 2018 Blizzard’s global data platform has become a driving force in both business and operational analytics. As more internal customers onboard with the system, there is increasing demand for custom applications to access this data in near real time. In order to avoid many independent teams with varying levels of Kafka expertise all accessing the firehose from our critical production Kafkas, we developed our own pub-sub system on top of Kafka to provide specific datasets to customers on their own cloud deployed Kafka clusters.

You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard

confluent

Flink has supported Apache Mesos officially since the 1.2 release and many users have been using them together even before that. The latest releases 1.4 and 1.5 (not released at the time of writing) add a deeper integration for resource schedulers, such as Mesos, which also resulted in many new features around this integration. But what does that mean in practice for operating large cluster? In this talk, we will discuss operational best practices-alongside with some pitfalls- for operating large Flink cluster on top of Apache Mesos, including topics such as: * Deployments, * Monitoring, * Scaling, * Upgrades, * Debugging.

Flink Forward San Francisco 2018: Jörg Schad and Biswajit Das - "Operating Fl...

Flink Forward

backgroundcommunicationandwaitevents-180124221026.pdf

ssuser785ce21

Presented at JavaOne 2015. JSR107, aka the Temporary Caching API for the Java Platform, has now been finalized almost 2 years ago. We've heard all about its ease of use and capabilities. But there is much left unaddressed. The good news is that the EG is looking at addressing many of the current shortcomings... But what do you do now? Go for proprietary APIs?! Ehcache, the de facto caching API for 10 years now, has gone through a major API revamp: Ehcache3. One major theme, beyond its usual ease of use, was JSR107. Natively integrating it, but also looking beyond. With close to no API tie-ins, Ehcache3 lets you extend the JSR107 API transparently to go beyond the specification: topology-wise: whether you want to go offheap and scale up, or scale out by clustering your caches; functionality-wise: using transactional caches, automatic resource control or even using a write-behind cache to scale out writes... Best of all is that this isn't only minimally intrusive, it is also all free to use and available as part of the open-source Ehcache v3 that has been GA'ed earlier this year...

Ehcache3 — JSR-107 on steroids

Alex Snaps

Similaire à Elastic Streams at Scale @ Flink Forward 2018 Berlin (20)

Jörg Schad - NO ONE PUTS Java IN THE CONTAINER - Codemotion Milan 2017

Azure storage deep dive

azure track -04- azure storage deep dive

Presto At Treasure Data

Search and analyze data in real time

Benchmarking Solr Performance at Scale

Performance

[若渴]Study on Side Channel Attacks and Countermeasures

K8s monitoring with elk

Mesos university - Devoxx France 2015 - part 2

Ecosysteme mesos university - devoxx france - 8 avril2015

Make Your Site SpEEdy

Ops Jumpstart: MongoDB Administration 101

keti companion classifier

Proving out flash storage array performance using swingbench and slob

Openstack meetup lyon_2017-09-28

You Must Construct Additional Pipelines: Pub-Sub on Kafka at Blizzard

Flink Forward San Francisco 2018: Jörg Schad and Biswajit Das - "Operating Fl...

backgroundcommunicationandwaitevents-180124221026.pdf

Ehcache3 — JSR-107 on steroids

Plus de Till Rohrmann

Apache Flink® Meets Apache Mesos® and DC/OS

Till Rohrmann

With Flink 1.3 being released, the Flink community is already working towards the upcoming release 1.4. Given Flink's high development pace, which manifested in Flink 1.3 being one of the feature-wise biggest releases in its recent history, it becomes more and more difficult to keep track of all development threads. Moreover, it requires more effort to learn about newly added features and which value they provide for your application. In this talk, I want to present and explain some of Flink's latest features, including incremental checkpointing, fine grained recovery, side outputs and many more. Furthermore, I want to put them in perspective with respect to Flink's future direction by giving some insights into ongoing development threads in the community. Thereby, I intend to give attendees a better picture about Flink's current and future capabilities.

From Apache Flink® 1.3 to 1.4

Till Rohrmann

In recent years, the generated and collected data is increasing at an almost exponential rate. At the same time, the data’s value has been identified in terms of insights that can be provided. However, retrieving the value requires powerful analysis tools, since valuable insights are buried deep in large amounts of noise. Unfortunately, analytic capacities did not scale well with the growing data. Many existing tools run only on a single computer and are limited in terms of data size by its memory. A very promising solution to deal with large-scale data is scaling systems and exploiting parallelism. In this presentation, we propose Gilbert, a distributed sparse linear algebra system, to decrease the imminent lack of analytic capacities. Gilbert offers a MATLAB-like programming language for linear algebra programs, which are automatically executed in parallel. Transparent parallelization is achieved by compiling the linear algebra operations first into an intermediate representation. This language-independent form enables high-level algebraic optimizations. Different optimization strategies are evaluated and the best one is chosen by a cost-based optimizer. The optimized result is then transformed into a suitable format for parallel execution. Gilbert generates execution plans for Apache Spark and Apache Flink, two massively parallel dataflow systems. Distributed matrices are represented by square blocks to guarantee a well-balanced trade-off between data parallelism and data granularity. An exhaustive evaluation indicates that Gilbert is able to process varying amounts of data exceeding the memory of a single computer on clusters of different sizes. Two well known machine learning (ML) algorithms, namely PageRank and Gaussian non-negative matrix factorization (GNMF), are implemented with Gilbert. The performance of these algorithms is compared to optimized implementations based on Spark and Flink. Even though Gilbert is not as fast as the optimized algorithms, it simplifies the development process significantly due to its high-level programming abstraction.

Gilbert: Declarative Sparse Linear Algebra on Massively Parallel Dataflow Sys...

Till Rohrmann

Streaming Analytics & CEP - Two sides of the same coin?

Till Rohrmann

Apache Flink: Streaming Done Right @ FOSDEM 2016

Till Rohrmann

Streaming Data Flow with Apache Flink @ Paris Flink Meetup 2015

Till Rohrmann

The talk explains how Apache Flink checkpoints stateful jobs using the asynchronous barrier snapshotting algorithm to give exactly once semantics in streaming. Furthermore, Flink's approach to master high availability (HA) is described which solves the problem of the JobManager being the single point of failure. Job checkpointing in combination with HA is the basis for Flink's fault tolerance mechanism to recover from occurring failures.

Fault Tolerance and Job Recovery in Apache Flink @ FlinkForward 2015

Till Rohrmann

Interactive Data Analysis with Apache Flink @ Flink Meetup in Berlin

Till Rohrmann

Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015

Till Rohrmann

Machine Learning with Apache Flink at Stockholm Machine Learning Group

Till Rohrmann

This presentation introduces Apache Flink, a massively parallel data processing engine which currently undergoes the incubation process at the Apache Software Foundation. Flink's programming primitives are presented and it is shown how easily a distributed PageRank algorithm can be implemented with Flink. Intriguing features such as dedicated memory management, Hadoop compatibility, streaming and automatic optimisation make it an unique system in the world of Big Data processing.

Introduction to Apache Flink - Fast and reliable big data processing

Till Rohrmann

Plus de Till Rohrmann (11)

Apache Flink® Meets Apache Mesos® and DC/OS

From Apache Flink® 1.3 to 1.4

Gilbert: Declarative Sparse Linear Algebra on Massively Parallel Dataflow Sys...

Streaming Analytics & CEP - Two sides of the same coin?

Apache Flink: Streaming Done Right @ FOSDEM 2016

Streaming Data Flow with Apache Flink @ Paris Flink Meetup 2015

Fault Tolerance and Job Recovery in Apache Flink @ FlinkForward 2015

Interactive Data Analysis with Apache Flink @ Flink Meetup in Berlin

Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015

Machine Learning with Apache Flink at Stockholm Machine Learning Group

Introduction to Apache Flink - Fast and reliable big data processing

Dernier

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Orbitshub

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Dubai, often portrayed as a shimmering oasis in the desert, faces its own set of challenges, including the occasional threat of flooding. Despite its reputation for opulence and modernity, the emirate is not immune to the forces of nature. In recent years, Dubai has experienced sporadic but significant floods, testing the resilience of its infrastructure and communities. Among the critical lifelines in this bustling metropolis is the Dubai International Airport, a bustling hub that connects the city to the world. This article explores the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Orbitshub

The action of the next cyber saga takes place in the mystical lands of the Asia-Pacific region, where the main characters began their digital activities in the middle of 2021 and qualitatively strengthened it in 2022. Corporate espionage, document theft, audio recordings, and data leaks from messaging platforms were all a matter of one day for Dark Pink. Their geographical focus may have started in the Asia-Pacific region, but their ambitions knew no bounds, targeting a European government ministry in a bold move to expand their portfolio. Their victim profile was as diverse as a UN meeting, targeting military organizations, government agencies, and even a religious organization. Because discrimination is not a fashionable agenda. In the world of cybercrime, they serve as a reminder that sometimes the most serious threats come in the most unassuming packages with a pink bow.

Cyberprint. Dark Pink Apt Group [EN].pdf

Overkill Security

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Tracing the root cause of a performance issue requires a lot of patience, experience, and focus. It’s so hard that we sometimes attempt to guess by trying out tentative fixes, but that usually results in frustration, messy code, and a considerable waste of time and money. This talk explains how to correctly zoom in on a performance bottleneck using three levels of profiling: distributed tracing, metrics, and method profiling. After we learn to read the JVM profiler output as a flame graph, we explore a series of bottlenecks typical for backend systems, like connection/thread pool starvation, invisible aspects, blocking code, hot CPU methods, lock contention, and Virtual Thread pinning, and we learn to trace them even if they occur in library code you are not familiar with. Attend this talk and prepare for the performance issues that will eventually hit any successful system. About authorWith two decades of experience, Victor is a Java Champion working as a trainer for top companies in Europe. Five thousands developers in 120 companies attended his workshops, so he gets to debate every week the challenges that various projects struggle with. In return, Victor summarizes key points from these workshops in conference talks and online meetups for the European Software Crafters, the world’s largest developer community around architecture, refactoring, and testing. Discover how Victor can help you on victorrentea.ro : company training catalog, consultancy and YouTube playlists.

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Victor Rentea

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Angeliki Cooney has spent over twenty years at the forefront of the life sciences industry, working out of Wynantskill, NY. She is highly regarded for her dedication to advancing the development and accessibility of innovative treatments for chronic diseases, rare disorders, and cancer. Her professional journey has centered on strategic consulting for biopharmaceutical companies, facilitating digital transformation, enhancing omnichannel engagement, and refining strategic commercial practices. Angeliki's innovative contributions include pioneering several software-as-a-service (SaaS) products for the life sciences sector, earning her three patents. As the Senior Vice President of Life Sciences at Avenga, Angeliki orchestrated the firm's strategic entry into the U.S. market. Avenga, a renowned digital engineering and consulting firm, partners with significant entities in the pharmaceutical and biotechnology fields. Her leadership was instrumental in expanding Avenga's client base and establishing its presence in the competitive U.S. market.

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Angeliki Cooney

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

MS Copilot expands with MS Graph connectors

Nanddeep Nachan

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

DBX First Quarter 2024 Investor Presentation

Dropbox

Dernier (20)

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

Corporate and higher education May webinar.pptx

Strategies for Landing an Oracle DBA Job as a Fresher

Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...

Cyberprint. Dark Pink Apt Group [EN].pdf

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Finding Java's Hidden Performance Traps @ DevoxxUK 2024

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

presentation ICT roal in 21st century education

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

How to Troubleshoot Apps for the Modern Connected Worker

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MS Copilot expands with MS Graph connectors

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

DBX First Quarter 2024 Investor Presentation

Elastic Streams at Scale @ Flink Forward 2018 Berlin

1. Elastic Streams at Scale @stsffap @joerg_schad

3. time Workload Resources time Workload Resources

5. • •

6. •

7. • • • •

8. • • •

9. • •

10. • • • #1 #2 #3

11. partitionId: 1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27 partitionId: 1, offset: 42 partitionId: 3, offset: 10 partitionId: 6, offset: 27 • Store offset for each partition • Individual entries are repartitionable

12. • • • •

13.

14. • • •

15. • • • • • • •

16.

17. • • • • • •

18.

19. • • • • • • • • • • • •

20. © 2018 Mesosphere, Inc. All Rights Reserved. Flink Mesos Integration Mesos Master Mesos Cluster Client (2) HTTP POST JobGraph/Jars Flink Master Process Flink Mesos ResourceManager JobManager (4) Start Process (and supervise) (8) Deploy Tasks (7) Register (5) Request slots Flink Mesos Dispatcher (3) Allocate container for Flink master (6) Allocate containers for TaskManagers Marathon (1) Start and monitor dispatcher Mesos Task TaskManager Mesos Task TaskManager

21. Mesos Master Mesos Cluster Cluster Client (1) Submit Mesos App. (JobGraph / JARs) Application Master Flink-Mesos ResourceManager JobManager TaskManager TaskManager TaskManager (2) Spawn Application Master (4) Start TaskManagers (6) Deploy Tasks (5) Register (3) Request slots

22. • Monitor

23. • • • • •

24. THANK YOU! @stsffap @dataArtisans @ApacheFlink WE ARE HIRING data-artisans.com/careers

Elastic Streams at Scale @ Flink Forward 2018 Berlin

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Elastic Streams at Scale @ Flink Forward 2018 Berlin

Similaire à Elastic Streams at Scale @ Flink Forward 2018 Berlin (20)

Plus de Till Rohrmann

Plus de Till Rohrmann (11)

Dernier

Dernier (20)

Elastic Streams at Scale @ Flink Forward 2018 Berlin