[DataSciCon] Divide, distribute and conquer stream v. batch

•

1 j'aime•354 vues

Data is flowing everywhere around us, from phones, credit cards, sensor-equipped buildings, vending machines, thermostats, trains, buses, planes, posts to social media, digital pictures and video and so on.... http://www.datascicon.tech

Logiciels

DIVIDE, DISTRIBUTE AND CONQUER: 
STREAM V. BATCH

Solutions Architect
Developer Advocate
Who am I?

Solutions Architect
Developer Advocate
@gamussa in internetz
Who am I?

Solutions Architect
Developer Advocate
@gamussa in internetz
Hey you, yes, you, go follow me in twitter ©
Who am I?

@gamussa @confluentinc @DataSciCon
BATCH PROCESSING
Data at rest

@gamussa @confluentinc @DataSciCon
Data and Queries
Origin and processing

@gamussa @confluentinc @DataSciCon
Data…

@gamussa @confluentinc @DataSciCon
✓ … inherently immutable
Data…
✓ … time-based

@gamussa @confluentinc @DataSciCon
CRUD -> CR

@gamussa @confluentinc @DataSciCon
Processing is a query

@gamussa @confluentinc @DataSciCon
Processing is a query
Function on full data set

@gamussa @confluentinc @DataSciCon
Processing is a query
Function on full data set
Projection

@gamussa @confluentinc @DataSciCon
Processing is a query
Function on full data set
Projection
Aggregations

@gamussa @confluentinc @DataSciCon
Processing is a query
Function on full data set
Projection
Aggregations
Joins

@gamussa @confluentinc @DataSciCon
Lambda architecture origins
http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html

@gamussa @confluentinc @DataSciCon
https://mapr.com/developercentral/lambda-architecture/
Lambda Architecture

@gamussa @confluentinc @DataSciCon
TFW Trying to explain modern big data
landscape

@gamussa @confluentinc @DataSciCon
STREAM PROCESSING
Data is motion

@gamussa @confluentinc @DataSciCon
Streaming Platform

@gamussa @confluentinc @DataSciCon
Interesting cases
Before You Go

I FOUND YOUR LACK OF FAULT TOLERANCE
DISTURBING

Data is too important to
store it in one computer

@gamussa @confluentinc @DataSciCon
How to process
«infinite» data?

@gamussa @confluentinc @DataSciCon
Time model

@gamussa @confluentinc @DataSciCon
Time model
Different use cases time semantics

@gamussa @confluentinc @DataSciCon
Time model
Different use cases time semantics
Majority of use cases require event-
time semantics

@gamussa @confluentinc @DataSciCon
Time model
Different use cases time semantics
Majority of use cases require event-
time semantics
Other use cases may require
processing-time or special variants
like ingestion-time

@gamussa @confluentinc @DataSciCon
Windowing
Input data, where
colors represent 
different users events
Rectangles denote 
different event-time 
windows
processing-time
event-time
windowing
alice
bob
dave

@gamussa @confluentinc @DataSciCon
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

@gamussa @confluentinc @DataSciCon
Windowing
Windowing is an operation that groups
events
Most commonly needed: time windows,
session windows
Examples:
✗Real-time monitoring: 5-minute averages
✗Reader behavior on a website: user browsing sessions

@gamussa @confluentinc @DataSciCon
Out-of-order and late data
Is very common in practice, not a rare
corner case
✗Related to time model discussion

@gamussa @confluentinc @DataSciCon
Out-of-order and late data

@gamussa @confluentinc @DataSciCon
Out-of-order and late data
Users with mobile phones enter 
airplane, lose Internet connectivity

@gamussa @confluentinc @DataSciCon
Out-of-order and late data
Users with mobile phones enter 
airplane, lose Internet connectivity
Emails are being written 
during the 10h flight

@gamussa @confluentinc @DataSciCon
Stream Processing: results

@gamussa @confluentinc @DataSciCon
Stream Processing: results
• Yes, it’s possible to get computation
results in real time

@gamussa @confluentinc @DataSciCon
Stream Processing: results
• Yes, it’s possible to get computation
results in real time
• Windows – finite view of infinite data
• Based on temporal characteristics of the evet

@gamussa @confluentinc @DataSciCon
DEMO
Let’s analyze flights

@gamussa @confluentinc @DataSciCon
https://www.confluent.io/blog/predicting-flight-arrivals-with-the-apache-kafka-streams-api/

@gamussa @confluentinc @DataSciCon
Example: Training Flight Prediction Model

@gamussa @confluentinc @DataSciCon
https://github.com/confluentinc/online-inferencing-blog-
application

@gamussa @confluentinc @DataSciCon
Thanks!
questions?
@gamussa
viktor@confluent.io

Contenu connexe

Similaire à [DataSciCon] Divide, distribute and conquer stream v. batch

Distributed caching for your next node.js project cf summit - 06-15-2017

Viktor Gamov

The right architecture is key for any IT project. This is especially the case for big data projects, where there are no standard architectures which have proven their suitability over years. This session discusses the different Big Data Architectures which have evolved over time, including traditional Big Data Architecture, Streaming Analytics architecture as well as Lambda and Kappa architecture and presents the mapping of components from both Open Source as well as the Oracle stack onto these architectures. The right architecture is key for any IT project. This is valid in the case for big data projects as well, but on the other hand there are not yet many standard architectures which have proven their suitability over years. This session discusses different Big Data Architectures which have evolved over time, including traditional Big Data Architecture, Event Driven architecture as well as Lambda and Kappa architecture. Each architecture is presented in a vendor- and technology-independent way using a standard architecture blueprint. In a second step, these architecture blueprints are used to show how a given architecture can support certain use cases and which popular open source technologies can help to implement a solution based on a given architecture.

Architecture of Big Data Solutions

Guido Schmutz

Managing Creativity

kamaelian

GrabCAD Print Announcement

GrabCAD

2020 06-03 cukenfest-bdd-and-sl_os

Abigail Bangser

Reactive data analysis with vert.x

Gerald Muecke

How to build simple web apps to automate your SEO tasks - BrightonSEO Spring ...

Charly Wargnier

Perchè potresti aver bisogno di un database NoSQL anche se non sei Google o F...

Codemotion

1) Migrating your on-prem #Enterprise #Data #Warehouse into the #Cloud? Here is what you need to learn (and unlearn) when designing a modern Cloud #DataWarehouse in #BigQuery! 2) Launching a #Startup? See how to supercharge your idea with #Firebase! Watch the recording at https://youtu.be/zezhXNqD0rs and more forward-looking talks on #Cloud #Architectures & #DataEngineering join http://ServerlessToronto.org User Group.

Lunch & Learn BigQuery & Firebase from other Google Cloud customers

Daniel Zivkovic

With Power BI you can bring your BI architecture to the next level. Architecture it's very important topic in a business intelligence project, let's discover which are right questions and possible scenarios to integrate Power BI in an existing environment or to build a new one from scratch. We'll talkabout how to choose the right Storage Modes, how to design a refreshing policy, how to use dataflows to decouple and to lift the transformation process on Cloud and more.

Damag - EmPower your BI Architecture

Riccardo Perico

Open Source Lambda Architecture with Hadoop, Kafka, Samza and Druid

DataWorks Summit

Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analysed, often with many consumers or systems interested in all or part of the events. Storing such huge event streams into HDFS or a NoSQL datastore is feasible and not such a challenge anymore. But if you want to be able to react fast, with minimal latency, you can not afford to first store the data and doing the analysis/analytics later. You have to be able to include part of your analytics right after you consume the data streams. Products for doing event processing, such as Oracle Event Processing or Esper, are avaialble for quite a long time and used to be called Complex Event Processing (CEP). In the past few years, another family of products appeared, mostly out of the Big Data Technology space, called Stream Processing or Streaming Analytics. These are mostly open source products/frameworks such as Apache Storm, Spark Streaming, Flink, Kafka Streams as well as supporting infrastructures such as Apache Kafka. In this talk I will present the theoretical foundations for Stream Processing, discuss the core properties a Stream Processing platform should provide and highlight what differences you might find between the more traditional CEP and the more modern Stream Processing solutions.

Introduction to Stream Processing

Guido Schmutz

In this talk Luke presents his team’s research comparing the evolution of Software Development & DevOps with that of Data Science & AI. Because it is more complex and has far more moving parts, Data Science & AI is where Software Development was in 1999: people are emailing and Slacking notebooks to each other, due to a lack of appropriate tooling. There are few CI/CD pipelines and model health monitoring is scarce. A lot that could be automated is still manual. And teams are siloed. This causes problems both for productivity: it’s hard to collaborate, and reproducibility: which impacts on governance and compliance. Luke presents a proposal for an architecture and a set of open source tools to solve both the collaboration and the governance problem in Data Science & AI. With live demos!

Inextricably linked: reproducibility and productivity in data science and AI

Luke Marsden

" As the capabilities of the web platform advance, and the pervasiveness of web content proliferates, fundamental design principles are more important than ever: How do you enhance your experience as new capabilities become available in Microsoft Edge and other browsers? You've considered how your content looks and operates on mobile, but how does that design adapt to desktop devices? Is it usable via ""headless"" interfaces like screen readers or Cortana? Does your content adapt gracefully to the constraints of its environment? In this session, Aaron Gustafson will help you think about user experience from its foundation, so you understand what it means to build adaptive web experiences – whether in an app, in a browser, or beyond. We’ll show you some of the newest capabilities coming to Microsoft Edge that let you build differentiated experiences on Windows 10. You’ll walk away practical, cross-platform code examples you can put to use in your projects today, and foundational principles to ensure that your projects are ready for anything. "

Designing for Everyone: Building great web experiences for any device

Windows Developer

Our application got popular and now it breaks

ColdFusionConference

Our application got popular and now it breaks

devObjective

Milestones, SHUV, Roadmaps - Oh My!

Jacqueline K. Sanders-Blackman, Agilist,CBAP,SSBB,CSM,SAFe

Milestones, SHUV, Roadmaps - Oh My!

Technology Expresso Expresso-Consulting

Yetizen (https://www.linkedin.com/company/yetizen/about/) was a gaming incubator that existed in San Francisco, roughly between 2011 and 2015. I thought it was an interesting experiment, and was happy to give a series of talks there, and advise the portfolio companies. This talk, from 2012, is an intro to "How to think about Analytics" -- many entrepreneurs know they need data, and they need analytics, but that they had best be able to claim that their startup is "data driven" But what, from an engineering perspective, does that really mean?

Crafting an Analytics Strategy

William Grosso

Introduction to the IBM Watson Data Platform

Margriet Groenendijk

Similaire à [DataSciCon] Divide, distribute and conquer stream v. batch (20)

Distributed caching for your next node.js project cf summit - 06-15-2017

Architecture of Big Data Solutions

Managing Creativity

GrabCAD Print Announcement

2020 06-03 cukenfest-bdd-and-sl_os

Reactive data analysis with vert.x

How to build simple web apps to automate your SEO tasks - BrightonSEO Spring ...

Perchè potresti aver bisogno di un database NoSQL anche se non sei Google o F...

Lunch & Learn BigQuery & Firebase from other Google Cloud customers

Damag - EmPower your BI Architecture

Open Source Lambda Architecture with Hadoop, Kafka, Samza and Druid

Introduction to Stream Processing

Inextricably linked: reproducibility and productivity in data science and AI

Designing for Everyone: Building great web experiences for any device

Our application got popular and now it breaks

Milestones, SHUV, Roadmaps - Oh My!

Crafting an Analytics Strategy

Introduction to the IBM Watson Data Platform

Plus de Viktor Gamov

You never know how your application performs once deployed to the production. Sure, you have unit tests, and your test coverage is sky high. However, you might depend on external resources like databases, web services, and distributed caches. Moreover, without a proper integration testing, you cannot be confident about the stability of your production environment. A long time ago, long before the invention of Docker, configuring the environment for integration testing was incredibly complicated and tedious. Developers and Ops were using fake database implementations, mocking servers, and all kinds of weird tricks to implement the environment without the real environment, but it was ugly, bulky, faulty and not cross-platform as well. Thanks to Docker, those days are long gone, now we can quickly prepare the environment with Docker for our tests. In this talk, I would like to spread the word about awesome project TestContainers – an open source library that exposes API for JUnit tests. It provides lightweight, disposable instances of shared databases, distributed caches or grids, or anything else that can run in a Docker container, all securely and reliably downloaded from your Docker Hub.

Testing containers with TestContainers @ AJUG 7/18/2017

Viktor Gamov

The more we work with Java 8, the more we go into the rabbit hole. Did they add all those streams, lambdas, monads, Optionals and CompletableFutures only to confuse us? It surely looks so! And Java 9 that heads our way brings even more of what we like the most, more puzzlers, of course! In this season we as usual have a great batch of the best Java WTF, great jokes to present them and great prizes for the winners!

[Philly ETE] Java Puzzlers NG

Viktor Gamov

Виктор возвращается с продолжением доклада про обработку данных в памяти! На этот раз он противопоставит «потоковый» подход «пакетному». Как мы знаем, информации, зараза, много! Мало того, что её много, так ещё никто не хочет сидеть и ждать, пока Hadoop-кластер посчитает всё. Big Data уже не интересно, Fast Data — наше новое «всё»! Так что же делать, если новые данные всё прибывают и прибывают по каналам связи (через сокет-сервер ли или модную нынче Кафку) ежесекундно (а может, даже и быстрее)? Можно было бы складывать их в какое-нибудь хранилище и считать по ночам MapReduce-ом. Или всё-таки прибегнем к потоковой обработке? У каждого подхода есть свои преимущества и недостатки: как очевидные, так и те, что не всегда лежат на поверхности. После небольшого введения и обзора терминов Виктор на основе примеров кода покажет, как использовать Hazelcast Jet для распределенной обработки потоковых данных.

Распределяй и властвуй — 2: Потоки данных наносят ответный удар

Viktor Gamov

Вы из тех, кто гонится за производительностью, запихивая в любой цикл Collection.parallelStream()? Задумываетесь, как обойтись без глобальной переделки кода и решить все проблемы с помощью большего количества серверов? Как получить распределённую обработку с линейным ростом производительности? Если подобные вопросы звучат у вас в голове, тогда этот доклад для вас. Оказывается, лишь немногие могут запускать Streams в распределенном режиме — Oracle Coherence, Infinispan, Hazelcast Jet — кто-то за деньги, кто-то бесплатно. На примере Hazelcast Jet Виктор расскажет обо всех деталях распределённой обработки. Кроме того, познакомит слушателей со своим другом Ориентированным Ациклическим Графом и покажет, как с его помощью организована обработка информации поверх API Java 8 Stream. Распределенная! Высокопроизводительная! «В памяти»!

[JBreak] Блеск И Нищета Распределенных Стримов - 04-04-2017

Viktor Gamov

Caching keeps the data in memory that either slow to calculate or originate from another underlying back-end system. In this presentation, you will see how to leverage Hazelcast as a distributed cache for Node.js applications. Hazelcast is an open source, elastic, self-balancing, self-healing and highly available in-memory computing platform provides fast, reliable access to scalable in-memory data. This hands-on talk will enable you to get started exploring Hazelcast for Node.js and give your project a jump start in speed and scale. In this talk, will be covered: - How distributed caching works in general. - Capabilities of Node.js client. - Real world use cases and best practices.

[OracleCode - SF] Distributed caching for your next node.js project

Viktor Gamov

Apache Spark is a distributed computation framework optimized to work in-memory, and heavily influenced by concepts from functional programming languages. Hazelcast - open source in-memory data grid capable of amazing feats of scale - provides wide range of distributed computing primitives computation, including ExecutorService, M/R and Aggregations frameworks. The nature of data exploration and analysis requires data scientists be able to ask questions that weren't planned to be asked—and get an answer fast! In this talk, Viktor will explore Spark and see how it works together with Hazelcast to provide a robust in-memory open-source big data analytics solution!

[OracleCode SF] In memory analytics with apache spark and hazelcast

Viktor Gamov

Java 8 introduced the Stream API as a modern, functional, and very powerful tool for processing collections of data. One of the main benefits of the Stream API is that it hides the details of iteration over the underlying data set, allowing for parallel processing within a single JVM, using a fork/join framework. I will talk about a Stream API implementation that enables parallel processing across many machines and many JVMs. With an explanation of internals of the implementation, I will give an introduction to the general design behind stream processing using DAG (directed acyclic graph) engines and how an actor-based implementation can provide in-memory performance while still leveraging industry-wide known frameworks as Java Streams API. https://www.jfokus.se/jfokus/talks.jsp#RidingtheJetStreams

[Jfokus] Riding the Jet Streams

Viktor Gamov

[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017

Viktor Gamov

Needing more performance from your Java applications? Is latency causing you stress? Repetitive loading of the data in applications, burning CPU time, taxing I/O / disk access? Too many applications caching the same data sets pushing the limits of your data management and application architecture? If so, take a look at JCache! This code-driven session demonstrates how to integrate Hazelcast Distributed Caches into your Spring or Java EE applications. Adding a bunch of annotations to a method can achieve an orders-of-magnitude speed improvement in applications with high-latency.

[Codemash] Caching Made "Bootiful"!

Viktor Gamov

Вы из тех, кто считает, что, распараллелив любой цикл, можно улучшить перформанс, и Collection.parallelStream() — ваш лучший друг? А как вам идея — вбросить ещё пачку машин и получить распределенную обработку? Интересно? Тогда для вас этот доклад обязателен к просмотру. Виктор познакомит слушателей со своим другом, Ориентированным (Направленным) Ациклическим Графом (или Маркизом?!), и покажет, как с его помощью была организована распределенная высокопроизводительная система обработки информации в памяти поверх нашего знакомого Java 8 Stream API.

[JokerConf] Верхом на реактивных стримах, 10/13/2016

Viktor Gamov

There is a perception in the Java community that JavaScript is a second-league interpreted language with the main purpose of making Web pages a little prettier. But JavaScript is a powerful, flexible, dynamically typed language. And today language has been experiencing its a revival driven by the interest in HTML5. Nashorn is a modern JavaScript engine available on JVM, and it’s already included with JDK8 builds. This presentation is about building polyglot application with Java and JavaScript.

JavaOne 2013: «Java and JavaScript - Shaken, Not Stirred»

Viktor Gamov

WebSockets: The Current State of the Most Valuable HTML5 API for Java Developers

Viktor Gamov

Functional UI testing of Adobe Flex RIA

Viktor Gamov

Testing Flex RIAs for NJ Flex user group

Viktor Gamov

Plus de Viktor Gamov (14)

Testing containers with TestContainers @ AJUG 7/18/2017

[Philly ETE] Java Puzzlers NG

Распределяй и властвуй — 2: Потоки данных наносят ответный удар

[JBreak] Блеск И Нищета Распределенных Стримов - 04-04-2017

[OracleCode - SF] Distributed caching for your next node.js project

[OracleCode SF] In memory analytics with apache spark and hazelcast

[Jfokus] Riding the Jet Streams

[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017

[Codemash] Caching Made "Bootiful"!

[JokerConf] Верхом на реактивных стримах, 10/13/2016

JavaOne 2013: «Java and JavaScript - Shaken, Not Stirred»

WebSockets: The Current State of the Most Valuable HTML5 API for Java Developers

Functional UI testing of Adobe Flex RIA

Testing Flex RIAs for NJ Flex user group

Dernier

Breaking the Code : A Guide to WhatsApp Business API.pdf

Meon Technology

Advanced Flow Concepts Every Developer Should Know

Peter Caitens

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Eric Wang (Software Engineer, @Uber) Uber has numerous deep learning models, most of which are highly complex with many layers and a vast number of features. Understanding how these models work is challenging and demands significant resources to experiment with various training algorithms and feature sets. With ML explainability, the ML team aims to bring transparency to these models, helping to clarify their predictions and behavior. This transparency also assists the operations and legal teams in explaining the reasons behind specific prediction outcomes. In this talk, Eric Wang will discuss the methods Uber used for explaining deep learning models and how we integrated these methods into the Uber AI Michelangelo ecosystem to support offline explaining.

AI/ML Infra Meetup | ML explainability in Michelangelo

Alluxio, Inc.

Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have. For more Tendenci AMS events, check out www.tendenci.com/events

Corporate Management | Session 3 of 3 | Tendenci AMS

Tendenci - The Open Source AMS (Association Management Software)

Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...

Abortion Clinic

This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.

top nidhi software solution freedownload

vrstrong314

A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf

kalichargn70th171

10 Essential Software Testing Tools You Need to Know About.pdf

kalichargn70th171

Agnieszka Andrzejewska - BIM School Course in Kraków

bim.edu.pl

Top Mobile App Development Companies 2024

XongoLab Technologies LLP

WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation

WSO2

The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month. The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies. However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News. Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!

SOCRadar Research Team: Latest Activities of IntelBroker

SOCRadar

Studiovity film pre-production and screenwriting software

info611746

Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...

rajkumar669520

Migrating your customer service data from Gorgias to Re:amaze can greatly enhance your support operations, but it requires detailed planning and precise execution. "A Guideline to Gorgias to Re:amaze Data Migration" is crafted to help businesses manage this transition smoothly and efficiently. This comprehensive guide ensures that all aspects of the data migration process are covered, from initial planning to post-migration support.

A Guideline to Gorgias to to Re:amaze Data Migration

Help Desk Migration

Secure Software Ecosystem Teqnation 2024

Soroosh Khodami

De mooiste recreatieve routes ontdekken met RouteYou en FME

Jelle | Nordend

Using IESVE for Room Loads Analysis - Australia & New Zealand

IES VE

Petr Matuska, Sales & Sales Engineering Lead, GraphAware Western Australia Police Force’s adoption of Neo4j and the GraphAware Hume graph analytics platform marks a significant advancement in data-driven policing. Facing the challenges of growing volumes of valuable data scattered in disconnected silos, the organisation successfully implemented Neo4j database and Hume, consolidating data from various sources into a dynamic knowledge graph. The result was a connected view of intelligence, making it easier for analysts to solve crime faster. The partnership between Neo4j and GraphAware in this project demonstrates the transformative impact of graph technology on law enforcement’s ability to leverage growing volumes of valuable data to prevent crime and protect communities.

GraphAware - Transforming policing with graph-based intelligence analysis

Neo4j

Data privacy is one of the most critical issues that businesses face. This presentation shares insights on the principles and best practices for ensuring the resilience and security of your workload. Drawing on a real-life project from the HR industry, the various challenges will be demonstrated: data protection, self-healing, business continuity, security, and transparency of data processing. This systematized approach allowed to create a secure AWS cloud infrastructure that not only met strict compliance rules but also exceeded the client's expectations.

Designing for Privacy in Amazon Web Services

KrzysztofKkol1

Dernier (20)

Breaking the Code : A Guide to WhatsApp Business API.pdf

Advanced Flow Concepts Every Developer Should Know

AI/ML Infra Meetup | ML explainability in Michelangelo

Corporate Management | Session 3 of 3 | Tendenci AMS

Abortion ^Clinic ^%[+971588192166''] Abortion Pill Al Ain (?@?) Abortion Pill...

top nidhi software solution freedownload

A Comprehensive Appium Guide for Hybrid App Automation Testing.pdf

10 Essential Software Testing Tools You Need to Know About.pdf

Agnieszka Andrzejewska - BIM School Course in Kraków

Top Mobile App Development Companies 2024

WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation

SOCRadar Research Team: Latest Activities of IntelBroker

Studiovity film pre-production and screenwriting software

Facemoji Keyboard released its 2023 State of Emoji report, outlining the most...

A Guideline to Gorgias to to Re:amaze Data Migration

Secure Software Ecosystem Teqnation 2024

De mooiste recreatieve routes ontdekken met RouteYou en FME

Using IESVE for Room Loads Analysis - Australia & New Zealand

GraphAware - Transforming policing with graph-based intelligence analysis

Designing for Privacy in Amazon Web Services

[DataSciCon] Divide, distribute and conquer stream v. batch

1. DIVIDE, DISTRIBUTE AND CONQUER:  STREAM V. BATCH

2. Stream v. Batch

3. Who am I?

4. Solutions Architect Who am I?

5. Solutions Architect Developer Advocate Who am I?

6. Solutions Architect Developer Advocate @gamussa in internetz Who am I?

8. @gamussa @confluentinc @DataSciCon BATCH PROCESSING Data at rest

9. @gamussa @confluentinc @DataSciCon Data and Queries Origin and processing

10. @gamussa @confluentinc @DataSciCon

11. @gamussa @confluentinc @DataSciCon Data…

12. @gamussa @confluentinc @DataSciCon Data…

13. @gamussa @confluentinc @DataSciCon ✓ … inherently immutable Data… ✓ … time-based

14. @gamussa @confluentinc @DataSciCon CRUD -> CR

15. @gamussa @confluentinc @DataSciCon Processing is a query

16. @gamussa @confluentinc @DataSciCon Processing is a query Function on full data set

17. @gamussa @confluentinc @DataSciCon Processing is a query Function on full data set Projection

18. @gamussa @confluentinc @DataSciCon Processing is a query Function on full data set Projection Aggregations

19. @gamussa @confluentinc @DataSciCon Processing is a query Function on full data set Projection Aggregations Joins

20. @gamussa @confluentinc @DataSciCon Lambda architecture origins http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html

21.

22.

23. @gamussa @confluentinc @DataSciCon https://mapr.com/developercentral/lambda-architecture/ Lambda Architecture

24. @gamussa @confluentinc @DataSciCon

25. @gamussa @confluentinc @DataSciCon TFW Trying to explain modern big data landscape

26. @gamussa @confluentinc @DataSciCon

27. @gamussa @confluentinc @DataSciCon STREAM PROCESSING Data is motion

28. @gamussa @confluentinc @DataSciCon Streaming Platform

29. @gamussa @confluentinc @DataSciCon Streaming Platform

30. @gamussa @confluentinc @DataSciCon

31. @gamussa @confluentinc @DataSciCon Interesting cases Before You Go

32. I FOUND YOUR LACK OF FAULT TOLERANCE DISTURBING

33. Data is too important to store it in one computer

34.

35.

36.

37.

38. @gamussa @confluentinc @DataSciCon How to process «infinite» data?

39. @gamussa @confluentinc @DataSciCon Time model

40. @gamussa @confluentinc @DataSciCon Time model Different use cases time semantics

41. @gamussa @confluentinc @DataSciCon Time model Different use cases time semantics Majority of use cases require event- time semantics

42. @gamussa @confluentinc @DataSciCon Time model Different use cases time semantics Majority of use cases require event- time semantics Other use cases may require processing-time or special variants like ingestion-time

43. @gamussa @confluentinc @DataSciCon Time Model

44. @gamussa @confluentinc @DataSciCon Time Model

45. @gamussa @confluentinc @DataSciCon Time Model

46. @gamussa @confluentinc @DataSciCon Windowing Input data, where colors represent  different users events Rectangles denote  different event-time  windows processing-time event-time windowing alice bob dave

47. @gamussa @confluentinc @DataSciCon https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101

48. @gamussa @confluentinc @DataSciCon Windowing Windowing is an operation that groups events Most commonly needed: time windows, session windows Examples: ✗Real-time monitoring: 5-minute averages ✗Reader behavior on a website: user browsing sessions

49. @gamussa @confluentinc @DataSciCon Out-of-order and late data Is very common in practice, not a rare corner case ✗Related to time model discussion

50. @gamussa @confluentinc @DataSciCon Out-of-order and late data

51. @gamussa @confluentinc @DataSciCon Out-of-order and late data Users with mobile phones enter  airplane, lose Internet connectivity

52. @gamussa @confluentinc @DataSciCon Out-of-order and late data Users with mobile phones enter  airplane, lose Internet connectivity Emails are being written  during the 10h flight

53. @gamussa @confluentinc @DataSciCon Out-of-order and late data Users with mobile phones enter  airplane, lose Internet connectivity Emails are being written  during the 10h flight Internet connectivity is restored,  phones will send queued emails now

54. @gamussa @confluentinc @DataSciCon Stream Processing: results

55. @gamussa @confluentinc @DataSciCon Stream Processing: results • Yes, it’s possible to get computation results in real time

56. @gamussa @confluentinc @DataSciCon Stream Processing: results • Yes, it’s possible to get computation results in real time • Windows – finite view of infinite data • Based on temporal characteristics of the evet

57. @gamussa @confluentinc @DataSciCon Stream Processing: results • Yes, it’s possible to get computation results in real time • Windows – finite view of infinite data • Based on temporal characteristics of the evet • Late event processing • You choose how long to wait

58. @gamussa @confluentinc @DataSciCon DEMO Let’s analyze flights

59. @gamussa @confluentinc @DataSciCon https://www.confluent.io/blog/predicting-flight-arrivals-with-the-apache-kafka-streams-api/

60. @gamussa @confluentinc @DataSciCon Example: Training Flight Prediction Model

61. @gamussa @confluentinc @DataSciCon https://github.com/confluentinc/online-inferencing-blog- application

62. @gamussa @confluentinc @DataSciCon Thanks! questions? @gamussa viktor@confluent.io

[DataSciCon] Divide, distribute and conquer stream v. batch

Recommandé

Recommandé

Contenu connexe

Similaire à [DataSciCon] Divide, distribute and conquer stream v. batch

Similaire à [DataSciCon] Divide, distribute and conquer stream v. batch (20)

Plus de Viktor Gamov

Plus de Viktor Gamov (14)

Dernier

Dernier (20)

[DataSciCon] Divide, distribute and conquer stream v. batch