Interactive Data Analysis with Apache Flink @ Flink Meetup in Berlin

•

16 j'aime•5,735 vues

This talk shows how we can use Apache Flink and Apache Zeppelin to do interactive data analysis. The examples show the usage of FlinkML to solve a linear regression and classification problem.

Technologie

Till Rohrmann
Flink PMC member
trohrmann@apache.org
@stsffap
Interactive Data Analysis
with Apache Flink

Exploratory Data Analysis
§  Visualize data
§  Calculate main
characteristics
§  Understand data and
ﬁnd possibly new
hypothesis
2

Read-Evaluate-Print Loop
§  New Scala shell offers REPL
§  Interactive queries
§  Let’s you explore data quickly
4

Problems
§  No visualization
§  No saving or replaying of written code
§  No assistance à Bad IDE
7

Notebooks
§  Web-based interactive
computation
environment
§  Combines rich text,
execution code, plots
and rich media
§  Storytelling
8

Apache Zeppelin
§  Web-based REPL with pluggable
interpreters
§  Since 2014 in the Apache Incubator
§  Supported interpreters:
•  Flink
•  Spark
•  Python
•  Markdown
•  Many more …
9

Word Count with Zeppelin
§  Find the 10 most frequent words with
more than 4 letters in the King James
version of the bible.
10

Linear regression
§  Let’s predict the inﬂuence of advertisement
spending on sales
§  Input data set:
http://www-bcf.usc.edu/~gareth/ISL/
Advertising.csv
§  Features:
•  TV advertisement money
•  Radio advertisement money
•  Newspaper advertisement money
§  Response:
•  Sales
15

Classiﬁcation
§  Let’s build a classiﬁer for insult detection
§  Kaggle challenge
https://www.kaggle.com/c/detecting-
insults-in-social-commentary
§  Label: 1 – Insult, 0 – No insult
§  Feature: Comment text
25

Conclusion
§  Interactive data analysis is really easy with
Apache Flink
§  Apache Zeppelin is great interactive
notebook
§  Zeppelin and Flink play well together to
solve machine learning tasks and more
28

Recommandé

Flink Community Update 2015 JuneMárton Balassi

Stateful Stream Processing at In-Memory SpeedJamie Grier

Extending the Yahoo Streaming BenchmarkJamie Grier

Slim Baltagi – Flink vs. SparkFlink Forward

A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)Robert Metzger

Marc Schwering – Using Flink with MongoDB to enhance relevancy in personaliza...Flink Forward

Jim Dowling – Interactive Flink analytics with HopsWorks and ZeppelinFlink Forward

Suneel Marthi – BigPetStore Flink: A Comprehensive Blueprint for Apache FlinkFlink Forward

Recommandé

Flink Community Update 2015 JuneMárton Balassi

Stateful Stream Processing at In-Memory SpeedJamie Grier

Extending the Yahoo Streaming BenchmarkJamie Grier

Slim Baltagi – Flink vs. SparkFlink Forward

A Data Streaming Architecture with Apache Flink (berlin Buzzwords 2016)Robert Metzger

Marc Schwering – Using Flink with MongoDB to enhance relevancy in personaliza...Flink Forward

Jim Dowling – Interactive Flink analytics with HopsWorks and ZeppelinFlink Forward

Suneel Marthi – BigPetStore Flink: A Comprehensive Blueprint for Apache FlinkFlink Forward

Apache Spark vs Apache FlinkAKASH SIHAG

Cascalog at May Bay Area Hadoop User Groupnathanmarz

Fabian Hueske – Cascading on FlinkFlink Forward

Rental Cars and Industrialized Learning to Rank with Sean DownesDatabricks

QCon London - Stream Processing with Apache FlinkRobert Metzger

January 2016 Flink Community Update & Roadmap 2016Robert Metzger

Extending the Yahoo Streaming Benchmark + MapR BenchmarksJamie Grier

Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & KafkaFlink Forward

January 2011 HUG: Kafka PresentationYahoo Developer Network

Community Update May 2016 (January - May) | Berlin Apache Flink MeetupRobert Metzger

Hopsworks - Self-Service Spark/Flink/Kafka/HadoopJim Dowling

Pinot: Near Realtime Analytics @ UberXiang Fu

Bullet: A Real Time Data Query EngineDataWorks Summit

Stream All Things—Patterns of Modern Data Integration with Gwen ShapiraDatabricks

Apache Flink community Update for March 2016 - Slim BaltagiSlim Baltagi

Presto at TwitterBill Graham

Building Data Pipelines in PythonC4Media

Apache Flink: Streaming Done Right @ FOSDEM 2016Till Rohrmann

SSR: Structured Streaming for R and Machine Learningfelixcss

K. Tzoumas & S. Ewen – Flink Forward KeynoteFlink Forward

Big Data visualization with Apache Spark and Zeppelinprajods

Taking a look under the hood of Apache Flink's relational APIs.Fabian Hueske

Contenu connexe

Tendances

Apache Spark vs Apache FlinkAKASH SIHAG

Cascalog at May Bay Area Hadoop User Groupnathanmarz

Fabian Hueske – Cascading on FlinkFlink Forward

Rental Cars and Industrialized Learning to Rank with Sean DownesDatabricks

QCon London - Stream Processing with Apache FlinkRobert Metzger

January 2016 Flink Community Update & Roadmap 2016Robert Metzger

Extending the Yahoo Streaming Benchmark + MapR BenchmarksJamie Grier

Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & KafkaFlink Forward

January 2011 HUG: Kafka PresentationYahoo Developer Network

Community Update May 2016 (January - May) | Berlin Apache Flink MeetupRobert Metzger

Hopsworks - Self-Service Spark/Flink/Kafka/HadoopJim Dowling

Pinot: Near Realtime Analytics @ UberXiang Fu

Bullet: A Real Time Data Query EngineDataWorks Summit

Stream All Things—Patterns of Modern Data Integration with Gwen ShapiraDatabricks

Apache Flink community Update for March 2016 - Slim BaltagiSlim Baltagi

Presto at TwitterBill Graham

Building Data Pipelines in PythonC4Media

Apache Flink: Streaming Done Right @ FOSDEM 2016Till Rohrmann

SSR: Structured Streaming for R and Machine Learningfelixcss

K. Tzoumas & S. Ewen – Flink Forward KeynoteFlink Forward

Tendances (20)

Apache Spark vs Apache Flink

Cascalog at May Bay Area Hadoop User Group

Fabian Hueske – Cascading on Flink

Rental Cars and Industrialized Learning to Rank with Sean Downes

QCon London - Stream Processing with Apache Flink

January 2016 Flink Community Update & Roadmap 2016

Extending the Yahoo Streaming Benchmark + MapR Benchmarks

Mohamed Amine Abdessemed – Real-time Data Integration with Apache Flink & Kafka

January 2011 HUG: Kafka Presentation

Community Update May 2016 (January - May) | Berlin Apache Flink Meetup

Hopsworks - Self-Service Spark/Flink/Kafka/Hadoop

Pinot: Near Realtime Analytics @ Uber

Bullet: A Real Time Data Query Engine

Stream All Things—Patterns of Modern Data Integration with Gwen Shapira

Apache Flink community Update for March 2016 - Slim Baltagi

Presto at Twitter

Building Data Pipelines in Python

Apache Flink: Streaming Done Right @ FOSDEM 2016

SSR: Structured Streaming for R and Machine Learning

K. Tzoumas & S. Ewen – Flink Forward Keynote

En vedette

Big Data visualization with Apache Spark and Zeppelinprajods

Taking a look under the hood of Apache Flink's relational APIs.Fabian Hueske

Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...Till Rohrmann

Step-by-Step Introduction to Apache Flink Slim Baltagi

Flink Gelly - Karlsruhe - June 2015Andra Lungu

Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015Till Rohrmann

Fault Tolerance and Job Recovery in Apache Flink @ FlinkForward 2015Till Rohrmann

Streaming Data Flow with Apache Flink @ Paris Flink Meetup 2015Till Rohrmann

Flink Streaming @BudapestDataGyula Fóra

Gelly-Stream: Single-Pass Graph Streaming Analytics with Apache FlinkVasia Kalavri

Introduction to Apache Flink - Fast and reliable big data processingTill Rohrmann

Flink Streaming Berlin MeetupMárton Balassi

Streaming Analytics & CEP - Two sides of the same coin?Till Rohrmann

The Stream Processor as the Database - Apache Flink @ Berlin buzzwords Stephan Ewen

Machine Learning with Apache Flink at Stockholm Machine Learning GroupTill Rohrmann

Stephan Ewen - Running Flink EverywhereFlink Forward

Unified Stream and Batch Processing with Apache FlinkDataWorks Summit/Hadoop Summit

Robert Metzger - Connecting Apache Flink to the World - Reviewing the streami...Flink Forward

Apache Flink at Strata San Jose 2016Kostas Tzoumas

Performance Comparison of Streaming Big Data PlatformsDataWorks Summit/Hadoop Summit

En vedette (20)

Big Data visualization with Apache Spark and Zeppelin

Taking a look under the hood of Apache Flink's relational APIs.

Dynamic Scaling: How Apache Flink Adapts to Changing Workloads (at FlinkForwa...

Step-by-Step Introduction to Apache Flink

Flink Gelly - Karlsruhe - June 2015

Computing recommendations at extreme scale with Apache Flink @Buzzwords 2015

Fault Tolerance and Job Recovery in Apache Flink @ FlinkForward 2015

Streaming Data Flow with Apache Flink @ Paris Flink Meetup 2015

Flink Streaming @BudapestData

Gelly-Stream: Single-Pass Graph Streaming Analytics with Apache Flink

Introduction to Apache Flink - Fast and reliable big data processing

Flink Streaming Berlin Meetup

Streaming Analytics & CEP - Two sides of the same coin?

The Stream Processor as the Database - Apache Flink @ Berlin buzzwords

Machine Learning with Apache Flink at Stockholm Machine Learning Group

Stephan Ewen - Running Flink Everywhere

Unified Stream and Batch Processing with Apache Flink

Robert Metzger - Connecting Apache Flink to the World - Reviewing the streami...

Apache Flink at Strata San Jose 2016

Performance Comparison of Streaming Big Data Platforms

Similaire à Interactive Data Analysis with Apache Flink @ Flink Meetup in Berlin

Pyspark vs Spark Let's Unravel the Bond!ankitbhandari32

GraphQL Europe RecapPhilipp Sporrer

Intro to Big Data Analytics using Apache Spark and Apache ZeppelinAlex Zeltov

Ruby on Rails (RoR) as a back-end processor for Apex Espen Brækken

ODSC East 2017 - Reproducible Research at Scale with Apache Zeppelin and SparkCarolyn Duby

Enabling exploratory data science with Spark and RDatabricks

Strata NYC 2015 - Supercharging R with Apache SparkDatabricks

Why scala for data scienceGuglielmo Iozzia

Unified Batch and Real-Time Stream Processing Using Apache FlinkSlim Baltagi

Intro to Spark with ZeppelinHortonworks

Integrating Apache Spark and NiFi for Data LakesDataWorks Summit/Hadoop Summit

Learn Apache Spark: A Comprehensive GuideWhizlabs

Apache Flink - A Stream Processing EngineAljoscha Krettek

20160512 apache-spark-for-everyoneAmanda Casari

Data science lifecycle with Apache ZeppelinDataWorks Summit/Hadoop Summit

Apache BookKeeper: A High Performance and Low Latency Storage ServiceSijie Guo

Flink Forward SF 2017: Dean Wampler - Streaming Deep Learning Scenarios with...Flink Forward

Taking Splunk to the Next Level - TechnicalSplunk

Taking Splunk to the Next Level – ArchitectureSplunk

sparklyr - Jeff AllenSri Ambati

Similaire à Interactive Data Analysis with Apache Flink @ Flink Meetup in Berlin (20)

Pyspark vs Spark Let's Unravel the Bond!

GraphQL Europe Recap

Intro to Big Data Analytics using Apache Spark and Apache Zeppelin

Ruby on Rails (RoR) as a back-end processor for Apex

ODSC East 2017 - Reproducible Research at Scale with Apache Zeppelin and Spark

Enabling exploratory data science with Spark and R

Strata NYC 2015 - Supercharging R with Apache Spark

Why scala for data science

Unified Batch and Real-Time Stream Processing Using Apache Flink

Intro to Spark with Zeppelin

Integrating Apache Spark and NiFi for Data Lakes

Learn Apache Spark: A Comprehensive Guide

Apache Flink - A Stream Processing Engine

20160512 apache-spark-for-everyone

Data science lifecycle with Apache Zeppelin

Apache BookKeeper: A High Performance and Low Latency Storage Service

Flink Forward SF 2017: Dean Wampler - Streaming Deep Learning Scenarios with...

Taking Splunk to the Next Level - Technical

Taking Splunk to the Next Level – Architecture

sparklyr - Jeff Allen

Plus de Till Rohrmann

Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...Till Rohrmann

Apache flink 1.7 and BeyondTill Rohrmann

Elastic Streams at Scale @ Flink Forward 2018 BerlinTill Rohrmann

Scaling stream data pipelines with Pravega and Apache FlinkTill Rohrmann

Modern Stream Processing With Apache Flink @ GOTO Berlin 2017Till Rohrmann

Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup BerlinTill Rohrmann

Apache Flink® Meets Apache Mesos® and DC/OSTill Rohrmann

From Apache Flink® 1.3 to 1.4Till Rohrmann

Apache Flink and More @ MesosCon Asia 2017Till Rohrmann

Redesigning Apache Flink's Distributed Architecture @ Flink Forward 2017Till Rohrmann

Gilbert: Declarative Sparse Linear Algebra on Massively Parallel Dataflow Sys...Till Rohrmann

Plus de Till Rohrmann (11)

Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink F...

Apache flink 1.7 and Beyond

Elastic Streams at Scale @ Flink Forward 2018 Berlin

Scaling stream data pipelines with Pravega and Apache Flink

Modern Stream Processing With Apache Flink @ GOTO Berlin 2017

Apache Flink Meets Apache Mesos And DC/OS @ Mesos Meetup Berlin

Apache Flink® Meets Apache Mesos® and DC/OS

From Apache Flink® 1.3 to 1.4

Apache Flink and More @ MesosCon Asia 2017

Redesigning Apache Flink's Distributed Architecture @ Flink Forward 2017

Gilbert: Declarative Sparse Linear Algebra on Massively Parallel Dataflow Sys...

Dernier

CNv6 Instructor Chapter 6 Quality of Servicegiselly40

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

Histor y of HAM Radio presentation slidevu2urc

Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer

Slack Application Development 101 Slidespraypatel2

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia

Salesforce Community Group Quito, Salesforce 101Paola De la Torre

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal

Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2

A Domino Admins Adventures (Engage 2024)Gabriella Davis

Developing An App To Navigate The Roads of BrazilV3cube

GenCyber Cyber Security Day PresentationMichael W. Hawkins

Scaling API-first – The story of a global engineering organizationRadu Cotescu

Automating Google Workspace (GWS) & more with Apps Scriptwesley chun

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge

Dernier (20)

CNv6 Instructor Chapter 6 Quality of Service

08448380779 Call Girls In Friends Colony Women Seeking Men

Injustice - Developers Among Us (SciFiDevCon 2024)

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Histor y of HAM Radio presentation slide

Axa Assurance Maroc - Insurer Innovation Award 2024

Slack Application Development 101 Slides

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...

Salesforce Community Group Quito, Salesforce 101

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service

Exploring the Future Potential of AI-Enabled Smartphone Processors

A Domino Admins Adventures (Engage 2024)

Developing An App To Navigate The Roads of Brazil

GenCyber Cyber Security Day Presentation

Scaling API-first – The story of a global engineering organization

Automating Google Workspace (GWS) & more with Apps Script

IAC 2024 - IA Fast Track to Search Focused AI Solutions

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

How to Troubleshoot Apps for the Modern Connected Worker

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Interactive Data Analysis with Apache Flink @ Flink Meetup in Berlin

1. Till Rohrmann Flink PMC member trohrmann@apache.org @stsffap Interactive Data Analysis with Apache Flink

2. Data Analysis 1

3. Exploratory Data Analysis §  Visualize data §  Calculate main characteristics §  Understand data and ﬁnd possibly new hypothesis 2

4. Data Analysts 3

5. Read-Evaluate-Print Loop §  New Scala shell offers REPL §  Interactive queries §  Let’s you explore data quickly 4

6. Scala Shell 5

7. Simple Scala Shell Example 6

8. Problems §  No visualization §  No saving or replaying of written code §  No assistance à Bad IDE 7

9. Notebooks §  Web-based interactive computation environment §  Combines rich text, execution code, plots and rich media §  Storytelling 8

10. Apache Zeppelin §  Web-based REPL with pluggable interpreters §  Since 2014 in the Apache Incubator §  Supported interpreters: •  Flink •  Spark •  Python •  Markdown •  Many more … 9

11. Word Count with Zeppelin §  Find the 10 most frequent words with more than 4 letters in the King James version of the bible. 10

12. 11

13. 12

14. 13

15. 14

16. Linear regression §  Let’s predict the inﬂuence of advertisement spending on sales §  Input data set: http://www-bcf.usc.edu/~gareth/ISL/ Advertising.csv §  Features: •  TV advertisement money •  Radio advertisement money •  Newspaper advertisement money §  Response: •  Sales 15

17. 16

18. 17

19. 18

20. 19

21. 20

22. 21

23. 22

24. 23

25. 24

26. Classiﬁcation §  Let’s build a classiﬁer for insult detection §  Kaggle challenge https://www.kaggle.com/c/detecting- insults-in-social-commentary §  Label: 1 – Insult, 0 – No insult §  Feature: Comment text 25

27. 26

28. 27

29. Conclusion §  Interactive data analysis is really easy with Apache Flink §  Apache Zeppelin is great interactive notebook §  Zeppelin and Flink play well together to solve machine learning tasks and more 28

30. 29

31. ﬂink.apache.org @ApacheFlink