Introduing spark

•

0 j'aime•160 vues

Taotao Li

a brief introduction to Apache Spark

Ingénierie

taotao.li@datayes.com
03/11/2016
Introducing Spark

Copyright © 2014 DataYes. All rights reserved
Agenda
Spark ! When, What, Why ?1
Basic Concepts in Spark2
Programming Model in Spark3
Demo & Next4
5 Q & A

Copyright © 2014 DataYes. All rights reserved
Spark ! When, What, Why ?
Top-level in Apache
2009 : Spark birth in AMPLab@UCB
2010 : open source
Into Apache incubator
2009~2010
2013
2014
New Stage : more than an open
source project

Copyright © 2014 DataYes. All rights reserved
Spark ! When, What, Why ?
From official: Apache Spark™ is a fast and general engine for large-scale data processing.
Key Points:
● A framework
● Birth for large-scale data processing
● Generalize programming model for data processing [ more than MR ]
● Provides high-level APIs : Scala, Python, R, Java
● Arm to teeth : SQL, Streaming, Machine Learning, GraphX
● Compatible with previous ecology : hadoop, mesos, hdfs, cassandra, hbase, s3 …

Copyright © 2014 DataYes. All rights reserved
Spark ! When, What, Why ?
● General
● Fast in develop
○ REPL explore
○ RDD operations
○ Less code
● Fast in processing
● Compatible
● Packges and 3-party packages
● Memory, cheaper and cheaper
● Company who accepts Spark

Copyright © 2014 DataYes. All rights reserved
Spark ! When, What, Why ?

Copyright © 2014 DataYes. All rights reserved
Spark ! When, What, Why ?
DDR4-3000 288-pin DIMM 4x4GB Price Trend

Copyright © 2014 DataYes. All rights reserved
Basic Concepts in Spark

Copyright © 2014 DataYes. All rights reserved
Basic Concepts in Spark
● Driver, Master, Worker, Executor
● Application
● SparkContext, i.e : sc
● RDD
● Transform & Action in RDD
need more ? check : 『 Spark 』2. spark 基本概念解析

Copyright © 2014 DataYes. All rights reserved
Programming Model in Spark

Copyright © 2014 DataYes. All rights reserved
Programming Model in Spark
Three basic steps to build a Spark Application
● load dataset
○ static dataset
○ dynamic dataset
● Processing
○ RDD operation
○ UDF
○ Cache
● Output Display
○ collect
○ store in database, file system ...

Copyright © 2014 DataYes. All rights reserved
Demo & Next
● Wrapper Spark for Uqer Use Cases
●
● Try Tungsten
●
● Dataframe & Datasets
●
● SQL & Mlib & Streaming
●
● 3-party package wrapper [sklearn, pandas, numpy ...etc]

Copyright © 2014 DataYes. All rights reserved
Demo & Next

Copyright © 2014 DataYes. All rights reserved
Demo & Next
● Monte Carlo in Spark
● Spark in finance : index similarity calculating
● Spark in finance : distributed backtesting strategy

Copyright © 2014 DataYes. All rights reserved
Demo, Demo, Demo
Q & A

Recommandé

Performant data processing with PySpark, SparkR and DataFrame APIRyuji Tamagawa

The SparkSQL things you maybe confusevito jeng

High Performance Python on Apache SparkWes McKinney

Apache spark linkedinYukti Kaura

Introduction to apache sparkUserReport

Performance of Spark vs MapReduceEdureka!

My Data Journey with Python (SciPy 2015 Keynote)Wes McKinney

Python Data Wrangling: Preparing for the FutureWes McKinney

Recommandé

Performant data processing with PySpark, SparkR and DataFrame APIRyuji Tamagawa

The SparkSQL things you maybe confusevito jeng

High Performance Python on Apache SparkWes McKinney

Apache spark linkedinYukti Kaura

Introduction to apache sparkUserReport

Performance of Spark vs MapReduceEdureka!

My Data Journey with Python (SciPy 2015 Keynote)Wes McKinney

Python Data Wrangling: Preparing for the FutureWes McKinney

Presto Fast SQL on AnythingAlluxio, Inc.

An Incomplete Data Tools Landscape for Hackers in 2015Wes McKinney

Spark for big data analyticsEdureka!

Big data Processing with Apache Spark & ScalaEdureka!

Intro to Apache SparkBTI360

Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)Spark Summit

Spark CoreTodd McGrath

The Computer Science Behind a modern Distributed DatabaseArangoDB Database

How Apache Arrow and Parquet boost cross-language interoperabilityUwe Korn

Koalas: Unifying Spark and pandas APIsTakuya UESHIN

Apache Arrow -- Cross-language development platform for in-memory dataWes McKinney

Operational Tips for Deploying SparkDatabricks

Apache Spark 101Abdullah Çetin ÇAVDAR

Introduction to SparkROlgun Aydın

Extending Pandas using Apache Arrow and NumbaUwe Korn

Introduction to Apache SparkSamy Dindane

Introduction to apache spark Aakashdata

Ibis: Scaling the Python Data ExperienceWes McKinney

PyData London 2017 – Efficient and portable DataFrame storage with Apache Par...Uwe Korn

First impressions of SparkR: our own machine learning algorithmInfoFarm

kapilumakkapil umak

Propuestas para pensar la enseñanza en la diversidadLauStok

Contenu connexe

Tendances

Presto Fast SQL on AnythingAlluxio, Inc.

An Incomplete Data Tools Landscape for Hackers in 2015Wes McKinney

Spark for big data analyticsEdureka!

Big data Processing with Apache Spark & ScalaEdureka!

Intro to Apache SparkBTI360

Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)Spark Summit

Spark CoreTodd McGrath

The Computer Science Behind a modern Distributed DatabaseArangoDB Database

How Apache Arrow and Parquet boost cross-language interoperabilityUwe Korn

Koalas: Unifying Spark and pandas APIsTakuya UESHIN

Apache Arrow -- Cross-language development platform for in-memory dataWes McKinney

Operational Tips for Deploying SparkDatabricks

Apache Spark 101Abdullah Çetin ÇAVDAR

Introduction to SparkROlgun Aydın

Extending Pandas using Apache Arrow and NumbaUwe Korn

Introduction to Apache SparkSamy Dindane

Introduction to apache spark Aakashdata

Ibis: Scaling the Python Data ExperienceWes McKinney

PyData London 2017 – Efficient and portable DataFrame storage with Apache Par...Uwe Korn

First impressions of SparkR: our own machine learning algorithmInfoFarm

Tendances (20)

Presto Fast SQL on Anything

An Incomplete Data Tools Landscape for Hackers in 2015

Spark for big data analytics

Big data Processing with Apache Spark & Scala

Intro to Apache Spark

Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)

Spark Core

The Computer Science Behind a modern Distributed Database

How Apache Arrow and Parquet boost cross-language interoperability

Koalas: Unifying Spark and pandas APIs

Apache Arrow -- Cross-language development platform for in-memory data

Operational Tips for Deploying Spark

Apache Spark 101

Introduction to SparkR

Extending Pandas using Apache Arrow and Numba

Introduction to Apache Spark

Introduction to apache spark

Ibis: Scaling the Python Data Experience

PyData London 2017 – Efficient and portable DataFrame storage with Apache Par...

First impressions of SparkR: our own machine learning algorithm

En vedette

kapilumakkapil umak

Propuestas para pensar la enseñanza en la diversidadLauStok

Democrats-Clinton&Obamaguest8d6534

Love Of A Tree. (Purple magosteen / Mangostán)Cachi Chien

Google Analytics Report on how to reduce bounce rateStreebo

Human Alphabets 4Sotirios Raptis

EMC World 2016 - code.05 Automating your Physical Data Center with RackHD{code}

SunGuard implementation kickoff meeting 07142016 pmWarren Payne PMP, M.S. IT Project Management

Resolución examen residentado 2016 26 de junio 2016Villamedic Group

En vedette (9)

kapilumak

Propuestas para pensar la enseñanza en la diversidad

Democrats-Clinton&Obama

Love Of A Tree. (Purple magosteen / Mangostán)

Google Analytics Report on how to reduce bounce rate

Human Alphabets 4

EMC World 2016 - code.05 Automating your Physical Data Center with RackHD

SunGuard implementation kickoff meeting 07142016 pm

Resolución examen residentado 2016 26 de junio 2016

Similaire à Introduing spark

Spark and Hadoop Technology Avinash Gautam

Spark 101Shahaf Azriely {TopLinked} ☁

Spark Summit EU 2015: Lessons from 300+ production usersDatabricks

Introduction to sparkHome

Meet SparkChicago Hadoop Users Group

Learn Apache Spark: A Comprehensive GuideWhizlabs

YARN Ready: Apache Spark Hortonworks

Spark1.0での動作検証 - Hadoopユーザ・デベロッパから見たSparkへの期待（Hadoop Conference Japan 2014）NTT DATA OSS Professional Services

Getting Started with Spark ScalaKnoldus Inc.

Apache Spark in Scientific ApplciationsDr. Mirko Kämpf

Apache Spark in Scientific ApplicationsDr. Mirko Kämpf

spark_v1_2Frank Schroeter

Started with-apache-sparkHappiest Minds Technologies

shark attack on sql-on-hadoop Talk at BerlinBuzzwords 2014Gerd König

Hive on spark berlin buzzwordsSzehon Ho

Introduction to Apache Sparkdatamantra

Detailed guide to the Apache Spark FrameworkAegis Software Canada

What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...Simplilearn

Review on Apache Spark TechnologyIRJET Journal

Spark Interview Questions and Answers | Apache Spark Interview Questions | Sp...Edureka!

Similaire à Introduing spark (20)

Spark and Hadoop Technology

Spark 101

Spark Summit EU 2015: Lessons from 300+ production users

Introduction to spark

Meet Spark

Learn Apache Spark: A Comprehensive Guide

YARN Ready: Apache Spark

Spark1.0での動作検証 - Hadoopユーザ・デベロッパから見たSparkへの期待（Hadoop Conference Japan 2014）

Getting Started with Spark Scala

Apache Spark in Scientific Applciations

Apache Spark in Scientific Applications

spark_v1_2

Started with-apache-spark

shark attack on sql-on-hadoop Talk at BerlinBuzzwords 2014

Hive on spark berlin buzzwords

Introduction to Apache Spark

Detailed guide to the Apache Spark Framework

What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...

Review on Apache Spark Technology

Spark Interview Questions and Answers | Apache Spark Interview Questions | Sp...

Dernier

Glass Ceramics: Processing and PropertiesPrabhanshu Chaturvedi

Online banking management system project.pdfKamal Acharya

Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...Dr.Costas Sachpazis

Booking open Available Pune Call Girls Koregaon Park 6297143586 Call Hot Ind...Call Girls in Nagpur High Profile

(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7Call Girls in Nagpur High Profile Call Girls

Call Girls in Nagpur Suman Call 7001035870 Meet With Nagpur EscortsCall Girls in Nagpur High Profile

UNIT-V FMM.HYDRAULIC TURBINE - Construction and workingrknatarajan

AKTU Computer Networks notes --- Unit 3.pdfankushspencer015

MANUFACTURING PROCESS-II UNIT-5 NC MACHINE TOOLSSIVASHANKAR N

The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...ranjana rawat

High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur EscortsCall Girls in Nagpur High Profile

Java Programming :Event Handling(Types of Events)simmis5

The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...ranjana rawat

(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...ranjana rawat

(SHREYA) Chakan Call Girls Just Call 7001035870 [ Cash on Delivery ] Pune Esc...ranjana rawat

Introduction to IEEE STANDARDS and its different types.pptxupamatechverse

Extrusion Processes and Their Limitations120cr0395

University management System project report..pdfKamal Acharya

Booking open Available Pune Call Girls Pargaon 6297143586 Call Hot Indian Gi...Call Girls in Nagpur High Profile

CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete RecordAsst.prof M.Gokilavani

Dernier (20)

Glass Ceramics: Processing and Properties

Online banking management system project.pdf

Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...

Booking open Available Pune Call Girls Koregaon Park 6297143586 Call Hot Ind...

(INDIRA) Call Girl Aurangabad Call Now 8617697112 Aurangabad Escorts 24x7

Call Girls in Nagpur Suman Call 7001035870 Meet With Nagpur Escorts

UNIT-V FMM.HYDRAULIC TURBINE - Construction and working

AKTU Computer Networks notes --- Unit 3.pdf

MANUFACTURING PROCESS-II UNIT-5 NC MACHINE TOOLS

The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...

High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur Escorts

Java Programming :Event Handling(Types of Events)

The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...

(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...

(SHREYA) Chakan Call Girls Just Call 7001035870 [ Cash on Delivery ] Pune Esc...

Introduction to IEEE STANDARDS and its different types.pptx

Extrusion Processes and Their Limitations

University management System project report..pdf

Booking open Available Pune Call Girls Pargaon 6297143586 Call Hot Indian Gi...

CCS335 _ Neural Networks and Deep Learning Laboratory_Lab Complete Record

Introduing spark

2. taotao.li@datayes.com 03/11/2016 Introducing Spark

4. Copyright © 2014 DataYes. All rights reserved Spark ! When, What, Why ? Top-level in Apache 2009 : Spark birth in AMPLab@UCB 2010 : open source Into Apache incubator 2009~2010 2013 2014 New Stage : more than an open source project

5. Copyright © 2014 DataYes. All rights reserved Spark ! When, What, Why ? From official: Apache Spark™ is a fast and general engine for large-scale data processing. Key Points: ● A framework ● Birth for large-scale data processing ● Generalize programming model for data processing [ more than MR ] ● Provides high-level APIs : Scala, Python, R, Java ● Arm to teeth : SQL, Streaming, Machine Learning, GraphX ● Compatible with previous ecology : hadoop, mesos, hdfs, cassandra, hbase, s3 …

6. Copyright © 2014 DataYes. All rights reserved Spark ! When, What, Why ? ● General ● Fast in develop ○ REPL explore ○ RDD operations ○ Less code ● Fast in processing ● Compatible ● Packges and 3-party packages ● Memory, cheaper and cheaper ● Company who accepts Spark

10. Copyright © 2014 DataYes. All rights reserved Basic Concepts in Spark ● Driver, Master, Worker, Executor ● Application ● SparkContext, i.e : sc ● RDD ● Transform & Action in RDD need more ? check : 『 Spark 』2. spark 基本概念解析

12. Copyright © 2014 DataYes. All rights reserved Programming Model in Spark Three basic steps to build a Spark Application ● load dataset ○ static dataset ○ dynamic dataset ● Processing ○ RDD operation ○ UDF ○ Cache ● Output Display ○ collect ○ store in database, file system ...

13. Copyright © 2014 DataYes. All rights reserved Demo & Next ● Wrapper Spark for Uqer Use Cases ● ● Try Tungsten ● ● Dataframe & Datasets ● ● SQL & Mlib & Streaming ● ● 3-party package wrapper [sklearn, pandas, numpy ...etc]

17. 谢谢