SlideShare une entreprise Scribd logo
1  sur  38
Télécharger pour lire hors ligne
Hadoop for Data Science
Donald Miner
Data Science MD
October 9, 2013
About Don
@donaldpminer
dminer@clearedgeit.com
I’ll talk about…
Intro to Hadoop (HDFS and MapReduce)
Some reasons why I think Hadoop is cool
(is this cliché yet?)
Step 1: Hadoop
Step 2: ????
Step 3: Data Science!
Some examples of data science work on Hadoop
What can Hadoop do to enable data science work?
Hadoop Distributed File System
HDFS
• Stores files in folders (that’s it)
– Nobody cares what’s in your files
• Chunks large files into blocks (~64MB-2GB)
• 3 replicates of each block (better safe than sorry)
• Blocks are scattered all over the place
FILE BLOCKS
MapReduce
• Analyzes raw data in HDFS where the data is
• Jobs are split into Mappers and Reducers
Reducers (you code this, too)
Automatically Groups by the
mapper’s output key
Aggregate, count, statistics
Outputs to HDFS
Mappers (you code this)
Loads data from HDFS
Filter, transform, parse
Outputs (key, value) pairs
Hadoop Ecosystem
• Higher-level languages like Pig and Hive
• HDFS Data systems like HBase and Accumulo
• Close friends like ZooKeeper, Flume, Storm,
Cassandra, Avro
Mahout
• Mahout is a Machine
Library
• Has both MapReduce
and non-parallel
implementations of a
number of algorithms:
– Recommenders
– Clustering
– Classification
Cool Thing #1: Linear Scalability
• HDFS and MapReduce
scale linearly
• If you have twice as
many computers, jobs
run twice as fast
• If you have twice as
much data, jobs run
twice as slow
• If you have twice as
many computers, you
can store twice as much
data
DATA LOCALITY!!
Cool Thing #2: Schema on Read
LOAD DATA FIRST, ASK QUESTIONS LATER
Data is parsed/interpreted as it is loaded out of HDFS
What implications does this have?
BEFORE:
ETL, schema design upfront,
tossing out original data,
comprehensive data study
Keep original data around!
Have multiple views of the same data!
Work with unstructured data sooner!
Store first, figure out what to do with it later!
WITH HADOOP:
Cool Thing #3: Transparent Parallelism
Network programming?
Inter-process communication?
Threading?
Distributed stuff?
With MapReduce, I DON’T CARE
Your solution
… I just have to be sure my solution fits into this tiny box
Fault tolerance?
Code deployment?
RPC?
Message passing?
Locking?
MapReduce
Framework
Data storage?
Scalability?
Data center fires?
Cool Thing #4: Unstructured Data
• Unstructured data:
media, text,
forms, log data
lumped structured data
• Query languages like SQL
and Pig assume some sort
of “structure”
• MapReduce is just Java:
You can do anything Java can
do in a Mapper or Reducer
The rest of the talk
• Four threads:
– Data exploration
– Classification
– NLP
– Recommender systems
I’m using these to illustrate some points
Exploration
• Hadoop is great at exploring data!
• I like to explore data in a couple ways:
– Filtering
– Sampling
– Summarization
– Evaluate cleanliness
• I like to spend 50% of my time
doing exploration
(but unfortunately it’s the
first thing to get cut)
Filtering
• Filtering is like a microscope:
I want to take a closer look at a subset
• In MapReduce, you do this in the mapper
• Identify nasty records you want to get rid of
• Examples:
– Only Baltimore data
– Remove gibberish
– Only 5 minutes
Sampling
• Hadoop isn’t the king of interactive analysis
• Sampling is a good way to grab a set of data
then work with it locally (Excel?)
• Pig has a handy SAMPLE keyword
• Types of sampling:
– Sample randomly across the entire data set
– Sub-graph extraction
– Filters (from the last slide)
Summarization
• Summarization is a bird’s-eye view
• MapReduce is good at summarization:
– Mappers extract the group-by keys
– Reducers do the aggregation
• I like to:
– Count number, get stdev, get average, get min/max of
records in several groups
– Count nulls in columns
(if applicable)
– Grab top-10 lists
Evaluating Cleanliness
• I’ve never been burned twice
• Things to check for:
– Fields that shouldn’t be null that are
– Duplicates (does unique records=records?)
– Dates (look for 1970; look at formats; time zones)
– Things that should be normalized
– Keys that are different because of trash
e.g. “ abc “ != “abc”
SQL/RDBMS =
What’s the point?
• Hadoop is really good at this stuff!
• You probably have a lot of data and a lot of it
is garbage!
• Take the time to do this and your further work
will be much easier
• It’s hard to tell what methods
you should use until you
explore your data
Classification
• Classification is taking feature vectors (derived from
your data), and then guessing some sort of label
– E.g.,
sunny, Saturday, summer -> play tennis
rainy, Wednesday, winter -> don’t play tennis
• Most classification algorithms aren’t easily
parallelizable or have good implementations
• You need a training set of true feature vectors and
labels… how often is your data labeled?
• I’ve found classification rather hard, except for when…
Overall Classification Workflow
EXPLORATION EXPERIMENTATION
WITH DIFFERENT METHODS
REFINING PROMISING
METHODS
The Model Training Workflow
FEATURE
EXTRACTION
MODEL
TRAINING USE MODEL
DATA FEATURE
VECTORS
MODEL OUTPUT
Data volumes in training
DATAVOLUME
DATA
I have a lot of data
Data volumes in training
DATAVOLUME
DATA
FEATURE
VECTORS
feature extraction
Is this result “big data”?
Examples:
- 10TB of network traffic distilled into 9K IP address FVs
- 10TB of medical records distilled into 50M patient FVs
- 10TB of documents distilled into 5TB of document FVs
Data volumes in training
DATAVOLUME
DATA
FEATURE
VECTORS
feature extraction Model
Training
MODEL
The model itself is usually pretty tiny
Data volumes in training
DATAVOLUME
DATA
FEATURE
VECTORS
feature extraction Model
Training
MODEL
Applying that model to all the
data is a big data problem!
Some hurdles
• Where do I run non-hadoop code?
• How do I host out results to the application?
• How do I use my model on streaming data?
• Automate performance measurement?
So what’s the point?
• Not all stages of the model training workflow
are Hadoop problems
• Use the right tool for the job in each phase
e.g., non-parallel model training in some cases
FEATURE
EXTRACTION
MODEL
TRAINING USE MODEL
DATA FEATURE
VECTORS
MODEL OUTPUT
Natural Language Processing
• A lot of classic tools in NLP are “embarrassingly
parallel” over an entire corpus since words split nicely.
– Stemming
– Lexical analysis
– Parsing
– Tokenization
– Normalization
– Removing stop words
– Spell check
Each of these apply to segments of text and
don’t have much to do with any other piece of
Text in the corpus.
Python, NLTK, and Pig
• Pig is a higher-level abstract over MapReduce
• NLTK is a popular natural language toolkit for Python
• Pig allows you to stream data through arbitrary
processes (including python scripts)
• You can use UDFs to wrap NLTK methods, but the need
to use Jython sucks
• Use Pig to move your data around, use a real package
to do the work on the records
postdata = STREAM data THROUGH `my_nltk_script.py`;
(I do the same thing with Scipy and Numpy)
OpenNLP and MapReduce
• OpenNLP is an Apache project is an NLP library
• “It supports the most common NLP tasks, such as
tokenization, sentence segmentation, part-of-
speech tagging, named entity extraction,
chunking, parsing, and coreference resolution.”
• Written in Java with reasonable APIs
• MapReduce is just Java, so you can link into just
about anything you want
• Use OpenNLP in the Mapper to enrich, normalize,
cleanse your data
So what’s the point?
• Hadoop can be used to glue together already
existing libraries
– You just have to figure out how to split the
problem up yourself
Recommender Systems
• Hadoop is good at recommender systems
– Recommender systems like a lot of data
– Systems want to make a lot of recommendations
• A number of methods available in Mahout
Collaborative Filtering:
Base recommendations on others
• Collaborative Filtering
is cool because it
doesn’t have to
understand the user or
the item… just the
relationships
• Relationships are easy
to extract, features and
labels not so much
What’s the point?
• Recommender systems parallelize and there is
a Hadoop library for it
• They use relationships, not features, so the
input data is easier to extract
• If you can fit your problem into the
recommendation framework, you can do
something interesting
Other stuff: Graphs
• Graphs are useful and a lot can be done with
Hadoop
• Check out Giraph
• Check out how Accumulo has been used to
store graphs (google: “Graph 500 Accumulo”)
• Stuff to do:
– Subgraph extraction
– Missing edge recommendation
– Cool visualizations
– Summarizing relationships
Other stuff: Clustering
• Provides interesting insight into group
• Some methods parallelize well
• Mahout has:
– Dirichlet process
– K-means
– Fuzzy K-means
Other stuff: R and Hadoop
• RHIPE and Rhadoop allow you to write
MapReduce jobs in R, instead of Java
• Can also use Hadoop streaming to use R
• This doesn’t magically parallelize all your R
code
• Useful to integrate into R more seamlessly
Wrap up
Hadoop can’t do everything and
you have to do the rest
THANKS!
dminer@clearedgeit.com
@donaldpminer

Contenu connexe

Tendances

Apache Pig: Making data transformation easy
Apache Pig: Making data transformation easyApache Pig: Making data transformation easy
Apache Pig: Making data transformation easyVictor Sanchez Anguix
 
Finding the needles in the haystack. An Overview of Analyzing Big Data with H...
Finding the needles in the haystack. An Overview of Analyzing Big Data with H...Finding the needles in the haystack. An Overview of Analyzing Big Data with H...
Finding the needles in the haystack. An Overview of Analyzing Big Data with H...Chris Baglieri
 
Geek camp
Geek campGeek camp
Geek campjdhok
 
Data Science with Spark - Training at SparkSummit (East)
Data Science with Spark - Training at SparkSummit (East)Data Science with Spark - Training at SparkSummit (East)
Data Science with Spark - Training at SparkSummit (East)Krishna Sankar
 
Pig Tutorial | Twitter Case Study | Apache Pig Script and Commands | Edureka
Pig Tutorial | Twitter Case Study | Apache Pig Script and Commands | EdurekaPig Tutorial | Twitter Case Study | Apache Pig Script and Commands | Edureka
Pig Tutorial | Twitter Case Study | Apache Pig Script and Commands | EdurekaEdureka!
 
Big Data Hadoop Training
Big Data Hadoop TrainingBig Data Hadoop Training
Big Data Hadoop Trainingstratapps
 
High Performance Machine Learning in R with H2O
High Performance Machine Learning in R with H2OHigh Performance Machine Learning in R with H2O
High Performance Machine Learning in R with H2OSri Ambati
 
Faster Faster Faster! Datamarts with Hive at Yahoo
Faster Faster Faster! Datamarts with Hive at YahooFaster Faster Faster! Datamarts with Hive at Yahoo
Faster Faster Faster! Datamarts with Hive at YahooMithun Radhakrishnan
 
Using the search engine as recommendation engine
Using the search engine as recommendation engineUsing the search engine as recommendation engine
Using the search engine as recommendation engineLars Marius Garshol
 
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlyData Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlySarah Guido
 
MapReduce Design Patterns
MapReduce Design PatternsMapReduce Design Patterns
MapReduce Design PatternsDonald Miner
 
Seattle Scalability Mahout
Seattle Scalability MahoutSeattle Scalability Mahout
Seattle Scalability MahoutJake Mannix
 
Scaling PyData Up and Out
Scaling PyData Up and OutScaling PyData Up and Out
Scaling PyData Up and OutTravis Oliphant
 
Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...
Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...
Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...Lucidworks
 
Big Data Science with H2O in R
Big Data Science with H2O in RBig Data Science with H2O in R
Big Data Science with H2O in RAnqi Fu
 
Pig programming is more fun: New features in Pig
Pig programming is more fun: New features in PigPig programming is more fun: New features in Pig
Pig programming is more fun: New features in Pigdaijy
 
Fast and Scalable Python
Fast and Scalable PythonFast and Scalable Python
Fast and Scalable PythonTravis Oliphant
 

Tendances (19)

Apache Pig: Making data transformation easy
Apache Pig: Making data transformation easyApache Pig: Making data transformation easy
Apache Pig: Making data transformation easy
 
Finding the needles in the haystack. An Overview of Analyzing Big Data with H...
Finding the needles in the haystack. An Overview of Analyzing Big Data with H...Finding the needles in the haystack. An Overview of Analyzing Big Data with H...
Finding the needles in the haystack. An Overview of Analyzing Big Data with H...
 
Geek camp
Geek campGeek camp
Geek camp
 
Data Science with Spark - Training at SparkSummit (East)
Data Science with Spark - Training at SparkSummit (East)Data Science with Spark - Training at SparkSummit (East)
Data Science with Spark - Training at SparkSummit (East)
 
Pig Tutorial | Twitter Case Study | Apache Pig Script and Commands | Edureka
Pig Tutorial | Twitter Case Study | Apache Pig Script and Commands | EdurekaPig Tutorial | Twitter Case Study | Apache Pig Script and Commands | Edureka
Pig Tutorial | Twitter Case Study | Apache Pig Script and Commands | Edureka
 
Big Data Hadoop Training
Big Data Hadoop TrainingBig Data Hadoop Training
Big Data Hadoop Training
 
High Performance Machine Learning in R with H2O
High Performance Machine Learning in R with H2OHigh Performance Machine Learning in R with H2O
High Performance Machine Learning in R with H2O
 
Faster Faster Faster! Datamarts with Hive at Yahoo
Faster Faster Faster! Datamarts with Hive at YahooFaster Faster Faster! Datamarts with Hive at Yahoo
Faster Faster Faster! Datamarts with Hive at Yahoo
 
Using the search engine as recommendation engine
Using the search engine as recommendation engineUsing the search engine as recommendation engine
Using the search engine as recommendation engine
 
Data Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at BitlyData Science at Scale: Using Apache Spark for Data Science at Bitly
Data Science at Scale: Using Apache Spark for Data Science at Bitly
 
Hadoop pig
Hadoop pigHadoop pig
Hadoop pig
 
MapReduce Design Patterns
MapReduce Design PatternsMapReduce Design Patterns
MapReduce Design Patterns
 
Seattle Scalability Mahout
Seattle Scalability MahoutSeattle Scalability Mahout
Seattle Scalability Mahout
 
Scaling PyData Up and Out
Scaling PyData Up and OutScaling PyData Up and Out
Scaling PyData Up and Out
 
R, Hadoop and Amazon Web Services
R, Hadoop and Amazon Web ServicesR, Hadoop and Amazon Web Services
R, Hadoop and Amazon Web Services
 
Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...
Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...
Where Search Meets Machine Learning: Presented by Diana Hu & Joaquin Delgado,...
 
Big Data Science with H2O in R
Big Data Science with H2O in RBig Data Science with H2O in R
Big Data Science with H2O in R
 
Pig programming is more fun: New features in Pig
Pig programming is more fun: New features in PigPig programming is more fun: New features in Pig
Pig programming is more fun: New features in Pig
 
Fast and Scalable Python
Fast and Scalable PythonFast and Scalable Python
Fast and Scalable Python
 

En vedette

Tosta matrimonio anchoas del cantábrico y boquerones olmeda orígenes
Tosta matrimonio  anchoas del cantábrico y boquerones olmeda orígenesTosta matrimonio  anchoas del cantábrico y boquerones olmeda orígenes
Tosta matrimonio anchoas del cantábrico y boquerones olmeda orígenesOlmeda Orígenes
 
OpSmile Art Auction Catalogue
OpSmile Art Auction CatalogueOpSmile Art Auction Catalogue
OpSmile Art Auction Catalogueopsmile
 
lunch and learn presentation
lunch and learn presentationlunch and learn presentation
lunch and learn presentationApriele Minott
 
Wildlife: Grand Canyon National Park
Wildlife: Grand Canyon National ParkWildlife: Grand Canyon National Park
Wildlife: Grand Canyon National ParkAustin Gratham
 
Linked In Pg Customer Contact Offering Base Mkt Ver Nov 091[1]
Linked In Pg Customer Contact Offering Base Mkt Ver Nov 091[1]Linked In Pg Customer Contact Offering Base Mkt Ver Nov 091[1]
Linked In Pg Customer Contact Offering Base Mkt Ver Nov 091[1]dshurts
 
中正行銷課
中正行銷課中正行銷課
中正行銷課bopomo
 
Linked Ocean Data - Exploring connections between marine datasets in a Big Da...
Linked Ocean Data - Exploring connections between marine datasets in a Big Da...Linked Ocean Data - Exploring connections between marine datasets in a Big Da...
Linked Ocean Data - Exploring connections between marine datasets in a Big Da...Adam Leadbetter
 
Abstraction Classes in Software Design
Abstraction Classes in Software DesignAbstraction Classes in Software Design
Abstraction Classes in Software DesignVladimir Tsukur
 
Обучение_детей_младшего_возраста_программированию
Обучение_детей_младшего_возраста_программированиюОбучение_детей_младшего_возраста_программированию
Обучение_детей_младшего_возраста_программированиюIvan Dementiev
 
Patologías del Sistema Nervioso
Patologías del Sistema NerviosoPatologías del Sistema Nervioso
Patologías del Sistema NerviosoHector Martínez
 
Broadband for Business
Broadband for BusinessBroadband for Business
Broadband for Businessmnrem
 
Kabinetschef van Willy Claes was mol voor Amerikanen
Kabinetschef van Willy Claes was mol voor AmerikanenKabinetschef van Willy Claes was mol voor Amerikanen
Kabinetschef van Willy Claes was mol voor AmerikanenThierry Debels
 
Mobile AR, OOH and the Mirror World
Mobile AR, OOH and the Mirror WorldMobile AR, OOH and the Mirror World
Mobile AR, OOH and the Mirror WorldChristopher Grayson
 
Why hotel direct bookers make better lovers - A Triptease + Tnooz slide prese...
Why hotel direct bookers make better lovers - A Triptease + Tnooz slide prese...Why hotel direct bookers make better lovers - A Triptease + Tnooz slide prese...
Why hotel direct bookers make better lovers - A Triptease + Tnooz slide prese...tnooz
 
Research into Film Magazines
Research into Film MagazinesResearch into Film Magazines
Research into Film Magazinesmeaglesham1
 

En vedette (20)

Tosta matrimonio anchoas del cantábrico y boquerones olmeda orígenes
Tosta matrimonio  anchoas del cantábrico y boquerones olmeda orígenesTosta matrimonio  anchoas del cantábrico y boquerones olmeda orígenes
Tosta matrimonio anchoas del cantábrico y boquerones olmeda orígenes
 
OpSmile Art Auction Catalogue
OpSmile Art Auction CatalogueOpSmile Art Auction Catalogue
OpSmile Art Auction Catalogue
 
lunch and learn presentation
lunch and learn presentationlunch and learn presentation
lunch and learn presentation
 
World Pangolin Day
World Pangolin DayWorld Pangolin Day
World Pangolin Day
 
Wildlife: Grand Canyon National Park
Wildlife: Grand Canyon National ParkWildlife: Grand Canyon National Park
Wildlife: Grand Canyon National Park
 
Po co ci content marketing
Po co ci content marketingPo co ci content marketing
Po co ci content marketing
 
Linked In Pg Customer Contact Offering Base Mkt Ver Nov 091[1]
Linked In Pg Customer Contact Offering Base Mkt Ver Nov 091[1]Linked In Pg Customer Contact Offering Base Mkt Ver Nov 091[1]
Linked In Pg Customer Contact Offering Base Mkt Ver Nov 091[1]
 
中正行銷課
中正行銷課中正行銷課
中正行銷課
 
Microsoft surface
Microsoft surfaceMicrosoft surface
Microsoft surface
 
Linked Ocean Data - Exploring connections between marine datasets in a Big Da...
Linked Ocean Data - Exploring connections between marine datasets in a Big Da...Linked Ocean Data - Exploring connections between marine datasets in a Big Da...
Linked Ocean Data - Exploring connections between marine datasets in a Big Da...
 
Abstraction Classes in Software Design
Abstraction Classes in Software DesignAbstraction Classes in Software Design
Abstraction Classes in Software Design
 
Обучение_детей_младшего_возраста_программированию
Обучение_детей_младшего_возраста_программированиюОбучение_детей_младшего_возраста_программированию
Обучение_детей_младшего_возраста_программированию
 
Boletim (2)
Boletim (2)Boletim (2)
Boletim (2)
 
Patologías del Sistema Nervioso
Patologías del Sistema NerviosoPatologías del Sistema Nervioso
Patologías del Sistema Nervioso
 
Broadband for Business
Broadband for BusinessBroadband for Business
Broadband for Business
 
Kabinetschef van Willy Claes was mol voor Amerikanen
Kabinetschef van Willy Claes was mol voor AmerikanenKabinetschef van Willy Claes was mol voor Amerikanen
Kabinetschef van Willy Claes was mol voor Amerikanen
 
Mobile AR, OOH and the Mirror World
Mobile AR, OOH and the Mirror WorldMobile AR, OOH and the Mirror World
Mobile AR, OOH and the Mirror World
 
Why hotel direct bookers make better lovers - A Triptease + Tnooz slide prese...
Why hotel direct bookers make better lovers - A Triptease + Tnooz slide prese...Why hotel direct bookers make better lovers - A Triptease + Tnooz slide prese...
Why hotel direct bookers make better lovers - A Triptease + Tnooz slide prese...
 
Daily Newsletter: 20th July, 2011
Daily Newsletter: 20th July, 2011Daily Newsletter: 20th July, 2011
Daily Newsletter: 20th July, 2011
 
Research into Film Magazines
Research into Film MagazinesResearch into Film Magazines
Research into Film Magazines
 

Similaire à Hadoop for Data Science

Scaling ETL with Hadoop - Avoiding Failure
Scaling ETL with Hadoop - Avoiding FailureScaling ETL with Hadoop - Avoiding Failure
Scaling ETL with Hadoop - Avoiding FailureGwen (Chen) Shapira
 
Big data Intro - Presentation to OCHackerz Meetup Group
Big data Intro - Presentation to OCHackerz Meetup GroupBig data Intro - Presentation to OCHackerz Meetup Group
Big data Intro - Presentation to OCHackerz Meetup GroupSri Kanajan
 
Hadoop World 2011: Data Mining in Hadoop, Making Sense of it in Mahout! - Mic...
Hadoop World 2011: Data Mining in Hadoop, Making Sense of it in Mahout! - Mic...Hadoop World 2011: Data Mining in Hadoop, Making Sense of it in Mahout! - Mic...
Hadoop World 2011: Data Mining in Hadoop, Making Sense of it in Mahout! - Mic...Cloudera, Inc.
 
BW Tech Meetup: Hadoop and The rise of Big Data
BW Tech Meetup: Hadoop and The rise of Big Data BW Tech Meetup: Hadoop and The rise of Big Data
BW Tech Meetup: Hadoop and The rise of Big Data Mindgrub Technologies
 
Hadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerHadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerIke Ellis
 
"R, Hadoop, and Amazon Web Services (20 December 2011)"
"R, Hadoop, and Amazon Web Services (20 December 2011)""R, Hadoop, and Amazon Web Services (20 December 2011)"
"R, Hadoop, and Amazon Web Services (20 December 2011)"Portland R User Group
 
Data Day Seattle 2015: Sarah Guido
Data Day Seattle 2015: Sarah GuidoData Day Seattle 2015: Sarah Guido
Data Day Seattle 2015: Sarah GuidoBitly
 
4. hadoop גיא לבנברג
4. hadoop  גיא לבנברג4. hadoop  גיא לבנברג
4. hadoop גיא לבנברגTaldor Group
 
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2tcloudcomputing-tw
 
Hadoop and Mapreduce for .NET User Group
Hadoop and Mapreduce for .NET User GroupHadoop and Mapreduce for .NET User Group
Hadoop and Mapreduce for .NET User GroupCsaba Toth
 
Dataiku hadoop summit - semi-supervised learning with hadoop for understand...
Dataiku   hadoop summit - semi-supervised learning with hadoop for understand...Dataiku   hadoop summit - semi-supervised learning with hadoop for understand...
Dataiku hadoop summit - semi-supervised learning with hadoop for understand...Dataiku
 
Agile analytics applications on hadoop
Agile analytics applications on hadoopAgile analytics applications on hadoop
Agile analytics applications on hadoopRussell Jurney
 
Machine Learning with Spark
Machine Learning with SparkMachine Learning with Spark
Machine Learning with Sparkelephantscale
 
Real time hadoop + mapreduce intro
Real time hadoop + mapreduce introReal time hadoop + mapreduce intro
Real time hadoop + mapreduce introGeoff Hendrey
 
2013 year of real-time hadoop
2013 year of real-time hadoop2013 year of real-time hadoop
2013 year of real-time hadoopGeoff Hendrey
 
Agile Data Science: Hadoop Analytics Applications
Agile Data Science: Hadoop Analytics ApplicationsAgile Data Science: Hadoop Analytics Applications
Agile Data Science: Hadoop Analytics ApplicationsRussell Jurney
 

Similaire à Hadoop for Data Science (20)

Scaling ETL with Hadoop - Avoiding Failure
Scaling ETL with Hadoop - Avoiding FailureScaling ETL with Hadoop - Avoiding Failure
Scaling ETL with Hadoop - Avoiding Failure
 
Hands On: Introduction to the Hadoop Ecosystem
Hands On: Introduction to the Hadoop EcosystemHands On: Introduction to the Hadoop Ecosystem
Hands On: Introduction to the Hadoop Ecosystem
 
Big data Intro - Presentation to OCHackerz Meetup Group
Big data Intro - Presentation to OCHackerz Meetup GroupBig data Intro - Presentation to OCHackerz Meetup Group
Big data Intro - Presentation to OCHackerz Meetup Group
 
Hadoop World 2011: Data Mining in Hadoop, Making Sense of it in Mahout! - Mic...
Hadoop World 2011: Data Mining in Hadoop, Making Sense of it in Mahout! - Mic...Hadoop World 2011: Data Mining in Hadoop, Making Sense of it in Mahout! - Mic...
Hadoop World 2011: Data Mining in Hadoop, Making Sense of it in Mahout! - Mic...
 
Bw tech hadoop
Bw tech hadoopBw tech hadoop
Bw tech hadoop
 
BW Tech Meetup: Hadoop and The rise of Big Data
BW Tech Meetup: Hadoop and The rise of Big Data BW Tech Meetup: Hadoop and The rise of Big Data
BW Tech Meetup: Hadoop and The rise of Big Data
 
Hadoop for the Absolute Beginner
Hadoop for the Absolute BeginnerHadoop for the Absolute Beginner
Hadoop for the Absolute Beginner
 
"R, Hadoop, and Amazon Web Services (20 December 2011)"
"R, Hadoop, and Amazon Web Services (20 December 2011)""R, Hadoop, and Amazon Web Services (20 December 2011)"
"R, Hadoop, and Amazon Web Services (20 December 2011)"
 
Data Day Seattle 2015: Sarah Guido
Data Day Seattle 2015: Sarah GuidoData Day Seattle 2015: Sarah Guido
Data Day Seattle 2015: Sarah Guido
 
4. hadoop גיא לבנברג
4. hadoop  גיא לבנברג4. hadoop  גיא לבנברג
4. hadoop גיא לבנברג
 
Architecting Your First Big Data Implementation
Architecting Your First Big Data ImplementationArchitecting Your First Big Data Implementation
Architecting Your First Big Data Implementation
 
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2
Tcloud Computing Hadoop Family and Ecosystem Service 2013.Q2
 
Hadoop and Mapreduce for .NET User Group
Hadoop and Mapreduce for .NET User GroupHadoop and Mapreduce for .NET User Group
Hadoop and Mapreduce for .NET User Group
 
Dataiku hadoop summit - semi-supervised learning with hadoop for understand...
Dataiku   hadoop summit - semi-supervised learning with hadoop for understand...Dataiku   hadoop summit - semi-supervised learning with hadoop for understand...
Dataiku hadoop summit - semi-supervised learning with hadoop for understand...
 
Agile analytics applications on hadoop
Agile analytics applications on hadoopAgile analytics applications on hadoop
Agile analytics applications on hadoop
 
Machine Learning with Spark
Machine Learning with SparkMachine Learning with Spark
Machine Learning with Spark
 
Not Just Another Overview of Apache Hadoop
Not Just Another Overview of Apache HadoopNot Just Another Overview of Apache Hadoop
Not Just Another Overview of Apache Hadoop
 
Real time hadoop + mapreduce intro
Real time hadoop + mapreduce introReal time hadoop + mapreduce intro
Real time hadoop + mapreduce intro
 
2013 year of real-time hadoop
2013 year of real-time hadoop2013 year of real-time hadoop
2013 year of real-time hadoop
 
Agile Data Science: Hadoop Analytics Applications
Agile Data Science: Hadoop Analytics ApplicationsAgile Data Science: Hadoop Analytics Applications
Agile Data Science: Hadoop Analytics Applications
 

Plus de Donald Miner

Machine Learning Vital Signs
Machine Learning Vital SignsMachine Learning Vital Signs
Machine Learning Vital SignsDonald Miner
 
Survey of Accumulo Techniques for Indexing Data
Survey of Accumulo Techniques for Indexing DataSurvey of Accumulo Techniques for Indexing Data
Survey of Accumulo Techniques for Indexing DataDonald Miner
 
An Introduction to Accumulo
An Introduction to AccumuloAn Introduction to Accumulo
An Introduction to AccumuloDonald Miner
 
Data, The New Currency
Data, The New CurrencyData, The New Currency
Data, The New CurrencyDonald Miner
 
The Amino Analytical Framework - Leveraging Accumulo to the Fullest
The Amino Analytical Framework - Leveraging Accumulo to the Fullest The Amino Analytical Framework - Leveraging Accumulo to the Fullest
The Amino Analytical Framework - Leveraging Accumulo to the Fullest Donald Miner
 

Plus de Donald Miner (6)

Machine Learning Vital Signs
Machine Learning Vital SignsMachine Learning Vital Signs
Machine Learning Vital Signs
 
Survey of Accumulo Techniques for Indexing Data
Survey of Accumulo Techniques for Indexing DataSurvey of Accumulo Techniques for Indexing Data
Survey of Accumulo Techniques for Indexing Data
 
An Introduction to Accumulo
An Introduction to AccumuloAn Introduction to Accumulo
An Introduction to Accumulo
 
SQL on Accumulo
SQL on AccumuloSQL on Accumulo
SQL on Accumulo
 
Data, The New Currency
Data, The New CurrencyData, The New Currency
Data, The New Currency
 
The Amino Analytical Framework - Leveraging Accumulo to the Fullest
The Amino Analytical Framework - Leveraging Accumulo to the Fullest The Amino Analytical Framework - Leveraging Accumulo to the Fullest
The Amino Analytical Framework - Leveraging Accumulo to the Fullest
 

Dernier

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
The Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementThe Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementNuwan Dias
 
UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5DianaGray10
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...Daniel Zivkovic
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Alexander Turgeon
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Juan Carlos Gonzalez
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 

Dernier (20)

Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
The Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API ManagementThe Kubernetes Gateway API and its role in Cloud Native API Management
The Kubernetes Gateway API and its role in Cloud Native API Management
 
UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5UiPath Studio Web workshop series - Day 5
UiPath Studio Web workshop series - Day 5
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 

Hadoop for Data Science

  • 1. Hadoop for Data Science Donald Miner Data Science MD October 9, 2013
  • 3. I’ll talk about… Intro to Hadoop (HDFS and MapReduce) Some reasons why I think Hadoop is cool (is this cliché yet?) Step 1: Hadoop Step 2: ???? Step 3: Data Science! Some examples of data science work on Hadoop What can Hadoop do to enable data science work?
  • 4. Hadoop Distributed File System HDFS • Stores files in folders (that’s it) – Nobody cares what’s in your files • Chunks large files into blocks (~64MB-2GB) • 3 replicates of each block (better safe than sorry) • Blocks are scattered all over the place FILE BLOCKS
  • 5. MapReduce • Analyzes raw data in HDFS where the data is • Jobs are split into Mappers and Reducers Reducers (you code this, too) Automatically Groups by the mapper’s output key Aggregate, count, statistics Outputs to HDFS Mappers (you code this) Loads data from HDFS Filter, transform, parse Outputs (key, value) pairs
  • 6. Hadoop Ecosystem • Higher-level languages like Pig and Hive • HDFS Data systems like HBase and Accumulo • Close friends like ZooKeeper, Flume, Storm, Cassandra, Avro
  • 7. Mahout • Mahout is a Machine Library • Has both MapReduce and non-parallel implementations of a number of algorithms: – Recommenders – Clustering – Classification
  • 8. Cool Thing #1: Linear Scalability • HDFS and MapReduce scale linearly • If you have twice as many computers, jobs run twice as fast • If you have twice as much data, jobs run twice as slow • If you have twice as many computers, you can store twice as much data DATA LOCALITY!!
  • 9. Cool Thing #2: Schema on Read LOAD DATA FIRST, ASK QUESTIONS LATER Data is parsed/interpreted as it is loaded out of HDFS What implications does this have? BEFORE: ETL, schema design upfront, tossing out original data, comprehensive data study Keep original data around! Have multiple views of the same data! Work with unstructured data sooner! Store first, figure out what to do with it later! WITH HADOOP:
  • 10. Cool Thing #3: Transparent Parallelism Network programming? Inter-process communication? Threading? Distributed stuff? With MapReduce, I DON’T CARE Your solution … I just have to be sure my solution fits into this tiny box Fault tolerance? Code deployment? RPC? Message passing? Locking? MapReduce Framework Data storage? Scalability? Data center fires?
  • 11. Cool Thing #4: Unstructured Data • Unstructured data: media, text, forms, log data lumped structured data • Query languages like SQL and Pig assume some sort of “structure” • MapReduce is just Java: You can do anything Java can do in a Mapper or Reducer
  • 12. The rest of the talk • Four threads: – Data exploration – Classification – NLP – Recommender systems I’m using these to illustrate some points
  • 13. Exploration • Hadoop is great at exploring data! • I like to explore data in a couple ways: – Filtering – Sampling – Summarization – Evaluate cleanliness • I like to spend 50% of my time doing exploration (but unfortunately it’s the first thing to get cut)
  • 14. Filtering • Filtering is like a microscope: I want to take a closer look at a subset • In MapReduce, you do this in the mapper • Identify nasty records you want to get rid of • Examples: – Only Baltimore data – Remove gibberish – Only 5 minutes
  • 15. Sampling • Hadoop isn’t the king of interactive analysis • Sampling is a good way to grab a set of data then work with it locally (Excel?) • Pig has a handy SAMPLE keyword • Types of sampling: – Sample randomly across the entire data set – Sub-graph extraction – Filters (from the last slide)
  • 16. Summarization • Summarization is a bird’s-eye view • MapReduce is good at summarization: – Mappers extract the group-by keys – Reducers do the aggregation • I like to: – Count number, get stdev, get average, get min/max of records in several groups – Count nulls in columns (if applicable) – Grab top-10 lists
  • 17. Evaluating Cleanliness • I’ve never been burned twice • Things to check for: – Fields that shouldn’t be null that are – Duplicates (does unique records=records?) – Dates (look for 1970; look at formats; time zones) – Things that should be normalized – Keys that are different because of trash e.g. “ abc “ != “abc” SQL/RDBMS =
  • 18. What’s the point? • Hadoop is really good at this stuff! • You probably have a lot of data and a lot of it is garbage! • Take the time to do this and your further work will be much easier • It’s hard to tell what methods you should use until you explore your data
  • 19. Classification • Classification is taking feature vectors (derived from your data), and then guessing some sort of label – E.g., sunny, Saturday, summer -> play tennis rainy, Wednesday, winter -> don’t play tennis • Most classification algorithms aren’t easily parallelizable or have good implementations • You need a training set of true feature vectors and labels… how often is your data labeled? • I’ve found classification rather hard, except for when…
  • 20. Overall Classification Workflow EXPLORATION EXPERIMENTATION WITH DIFFERENT METHODS REFINING PROMISING METHODS The Model Training Workflow FEATURE EXTRACTION MODEL TRAINING USE MODEL DATA FEATURE VECTORS MODEL OUTPUT
  • 21. Data volumes in training DATAVOLUME DATA I have a lot of data
  • 22. Data volumes in training DATAVOLUME DATA FEATURE VECTORS feature extraction Is this result “big data”? Examples: - 10TB of network traffic distilled into 9K IP address FVs - 10TB of medical records distilled into 50M patient FVs - 10TB of documents distilled into 5TB of document FVs
  • 23. Data volumes in training DATAVOLUME DATA FEATURE VECTORS feature extraction Model Training MODEL The model itself is usually pretty tiny
  • 24. Data volumes in training DATAVOLUME DATA FEATURE VECTORS feature extraction Model Training MODEL Applying that model to all the data is a big data problem!
  • 25. Some hurdles • Where do I run non-hadoop code? • How do I host out results to the application? • How do I use my model on streaming data? • Automate performance measurement?
  • 26. So what’s the point? • Not all stages of the model training workflow are Hadoop problems • Use the right tool for the job in each phase e.g., non-parallel model training in some cases FEATURE EXTRACTION MODEL TRAINING USE MODEL DATA FEATURE VECTORS MODEL OUTPUT
  • 27. Natural Language Processing • A lot of classic tools in NLP are “embarrassingly parallel” over an entire corpus since words split nicely. – Stemming – Lexical analysis – Parsing – Tokenization – Normalization – Removing stop words – Spell check Each of these apply to segments of text and don’t have much to do with any other piece of Text in the corpus.
  • 28. Python, NLTK, and Pig • Pig is a higher-level abstract over MapReduce • NLTK is a popular natural language toolkit for Python • Pig allows you to stream data through arbitrary processes (including python scripts) • You can use UDFs to wrap NLTK methods, but the need to use Jython sucks • Use Pig to move your data around, use a real package to do the work on the records postdata = STREAM data THROUGH `my_nltk_script.py`; (I do the same thing with Scipy and Numpy)
  • 29. OpenNLP and MapReduce • OpenNLP is an Apache project is an NLP library • “It supports the most common NLP tasks, such as tokenization, sentence segmentation, part-of- speech tagging, named entity extraction, chunking, parsing, and coreference resolution.” • Written in Java with reasonable APIs • MapReduce is just Java, so you can link into just about anything you want • Use OpenNLP in the Mapper to enrich, normalize, cleanse your data
  • 30. So what’s the point? • Hadoop can be used to glue together already existing libraries – You just have to figure out how to split the problem up yourself
  • 31. Recommender Systems • Hadoop is good at recommender systems – Recommender systems like a lot of data – Systems want to make a lot of recommendations • A number of methods available in Mahout
  • 32. Collaborative Filtering: Base recommendations on others • Collaborative Filtering is cool because it doesn’t have to understand the user or the item… just the relationships • Relationships are easy to extract, features and labels not so much
  • 33. What’s the point? • Recommender systems parallelize and there is a Hadoop library for it • They use relationships, not features, so the input data is easier to extract • If you can fit your problem into the recommendation framework, you can do something interesting
  • 34. Other stuff: Graphs • Graphs are useful and a lot can be done with Hadoop • Check out Giraph • Check out how Accumulo has been used to store graphs (google: “Graph 500 Accumulo”) • Stuff to do: – Subgraph extraction – Missing edge recommendation – Cool visualizations – Summarizing relationships
  • 35. Other stuff: Clustering • Provides interesting insight into group • Some methods parallelize well • Mahout has: – Dirichlet process – K-means – Fuzzy K-means
  • 36. Other stuff: R and Hadoop • RHIPE and Rhadoop allow you to write MapReduce jobs in R, instead of Java • Can also use Hadoop streaming to use R • This doesn’t magically parallelize all your R code • Useful to integrate into R more seamlessly
  • 37. Wrap up Hadoop can’t do everything and you have to do the rest

Notes de l'éditeur

  1. Donald's talk will cover how to use native MapReduce in conjunction with Pig, including a detailed discussion of when users might be best served to use one or the other.