2. LIVE On-line Class
Class Recording in LMS
24/7 Post Class Support
Module Wise Quiz
Project Work on Large Data Base
Verifiable Certificate
How it Works?
Slide 2 www.edureka.in/data-science
3. Slide 3 www.edureka.in/data-science
Module 1
» Introduction to Data Science
Module 2
» Basic Data Manipulation using R
Module 3
» Machine Learning Techniques using R Part -1
- Clustering
- TF-IDF and Cosine Similarity
- Association Rule Mining
Module 4
» Machine Learning Techniques using R Part -2
- Supervised and Unsupervised Learning
- Decision Tree Classifier
Course Topics
Module 5
» Machine Learning Techniques using R Part -3
- Random Forest Classifier
- Naïve Bayer’s Classifier
Module 6
» Introduction to Hadoop Architecture
Module 7
» Integrating R with Hadoop
Module 8
» Mahout Introduction and Algorithm
Implementation
Module 9
» Additional Mahout Algorithms and Parallel
Processing in R
Module 10
» Project
4. Topics for the Day
Slide 4 www.edureka.in/data-science
Big Data
Big Data Scenarios
Big Data Challenges
Introduction to Data Science
Data Science: Components
Types of Data Scientists
Data Science: Core Components
Use-Cases
Introduction to Hadoop and R
R and Hadoop Integration
Machine Learning with Mahout
Assignment, Pre-work and Agenda for the Next Class
What’s Within the LMS
References
5. Objectives
At the end of this module, you will be able to
Understand Big Data and its challenges
Implement Big Data in real time scenarios
List and explain the components and prospects of Data Science
Learn the implementation of Hadoop on Big data
Analyze some real world use-cases with the help of R programming Language
Understand machine learning concepts
8. What is Big Data?
Lots of Data
(Terabytes or
Petabytes)
Systems/Enterprises
generate huge amount
of data from Terabytes
to and even Petabytes
of information
Slide 7 www.edureka.in/data-sciencehttp://www.today.mccombs.utexas.edu/2012/04/the-big-data-machine
10. Big Data Scenarios: Sports
Slide 10 www.edureka.in/data-sciencehttp://www.espncricinfo.com/
11. Big Data Scenarios: Sports
Sports teams are using data for tracking ticket
sales and even for tracking team strategies.
Advertising and marketing agencies are tracking
social media to understand responsiveness to
campaigns, promotions, and other advertising
mediums
Slide 11 www.edureka.in/data-sciencehttp://www.espncricinfo.com/
12. Big Data Scenarios : Hospital Care
Slide 11 www.edureka.in/data-sciencehttp://www.majorprojects.vic.gov.au/our-projects/our-past-projects/austin-hospital
13. Big Data Scenarios : Hospital Care
Hospitals are analyzing medical data and patient
records to predict those patients that are likely to seek
readmission within a few months of discharge. The
hospital can then intervene in hopes of preventing
another costly hospital stay.
Medical diagnostics company analyzes millions of lines
of data to develop first non-intrusive test for
predicting coronary artery disease. To do so,
researchers at the company analyzed over 100 million
gene samples to ultimately identify the 23 primary
predictive genes for coronary artery disease
Slide 12 www.edureka.in/data-science
14. Big Data Scenarios : Amazon.com
Slide 14 www.edureka.in/data-sciencehttp://wp.streetwise.co/wp-content/uploads/2012/08/Amazon-Recommendations.png
15. Amazon has an unrivalled bank of data on online consumer
purchasing behaviour that it can mine from its 152
million customer accounts.
Amazon also uses Big Data to monitor, track and secure its 1.5
billion items in its retail store that are laying around it 200
fulfilment centres around the world. Amazon stores the
product catalogue data in S3.
S3 can write, read and delete objects up to 5 TB of data each.
The catalogue stored in S3 receives more than 50 million
updates a week and every 30 minutes all data received is
crunched and reported back to the different warehouses and
the website.
Big Data Scenarios : Amazon.com
Slide 15 www.edureka.in/data-sciencehttp://wp.streetwise.co/wp-content/uploads/2012/08/Amazon-Recommendations.png
16. Big Data Scenarios: NetFlix
Slide 16 www.edureka.in/data-sciencehttp://smhttp.23575.nexcesscdn.net/80ABE1/sbmedia/blog/wp-content/uploads/2013/03/netflix-in-asia.png
17. Netflix uses 1 petabyte to store the videos for streaming.
BitTorrent Sync has transferred over 30 petabytes of data
since its pre-alpha release in January 2013.
The 2009 movie Avatar is reported to have taken over 1
petabyte of local storage at Weta Digital for the rendering
of the 3D CGI effects.
One petabyte of average MP3-encoded songs (for mobile,
roughly one megabyte per minute), would require 2000
years to play.
Big Data Scenarios: NetFlix
Slide 17 www.edureka.in/data-sciencehttp://smhttp.23575.nexcesscdn.net/80ABE1/sbmedia/blog/wp-content/uploads/2013/03/netflix-in-asia.png
18. Big Data Scenarios: The Large Hadron Collider
Slide 17 www.edureka.in/data-sciencehttp://www.crowdsourcing.org/article/-nasa-tries-to-free-creativity-with-big-data-challenge/19984
19. The experiments in the Large Hadron Collider produce
about 15 petabytes of data per year, which are
distributed over the Worldwide LHC Computing Grid.
One petabyte is enough to store the DNA of the
entire population of the USA - with cloning it twice.
Big Data Scenarios: The Large Hadron Collider
Slide 18 www.edureka.in/data-sciencehttp://en.wikipedia.org/wiki/Large_Hadron_Collider
20. IBM’s Definition
IBM’s Definition – Big Data Characteristics
http://www-01.ibm.com/software/data/bigdata/
Web
logs
Images
Videos
Sensor
Data
Audios
VOLUME VELOCITY VARIETY
Slide 20 www.edureka.in/data-science
21. IBM’s Definition
Structured
Unstructured
Semi structured
All the above
Variety
3 Vs of
Big data
Batch
Near Time
Real Time
Streams
Velocity
Terabytes
Records
Transactions
Tables, files
Volume
IBM’s Definition – Big Data Characteristics
http://www-01.ibm.com/software/data/bigdata/
Slide 21 www.edureka.in/data-science
22. What about ‘Veracity’?
Slide 21 www.edureka.in/data-sciencehttp://whatsthebigdata.files.wordpress.com/2013/11/batman-on-big-data.jpg
23. HelHoello Teh!e!re!!
My namiseAinsnAien.nie.
I lovIeloqvueizqzueiszzaensdand
puzpzluezszlaensdaIndamI am hetroemtoakmake
youygouuygsutyhsintkhiannkdaanndsawnsweyr my
quequestions.
Slide 23 www.edureka.in/data-science
Annie’s Introduction
24. Map the following to corresponding type: Structured/ Unstructured/ Semi-
structured.
- XML Files
- Word Docs, PDF files, Text files
- E-Mail body
- Data from Enterprise systems (ERP, CRM etc.)
Slide 24 www.edureka.in/data-science
Annie’s Question
25. XML Files -> Semi-structured data
Word Docs, PDF files, Text files -> Unstructured Data
E-Mail body -> Unstructured Data
Data from Enterprise systems (ERP, CRM etc.) -> Structured Data
Slide 25 www.edureka.in/data-science
Annie’s Answer
26. Big Data: Challenges
Slide 25 www.edureka.in/data-sciencehttp://spinnakr.com/blog/wp-content/uploads/2013/08/Using-Big-Data-.jpg
27. Big Data
Challenges
Data security and
Privacy
High variety of
Information
High veracity of
Data
Data Acquisition
High velocity of
processed Data
Information search
and Analytics
High volume of
Data
Information storage
and Analytics
Slide 26 www.edureka.in/data-science
Big Data: Challenges
29. Data Science
Slide 28 www.edureka.in/data-sciencehttp://escience.washington.edu/blog/uw-berkeley-nyu-collaborate-378m-data-science-initiative
30. Data Science
“More data usually beats better algorithms,”
Such as: Recommending movies or music based on past preferences.
Slide 30 www.edureka.in/data-science
31. No matter how extremely unpleasant your algorithm is, they can often be beaten simply by having
more data (and a less sophisticated algorithm).
Big Data is here
Bad News We are struggling to
store and analyze it.
Good News
Data Science
Slide 31 www.edureka.in/data-science
35. Types of Data Scientists
Based on clustering the ways that data is handled by Data Scientists, the following 4 categories can be created:
Data Businesspeople are the product and profit-focused data scientists. They’re leaders, managers, and
entrepreneurs, but with a technical bent. A common educational path is an engineering degree paired with an
MBA.
Data Creatives are eclectic jacks-of-all-trades, able to work with a broad range of data and tools. They may
think of themselves as artists or hackers, and excel at visualization and open source technologies.
Data Developers are focused on writing software to do analytic, statistical, and machine learning tasks, often
in production environments. They often have computer science degrees, and often work with so-called “big
data”.
Data Researchers apply their scientific training, and the tools and techniques they learned in academia, to
organizational data. They may have PhDs, and their creative applications of mathematical tools yields valuable
insights and products.
Slide 34 www.edureka.in/data-sciencehttp://datacommunitydc.org/blog/2013/06/there-is-more-than-one-kind-of-data-scientist/
36. Relationships - Four Categories and the Five Skill Groups
Slide 35 www.edureka.in/data-sciencehttp://datacommunitydc.org/blog/wp-content/uploads/2012/08/SkillsSelfIDMosaic-edit-500px.png
37. Data Science: Core Components
Data Science
Data Architecture
Tool: Hadoop
Machine Learning
Tool: Mahout
Analytics
Tool: R
Slide 37 www.edureka.in/data-science
41. Understanding the
Machine Learning
algorithm to be
used Implementing Machine
Learning in Hadoop on Big
Data Visualisation of
the analysis
Understanding the
problem statement
and defining the
solution
Exploring ways to
integrate R with
Hadoop
Implementing Machine
Learning algorithm in R on
the smaller dataset
Use-Case Implementation:Process Flow Diagram
Slide 41 www.edureka.in/data-science
42. Domain of the Dataset:
Communications and Media. However, the
application of the algorithm is not limited to only
Communications and Media. The technique is
useful for any domain which requires organizing
documents to improve retrieval and support
browsing.
Problem Statement:
A top media company wants to browse through
the popular news from a collection that appeared
on the Reuters newswire in 1987.
Clustering / Grouping documents based on their
contents will make the analysis easier.
Media Use-Case
The Reuters-21578 data set composition
Slide 42 www.edureka.in/data-science
43. Media Use-Case: K-means Clustering
First we will
understand the
implementation of the
technique in R on a
smaller dataset
Then we will understand how
to achieve document
clustering on Big Data using
Mahout libraries on Hadoop
K-Means Clustering can
be implemented on this
dataset
Communications and
Media Dataset to be
Clustered based on
their contents
R Implementation
Hadoop
Implementation
Machine Learning
Implementation
Content-wise
Clustered/Grouped
documents
Slide 43 www.edureka.in/data-science
44. Domain of the Dataset:
Products and Retail. However, the application of the
algorithm is not limited to only Products and Retail. The
technique can be applied wherever we want to discover
the co-occurrence relationship amongst various
activities.
Problem Statement:
Market Basket Analysis.
A retail outlet wants understand the purchase behavior
of a buyer. This information will enable the retailer to
understand the buyer's needs.
The analysis might tell a retailer that customers often
purchase shampoo and conditioner together, so putting
both items on promotion at the same time would create
a significant increase in profit, while a promotion
involving just one of the items would likely drive sales of
the other.
Market Basket Use-Case
Market Basket Analysis
98% of people
who purchased
items A and B
also purchased
item C
Slide 44 www.edureka.in/data-science
45. Market Basket Use-Case: Association Rule Mining
Product and Retail
Dataset
Understand the
implementation of the
technique on a smaller
dataset
Understand how to
achieve the same on
Big Data using Mahout
libraries on Hadoop
The technique used is
Affinity Analysis or
Association Rule Mining
R Implementation
Hadoop
Implementation
Machine Learning
Implementation
Market Basket
Analysis
Slide 45 www.edureka.in/data-science
46. Slide 45 www.edureka.in/data-science
Domain of the Dataset:
Life Science and Health Care. However, the application
of the algorithm is not limited to only Life Science and
Health Care . The technique can be applied wherever
we want to forecast the occurrence of a event on the
basis of certain conditions.
Problem Statement:
A health care organization wants to forecast the onset
of diabetes mellitus in Indians using certain set of
attributes of patients as input such as:
Plasma glucose concentration
Diastolic blood pressure
Triceps skin fold thickness
etc.
Health Care Use-Case
http://www.thenewstribe.com/2013/11/15/diabetes-is-killing-one-patient-every-six-seconds/
47. Slide 46 www.edureka.in/data-science
Understand the basic
implementation of the technique
on a smaller dataset using R
Achieve parallel processing on
the same algorithm using a
parallel processing library
provided by Revolution R.
Understand how to
achieve the same on Big
Data using Mahout
libraries on Hadoop
The technique used
is Affinity Analysis or
Association Rule
Mining.
R Implementation
Hadoop
Implementation
Machine Learning
Implementation
Forecast the onset of
diabetes mellitus in
Indians
Life Science and
Health Care Dataset
with some attributes
of patients as input.
Health Care Use-Case: Parallel Processing
48. Slide 47 www.edureka.in/data-science
Domain of the Dataset:
Social Media. However, the application of the
algorithm is not limited to only Social Media. The
technique can be applied wherever we want to put
documents into category without going through
the contents of all the documents.
Problem Statement:
A Social Media research firm wants to know the
trends of topics discussed on Twitter. For easy
analysis it wants to classify them in the following
categories:
apparel (clothes, shoes, watches, …)
art (Book, DVD, Music, …)
camera
event (travel, concert, …)
health (beauty, spa, …)
home (kitchen, furniture, garden, …)
tech (computer, laptop, tablet, …)
http://www.mobigyaan.com/images/stories/Miscellaneous/mobigyaan-twitter-chat.jpg
Social Media Use-Case
49. Social Media Use-Case: Naïve Bayes Classifier
Understand the basic
implementation of the
technique on a smaller
dataset using R.
Understand how to
achieve the same on
Big Data using Mahout
libraries on Hadoop.
The technique used is
Naïve Bayes Classifier.
Social Media
dataset
R Implementation
Hadoop
Implementation
Machine Learning
Implementation
Categorical
classification of
the tweets
Slide 49 www.edureka.in/data-science
50. Going forward with the class, we will throw some light on the concepts of
Hadoop, R and Machine Learning respectively.
These topics will be vividly covered in their respective modules during the course.
Data Science: Core Components
Slide 50 www.edureka.in/data-science
52. Apache Hadoop is a framework that allows for the distributed
processing of large data sets across clusters of commodity
computers using a simple programming model.
It is an Open-source Data Management with scale-out
storage & distributed processing.
In 2004, Google published a paper on a process
called MapReduce.
MapReduce framework provides a
and associated implementation to
data.
parallel processing model
process huge amount of
Therefore, an implementation of MapReduce framework was
adopted by an Apache open source project named Hadoop.
Introduction to Hadoop
Slide 52 www.edureka.in/data-science
54. Hadoop Core Components
Data Node
Task
Tracker
Data Node
Task
Tracker
Data Node
Task
Tracker
Data Node
Task
Tracker
MapReduce
Engine
HDFS
Cluster
Job Tracker
Admin Node
Name node
Slide 54 www.edureka.in/data-science
55. Hadoop is a framework that allows for the distributed processing of:
- Small Data Sets
- Large Data Sets
Slide 55 www.edureka.in/data-science
Annie’s Question
56. Large Data Sets. It is also capable to process small data-sets however to
experience the true power of Hadoop one needs to have data in Tb’s because
this where RDBMS takes hours and fails whereas Hadoop does the same in
couple of minutes.
Slide 56 www.edureka.in/data-science
Annie’s Answer
57. For setting-up Hadoop on your system you can follow the “Hadoop Installation Guide” present in the LMS.
Slide 57 www.edureka.in/data-science
60. R : Characteristics
Slide 60 www.edureka.in/data-science
R is open source and free.
R has lots of packages and multiple ways of doing the same thing.
By default stores memory in RAM.
R has the most advanced graphics. You need much better programming skills.
R has GUI to help make learning easier.
Customization needs command line.
R can connect to many database and data types.
61. Comparing R and others
http://r4stats.com/articles/popularity/
Comparing R
Slide 61 www.edureka.in/data-science
63. Annie’s Question
R Provides support in terms of:
1. Dedicated Support and Documentation
2. Email-lists, twitter, etc.
Slide 63 www.edureka.in/data-science
67. Annie’s Question
Most of the functions in R are written in :
- Java
- R
- C
- Fortran
Slide 67 www.edureka.in/data-science
68. Annie’s Answer
Most of the user-visible functions in R are written in R.
It is possible for the user to interface to procedures written in the C, C++, or
FORTRAN languages for efficiency.
Slide 68 www.edureka.in/data-science
69. Introduction to R Programming language
www.r-project.org/about.html
Slide 69 www.edureka.in/data-science
History Open Source Official Website R Journal
Evolution Free R Core
Current State Widely Recognized Creators
70. R and Hadoop Integration
R and Hadoop are a natural match in Big Data Analytics and visualization.
One of the most well-known R packages to support Hadoop functionalities is : RHadoop
Rhadoop was developed by Revolution Analytics.
RHadoop is a collection of three R packages: rmr, rhdfs and rhbase.
rmr package provides Hadoop MapReduce functionality in R, rhdfs provides HDFS
management in R and rhbase provides HBase database management from within R.
file
+
Slide 70 www.edureka.in/data-science
71. For setting-up R on your system you can follow the “R Installation Guide” present in the LMS.
Slide 71 www.edureka.in/data-science
73. Machine Learning: Mahout
Machine Learning is a class of algorithms which is data-driven, i.e. unlike "normal" algorithms it is
the data that "tells" what the "good answer" is.
Example:
An hypothetical non-machine learning
algorithm for face recognition in images
would try to define
what a face is (round skin-like-colored
disk, with dark area where you expect the
eyes etc).
A machine learning algorithm would not
have such coded definition, but will
"learn-by-examples": you'll show several
images of faces and not-faces and a good
algorithm will
eventually learn and be able to predict
whether or not an unseen image is a face.
Slide 72 www.edureka.in/data-sciencehttp://endthelie.com/2012/08/24/fbi-sharing-facial-recognition-software-with-police-departments-across-america/
74. Mahout Overview
Mahout is about scalable
Machine Learning
Mahout has functionality
for many of today’s
common machine
learning tasks
Machine Learning is all
over the web today
MapReduce magic in
action
Slide 74 www.edureka.in/data-science
75. Hadoop and
MapReduce magic in
action
Write intelligent applications using Apache Mahout
LinkedIn Recommendations
https://cwiki.apache.org/confluence/display/MAHOUT/Powered+By+Mahout
Machine Learning: LinkedIn Recommendations
Slide 75 www.edureka.in/data-science
76. Annie’s Question
Mahout Algorithms for clustering, classification and collaborative filtering are
implemented on top of Apache Hadoop using :
- Flume
- MapReduce
- Sqoop
- Hive
Slide 76 www.edureka.in/data-science
77. Annie’s Answer
Mahout Algorithms are implemented on top of Apache Hadoop using the
Map/Reduce paradigm.
Slide 77 www.edureka.in/data-science
78. 1. Install R with the help of “R Installation Steps” guide in the LMS. This is a step wise guide which will help you in
installing and setting up R on your system
Slide 78 www.edureka.in/data-science
Assignment
79. Agenda for Next Class
Slide 79 www.edureka.in/data-science
In the next class you will be able to
Understand what is R
Describe why R is used?
Implement R Programming Concepts
Learn Data Import Techniques
Analyze the Processing of Data
80. Pre-work
Go through the “R Essentials for Data Science” section in the LMS. Watch the recordings present in the
section to gain an understanding of the R environment.
Slide 80 www.edureka.in/data-science
81. What’s Within the LMS
This section will
give some
prerequisites on
R Programming
Language.
This section will give
you an insight of
machine learning
This section will
give you an
insight of data
science course
This section provides
an insight in Hadoop
HDFS and MapReduce
framework to manage
data.
Old Batch
recordings –
Handy for you
Click here to expand
and view all the
elements of this
Module
Slide 81 www.edureka.in/data-science
82. What’s Within the LMS
Recording of
the Class
Presentation
Assignment
Quiz
Slide 82 www.edureka.in/data-science
83. What’s Within the LMS
Further
Reading
Pre-work
Installation
Guide
Slide 83 www.edureka.in/data-science
