SlideShare une entreprise Scribd logo
1  sur  16
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
PySpark RDD
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
Today’s Training Topics
❖ Need for RDD
❖ What are RDD?
❖ PySpark RDD Operations
❖ Features of RDD
❖ PySpark RDD Operations - Demo
❖ Finding Page Rank – PySpark RDD Demo
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
Why RDD?
Stable Storage
(HDFS)
Stable Storage
(HDFS)
Stable Storage
(HDFS)
JOB #1 JOB #2
Iterative Process
Reusing Data
Sharing Data
S L O W
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
What are RDD?
R
D
D
esilient
istributed
ataset
Backbone of Apache Spark
One of the First Fundamental Data Structures
Transformations
Actions
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
Transformations and Actions
Transformations
map
flatMap
filter
distinct
reduceByKey
mapPartitions
Actions
collect
collectAsMap
reduce
countByKey
take
countByValue
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
Features of RDDs
In-Memory
Computation
Lazy
Evaluations
Fault
Tolerance
Immutability
Partitioning
Persistence
Coarse
Grained
Operations
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
RDD Creation & Operations
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
Transformations
map
flatMap
filter
distinct
reduceByKey
mapPartitions
Actions
collect
reduce
groupByKey
take
countByValue
groupBy
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
Finding Page Rank
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
How it works ?
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
How it works ?
Page Rank of Site
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
How it works ?
Page Rank of Site
Page Rank of Inbound Link
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
How it works ?
Page Rank of Site
Page Rank of Inbound Link
Number of Links
On that Page
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
How it works ?
Netflix
Amazon
Wikipedia
Google
Iter - 0 Iter - 1 Iter - 2 Rank
1/4
1/4
1/4
1/4
1/12
2.5/12
4.5/12
4/12
1.5/12
2/12
4.5/12
4/12
1
2
3
4
PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training
Problem Statement
PySpark RDD Tutorial | PySpark Tutorial for Beginners | PySpark Online Training | Edureka

Contenu connexe

Plus de Edureka!

Plus de Edureka! (20)

How to Deploy Java Web App in AWS| Edureka
How to Deploy Java Web App in AWS| EdurekaHow to Deploy Java Web App in AWS| Edureka
How to Deploy Java Web App in AWS| Edureka
 
Importance of Digital Marketing | Edureka
Importance of Digital Marketing | EdurekaImportance of Digital Marketing | Edureka
Importance of Digital Marketing | Edureka
 
RPA in 2020 | Edureka
RPA in 2020 | EdurekaRPA in 2020 | Edureka
RPA in 2020 | Edureka
 
Email Notifications in Jenkins | Edureka
Email Notifications in Jenkins | EdurekaEmail Notifications in Jenkins | Edureka
Email Notifications in Jenkins | Edureka
 
EA Algorithm in Machine Learning | Edureka
EA Algorithm in Machine Learning | EdurekaEA Algorithm in Machine Learning | Edureka
EA Algorithm in Machine Learning | Edureka
 
Cognitive AI Tutorial | Edureka
Cognitive AI Tutorial | EdurekaCognitive AI Tutorial | Edureka
Cognitive AI Tutorial | Edureka
 
AWS Cloud Practitioner Tutorial | Edureka
AWS Cloud Practitioner Tutorial | EdurekaAWS Cloud Practitioner Tutorial | Edureka
AWS Cloud Practitioner Tutorial | Edureka
 
Blue Prism Top Interview Questions | Edureka
Blue Prism Top Interview Questions | EdurekaBlue Prism Top Interview Questions | Edureka
Blue Prism Top Interview Questions | Edureka
 
Big Data on AWS Tutorial | Edureka
Big Data on AWS Tutorial | Edureka Big Data on AWS Tutorial | Edureka
Big Data on AWS Tutorial | Edureka
 
A star algorithm | A* Algorithm in Artificial Intelligence | Edureka
A star algorithm | A* Algorithm in Artificial Intelligence | EdurekaA star algorithm | A* Algorithm in Artificial Intelligence | Edureka
A star algorithm | A* Algorithm in Artificial Intelligence | Edureka
 
Kubernetes Installation on Ubuntu | Edureka
Kubernetes Installation on Ubuntu | EdurekaKubernetes Installation on Ubuntu | Edureka
Kubernetes Installation on Ubuntu | Edureka
 
Introduction to DevOps | Edureka
Introduction to DevOps | EdurekaIntroduction to DevOps | Edureka
Introduction to DevOps | Edureka
 
ITIL® Tutorial for Beginners | ITIL® Foundation Training | Edureka
ITIL® Tutorial for Beginners | ITIL® Foundation Training | EdurekaITIL® Tutorial for Beginners | ITIL® Foundation Training | Edureka
ITIL® Tutorial for Beginners | ITIL® Foundation Training | Edureka
 
Difference between ITIL v3 and ITIL 4 | ITIL® Foundation Training | Edureka
Difference between ITIL v3 and ITIL 4 | ITIL® Foundation Training | EdurekaDifference between ITIL v3 and ITIL 4 | ITIL® Foundation Training | Edureka
Difference between ITIL v3 and ITIL 4 | ITIL® Foundation Training | Edureka
 
Jenkins vs Bamboo | Differences Between Jenkins and Bamboo | Edureka
Jenkins vs Bamboo | Differences Between Jenkins and Bamboo | EdurekaJenkins vs Bamboo | Differences Between Jenkins and Bamboo | Edureka
Jenkins vs Bamboo | Differences Between Jenkins and Bamboo | Edureka
 
What Is Digital Marketing? | Digital Marketing Tutorial | Edureka
What Is Digital Marketing? | Digital Marketing Tutorial | EdurekaWhat Is Digital Marketing? | Digital Marketing Tutorial | Edureka
What Is Digital Marketing? | Digital Marketing Tutorial | Edureka
 
What is JUnit? | Edureka
What is JUnit? | EdurekaWhat is JUnit? | Edureka
What is JUnit? | Edureka
 
Machine Learning in 10 Minutes | What is Machine Learning? | Edureka
Machine Learning in 10 Minutes | What is Machine Learning? | EdurekaMachine Learning in 10 Minutes | What is Machine Learning? | Edureka
Machine Learning in 10 Minutes | What is Machine Learning? | Edureka
 
Web Development Projects | Web Dev Project Ideas For Beginners | Edureka
Web Development Projects | Web Dev Project Ideas For Beginners | EdurekaWeb Development Projects | Web Dev Project Ideas For Beginners | Edureka
Web Development Projects | Web Dev Project Ideas For Beginners | Edureka
 
What's new in python 3.8? | Python 3.8 New Features | Edureka
What's new in python 3.8? | Python 3.8 New Features | EdurekaWhat's new in python 3.8? | Python 3.8 New Features | Edureka
What's new in python 3.8? | Python 3.8 New Features | Edureka
 

Dernier

Dernier (20)

Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 

PySpark RDD Tutorial | PySpark Tutorial for Beginners | PySpark Online Training | Edureka

  • 1. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training PySpark RDD
  • 2. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training Today’s Training Topics ❖ Need for RDD ❖ What are RDD? ❖ PySpark RDD Operations ❖ Features of RDD ❖ PySpark RDD Operations - Demo ❖ Finding Page Rank – PySpark RDD Demo
  • 3. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training Why RDD? Stable Storage (HDFS) Stable Storage (HDFS) Stable Storage (HDFS) JOB #1 JOB #2 Iterative Process Reusing Data Sharing Data S L O W
  • 4. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training What are RDD? R D D esilient istributed ataset Backbone of Apache Spark One of the First Fundamental Data Structures Transformations Actions
  • 5. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training Transformations and Actions Transformations map flatMap filter distinct reduceByKey mapPartitions Actions collect collectAsMap reduce countByKey take countByValue
  • 6. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training Features of RDDs In-Memory Computation Lazy Evaluations Fault Tolerance Immutability Partitioning Persistence Coarse Grained Operations
  • 7. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training RDD Creation & Operations
  • 8. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training Transformations map flatMap filter distinct reduceByKey mapPartitions Actions collect reduce groupByKey take countByValue groupBy
  • 9. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training Finding Page Rank
  • 10. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training How it works ?
  • 11. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training How it works ? Page Rank of Site
  • 12. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training How it works ? Page Rank of Site Page Rank of Inbound Link
  • 13. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training How it works ? Page Rank of Site Page Rank of Inbound Link Number of Links On that Page
  • 14. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training How it works ? Netflix Amazon Wikipedia Google Iter - 0 Iter - 1 Iter - 2 Rank 1/4 1/4 1/4 1/4 1/12 2.5/12 4.5/12 4/12 1.5/12 2/12 4.5/12 4/12 1 2 3 4
  • 15. PYSPARK CERTIFICATION TRAINING www.edureka.co/pyspark-certification-training Problem Statement