Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010

•

8 j'aime•1,482 vues

Hadoop Summit 2010 - Research Track Efficient Parallel Set-Similarity Joins Using Hadoop Chen Li, University of California, Irvine

Technologie Divertissement et humour

Efficient Parallel Set-Similarity Joins Using Hadoop ,[object Object],Joint work with Michael Carey and Rares Vernica

Motivation: Data Cleaning Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Tom Hanks Toy Story 3 2010 Animation Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime Find movies starring Tom Hanks

Movies starring S..warz…ne…ger? Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Tom Hanks Toy Story 3 2010 Animation Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime

Similarity Search Find movies with a star “ similar to ” Schwarrzenger . Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Samuel Jackson Iron man 2008 Sci-Fi Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime

Record linkage Table R Table S Star Keanu Reeves Samuel Jackson Schwarzenegger … Star Keanu Reeves Samuel L. Jackson Schwarzenegger …

Two-step solution Table R Table S Step 2: Verification Star … Star … Step 1: Similarity Join

[object Object],[object Object],[object Object],[object Object],Focus of this talk

[object Object],[object Object],[object Object],[object Object],Talk Outline

Set-Similarity Join Finding pairs of records with a similarity on their join attributes > t

Why this formulation? ,[object Object],[object Object],“ Samuel L. Jackson”  {Samuel, L., Jackson} “ Samuel Jackson”  {Samuel, Jackson} S c h w a r z e n e g g e r

Set-similarity functions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],Talk Outline

[object Object],[object Object],[object Object],[object Object],[object Object],Why Hadoop?

[object Object],A naïve solution ,[object Object],[object Object],[object Object]

Solving frequency skew: prefix filtering ,[object Object],[object Object],[object Object],prefix r1 r2 Sorted by frequency Chaudhuri, Ganti, Kaushik: A Primitive Operator for Similarity Joins in Data Cleaning. ICDE 2006: 5

Prefix filtering: example ,[object Object],[object Object],[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],Hadoop Solution: Overview

Stage 1: Sort tokens by frequency ,[object Object],[object Object],MapReduce phase 1 MapReduce phase 2

Stage 2: Find “similar” id pairs ,[object Object],[object Object]

Stage 3: id pairs  record pairs (phase 1) ,[object Object]

Stage 3: id pairs  record pairs (phase 2) ,[object Object]

[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Experimental Setting

Running time ,[object Object],[object Object],[object Object]

[object Object],[object Object],[object Object],Additional results

[object Object],[object Object],[object Object],Summary

Thank you ,[object Object],[object Object],[object Object],Acknowledgements: NSF, Google, IBM.

Contenu connexe

Similaire à Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010

Deduplication on large amounts of codesource{d}

Tokenamooool2000

Tg noh jeju_workshopTae-Gil Noh

Class 9: Consistent HashingDavid Evans

R language introductionShashwat Shriparv

Hands on Mahout!OSCON Byrum

Dcn 20170823 yjy재연 윤

(Talk in Powerpoint Format)butest

Lambdas: Myths and MistakesRichardWarburton

Language Technology Enhanced Learningtelss09

PyGotham NY 2017: Natural Language Processing from ScratchNoemi Derzsy

GhostJhih-Ming Chen

R Analytics in the CloudDataMine Lab

Stack_Overflow-Network_GraphYaopeng (Gyoho) Wu

Clusterguest1babda

P3 2018 python_regexesProf. Wim Van Criekinge

Machine Learning Applications in NLP.pptbutest

Schemaless DatabasesDan Gunter

Дмитрий Селиванов, OK.RU. Finding Similar Items in high-dimensional spaces: L...Mail.ru Group

So Far (Schematically) yet So Near (Semantically)Amit Sheth

Similaire à Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010 (20)

Deduplication on large amounts of code

Token

Tg noh jeju_workshop

Class 9: Consistent Hashing

R language introduction

Hands on Mahout!

Dcn 20170823 yjy

(Talk in Powerpoint Format)

Lambdas: Myths and Mistakes

Language Technology Enhanced Learning

PyGotham NY 2017: Natural Language Processing from Scratch

Ghost

R Analytics in the Cloud

Stack_Overflow-Network_Graph

Cluster

P3 2018 python_regexes

Machine Learning Applications in NLP.ppt

Schemaless Databases

Дмитрий Селиванов, OK.RU. Finding Similar Items in high-dimensional spaces: L...

So Far (Schematically) yet So Near (Semantically)

Plus de Yahoo Developer Network

Developing Mobile Apps for Performance - Swapnil Patel, Verizon MediaYahoo Developer Network

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...Yahoo Developer Network

Athenz & SPIFFE, Tatsuya Yano, Yahoo JapanYahoo Developer Network

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...Yahoo Developer Network

CICD at Oath using ScrewdriverYahoo Developer Network

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, OathYahoo Developer Network

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenuYahoo Developer Network

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, AmpoolYahoo Developer Network

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...Yahoo Developer Network

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...Yahoo Developer Network

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, OathYahoo Developer Network

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...Yahoo Developer Network

Moving the Oath Grid to Docker, Eric Badger, OathYahoo Developer Network

Architecting Petabyte Scale AI ApplicationsYahoo Developer Network

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...Yahoo Developer Network

Jun 2017 HUG: YARN Scheduling – A Step BeyondYahoo Developer Network

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies Yahoo Developer Network

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...Yahoo Developer Network

February 2017 HUG: Exactly-once end-to-end processing with Apache ApexYahoo Developer Network

February 2017 HUG: Data Sketches: A required toolkit for Big Data AnalyticsYahoo Developer Network

Plus de Yahoo Developer Network (20)

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

CICD at Oath using Screwdriver

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Moving the Oath Grid to Docker, Eric Badger, Oath

Architecting Petabyte Scale AI Applications

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

Dernier

Anypoint Exchange: It’s Not Just a Repo!Manik S Magar

DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays

Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson

Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed

Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity

From Family Reminiscence to Scholarly Archive .Alan Dix

Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst

Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdfPrecisely

SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero

TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc

Powerpoint exploring the locations used in television show Time Clashcharlottematthew16

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxnull - The Open Security Community

Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University

"ML in Production",Oleksandr BaganFwdays

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos

What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett

Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro

How to write a Business Continuity PlanDatabarracks

CloudStudio User manual (basic edition):comworks

Dernier (20)

Anypoint Exchange: It’s Not Just a Repo!

DevoxxFR 2024 Reproducible Builds with Apache Maven

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...

Are Multi-Cloud and Serverless Good or Bad?

Scanning the Internet for External Cloud Exposures via SSL Certs

Dev Dives: Streamline document processing with UiPath Studio Web

From Family Reminiscence to Scholarly Archive .

Human Factors of XR: Using Human Factors to Design XR Systems

Hyperautomation and AI/ML: A Strategy for Digital Transformation Success.pdf

SIP trunking in Janus @ Kamailio World 2024

TrustArc Webinar - How to Build Consumer Trust Through Data Privacy

Powerpoint exploring the locations used in television show Time Clash

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx

Nell’iperspazio con Rocket: il Framework Web di Rust!

"ML in Production",Oleksandr Bagan

Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)

What's New in Teams Calling, Meetings and Devices March 2024

Unraveling Multimodality with Large Language Models.pdf

How to write a Business Continuity Plan

CloudStudio User manual (basic edition):

Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010

2. Motivation: Data Cleaning Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Tom Hanks Toy Story 3 2010 Animation Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime Find movies starring Tom Hanks

3. Movies starring S..warz…ne…ger? Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Tom Hanks Toy Story 3 2010 Animation Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime

4. Similarity Search Find movies with a star “ similar to ” Schwarrzenger . Star Title Year Genre Keanu Reeves The Matrix 1999 Sci-Fi Samuel Jackson Iron man 2008 Sci-Fi Schwarzenegger The Terminator 1984 Sci-Fi Samuel Jackson The man 2006 Crime

5. Record linkage Table R Table S Star Keanu Reeves Samuel Jackson Schwarzenegger … Star Keanu Reeves Samuel L. Jackson Schwarzenegger …

6. Two-step solution Table R Table S Step 2: Verification Star … Star … Step 1: Similarity Join

9. Set-Similarity Join Finding pairs of records with a similarity on their join attributes > t

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25. Speedup

26.

27.

28.

29.

30.

Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010

Recommandé

Recommandé

Contenu connexe

Similaire à Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010

Similaire à Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010 (20)

Plus de Yahoo Developer Network

Plus de Yahoo Developer Network (20)

Dernier

Dernier (20)

Efficient Parallel Set-Similarity Joins Using Hadoop__HadoopSummit2010