WebHDFS at King - May 2014 Hadoop MeetUp

•

1 j'aime•1,185 vues

huguk

The latest developments at King on their work with WebHDFS .

Technologie Sports

2
How to turbo charge
your data transfers with
WebHDFS
Andy Done, Data Platform Lead
andy.done@king.com

Old way
hadoop fs –cat /some/path/* | bulk_load my_table
WebHDFS

WebHDFS way
IMPORT INTO TABLE my_table FROM
FILE ‘http://namenode/webhdfs/v1/some/path/file_1’
FILE ‘http://namenode/webhdfs/v1/some/path/file_2’
…
FILE ‘http://namenode/webhdfs/v1/some/path/file_n’
WebHDFS

WebHDFS benefits
•  Simple
•  Efficient
•  Ubiquitous
•  Parallelisable
•  Bidirectional
•  Fast
WebHDFS

Contenu connexe

Tendances

GlusterFS As an Object StorageKeisuke Takahashi

Windows Server 2016 WebinarMen and Mice

dNFS for DBA'sMarcin Przepiórowski

Apache Traffic Serversupertom

5-WebServers.pptwebhostingguy

are available herewebhostingguy

Cf camp 2019 cfconfig - a new way to manage your cold-fusion engine configOrtus Solutions, Corp

Rubyspec y el largo camino hacia Ruby 1.9David Calavera

[MathWorks] Versioning InfrastructurePerforce

RedisMarc Beaupré-Pham

Apache Traffic Server & LuaKit Chan

WE18_Performance_Up.pptwebhostingguy

HAProxy tech talkicebourg

Curl Tutorial Ankireddy Polu

Hands On Gluster with Jeff DarcyGluster.org

What is new in BIND 9.11?Men and Mice

Web Server Load BalancerMobME Technical

Mini-Training: To cache or not to cacheBetclic Everest Group Tech Team

Clug 2012 March web server optimisationgrooverdan

NGINX: High Performance Load BalancingNGINX, Inc.

Tendances (20)

GlusterFS As an Object Storage

Windows Server 2016 Webinar

dNFS for DBA's

Apache Traffic Server

5-WebServers.ppt

are available here

Cf camp 2019 cfconfig - a new way to manage your cold-fusion engine config

Rubyspec y el largo camino hacia Ruby 1.9

[MathWorks] Versioning Infrastructure

Redis

Apache Traffic Server & Lua

WE18_Performance_Up.ppt

HAProxy tech talk

Curl Tutorial

Hands On Gluster with Jeff Darcy

What is new in BIND 9.11?

Web Server Load Balancer

Mini-Training: To cache or not to cache

Clug 2012 March web server optimisation

NGINX: High Performance Load Balancing

Plus de huguk

Data Wrangling on Hadoop - Olivier De Garrigues, Trifactahuguk

ether.camp - Hackathon & ether.camp introhuguk

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoophuguk

Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...huguk

Extracting maximum value from data while protecting consumer privacy. Jason ...huguk

Intelligence Augmented vs Artificial Intelligence. Alex Flamant, IBM Watsonhuguk

Streaming Dataflow with Apache Flink huguk

Lambda architecture on Spark, Kafka for real-time large scale MLhuguk

Today’s reality Hadoop with Spark- How to select the best Data Science approa...huguk

Jonathon Southam: Venture Capital, Funding & Pitchinghuguk

Signal Media: Real-Time Media & News Monitoringhuguk

Dean Bryen: Scaling The Platform For Your Startuphuguk

Peter Karney: Intro to the Digital catapulthuguk

Cytora: Real-Time Political Risk Analysishuguk

Cubitic: Predictive Analyticshuguk

Bird.i: Earth Observation Data Made Socialhuguk

Aiseedo: Real Time Machine Intelligencehuguk

Secrets of Spark's success - Deenar Toraskar, Think Reactive huguk

TV Marketing and big data: cat and dog or thick as thieves? Krzysztof Osiewal...huguk

Hadoop - Looking to the Future By Arun Murthyhuguk

Plus de huguk (20)

Data Wrangling on Hadoop - Olivier De Garrigues, Trifacta

ether.camp - Hackathon & ether.camp intro

Google Cloud Dataproc - Easier, faster, more cost-effective Spark and Hadoop

Using Big Data techniques to query and store OpenStreetMap data. Stephen Knox...

Extracting maximum value from data while protecting consumer privacy. Jason ...

Intelligence Augmented vs Artificial Intelligence. Alex Flamant, IBM Watson

Streaming Dataflow with Apache Flink

Lambda architecture on Spark, Kafka for real-time large scale ML

Today’s reality Hadoop with Spark- How to select the best Data Science approa...

Jonathon Southam: Venture Capital, Funding & Pitching

Signal Media: Real-Time Media & News Monitoring

Dean Bryen: Scaling The Platform For Your Startup

Peter Karney: Intro to the Digital catapult

Cytora: Real-Time Political Risk Analysis

Cubitic: Predictive Analytics

Bird.i: Earth Observation Data Made Social

Aiseedo: Real Time Machine Intelligence

Secrets of Spark's success - Deenar Toraskar, Think Reactive

TV Marketing and big data: cat and dog or thick as thieves? Krzysztof Osiewal...

Hadoop - Looking to the Future By Arun Murthy

Dernier

Scaling API-first – The story of a global engineering organizationRadu Cotescu

Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik

Axa Assurance Maroc - Insurer Innovation Award 2024The Digital Insurer

Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science

Developing An App To Navigate The Roads of BrazilV3cube

How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes

Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge

Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

Partners Life - Insurer Innovation Award 2024The Digital Insurer

[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745

How to convert PDF to text with Nanonetsnaman860154

Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700

A Domino Admins Adventures (Engage 2024)Gabriella Davis

From Event to Action: Accelerate Your Decision Making with Real-Time AutomationSafe Software

A Call to Action for Generative AI in 2024Results

Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge

Dernier (20)

Scaling API-first – The story of a global engineering organization

Injustice - Developers Among Us (SciFiDevCon 2024)

Axa Assurance Maroc - Insurer Innovation Award 2024

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx

Developing An App To Navigate The Roads of Brazil

How to Troubleshoot Apps for the Modern Connected Worker

Driving Behavioral Change for Information Management through Data-Driven Gree...

Breaking the Kubernetes Kill Chain: Host Path Mount

08448380779 Call Girls In Friends Colony Women Seeking Men

Partners Life - Insurer Innovation Award 2024

[2024]Digital Global Overview Report 2024 Meltwater.pdf

How to convert PDF to text with Nanonets

Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...

A Domino Admins Adventures (Engage 2024)

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

A Call to Action for Generative AI in 2024

Tata AIG General Insurance Company - Insurer Innovation Award 2024

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf