SlideShare une entreprise Scribd logo
1  sur  53
Télécharger pour lire hors ligne
Apache Hadoop for System Administrators
Allen Wittenauer
Twitter: @_a__w_
Email: aw@apache.org
Hadoop Deployed Now?
Planning Hadoop Deployment?
Needed some place
to sit before lunch?
An Extremely Quick &
Incomplete Intro to Hadoop
 Map (“transform”)
– Perl:

@items=(1,2,3,4,5);
sub sqr {return $_**2);
print join(‘,’,map(sqr,@items));
1,4,9,16,25
– Python:

items = [1,2,3,4,5]
def sqr(x) : return x**2
print list(map(sqr,items))
[1, 4, 9, 16, 25]
 Reduce (“compress” or “fold”)
– Perl

use List::Util qw/reduce/;
@items=(1,4,9,16,25);
print reduce {$a>$b ? $a:$b} @items;
25
– Python

from functools import reduce
items = [1,4,9,16,25]
print reduce ((lambda x,y: x if
(x>y) else y), items)
25
NEVER

GIVE

GONNA

YOU UP
Hadoop
(‘common’ or ‘core’)

MapReduce

HDFS
Hadoop
(‘common’ or ‘core’)

MapReduce

S3
Hadoop
(‘common’ or ‘core’)

MapReduce

Gluster
Hadoop
(‘common’ or ‘core’)

HBase

HDFS
NameNode

DataNode

DataNode

DataNode

D

D

D

D

D

D

ext4

ext4

ext4

ext4

ext4

ext4
JobTracker

TaskTracker
M

M

R

R

TaskTracker
M

M

R

R

TaskTracker
M

M

R

R
JobTracker

TaskTracker
M

M

R

R

TaskTracker
M

M

R

R

TaskTracker
M

M

R

R
Name
Node

DN

DN

DN

DN

DN

D D D D
M M R R

D D D D
M M R R

D D D D
M M R R

D D D D
M M R R

D D D D
M M R R

TT

TT

TT

TT

TT

Job
Tracker
Hadoop isn’t designed for
system administrators and/or support staff.
“Hadoop is not a developer problem;
it’s an operations problem.”
-- Hadoop vendor ex-employee
Don’t Make Assumptions
tail’ing the logs won’t
tell you the whole story.
%
Monitor the masters!
 LinkedIn’s Configuration
– 30+ Health Checks per Grid
 Masters, canary report, daily fsck, etc

Nagios

– 10+ Health Checks per DC
 LDAP, Kerberos, etc ...

– Cross-DC Nagios Server Checks

ZK

 Warn: 5% down nodes
 Panic: 30% down
 HDFS: 20% Free Space
 Gateway home dir: 10% free
space
 ...

VD

NN

JT

Compute
Nodes

AZ

GW
 Health Check Script
– “OK” - good status
– “ERROR (message)” - bad status

mapred.healthChecker.script.path

 Consider checking ...
– critical software
– ownership & permissions
– network connection speed
– drive count

– file system space
– RO file systems
– IO errors
– missing memory
 Use the tools most of your user’s code is written in!
 Pig
– testfile:

100
– Code:

A = load 'testfile' using PigStorage(',')
as (i: int);
B = foreach C generate i;
C = distinct B;
dump C;
– Output:

(100)
Reactive
Proactive
Resource Controls
 JobTracker Memory Resource Controls
– Limit jobs stored in JT heap:
mapred.jobtracker.completeuserjobs.maximum
– Limit total # of job tasks: mapred.jobtracker.maxtasks.per.job

 Job Memory Resource Controls
– Scheduler-level: mapred.cluster.*.memory.mb
– TT-level: auto-calculated based upon MR slot counts & scheduler level settings
– MR Job-level: mapred.job.*.memory.mb
– Linux only: /proc memory calculator and task killer
“I set the heap to 1G but my
process ran out of memory?”
Treat HDFS like any other multi-tenant FS
 Quota everything
– Yes, including /tmp
– No “show me all quotas” functionality

dfsadmin -setQuota
dfsadmin -setSpaceQuota

 Be consistent:
– /user/* all get same quota

 Be flexible:
– Make another dir for user’s to store big projects (e.g., /project)

 Be smart:
– Have a policy that content in /tmp gets deleted after X days. Automate this!
– Build reporting that shows files that are replicated less than 3 times
Compute Node Disk
Partitioning as Protective Measure
 root partitioning

20 GB /, ...

200 GB task space

(rest) HDFS

 non-root partinioning

5 GB
swap

200 GB task space

(rest) HDFS
Security!
 Queue Level ACLs

mapred-queue-acls.xml

– users
– groups
– netgroups

 Service Level ACLs
– hosts
– users
– groups
– netgroups

hadoop-policy.xml

– Limitation: Web services are all or nothing! :(
– Be aware: Hadoop uses ephemeral ports all over the place! :(
Kerberos!
Corp IT
Active Directory
@CORP

krbtgt/GRID@CORP

krbtgt/host@GRID
krbtgt/service@GRID

Password

Client Node

Grid Realm
@GRID

krbtgt/user@CORP
krbtgt/GRID@CORP

Hadoop
Services
http://data.linkedin.com/opensource/white-elephant
 Fonzi: http://www.flickr.com/photos/elzey/7224689810
 Captain Obvious by artist Stuart McGhee. http://stuartmcghee.com/
 Ant on flower: http://www.flickr.com/photos/bolonski/6116358907
 Ant Colony: http://www.flickr.com/photos/klearchos/2821230516
 Ant Queen: http://commons.wikimedia.org/wiki/
File:Camponotus_crispulus_queen_ant.jpg
 Canary: http://www.flickr.com/photos/nathan_and_jenny/2454127424
 Mona Lisa: Leonardo Da Vinci
 White Elephant: http://data.linkedin.com/opensource/white-elephant
 Ecce Homo:
– Elías García Martínez (original)
– Cecilia Giménez (restored)
Thanks!
Contact:
Twitter: @_a__w_
Email: aw@apache.org
More info:
Quora: www.quora.com/user/allenwittenauer
SlideShare: www.slideshare.net/allenwittenauer

Contenu connexe

Tendances

"Ops Tools with Perl" 2012/05/12 Hokkaido.pm
"Ops Tools with Perl" 2012/05/12 Hokkaido.pm"Ops Tools with Perl" 2012/05/12 Hokkaido.pm
"Ops Tools with Perl" 2012/05/12 Hokkaido.pm
Ryosuke IWANAGA
 

Tendances (17)

Lua: the world's most infuriating language
Lua: the world's most infuriating languageLua: the world's most infuriating language
Lua: the world's most infuriating language
 
KubeCon EU 2016: Custom Volume Plugins
KubeCon EU 2016: Custom Volume PluginsKubeCon EU 2016: Custom Volume Plugins
KubeCon EU 2016: Custom Volume Plugins
 
Perl on Amazon Elastic MapReduce
Perl on Amazon Elastic MapReducePerl on Amazon Elastic MapReduce
Perl on Amazon Elastic MapReduce
 
HAB Software Woes
HAB Software WoesHAB Software Woes
HAB Software Woes
 
"Ops Tools with Perl" 2012/05/12 Hokkaido.pm
"Ops Tools with Perl" 2012/05/12 Hokkaido.pm"Ops Tools with Perl" 2012/05/12 Hokkaido.pm
"Ops Tools with Perl" 2012/05/12 Hokkaido.pm
 
Hvordan sette opp en OAI-PMH metadata-innhøster
Hvordan sette opp en OAI-PMH metadata-innhøsterHvordan sette opp en OAI-PMH metadata-innhøster
Hvordan sette opp en OAI-PMH metadata-innhøster
 
Apache beam — promyk nadziei data engineera na Toruń JUG 28.03.2018
Apache beam — promyk nadziei data engineera na Toruń JUG 28.03.2018Apache beam — promyk nadziei data engineera na Toruń JUG 28.03.2018
Apache beam — promyk nadziei data engineera na Toruń JUG 28.03.2018
 
My life as a beekeeper
My life as a beekeeperMy life as a beekeeper
My life as a beekeeper
 
Perl Memory Use 201207 (OUTDATED, see 201209 )
Perl Memory Use 201207 (OUTDATED, see 201209 )Perl Memory Use 201207 (OUTDATED, see 201209 )
Perl Memory Use 201207 (OUTDATED, see 201209 )
 
Hadoop spark performance comparison
Hadoop spark performance comparisonHadoop spark performance comparison
Hadoop spark performance comparison
 
Ansible for Beginners
Ansible for BeginnersAnsible for Beginners
Ansible for Beginners
 
Redis as a message queue
Redis as a message queueRedis as a message queue
Redis as a message queue
 
Dtalk shell
Dtalk shellDtalk shell
Dtalk shell
 
agri inventory - nouka data collector / yaoya data convertor
agri inventory - nouka data collector / yaoya data convertoragri inventory - nouka data collector / yaoya data convertor
agri inventory - nouka data collector / yaoya data convertor
 
Linux Command Line
Linux Command LineLinux Command Line
Linux Command Line
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
Building a DSL with GraalVM (VoxxedDays Luxembourg)
Building a DSL with GraalVM (VoxxedDays Luxembourg)Building a DSL with GraalVM (VoxxedDays Luxembourg)
Building a DSL with GraalVM (VoxxedDays Luxembourg)
 

En vedette

En vedette (7)

Let's Talk Operations! (Hadoop Summit 2014)
Let's Talk Operations! (Hadoop Summit 2014)Let's Talk Operations! (Hadoop Summit 2014)
Let's Talk Operations! (Hadoop Summit 2014)
 
Apache Yetus: Intro to Precommit for HBase Contributors
Apache Yetus: Intro to Precommit for HBase ContributorsApache Yetus: Intro to Precommit for HBase Contributors
Apache Yetus: Intro to Precommit for HBase Contributors
 
Apache Yetus: Helping Solve the Last Mile Problem
Apache Yetus: Helping Solve the Last Mile ProblemApache Yetus: Helping Solve the Last Mile Problem
Apache Yetus: Helping Solve the Last Mile Problem
 
Deploying Grid Services Using Apache Hadoop
Deploying Grid Services Using Apache HadoopDeploying Grid Services Using Apache Hadoop
Deploying Grid Services Using Apache Hadoop
 
Hadoop Operations at LinkedIn
Hadoop Operations at LinkedInHadoop Operations at LinkedIn
Hadoop Operations at LinkedIn
 
Hadoop Performance at LinkedIn
Hadoop Performance at LinkedInHadoop Performance at LinkedIn
Hadoop Performance at LinkedIn
 
A Multi Colored YARN
A Multi Colored YARNA Multi Colored YARN
A Multi Colored YARN
 

Similaire à Apache Hadoop for System Administrators

Aws dc elastic-mapreduce
Aws dc elastic-mapreduceAws dc elastic-mapreduce
Aws dc elastic-mapreduce
beaknit
 
Aws dc elastic-mapreduce
Aws dc elastic-mapreduceAws dc elastic-mapreduce
Aws dc elastic-mapreduce
beaknit
 
r,rstats,r language,r packages
r,rstats,r language,r packagesr,rstats,r language,r packages
r,rstats,r language,r packages
Ajay Ohri
 
Hadoop tutorial hand-outs
Hadoop tutorial hand-outsHadoop tutorial hand-outs
Hadoop tutorial hand-outs
pardhavi reddy
 

Similaire à Apache Hadoop for System Administrators (20)

Aws dc elastic-mapreduce
Aws dc elastic-mapreduceAws dc elastic-mapreduce
Aws dc elastic-mapreduce
 
Aws dc elastic-mapreduce
Aws dc elastic-mapreduceAws dc elastic-mapreduce
Aws dc elastic-mapreduce
 
Using R on High Performance Computers
Using R on High Performance ComputersUsing R on High Performance Computers
Using R on High Performance Computers
 
10 things I learned building Nomad packs
10 things I learned building Nomad packs10 things I learned building Nomad packs
10 things I learned building Nomad packs
 
Calling r from sas (msug meeting, feb 17, 2018) revised
Calling r from sas (msug meeting, feb 17, 2018)   revisedCalling r from sas (msug meeting, feb 17, 2018)   revised
Calling r from sas (msug meeting, feb 17, 2018) revised
 
Processing massive amount of data with Map Reduce using Apache Hadoop - Indi...
Processing massive amount of data with Map Reduce using Apache Hadoop  - Indi...Processing massive amount of data with Map Reduce using Apache Hadoop  - Indi...
Processing massive amount of data with Map Reduce using Apache Hadoop - Indi...
 
Hadoop Introduction
Hadoop IntroductionHadoop Introduction
Hadoop Introduction
 
Meethadoop
MeethadoopMeethadoop
Meethadoop
 
r,rstats,r language,r packages
r,rstats,r language,r packagesr,rstats,r language,r packages
r,rstats,r language,r packages
 
Hadoop 101 for bioinformaticians
Hadoop 101 for bioinformaticiansHadoop 101 for bioinformaticians
Hadoop 101 for bioinformaticians
 
Implementation of k means algorithm on Hadoop
Implementation of k means algorithm on HadoopImplementation of k means algorithm on Hadoop
Implementation of k means algorithm on Hadoop
 
k-means algorithm implementation on Hadoop
k-means algorithm implementation on Hadoopk-means algorithm implementation on Hadoop
k-means algorithm implementation on Hadoop
 
L3.fa14.ppt
L3.fa14.pptL3.fa14.ppt
L3.fa14.ppt
 
Spark 4th Meetup Londond - Building a Product with Spark
Spark 4th Meetup Londond - Building a Product with SparkSpark 4th Meetup Londond - Building a Product with Spark
Spark 4th Meetup Londond - Building a Product with Spark
 
Hadoop & HDFS for Beginners
Hadoop & HDFS for BeginnersHadoop & HDFS for Beginners
Hadoop & HDFS for Beginners
 
Introduction to Slurm
Introduction to SlurmIntroduction to Slurm
Introduction to Slurm
 
Hadoop - Lessons Learned
Hadoop - Lessons LearnedHadoop - Lessons Learned
Hadoop - Lessons Learned
 
Learning Puppet basic thing
Learning Puppet basic thing Learning Puppet basic thing
Learning Puppet basic thing
 
Hadoop tutorial hand-outs
Hadoop tutorial hand-outsHadoop tutorial hand-outs
Hadoop tutorial hand-outs
 
Flux and InfluxDB 2.0 by Paul Dix
Flux and InfluxDB 2.0 by Paul DixFlux and InfluxDB 2.0 by Paul Dix
Flux and InfluxDB 2.0 by Paul Dix
 

Dernier

Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
giselly40
 

Dernier (20)

Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
08448380779 Call Girls In Diplomatic Enclave Women Seeking Men
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
CNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of ServiceCNv6 Instructor Chapter 6 Quality of Service
CNv6 Instructor Chapter 6 Quality of Service
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 

Apache Hadoop for System Administrators