2014 International Software Testing Conference in Seoul

Testing Big Data:
Unit Test in Hadoop (Part II)
Jongwook Woo (PhD)
High-Performance Internet Computing Center (HiPIC)
Educational Partner with Cloudera and Grants Awardee of Amazon AWS
Computer Information Systems Department
California State University, Los Angeles

Seoul Software Testing Conference

Contents

• Test in General
• Use Cases: Big Data in Hadoop and Ecosystems
• Unit Test in Hadoop


Test in general

• Quality Assurance
– TDD (Test Driven Development)
• Unit Test
– Test functional units of the S/W

– BDD (Behavior Driven Development)
• Based on TDD
• Test behavior of the S/W

– Integration Test:
• integrated components
– Group of unit tests

• CI (Continuous Integration) Server
– Hudson, Jenkins etc

CI Server

• Continuous Integration Server
– TDD (Test Driven Development) based
• All developers commit the update everyday
• CI server compile and run the unit tests
• If a test fails, all receive the failure email
– Know who committed a bad code

– Hudson, Jenkins etc
• Supports SCM version control tools
– CVS, Subversion, Git


Test in Hadoop

• Much harder
– JUnit cannot be used in Hadoop
– Cluster
– Server
– Parallel Computing


Use Cases: Shopzilla

• Hadoop’s Elephant In The Room
– Hadoop testing
• Quality Assurance
– Unit Test: functional units of the S/W
– Integration Test: integrated components
– BDD Test: Behavior of the S/W

• Augmented Development
– Use a dev cluster?
» Too long per day
– Hadoop-In-A-Box


Use Cases: Shopzilla

• Hadoop-In-A-Box
– Fully compatible Mock Environment
• Without a cluster
• Mock cluster state

– Test Locally
• Single Node Pseudo Cluster
• MiniMRCluster
• => can test HDFS, Pig


Use Cases: Yahoo

• Developer
– Wants to run Hadoop codes in the local machine
• Does not want to run Hadoop codes at the Hadoop cluster

• Yahoo HIT
– Hadoop Integration Test
– Run Hadoop tests in the Hadoop Ecosystems
• Deploy HIT on a Hadoop single or cluster
• Run tests in Hadoop, Pig, Hive, Oozie,…


Unit Test in Hadoop

• MRUnit testing framework
– is based on JUnit
– Cloudera donated to Apache
– can test Map Reduce programs
• written on 0.20 , 0.23.x , 1.0.x , 2.x version of Hadoop

– Can test Mapper, Reducer, MapperReducer


Unit Test in Hadoop

• WordCount Example
– reads text files and counts how often words occur.
• The input and the output are text files,

– Need three classes
• WordCount.java
– Driver class with main function

• WordMapper.java
– Mapper class with map method

• SumReducer.java
– Reducer class with reduce method


WordCount Example

• WordMapper.java
– Mapper class with map function
– For the given sample input
• assuming two map nodes
– The sample input is distributed to the maps

• the first map emits:
– <Hello, 1> <World, 1> <Bye, 1> <World, 1>

• The second map emits:
– <Hello, 1> <Hadoop, 1> <Goodbye, 1> <Hadoop, 1>


WordCount Example

• SumReducer.java
– Reducer class with reduce function
– For the input from two Mappers
• the reduce method just sums up the values,
– which are the occurrence counts for each key

• Thus the output of the job is:
– <Bye, 1> <Goodbye, 1> <Hadoop, 2> <Hello, 2> <World, 2>


WordCount.java (Driver)
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
public class WordCount {
public static void main(String[] args) throws Exception {
if (args.length != 2) {
System.out.println("usage: [input] [output]");
System.exit(-1);
}

Job job = Job.getInstance(new Configuration());
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(WordMapper.class);
job.setReducerClass(SumReducer.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setJarByClass(WordCount.class);
job.submit();
}
}


WordCount.java
System.exit(-1);
}

Check Input and Output files

job.submit();
}
}


WordCount.java
System.exit(-1);
}

Set output (key, value) types

job.submit();
}
}


WordCount.java
System.exit(-1);
}

Set Mapper/Reducer classes

job.submit();
}
}


WordCount.java
System.exit(-1);
}

Set Input/Output format classes

job.submit();
}
}


WordCount.java
System.exit(-1);
}

Set Input/Output paths

job.submit();
}
}


WordCount.java
System.exit(-1);
}
Set

Driver class

job.submit();
}
}


WordCount.java
System.exit(-1);
}

Submit the job

to the master node

job.submit();
}
}


WordMapper.java (Mapper class)
import java.io.IOException;
import java.util.StringTokenizer;
import org.apache.hadoop.mapreduce.Mapper;
public class WordMapper extends Mapper<Object, Text, Text, IntWritable> {
private Text word = new Text();
private final static IntWritable one = new IntWritable(1);
@Override
public void map(Object key, Text value,
Context contex) throws IOException, InterruptedException {
// Break line into words for processing
StringTokenizer wordList = new StringTokenizer(value.toString());
while (wordList.hasMoreTokens()) {
word.set(wordList.nextToken());
contex.write(word, one);
}
}
}

WordMapper.java

Extends mapper class with input/
output keys and values

@Override
}
}
}

WordMapper.java
Output (key, value) types
@Override
}
}
}

WordMapper.java
Input (key, value) types
Output as Context type

@Override
}
}
}


WordMapper.java
@Override
Read words from each line
Context contex) throws IOException, InterruptedException { input file
of the
}
}
}

WordMapper.java
@Override
Count each word

}
}
}

Shuffler/Sorter

• Maps emit (key, value) pairs
• Shuffler/Sorter of Hadoop framework
– Sort (key, value) pairs by key
– Then, append the value to make (key, list of values) p
air
– For example,
• The first, second maps emit:
– <Hello, 1> <World, 1> <Bye, 1> <World, 1>
– <Hello, 1> <Hadoop, 1> <Goodbye, 1> <Hadoop, 1>

• Shuffler produces and it becomes the input of the reducer
– <Bye, 1>, <Goodbye, 1>, <Hadoop, <1,1>>, <Hello, <1, 1>>
, <World, <1,1>>


SumReducer.java (Reducer class)
import java.io.IOException;
import java.util.Iterator;
import org.apache.hadoop.mapreduce.Reducer;

public class SumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
private IntWritable totalWordCount = new IntWritable();
@Override
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int wordCount = 0;
Iterator<IntWritable> it=values.iterator();
while (it.hasNext()) {
wordCount += it.next().get();
}
totalWordCount.set(wordCount);
context.write(key, totalWordCount);
}
}


SumReducer.java

Extends Reducer class with input/
output keys and values

@Override
int wordCount = 0;
}
}
}

SumReducer.java
Set output value type

@Override
int wordCount = 0;
}
}
}

SumReducer.java
Set input (key, list of values) type
and output as Context class

@Override
int wordCount = 0;
}
}
}


SumReducer.java
@Override
int wordCount = 0;
For each word,
Iterator<IntWritable> it=values.iterator(); Count/sum the number of values
}
}
}

SumReducer.java
@Override
int wordCount = 0;
}
For each word,
Total count becomes the value
}
}

SumReducer

• Reducer
– Input: Shuffler produces and it becomes the input of
the reducer
• <Bye, 1>, <Goodbye, 1>, <Hadoop, <1,1>>, <Hello, <1, 1>>
, <World, <1,1>>

– Output
• <Bye, 1>, <Goodbye, 1>, <Hadoop, 2>, <Hello, 2>, <World,
2>


MRUnit Test

• How to UnitTest in Hadoop
– Extending JUnit test
• With org.apache.hadoop.mrunit.* API

– Needs to test Driver, Mapper, Reducer
• MapReduceDriver, MapDriver, ReduceDriver
• Add input with expected output


MRUnit Test
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.mrunit.MapDriver;
import org.apache.hadoop.mrunit.MapReduceDriver;
import org.apache.hadoop.mrunit.ReduceDriver;
import org.junit.Before;
import org.junit.Test;
public class TestWordCount {
MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver;
MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
@Before
public void setUp() {
WordMapper mapper = new WordMapper();
SumReducer reducer = new SumReducer();
mapDriver = new MapDriver<LongWritable, Text, Text, IntWritable>();
mapDriver.setMapper(mapper);
reduceDriver = new ReduceDriver<Text, IntWritable, Text, IntWritable>();
reduceDriver.setReducer(reducer);
mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable>();
mapReduceDriver.setMapper(mapper);
mapReduceDriver.setReducer(reducer);
}


MRUnit Test
@Test
public void testMapper() {
mapDriver.withInput(new LongWritable(1), new Text("cat cat dog"));
mapDriver.withOutput(new Text("cat"), new IntWritable(1));
mapDriver.withOutput(new Text("dog"), new IntWritable(1));
mapDriver.runTest();
}
@Test
public void testReducer() {
List<IntWritable> values = new ArrayList<IntWritable>();
values.add(new IntWritable(1));
reduceDriver.withInput(new Text("cat"), values);
reduceDriver.withOutput(new Text("cat"), new IntWritable(2));
reduceDriver.runTest();
}
@Test
public void testMapReduce() {
mapReduceDriver.withInput(new LongWritable(1), new Text("cat cat dog"));
mapReduceDriver.addOutput(new Text("cat"), new IntWritable(2));
mapReduceDriver.addOutput(new Text("dog"), new IntWritable(1));
mapReduceDriver.runTest();
}
}


MRUnit Test

Using MRUnit API, declare MapReduce,
Mapper, Reducer drivers
with input/output (key, value)

MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDr
iver;
@Before
mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, I
ntWritable>();
}


MRUnit Test
iver;
Run setUp() before executing each test method
@Before
ntWritable>();
}


MRUnit Test
iver;
@Before
Instantiate WordCount Mapper, Reducer
ntWritable>();
}


MRUnit Test
iver;
@Before

Instantiate and set Mapper
driver

ntWritable>();
}


MRUnit Test
iver;
@Before

Instantiate and set Reducer
driver

ntWritable>();
}


MRUnit Test
iver;
@Before

Instantiate and set
MapperReducer driver

ntWritable>();
}


MRUnit Test
Mapper test:
Define sample input with expected output

@Test
}
@Test
}


MRUnit Test
@Test
}
Reducer test:

@Test
}


MRUnit Test

MapperReducer test:

@Test
public void testMapReduce() {
mapReduceDriver.withInput(new LongWritable(1), new Text("cat cat dog"));
mapReduceDriver.addOutput(new Text("cat"), new IntWritable(2));
mapReduceDriver.addOutput(new Text("dog"), new IntWritable(1));
mapReduceDriver.runTest();
}
}


MRUnit Test in real

• Need to implement unit tests
– How many?
• all Map, Reduce, Driver

– Problems?
• Mostly work
– But it does not support complicated Map, Reduce APIs

– How many problems you can detect
• Depends on how well you implement MRUnit code


Conclusion

•
•
•
•

MRUnit for Hadoop Unit Test
Development
Integrate with QA site with CI server
Need to use it


Question?


References
1. Hadoop WordCount example with new map reduce api (http://codesfusion.blogspot.com/2013/1
0/hadoop-wordcount-with-new-map-reduce-api.html)
2. Hadoop Word Count Example (http://wiki.apache.org/hadoop/WordCount )
3. Example: WordCount v1.0, Cloudera Hadoop Tutorial (http://www.cloudera.com/content/clouder
a-content/cloudera-docs/HadoopTutorial/CDH4/Hadoop-Tutorial/ht_walk_through.html )
4. Testing Word Count (https://cwiki.apache.org/confluence/display/MRUNIT/Testing+Word+Count)
5. Apache MRUnit Tutorial (https://cwiki.apache.org/confluence/display/MRUNIT/MRUnit+Tutorial )
6. Hadoop Integration Test Suite, Shopzilla (https://github.com/shopzilla/hadoop-integration-test-sui
te )
7. Hadoop’s Elepahnt in the Room, Jeremy Lucas, Shopzilla (http://tech.shopzilla.com/2013/04/hado
ops-elephant-in-the-room/ )
8. Facebook Test MapReduce Local (https://github.com/facebook/hadoop-20/blob/master/src/test/
org/apache/hadoop/mapreduce/TestMapReduceLocal.java )
9. Yahoo HIT Hadoop Integrated Testing (http://www.slideshare.net/ydn/hi-tv3?from_search=1 )


2014 International Software Testing Conference in Seoul

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (6)

Similaire à 2014 International Software Testing Conference in Seoul

Similaire à 2014 International Software Testing Conference in Seoul (20)

Plus de Jongwook Woo

Plus de Jongwook Woo (20)

Dernier

Dernier (20)

2014 International Software Testing Conference in Seoul