1. Your natural partner to develop innovative
solutions
Nokia Institute of Technology
Nokia Internal Use Only

2. Agenda
Agenda
•
•
•
MapReduce Summarization Patterns
MapReduce Coding Best Practices
Ctrending MR Performance Evaluation
• Ctrending MR Execution Summary
• Code Profiling
• Profiling Results
• Code Tuning
• Hbase Configuration Tuning
• Tuning Results
• Refactoring Proposal
Nokia Internal Use Only

3. MapReduce Summarization Patterns
MapReduce Summarization Patterns
•
Numerical Summarizations
• General counting of data set records
• Groups records by a custom key, calculating numerical values per
group
• Known Uses
• Word count, record count, min/max count, avg/median/standard
deviation
Nokia Internal Use Only

4. MapReduce Summarization Patterns
MapReduce Summarization Patterns
•
Inverted Index
• Indexes large data set into keywords
• Mapper emits keywords/ids values and the framework handles most of
the work
• May use IdentityReducer
• Should benefit from Partitioner for load balance
Nokia Internal Use Only

5. MapReduce Summarization Patterns
MapReduce Summarization Patterns
•
•
Counting with Counters
• Leverages MapReduce framework’s counters.
• Counters are all stored in-memory locally on each Mapper, then aggregated by the
framework.
• Better performance, however may not exceed tens of counters definition.
Known Uses
• Count number of records, count small number of groups, summations
Nokia Internal Use Only

6. MapReduce Coding Best Practices
MapReduce Coding Best Practices
•
•
•
Define Output Values
• Create custom Writable extending classes to be used as output from
Mappers;
• Provides cleaner Mapper code and avoids String parsing on Reducer
code side;
Avoid Local Object Creation
• Map and Reduce methods are invoked on very large loops;
• Creating local objects inside map or reduce leads to huge number of
objects being attached to Eden space of Young Generation JVM’s
Heap;
• Reuse Global instances to decrease Young GC Activity;
Use Combiners on Counting Summarizations
• Combiners reduce bandwidth consuption, as it applies aggregations
locally to mappers node, before mapper output is sent to shuffle and
sort phase, then made available for reducers
Nokia Internal Use Only

7. Ctrending MR Performance Evaluation
Ctrending MR Performance Evaluation
• Ctrending MR Execution Summary
• Total MR Jobs Running: 8
• Avg of processed tweets: 2.2 Million
• Tweets identified as Music related: 10.5%
• Total Execution Time: 2 hours and 20 minutes
• Slowest MapReduces:
• Tweets Counter: 46 minutes
• Nokia Entity Id Join: 1 hour and 10 minutes
Nokia Internal Use Only

8. Ctrending MR Code Profiling
Ctrending MR Code Profiling
• Mainly applied to Nokia Id Join Mapper
• Added usage of MapReduce framework’s Counters to collect execution
time metrics
• Also used Counters to sum total of entities id being found in Nokia Id Join
mapper
• Needed to create Static fields in search strategy implementations to
collect execution time metric
Nokia Internal Use Only

9. Ctrending MR Profiling Results
Ctrending MR Profiling Results
TOTAL_ARTISTS_NMS_FOUND
77
TOTAL_ARTISTS_NOT_IN_CACHE
TOTAL_CANDIDATES_FORMATTING_TIME
TOTAL_HBASE_GET_TIME
1,904
67,873
262,647
TOTAL_NORMALIZATION_TIME
22,452
TOTAL_SEARCH_ARTIST_TIME
611,066
TOTAL_SEARCH_CALCULATION_TIME
5,605
TOTAL_SEARCH_NMS_TIME
3,740,552
TOTAL_SEARCH_TIME
4,098,270
TOTAL_SEARCH_TRACK_TIME
3,486,978
TOTAL_TRACKS_NMS_FOUND
145
TOTAL_TRACKS_NOT_IN_CACHE
Nokia Internal Use Only
4,635

10. Ctrending MR Code Tuning
Ctrending MR Code Tuning
• Tuning Tweets Count MapReduce
• Applied IntSumReducer as combiner.
• Ajusted Hbase Scan to fetch and copy records on blocks of
thousands, in order to optimize network usage between nodes.
• Also set blockCache to false, as this table will always be read
sequentially at once.
Nokia Internal Use Only

11. Ctrending MR Code Tuning
Ctrending MR Code Tuning
• Tuning Entity Id Search MapReduce
• Removed unnecessary split/indexof calls
• Removed redundant object creation from map method
Nokia Internal Use Only

12. Ctrending MR Code Tuning
Ctrending MR Code Tuning
• Tuning Entity Id Search MapReduce
• Profiling results shows that NMS Search is the bottleneck
• It costs more than 90% of all MapReduce execution time
• It also shows that NMS Search is not adding enough value
• It founds only 4% of Artists Ids not in cache
• It founds only 3% of Tracks Ids not in cache
• This drove the decision to remove NMS search by simply referencing
CustomCache ISearchStrategy implementation on Mapper setup
method
Nokia Internal Use Only

13. Hbase Configuration Tuning
Hbase Configuration Tuning
• Artists and Tracks Cache is an inverted indexes structure stored on
Hbase tables.
• These tables present high level of random access to it’s records (Get
operations), while Entity Id Search MapReduce performs searches on the
cache.
• This could have performance optimized if Cache table blocks were made
available in RegionServer’s memory.
• Hbase provides Table level configuration property that increases blocks
priority to be stored on RegionServer’s memory
Nokia Internal Use Only

14. Hbase Configuration Tuning
Hbase Configuration Tuning
• Additional configuration is required on Hbase RegionServer, so that
block cache is possible most part of the time.
• hbase.regionserver.global.memstore.upperLimit -> defines maximum
% of Heap available for writing in memstores, before put operations
are actually written to disk files.
• hbase.regionserver.global.memstore.lowerLimit -> defines minimum
% of Heap available for writing in memstores. Flush operations will
free memstore until this limit is reached.
• hfile.block.cache.size -> % of Heap to be used to store blocks inmemory
Nokia Internal Use Only

15. Hbase Configuration Tuning
Hbase Configuration Tuning
• Most Ctrending Hbase put operations are done in batch jobs
(Twitter Crawler).
• Music entities cache requires many Get operations, while
EntityIdSearchMR is executing.
• Simply setting cache tables to be maintained in-memory does not
work, if there is not enough memory available.
• More memory can be made available to cache tables blocks on
RegionServers by decreasing % of Heap reserved to memstore
and increasing it for block cache.
Nokia Internal Use Only

16. Ctrending MR Tuning Results
Ctrending MR Tuning Results
• TweetsCountMR
• Total Execution Time Prior Tuning: 46 minutes (average)
• Total Execution Time After Tuning: 20 minutes (average)
• EntityIdSearchMR
• Total Execution Time Prior Tuning: 1 hour and 10 minutes (average)
• Total Execution Time Adter Tuning: 6 minutes (average)
• CONCLUSION: Do not ever perform HTTP Requests on MapReduces
again!!!
Nokia Internal Use Only

17. Refactoring
Refactoring
• Write batch process to read generated rankings and perform requests
to NMS for music entities which ID was not found.
• Better implement this as a Java multi-thread standalone process,
instead of MapReduce
• As input file is small (the filtered rank), Hadoop default InputFormat
implementations will not split it in many Map tasks.
• Unless a custom InputFormat be implemented, develop a
MapReduce for this will probably take long time to execute, as it will
end up with a single Map task to request NMS for all unknown Ids
• Optimize Heap usage on other MRs by avoiding Object creation on
Map methods.
• Enhance code quality (and even performance), by defining
OutputValues for Trending MRs
Nokia Internal Use Only

18. References
References
• HBase, The Definitive Guide, Lars George, O'Reilly
• MapReduce Design Patterns, Donald Miner, Adam Shook
• Hadoop Official WebSite
• http://hadoop.apache.org/
Nokia Internal Use Only