30B events a day with hadoop

30 Billion Events a Day with Hadoop

Michael Brown, CTO, comScore, Inc.
May 10th, 2012

comScore is a Global Leader in Measuring the Digital World

NASDAQ SCOR
Clients 1860+ worldwide
Employees 1000+
Headquarters Reston, VA
170+ countries under measurement;
Global Coverage
43 markets reported

Local Presence 32 locations in 23 countries

© comScore, Inc. Proprietary. 2 V1011

Some of our Clients
Media Agencies Telecom/Mobile Financial Retail Travel CPG Pharma Technology


The Trusted Source for Digital Intelligence Across Vertical Markets

9 out of the top 10 9 out of the top 10
INVESTMENT BANKS AUTO INSURERS

WIRELESS CARRIERS INTERNET SERVICE
PROVIDERS

ONLINE PROPERTIES PHARMACEUTICAL
COMPANIES

ADVERTISING AGENCIES CONSUMER FINANCE
COMPANIES

MAJOR MEDIA COMPANIES CPG COMPANIES


Unified Digital Measurement™ (UDM) Establishes Platform For
Panel + Census Data Integration

Global PERSON Global DEVICE
Measurement Measurement

PANEL CENSUS

Unified Digital Measurement (UDM)
Patent-Pending Methodology
Adopted by 90% of Top 100 U.S. Media Properties


Beacon Heat Map

© comScore, Inc. Proprietary. 6

Worldwide Tags per Month

Monthly Records Collection
1,000,000,000,000

900,000,000,000

800,000,000,000

700,000,000,000

600,000,000,000
# of records

500,000,000,000

400,000,000,000

300,000,000,000

200,000,000,000

100,000,000,000

0
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb

Apr

Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb

Apr

Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb

Apr
Mar

Mar

Mar
May

May

May
2009 2010 2011 2012

Panel Records Beacon Records

Our Event Volume in Perspective

Property Page Views (MM)

FACEBOOK.COM 472,814

Google Sites 302,802

Yahoo! Sites 90,448

Total 866,064

Source: comScore MediaMetrix Worldwide April 2012


Growth Slides
1,600,000,000,000

R² = 0.9335
1,400,000,000,000

1,200,000,000,000

1,000,000,000,000

800,000,000,000

600,000,000,000

400,000,000,000

200,000,000,000

-


The Project:
Census Web Agg


The Problem Statement

§  Calculate the number of events and unique cookies for each key
§  Key take aways
–  Data on input will be sessionized daily
–  Need to process all data for a month
–  Need to calculate values for Total Internet and for each site under
measurement


Counting Uniques from a Time Ordered Log File

A Major Downsides:
Need to keep all key elements in memory.
D Constrained to one machine for final aggregation.

B

C

B

A

A


Counting Uniques from a Key Ordered Log File

A Major Downsides:
Need to sort data in advance.
A The sort time increases as volume grows.

A

B

B

C

D


Scaling Issue

§  As our volume has grown we have the following stats:
–  Over 900 billion events per month
–  Over 150 billion sessions per month
–  Over 5,000 reportable sites
–  Over 50 countries
–  We see 15 billion distinct cookies in a month
–  5 sites have over 1 billion cookies in a month
–  The sum of all distinct cookies is 377 billion
–  We only need to output 15 million rows


Counting Uniques from a Key Ordered Log File


Windows v1 (Single Server)

§  Time to process data for first few months
Month Wall Time (hours)

Jul 2009 8
Aug 2009 10
Sep 2009 11
Oct 2009 16
Nov 2009 37

§  V1 Processed sessions at roughly 250K rows/sec

§  Problems with this version:
–  Slow
–  Not Scalable
–  Dedicated Server
–  Bottleneck for delivering production


Counting Uniques from Sharded Key Ordered Log Files


Windows v2

§  Features of this version
–  Distributed (32 servers)
–  Multithreaded
–  Data Localization
–  Very low network data transfer
–  Handling the data growth

§  The V2 code processed data over 8 million rows/sec
–  1 hour for Dec 2009; 5 hours for April 2012

§  Issues
–  Data is distributed by ID into 64 parts
–  Possibilities for skew in distribution key, that impacts performance and high disk usage on a node
–  All data replication is manual, along with recovery
–  Results cannot be calculated if any node is down
–  Adding new servers or change in parts is a ton of effort
–  Overhead to maintain framework to run distributed jobs


Enter the Elephant

§  Why Hadoop?
–  Scalable
–  Low risk to lose data due to replication
–  Run on a shared production cluster
–  No overhead to maintain framework
–  Easy job submission and management


Basic Approach

§  Leverage Pig for POC
–  Pig Latin is easy for developers and data analysts to learn
–  Rapid application development vs. M/R applications (i.e. 1 line of Pig Latin = 20 lines in Java Map/
Reduce)
–  Extendable via UDFs


Performance of Basic Approach on Various Samples

Aggregation Performance
80.00

70.00

60.00

50.00
Time (minutes)

40.00

30.00

20.00

10.00

0.00
372 GB (3%) 744 GB (6%) 1116 GB (9%)
Input data size

© comScore, Inc. Proprietary. 21 Note: Target data size is over 10 TB

M/R Data Flow

B C A B C A

Mapper
Map Mapper Mapper
Map Map

A A B B C C

Reduce Reduce Reduce

A B C


Basic Approach Retrospective

§  Processing speed is not scaling to our needs on a sample of the input data
§  Diagnosis
–  Most aggregations could not take significant advantage of combiners. Not a Pig issue.
–  Large shuffles caused poor job performance. In some cases large aggregations ran slower on the
Hadoop cluster compared to the current architecture

§  Diagnosis
–  A new approach is required to reduce the shuffle


Solution to reduce the shuffle

§  The Problem:
–  Most aggregations within comScore can not take advantage of combiners, leading to large shuffles
and job performance issues

§  The Idea:
–  Partition and sort data on a daily basis
–  Create a custom input format to merge daily partitions for monthly aggregations


Custom Input Format with Map Side Aggregation

B C A B C A

A Mapper
Map B Mapper
Map C Mapper
Map

Combiner Combiner Combiner

A B C

Reduce Reduce Reduce

A B C


Performance of v2 on Various Samples

Aggregation Performance
120.00

100.00

80.00
Time (minutes)

60.00

40.00

20.00

0.00
372 GB (3%) 744 GB (6%) 1116 GB (9%) 10304 GB (100%)
Input data size

Pig Custom Input Format


Partitioning Summary

§  Benefits:
–  A large portion of the aggregation can be completed in the map phase
–  Applications can now take advantage of combiners
–  Shuffles sizes are minimal

§  Risks:
–  Data locality loss
–  Map failures might result in long run times. This is dependent on the size of the partitions


Full Sample Performance

§  Full set of data analysis
–  10 TB of input data
–  150 billion session rows

§  Total Time
–  1 hour, 45 minutes
–  Over 23,000,000 rows/sec


Future Ideas

§  HBase
–  Unique cookie calculations are free as data is more organized
–  How will data loading fare?

§  Data Locality
–  Ideally it would be great to provide additional clues to the storage of the data
–  Not sure if it will be included in Hadoop

§  Connection to a MPP DB
–  We also leverage Greenplum DB, we could connect to each sharded instance


Hadoop Cluster

§  Production Hadoop Cluster
–  80 nodes: Mix of Dell R710 and R510
–  Each R510 has (12x2TB drives; 64GB RAM; 24 cores)
–  1768 total CPUs
–  4.7TB total memory
–  1200TB total disk space
–  Our distro is MapR M5 1.2.7


Useful Factoids
Colorful, bite-sized graphical representations of the best discoveries we unearth.

Visit www.comscoredatamine.com or follow @datagems for the latest gems.


Thank You!

Michael Brown
CTO
comScore, Inc.

mbrown@comscore.com


30B events a day with hadoop

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (7)

Similaire à 30B events a day with hadoop

Similaire à 30B events a day with hadoop (7)

Plus de DataWorks Summit

Plus de DataWorks Summit (20)

Dernier

Dernier (20)

30B events a day with hadoop