At a large European publisher, we had the challenge of receiving external data from 10+ different sources, adding up to tens of GBs per day. Add to this that the file formats will sometimes change without notifications and that sometimes connections go bad or files go missing. This is while trying to maintain that at least the amount of data is near correct in an environment where the 'correct' amount of data for a source is often a difficult to predict number somewhere between 20M and 50M records for a particular day. We built a extracting and loading pipeline to get data into Hadoop en expose it via Hive tables, which includes scheduling, reporting, monitoring, transforming and, above all, the ability to respond to changes very quickly. After all, responding to a file format change within the same day or adding a new source in a day are very reasonable user requests (right). We were focused on developer friendliness and rely on a fully open source stack, using Hadoop, Hive, Jenkins, various scripting languages and more. This is my talk about the setup and our lessons learned. In our quest for data quality, we also did work on attempting to predict the expected data volumes, based on seasonality and weather information, in order to proactively alert when a data import appears to fall short of the expected volume. I will include these results in the talk.

1. GoDataDriven
PROUDLY PART OF THE XEBIA GROUP
@fzk
frisovanvollenhoven@godatadriven.com
Dirty Data
Friso van Vollenhoven
It’s a mess. It’s your problem.

7. 'februari-22 2013'

9. A: Yes, sometimes as
often as 1 in every 10K
calls. Or about once a
week at 3K files / day.

12. þ

13. þ

14. TSV ==
thorn separated values?

15. þ == 0xFE

16. or -2, in Hive
CREATE TABLE browsers (
browser_id STRING,
browser STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY '-2';

23. • The format will change
• Faulty deliveries will occur
• Your parser will break
• Records will be mistakingly produced (over-logging)
• Other people test in production too (and you get the
data from it)
• Etc., etc.

24. •Simple deployment of ETL code
•Scheduling
•Scalable
•Independent jobs
•Fixable data store
•Incremental where possible
•Metrics

25. EXTRACT
TRANSFORM
LOAD

31. • No JVM startup overhead for Hadoop API usage
• Relatively concise syntax (Python)
• Mix Python standard library with any Java libs

33. • Flexible scheduling with dependencies
• Saves output
• E-mails on errors
• Scales to multiple nodes
• REST API
• Status monitor
• Integrates with version control

35. Deployment
git push jenkins master

36. Independent jobs
source (external)
staging (HDFS)
hive-staging (HDFS)
Hive
HDFS upload + move in place
MapReduce + HDFS move
Hive map external table + SELECT INTO

37. Out of order jobs
• At any point, you don’t really know what ‘made it’
to Hive
• Will happen anyway, because some days the data
delivery is going to be three hours late
• Or you get half in the morning and the other half
later in the day
• It really depends on what you do with the data
• This is where metrics + fixable data store help...

38. Fixable data store
• Using Hive partitions
• Jobs that move data from staging create partitions
• When new data / insight about the data arrives,
drop the partition and re-insert
• Be careful to reset any metrics in this case
• Basically: instead of trying to make everything
transactional, repair afterwards
• Use metrics to determine whether data is fit for
purpose

39. Metrics

40. Metrics service
• Job ran, so may units processed, took so much
time
• e.g. 10GB imported, took 1 hr
• e.g. 60M records transformed, took 10 minutes
• Dropped partition
• Inserted X records into partition

41. GoDataDriven
We’re hiring / Questions? / Thank you!
@fzk
frisovanvollenhoven@godatadriven.com
Friso van Vollenhoven