Moving Past Infrastructure Limitations Presented by MediaMath This presentation was given at a Big Data Warehousing Meetup with Caserta Concepts, MediaMath and Qubole. You can learn more about the event here: http://www.meetup.com/Big-Data-Warehousing/events/228372516/ Event description: At Caserta Concepts, we are firm believers in big data thriving on the cloud. The instant-on, nearly unlimited storage and computing capabilities of AWS has made it the defacto solution for a full spectrum of organizations needing to process large amounts of data. What's more, an ecosystem of value-added platforms has emerged to further ease and democratize the implementation of cloud based solutions. Qubole has developed a great platform for easily deploying and managing ephemeral and long-lived Hadoop and Spark clusters on AWS. Moving Past Infrastructure Limitations: Data Warehousing at MediaMath Over the past year and a half, MediaMath has undertaken a “data liberation” effort in an attempt to leave their bigbox, monolithic data warehouse behind. In this talk, Rory Sawyer, Software Engineer at MediaMath, will describe how this effort transformed MediaMath’s legacy architecture and legacy mindset, which imposed harsh inefficiencies on data sharing and utilization. The current mindset removes these inefficiencies and allows them to say “yes” to more projects and ideas. Rory will also demo how MediaMath uses Amazon Web Services and Qubole so that infrastructure is no longer a limiting factor on what and how users query. This combination allows them to scale their resources up and down as needed while bridging different data sources and execution engines. Using and extending MediaMath’s data warehousing is no longer a privileged activity but an ability that every employee and client has.

1. ©2016 MediaMath Inc. 1
Moving Past
Infrastructure Limitations
Presented by MediaMath
FEB 16, 2016

2. ©2016 MediaMath Inc. 2
02.16.2016
Rory Sawyer – Software Engineer, Data Platform
Moving Past Infrastructure
Limitations

3. ©2016 MediaMath Inc. 3

4. ©2016 MediaMath Inc. 4
Massive Volume of Data
 180 billion impression opportunities a day

5. ©2016 MediaMath Inc. 5
Massive Volume of Data
 180 billion impression opportunities a day
 3+ million peak qps

6. ©2016 MediaMath Inc. 6
Massive Volume of Data
 180 billion impression opportunities a day
 3+ million peak qps
 3+ TB of data per day (compressed)

7. ©2016 MediaMath Inc. 7
Massive Volume of Data
 180 billion impression opportunities a day
 3+ million peak qps
 3+ TB of data per day (compressed)
 Logs represent financial transactions
Every record counts!

8. ©2016 MediaMath Inc. 8
MediaMath’s Data Platform
 Centralized location for data at MM
Collect data from across the company
Standardize access for internal and external clients
 End-result of data warehouse transformation

9. ©2016 MediaMath Inc. 9
Once Upon A Time
The Old Days
Etc..

10. ©2016 MediaMath Inc. 10
Architecture – 2013

11. ©2016 MediaMath Inc. 11
Data Warehousing at MM Circa 2013

12. ©2016 MediaMath Inc. 12
Data Warehousing at MM Circa 2013
 No proper QA/testing environment

13. ©2016 MediaMath Inc. 13
Data Warehousing at MM Circa 2013
 No proper QA/testing environment
 Production workflows and ad-hoc analytics ran side-by-side

14. ©2016 MediaMath Inc. 14
Data Warehousing at MM circa 2013
 No proper QA/testing environment
 Production workflows and ad-hoc analytics ran side-by-side
 Scaling becomes an issue
Developing/testing/deploying changes to workflows frustrating
Copying data to more monolithic systems
More shell, more problems

15. ©2016 MediaMath Inc. 15
Data Access circa 2013 – Users and Consumers
 Tools: SQL, shell
 Consumers: Data analysts, data engineers

16. ©2016 MediaMath Inc. 16
Data Access circa 2013
 Logs: Custom FTP transfers
Merely extracting data could cause production problems
FTP could run out of space

17. ©2016 MediaMath Inc. 17
Data Access circa 2013
 Logs: Custom FTP transfers
Merely extracting data could cause production problems
FTP could run out of space
 Heavy reliance on canned reports
Served via reporting API
Updated at most three times a day, usually just once a day

18. ©2016 MediaMath Inc. 18
Data Access circa 2013
 Logs: Custom FTP transfers
Merely extracting data could cause production problems
FTP could run out of space
 Heavy reliance on canned reports
Served via reporting API
Updated at most three times a day, usually just once a day
 Hard to keep pace with growing demands
Internal Clients
External Clients

19. ©2016 MediaMath Inc. 19
Data Liberation

20. ©2016 MediaMath Inc. 20
Moving Past Infrastructure
 Resource flexibility

21. ©2016 MediaMath Inc. 21
Moving Past Infrastructure
 Resource flexibility
 Fully own our conceptual problems
Can’t just get a bigger box or a higher support license

22. ©2016 MediaMath Inc. 22
Moving Past Infrastructure
 Resource flexibility
 Fully own our conceptual problems
Can’t just get a bigger box or a higher support license
 Lower barrier to entry

23. ©2016 MediaMath Inc. 23
Moving Past Infrastructure
 Resource flexibility
 Fully own our conceptual problems
Can’t just get a bigger box or a higher support license
 Lower barrier to entry
Decouple storage and computation

24. ©2016 MediaMath Inc. 24
Move to the Cloud

25. ©2016 MediaMath Inc. 25
Move to the Cloud
 Simple Storage Service (S3):
Primary data store; source of truth
Append-only. Update = delete + append

26. ©2016 MediaMath Inc. 26
Move to the Cloud
 Simple Storage Service (S3):
Primary data store; source of truth
Append-only. Update = delete + append
 Elastic Map Reduce (EMR):
Transient hadoop clusters
Spot instances – save money

27. ©2016 MediaMath Inc. 27
Move to the Cloud
 Simple Storage Service (S3):
Primary data store; source of truth
Append-only. Update = delete + append
 Elastic Map Reduce (EMR):
Transient hadoop clusters
Spot instances – save money
 Redshift:
Columnar storage for efficient querying

28. ©2016 MediaMath Inc. 28
Data Platform – Today

29. ©2016 MediaMath Inc. 29
Data Platform – Today

30. ©2016 MediaMath Inc. 30
Data Access – Today

31. ©2016 MediaMath Inc. 31
Developer Experience

32. ©2016 MediaMath Inc. 32
Developer Experience
 Get to say “yes” more
Rapid development/testing/deployment removes inertia

33. ©2016 MediaMath Inc. 33
Developer Experience
 Get to say “yes” more
Rapid development/testing/deployment removes inertia
 Clearly distinct, perfectly synced QA environment
Run multiple versions of workflows simultaneously on same source data

34. ©2016 MediaMath Inc. 34
Developer Experience
 Get to say “yes” more
Rapid development/testing/deployment removes inertia
 Clearly distinct, perfectly synced QA environment
Run multiple versions of workflows simultaneously on same source data
 More control over components

35. ©2016 MediaMath Inc. 35
Developer Experience
 Get to say “yes” more
Rapid development/testing/deployment removes inertia
 Clearly distinct, perfectly synced QA environment
Run multiple versions of workflows simultaneously on same source data
 More control over components
 Localized impact of processing
Each team uses their own compute environment

36. ©2016 MediaMath Inc. 36
We don’t worry about this like we used to

37. ©2016 MediaMath Inc. 37
Improved User Experience

38. ©2016 MediaMath Inc. 38
Improved User Experience
 Augmented standard reporting with easily-accessible data
warehouse
AWS + Qubole provides value to all skill levels

39. ©2016 MediaMath Inc. 39
Improved User Experience
 Augmented standard reporting with easily-accessible data
warehouse
AWS + Qubole provides value to all skill levels
 Transparently handle different data sources
Bridge storage types and AWS accounts

40. ©2016 MediaMath Inc. 40
Improved User Experience
 Augmented standard reporting with easily-accessible data
warehouse
AWS + Qubole provides value to all skill levels
 Transparently handle different data sources
Bridge storage types and AWS accounts
 Choose your preferred query method
Spark, MapReduce, Flink, or BI tool

41. ©2016 MediaMath Inc. 41
Improved User Experience
 Augmented standard reporting with easily-accessible data
warehouse
AWS + Qubole provides value to all skill levels
 Transparently handle different data sources
Bridge storage types and AWS accounts
 Choose your preferred query method
Spark, MapReduce, Flink, or BI tool
 All barriers removed

42. ©2016 MediaMath Inc. 42
Productize it, cap’n

43. ©2016 MediaMath Inc. 43
Productize it, cap’n
 Log level data API
Direct log access on S3
 Interactive Query
Scalable data processing with Qubole

44. ©2016 MediaMath Inc. 44
Hive

45. ©2016 MediaMath Inc. 45
Spark

46. ©2016 MediaMath Inc. 46
SmartQuery

47. ©2016 MediaMath Inc. 47
Clusters

48. ©2016 MediaMath Inc. 48
Qubole’s Greatest Hits

49. ©2016 MediaMath Inc. 49
Hybrid Life

50. ©2016 MediaMath Inc. 50
New and Old

51. ©2016 MediaMath Inc. 51
Managing a Hybrid Warehouse

52. ©2016 MediaMath Inc. 52
Managing a Hybrid Warehouse
 Upfront effort to keep old and new consistent
After that, could migrate in pieces

53. ©2016 MediaMath Inc. 53
Managing a Hybrid Warehouse
 Upfront effort to keep old and new consistent
After that, could migrate in pieces
 Keeping datasets in sync
Store metadata about datasets and processes
Keep record of what data was processed by which batches

54. ©2016 MediaMath Inc. 54
Managing a Hybrid Warehouse
 Upfront effort to keep old and new consistent
After that, could migrate in pieces
 Keeping datasets in sync
Store metadata about datasets and processes
Keep record of what data was processed by which batches

55. ©2016 MediaMath Inc. 55
Ch-ch-ch-challenges

56. ©2016 MediaMath Inc. 56
Ch-ch-ch-challenges
 Spot instances: bid too low, jobs never start
Build processes around selecting best/cheapest zones

57. ©2016 MediaMath Inc. 57
Ch-ch-ch-challenges
 Spot instances: bid too low, jobs never start
Build processes around selecting best/cheapest zones
 Maintaining two systems at once
Consistency, monitoring, updates…

58. ©2016 MediaMath Inc. 58
Ch-ch-ch-challenges
 Spot instances: bid too low, jobs never start
Build processes around selecting best/cheapest zones
 Maintaining two systems at once
Consistency, monitoring, updates…
 Migrating mindset
New set of questions to answer

59. ©2016 MediaMath Inc. 59
What we’ve learned

60. ©2016 MediaMath Inc. 60
Life after Liberation

61. ©2016 MediaMath Inc. 61
Life after Liberation
 Decentralize all the things
Single-machine -> distributed computing
Single data team -> data engineers on all the teams

62. ©2016 MediaMath Inc. 62
Life after Liberation
 Decentralize all the things
Single-machine -> distributed computing
Single data team -> data engineers on all the teams
 Engineers on every team
Data Science – Spark (Scala)
Analytics – Spark/Hive (with Redshift connector)
Product – Hive
Engineering – Spark/Hive/MapReduce
Business analysts – SmartQuery

63. ©2016 MediaMath Inc. 63
Data Access circa 2013 – Users and Consumers
 Tools: SQL, shell
 Consumers: Data analysts, data engineers

64. ©2016 MediaMath Inc. 64
Data Access Today – Users and Consumers
 Tools: Hadoop (Scalding, Hive), Spark, RDBMS
 Consumers: Engineers, product managers, business
analysts, etc.

65. ©2016 MediaMath Inc. 65
The Cost of Decentralization

66. ©2016 MediaMath Inc. 66
The Cost of Decentralization
 Different producers and consumers have different priorities
File format, end-to-end latency, correctness, etc…

67. ©2016 MediaMath Inc. 67
The Cost of Decentralization
 Different producers and consumers have different priorities
File format, end-to-end latency, correctness, etc…
 Adding a platform layer could add friction

68. ©2016 MediaMath Inc. 68
Not Abandoning Managed Infrastructure
or: There and Back Again

69. ©2016 MediaMath Inc. 69
Not Abandoning Managed Infrastructure
or: There and Back Again
 Managed hardware is still important
On-premises Hadoop cluster
Clients ETL into managed hardware

70. ©2016 MediaMath Inc. 70
Not Abandoning Managed Infrastructure
or: There and Back Again
 Managed hardware is still important
On-premises Hadoop cluster
Clients ETL into managed hardware
 Experience with Data Liberation broke down “walled garden” feel
of AWS

71. ©2016 MediaMath Inc. 71
Some sort of “last slide” title

72. ©2016 MediaMath Inc. 72
Some sort of “last slide” title
 Moving DW to cloud has proven itself
Quick development allows us to keep pace
Ease of use helps teams and clients fine tune their own reporting

73. ©2016 MediaMath Inc. 73
Some sort of “last slide” title
 Moving DW to cloud has proven itself
Quick development allows us to keep pace
Ease of use helps teams and clients fine tune their own reporting
 Re-thinking the tools and skills needed for data warehousing

74. ©2016 MediaMath Inc. 74
Some sort of “last slide” title
 Moving DW to cloud has proven itself
Quick development allows us to keep pace
Ease of use helps teams and clients fine tune their own reporting
 Re-thinking the tools and skills needed for data warehousing
 Avoid tech debt by evolving our software and ideas before
committing to hardware

75. ©2016 MediaMath Inc. 75
Some sort of “last slide” title
 Moving DW to cloud has proven itself
Quick development allows us to keep pace
Ease of use helps teams and clients fine tune their own reporting
 Re-thinking the tools and skills needed for data warehousing
 Avoid tech debt by evolving our software and ideas before
committing to hardware
 Move away from trickle-down data

76. ©2016 MediaMath Inc. 76
THANK YOU!
Rory Sawyer
Software Engineer
Data Platform