Let's face it: data warehousing in the traditional sense has been tedious, lacking agility, and slow. Often designed and built around a static set of up front questions, they are basically a big database tailored to the application of populating dashboards. The fact that the volume of data that we need to deal with recently exploded by several order of magnitude isn't improving the situation. It's no surprise that we see a new class of data warehousing setups emerge, using big data technologies en NoSQL stores. A nice side effect is that these solutions are usually not only the domain of the BI crowd, but can also be developer friendly and allow development of more data driven apps. In this session I will present about experiences using Hadoop and other tools from the Hadoop ecosystem, such as Hive, Pig and bare MapReduce, to handle data that grows tens of GBs per day. We create a system where data is captured, stored and made available to different users and use cases, ranging from end users that write SQL queries to software developers that access the underlying data to create data driven products. I will cover topics like ETL, querying, development and deployment and reporting; using a fully open source stack, of course.

1. GoDataDriven
PROUDLY PART OF THE XEBIA GROUP
@mids106
jorisbontje@godatadriven.com
Building a Big Data
Warehouse
Joris Bontje
Big Data Hacker

2. About Me
Big Data Hacker
Data Driven Solution Architect
HadoopTrainer

4. About GoDataDriven

5. Data Warehouse
Evolution

6. http://en.wikipedia.org/wiki/Data_warehouse
In computing, a data warehouse is a
database used for reporting and data
analysis.

7. Database Architecture (1.0)
Products)
Customers)
Orders)
Inventory)
Sales)DB)

8. Analytical Database (2.0)
Sales&
Inventory&Customers&
Products&
Orders&

9. Basic DWH Architecture
TX#
DB#
Analy+cal#
DB#
BI#ETL

10. Data Marts
TX#
DB#
DW#
Sales#
Mktg#
Prch#
BI#

11. Multiple Data-Sources
other&
Files&
TX&
DB&
DW&
Sales&
Mktg&
Prch&
BI&

12. Operational Data Store
DW#ODS#
other#
Files#
TX#
DB# Sales#
Mktg#
Prch#
BI#

13. Hadoop

14. GoDataDriven
No Hadoop
DW#ODS#
other#
Files#
TX#
DB# Sales#
Mktg#
Prch#
BI#

15. GoDataDriven
ETL Engine
other&
Files&
TX&
DB& Sales&
Mktg&
Prch&
DW
BI&

16. GoDataDriven
Tiered Data Warehouse
other&
Files&
TX&
DB& Sales&
Mktg&
Prch&
BI&

17. GoDataDriven
Analytical Query Engine
other&
Files&
TX&
DB&
BI&

18. Tools

19. Tools

20. Tools Applied

21. Tools Applied

22. Considerations

23. Considerations
Big Data is dirty
Automate everything
Monitoring and QA become the same thing

24. My Past Trends
Big Data Forum 2012

25. My PastTrends
Cloud / On-demand

26. My PastTrends
Hadoop Hardware

27. My PastTrends
Batch → Real-Time

28. New Trends
XebiCon 2013

29. Trends
Impala
Open Source, Real-time Query engine
for Hadoop

30. Trends
Defacto standard for Hadoop metadata

31. Simple Database Architecture
Products)
Customers)
Orders)
Inventory)
Sales)DB)

32. The future?
Products)
Customers)
Orders)
Inventory)
Sales)

33. GoDataDriven
We’re hiring / Questions? / Thank you!
@mids106
jorisbontje@godatadriven.com
Joris Bontje
Big Data Hacker