This document discusses the evolution of data analytics and modeling. It describes three waves: the first with slow hardware and manual entry; the second with faster PCs but tool explosions; and the third wave now with big data, cloud warehouses, and data-driven tools like Looker and BigQuery. It argues that in this current wave, having a flexible yet performant data model built on SQL in a warehouse, and using a language like LookML to define relationships and translate questions, allows gaining reliable answers with agility without worrying about low-level syntax or tools.

1. : Blue Apron uses Looker and Big Query to
Advance their Analytics
May 2, 2017

2. Data Modeling in the Age of
Cloud Warehouses
Daniel Mintz
Chief Data Evangelist

3. 3
Some History

4. 4

5. 5
First wave
● Slow, expensive
hardware
● IT-centric
● Manual data entry
Why? Advantages
● Reliable answers
● Pixel-perfect
● Fast (for 1990)
Disadvantages
● Inflexible
● Locked down
● Low-resolution

6. 6

7. 7
Second wave
● Faster PCs
● IT bottleneck
● Growing data
volumes
Why? Advantages
● Agility
● Faster time-to-
insight
● Higher-resolution
Disadvantages
● No shared model
● Dependent on
data extracts
● Tool explosion

8. 8
But…

9. 9
...what if?
1. Storing big data got cheap?
2. Querying big data was fast?
3. Warehouses scaled elastically?
4. Ops was handled?
5. You paid for what you used?

10. 10
Duplicate
records are
“free”
● Repeated data
costs ~$0
● It’s OK to duplicate
if it doesn’t create
inconsistency

11. 11
Distributed
systems are the
new normal
● JOIN/Shuffle
incurs a new cost
● What does any
given JOIN
improve?

12. 12
Then...

13. 13
...you would want?
1. A model that
a. aids performance
b. is flexible and easy to update
c. reflects the real world and is easy to understand
2. A tool that could leverage the warehouse directly
3. A language to abstract away low-level concerns

14. 14
Detour!

15. 1940 1950 1960 1970 1980 1990 2000 2010
Machine Code
Assembly
FORTRAN
COBOL
BASIC C C++
Objective-C
Python
Java
Javascript
PHP
C#
Go
Ruby
1940 1950 1960 1970 1980 1990 2000 2010
Data Written to Files
Roll Your Own b-tree
Codd SQL
Oracle V2
IBM DB2
T-SQL MySQL
PostgreSQL
??????
Programming Language Development
Data Language Development

16. 16
What do we want from data language?
1. Define relationships and definitions once
2. Retain agility
3. Translate business questions to data queries
4. Stay performant
5. Stop worrying about syntax

17. 17
“What about SQL? I love SQL!”
It’s proven, powerful and versatile
Except:
SELECT
DATE(orders.timestamp),
SUM(orders.total)
FROM
orders
WHERE
orders.status = ‘completed’
GROUP BY 1

18. 18
LookML is one solution: builds on SQL
● Reusable
● Collaborative
● Flexible
● Organized
● Version-controlled

19. 19

20. 20
Third wave
● Big Data
Revolution
● Too many tools
● The Internet
Why? Advantages
● Reliable answers
● Agility
● Best-in-class
tools
● Full resolution
Disadvantages
● Shift in thinking
Need a powerful
warehouse
Insight
abundance

21. 21
?