Over the last decade, the 3Vs of data - Volume, Velocity & Variety has grown massively. The Big Data revolution has completely changed the way companies collect, analyze & store data. Advancements in cloud-based data warehousing technologies have empowered companies to fully leverage big data without heavy investments both in terms of time and resources. But, that doesn’t mean building and managing a cloud data warehouse isn’t accompanied by any challenges. From deciding on a service provider to the design architecture, deploying a data warehouse tailored to your business needs is a strenuous undertaking. Looking to deploy a data warehouse to scale your company’s data infrastructure or still on the fence? In this presentation you will gain insights into the current Data Warehousing trends, best practices, and future outlook. Learn how to build your data warehouse with the help of real-life use-cases and discussion on commonly faced challenges. In this session you will learn: - Choosing the best solution - Data Lake vs. Data Warehouse vs. Data Mart - Choosing the best Data Warehouse design methodologies: Data Vault vs. Kimball vs. Inmon - Step by step approach to building an effective data warehouse architecture - Common reasons for the failure of data warehouse implementations and how to avoid them

1. Data Warehousing Trends,
Best Practices, and Future
Outlook
James Serra
Data Platform Architecture Lead
EY
JamesSerra3@gmail.com
Blog: JamesSerra.com

2. About Me
 EY, Data Platform Architecture Lead
 Was previously a Data & AI Architect at Microsoft for seven years
 In IT for 35 years, worked on many BI and DW projects
 Worked as desktop/web/database developer, DBA, BI and DW architect and developer, MDM
architect, PDW/APS developer
 Been perm employee, contractor, consultant, business owner
 Presenter at PASS Summit, SQLBits, Enterprise Data World conference, Big Data Conference
Europe, SQL Saturdays
 Blog at JamesSerra.com
 Former SQL Server MVP
 Author of book “Reporting with Microsoft SQL Server 2012”

3. Agenda
 Why Data Warehouse
 Data Warehouse vs Data Mart
 Kimball vs Inmon
 Data Vault
 Data Lake
 Why Data Warehouses fail
 How Data Warehouses succeed
 Data Warehouse concepts
 Modern Data Warehouse
 Data Fabric
 Data Lakehouse
 Data Mesh

4. I tried to build a data warehouse on my own…
And ended up passed-out drunk in a Denny’s
parking lot
Let’s prevent that from happening…

5. What a Data Warehouse is not
• A data warehouse is not a copy of a source database with the name prefixed with “DW”
• It is not a copy of multiple tables (i.e. customer) from various sources systems unioned
together in a view
• It is not a dumping ground for tables from various sources with not much design put into it

6. What is a Data Warehouse and why use one?
A data warehouse is where you store data from multiple data sources to be used for historical and
trend analysis reporting. It acts as a central repository for many subject areas and contains the
"single version of truth". It is NOT to be used for OLTP applications.
Reasons for a data warehouse:
 Reduce stress on production system
 Optimized for read access, sequential disk scans
 Integrate many sources of data
 Keep historical records (no need to save hardcopy reports)
 Restructure/rename tables and fields, model data
 Protect against source system upgrades
 Use Master Data Management, including hierarchies
 No IT involvement needed for users to create reports
 Improve data quality and plugs holes in source systems
 One version of the truth
 Easy to create BI solutions on top of it (i.e. Azure Analysis Services Cubes)

7. Why use a Data Warehouse?
Legacy applications + databases = chaos
Production
Control
MRP
Inventory
Control
Parts
Management
Logistics
Shipping
Raw Goods
Order Control
Purchasing
Marketing
Finance
Sales
Accounting
Management
Reporting
Engineering
Actuarial
Human
Resources
Continuity
Consolidation
Control
Compliance
Collaboration
Enterprise data warehouse = order
Single version
of the truth
Enterprise Data
Warehouse
Every question = decision
Two purposes of data warehouse: 1) save time building reports; 2) slice in dice in ways you could not do before

8. Enterprise Data Maturity Stages
Structured data is
transacted and
locally managed.
Data used
reactively
STAGE 2:
Informative
STAGE 1:
Reactive
Structured data is
managed and
analyzed centrally
and informs the
business
Data capture is
comprehensive and
scalable and leads
business decisions
based on advanced
analytics
STAGE 4:
Transformative
STAGE 3:
Predictive Data transforms
business to drive
desired outcomes.
Real-time
intelligence
Rear-view
mirror
Any data, any
source, anywhere at
scale

9. Data Warehouse vs Data Mart
 Data Warehouse: A single organizational repository of enterprise wide data
across many or all subject areas
 Holds multiple subject areas
 Holds very detailed information
 Works to integrate all data sources
 Feeds data mart
 Data Mart: Subset of the data warehouse that is usually oriented to specific
subject (finance, marketing, sales)
• The logical combination of all the data marts is a data warehouse
In short, a data warehouse as contains many subject areas, and a data mart
contains just one of those subject areas

10. Kimball and Inmon Methodologies
Two approaches for building data warehouses

11. Kimball and Inmon Myths
 Myth: Kimball is a bottom-up approach without enterprise focus
 Really top-down: BUS matrix (choose business processes/data sources), conformed
dimensions, MDM
 Myth: Inmon requires a ton of up-front design that takes a long time
 Inmon says to build DW iteratively, not big bang approach (p. 91 BDW, p. 21 Imhoff)
 Myth: Star schema data marts are not allowed in Inmon’s model
 Inmon says they are good for direct end-user access of data (p. 365 BDW), good for data
marts (p. 12 TTA)
 Myth: Very few companies use the Inmon method
 Survey had 39% Inmon vs 26% Kimball. Many have a EDW
 Myth: The Kimball and Inmon architectures are incompatible
 They can work together to provide a better solution

12. Kimball and Inmon Methodologies
 Relational (Inmon) vs Dimensional (Kimball)
 Relational Modeling:
 Entity-Relationship (ER) model
 Normalization rules
 Many tables using joins
 History tables, natural keys
 Good for indirect end-user access of data
 Dimensional Modeling:
 Facts and dimensions, star schema
 Less tables but have duplicate data (de-normalized)
 Easier for user to understand (but strange for IT people used to relational)
 Slowly changing dimensions, surrogate keys
 Good for direct end-user access of data

13. Relational Model vs Dimensional Model
Relational Model Dimensional Model
If you are a business user, which model is easier to use?

14. Kimball and Inmon Methodologies
• Kimball:
• Logical data warehouse (BUS), made up of subject areas (data marts)
• Business driven, users have active participation
• Decentralized data marts (not required to be a separate physical data store)
• Independent dimensional data marts optimized for reporting/analytics
• Integrated via Conformed Dimensions (provides consistency across data sources)
• 2-tier (data mart, cube), less ETL, no data duplication
• Inmon:
• Enterprise data model (CIF) that is a enterprise data warehouse (EDW)
• IT Driven, users have passive participation
• Centralized atomic normalized tables (off limit to end users)
• Later create dependent data marts that are separate physical subsets of data and can
be used for multiple purposes
• Integration via enterprise data model
• 3-tier (data warehouse, data mart, cube), duplication of data

15. Kimball Model
Why staging: Limit source contention (ELT), Recoverability, Backup, Auditing

16. Inmon Model

17. Reasons to add a Enterprise Data Warehouse
 Single version of the truth
 May make building dimensions easier using lightly denormalized tables in EDW instead of
going directly from the OLTP source
 Normalized EDW results in enterprise-wide consistency which makes it easier to spawn-off
the data marts at the expense of duplicated data
 Less daily ETL refresh and reconciliation if have many sources and many data marts in
multiple databases
 One place to control data (no duplication of effort and data)
 Reason not to: If have a few sources that need reporting quickly

18. Which model to use?
 Models are not that different, having become similar over the years, and can compliment
each other
 Boils down to Inmon creates a normalized DW before creating a dimensional data mart and
Kimball skips the normalized DW
 With tweaks to each model, they look very similar (adding a normalized EDW to Kimball,
dimensionally structured data marts to Inmon)
 Bottom line: Understand both approaches and pick parts from both for your situation – no
need to just choose just one approach
 BUT, no solution will be effective unless you possess soft skills (leadership, communication,
planning, and interpersonal relationships)

19. Hybrid Model
Advice: Use SQL Server Views to interface between each level in the model (default values, computations, rename fields, etc)
In the DW Bus Architecture, each data mart could be a schema (broken out by business process subject areas), all in one
database. Another option is to have each data mart in its own database with all databases on one server or spread among
multiple servers. Also, the staging areas, CIF, and DW Bus can all be on the same powerful server (MPP)

20. Kimball Methodology
From: Kimball’s The Microsoft Data Warehouse Toolkit
Kimball defines a development lifecycle, where Inmon is just about the data warehouse (not “how” used)

21. Data Vault
• Daniel Linstedt, author of Building Scalable Data Warehouse with Data Vault 2.0
• Not just a modeling technique, but entire methodology for DW projects
• Layer between Star Schema and Staging (use in combination with Kimball model, modeled
somewhere between the 3NF and star schema)
• Improves audit and historical tracking
• Hubs (business entities)
• Links (relationships)
• Satellites (descriptive attributes)
https://www.indellient.com/blog/data-vault-what-is-it-and-when-should-it-be-used/

22. Observation
Pattern
Theory
Hypothesis
What will
happen?
How can we
make it happen?
Predictive
Analytics
Prescriptive
Analytics
What
happened?
Why did
it happen?
Descriptive
Analytics
Diagnostic
Analytics
Confirmation
Theory
Hypothesis
Observation
Two Approaches to getting value out of data: Top-Down +
Bottoms-Up

23. Implement Data Warehouse
Physical Design
ETL
Development
Reporting &
Analytics
Development
Install and Tune
Reporting &
Analytics Design
Dimension Modelling
ETL Design
Setup Infrastructure
Understand
Corporate
Strategy
Data Warehousing Uses A Top-Down Approach
Data sources
Gather
Requirements
Business
Requirements
Technical
Requirements

24. The “data lake” Uses A Bottoms-Up Approach
Ingest all data
regardless of requirements
Store all data
in native format without
schema definition
Do analysis
Using analytic engines
like Hadoop
Interactive queries
Batch queries
Machine Learning
Data warehouse
Real-time analytics
Devices

25. Data Lake + Data Warehouse Better Together
Data sources
What happened?
Descriptive
Analytics
Diagnostic
Analytics
Why did it happen?
What will happen?
Predictive
Analytics
Prescriptive
Analytics
How can we make it happen?

26. Exactly what is a data lake?
A storage repository, usually Hadoop, that holds a vast amount of raw data in its native format until
it is needed.
Reasons to use a data lake:
• Inexpensively store unlimited data
• Centralized place for multiple subjects (single version of the truth)
• Collect all data “just in case” (data hoarding). The data lake is a good place for data that you “might” use down the road
• Easy integration of differently-structured data
• Store data with no modeling – “Schema on read”
• Complements enterprise data warehouse (EDW)
• Frees up expensive EDW resources for queries instead of using EDW resources for transformations (avoiding user contention)
• Wanting to use technologies/tools (i.e Databricks) to refine/filter data that do the refinement quicker/better than your EDW
• Quick user access to data for power users/data scientists (allowing for faster ROI)
• Data exploration to see if data valuable before writing ETL and schema for relational database, or use for one-time report
• Allows use of Hadoop tools such as ETL and extreme analytics
• Place to land IoT streaming data
• On-line archive or backup for data warehouse data
• With Hadoop/ADLS, high availability and disaster recovery built in
• It can ingest large files quickly and provide data redundancy
• ELT jobs on EDW are taking too long because of increasing data volumes and increasing rate of ingesting (velocity), so offload some of them to the Hadoop data lake
• Have a backup of the raw data in case you need to load it again due to an ETL error (and not have to go back to the source). You can keep a long history of raw data
• Allows for data to be used many times for different analytic needs and use cases
• Cost savings and faster transformations: storage tiers with lifecycle management; separation of storage and compute resources allowing multiple instances of different
sizes working with the same data simultaneously vs scaling data warehouse; low-cost storage for raw data saving space on the EDW
• Extreme performance for transformations by having multiple compute options each accessing different folders containing data
• The ability for an end-user or product to easily access the data from any location

27. Data Warehouse
Serving, Security & Compliance
• Business people
• Low latency
• Complex joins
• Interactive ad-hoc query
• High number of users
• Additional security
• Large support for tools
• Dashboards
• Easily create reports (Self-service BI)
• Know questions

28. Why Data Warehouses fail
• 70-80% corporate BI projects fail (Gartner)
• The executive who thinks data warehousing is “easy”
• Starting with an end date and working backwards
• Lack of user involvement
• Insufficient business requirements or too much
• Insufficient architect/design or too much
• Not enough planning for data governance
• Chose the wrong technology

29. Why Data Warehouses fail
• Poor communication between IT and the business
• Not setting up time to “validate the numbers”
• Performance issues (slow response times)
• Inexperience and lack of technical knowledge
• Modeling the data warehouse to reflect the source data rather than the
business

30. Why Data Warehouses fail
• The wrong consulting company is hired
• Lack of experience
• Using offshore developers
• Owning the whole project
• No mentoring or knowledge transfer
• Budget runs out
• Developers
• Testers
• DBA
• PM
• Cloud costs
• Software licenses
• Training

31. How DW project succeed
• Choosing the right technology
• Understand all aspects of data warehouse architectures
• Kimball vs Inmon
• Star Schema vs Relational
• Modern data warehouse vs data fabric vs data lakehouse vs data mesh
• Find a project champion/sponsor
• Have DW director who keeps everyone running at 80% efficiency
• Interative/Agile approach to delivery with frequent releases
• Break project into phases
• Start with a high-priority subject but one that is small and easy
• Get a quick win that shows lot’s of benefits and market it
• Build a solid foundation with a repeatable process
• Get people excited about DW thru training
• User Power BI to prototype/requirements

32. Modern Data Warehouse
Data Fabric adds: data access, data policies, data catalog, MDM, data virtualization,
data scientist tools, APIs, building blocks, products

33. Data Lakehouse

34. Data Mesh

35. Data Mesh
Credit to Zhamak Dehghani

36. Q & A ?
James Serra, Big Data Evangelist
Email me at: JamesSerra3@gmail.com
Follow me at: @JamesSerra
Link to me at: www.linkedin.com/in/JamesSerra
Visit my blog at: JamesSerra.com