This deck covers the journey of starting with BigQuery, adding more data sources and building a process around your data warehouse. It covers the three phases greenfield, dashboards and operational analytics and the necessary data components. The code for uploading your product feed can be found here: https://gist.github.com/ChrisGutknecht/fde93092e21039299ab76715596eac01 If you have any questions, reach out to me on Linkedin!

1. Building a Marketing Data
Warehouse from Scratch
Christopher Gutknecht | @chrisgutknecht | Bergzeit

2. 2. Dashboards
1. Greenfield
Our Plan: The Three Phases of a Data Platform
3. Operational Analytics

3. What You Will Take Away from this Session
1. When and why you should invest in a Marketing DWH
3. Interesting use cases by combining data sources
5. Design outcome-oriented questions for analytics projects
2. Learn the data ecosystem and the benefits of BigQuery
4. Many tactical tips for daily use

4. About Chris: Head of Acquisition & Optimization
Digital Marketer
Tech nerd
Climber
1997 2008 2013 2020
Dad of 2
Big Thanks to Steffi,
our Data Scientist

5. Bergzeit: Combining Love for Mountains & Data
Online Store for Mountain Gear
122 M Revenue in Financial Year 20/21
14 Countries & 5 Languages
Commerce. Content. Guided Tours

6. Let’s Set Clear Expectations for this Session
Technical deep dive
Intro to data ecosystem
What this session IS What it’s NOT
Machine learning
Customer data focused
Google Cloud focused
Practical tips & mistakes
Ecommerce use cases

7. Why Is Data Knowledge Important for You?
Behavioural data -> digital success
Operational analytics on the rise
Requirement Privacy by design

8. Most Desirable Digital Marketing Skills 20/21
source: marketingcharts.com

9. The
Components of a
Modern Data
Platform

10. Not so fast… Why All This Complexity?

11. Why Do I Need a Marketing Data Warehouse?
Connectors

12. If your Frustration Grows with Reporting Tools
Transformations
Volume
Operational Use Cases
Complexity
More Data Sources

13. Don’t Be this Guy - Know When To Scale Up

14. But Don’t Overengineer - Gradually Scale

15. Infrastructure Maintenance
Long upfront planning
Past: The Old On-Premise Data Veterans
Manual Scaling
Strict Dimensional Modeling

16. Today: The Cool Cloud Kids on the Block
100% cloud, no Ops
Seamless scaling
Instantly ready

17. The Components of a Modern Data Platform
Data Warehouse
Data Ingestion
Data Catalog & Governance
Activation
Data Quality
Job Orchestration
Visualization
Transformation

18. Analysts are Turning into Analytics Engineers
Data Warehouse
Data Ingestion
Data Catalog & Governance
Activation
Data Quality
Job Orchestration
Visualization
Transformation

19. Phase 1: Greenfield

20. Let’s Start from The Beginning: Data Sources
Data Ingestion
Google Ads Data Transfer
GA4 Export
Google Merchant Center
Paid Connectors, e.g. Fivetran
Custom Ingestion Scripts
Google Sheets
Cost Connectors, e.g. Funnel.io

21. 1. Navigate to Data transfers
Set up a Big Query Ads Transfer in One Minute
2. Configure the Transfer details

22. 2. Storage & Transfer
Get Your GA4 Data into BQ in Two Minutes
1. BQ Linking in Admin UI

23. Easily Connect Your Sheets as BigQuery Tables

24. Data Loader Vendors: Easy Setup, Instant Results

25. Suggestions for Interesting Data Sources
Domain Data Source Available in Data Loaders?
SEO Google Search Console
SEO Pagespeed Insights & Lighthouse
SEO Google Bot Logfiles
Ecom Inventory Data & Attributes
Ecom Trusted Shops Reviews
Ecom Awin Open Orders
Social Instagram
Social Facebook
.... ...

26. 2. Storage & Transfer
The Best Custom Way to Ingest Data into BQ
Data Source 3. BigQuery
1. Cloud Function
Data Transfer handles ingest job
Observability via alerts
Data Fetch with Python and Pandas

27. How Do We Batch-Ingest New Data?
Change Data Capture
Snapshots Copies
Full History Daily State
very easy
For <300k rows
rather easy
duplicate rows
storage efficient
complex architecture

28. Phase 2: Dashboards

29. We’ve Got Data: Show Me Shiny Reports!
Data Warehouse
Data Ingestion Visualization

30. How do Price Discounts affect Sales? Sources:

31. Price Discounts Data Model Sources:
date
sku
detail_views
product order value
ga sessions
date
sku
price
sale_price
diff_price_to_sale
diff_price_to_sale_grouped
products

32. How do Season Categories Affect Sales?

33. Season Category Data Model Sources:
date
sku
detail_views
product order value
ga sessions
date
sku
season_category
products

34. Category Revenue Share without Ratings

35. Category Rev & Ratings Data Model Sources:
date
sku
top_category
product order value
ga sessions
date
sku
rating_count
rating_count_grouped
products ratings

36. Which Category Has More Selection Orders?

37. Selection Orders Data Model Sources:
date
transaction_id
sub_category
is_multi_same_category
is_multi_same_color
is_multi_same_size
ga sessions

38. Wait… Don’t You Need to Know SQL for This?

39. You Need to Learn SQL: Take a BigQuery Course
Self-paced
45€ / User / Month
Takes 4-8 weeks
Certificate

40. WAIT! Before you build 100s of Dashboards...
Avoid code repetition
Style Guidelines
Test Coverage
Apply DEV Best Practices
SQL Version Control

41. Warning Signs You’re Doing Analytics Wrong

42. Don’t Create Datasets in the US, use EU
Always keep in EU
Can’t join with EU datasets

43. Avoid Saved Queries - They Don’t Scale

44. This Report is Broken. Who’s the Owner?
No Idea,
Help Yourself!

45. Don’t Use Custom Queries as Data Sources

46. Each Analyst uses a different SQL Code & Naming

47. Instead, Apply “Data Product” Thinking
Data Product Owner
Scrum Process
Data SLAs
Treat Data as a Product

48. #1 Pick And Implement A SQL Styleguide
https://github.com/mattm/sql-style-guide

49. #2 Define A Dataset & Table Naming Convention
Fields
Domains
Datasets
Tables
product
seo
seo_google_search_console
query_by_page_daily
ga_product_order_value
page

50. #3 Use dbt for all SQL Transformations

51. Build Your Data Product like a German House

52. Phase 3: Operational Analytics in Production

53. You Need Clean Data for Operational Analytics

54. What is Operational Analytics for Bergzeit?
ML
Products
Profit Bidding
Rule-based
Products
Data Uploads Attribution Model
Updating Affiliate Sales

55. Case: Upload Your Shopping Feed Every 10 Min
2. Cloud Function
with 15 lines of code
3. Schedule
Cronjob
1. Get GCS Bucket
Name
Code samples: https://gist.github.com/ChrisGutknecht/fde93092e21039299ab76715596eac01

56. Case: Profit Bidding & Report
More Details: https://www.slideshare.net/ChristopherGutknecht/gross-profit-bidding-for-ecommerce-smx-virtual-2021

57. Operational Data Errors Can be Really Costly

58. We Need To Prepare for Data Pipeline Errors
2. Execution
1. Source Data 3. Target Table
Test Coverage Test Coverage
Retry Policies

59. How Can We Solve Our Missing Data Problem?

60. Solution: Define a Table Freshness Alert with dbt

61. Define All Data Quality Tests in dbt
Non Null Values
Restricted Values
Uniqueness
Data Freshness

62. Simple: Cloud Scheduler for Retry Execution

63. Advanced: Use Airflow For Data Task Graphs
Retries on Every Task
Alerting & Monitoring
Data Tasks in DAGs
Snaspshots & Backfills

64. Let’s Finish With a Strategic View From the Peak

65. How Can We Generate Value? Focus on Actions
1. Define Actions 3. Factors
2. Success Metrics
What Will You
Do Differently If
You Have the
Data?
What Would
Success in
Metrics Look
Like?
Which Factors
Influence
Success?
4. Tests
How Can We
Test Actions on
These Factors?

66. Who Should Be Your Data Hire?
1. Focus:
SQL & Warehouse
3. Focus: ML Models
2. Focus: Data Pipelines

67. Your Takeaways from this Session
1. When and why you should invest in a Marketing DWH
3. How to explore use cases by combining data sources
5. Design outcome-oriented questions for analytics projects
2. Learn the data ecosystem and the benefits of BigQuery
4. Many tactical tips for daily use

68. Thanks for Your Time.
Looking Forward To Questions!
Chris Gutknecht | Teamlead A&O | Hiring a PPC!

69. 2. Dashboards
1. Greenfield
ANNEX: The Three Phases of a Data Platform
3. Operational Analytics

70. Data Warehouse vs Data Lake
Structured Data
Table Schemas
Transactions
Sharded Files
Unstructured Data
Lakehouse

71. Why Focus On Google Cloud & Big Query?
Market Leader in Data Analytics*
Free Google Data Connectors
Seamless low-tech scaling
source:: Forrester Research 2021

72. The Best Cloud Data Warehouse? It Depends
Source: https://medium.com/pocket-gems/a-comparative-analysis-between-bigquery-redshift-and-snowflake-8d194fdf5693
Google Data Sources BigQuery = Google Cloud

73. How often do we Ingest Data?
Real-Time
Stream Processing
Batch Processing
or

74. Connected Sheets = ‘NoCode’ Analysis on BQ

75. Dom Woodman’s Search Console Downloader
https://www.pipedout.com/resources/tools/download-search-console/

76. Data Modeling: Star Schema & 3rd Normal Form
Third Normal
Form

77. Data Modeling Choices: Denormalized
Third Normal
Form
Denormalized

78. 2. Dashboards
1. Greenfield
ANNEX: The Three Phases of a Data Platform
3. Operational Analytics

79. Or Pick the Official “Data Analytics” Certificate
8 Courses
4-6 Months (longer)
Intro to R
Certificate

80. Export Your Your GPC Costs to BigQuery

81. Monitor Your BigQuery Costs in a Dashboard
Monitor Expensive Queries: https://www.pascallandau.com/bigquery-snippets/monitor-query-costs/

82. 2. Dashboards
1. Greenfield
ANNEX: The Three Phases of a Data Platform
3. Operational Analytics

83. How To Sync Segments: CDP vs Reverse ETL?
Customer Data Platform
Reverse ETL
(DWH = Central)