3D: DBT using Databricks and Delta
•
3 j'aime•2,286 vues
Data Build Tool (DBT) is an open source technology to set up your data lake using best practices from software engineering. This SQL first technology is a great marriage between Databricks and Delta. This allows you to maintain high quality data and documentation during the entire datalake life-cycle. In this talk I’ll do an introduction into DBT, and show how we can leverage Databricks to do the actual heavy lifting. Next, I’ll present how DBT supports Delta to enable upserting using SQL. Finally, we show how we integrate DBT+Databricks into the Azure cloud. Finally we show how we emit the pipeline metrics to Azure monitor to make sure that you have observability over your pipeline.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à 3D: DBT using Databricks and Delta
Similaire à 3D: DBT using Databricks and Delta (20)
Plus de Databricks
Plus de Databricks (20)
Dernier
Dernier (20)
3D: DBT using Databricks and Delta
- 1. 3D: DBT, Databricks and Delta Fokko Driesprong Principal Code Connaisseur
- 2. whoami ▪ Fokko Driesprong ▪ Master Distributed Systems & Software Engineering ▪ Code Connaisseur at GoDataDriven ▪ Mostly doing {data,software} engineering ▪ Open source enthousiast ▪ ASF Member ▪ Committer + PMC member on Apache {Airflow, Avro, Druid} ▪ Committer on Apache Parquet
- 3. GoDataDriven ▪ Amsterdam based consultancy ▪ Data {Engineer,Science,Strategy} ▪ And now also analytics engineering! ▪ Around 50 consultants ▪ Used to do Hadoop, now Cloud
- 4. Agenda What is DBT? DBT + Delta Lake DBT + Azure Databricks
- 5. What’s DBT? And why I ❤ it so much
- 6. Data Build Tool ▪ Tool for building data pipelines following the DataOps principles ▪ Simple tool to build complex pipelines ▪ Best practices from software engineering ▪ Linting / Peer reviews / DRY principle / Data testing ▪ SQL First ▪ Encodes organisational knowledge into the pipeline Try it yourself: https://godatadriven.com/blog/tutorial-for-dbt-analytics-engineering-made-easy/ But everyone just calls it DBT
- 7. Data Build Tool ▪ Main focus on integration with DWH platforms ▪ Postgres, Redshift, Snowflake and Bigquery ▪ Support for Spark / Databricks ▪ Created by Fishtown Analytics ▪ Huge open source community ▪ Apache 2.0 Open Source license But everyone just calls it DBT
- 8. DBT The T in Extract-Transform-Load (ELT) Analysts using dbt can transform their data by simply writing select statements, while dbt handles turning these statements into tables and views in a data warehouse.
- 9. Let’s go through at a small example Combine orders and order_lines into a revenue table
- 10. SQL with some Ninja2 sauce Revenue table
- 11. DBT as a SQL Runner Executes the pipeline from the command line
- 12. Seamless integration with the Databricks Metastore ▪ Requires a Hive Metastore ▪ Analyize table
- 13. DBT as a SQL Compiler Compiled SQL: target/compiled/dbtpreprocessing/models/revenue.sql
- 14. Next to the SQL there is documentation Give meaning to the columns and add constraints
- 15. Looking at the docs dbt docs generate dbt docs serve ▪ Columns including types ▪ Test constraints ▪ Statistics ▪ The compiled query
- 16. Testing dbt test ▪ Not-null ▪ Uniqueness ▪ Accepted values ▪ Referential constraints ▪ Custom tests
- 17. How does DBT communicate with Spark? ▪ SQL Over HTTP ▪ Authenticate using the token ▪ Parallel execution
- 18. DBT with Delta Lake
- 19. Switch to incremental ingestion Using the Delta format ▪ ACID dataformat by Databricks ▪ Linux Software Foundation ▪ Allows MERGE INTO ▪ Enabled incremental imports
- 20. Switch to incremental Delta If the table doesn’t exists (yet)
- 21. Switch to incremental Delta Incremental MERGE INTO if the table exists
- 23. In practice Incremental imports ▪ Watermark column ▪ Only load the changed orders ▪ Also interesting for Users table
- 24. DBT Macro’s Running it a second time ▪ Don’t Repeat Yourself ▪ Write a Macro instead
- 25. DBT with Azure Databricks
- 26. Observability is king Keeping track of your pipelines ▪ Building trust ▪ Track aggregated metrics computed by Spark ▪ Application insights ▪ Centralized system
- 27. Very simple Hive UDF Keep track of stats over time
- 28. Small snippet of Scala Sends the metrics to Application Insights
- 29. Use the UDF in DBT Sends the metrics to Application Insights ▪ Register the UDF ▪ Keep track of ▪ Seconds since last order ▪ Number of orders
- 30. Be proactive Before there are angry managers at your desk ▪ Keep track of the metric ▪ Send alerts on business rules ▪ Outlier based on historical distributions
- 31. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions. ▪ Code available at: ▪ https://github.com/godatadriven/dbt-data-ai-summit ▪ https://github.com/godatadriven/azure-dbt-logger
- 33. Code example
- 34. Two Columns ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format ▪ Bulleted list format Headline FormatHeadline Format
- 35. Attribution Format Second line of attribution This is a template for a quote slide. This is where the quote goes. Attribute the source below…
- 36. Databricks simplifies data and AI so data teams can innovate faster
- 37. Logos
- 38. Databricks Logos