Creating Visual Transformations in Azure Data Factory (dataMinds Connect)

•

3 likes•1,171 views

This document discusses visual data transformations in Azure Data Factory. It introduces Mapping Data Flows, which allow transforming data at scale visually without code. It explores the capabilities of Mapping Data Flows, the various transformations available, using expressions, debugging transformations, and handling schema drift. The document includes a demonstration of transforming sample data using Mapping Data Flows.

Data & Analytics

Creating Visual Transformations in Azure Data Factory@cathrinew
Azure Data Factory v2 came with many new capabilities and
improvements. One of biggest game-changers is the Mapping
Data Flows feature, allowing you to transform data at scale -
without having to write a single line of code!
In this session, we will first go through the capabilities and
use cases for Mapping Data Flows. Then, we will explore the
various transformations available, as well as the expression
language and how to use the visual expression builder.
Finally, we will look at how to debug, monitor, and optimize
our data transformations.
Cathrine Wilhelmsen

Visual Transformations
in Azure Data Factory
Cathrine Wilhelmsen
dataMinds Connect · Oct 8, 2019

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Data Warehousing Business Intelligence
Artificial Intelligence
Big Data and Analytics
Machine Learning
Data Science

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What?
When?
Why?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Collect
Store
Transform
Integrate
Prepare

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Azure Data Factory?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What can you do in Azure Data Factory?
Copy Data Transform Data

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Mapping Data Flows?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What do Mapping Data Flows look like?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Why use Mapping Data Flows?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
When to use Mapping Data Flows?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What are Data Flow Transformations?
18

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Source
1 or more sources per data flow

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Sink
1 or more sinks per data flow

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Branch

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Join

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Conditional Split

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Exists

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Union

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Lookup

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Derived Column

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Select

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Aggregate

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Surrogate Key

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Pivot

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Unpivot

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Window

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Filter

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Sort

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Alter Row

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Kamil Nowinski's ADF Cheat Sheet
github.com/SQLPlayer/CheatSheets/blob/master/ADFDF-Cheat-Sheet-sqlplayer.pdf

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Expression Language

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Expression Language: Two Flavors
||
&&
==
>
>=
<
<=
!=

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Visual Expression Builder

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Debug Mode?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you execute a Data Flow?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Schema Drift?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How do you handle Schema Drift?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Column Pattern Matching?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Column Pattern Matching work?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
What is Rule-Based Mapping?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How does Rule-Based Mapping work?

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
How much do Mapping Data Flows cost?
Compute Optimized: $0.075 per vCore-hour
General Purpose: $0.109 per vCore-hour
Memory Optimized: $0.148 per vCore-hour

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Advanced Analytics on Big Data
https://azure.microsoft.com/en-us/solutions/architecture/advanced-analytics-on-big-data/

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Modern Data Warehouse
https://azure.microsoft.com/en-us/solutions/architecture/modern-data-warehouse/

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Ingest
Azure
Data Factory
Serve
Azure SQL Data
Warehouse
Visualize
Power BI
Store
Azure Data Lake Storage Gen2
Data Pipeline Orchestration and Monitoring
Azure Data Factory
Sources
Cloud
SaaS
Prepare
Wrangling
Data Flows
Transform
Mapping
Data Flows
On-Premises

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Resources:
aka.ms/dflinks

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Videos:
aka.ms/adfvideos

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Mark Kromer on Twitter:
@KromerBigData @mssqldude

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
Announcement Blog Post
https://azure.microsoft.com
/en-us/blog/azure-data-
factory-mapping-data-
flows-are-now-generally-
available/

© 2019 Cathrine Wilhelmsen (hi@cathrinew.net)
@cathrinew
cathrinew.net
hi@cathrinew.net
thank you!

What's hot

Azure Data Factory presentation with links

Chris Testa-O'Neill

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Cathrine Wilhelmsen

Next Generation Data Integration with Azure Data Factory

Tom Kerkhove

Azure data factory

BizTalk360

Azure Data Factory Data Flows Training (Sept 2020 Update)

Mark Kromer

Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)

Cathrine Wilhelmsen

Comcast is one of the leading providers of communications, entertainment, and cable products and services. At the heart of it is Comcast RDK providing the backbone of telemetry to the industry. RDK (Reference Design Kit) is pre-bundled opensource firmware for a complete home platform covering video, broadband and IoT devices. RDK team at Comcast analyzes petabytes of data, collected every 15 minutes from 70 million devices (video and broadband and IoT devices) installed in customer homes. They run ETL and aggregation pipelines and publish analytical dashboards on a daily basis to reduce customer calls and firmware rollout. The analysis is also used to calculate WIFI happiness index which is a critical KPI for Comcast customer experience. In addition to this, RDK team also does release tracking by analyzing the RDK firmware quality. SQL Analytics allows customers to operate a lakehouse architecture that provides data warehousing performance at data lake economics for up to 4x better price/performance for SQL workloads than traditional cloud data warehouses. We present the results of the “Test and Learn” with SQL Analytics and the delta engine that we worked in partnership with the Databricks team. We present a quick demo introducing the SQL native interface, the challenges we faced with migration, The results of the execution and our journey of productionizing this at scale.

SQL Analytics Powering Telemetry Analysis at Comcast

Databricks

Azure Data Factory

HARIHARAN R

It can be quite challenging keeping up with the frequent updates to the Microsoft products and understanding all their use cases and how all the products fit together. In this session we will differentiate the use cases for each of the Microsoft services, explaining and demonstrating what is good and what isn't, in order for you to position, design and deliver the proper adoption use cases for each with your customers. We will cover a wide range of products such as Databricks, SQL Data Warehouse, HDInsight, Azure Data Lake Analytics, Azure Data Lake Store, Blob storage, and AAS as well as high-level concepts such as when to use a data lake. We will also review the most common reference architectures (“patterns”) witnessed in customer adoption.

Differentiate Big Data vs Data Warehouse use cases for a cloud solution

James Serra

Azure data factory

David Giard

At wetter.com we build analytical B2B data products and heavily use Spark and AWS technologies for data processing and analytics. I explain why we moved from AWS EMR to Databricks and Delta and share our experiences from different angles like architecture, application logic and user experience. We will look how security, cluster configuration, resource consumption and workflow changed by using Databricks clusters as well as how using Delta tables simplified our application logic and data operations.

Moving to Databricks & Delta

Databricks

Azure Data Factory v2

Sergio Zenatti Filho

It’s very easy to be distracted by the latest and greatest approaches with technology, but sometimes there’s a reason old approaches stand the test of time. Star Schemas & Kimball is one of those things that isn’t going anywhere, but as we move towards the “Data Lakehouse” paradigm – how appropriate is this modelling technique, and how can we harness the Delta Engine & Spark 3.0 to maximise it’s performance?

Achieving Lakehouse Models with Spark 3.0

Databricks

Azure Databricks - An Introduction (by Kris Bock)

Daniel Toomey

Data Quality Patterns in the Cloud with Azure Data Factory

Mark Kromer

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing Startup (Presented at SQLBits on March 11th, 2022) What happens when you mix one rapidly-changing startup, one data analyst, one data engineer, and one hypothesis that Azure Synapse Analytics could be the right tool of choice for gaining business insights? We had no idea, but we gave it a go! Our ambition was to think big, start small, and act fast – to deliver business value early and often. Did we succeed? Join us for an honest conversation about why we decided to implement Azure Synapse Analytics alongside Power BI, how we got started, which areas we completely messed up at first, what our current solution looks like, the lessons learned along the way, and the things we would have done differently if we could start all over again.

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...

Cathrine Wilhelmsen

Azure data factory security

MikeBrassil1

Azure Synapse Analytics is Azure SQL Data Warehouse evolved: a limitless analytics service, that brings together enterprise data warehousing and Big Data analytics into a single service. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources, at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. This is a huge deck with lots of screenshots so you can see exactly how it works.

Azure Synapse Analytics Overview (r2)

James Serra

Azure Data Factory Introduction.pdf

MaheshPandit16

Data Con LA 2020 Description In this session, I introduce the Amazon Redshift lake house architecture which enables you to query data across your data warehouse, data lake, and operational databases to gain faster and deeper insights. With a lake house architecture, you can store data in open file formats in your Amazon S3 data lake. Speaker Antje Barth, Amazon Web Services, Sr. Developer Advocate, AI and Machine Learning

Owning Your Own (Data) Lake House

Data Con LA

What's hot (20)

Azure Data Factory presentation with links

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Next Generation Data Integration with Azure Data Factory

Azure data factory

Azure Data Factory Data Flows Training (Sept 2020 Update)

Building Dynamic Pipelines in Azure Data Factory (SQLSaturday Oslo)

SQL Analytics Powering Telemetry Analysis at Comcast

Azure Data Factory

Differentiate Big Data vs Data Warehouse use cases for a cloud solution

Azure data factory

Moving to Databricks & Delta

Azure Data Factory v2

Achieving Lakehouse Models with Spark 3.0

Azure Databricks - An Introduction (by Kris Bock)

Data Quality Patterns in the Cloud with Azure Data Factory

Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...

Azure data factory security

Azure Synapse Analytics Overview (r2)

Azure Data Factory Introduction.pdf

Owning Your Own (Data) Lake House

Similar to Creating Visual Transformations in Azure Data Factory (dataMinds Connect)

Azure Data Factory for the SSIS Developer (SentryOne Webinar)

Cathrine Wilhelmsen

Understanding Azure Data Factory: The What, When, and Why (NIC 2020)

Cathrine Wilhelmsen

Pipelines and Packages: Introduction to Azure Data Factory (24HOP)

Cathrine Wilhelmsen

Pipelines and Data Flows: Introduction to Data Integration in Azure Synapse A...

Cathrine Wilhelmsen

Choosing Between Microsoft Fabric, Azure Synapse Analytics and Azure Data Fac...

Cathrine Wilhelmsen

Biml for Beginners: Script and Automate SSIS development (SQLSaturday Finland)

Cathrine Wilhelmsen

Biml for Beginners: Script and Automate SSIS development (SQLSaturday Oslo)

Cathrine Wilhelmsen

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLGrillen 2018)

Cathrine Wilhelmsen

Biml for Beginners: Script and Automate SSIS development (Hybrid VC)

Cathrine Wilhelmsen

Whether you are moving a small application or entire datacenters, migrating to the cloud can be a complex process. In this session, we will share some of the common challenges that our customers face on their journey to the cloud and discuss how these challenges can be overcome. We will outline the patterns of success that we have observed from partnering with hundreds of customers on their large-scale migrations as well as highlight the mechanisms we have created to help our customers migrate faster. About the Event: AWS Transformation Day is designed for enterprise organizations migrating to the cloud to become more responsive, agile and innovative, while staying secure and compliant. Join us for this one-day event and we’ll share our experiences of helping enterprise customers accelerate the pace of migration and adoption of strategic services. Who should attend? This event is recommended for IT and business leaders who are looking to create sustainable benefits and a competitive advantage by using the AWS Cloud. CIOs, CTOs, CISOs, CDOs, CFOs, IT leaders and IT professionals, enterprise developers, business decision makers, and finance executives.

An Overview of Best Practices for Large Scale Migrations - AWS Transformation...

Amazon Web Services

Mass Migrations to AWS

Amazon Web Services

Large Scale Migrations - Transformation Day Montreal 2018

Amazon Web Services

企業雲端化之旅

Amazon Web Services

Overview of AWS Cloud Migration 企業雲端化之旅 Level: 100/200, 中文演講講師: Xia Zhang, Principal Evangelist, AWS GCR 企業客戶的雲端遷移實務綜觀。 1. 利用雲端服務優勢，能帶給企業哪些營運成本、人力及物力的精省 2. 如何將 Oracle、 SQL Server、與VMware 上的企業資料轉移至雲端，取得更完善強大產能 3. 如何部屬以 Container 為基礎的混合雲架構 4. 以及雲端資安實務分享聯繫銷售: https://aws.amazon.com/tw/contact-us/ 與銷售線上聊天: https://pages.awscloud.com/tw-hkt-sales-chat.html

企業雲端化之旅

Amazon Web Services

Speakers: Alessandro Negro, Chief Scientist, GraphAware Abstract: In recent years, knowledge graphs are gaining significant traction across industries. This kind of graph excels uniquely at providing in-depth contextual knowledge so needed by organizations to successfully differentiate and compete today- particularly when leveraging unstructured data using NLP/U and Machine Learning on the graph. But where is the knowledge that is created from the combined sources? And how can we start surfacing new, high value insights that we could not obtain before? An example of how this can work is Hume, a graph-powered insights engine. It uses an innovative approach, creating what we call a Collaborative Knowledge Graph (CKG). In the CKG, Hume connects, enriches, and transforms data from structured and especially unstructured sources in order to liberate knowledge, discover hidden insights, and unlock new opportunities difficult or impossible to detect before. By ingesting and analyzing the details of your domain and the nuances of your most complex business challenges, and by enriching the CKG with external knowledge sources such as Wikidata, private knowledge bases and more, Hume shows one way to deliver on the value and potential of knowledge graphs.

Using Knowledge Graphs to Predict Customer Needs, Improve Product Quality an...

Neo4j

Cloud migration is more than simply a business efficiency or a cost-saving measure. It’s a critical step towards digital transformation, innovation and operational resilience that has opened up opportunities for those who’ve embraced cloud adoption. Whether you are looking to embark on your cloud migration or scaling the number of applications you’re moving to the cloud, it does not need to be a daunting task or one that you go at alone. AWS offers 10 years of experience helping businesses to efficiently move their legacy on-premises systems to the cloud. We work closely alongside numerous local delivery partners to help you meet your business needs. Our Cloud Migration insights forum helps you to learn how to simplify your cloud journey with AWS.

AWS Cloud Migration Insights Forum

Amazon Web Services

Best Practices for Large Scale Migrations - AWS Transformation Day Boston 2018

Amazon Web Services

Large-Scale Migration: Best Practices - ENT210 - Chicago AWS Summit

Amazon Web Services

Using Knowledge Graphs to Predict Customer Needs and Improve Quality

Neo4j

Biml Tips and Tricks: Not Just for SSIS Packages! (SQLBits 2019)

Cathrine Wilhelmsen

Similar to Creating Visual Transformations in Azure Data Factory (dataMinds Connect) (20)