Azure Data Factory now allows users to perform data wrangling tasks through Power Query activities, translating M scripts into ADF data flow scripts executed on Apache Spark. This enables code-free data exploration, preparation, and operationalization of Power Query workflows within ADF pipelines. Examples of use cases include data engineers building ETL processes or analysts operationalizing existing queries to prepare data for modeling, with the goal of providing a data-first approach to building data flows and pipelines in ADF.

1. Azure Data Factory: Data Wrangling
Power Query in ADF
Updated Public Preview Q1 CY21

2. What is Data Wrangling?
 Code-free data exploration and data prep
 Operationalize Power Query as an activity by
translating M script in ADF data flow script
 Execute Power Query as a pipeline activity using the
ADF data flow serverless, scaled-out, ADF-managed
Apache Spark engine
 Essentially acts as a data-first entry point to building
ADF data flows

3. ADF Data Wrangling Use Cases
 Data Engineer is building an ETL process in ADF uses PQ to explore data using data profiling
 Business Analyst is a PQ desktop user and wishes to operationalize their M query in a data
pipeline that sinks data in the Lake
 Data Engineer needs to prep data for modeling and ETL by using a data-first approach. Creates a
PQ wrangling activity and adds it to pipeline.
 Trimming strings
 Data type conversions
 Rename columns
 Remove columns
 Value prop
 “Data Wrangling in ADF”, not “Power Query lift-and-shift”

4. ADF Data Wrangling Roadmap – PQ Activity
 Continue to add more M functions to fold into Spark
 Add more native connectors that work in both ADF & Power Query
 Enable V-Net in Power Query Online data wrangling experience in ADF
 Launch PQ activity in Synapse Pipelines
 Enable interactive monitoring similar to Copy and Data Flow

5. Additional
resources
Documentation
List of tutorial videos
Expression language reference
Performance guide
ADF twitter
ADF tech community blog