SlideShare a Scribd company logo
1 of 36
Analysing StackExchange data
with Azure Data Lake
Analysing StackExchange data with Azure Data Lake
Nice to meet you
Tom KERKHOVE
➔ Integration Professional
➔ IoT Competency Lead
➔ Windows Development &
Microsoft Azure MVP
tom.kerkhove@codit.eu
+32 473 701 074
@TomKerkhove
be.linkedin.com/in/tomkerkhove
github.com/tomkerkhove
Agenda
• Why should we care about Big Data?
• Big Data in Azure
• Azure Data Lake
• Demo
• Q & A
3
Integration of ThingsInternet of Things
5
Connect and scale
with efficiency
Analyze and act
on new data
Integrate and transform
business processes
Event producers & gateways Ingestion & transformation Report, Act, Predict
Microsoft Patterns & Practices – IoT Journey
9
10
Cluster Management
11
Languages
Platform Services
Infrastructure Services
Web Apps
Mobile
Apps
API
Management
API Apps
Logic Apps
Notification
Hubs
Content
Delivery
Network (CDN)
Media
Services
BizTalk
Services
Hybrid
Connections
Service Bus
Storage
Queues
Hybrid
Operations
Backup
StorSimple
Azure Site
Recovery
Import/Export
SQL
Database
DocumentDB
Redis
Cache
Azure
Search
Storage
Tables
Data
Warehouse Azure AD
Health Monitoring
AD Privileged
Identity
Management
Operational
Analytics
Cloud
Services
Batch
RemoteApp
Service
Fabric
Visual Studio
App
Insights
Azure
SDK
VS Online
Domain Services
HDInsight Machine
Learning
Stream
Analytics
Data
Factory
Event
Hubs
Mobile
Engagement
Data
Lake
IoT Hub
Data
Catalog
Security &
Management
Azure Active
Directory
Multi-Factor
Authentication
Automation
Portal
Key Vault
Store/
Marketplace
VM Image Gallery
& VM Depot
Azure AD
B2C
Scheduler
Overview in Azure
13
DocumentDB
Data Factory Stream Analytics Data Lake HDInsight Data Lake
(Store & Analytics)
Virtual Machine
IoT Hub SQL Data
Warehouse
SQL DatabaseStorageEvent Hubs
Document Db
Data Ingestion Data Storage
Data Pipelines
Machine Learning
Data Analytics
Cortana Analytics Suite
15
Analysing Big Data in Azure
Azure Data Lake Family
HDInsight Data Lake Store Data Lake Analytics
• Unlimited storage
• WebHDFS Store
• Managed cluster service
• Open-source technology
• Runs on Windows or Linux
• Managed job service
• U-SQL batch-processing
Azure Data Lake Store
➔ WebHDFS compatible
➔ Any size
➔ Any format as-is
➔ Write-once-read-many
➔ Enterprise-grade security
➔ Thé big data store in Azure
17
Characteristics
➔ Data Warehousing
➔ Structured data
➔ Defined set of schemas
➔ Requires Extract-Transform-
Load (ETL) before storing
➔ Known for some of us
➔ Exploratory analysis is hard
because of transforming the
data
18
Data Lake vs DataWarehousing
➔ Data Lake
➔ Raw data
(unstructured/semi-structured/structured)
➔ “Dump” all your data in the
lake
➔ Data scientists will
interpret data from the lake
➔ Without metadata, turns in
a data swamp pretty fast
19Martin Fowler on Data Lake & Data Warehouses(link)
Azure Data Lake Analytics
➔ Run analytics jobs on managed clusters
➔ Don’t worry about scale
➔ Written in U-SQL
➔ SQL Syntax
➔ Extensibility in C#
➔ Easily scaled with Analytics Units
➔ Pay for processing time only
20
Writing U-SQL scripts
21
Extract from data source by
using built-in or custom
extractors.
Transform / Analyse the data
using SQL-syntax, in-line C# or
C# method calls
Output the result to a data
source by using built-in or
custom extractors
22
Data Lake Analytics - Data Sources
U-SQL
Query Query
Azure
Storage Blobs
Azure
Data Lake Store
Azure
SQL Database
Azure
SQL Data Warehouse
Azure SQL
in VMs
Azure Data Lake Analytics
24
Meet StackExchange
➔ Over 280 subwebsites
➔ 150+ GB of open-source data
➔ Different kinds of data
➔ Posts
➔ Users
➔ Votes
➔ ...
➔ A big data sample data set
What AreWe GoingTo Do?
• Downloading the
original data set
Acquiring The
Data
• Upload data set to
Azure
• Determine what
service to use
Moving The
Data • Merging data from
each site into one
file
• Conversion from
XML to CSV
Aggregating
The Data
• Run business logic
on it
• Attempt to gain
knowledge from it
Analyzing The
Data • Visualize what we’ve
learned
Visualizing The
Data
26
Azure Data Lake tools forVisual Studio
➔ Projects / Solutions / Source control
➔ Store Explorer
➔ Browse store
➔ Download complete / subset of file
➔ Preview
➔ JobVisualizer
➔ Determine bottlenecks by using heatmaps
➔ Playback jobs based on telemetry
➔ Query optimization
➔ Job Profiler
➔ Off-Line execution
27
Integration with Azure Services
➔ Integrate in your data pipelines in Azure Data Factory
➔ Move data from Azure Data Lake Store to other store
➔ Move data to Azure Data Lake Store
➔ Run U-SQL query within pipeline
➔ Integration with Azure Data Catalog
➔ Register your Azure Data Lake Store assets
28
Pricing
➔ Data Lake Store
➔ $0,08/GB stored per month
➔ $0,14 per 1M transactions
• 1 transaction is block of up to 128 kB
➔ Egress will be billed but not know yet
➔ Data Lake Analytics
➔ $0,05 per job
➔ $0,05 per minute per Analytics Unit for processing time
29
Azure Data Lake Store vs Blob Storage
30
No Limitations
Store whatever you
want in any format
Security
Built-in Azure Active
Directory support
Pricing
More expensive than
Storage RA-GRS
Redundancy
It’s there but no control
over it
Built for Scale
Optimized for high-
scale reads
Integration
With Data Factory, Data
Catalog & HDInsight
31
Summary
➔ Big Data is not just a hype so get ready
➔ Azure Data Lake Store
➔ Analyse today & explore tomorrow
➔ Data Swamps
➔ Data Lake Analytics
➔ No cluster management
➔ Re-use existing skills
➔ Pay for what we use
➔ Big Data in Azure? Azure Data Lake family and it’s easy!
34
35
36

More Related Content

What's hot

What's hot (20)

Part 3 - Modern Data Warehouse with Azure Synapse
Part 3 - Modern Data Warehouse with Azure SynapsePart 3 - Modern Data Warehouse with Azure Synapse
Part 3 - Modern Data Warehouse with Azure Synapse
 
Azure Data Factory for Azure Data Week
Azure Data Factory for Azure Data WeekAzure Data Factory for Azure Data Week
Azure Data Factory for Azure Data Week
 
Azure data lake sql konf 2016
Azure data lake   sql konf 2016Azure data lake   sql konf 2016
Azure data lake sql konf 2016
 
Azure Data Factory V2; The Data Flows
Azure Data Factory V2; The Data FlowsAzure Data Factory V2; The Data Flows
Azure Data Factory V2; The Data Flows
 
Unleash the power of Azure Data Factory
Unleash the power of Azure Data Factory Unleash the power of Azure Data Factory
Unleash the power of Azure Data Factory
 
Running cost effective big data workloads with Azure Synapse and ADLS (MS Ign...
Running cost effective big data workloads with Azure Synapse and ADLS (MS Ign...Running cost effective big data workloads with Azure Synapse and ADLS (MS Ign...
Running cost effective big data workloads with Azure Synapse and ADLS (MS Ign...
 
Microsoft Azure BI Solutions in the Cloud
Microsoft Azure BI Solutions in the CloudMicrosoft Azure BI Solutions in the Cloud
Microsoft Azure BI Solutions in the Cloud
 
Building Data Lakes with Apache Airflow
Building Data Lakes with Apache AirflowBuilding Data Lakes with Apache Airflow
Building Data Lakes with Apache Airflow
 
Modern data warehouse
Modern data warehouseModern data warehouse
Modern data warehouse
 
Azure Data Lake and Azure Data Lake Analytics
Azure Data Lake and Azure Data Lake AnalyticsAzure Data Lake and Azure Data Lake Analytics
Azure Data Lake and Azure Data Lake Analytics
 
Digital Transformation with Microsoft Azure
Digital Transformation with Microsoft AzureDigital Transformation with Microsoft Azure
Digital Transformation with Microsoft Azure
 
Azure Data Factory v2
Azure Data Factory v2Azure Data Factory v2
Azure Data Factory v2
 
ETL in the Cloud With Microsoft Azure
ETL in the Cloud With Microsoft AzureETL in the Cloud With Microsoft Azure
ETL in the Cloud With Microsoft Azure
 
An intro to Azure Data Lake
An intro to Azure Data LakeAn intro to Azure Data Lake
An intro to Azure Data Lake
 
Develop scalable analytical solutions with Azure Data Factory & Azure SQL Dat...
Develop scalable analytical solutions with Azure Data Factory & Azure SQL Dat...Develop scalable analytical solutions with Azure Data Factory & Azure SQL Dat...
Develop scalable analytical solutions with Azure Data Factory & Azure SQL Dat...
 
Azure Databricks—Apache Spark as a Service with Sascha Dittmann
Azure Databricks—Apache Spark as a Service with Sascha DittmannAzure Databricks—Apache Spark as a Service with Sascha Dittmann
Azure Databricks—Apache Spark as a Service with Sascha Dittmann
 
Azure Data Factory v2
Azure Data Factory v2Azure Data Factory v2
Azure Data Factory v2
 
Modern Data architecture Design
Modern Data architecture DesignModern Data architecture Design
Modern Data architecture Design
 
Spark as a Service with Azure Databricks
Spark as a Service with Azure DatabricksSpark as a Service with Azure Databricks
Spark as a Service with Azure Databricks
 
Intro to Azure Data Factory v1
Intro to Azure Data Factory v1Intro to Azure Data Factory v1
Intro to Azure Data Factory v1
 

Viewers also liked

Viewers also liked (7)

IBM - Transformation digitale et le SI des banques
IBM - Transformation digitale et le SI des banquesIBM - Transformation digitale et le SI des banques
IBM - Transformation digitale et le SI des banques
 
Incorporating the Data Lake into Your Analytic Architecture
Incorporating the Data Lake into Your Analytic ArchitectureIncorporating the Data Lake into Your Analytic Architecture
Incorporating the Data Lake into Your Analytic Architecture
 
Data Lake vs. Data Warehouse: Which is Right for Healthcare?
Data Lake vs. Data Warehouse: Which is Right for Healthcare?Data Lake vs. Data Warehouse: Which is Right for Healthcare?
Data Lake vs. Data Warehouse: Which is Right for Healthcare?
 
Public v1 real world example of azure functions serverless conf london 2016
Public v1 real world example of azure functions serverless conf london 2016 Public v1 real world example of azure functions serverless conf london 2016
Public v1 real world example of azure functions serverless conf london 2016
 
Hands-On with U-SQL and Azure Data Lake Analytics (ADLA)
Hands-On with U-SQL and Azure Data Lake Analytics (ADLA)Hands-On with U-SQL and Azure Data Lake Analytics (ADLA)
Hands-On with U-SQL and Azure Data Lake Analytics (ADLA)
 
Building the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake AnalyticsBuilding the Data Lake with Azure Data Factory and Data Lake Analytics
Building the Data Lake with Azure Data Factory and Data Lake Analytics
 
Big data architectures and the data lake
Big data architectures and the data lakeBig data architectures and the data lake
Big data architectures and the data lake
 

Similar to Integration Monday - Analysing StackExchange data with Azure Data Lake

SQL Analytics Powering Telemetry Analysis at Comcast
SQL Analytics Powering Telemetry Analysis at ComcastSQL Analytics Powering Telemetry Analysis at Comcast
SQL Analytics Powering Telemetry Analysis at Comcast
Databricks
 

Similar to Integration Monday - Analysing StackExchange data with Azure Data Lake (20)

Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...
Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...
Analyzing StackExchange Data with Azure Data Lake (Tom Kerkhove @ Integration...
 
NDC Sydney - Analyzing StackExchange with Azure Data Lake
NDC Sydney - Analyzing StackExchange with Azure Data LakeNDC Sydney - Analyzing StackExchange with Azure Data Lake
NDC Sydney - Analyzing StackExchange with Azure Data Lake
 
NDC Minnesota - Analyzing StackExchange data with Azure Data Lake
NDC Minnesota - Analyzing StackExchange data with Azure Data LakeNDC Minnesota - Analyzing StackExchange data with Azure Data Lake
NDC Minnesota - Analyzing StackExchange data with Azure Data Lake
 
SQL Analytics Powering Telemetry Analysis at Comcast
SQL Analytics Powering Telemetry Analysis at ComcastSQL Analytics Powering Telemetry Analysis at Comcast
SQL Analytics Powering Telemetry Analysis at Comcast
 
Unlocking the Value of Your Data Lake
Unlocking the Value of Your Data LakeUnlocking the Value of Your Data Lake
Unlocking the Value of Your Data Lake
 
Intelligent Cloud Conference 2018 - Next Generation of Data Integration with ...
Intelligent Cloud Conference 2018 - Next Generation of Data Integration with ...Intelligent Cloud Conference 2018 - Next Generation of Data Integration with ...
Intelligent Cloud Conference 2018 - Next Generation of Data Integration with ...
 
Lakehouse in Azure
Lakehouse in AzureLakehouse in Azure
Lakehouse in Azure
 
Azure SQL Data Warehouse
Azure SQL Data Warehouse Azure SQL Data Warehouse
Azure SQL Data Warehouse
 
Azure Lowlands: An intro to Azure Data Lake
Azure Lowlands: An intro to Azure Data LakeAzure Lowlands: An intro to Azure Data Lake
Azure Lowlands: An intro to Azure Data Lake
 
10 Reasons Snowflake Is Great for Analytics
10 Reasons Snowflake Is Great for Analytics10 Reasons Snowflake Is Great for Analytics
10 Reasons Snowflake Is Great for Analytics
 
Building End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCPBuilding End-to-End Delta Pipelines on GCP
Building End-to-End Delta Pipelines on GCP
 
Building a Turbo-fast Data Warehousing Platform with Databricks
Building a Turbo-fast Data Warehousing Platform with DatabricksBuilding a Turbo-fast Data Warehousing Platform with Databricks
Building a Turbo-fast Data Warehousing Platform with Databricks
 
Azure Data Platform Overview.pdf
Azure Data Platform Overview.pdfAzure Data Platform Overview.pdf
Azure Data Platform Overview.pdf
 
Azure fundamental -Introduction
Azure fundamental -IntroductionAzure fundamental -Introduction
Azure fundamental -Introduction
 
Afternoons with Azure - Azure Data Services
Afternoons with Azure - Azure Data ServicesAfternoons with Azure - Azure Data Services
Afternoons with Azure - Azure Data Services
 
IBM Cloud Day January 2021 - A well architected data lake
IBM Cloud Day January 2021 - A well architected data lakeIBM Cloud Day January 2021 - A well architected data lake
IBM Cloud Day January 2021 - A well architected data lake
 
Big Data Analytics from Azure Cloud to Power BI Mobile
Big Data Analytics from Azure Cloud to Power BI MobileBig Data Analytics from Azure Cloud to Power BI Mobile
Big Data Analytics from Azure Cloud to Power BI Mobile
 
1 Introduction to Microsoft data platform analytics for release
1 Introduction to Microsoft data platform analytics for release1 Introduction to Microsoft data platform analytics for release
1 Introduction to Microsoft data platform analytics for release
 
Accelerating Business Intelligence Solutions with Microsoft Azure pass
Accelerating Business Intelligence Solutions with Microsoft Azure   passAccelerating Business Intelligence Solutions with Microsoft Azure   pass
Accelerating Business Intelligence Solutions with Microsoft Azure pass
 
Cepta The Future of Data with Power BI
Cepta The Future of Data with Power BICepta The Future of Data with Power BI
Cepta The Future of Data with Power BI
 

More from Tom Kerkhove

More from Tom Kerkhove (20)

Techorama 2022 - Adventures of building Promitor, an open-source product
Techorama 2022 - Adventures of building Promitor, an open-source productTechorama 2022 - Adventures of building Promitor, an open-source product
Techorama 2022 - Adventures of building Promitor, an open-source product
 
Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...
Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...
Microsoft Partners - Application Autoscaling Made Easy With Kubernetes Event-...
 
Introduction to Promitor
Introduction to PromitorIntroduction to Promitor
Introduction to Promitor
 
Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...
Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...
Azure Lowlands 2020 - API management for microservices in a hybrid and multi-...
 
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...
 
Global Azure Virtual - Application Autoscaling with KEDA
Global Azure Virtual - Application Autoscaling with KEDAGlobal Azure Virtual - Application Autoscaling with KEDA
Global Azure Virtual - Application Autoscaling with KEDA
 
Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...
Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...
Building Bruges 2020 - Adventures of building a multi-tenant PaaS on Microsof...
 
AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...
AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...
AZUG Lightning Talk - Application autoscaling on Kubernetes with Kubernetes E...
 
IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...
IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...
IglooConf 2020 - API management for microservices in a hybrid and multi-cloud...
 
IglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft Azure
IglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft AzureIglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft Azure
IglooConf 2020 - Adventures of building a multi-tenant PaaS on Microsoft Azure
 
Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...
Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...
Microsoft Ignite 2019 - API management for microservices in a hybrid and mult...
 
Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...
Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...
Integrate UK 2019 - Adventures of building a (multi-tenant) PaaS on Microsoft...
 
Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...
Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...
Techdays Finland 2019 - Adventures of building a (multi-tenant) PaaS on Micro...
 
Azure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Azure Low Lands 2019 - Building secure cloud applications with Azure Key VaultAzure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
Azure Low Lands 2019 - Building secure cloud applications with Azure Key Vault
 
Next Generation Data Integration with Azure Data Factory
Next Generation Data Integration with Azure Data FactoryNext Generation Data Integration with Azure Data Factory
Next Generation Data Integration with Azure Data Factory
 
Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...
Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...
Intelligent Cloud Conference 2018 - Automatically scaling Kubernetes pods bas...
 
Intelligent Cloud Conference 2018 - Building secure cloud applications with A...
Intelligent Cloud Conference 2018 - Building secure cloud applications with A...Intelligent Cloud Conference 2018 - Building secure cloud applications with A...
Intelligent Cloud Conference 2018 - Building secure cloud applications with A...
 
Techdays Finland 2018 - Building secure cloud applications with Azure Key Vault
Techdays Finland 2018 - Building secure cloud applications with Azure Key VaultTechdays Finland 2018 - Building secure cloud applications with Azure Key Vault
Techdays Finland 2018 - Building secure cloud applications with Azure Key Vault
 
ITProceed 2015 - Securing Sensitive Data with Azure Key Vault
ITProceed 2015 - Securing Sensitive Data with Azure Key VaultITProceed 2015 - Securing Sensitive Data with Azure Key Vault
ITProceed 2015 - Securing Sensitive Data with Azure Key Vault
 
Securing sensitive data with Azure Key Vault
Securing sensitive data with Azure Key VaultSecuring sensitive data with Azure Key Vault
Securing sensitive data with Azure Key Vault
 

Recently uploaded

NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
Amil baba
 
一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理
cyebo
 
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Valters Lauzums
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
pyhepag
 
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotecAbortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 
Exploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxExploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptx
DilipVasan
 

Recently uploaded (20)

AI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfAI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdf
 
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictSupply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
 
How I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonHow I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prison
 
2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting
 
Easy and simple project file on mp online
Easy and simple project file on mp onlineEasy and simple project file on mp online
Easy and simple project file on mp online
 
basics of data science with application areas.pdf
basics of data science with application areas.pdfbasics of data science with application areas.pdf
basics of data science with application areas.pdf
 
The Significance of Transliteration Enhancing
The Significance of Transliteration EnhancingThe Significance of Transliteration Enhancing
The Significance of Transliteration Enhancing
 
一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理一比一原版麦考瑞大学毕业证成绩单如何办理
一比一原版麦考瑞大学毕业证成绩单如何办理
 
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
 
Pre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptxPre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptx
 
Formulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfFormulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdf
 
Artificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfArtificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdf
 
Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
 
Slip-and-fall Injuries: Top Workers' Comp Claims
Slip-and-fall Injuries: Top Workers' Comp ClaimsSlip-and-fall Injuries: Top Workers' Comp Claims
Slip-and-fall Injuries: Top Workers' Comp Claims
 
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
 
2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call2024 Q1 Tableau User Group Leader Quarterly Call
2024 Q1 Tableau User Group Leader Quarterly Call
 
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotecAbortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
 
Exploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptxExploratory Data Analysis - Dilip S.pptx
Exploratory Data Analysis - Dilip S.pptx
 

Integration Monday - Analysing StackExchange data with Azure Data Lake

  • 1. Analysing StackExchange data with Azure Data Lake Analysing StackExchange data with Azure Data Lake
  • 2. Nice to meet you Tom KERKHOVE ➔ Integration Professional ➔ IoT Competency Lead ➔ Windows Development & Microsoft Azure MVP tom.kerkhove@codit.eu +32 473 701 074 @TomKerkhove be.linkedin.com/in/tomkerkhove github.com/tomkerkhove
  • 3. Agenda • Why should we care about Big Data? • Big Data in Azure • Azure Data Lake • Demo • Q & A 3
  • 4.
  • 6. Connect and scale with efficiency Analyze and act on new data Integrate and transform business processes
  • 7. Event producers & gateways Ingestion & transformation Report, Act, Predict
  • 8. Microsoft Patterns & Practices – IoT Journey
  • 9. 9
  • 12. Platform Services Infrastructure Services Web Apps Mobile Apps API Management API Apps Logic Apps Notification Hubs Content Delivery Network (CDN) Media Services BizTalk Services Hybrid Connections Service Bus Storage Queues Hybrid Operations Backup StorSimple Azure Site Recovery Import/Export SQL Database DocumentDB Redis Cache Azure Search Storage Tables Data Warehouse Azure AD Health Monitoring AD Privileged Identity Management Operational Analytics Cloud Services Batch RemoteApp Service Fabric Visual Studio App Insights Azure SDK VS Online Domain Services HDInsight Machine Learning Stream Analytics Data Factory Event Hubs Mobile Engagement Data Lake IoT Hub Data Catalog Security & Management Azure Active Directory Multi-Factor Authentication Automation Portal Key Vault Store/ Marketplace VM Image Gallery & VM Depot Azure AD B2C Scheduler
  • 13. Overview in Azure 13 DocumentDB Data Factory Stream Analytics Data Lake HDInsight Data Lake (Store & Analytics) Virtual Machine IoT Hub SQL Data Warehouse SQL DatabaseStorageEvent Hubs Document Db Data Ingestion Data Storage Data Pipelines Machine Learning Data Analytics
  • 15. 15
  • 16. Analysing Big Data in Azure Azure Data Lake Family HDInsight Data Lake Store Data Lake Analytics • Unlimited storage • WebHDFS Store • Managed cluster service • Open-source technology • Runs on Windows or Linux • Managed job service • U-SQL batch-processing
  • 17. Azure Data Lake Store ➔ WebHDFS compatible ➔ Any size ➔ Any format as-is ➔ Write-once-read-many ➔ Enterprise-grade security ➔ Thé big data store in Azure 17
  • 18. Characteristics ➔ Data Warehousing ➔ Structured data ➔ Defined set of schemas ➔ Requires Extract-Transform- Load (ETL) before storing ➔ Known for some of us ➔ Exploratory analysis is hard because of transforming the data 18 Data Lake vs DataWarehousing ➔ Data Lake ➔ Raw data (unstructured/semi-structured/structured) ➔ “Dump” all your data in the lake ➔ Data scientists will interpret data from the lake ➔ Without metadata, turns in a data swamp pretty fast
  • 19. 19Martin Fowler on Data Lake & Data Warehouses(link)
  • 20. Azure Data Lake Analytics ➔ Run analytics jobs on managed clusters ➔ Don’t worry about scale ➔ Written in U-SQL ➔ SQL Syntax ➔ Extensibility in C# ➔ Easily scaled with Analytics Units ➔ Pay for processing time only 20
  • 21. Writing U-SQL scripts 21 Extract from data source by using built-in or custom extractors. Transform / Analyse the data using SQL-syntax, in-line C# or C# method calls Output the result to a data source by using built-in or custom extractors
  • 22. 22
  • 23. Data Lake Analytics - Data Sources U-SQL Query Query Azure Storage Blobs Azure Data Lake Store Azure SQL Database Azure SQL Data Warehouse Azure SQL in VMs Azure Data Lake Analytics
  • 24. 24
  • 25. Meet StackExchange ➔ Over 280 subwebsites ➔ 150+ GB of open-source data ➔ Different kinds of data ➔ Posts ➔ Users ➔ Votes ➔ ... ➔ A big data sample data set
  • 26. What AreWe GoingTo Do? • Downloading the original data set Acquiring The Data • Upload data set to Azure • Determine what service to use Moving The Data • Merging data from each site into one file • Conversion from XML to CSV Aggregating The Data • Run business logic on it • Attempt to gain knowledge from it Analyzing The Data • Visualize what we’ve learned Visualizing The Data 26
  • 27. Azure Data Lake tools forVisual Studio ➔ Projects / Solutions / Source control ➔ Store Explorer ➔ Browse store ➔ Download complete / subset of file ➔ Preview ➔ JobVisualizer ➔ Determine bottlenecks by using heatmaps ➔ Playback jobs based on telemetry ➔ Query optimization ➔ Job Profiler ➔ Off-Line execution 27
  • 28. Integration with Azure Services ➔ Integrate in your data pipelines in Azure Data Factory ➔ Move data from Azure Data Lake Store to other store ➔ Move data to Azure Data Lake Store ➔ Run U-SQL query within pipeline ➔ Integration with Azure Data Catalog ➔ Register your Azure Data Lake Store assets 28
  • 29. Pricing ➔ Data Lake Store ➔ $0,08/GB stored per month ➔ $0,14 per 1M transactions • 1 transaction is block of up to 128 kB ➔ Egress will be billed but not know yet ➔ Data Lake Analytics ➔ $0,05 per job ➔ $0,05 per minute per Analytics Unit for processing time 29
  • 30. Azure Data Lake Store vs Blob Storage 30 No Limitations Store whatever you want in any format Security Built-in Azure Active Directory support Pricing More expensive than Storage RA-GRS Redundancy It’s there but no control over it Built for Scale Optimized for high- scale reads Integration With Data Factory, Data Catalog & HDInsight
  • 31. 31
  • 32. Summary ➔ Big Data is not just a hype so get ready ➔ Azure Data Lake Store ➔ Analyse today & explore tomorrow ➔ Data Swamps ➔ Data Lake Analytics ➔ No cluster management ➔ Re-use existing skills ➔ Pay for what we use ➔ Big Data in Azure? Azure Data Lake family and it’s easy!
  • 33.
  • 34. 34
  • 35. 35
  • 36. 36