--session donnée lors du SQL Saturday Torino 2015-- http://www.sqlsaturday.com/454/EventHome.aspx Data Management Gateway (classic, personal, AS connector) is what make modern Microsoft BI stack hybrid. Power BI (v1 and v2) and Azure Data Factory use that component to interact with On-Prem Data assets. That session is a Deep dive into the DMG and the hybrid architecture involved by Power BI and ADF. How does it work ? Security, Firewall, Certificates, Multiple gateways, Admin delegation, Scale out, Disaster Recovery…. All that topics will be covered during that technical session.

1. #SQLSAT454
Deep Dive DMG
Data Management Gateway(s)
Jean-Pierre Riehl - @djeepy1

2. #SQLSAT454
Sponsors

3. #SQLSAT454
Jean-Pierre Riehl
Practice Manager Data & BI
@djeepy1 - http://blog.djeepy1.net
Who am I ?
Pure-Player Microsoft
 Self-Service BI experts
 « Agile BI » projects
 Data Strategy
 Hybrid-BI
MVP SQL Server
French PASS
Chapter Leader

4. #SQLSAT454
La communauté Data Microsoft
Webcasts, Conférences, Afterworks
.Pro
Save the Date
11/30th & 12/1st
Campus Microsoft
Paris
Self-promotion
2 days
40 sessions
50 speakers
100% Free
5th edition

5. #SQLSAT454
An hybrid story…

6. #SQLSAT454
Microsoft Cloud-First strategy…

7. #SQLSAT454
But your data remains On-Prem…

8. #SQLSAT454
Here it comes DMG

9. #SQLSAT454
DMG-aware tools
Power BI for O365
PowerBI.com
Azure Data Factory

10. #SQLSAT454
One DMG to rule them all ?
DMZ
SQL
Cloud On-Prem
Oracle
…
Power BI for O365
Data
Management
Gateway
OnPrem Data Sources
PowerBI.com
Azure Data Factory
Excel

11. #SQLSAT454
Versions of DMG
 There are multiple versions of DMG
Data Management Gateway
Power BI (v1)
ADF
1.6.5735.1 23/9/2015
Power BI Personal Gateway Power BI 1.6.5715.1 3/9/2015
Analysis Services Connector Power BI 1.6.5696.2 2/9/2015
All with the same base code

12. #SQLSAT454
Power BI v1
(for O365)
31/12
24/07
Power BI v2
powerbi.com
Azure
Data Factory
Classic DMG ??
SSAS Connector
Personal Gateway
Versions timeline
Classic DMG ??

13. #SQLSAT454
GATEWAYS USAGES

14. #SQLSAT454
Power BI
Excel
Power Query Power Pivot
Power View Power Map
Power BI Tenant SQL
Cloud On-Prem
Oracle …
Power BI
Admin Center
Data
Catalog
Q&A
Power BI Sites
Data Refresh
Indexing / Search
OData
Access

15. #SQLSAT454
PowerBI.com
Live Power BI
reports &
dashboards
Live
Query
Scheduled
Refresh
Cloud
On-premises

16. #SQLSAT454
Azure Data Factory

17. #SQLSAT454
Hybrid features
 Data Refresh
 Data Movement
 Live Query
 Indexing Sources
 OData Access
(Data Virtualization)
Initiate by Cloud

18. #SQLSAT454
DMG DETAILS

19. #SQLSAT454
What’s that
 A Windows Service
 DIAHostService (Data Management Gateway)
 A configuration application
 Managed by the cloud
 Power BI Admin Center
 Azure Portal (new version)
Manage by User
An application

20. #SQLSAT454
FIRST LOOK AT DMG

21. #SQLSAT454

22. #SQLSAT454
Data Sources Supported
Source Authentification
SQL Server / Oracle
Sybase, Teradata, DB2, MySQL
Windows, Basic
Custom ODBC
Files (CSV, XML, Text, Excel, Access)
Folders
Windows
SharePoint List (Online)
SharePoint List (On-prem)
Anonymous
Anonymous, Windows
OData Feed
Anonymous, Windows, Basic
Key (Data Market)
Azure Marketplace Key (Data Market)
Azure HDInsight
Azure Blob Storage
Azure Table Storage
Key (Azure Account)
SQL Azure
PostgreSQL
Basic
SSAS Tabular/Multidim (Power Query)
SSAS Tabular (Live Query)
Windows
+SSIS Data
Feed

23. #SQLSAT454
Data Sources supported
 Details on data sources supported
 https://support.office.com/en-US/article/Supported-Data-Sources-
and-Data-Types-CB69A30A-2225-451F-A9D0-59D24419782E
 Prequisites
 Beware of versions
 Beware of types
 Drivers installed
 https://support.office.com/en-us/article/Data-source-prerequisites-
Power-Query
and check version…

24. #SQLSAT454
HOW DOES IT WORK

25. #SQLSAT454
SCHEDULE A DATA REFRESH

26. #SQLSAT454
Data Source Discovery
 Data Sources are declared on « Admin
Center »
 They are associated to a Gateway
 Power BI Service resolves Data Source and
find right Gateway
 Resolution = Provider + Server + Catalog
 Warning: Case sensitive
Personal
& Self-Registered

27. #SQLSAT454
Cloud On-Prem
Data Refresh explained
Power BI
OnPrem
Sources
DMG
Azure Service
Bus
0
Register DMG
Register Data Sources
1
Ask for Data Refresh
Check permissions
Find Gateway
4 Execute Query
5 Get Data
6 Compress, Chunk
Queue
request2
3 Read
request
Gateway
Cloud Service
7 Update Model

28. #SQLSAT454
Cloud On-Prem
OData access explained
Power BI
OnPrem
Sources
DMG
0 Index Sources
2
Resolve Data Source
Redirect to local Gateway
4 Execute Query
3 Request Data
1 Search Source
Excel
Power Query
Hybrid Proxy

29. #SQLSAT454
SECURITY

30. #SQLSAT454
Excel
Power Query
Power BI
DMG
Data sources
• SQL (1433)
• FileShare
• Etc.
+DNS resolution
Azure Service Bus (outbound)
• 9350-9354 / 5671,5672
Power BI Admin Center (outbound)
• 80/443
Direct HTTP
(inbound)
• HTTPS (8050)
• HTTP (8051)
HTTPS
(443)
Cloud
On-Prem
Network Security
Credential
Manager

31. #SQLSAT454
Identity Management
2 flavors :
 Impersonation by the DMG
 Impersonation by the Source
How is query authentified on Data Source ?

32. #SQLSAT454
Impersonation by DMG
 Connection identity is impersonated by DMG
 Peer-to-peer connection
 Credentials are stored by DMG for each
source
 Credentials can be stored encrypted locally
 Credentials can be stored encrypted on Cloud
 Credentials are set up locally
 Credential Manager is a Click Once application
 Credential Manager discuss with DMG
Or set up on the cloud

33. #SQLSAT454
CREDENTIAL MANAGER

34. #SQLSAT454
Impersonation by the Source
 Used by Live Query
 Keep identity of original user (Kerberos-like)
 Use the EffectiveUsername method
 DMG authenticates to SSAS
 DMG passes EffectiveUsername property
 SSAS executes the query as if it was original user
 DMG must be SSAS Admin
 UPN must be declared in AD

35. #SQLSAT454
EffectiveUsername

36. #SQLSAT454
HIGH AVAILABILITY &
DISASTER RECOVERY

37. #SQLSAT454
As easy as scale-out
DMGs
Cloud
On-Prem
Power BI
…
Instance 1 Instance 2
Server A Server B

38. #SQLSAT454
HA & DR
 Gateways and Data sources are managed by
Cloud
 Credentials are encrypted by MDG
 Use of Certificate, self-generated or yours
 Need Certificate when registering a new instance
 Warning when stored localy
 Load Balancing

39. #SQLSAT454
HA & DR

40. #SQLSAT454
Last thought
« Classic » DMG
 IT managed
 Data source granularity
 Data sources registration
 Credentials bound to source
 HA & DR
Personal Gateway
 User Managed
 User Granularity
 Self data sources registration
 Credentials bound to user
 No service continuity
Waiting for merge !!!!!!

41. #SQLSAT454
Going further
 Express Route
 Mashup Container (aka Power Query)
 Monitoring & Logging

42. #SQLSAT454
Resources
 Getting Started : https://support.office.com/en-us/article/Power-BI-for-Office-365-Admin-Center-Help-
5e391ecb-500c-47a3-bd0f-a6173b541044
 Data Factory Reference : https://msdn.microsoft.com/en-us/library/azure/dn834987.aspx
 Supported Data sources : https://support.office.com/en-us/article/Supported-Data-Sources-and-Data-
Types
 DMG Prerequisites : https://support.office.com/en-us/article/Data-source-prerequisites-Power-Query
 SSIS Data Feed : http://www.microsoft.com/en-us/download/details.aspx?id=39931
 How does DMG work - sequence diagram :
 https://support.office.com/en-us/article/An-Inside-look-at-Microsoft-Data-Management-Gateway-
078D306D-2D28-409C-962B-D80B69A1D0B2
 Azure IP adresses : http://www.microsoft.com/en-us/download/details.aspx?id=41653
 Store encrypted credentials in the cloud for business continuity : https://support.office.com/en-
us/article/Business-Continuity-for-a-Data-Management-Gateway-00106c58-a0a1-4cea-8886-
da51b8cadc5b
 Matt Masson Session at SPC : http://channel9.msdn.com/Events/SharePoint-
Conference/2014/SPC397
 Mini Nay & Luming Han Session at PASS Summit :
http://www.sqlpass.org/summit/2014/Sessions.aspx#sid7231

43. #SQLSAT454@djeepy1 - http://blog.djeepy1.net
Any questions ?
Thank You!
And don’t forget to compile evaluations form :