Contenu connexe Similaire à Data Catalog as the Platform for Data Intelligence (20) Data Catalog as the Platform for Data Intelligence1. The Data Catalog as the Platform
for Data Intelligence
Satyen Sangani
August 18, 2020
Twitter: @satyx
2. 2 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Topics
• The drive for data culture
• The search for (data) intelligence
• The state of the data catalog
• The data catalog as a platform for data intelligence
3. 3 | Alation Confidential & Proprietary.The Catalog is the Platform™
The World is Changing Fast...
4. 4 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
...And Your Organization is Changing Too.
Used to have too little
data, now too much
Capture tribal knowledge,
enable collaboration
Comply with changing
laws and regulations
Data Explosion
• 63 ZB in 2021 (volume)
• 300+ DBMSs (variety)
• IoT and Edge (velocity)
Evolving Laws
• Privacy: GDPR, CCPA, HIPAA
• Risk: BCBS 239
• Industry-specific: FedRAMP
Changing Workforce
• Turnover, restructuring
• New SMEs, lost stewards
• Remote work (WFH)
5. 5 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
In a Changing World, Everyone is Making Decisions
People are constantly changing
Information doesn’t flow fast enough
Expertise travels even more slowly than information
6. 6 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
How Do You Want Them to Make Decisions?
Simon Says? Their Gut?
Based on data
and evidence?
7. 7 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Characteristics of a Data Culture
Hypothesis/
Test Oriented
Knowledge Systematically
Collected/Documented
Methodological
Transparency
Distributed Evidence Based Non-Hierarchical
8. 8 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
8 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Listen like you’re wrong.
Measure ourselves through customer impact.
Move the ball.
Build for the long term.
Alation’s Values Intended to Drive a Data Culture
9. 9 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Data Culture Is an Imperative
The Benefits
are Compelling
• Insights-driven
businesses on track to
earn $1.8T by 2021
• 7x more likely to
increase revenue as
result of big data
• 2.8x more likely to have
double-digit growth
The Stakes are High
& Mistakes Costly
• Fines of 20M Euros or
4% of annual revenue
(GDPR)
• $5M fines for misleading
financial reports (SOX)
• Incorrect decisions
made on bad data or
bad models
Data Culture is a Priority
Forrester: Insights-Driven Businesses Set the Pace
for Global Growth. October 2018
2018 Gartner Fourth Annual Chief Data Officer Study
10. 10 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
The Alation Vision
To empower a curious and rational world
Data Culture
Data
Literacy
Enable proper
interpretation & analysis
Data
Governance
Take responsibility
& authority
Data Search
& Discovery
Find &
understand
11. 11 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Topics
• The drive for data culture
• The search for (data) intelligence
• The state of the data catalog
• The data catalog as a platform for data intelligence
12. 12 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Data Intelligence
Intelligence (as defined by Google):
in •tel •li•gence
/inˈteləjəns/
The ability to acquire and apply knowledge and skills.
Data Intelligence (as defined by me):
da•ta in •tel •li•gence
/ˈdādə/ /inˈteləjəns/
The organizational ability to acquire and apply
knowledge, skills, and the resulting competitive
advantage by leveraging data & analytics.
13. 13 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
We’ve Been Searching for a While...
14. 14 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
The “Truth” Depends on Your Point of View
15. 15 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Academia Gets This
16. 16 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Intelligence Comes From
The ability to test
multiple scenarios
Documented prose,
not just reports
Traceable claims,
data and reports
Having the right info
at the right time
Links to
evidence
Open
Access
17. 17 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
You don’t need a single source of truth...
...you need a single system of reference.
18. 18 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Topics
• The drive for data culture
• The search for (data) intelligence
• The state of the data catalog
• The data catalog as a platform for data intelligence
19. 19 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
What is a Data Catalog?
• A repository of metadata on information sources across
an organization
- Search & discovery
- Data governance & curation
- Collaboration & analysis
• Catalogs a broad range of information assets
- Data sets, tables, articles, reports, queries, visualizations, conversations
• Includes common functionality such as:
Answers these
core questions:
How to find information?
Can it be used?
Should it be used?
How should it be used?
Business Glossary Lineage Catalog Pages Search
20. 20 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Alation Commercialized the Data Catalog
“Alation started the ML Data Catalog trend.”
-Forrester Wave, Machine Learning Data Catalogs, 2018
• Founded 8 years ago in 2012
• Born out of my experience
at Oracle
• Envisioned a collaborative catalog
• Born in the era of “big” data
21. 21 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
But, Metadata Management and Data Catalogs
Have Been Around for Years.
CatalogMetadata Management
Used by: Information Suppliers
Model: System of Record
Inspiration: Chart of Accounts/
Inventory Management
Intelligence: Transactional: Based on
Human Assertion
Coverage: Systems with “physical” data
Used by: Information Consumers
Model: System of Engagement
Inspiration: Collaborative/Social
Intelligence: Automated: Based on Social
Behaviors
Coverage: add. Reports, Algorithms,
ETLs, Streams
22. 22 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Living Catalogs Enable You to Quickly Self-Serve
is a trusted catalog that
helps you quickly find a
professional.
is trusted catalog that helps
you quickly learn about
products sold on the web.
is a trusted catalog
that helps you
quickly surf the web.
You used to have a
rolodex and a network.
Experts in Data Management New OLED TV What is PageRank?
You used to go
to a store.
You used to get help
from a librarian.
23. 23 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Amazon Reinvented the Supply Chain
They did this because they understood demand
24. 24 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Topics
• The drive for data culture
• The search for (data) intelligence
• The state of the data catalog
• The data catalog as a platform for data intelligence
25. 25 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Early Uses of Data Catalogs Were Search/Discovery
Data
Scientist
Business
Analyst
26. 26 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
The Importance of Machine Learning
Plot the dots
Connect
Make Connections
Compute
Draw Conclusions
Conclude
Sue
Bob
Query #2
Query #3
Query #1
Top Users: Bob and Sue
Popularity: II
27. 27 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Collaborative
data catalog
Virtuous Cycle of Adoption
(More) people
use data catalog
Data catalog
becomes more
valuable
27 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
28. 28 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
The “Truth” Depends on Your Point of View
29. 29 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
The Data Catalog Is that Source of Reference
FIND | UNDERSTAND | TRUST
30. 30 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
The Catalog as Platform for Data Intelligence
31. 31 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
31 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
“Alation helps the business uphold our
governance standards so anybody coming in
can get their data quickly and efficiently,
and make the right decisions.”
AMY KEELTY
Information Strategy & Governance Director
American Family Insurance
Data Governance Naturally Followed
32. 32 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Data Governance is Evolving from Centralized
to Distributed Responsibility
Guided Community Driven
Social Collaboration
Centralized
Command & Control
Centralized
Command & Control
Guided Community Driven
Social Collaboration
33. 33 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
A Different, Value-Driven Approach
People-First
Exercised at
Point of Use Collaborative Intelligent
Connects policy to
action in users’
day-to-day activities
Guides people’s
behavior by discovering
and formalizing their
relationship with data
Automation powered
by AI and ML
Crowdsourcing and
community-driven
Iterative,
Agile
Incremental, not a
big-bang approach
Active Data Governance
34. 34 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
The Accelerating Pace of Cloud Data Migration
These systems are increasingly headed to the cloud,
and by 2022, 75% of all databases will be deployed
or migrated to a cloud platform, the report said.”
1990s:
• Spreadsheets
2005:
• Data Lakes
2000s:
• Data Warehouse
2010:
• Cloud Hosted Data Warehouses
• Cloud Managed Data Lakes
”
35. 35 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
What Data is Most-Used?
1. In this example, of the 145
tables, only 13 have ever been
queried by Database users
2. Reduce storage & compute
costs through migrating what
users care about
3. Move related Datasets by
understanding commonly
joined tables
36. 36 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Catalogs Provide Intelligence...
The ability to test
multiple scenarios
Documented prose,
not just reports
Traceable claims,
data and reports
Having the right info
at the right time
Links to
evidence
Open
Access
37. 37 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Where Do We Go Next?
More intelligence about:
Semantics Third Party Data
How data can be
merged/joined
Algorithms
38. 38 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Scale Data Discovery & Automate Data Tagging
Delivers CCPA &
GDPR templates
to tag data sets
It’s hard to identify
and classify private
or sensitive data
Solution
Automate data discovery
and sensitive data tagging
Gathers technical
metadata across
enterprise data
sources
1
2
3 Understands
unique individuals’
data across disparate
data sources
39. 39 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Accelerate Governance & Stewardship of Sensitive Data
Classifies sensitive
data and correlates
it to the related
datasets
We can’t easily flag
relevant policies when
users interact with data
Solution
Accelerate governance
of sensitive data
Ingests metadata
and is the single
catalog to find
trusted data
1
2
3 Groups the data
and uses a template
to assign policies to
that group to scale
policy notification
across dataset
40. 40 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Ensure Ongoing Compliance
Scales & automates
discovery and tagging
by connecting to,
profiling and tagging
relational, non relational
and unstructured data
It’s hard to maintain
as business and
regulations change
Solution
Single point of reference
for policy & usage
guidelines at the point
of consumption
Surfaces metadata and
usage patterns for insights
from various data sources
1
2
3 Single interface for users to
find policies along with data
to break down silos
41. 41 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
How Alation and BigID Address the Challenges
Use - Data workers
access data with the
associated policies
5
Classify - BigID profiles
data and classifies data into
a personal and sensitive
category with associated
policy templates
2
Tag – Alation imports the data
classification from BigID and
tags Alation catalog pages to
associate the metadata with
the correct policies
31
Ingest - Alation ingests
metadata from all
enterprise data sources
for stewards to tag for
contextual classification
Scale - BigID uses Alation
documentation to
automate data association
across data landscape
4
41 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
42. 42 | © 2020 Alation, Inc. – All Rights Reserved.The Catalog is the Platform™
Satyen Sangani
@satyx