Data Modeling & Data Integration

Data Modeling & Data Integration
Donna Burbank
Global Data Strategy Ltd.
Lessons in Data Modeling DATAVERSITY Series
August 24th, 2017

Global Data Strategy, Ltd. 2017
Donna Burbank
Donna is a recognised industry expert in
information management with over 20
years of experience in data strategy,
information management, data modeling,
metadata management, and enterprise
architecture. Her background is multi-
faceted across consulting, product
development, product management, brand
strategy, marketing, and business
leadership.
She is currently the Managing Director at
Global Data Strategy, Ltd., an international
information management consulting
company that specializes in the alignment
of business drivers with data-centric
technology. In past roles, she has served in
key brand strategy and product
management roles at CA Technologies and
Embarcadero Technologies for several of
the leading data management products in
the market.
As an active contributor to the data
management community, she is a long
time DAMA International member, Past
President and Advisor to the DAMA Rocky
Mountain chapter, and was recently
awarded the Excellence in Data
Management Award from DAMA
International in 2016. She was on the
review committee for the Object
Management Group’s (OMG) Information
Management Metamodel (IMM) and the
Business Process Modeling Notation
(BPMN). Donna is also an analyst at the
Boulder BI Train Trust (BBBT) where she
provides advices and gains insight on the
latest BI and Analytics software in the
market.
She has worked with dozens of Fortune
500 companies worldwide in the Americas,
Europe, Asia, and Africa and speaks
regularly at industry conferences. She has
co-authored two books: Data Modeling for
the Business and Data Modeling Made
Simple with ERwin Data Modeler and is a
regular contributor to industry
publications. She can be reached at
donna.burbank@globaldatastrategy.com
Donna is based in Boulder, Colorado, USA.
2
Follow on Twitter @donnaburbank
Today’s hashtag: #LessonsDM

Lessons in Data Modeling Series
• January 26th How Data Modeling Fits Into an Overall Enterprise Architecture
• February 23rd Data Modeling and Business Intelligence
• March Conceptual Data Modeling – How to Get the Attention of Business Users
• April The Evolving Role of the Data Architect – What does it mean for your Career?
• May Data Modeling & Metadata Management
• June Self-Service Data Analysis, Data Wrangling, Data Munging, and Data Modeling
• July Data Modeling & Metadata for Graph Databases
• August Data Modeling & Data Integration
• September Data Modeling & MDM
• October Agile & Data Modeling – How Can They Work Together?
• December Data Modeling, Data Quality & Data Governance
3
This Year’s Line Up

Both Business & Technical Drivers Require Data Integration
4
A Data Model is a Common Reference Hub for Business & Technical Rules
Business Drivers
Technology Drivers
Enterprise Knowledge
Inventory
Mergers &
Acquisitions
Innovation &
Collaboration
Efficiency &
Agility
Etc…
Data ModelData Warehousing
Master Data
Management (MDM)
Data Lake
APIs & Application
Integration
Etc…
The Data Model
is the Common
Reference

Levels of Data Models
5
Conceptual
Logical
Physical
Purpose
Communication & Definition of
Business Concepts & Rules
Clarification & Detail
of Business Rules &
Data Structures
Technical
Implementation on
a Physical Database
Audience
Business Stakeholders
Data Architects
Data Architects
Business Analysts
DBAs
Developers
Business Concepts
Data Entities
Physical Tables
Business Stakeholders
Data Architects
Enterprise
Subject Areas
Organization & Scoping of main
business domain areas
Data
Integration
Team
Data Models are helpful for data integration at each level.

Enterprise Knowledge Inventory
• A data model describes a business, particularly at the conceptual & logical levels. It provides:
• Inventory of the key data assets that run the organization
• Clarification of core terminology & business definitions (e.g. what do we mean by Location…)
• Definition of core business rules & practices (e.g. Staff are assigned to only one Location…)
6
In a data-driven business, data is your core Intellectual Property (IP)
A “one page” enterprise business model should provide an overview of what
the business does & how it operates.
- i.e. The model to the left is most likely a retail organization, not a health care provider.

• A data model describes YOUR business, with your unique business rules, terminology, & definitions.
• It describes the unique way your organization operates
• It is your IP -- protect & manage it accordingly
7
Your Organization
Integration
You’ll need to integrate with
applications, partners, agencies, etc.
-- but they should not necessarily
re-define how your organization
operates.
Etc.
Applications & Partners should NOT:
• Hold your data “captive”
• Re-define how you do business, just to fit
their canned data model. (There may be a
reason to change, but do it purposefully).

• A common question is whether to use an industry standard data model.
• Industry models can be a helpful reference & guide
• But don’t blindly follow them, without customizing for your unique organization.
• Just as your organization is unique, so is its data.
8
Your Organization
Reference
Industry data models can be a great
reference, but you’ll likely want to
customize them to fit your unique
organization.
Industry Standard Model

Mergers & Acquisitions
• Data is a large part of the value of a business acquisition
• Increasingly, data is a key driver for acquisitions – obtaining the data that another firm maintains about
customers, products, recipes, innovations, etc.
• The data holds the rules, history & IP of the business.
• Just as you would take an inventory of products, you need to data a data inventory – via a data model
(reverse engineering).
• Disparate business processes are often manifested in the data. Ignoring these key business process
differences can wreak havoc on operations.
9
Organization BOrganization A
What issues might arise in integrating the customer accounts from the two organizations?

Efficiency & Agility
10
• In many organizations, a great deal of time, energy, and brainpower is wasted:
• Reformatting or re-working data from disparate sources
• Searching to understand the meaning of data
• Looking for data that is unavailable
• Siloes often exist, with key information not being shared – not out of malice, but because
common, published inventory & standards don’t exist – i.e. lacking a common data model.
I’m just about done with my spreadsheet – Customers
by Region, Age, and Income Level. Great!
If I have to reformat this spreadsheet one
more time to account for mismatched
Region Codes, I’m going to shoot myself.
Etc.!
Why can’t we get Income Levels for
our customers? This is so dumb.

Innovation & Collaboration
• An Enterprise Data Model provides a “catalogue” of an organization’s data asset.
• Staff are able to see all of the data available across the organization – spurring innovation & collaboration.
11
Sharing the catalogue of enterprise data assets
I didn’t realize that the Insurance
Dept was tracking Weather
Events. I could use that to link
Weather to Product Sales for
Trend Analysis!! Cool!

12
Technical Data Integration
Many styles & methods

Data Modeling for Data Warehousing & Business Intelligence
• What is the definition of customer?
• Where is the data stored?
• How is it structured?
• Who uses or owns the data?
Data Warehouse BI Report:
Customers by Region
• What are the definitions of key business terms?
• What do I want to report on?
• How do I optimize the database for these reports?
Data Modeling helps answer:
For Data Warehousing For BI Reporting
Data Modeling helps answer:
• Data Modeling is the “Intelligence behind Business Intelligence”
• Creating business meaning & context
• Understand source and target data systems
• Optimize data structures to align queries with reports
Show me all
customers by region
Source Systems
Relational Model
Dimensional Model

The Need for Data Warehousing
True or False: “We don’t need data warehousing any more because storage is so
cheap and processing power is so fast with today’s modern hardware.”
14

The Need for Data Warehousing
• True or False: “We don’t need data warehousing any more because storage is so cheap
and processing power is so fast with today’s modern hardware.”
15
I can’t find anything in this file cabinet.
It’s just a bunch of papers without any
folders or organization!
Don’t worry—just get more file cabinets.
Much of the value in data warehousing is making data consumable &
understandable for ease of reporting.

Metadata Matters
Even with today’s advanced hardware & storage options, self-service BI tools, and data
science skills & tools, attention needs to be paid to the quality, context, & structure of data
Raw data used in Self-Service Analytics and BI environments is
often so poor that many data scientists and BI professionals
spend an estimated 50 – 90% of their time cleaning and
reformatting data to make it fit for purpose.(4
Source: DataCenterJournal.com
Correcting poor data quality is a Data Scientist’s least favorite
task, consuming on average 80% of their working day
Source: Forbes 2016
(aka Data Models & Metadata)
If I have to reformat this spreadsheet one
more time to account for mismatched
Region Codes, I’m going to shoot myself.

Master Data Management (MDM)
• Master Data Management (MDM) is the practice of identifying, cleansing, storing & governing
core data assets of the organization (e.g. customer, product, etc.)
• There are many architectural approaches to MDM. Two are the following:
17
Centralized Virtualized/Registry
MDM
Virtualization Layer
• Core data stored in
a common schema
in a centralized
“hub”.
• Used as a common
reference for
operational systems,
DW, etc.
• Data remains in
source systems.
• Referenced through
a common
virtualization layer.
BOTH require a Data Model

MDM Data Models
• In an MDM Model, the core attributes for master
data entities can be identified.
• In some cases, Stewardship can be defined at the
attribute level.
• Multiple groups update/create/monitor certain field
values.
• Certain attributes are core to all (e.g. demographics
info)
• (More on this next month…) 
18
Core, Shared
Attributes
Team A
Team B
Team C
Patient
Patient ID
Date of Birth
SSN
First Name
Last Name
MaidenName
Middle Name
Name Prefix
Name Suffix
Date of Death
Phone Number
Email
Gender
Marital Status
Race
Ethnicity
Religion
Primary Language
Secondary Language
Primary Diagnosis Group
Secondary Diagnosis Group
Competency Status
Education Level
Need ofDetox
Risk of Harm to Self
Special Needs Requirement
Current Risk
Veteran Status
National Guard/Military Reserve
Pregnant
Employment Status
Number in Household
Household Income
Living Arrangement
No Mailings

Data Lake Big Data Model
• With the Big Data and NoSQL paradigm, “Schema-on-Read” means you do not need to know how you will
use your data when you are storing it.
19
File system
hdfs dfs -put /local/path/userdump /hdfs/path/data/users
Table Structures
Create table …
Analysis
Analyze & understand the data. Build a data structure to suite
your needs.
• You do need to know how you will use your data when you are using
it and model accordingly.
• For example, you may first place the data on HDFS in files, then apply a table
structure in Hive.
• Apache Hive provides a mechanism to project structure onto the
data in Hadoop.
Hive
HDFS
Exploration

An Enterprise Data Inventory for the Data Lake
• An Enterprise Data Model can help provide an Inventory for what data resides in the Data Lake.
20
Twitter Feeds
Hive Table for Staff
NOAA Weather Feeds
Hive Table for Product
Sensor Data
Allowing for Innovation & Discovery

APIs and Application Integration
• APIs are a standard ways to share data to/from applications.
• These should be mapped to the Enterprise model, but the API model/design should focus on the
User Perspective.
21
Enterprise Perspective User Perspective
PersonObject
Person
PersonID: string
PersonFirstName: string
PersonLastName: string
GetPersonObject (GET)
PutPersonObject (PUT)
Application

Summary
• Business Data Models help create an Enterprise Knowledge Inventory that can help with business
drivers such as:
• Innovation & Collaboration
• Mergers & Acquisitions
• Efficiency & Agility
• Business Data Models help define the core definitions & rules for your enterprise data
• Data is your organization’s IP
• A Data model is an inventory of the data asset
• Technical Data Models support the various ways to integrate data from Data Warehousing, Data Lakes,
MDM, to APIs.
• Technical source & target formats are key
• Helps define stewardship & ownership
• Format should suit the purpose & audience
• Using Data Models are a core part of data integration helps provide both structure & meaning for both
business & technical team members

About Global Data Strategy, Ltd
• Global Data Strategy is an international information management consulting company that specializes
in the alignment of business drivers with data-centric technology.
• Our passion is data, and helping organizations enrich their business opportunities through data and
information.
• Our core values center around providing solutions that are:
• Business-Driven: We put the needs of your business first, before we look at any technology solution.
• Clear & Relevant: We provide clear explanations using real-world examples.
• Customized & Right-Sized: Our implementations are based on the unique needs of your organization’s
size, corporate culture, and geography.
• High Quality & Technically Precise: We pride ourselves in excellence of execution, with years of
technical expertise in the industry.
23
Data-Driven Business Transformation
Business Strategy
Aligned With
Data Strategy
Visit www.globaldatastrategy.com for more information

Contact Info
• Email: donna.burbank@globaldatastrategy.com
• Twitter: @donnaburbank
@GlobalDataStrat
• Website: www.globaldatastrategy.com
24

Lessons in Data Modeling Series
• January 26th How Data Modeling Fits Into an Overall Enterprise Architecture
• February 23rd Data Modeling and Business Intelligence
• March Conceptual Data Modeling – How to Get the Attention of Business Users
• April The Evolving Role of the Data Architect – What does it mean for your Career?
• May Data Modeling & Metadata Management
• June Self-Service Data Analysis, Data Wrangling, Data Munging, and Data Modeling
• July Data Modeling & Metadata for Graph Databases
• August Data Modeling & Data Integration
• September Data Modeling & MDM
• October Agile & Data Modeling – How Can They Work Together?
• December Data Modeling, Data Quality & Data Governance
25
This Year’s Line Up

Questions?
26
Thoughts? Ideas?

Data Modeling & Data Integration

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (15)

Similaire à Data Modeling & Data Integration

Similaire à Data Modeling & Data Integration (20)

Plus de DATAVERSITY

Plus de DATAVERSITY (20)

Dernier

Dernier (20)

Data Modeling & Data Integration