Mastering Data Modeling for NoSQL Platforms

1© 2017 IDERA, Inc. All rights reserved.
MASTERING DATA MODELING FOR NOSQL PLATFORMS
SEPTEMBER 19, 2017
Ron Huizenga - Senior Product Manager
@DataAviator

2© 2016 IDERA, Inc. All rights reserved. Proprietary and confidential. 2© 2017 IDERA, Inc. All rights reserved.
AGENDA
 Data trends & usage
 Why we need models
 Data model types
 Metadata extensions
 NoSQL platforms
• Examples
• Hive
• MongoDB
 Modeling techniques
• Modeling for graph database
 Summary
 Q & A

ENTERPRISE DATA TRENDS
Increasing volumes, velocity,
and variety of Enterprise Data
30% - 50% year/year growth
Decreasing % of enterprise
data which is effectively
utilized
5% of all Enterprise data
fully utilized
Increased risk from data
misunderstanding and non-
compliance
$600bn/annual cost for
data clean-up in U.S.

BUSINESS STAKEHOLDERS’ DATA USAGE
Suspect that business stakeholders
INTERPRET DATA INCORRECTLY
Yes, frequently
14%
Yes, occasionally
67%
No, never
9%
I don’t know
10%
Suspect that business stakeholders make decisions
USING THE WRONG DATA?
Yes, frequently
11%
Yes, occasionally
64%
No, never
13%
I don’t know
12%

… AND WE STILL WANT MORE

DATA LAKE
What’s
in your
data
lake
swamp?

VALUE AND THE NEW LIFECYCLE
Discover
DocumentIntegrate
ER/Studio
 Key Skill Sets
• Data Design & Management
• ETL and Software Development
• Data Analysis / Stats
• Business Analysis & Discovery
 Value Delivered
• Validation
• Integration
• Enrichment
• Usability
 New Data Sources & TCO

WHY DO WE NEED DATA MODELS?
 Evolution
• 38 years of construction
• 147 builders
• No Blueprints
• No Planning
 Result
• 7 stories
• 65 doors to blank walls
• 13 staircases abandoned
• 24 skylights in floors
• 160 rooms, 950 doors
• 47 fireplaces, 17 chimneys
• Miles of hallways
• Secret passages in walls
• 10,000 window panes (all
bathrooms are fitted with windows)

SOME QUESTIONS MODELING CAN ANSWER
 To understand organizational data
• What’s important?
• Where is it? (can be may places)
• Where did it come from?
• How is it used?
• Chain of custody
• Business Rules
 Governance
• How do I identify private information?
• How long should I keep the information?
• Master Data Management classification
• Data quality
• Is it fit for purpose?
• What changed and why?
Data
Governance
Data
Architecture
Management
Data
Development
Database
Operations
Management
Data Security
Management
Reference &
Master Data
Management
Data
Warehousing
& Business
Intelligence
Management
Document &
Content
Management
Metadata
Management
Data Quality
Management

MODEL TYPES & DIAGRAMS
 Data Model Types
• Conceptual Models
• Logical Models
• Physical Models
 Specialized Data Models
• Dimensional
• NoSQL
 Data lineage
 Business process models

DATA MODELS
 Conceptual
• Technology-neutral, high-level layout of entities and
their relationships
• Used to establish contextual consensus among
modeling domain stakeholders
 Logical
• Adds detail to conceptual models in a technology-
neutral rendering
• More context on the entity relationships, including terms
and definitions
 Physical
• Tied to a particular database implementation
• Includes implementation-level details such as indexing
and federation
ABSTRACTION
ELABORATION

DATA MODEL CONSTRUCTS
 Full Specification
• Logical
• Physical
 Persistence Boundaries
• Business Data Objects
 Descriptive metadata
• Names
• Definitions (data dictionary)
• Notes
 Implementation characteristics
• Data types
• Keys
• Indexes
• Views
 Business Rules
• Relationships (referential constraints)
• Value Restrictions (constraints)
 Security Classifications + Rules
 Governance Metadata
• Master Data Management classes
• Data Quality classifications
• Retention policies

DATA DICTIONARY – METADATA EXTENSIONS (ATTACHMENTS)

HIVE

JSON: SINGLE NESTED OBJECT
db.publisher.insert(
{
"_id" :
ObjectId("5367dd99228cd006ab2bc60b"),
name: "O'Reilly Media",
founded: 1980,
location: "CA",
address:
{
street: "987 Oak Ave",
city: "Lincoln",
state: "CA",
zip: "98765"
}
})

JSON: ARRAY OF NESTED OBJECTS
db.patron.insert(
{
"_id" :
ObjectId("5367ddc4228cd006ab2bc60c"),
name: "Joe Bookreader",
address: [
{
street: "123 Fake Street",
city: "Faketon",
state: "MA"
},
{
street: "1 Someother Street",
zip: "12345"
} ]
})
db.book.insert([
{
title: "MongoDB: The Definitive Guide",
author: [ "Kristina Chodorow", "Mike Dirolf" ],
published_date: ISODate("2010-09-24"),
pages: 216,
language: "English",
publisher_id: ObjectId("5367dd99228cd006ab2bc60b"),
available: 3,
checkout: [ { by: ObjectId("5367ddc4228cd006ab2bc60c"), date: ISODate("2012-
10-15") } ]
},
{
title: "50 Tips and Tricks for MongoDB Developer",
author: [ "Kristina Chodorow" ],
pages: 68,
publisher_id: ObjectId("5367dd99228cd006ab2bc60b")
}])

REFERENCING RELATIONSHIPS
db.book.insert([
{
title: "MongoDB: The Definitive Guide",
author: [ "Kristina Chodorow", "Mike Dirolf" ],
pages: 216,
publisher_id: ObjectId("5367dd99228cd006ab2bc60b"),
available: 3,
checkout: [ { by: ObjectId("5367ddc4228cd006ab2bc60c"), date: ISODate("2012-10-
15") } ]
},
{
title: "50 Tips and Tricks for MongoDB Developer",
author: [ "Kristina Chodorow" ],
pages: 68,
publisher_id: ObjectId("5367dd99228cd006ab2bc60b")
}])

ER/STUDIO: EXTENDED NOTATION FOR MONGODB

MODELING TECHNIQUES (GRAPH)
 NoSQL is touted as radically different
• But is it really?
• Implementation/deployment is different
• Modeling benefits and principles still apply
 Conceptual Modeling
• Business concepts
• High level of abstraction
• No implementation detail
 Logical Modeling
• Represents data and the business rules of that data
• Technology agnostic
 Physical modeling
• Defines the platform specific implementation constructs and details

CONCEPTUAL MODEL

LOGICAL MODEL

PHYSICAL MODEL - RELATIONAL

LOGICAL REPRESENTATION FOR GRAPH (SHOWS NODES)

PHYSICAL MODEL GENERATION
 Need to resolve the non-specific relationships
• Can be done automatically!

PHYSICAL MODEL (GRAPH)
 10 Edge tables have been added
between the 6 original Nodes
• Color coded the associative entities
(edges) to make them stand out on the
diagram
• Value of the “Graph Table Type”
attachment = “Edge” for the associative
tables that were generated
 Connector lines have been left in the
model, purely for documentation
• Easier to visually trace which specific
nodes a particular edge connects.
• The generate property has been turned
off for all the connectors (since there is
no such thing as a referential constraint
in a graph data store)
 Each table in the physical model has
a synthetic key
• This is the practice for nodes and edges
in graph data stores.

ENTERPRISE – PUTTING TOGETHER THE PIECES
 Entity Instances – Identify and correlate
• Duplicates
• Complimentary
• Extensions
 Data Lineage
• Chain of custody
• Transactional, Staging, ETL, Warehouse, Marts …
• Transformations
• Data Flow
 Data Usage
• Business processes
• Data Creation and consumption (CRUD)
• Business Rules, Ownership
 Business Meaning
• Business glossaries
• Terms

ENTITY INSTANCES
Repository
Universal Mappings

UNIVERSAL MAPPINGS
 Ability to link “like” or related objects
• Within same model file
• Across separate model files
• Conceptual
• Logical
• Physical
 Entity/Table level
 Attribute/Column level

DATA LINEAGE

BUSINESS GLOSSARIES

ENHANCED COMMUNICATION: GLOSSARY INTEGRATION

SUMMARY
 Organizations are facing huge increases in data volume
• Effective data utilization is decreasing
• Users often make incorrect decisions with data, or use the wrong data
• Big Data adds significant volume and complexity
 NoSQL modeling differences vs. relational
• Different physical deployment characteristics
• New physical constructs such as embedded objects and arrays
• Current conceptual and logical modeling techniques still apply
• Current techniques can be applied to new technologies such as graph database
 Modeling is more important than ever before!
• Must have comprehensive enterprise models to understand the data landscape and
business context
• Data modeling
• Process modeling
• Data lineage
• Universal mappings
• Metadata
• Business glossaries

THANKS!
Any questions?
You can find me at:
ron.huizenga@idera.com
@DataAviator

Mastering Data Modeling for NoSQL Platforms

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Mastering Data Modeling for NoSQL Platforms

Similaire à Mastering Data Modeling for NoSQL Platforms (20)

Plus de DATAVERSITY

Plus de DATAVERSITY (20)

Dernier

Dernier (20)

Mastering Data Modeling for NoSQL Platforms