This document provides an overview of SQL and NoSQL databases. It discusses how relational databases using SQL emerged as the dominant data storage approach but faced challenges in scaling to big data workloads. NoSQL databases were developed to address these scaling needs by using non-relational data models like key-value, document, and column-oriented structures that are better suited to distributed architectures. The document outlines the history and characteristics of SQL and relational databases and how NoSQL databases address needs like scalability that drove their emergence in the big data era.
2. Topics
What is SQL? What is NoSQL?
Why have relational databases been
successful?
Why did NoSQL databases emerge?
How are their data models different?
3.
4. SQL & relational databases
Relational databases are software
applications that store data
Data is stored in tables that have rows &
columns : think excel spreadsheets
FirstName LastName Age
Zipcode
Gender
Bob
Smith
45
38444
M
Jane
Happy
23
15122
F
Fred
Jones
55
92102
M
Johnny
Appleseed
26
90025
M
5. SQL & relational databases
Relational databases typically have
many tables that are “related” to one
another
6. SQL & relational databases
Relational databases support access to
data in tables through a language called
“SQL” – Structured Query Language
SQL supports “set” based operations on
tables – selection, projection, joining
SQL is based on relational algebra
7. SQL & relational databases
Relational databases were developed in
the late 1970s at IBM
They have been the dominant approach
to data management in the enterprise
through the early 2000’s
Examples include
Oracle
Sybase
MySQL
Postgress
8.
9. NoSQL databases
NoSQL are software applications that
store data
They, not surprisingly, do not use SQL or
the relational model (interrelated tables)
They are “less strict” about data
definition
They were developed in a “big-data”
world for applications needing massive
scalability (clustering)
12. RDBMS value - persistence
During the 90’s and 2000’s as pc’s
became ubiquitous, distributed
computing took off.
In the 1990’s, client-server and n-tier
architectures dominated enterprise
development
The late 90’s and 2000’s saw the
dominance of the web and distributed
applications that broke out of enterprise
13. RDBMS value - persistence
In this distributed world where
applications needed to keep data
around for
Many users
Extended periods
RDBMS emerged as the defacto choice for
persisting data.
14. RDBMS value - concurrency
Another challenge that distributed
applications presented was
concurrency:
many users viewing and potentially updating
the same data at the same time
Concurrency is notoriously difficult to
get right for even the best engineers.
Relational databases “helped” by
controlling data access with transactions
15. RDBMS value - integration
Enterprise application eco-systems
necessitate multiple integrated software
applications. Example
Customer Service app
Biz Intel app
E-Commerce app
Inventory management apps
Common approach was to use a shared
rdbms database integration approach.
16. RDBMS value – SQL
RDBMS providers all supported a core
SQL standard
In theory this would allow developers to
switch reliance on different RDBMS
providers without problems
In fact, different providers (Oracle,
Sybase, Microsoft) developed different
“dialects” or SQL extensions (pl SQL vs.
T-SQL)
17.
18. Crack #1– impedance mismatch
Impedance mismatch is the difference
between the relational model and inmemory data structures
19. Crack #1– impedance mismatch
In the late 1990s people believed that
impedance mismatch would lead to
RDBMS being replaced by databases
that replicated in-memory structures to
disk (OODBMS)
While the 1990s saw the rise of OO
programming languages, OODBMS
never took gained real traction
20. Crack #1– impedance mismatch
OODBMS didn’t gain traction because
Impedance mismatch had been made easier
to deal with by Object-Relational (OR)
mapping frameworks like Hibernate, iBatis,
& Cocoon
There was a growing professional divide
between application developers and
database administrators
The value of RDBMS as an app integration
mechanism was large
21. Crack #2– SOA
The 2000’s saw a shift in how enterprise
applications interacted
Historically, many applications interacted
through a shared RDBMS.
This approach – shared integration
RDBMS – has serious problems
Overly complex schema
Cant change tables or add indices easily
Database has to preserve integrity
22. Crack #2– SOA
Interactions between applications shifted
to web-services
Web-services constituted protocols for
moving documents (XML, JSON) over
HTTP using SOAP or REST based
approaches
SOA allowed applications to
encapsulate data and expose it through
services
23. The Final Crack #3– Clusters
The internet saw several large web
properties dramatically increase in scale
Websites started tracking activity and
structure in a very detailed way
Social gestures
Social links
Log data
Purchase gestures
Increasing numbers of users appeared
using more devices
24. The Final Crack #3– Clusters
The problem with scaling out (clustering)
is that RDBMS are not designed to run
on clusters.
Oracle RAC & MS SQL Server all use
the concept of a shared disk sub-system
Still single point of failure and scaling
limitation
The final crack – mismatch between
RDBMS & clusters
25. NoSQL Emergence
The emergence of NoSQL was really
about needing databases that run on
clusters
One exception is Graph databases
Though problems with shared database
integration and impedance mismatch
existed, it was the need for scale that
drove the emergence of NoSQL
databases
26.
27. Aggregate Data Models
A key characteristic of NoSQL
databases is that they do not use the
Relational data metamodel (relations &
tuples)
There are four types of data
metamodels in the NoSQL eco-system
Key-value
Document
Column-family
Graph
28. Aggregate Data Models
Key-value, document, and columnfamily NoSQL databases share a
common characteristic of their data
models called “aggregate orientation”
We ill not cover graph based data metamodels in this presentation
29. Aggregates
The relational model takes information
you want to store and divides it into
rows.
Rows are lists of simple data values.
Rows are the unit of data operation
Aggregate orientation recognizes that
often times data units can be more
complex and can have nested lists and
record structures
30. Aggregates
The relational model takes information you
want to store and divides it into rows.
In RDBMS rows are lists of simple data
values.
In RDBMS rows are the unit of data
operation
Aggregate orientation recognizes that often
times data units can be more complex and
can have nested lists and record structures
With Aggregates, aggregates are the unit
of data operation
33. Consequences of Aggregate
Orientation
Relations capture data elements and
relations, but not aggregates.
Aggregates are really “chunks” of data
that are typically retrieved and operated
on as an interaction unit.
Aggregates are about how the data is
being used.
RDBMS do not have knowledge of
aggregate structure and cant use it to
store and distribute data
34. Consequences of Aggregate
Orientation
So, RDBMS are aggregate-ignorant. Is
that a bad or good thing? Its both
Its good if you need to access and use
the data in many different ways – if you
don’t have a primary structure for
manipulating your data
Its bad if you want to run on a cluster.
Aggregates are great on clusters
because you can distribute them across
nodes
35. Consequences of Aggregate
Orientation
Aggregate orientation allows you to
operate many logical data items (in the
aggregate) by updating the aggregate
atomically
Aggregate oriented NoSQL databases
can be said to support transactions on
single aggregates, but not across
aggregates
36. Key-Value & Document Data
Models
Both types of databases have a key or
Id that is mapped to an aggregate data
structure in a virtual table
With key-value NoSQL dbs, we can only
access the aggregate by looking up its
key
With document databases we can also
look up aggregates by fields in the
aggregate
37. Key-Value & Document Data
Models
Examples of Key-Value NoSQL dbs are
Redis
Examples of Document NoSQL dbs are
Mongodb
Couchbase
SimpleDB
38. Column-Family Data Models
These NoSQL databases where
influenced by Google’s BigTable
The Columnar is a two-level aggregate
structure
There is a key (row identifier) that maps to
the aggregate of interest
The aggregate is a map of more detailed
values – these are referred to as columns
40. Column-Family Data Models
Column-family dbs organize columns
into families
The data is row-oriented
Each row is an aggregate (eg. Customer
with id 1234)
The data is column-oriented
Each column family defines a record type
(customer profile)
But, columns can also be dynamic and
unique (to model lists)