NoSQL intro for YaJUG / NoSQL UG Luxembourg

NOSQL
BIG DATA
NEW DATABASES
Stuff.
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

Hey!

I won’t do
a demo.


I’m @stevenn
or
stevenn@outerthought.org


Houston,
we have
a problem.

We’re drowning.


Drowning
in a
Sea
of
Data.

Mountains of
Metadata.

The firehose
of UGC.

Still, we
can’t make
much sense
of it.


... and we throw
a lot of it away.

We regard
content as cost.


But data is an
opportunity.


Think about it.


advertisements
IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 15

recommendations

anything that sells

proﬁle harvesting

The future is for
datanerds.


Nerdy
enough?

This is what Big
Data is about:
new insights,
new business.

But first,
some history.


How did NOSQL happen?

2. simpliﬁcation

1. standardization
hierarchical databases

IMS
XMLDB RDBMS

OODBMS


How did NOSQL happen?

4. rethinking
the problem

RDBMS NOSQL

caching
denormalisation
sharding
replication ...
3. pain


Numbers of scale

http://qos.doubleclick.net/counters/


Numbers of scale

» Twitter does 12 M tweet displays

» ... per second.


Types of scaling
» scaling for usage » scaling types of ops
» volume of users » concurrent read
» volume of data » concurrent write

availability partioning
replication consistency

distributed systems


... and
distributed
systems are
HARD.

8 fallacies of distributed computing
» The network is reliable.

» Latency is zero.

» Bandwidth is inﬁnite.

Peter Deutsch and James Gosling
» The network is secure.

» Topology doesn't change.

» There is one administrator.

» Transport cost is zero.

» The network is homogeneous.


Data.

Trend 1: Data size
ExaBytes (10!") of data stored per year
988
1000
Each year more and
more digital data is
created. Over t wo
750 years we create more
digital data than all 623
the data created in
history before that.
500
397

253
250 161

0
2006 2007 2008 2009 2010
Data source: IDC 2007 3


Trend 2: Connectedness
Giant
Global
Graph (GGG)

Over time data has evolved to Ontologies
be more and more interlinked
and connected.
RDF
Hypertext has links,
Blogs have pingback,
Tagging groups all related data Folksonomies
Information connectivity

Tagging

Wikis User-generated
content
Blogs

RSS

Hypertext

Text documents
web 1.0 web 2.0 “web 3.0”

1990 2000 2010 2020 4


Trend 3: Semi-structure
! Individualization of content
• In the salary lists of the 1970s, all elements had exactly one job
• In Or 15? lists of the 2000s, we need 5 job columns! Or 8?
the salary

! All encompassing “entire world views”
• Store more data about each entity
! Trend accelerated by the decentralization of content generation
that is the hallmark of the age of participation (“web 2.0”)

5


Trend 4: Architecture

1980s: Mainframe applications

Application

DB

6



1990s: Database as integration hub

Application Application Application

DB

7



2000s: (moving towards) Decoupled services
with their own backend


DB DB DB

8



2000s: (moving towards) Decoupled services
with their own backend


DB DATA TIER
DB DB

8


For years, we
tried to squeeze
data into a
one-size-fits-all
container.

Also, the cost perspective


NOSQL,
the Data
Liberation
Front
(or: Polyglot Persistency)

Cambrian Explosion


Cambrian Explosion

N-O-SQL


The NOSQL footprint
free-structured or sparse data

NOSQL

MongoDB
CouchDB
neo4j

Cassandra

available (complexity)
simple operational

HBase

highly scalable and
constraints
ACID,

SQL

referential integrity,
typed data


Other axes of classiﬁcation
» Data Model

» Consistency

» Atomic test-and-set

» Secondary indexes

» Manageability

» Latency vs. Durability

» Read vs. Write Performance

» Dynamic Scaling

» Auto failover

» Compression Support

» Range Scanning

» Failure Scenarios


Data Model

» Key/Value

» Document

» Row Stores with Column Families

» Graphs


Other axes of classiﬁcation
» Data Model

» Consistency

» Atomic test-and-set

» Secondary indexes

» Manageability

» Latency vs. Durability

» Read vs. Write Performance

» Dynamic Scaling

» Auto failover

» Compression Support

» Range Scanning
http://huanliu.wordpress.com/2011/01/21/
» Failure Scenarios dimensions-to-use-to-compare-nosql-data-stores/


Hire a good
consultant.
(or become one, like Xebia, SFEIR,
Cloudera ...)


Data Processing


Map + Reduce


Hadoop: HDFS + MapReduce
» single ﬁlesystem + single execution-space


Processing large datasets with MR

» Beneﬁt from parallellisation

» Less modelling upfront (ad-hoc processing)

» Compartmentalized approach reduces
operational risks (aka robustness)
» AsterData et al. have SQL/MR hybrids for
huge-scale BI


LILY

Cloud-scale
content
storage & search

LILY
+

58

Lily

» provides scalable storage

» and scalable search

» with a fault-tolerant, distributed architecture

» automated index maintenance

» versioning, rich data types, Java+REST API

» based on HBase (NOSQL) and SOLR (Lucene)


Choosing a NoSQL store for Lily: step I
» automatic scaling to large data sets

» fault-tolerance

» ﬂexible datamodel with sparse data

» commodity hardware

» efﬁcient random access

» community-based open source

» Java if possible


Choosing a NoSQL store for Lily: step II

» need for consistency

» atomic single-row updates

» M/R for index regeneration


Choosing a NoSQL store for Lily: step III

HBase
» datamodel with column families and cell
versioning
» ordered tables with range scans

» HDFS for blob storage

» Apache


Lily

» scales to inﬁnity, and beyond
» open source
» Apache license (no strings attached)
» Java and REST API

» www.lilyproject.org

» subscription- and partnership-based
business model


distributed process coordination
and conﬁguration (ZooKeeper)

}
query update indexer
Lily
Lily Lily Store Server
store
client
node WAL MQ M/R

client

}
store
node 2ary WAL / HBase Region Server
documents
indexes MQ
client

store
node

} Hadoop DFS

REST

index
replica
inverted index

replica replica
} SOLR

lily simpliﬁed architecture

Key lessons learned
» unlearning normalization is very difﬁcult

» integrity checking in code = not so bad

» doing joins in code can be very liberating

» importance of keyspace design
» secondary indexing
» data de-normalization = size! (x3)

» schema vs. code ﬂexibility?

» distribution is everywhere
and you shouldn’t forget about it

Pssst. :-)
If you absolutely, positively want to see a
demo, go check http://outerthought.blip.tv/


Reading material

» Amazon Dynamo, Google BigTable, CAP

» http://nosql.mypopescu.com/

» http://nosql-database.org/

» http://twitter.com/nosqlupdate

» http://highscalability.com/


We’re growing
We’re hiring


Thank you !
for your attention
for your questions
» stevenn@outerthought.org

» @stevenn


NoSQL intro for YaJUG / NoSQL UG Luxembourg

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (6)

Similaire à NoSQL intro for YaJUG / NoSQL UG Luxembourg

Similaire à NoSQL intro for YaJUG / NoSQL UG Luxembourg (20)

Plus de NGDATA

Plus de NGDATA (8)

Dernier

Dernier (20)

NoSQL intro for YaJUG / NoSQL UG Luxembourg