Googol Data

Googol records (with MySQL)
IPC | October 2008 | Alex Aulbach

Definition: Googol

10100

or

10 000 000 000 000 000 000 000 000 000 000 000 000
000 000 000 000 000 000 000 000 000 000 000 000
000 000 000 000 000 000 000 000 000

or

“Imaginable big number”

„Googol records“
© MAYFLOWER GmbH 2008 2

Overview

What will the future bring for databases?

Is the principal way to access data the best?

Patterns (or suggestions)

and showing how that could work with MySQL.

Discuss!


The (performance) future of the web

Only 10-20 % of world population are “in the Internet”.

How should it be with 80 % ?



World population is growing and people get older.



More specialized databases

More ways to access them

Much easier to access

Sharing knowledge vs. closed knowledge: Who wins?

Services become more dependent to others

The web grows faster than Moores Law!
(Moores Law: “Only” Factor 1000 in 20 years.)


What does this mean us?

We will surely come into problems

But cannot say when, where and why

No Boss. The data belongs to everyone

It’s like new roads

It’s “Real-live”!


Consequences of growth

New hardware will no longer solve speed problems

Even new database will not

Even a rewrite of the application won’t

Need to rethink the problems from scratch!


Of course...

… need for splitting, sharding, partitioning, cluster etc.

… need to plan growth from beginning of the project.

… hardware resources can no longer be planned.

… distinct importance of data.

… estimate instead of being correct.


But ...

Is this enough?


Patterns (or better: suggestions)

Brain storage engine.

Reading differs from writing.

Redundancy and specialization.

The storage itself can keep the information.

Time (and sleep).

The journey is the reward.


1 :: Brain storage engine :: 1

Short term memory (working memory)

Unsorted, unfiltered, any data
Fast read
Very much fast updates/changes
Remembers which data is changed/invalid
Limited


1 :: Brain storage engine :: 2

Long-term memory

Presorted, well filtered data
Unlimited (well, more or less)
Extremely fast read access (sometimes)
Updates/inserts by repeating in working memory
Sleep helps to better store


How does that model fit into real life?

Nobody awaits to find old things fast

Telephone-books

90/10-Problems


Show

Searching in long term memory.
Scaling of working/long-term memory
vs.
one table with inserts/updates/deletes.


2 :: Reading differs from writing

Look at the physical processes

Reading with the fingertips:
No read and write at the same time

Handling reading and writing as different aspects of the
same thing is a compromise

Only specialization enables good optimization


Reader/Writer: Simplest layout


The web as storage?


Web can work like this


Recursive definition of the catalog


Scaling, setup as “black box”


Share everything


Comments

How does this scale?

What doesn’t work with this?


3 :: Redundancy and specialization :: 1

We cannot backup a googol

Nobody needs backup, but everybody needs to restore



Redundancy:

Store the information on many places
Store more important information on more places

Specialization:
“Materialized views”
EAV modeling and pivoting
Take ideas from data warehouses
and repositories



The wheel comes full circle:

More important: more access.
More access: More need for redundancy.
More redundancy: more speed and reliability.
More speed and reliability: more important.


Implementation with Reader/Writer


4 :: The storage itself
can keep the information.

“A storage has always physical limitations.
A logical information of data which belongs together
doesn't have any physical limitations.”

Alex Aulbach, Sept. 2008


The index is the problem!

The googol-universe is limited.
The index can take “half of the galaxies”.
Only the “rest” can be used for the data.

Less index means:
Faster search in the “needed” index.
Less time to write data and index.
Less time to warm up.
More space for the records.


Show

Access full table or split data into several parts.
Index-size
Write
Presorted tables


5 :: Time (and sleep) :: 1

Human brain: Only three bits per second!

We all have been babies.

Trust! Just wait and see.

Developers (and customers) need to think in decades
not in days till to the project-end.


5 :: Sleep (and time) :: 2

Again human brain: Learns while sleeping!
Why not apply this for databases?

Premise: Redundancy!
Dolphins sleep only with one hemisphere at a time.

The wheel comes full circle
Redundancy.
Distinct read and write.


Show

Well, I can’t show this, because it takes … time.


6 :: The journey is the reward

Future: Not so important how to search, but where.

Store step by step where to find the result, not the result.

You can find faster ways only by trying a shortcut.

It comes full circle:
Search many different ways and take the fastest.
While sleeping try out new things (dreaming).


Conclusion

Dreams may come true while sleeping.

We must invent now the tools
to solve the problems of the future.

Speed is not a matter of hardware
but of how things are done.

Never take speed as stated:
In a googol-universe wormholes exists!

Moores Law may help, but do not trust em.


Thank you!

Alex Aulbach
Mayflower GmbH
Pleichertorstr. 2
97070 Würzburg, Germany
+49 (931) 35 9 65 - 0
alex.aulbach@mayflower.de

Googol Data

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (11)

Similaire à Googol Data

Similaire à Googol Data (20)

Plus de Mayflower GmbH

Plus de Mayflower GmbH (20)

Dernier

Dernier (20)

Googol Data