9. OLTP vs DW
● many single-row
writes
● current data
● queries generated
by user activity
● < 1s response
times
● 1000's of users
● few large batch
imports
● years of data
● queries generated
by large reports
● queries can run for
minutes/hours
● 10's of users
10. OLTP vs DW
big data for
many
concurrent
requests to
small amounts
of data each
big data
for low
concurrency
requests to very
large amounts
of data each
15. data mining
the database where you don't know what's in
there, but you want to find out
● lots of data (TB to PB)
● mostly “semi-structured”
● data produced as a side effect of other
business processes
● needs CPU-intensive processing
18. BI/DSS/OLAP/Analytics
databases which support visualization of
large amounts of data
● data is fairly well understood
● most data can be reduced to categories,
geography, and taxonomy
● primarily about indexing
24. Extract, Transform, Load
● how you turn external raw data into useful
database data
● Apache logs → web analytics DB
● CSV POS files → financial reporting DB
● OLTP server → 10-year data warehouse
● also called ELT when the transformation is
done inside the database
25. Purpose of ETL/ELT
getting data into the data warehouse
● clean up garbage data
● split out attributes
● “normalize” dimensional data
● deduplication
● calculate materialized views / indexes
29. ELT Tips
think volume
● bulk processing or parallel processing
● no row-at-a-time, document-at-a-time
● insert into permanent storage should be
the last step
● no updates
34. standard relational
the all-purpose solution for not-that-big data
● adequate for all tasks
● but not excellent at any of them
● easy to use
● low resource requirements
● well-supported by all software
● familiar
● not suitable for really big data
39. MPP
cpu-intensive data warehousing
● data mining, some analytics
● supporting complex query logic
● moderately big data (1-200TB)
● drawbacks: proprietary, expensive
● now hybridizes
● with other types
44. column stores
for aggregations and transformations of
highly structured data
● good for BI, analytics, some archiving
● moderately big data (0.5-100TB)
● bad for data mining
● slow to add new data / purge data
● usually support compression
49. map/reduce
// Map
function (doc) {
emit(doc.val, doc.val)
}
// Reduce
function (keys, values, rereduce) {
// This computes the standard deviation of the mapped results
var stdDeviation=0.0;
var count=0;
var total=0.0;
var sqrTotal=0.0;
if (!rereduce) {
// This is the reduce phase, we are reducing over emitted values from
// the map functions.
for(var i in values) {
total = total + values[i];
sqrTotal = sqrTotal + (values[i] * values[i]);
}
count = values.length;
}
else {
// This is the rereduce phase, we are re-reducing previosuly
// reduced values.
for(var i in values) {
count = count + values[i].count;
total = total + values[i].total;
sqrTotal = sqrTotal + values[i].sqrTotal;
}
}
var variance = (sqrTotal - ((total * total)/count)) / count;
stdDeviation = Math.sqrt(variance);
// the reduce result. It contains enough information to be rereduced
// with other reduce results.
return {"stdDeviation":stdDeviation,"count":count,
"total":total,"sqrTotal":sqrTotal};
};
50. map/reduce vs. MPP
● open source
● petabytes
● write routines by
hand
● inefficient
● generic
● cheap HW / cloud
● DIY tools
● proprietary
● terabytes
● advanced query
support
● efficient
● specific
● needs good HW
● integrated tools
53. enterprise search
when you need to do DW with a huge pile of
partly processed “documents”
● does: light data mining, light BI/analytics
● best “full text” and keyword search
● supports “approximate results”
● lots of special features for web data
54. E.S. vs. C-Store
● batch load
● semi-structured
data
● uncompressed
● star schema
● sharding
● approximate results
● batch load
● fully normalized
data
● compressed
● snowflake schema
● parallel query
● exact results
59. SELECT MAX(concurrent)
FROM (
SELECT SUM(tally)
OVER (ORDER BY start)
AS concurrent
FROM (
SELECT start, 1::INT as tally
FROM events
UNION ALL
SELECT (start + duration), -1
FROM events )
AS event_vert) AS ec;
60. stream processing SQL
● replace multiple queries with a single
query
● avoid scanning large tables multiple times
● replace pages of application code
● and MB of data transmission
● SQL alternative to map/reduce
● (for some data mining tasks)
62. query results as table
● calculate once, read many time
● complex/expensive queries
● frequently referenced
● not necessarily a whole query
● often part of a query
● might be manually or automatically
updated
● depends on product
63. non-relational matviews
● CouchDB Views
● cache results of map/reduce jobs
● updated on data read
● Solr / Elastic Search “Faceted Search”
● cached indexed results of complex searches
● updated on data change
64. maintaining matviews
BEST: update matviews
at batch load time
GOOD: update matview according
to clock/calendar
FAIR: update matview on data
request
BAD for DW: update matviews
using a trigger
65. matview tips
● matviews should be small
● 1/10 to ¼ of RAM on each node
● each matview should support several
queries
● or one really really important one
● truncate + append, don't update
● index matviews like crazy
● if they are not indexes themselves
69. OLAP
● OnLine Analytical Processing
● Visualization technique
● all data as a multi-dimensional space
● great for decision support
● CPU & RAM intensive
● hard to do on really big data
● Works well with column stores
70. Contact
● Josh Berkus: josh@pgexperts.com
● blog: blogs.ittoolbox.com/database/soup
● twitter: @fuzzychef
● PostgreSQL: www.postgresql.org
● pgexperts: www.pgexperts.com
This talk is copyright 2011 Josh Berkus and is licensed under the Creative Commons Attribution
license. Many images were taken from google images and are copyright their original creators,
whom I don't actually know. Logos are trademark their respective owners, and are used here
under fair use.
