Contenu connexe
Similaire à 2012 summarytables (20)
2012 summarytables
- 2. 2 Copyright © 2012, Oracle and/or its affiliates. All rights
reserved.
Insert Information Protection Policy Classification from Slide 8
ORACLE
PRODUCT
LOGO
MySQL: Summary Tables
Keith Larson
keith.larson@oracle.com
MySQL Community Manager
sqlhjalp.blogspot.com
sqlhjalp.com/pdf/2012_summarytables.pdf
- 3. 3 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Safe Harbor Statement
The following is intended to outline our general product
direction. It is intended for information purposes only,
and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or
functionality, and should not be relied upon in making
purchasing decisions.
The development, release, and timing of any features or
functionality described for Oracle’s products remains at
the sole discretion of Oracle.
- 4. 4 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
localhost ~]$ whoami
Keith Larson
keith.larson@oracle.com
MySQL Community Manager
sqlhjalp.blogspot.com
Started with MySQL 3.23 during the dot.com days.
Primary real world work was with a MySQL InnoDB delayed replicated environment
that easily held over 4 billion rows of user data. Did a lot of data mining of that data
over the years.
Numerous other sites developed on LAMP stack over the last 13 years.
Also a movie buff :)
Who are you?
DBAs?
Developers?
sqlhjalp.blogspot.com/2012/02/summary-tables-with-mysql.html
- 5. 5 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Program Agenda
MySQL Summary Tables are not a new concept. They are
exactly what they sound like, basically a summary of existing
data. We will review how to take advantage of summary table
information and ETL the processes to provide quick results for
your application.
• A Situation
• Some Solutions
• Resources to learn more
- 6. 6 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
“I know why you're here, ... I know what you've been
doing... why you hardly sleep, why you live alone, and why
night after night, you sit by your computer. You're looking
for him. I know because I was once looking for the same
thing. And when he found me, he told me I wasn't really
looking for him. I was looking for an answer. It's the
question that drives us.... “
-- Trinity , The Matrix
“Perhaps we are asking the wrong questions.”
-- Agent Brown , The Matrix
http://keikonium.deviantart.com/art/The-Matrix-Background-32136837
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
- 7. 7 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Temet Nosce - thine own-self you must know.
How well do you know your data....
What if your data was as big as the matrix?
http://thematrix101.com/media/still/photo-mat_temetnosce.jpg
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
●
How to sort data efficiently
●
How to data mine all of this data
●
Reporting of data trends takes a long time
●
How would you gather data for rendering
A Situation
- 8. 8 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
http://www.grinningplanet.com/3001/mp3news_other/matrix-pods_222x127.jpg
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
Table of all the structures
CREATE TABLE `structure` (
`structure_id` int(5) unsigned NOT NULL auto_increment,
`latitude` Float (10,6),
`longitude` Float (10,6),
`date_online` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`levels` int(5) unsigned DEFAULT '0',
`max_pods` int(5) unsigned DEFAULT '0',
`ip` INT UNSIGNED NOT NULL,
`communication_protocol`
enum('OFF','IP.PIPE','IP.UDP','IP6.PIPE','IP6.UDP','IP.SPIPE','IP6.SPIPE','SNA.PIPE'
) default NULL,
PRIMARY KEY (`structure_id`),
KEY `date_online` (`date_online`)
) ENGINE=InnoDB ;
The Situation
- 9. 9 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
The Situation
http://www.tony5m17h.net/MatrixNet.gif
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
Table of all the pods
CREATE TABLE `pod` (
`pod_id` int(5) unsigned NOT NULL auto_increment,
`structure_id` int(5) unsigned DEFAULT '0',
`online_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`offline_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`level` int(5) unsigned DEFAULT '0',
`position` int(5) unsigned DEFAULT '0',
`ip` INT UNSIGNED NOT NULL,
`communication_protocol`
enum('OFF','IP.PIPE','IP.UDP','IP6.PIPE','IP6.UDP','IP.SPIPE','IP6.SPIPE','SNA.PIPE'
) default NULL,
UNIQUE KEY `pods` (`pod_id`,`structure_id`,`online_date`),
KEY `online_date` (`online_date`) ,
KEY `offline_date` (`offline_date`)
) ENGINE=InnoDB ;
- 10. 10 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE `human` (
`human_id` int(5) unsigned NOT NULL auto_increment,
`human_key` varchar(32) default NULL,
`pod_id` int(5) unsigned DEFAULT '0',
`online_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`ip` INT UNSIGNED NOT NULL,
`offline_date` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`human_id`),
UNIQUE KEY `human` (`human_id`,`pod_id`,`online_date`),
KEY `online_date` (`online_date`) ,
KEY `offline_date` (`offline_date`)
) ENGINE=InnoDB ;
www.amazon.com/ANDERSON-REEVES-Action-Figure-Accessories/dp/B003YR64HY
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
The Situation
Table of all the humans
- 11. 11 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE `data` (
`data_id` int(5) unsigned NOT NULL auto_increment,
`question` varchar(100) default NULL,
`create_date` datetime default NULL,
`date_updated` datetime NOT NULL default '0000-00-00 00:00:00',
PRIMARY KEY (`data_id`),
KEY `question` (`question`(5))
) ENGINE=InnoDB ;
mysql> select data_id , question from data limit 7;
+---------+-------------+
| data_id | question |
+---------+-------------+
| 1 | First Name |
| 2 | Last Name |
| 3 | Gender |
| 4 | Address |
| 5 | City |
| 6 | State |
| 7 | Zip |
+---------+------------+
The Situation
www.amazon.com/ANDERSON-REEVES-Action-Figure-Accessories/dp/B003YR64HY
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
Table of all the humans
- 12. 12 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE `human_data` (
`human_data_id` bigint(12) unsigned NOT NULL auto_increment,
`human_id` int(9) unsigned default NULL,
`data_id` int(5) unsigned default NULL,
`value` varchar(100) default NULL,
PRIMARY KEY (`human_data_id`),
UNIQUE KEY `hd_id` (`human_id`,`data_id`),
KEY `dv` (`data_id`,`value`(5))
) ENGINE=InnoDB;
Table of all the humans
The Situation
www.amazon.com/ANDERSON-REEVES-Action-Figure-Accessories/dp/B003YR64HY
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
+---------------+----------+---------+------------------------------+
| human_data_id | human_id | data_id | value |
+---------------+----------+---------+------------------------------+
| 1 | 1 | 1 | Thomas |
| 2 | 1 | 2 | Anderson |
| 3 | 1 | 3 | M |
| 13 | 1 | 4 | 1060 West Addison Street |
| 4 | 1 | 5 | Chicago |
| 5 | 1 | 6 | Illinois |
| 6 | 1 | 7 | 60613 |
| 7 | 1 | 16 | Lower Downtown, Capitol City |
| 8 | 1 | 17 | 03 |
| 9 | 1 | 18 | 11 |
| 10 | 1 | 19 | 1962 |
| 11 | 1 | 20 | Central West Junior High |
| 12 | 1 | 21 | Owen Paterson High School |
+---------------+----------+---------+------------------------------+
- 13. 13 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
www.amazon.com/ANDERSON-REEVES-Action-Figure-Accessories/dp/B003YR64HY
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
CREATE TABLE `job` (
`job_id` int(5) unsigned NOT NULL DEFAULT '0',
`title` varchar(15) DEFAULT NULL,
`salary` decimal(5,3) default '0.000',
PRIMARY KEY (`job_id`)
) ENGINE=InnoDB ;
CREATE TABLE `color` (
`color_id` int(5) unsigned NOT NULL DEFAULT '0',
`color` enum('Brown','Blue','Hazel','Green','Red','Yellow','Blonde','Gold') default NULL,
PRIMARY KEY (`color_id`)
) ENGINE=InnoDB ;
The Situation
Table of all the humans
- 14. 14 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE `eyes` (
`eyes_id` int(5) unsigned NOT NULL auto_increment,
`color_id` int(5) unsigned DEFAULT '0',
`species_id` int(5) unsigned DEFAULT '0',
`style_id` int(5) unsigned DEFAULT '0',
`shape_id` int(5) unsigned DEFAULT '0',
PRIMARY KEY (`eyes_id`)
) ENGINE=InnoDB ;
CREATE TABLE `hair` (
`hair_id` int(5) unsigned NOT NULL auto_increment,
`style_id` int(5) unsigned DEFAULT '0',
`color_id` int(5) unsigned DEFAULT '0',
`shape_id` int(5) unsigned DEFAULT '0',
PRIMARY KEY (`hair_id`)
) ENGINE=InnoDB ;
The Situation
www.amazon.com/ANDERSON-REEVES-Action-Figure-Accessories/dp/B003YR64HY
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
Table of all the humans
- 15. 15 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
SELECT h.human_key , hd1.value as 'First Name' , hd2.value as 'Last Name' , hd3.value as
'Gender' , hd4.value as 'Address' , hd5.value as 'City' , hd6.value as 'State' , hd7.value as
'ZipCode' , hd16.value as 'Place of Birth' , CONCAT(hd17.value, '-',hd18.value,'-', hd19.value )
AS DOB, hd20.value as 'Junior High' , hd21.value as 'High School', INET_NTOA(h.ip) as ip
FROM human h
INNER JOIN human_data hd1 ON h.human_id = hd1.human_id AND hd1.data_id = 1
INNER JOIN human_data hd2 ON h.human_id = hd2.human_id AND hd2.data_id = 2
INNER JOIN human_data hd3 ON h.human_id = hd3.human_id AND hd3.data_id = 3
INNER JOIN human_data hd4 ON h.human_id = hd4.human_id AND hd4.data_id = 4
INNER JOIN human_data hd5 ON h.human_id = hd5.human_id AND hd5.data_id = 5
INNER JOIN human_data hd6 ON h.human_id = hd6.human_id AND hd6.data_id = 6
INNER JOIN human_data hd7 ON h.human_id = hd7.human_id AND hd7.data_id = 7
LEFT JOIN human_data hd16 ON h.human_id = hd16.human_id AND hd16.data_id = 16
LEFT JOIN human_data hd17 ON h.human_id = hd17.human_id AND hd17.data_id = 17
LEFT JOIN human_data hd18 ON h.human_id = hd18.human_id AND hd18.data_id = 18
LEFT JOIN human_data hd19 ON h.human_id = hd19.human_id AND hd19.data_id = 19
LEFT JOIN human_data hd20 ON h.human_id = hd20.human_id AND hd20.data_id = 20
LEFT JOIN human_data hd21 ON h.human_id = hd21.human_id AND hd21.data_id = 21
WHERE h.human_id = 1G
Lots of Data Across lots of Tables
The Situation
- 16. 16 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
*************************** 1. row ***************************
human_key: 63b28bb2be0bf40085646decce81ae9f
First Name: Thomas
Last Name: Anderson
Gender: M
Address: 1060 West Addison Street
City: Chicago
State: Illinois
ZipCode: 60613
Place of Birth: Lower Downtown, Capitol City
DOB: 03-11-1962
Junior High: Central West Junior High
High School: Owen Paterson High School
ip: 192.168.5.221
1 row in set (0.00 sec)
The Situation
Lots of Data Across lots of Tables
www.amazon.com/ANDERSON-REEVES-Action-Figure-Accessories/dp/B003YR64HY
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
- 17. 17 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
SELECT h.human_key , hd1.value as 'First Name' , hd2.value as 'Last Name' ,
hd3.value as 'Gender'
FROM human h
INNER JOIN human_data hd1 ON h.human_id = hd1.human_id AND hd1.data_id = 1
INNER JOIN human_data hd2 ON h.human_id = hd2.human_id AND hd2.data_id = 2
INNER JOIN human_data hd3 ON h.human_id = hd3.human_id AND hd3.data_id = 3
WHERE hd2.value like 'Mc%';
Lots of Data Across lots of Tables
The Situation
- 18. 18 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
+----------------------------------+----------------+-------------+--------+
| human_key | First Name | Last Name | Gender |
+----------------------------------+----------------+-------------+--------+
| 5631e6ee59a4175cd06c305840562ff3 | Shunichi | McAffer | M |
| d28c252abf26d8ffa69ad522c66b698e | Xudong | McAffer | F |
| 80d4c0e3d8705d1ae4cfd19dd89fa34a | Taizo | McAffer | M |
| 25a714b921e3a2ba54c452debb9b5584 | Yuuichi | McAffer | M |
...
| 601fa6a61d57b1f8a800d15eedfbd370 | Martien | McAlpine | F |
...
| 657e31ff3231b847d7604f6647a2dfc9 | Shir | McClurg | M |
..
| 72db86e4c73b9fabb4810562b236488e | Youpyo | McClure | M |
| 83b7afcfe4ad452c54ea6427b18960f8 | Khaled | McClurg | F |
..
| 2abbe47df459c35281fb6e86f19ea85e | Maria | McDermid | F |
...
| 999df4ce78b966de17aee1dc87111044 | Berhard | McFarlin | M |
1482 rows in set (0.15 sec)
Lots of Data Across lots of Tables
The Situation
- 19. 19 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Define snowflake
A snowflake schema is a logical arrangement of tables
in a multidimensional database such that the entity
relationship diagram resembles a snowflake in shape. ...
-- en.wikipedia.org/wiki/Snowflake_schema
http://thumbs.dreamstime.com/thumblarge_316/12227643929M1f6o.jpg
http://www.informix.com.ua/articles/rolap/rolap.htm
- 20. 20 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Solutions
Data Warehouse**
Noun:
A large store of data accumulated from a wide range of
sources within a company and used to guide
management decisions.
-- Google.com
** Same place, same time on Saturday for more on Data Warehouses.
- 21. 21 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Solutions
Bill Inmon's paradigm: Data warehouse is one part of the
overall business intelligence system. An enterprise has
one data warehouse, and data marts source their
information from the data warehouse. In the data
warehouse, information is stored in 3rd normal form.
Ralph Kimball's paradigm: Data warehouse is the
conglomerate of all data marts within the enterprise.
Information is always stored in the dimensional model.
http://www.1keydata.com/datawarehousing/inmon-kimball.html
- 22. 22 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Define Summary Table
A collection of one or more data elements that are
classified into some logical structure, either as
dimensions or data points
-- webster's dictionary
Solutions
- 23. 23 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE `the_matrix` (
`the_matrix_id` bigint(12) unsigned NOT NULL AUTO_INCREMENT,
`human_id` int(9) unsigned DEFAULT NULL,
`firstname` varchar(100) DEFAULT NULL,
`lastname` varchar(100) DEFAULT NULL,
`alias` varchar(50) DEFAULT NULL,
`gender` varchar(2) DEFAULT NULL,
`address` varchar(100) DEFAULT NULL,
`city` varchar(100) DEFAULT NULL,
`state` varchar(100) DEFAULT NULL,
`zipcode` varchar(100) DEFAULT NULL,
`dob_month` varchar(5) DEFAULT NULL,
`dob_day` varchar(5) DEFAULT NULL,
`dob_year` varchar(5) DEFAULT NULL,
`hair_color` varchar(50) DEFAULT NULL,
`hair_style` varchar(50) DEFAULT NULL,
`eye_color` varchar(50) DEFAULT NULL,
`date_updated` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
PRIMARY KEY (`the_matrix_id`),
UNIQUE KEY `human_id` (`human_id`),
KEY `date_updated` (`date_updated`)
) ENGINE=InnoDB
Solutions: Summary Table
- 24. 24 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
DELIMITER //
CREATE PROCEDURE build_the_matrix()
BEGIN
SELECT MAX(date_updated) INTO @matrix_date FROM the_matrix ;
REPLACE INTO the_matrix SELECT NULL, h.human_id, hd1.value as 'First Name' , hd2.value as 'Last Name',
hd13.value as alias , hd3.value as 'Gender' , hd4.value as 'Address' , hd5.value as 'City' , hd6.value as 'State' ,
hd7.value as 'ZipCode' , hd17.value as dob_month, hd18.value as dob_day, hd19.value as dob_year, hd23.value as
hair_color, hd24.value as hair_style, hd25.value as eye_color, hdd.date_updated
FROM human h
INNER JOIN human_data hdd ON h.human_id = hdd.human_id
INNER JOIN human_data hd1 ON h.human_id = hd1.human_id AND hd1.data_id = 1
INNER JOIN human_data hd2 ON h.human_id = hd2.human_id AND hd2.data_id = 2
LEFT JOIN human_data hd3 ON h.human_id = hd3.human_id AND hd3.data_id = 3
LEFT JOIN human_data hd4 ON h.human_id = hd4.human_id AND hd4.data_id = 4
LEFT JOIN human_data hd5 ON h.human_id = hd5.human_id AND hd5.data_id = 5
LEFT JOIN human_data hd6 ON h.human_id = hd6.human_id AND hd6.data_id = 6
LEFT JOIN human_data hd7 ON h.human_id = hd7.human_id AND hd7.data_id = 7
LEFT JOIN human_data hd13 ON h.human_id = hd13.human_id AND hd13.data_id = 13
LEFT JOIN human_data hd16 ON h.human_id = hd16.human_id AND hd16.data_id = 16
LEFT JOIN human_data hd17 ON h.human_id = hd17.human_id AND hd17.data_id = 17
LEFT JOIN human_data hd18 ON h.human_id = hd18.human_id AND hd18.data_id = 18
LEFT JOIN human_data hd19 ON h.human_id = hd19.human_id AND hd19.data_id = 19
LEFT JOIN human_data hd20 ON h.human_id = hd20.human_id AND hd20.data_id = 20
LEFT JOIN human_data hd21 ON h.human_id = hd21.human_id AND hd21.data_id = 21
LEFT JOIN human_data hd23 ON h.human_id = hd23.human_id AND hd23.data_id = 23
LEFT JOIN human_data hd24 ON h.human_id = hd24.human_id AND hd24.data_id = 24
LEFT JOIN human_data hd25 ON h.human_id = hd25.human_id AND hd25.data_id = 25
WHERE hdd.date_updated >= @matrix_date ;
END //
DELIMITER ;
Solutions: ExtractLoad EL
- 25. 25 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE EVENT build_the_matrix_10minruns
ON SCHEDULE
EVERY 10 MINUTE
COMMENT ' it is the question that drives us '
DO
CALL build_the_matrix();
show variables like '%event_s%';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| event_scheduler | ON |
+-----------------+-------+
mysql> set GLOBAL event_scheduler = ON || OFF ;
mysql> show processlistG
.....
User: event_scheduler
Host: localhost
db: NULL
Command: Daemon
Time: 26
State: Waiting for next activation
Info: NULL
Solutions: ExtractLoad EL
- 26. 26 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Solutions: Data Mining
mysql> SELECT SQL_no_cache COUNT(h.human_id) , g.value , s.value , c.value
FROM human h
INNER JOIN human_data g ON g.human_id = h.human_id AND g.data_id = 3
LEFT JOIN human_data s ON s.human_id = h.human_id AND s.data_id = 6
LEFT JOIN human_data c ON c.human_id = h.human_id AND c.data_id = 25
GROUP BY g.value , s.value , c.value;
+-------------------+-------+----------+-------+
| COUNT(h.human_id) | value | value | value |
+-------------------+-------+----------+-------+
| 119932 | F | NULL | NULL |
...
| 14 | M | New York | Blue |
| 13 | M | New York | Brown |
| 7 | M | New York | Green |
| 5 | M | New York | Hazel |
| 7 | M | Oregon | Blue |
| 8 | M | Oregon | Brown |
| 9 | M | Oregon | Green |
| 4 | M | Oregon | Hazel |
+-------------------+-------+------------+-------+
51 rows in set (5.26 sec)
- 27. 27 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Solutions: Data Mining
Data Mine the matrix
mysql> SELECT SQL_no_cache COUNT(human_id), gender , state , hair_color
FROM the_matrix
GROUP BY gender , state , hair_color
ORDER BY gender , state , hair_color ;
| COUNT(human_id) | gender | state | hair_color |
+-----------------+--------+------------+------------+
| 0 | NULL | NULL | NULL |
| 109196 | F | NULL | NULL |
| 163435 | M | NULL | NULL |
.....
| 1 | M | Illinois | BLACK |
| 25 | M | New York | BLACK |
| 14 | M | New York | BLOND |
| 12 | M | Oregon | BLACK |
| 16 | M | Oregon | BLOND |
+-----------------+--------+------------+------------+
28 rows in set (0.66 sec)
- 28. 28 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Solutions: Data Mining
mysql> SELECT SQL_no_cache COUNT(h.human_id) , g.value , s.value , c.value
FROM human h
INNER JOIN human_data g ON g.human_id = h.human_id
LEFT JOIN human_data s ON s.human_id = h.human_id
LEFT JOIN human_data c ON c.human_id = h.human_id
WHERE g.data_id = 3 AND s.data_id = 6 AND c.data_id = 25
GROUP BY g.value , s.value , c.value;
+-------------------+-------+----------+-------+
| COUNT(h.human_id) | value | value | value |
+-------------------+-------+----------+-------+
...
| 14 | M | New York | Blue |
| 13 | M | New York | Brown |
| 7 | M | New York | Green |
| 5 | M | New York | Hazel |
| 7 | M | Oregon | Blue |
| 8 | M | Oregon | Brown |
| 9 | M | Oregon | Green |
| 4 | M | Oregon | Hazel |
+-------------------+-------+------------+-------+
41 rows in set (0.14 sec)
- 29. 29 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE VIEW matrix.matrix_view AS
SELECT SQL_NO_CACHE COUNT(h.human_id) , g.value as gender, s.value as state , c.value as hair_color
FROM human h
LEFT JOIN human_data g ON g.human_id = h.human_id
LEFT JOIN human_data s ON s.human_id = h.human_id
LEFT JOIN human_data c ON c.human_id = h.human_id
WHERE g.data_id = 3 AND s.data_id = 6 AND c.data_id = 25
GROUP BY g.value , s.value , c.value;
+-------------------+--------+------------+------------+
| COUNT(h.human_id) | gender | state | hair_color |
+-------------------+--------+------------+------------+
.....
| 14 | M | New York | Blue |
| 13 | M | New York | Brown |
| 7 | M | New York | Green |
| 5 | M | New York | Hazel |
| 7 | M | Oregon | Blue |
| 8 | M | Oregon | Brown |
| 9 | M | Oregon | Green |
| 4 | M | Oregon | Hazel |
+-------------------+--------+------------+------------+
41 rows in set (0.04 sec)
Solutions: A View
- 30. 30 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
DELIMITER //
CREATE PROCEDURE build_the_matrix()
BEGIN
SELECT MAX(date_updated) INTO @matrix_date FROM the_matrix ;
REPLACE INTO the_matrix SELECT NULL, h.human_id, hd1.value as 'First Name' , hd2.value as 'Last Name', hd13.value as alias ,
hd3.value as 'Gender' , hd4.value as 'Address' , hd5.value as 'City' , hd6.value as 'State' , hd7.value as 'ZipCode' , hd17.value as dob_month,
hd18.value as dob_day, hd19.value as dob_year, hd23.value as hair_color, hd24.value as hair_style, hd25.value as eye_color,
hdd.date_updated
FROM human h
INNER JOIN human_data hdd ON h.human_id = hdd.human_id
INNER JOIN human_data hd1 ON h.human_id = hd1.human_id AND hd1.data_id = 1
INNER JOIN human_data hd2 ON h.human_id = hd2.human_id AND hd2.data_id = 2
LEFT JOIN human_data hd3 ON h.human_id = hd3.human_id AND hd3.data_id = 3
LEFT JOIN human_data hd4 ON h.human_id = hd4.human_id AND hd4.data_id = 4
LEFT JOIN human_data hd5 ON h.human_id = hd5.human_id AND hd5.data_id = 5
LEFT JOIN human_data hd6 ON h.human_id = hd6.human_id AND hd6.data_id = 6
LEFT JOIN human_data hd7 ON h.human_id = hd7.human_id AND hd7.data_id = 7
LEFT JOIN human_data hd13 ON h.human_id = hd13.human_id AND hd13.data_id = 13
LEFT JOIN human_data hd16 ON h.human_id = hd16.human_id AND hd16.data_id = 16
LEFT JOIN human_data hd17 ON h.human_id = hd17.human_id AND hd17.data_id = 17
LEFT JOIN human_data hd18 ON h.human_id = hd18.human_id AND hd18.data_id = 18
LEFT JOIN human_data hd19 ON h.human_id = hd19.human_id AND hd19.data_id = 19
LEFT JOIN human_data hd20 ON h.human_id = hd20.human_id AND hd20.data_id = 20
LEFT JOIN human_data hd21 ON h.human_id = hd21.human_id AND hd21.data_id = 21
LEFT JOIN human_data hd23 ON h.human_id = hd23.human_id AND hd23.data_id = 23
LEFT JOIN human_data hd24 ON h.human_id = hd24.human_id AND hd24.data_id = 24
LEFT JOIN human_data hd25 ON h.human_id = hd25.human_id AND hd25.data_id = 25
WHERE hdd.date_updated >= @matrix_date ;
END //
DELIMITER ;
Solutions: Populating Tables
- 31. 31 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE EVENT build_the_matrix
ON SCHEDULE
EVERY 2 MINUTE
COMMENT 'It is the question that drives us'
DO
CALL build_the_matrix();
mysql> show events;
Db: matrix
Name: build_the_matrix
Definer: root@localhost
Time zone: SYSTEM
Type: RECURRING
Execute at: NULL
Interval value: 2
Interval field: MINUTE
Starts: 2012-05-23 23:09:24
Ends: NULL
Status: ENABLED
Originator: 1
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: latin1_swedish_ci
mysql> select * from information_schema.eventsG
Solutions: Events
- 32. 32 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Triggers
delimiter |
CREATE TRIGGER some_trigger BEFORE INSERT ON Table_A
FOR EACH ROW BEGIN
INSERT INTO TEST_B SET a2 = NEW.a1;
DELETE FROM TEST_C WHERE a3 = NEW.a1;
UPDATE TEST_D SET b4 = b4 + 1 WHERE a4 = NEW.a1;
END; |
delimiter ;
mysql> show triggers;
Solutions: Populating Tables
- 33. 33 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Transform---
– This is your chance to clean data
– Enable the special needs of others
– Rectify Problems
– Know your data....
• Events
• Procedures
• Triggers
• Scripts
Solutions: ExtractTransformLoad ETL
- 34. 34 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
mysql> SELECT SQL_no_cache
COUNT(m.human_id) as Amount , UPPER(m.gender) as gender
FROM human h
INNER JOIN the_matrix m ON m.human_id = h.human_id
INNER JOIN pod p ON p.pod_id = h.pod_id
INNER JOIN structure s ON s.structure_id = p.structure_id
WHERE s.date_online BETWEEN 19850101000000 AND 19890101000000
GROUP BY m.gender;
+--------+--------+
| Amount | gender |
+--------+--------+
| 23783 | F |
| 35472 | M |
+--------+--------+
2 rows in set (2.08 sec)
Solutions: Reporting
- 35. 35 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
First start with our lowest common denominator beyond your root table.
CREATE TABLE `structure` (
`structure_id` int(5) unsigned NOT NULL DEFAULT '0',
`latitude` float(10,6) DEFAULT NULL,
`longitude` float(10,6) DEFAULT NULL,
`date_online` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
`levels` int(5) unsigned DEFAULT '0',
`max_pods` int(5) unsigned DEFAULT '0',
`ip` INT UNSIGNED NOT NULL,
`communication_protocol` enum('OFF','EMAIL','FTP','BATCH POST','BATCH GET','REAL
TIME GET','TRACKTAGS','FTP-SSL','REAL TIME POST','CUSTOM','POST') DEFAULT
NULL,
PRIMARY KEY (`structure_id`),
KEY `date_online` (`date_online`)
) ENGINE=InnoDB DEFAULT
Solutions: Reporting
- 36. 36 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE `report_structure_24hour` (
`report_structure_24hour_id` int(5) unsigned NOT NULL AUTO_INCREMENT,
`totals` int(5) unsigned DEFAULT '0',
`gender` varchar(15) DEFAULT NULL,
`date_online` date NOT NULL DEFAULT '0000-00-00',
PRIMARY KEY (`report_structure_24hour_id`),
KEY `date_online` (`date_online`)
) ENGINE=MEMORY;
REPLACE INTO report_structure_24hour
SELECT SQL_no_cache
NULL,
COUNT(m.human_id) as Amount , UPPER(m.gender) as gender ,
DATE_FORMAT(s.date_online, '%Y-%m-%d %h ') as date
FROM human h
INNER JOIN the_matrix m ON m.human_id = h.human_id
INNER JOIN pod p ON p.pod_id = h.pod_id
INNER JOIN structure s ON s.structure_id = p.structure_id
WHERE s.date_online >= NOW() - interval 24 hour
GROUP BY m.gender, date;
Solutions: Reporting
- 37. 37 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE `report_structure_perday` (
`report_structure_perday_id` int(5) unsigned NOT NULL AUTO_INCREMENT,
`totals` int(5) unsigned DEFAULT '0',
`gender` varchar(15) DEFAULT NULL,
`date_online` date NOT NULL DEFAULT '0000-00-00',
PRIMARY KEY (`report_structure_perday_id`),
KEY `date_online` (`date_online`)
) ENGINE=InnoDB;
REPLACE INTO report_structure_perday
SELECT SQL_no_cache
NULL,
COUNT(m.human_id) as Amount , UPPER(m.gender) as gender ,
DATE_FORMAT(s.date_online, '%Y-%m-%d') as date
FROM human h
INNER JOIN the_matrix m ON m.human_id = h.human_id
INNER JOIN pod p ON p.pod_id = h.pod_id
INNER JOIN structure s ON s.structure_id = p.structure_id
WHERE s.date_online >= NOW() - interval 1 week
GROUP BY m.gender, date;
Solutions: Reporting
- 38. 38 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE `report_structure_permonth` (
`report_structure_permonth_id` int(5) unsigned NOT NULL AUTO_INCREMENT,
`totals` int(5) unsigned DEFAULT '0',
`gender` varchar(15) DEFAULT NULL,
`date_online` date NOT NULL DEFAULT '0000-00-00',
PRIMARY KEY (`report_structure_permonth_id`),
KEY `date_online` (`date_online`)
) ENGINE=InnoDB;
REPLACE INTO report_structure_permonth
SELECT SQL_no_cache
NULL,
SUM(totals)as Amount, gender , DATE_FORMAT(date_online, '%Y-%m-00') as date
FROM report_structure_perday
WHERE date_online >= NOW() - interval 2 month
GROUP BY gender, date;
Solutions: Reporting
- 39. 39 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
CREATE TABLE `report_structure_per_year` (
`report_structure_year_id` int(5) unsigned NOT NULL AUTO_INCREMENT,
`totals` int(5) unsigned DEFAULT '0',
`gender` varchar(15) DEFAULT NULL,
`date_online` int(5) unsigned DEFAULT '0000',
PRIMARY KEY (`report_structure_year_id`),
KEY `date_online` (`date_online`)
) ENGINE=InnoDB;
REPLACE INTO report_structure_per_year
SELECT SQL_no_cache
NULL,
SUM(totals)as Amount, gender , DATE_FORMAT(date_online, '%Y') as date
FROM report_structure_permonth
WHERE date_online >= NOW() - interval 2 YEAR
GROUP BY gender, date;
Solutions: Reporting
- 40. 40 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Populate tables with :
DELIMITER //
CREATE PROCEDURE reporting_perday()
BEGIN
REPLACE INTO report_structure_perday
SELECT SQL_no_cache
NULL,
COUNT(m.human_id) as Amount , UPPER(m.gender) as gender , DATE_FORMAT(s.date_online, '%Y-%m-%d')
as date
FROM human h
INNER JOIN the_matrix m ON m.human_id = h.human_id
INNER JOIN pod p ON p.pod_id = h.pod_id
INNER JOIN structure s ON s.structure_id = p.structure_id
WHERE s.date_online >= NOW() - interval 1 week
GROUP BY m.gender, date;
END //
DELIMITER ;
CREATE EVENT reporting_perday
ON SCHEDULE
EVERY 24 HOUR
COMMENT 'It is the question that drives us'
DO
CALL reporting_perday();
Solutions: Reporting
- 41. 41 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Populate tables with :
DELIMITER //
CREATE PROCEDURE reporting_permonth()
BEGIN
REPLACE INTO report_structure_permonth
SELECT SQL_no_cache
NULL,
SUM(totals)as Amount, gender , DATE_FORMAT(date_online, '%Y-%m-00') as date
FROM report_structure_perday
WHERE date_online >= NOW() - interval 2 month
GROUP BY gender, date;
END //
DELIMITER ;
CREATE EVENT reporting_permonth
ON SCHEDULE EVERY 2 WEEK
COMMENT 'It is the question that drives us'
DO
CALL reporting_permonth();
Solutions: Reporting
- 42. 42 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Populate tables with :
DELIMITER //
CREATE PROCEDURE reporting_peryear()
BEGIN
REPLACE INTO report_structure_per_year
SELECT SQL_no_cache
NULL,
SUM(totals)as Amount, gender , DATE_FORMAT(date_online, '%Y') as date
FROM report_structure_permonth
WHERE date_online >= NOW() - interval 2 YEAR
GROUP BY gender, date;
END //
DELIMITER ;
CREATE EVENT reporting_peryear
ON SCHEDULE
EVERY 2 MONTH
COMMENT 'It is the question that drives us'
DO
CALL reporting_peryear();
Solutions: Reporting
- 43. 43 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
It all comes down to who do
you want to be?
The One or just someone ?
The Solutions are up to you.
http://images1.wikia.nocookie.net/__cb20070215050459/uncyclopedia/images/7/75/Neo.JPG
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
Solutions - Solution
www.amazon.com/ANDERSON-REEVES-Action-Figure-Accessories/dp/B003YR64HY
The Matrix copyright © 1999 - 2012, Warner Bros. Warner Bros. is the owner of all copyrights and trademark rights in The Matrix.
- 44. 44 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.
Resources
http://code.google.com/p/flexviews/
http://code.google.com/p/flexviews/
http://code.google.com/p/flexviews/
http://code.google.com/p/flexviews/
http://code.google.com/p/flexviews/
http://code.google.com/p/flexviews/
- 45. 45 Copyright © 2012, Oracle and/or its affiliates. All rights reserved.