Presentation by Stewart Bryson, Rittman Mead, on real-time data warehousing techniques using the Oracle Database and Oracle Business Intelligence Enterprise Edition. As delivered at Collaborate'11, Orlando, April 2011.
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Real-Time Data Warehousing using Oracle Database and OBIEE - Collaborate'11
1. Real-Time DW with OBIEE and Oracle Database
Stewart Bryson, Managing Director, Rittman Mead America
Collaborate, Orlando, April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 1
2. Who am I?
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 2
3. • Stewart Bryson, Managing Director, Rittman
Mead America
• Oracle BI/DW Architect and Delivery Specialist
• Oracle Database and BI Stack since 1996
• Regular Community Speaker
• Writer for Rittman Mead Blog:
http://www.rittmanmead.com/blog
• Developer of Transcend Framework
• Email : stewart.bryson@rittmanmead.com
• Twitter : @stewartbryson
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 3
4. Who is Rittman Mead?
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 4
5. • Oracle BI&DW Experts
• Consulting, Training and Support
• Global Company, Offices in the US, UK and India
• Clients in the UK, USA, Europe, Middle-East
• Voted UKOUG BI Partner of the Year 2008
• 16 Sessions at Oracle Open World 2010
• Twitter : @rittmanmead
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 5
6. Rittman Mead BI Forum, May 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 6
7. Registrations open for the Rittman Mead BI Forum, May 2011
• Brighton, UK, May 18-20th 2011
• Atlanta, GA, May 25-27th 2011
• Covers OBIEE, DW, EPM
• Guest speakers including:
‣ Jeff McQuigg
‣ Kevin McGinley
‣ Jean-Pierre Dijcks
‣ Mike Durran
‣ Shyam Varan Nath
‣ Philippe Lions
‣ Emiel van Bockel
• OBIEE 11g Masterclass with
Mark Rittman and Tony Heljula
• Full details at http://www.rittmanmead.com/biforum2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 7
8. What will this session answer?
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 8
9. • What do I mean by “real-time data warehousing?”
‣ Zero latency, or just reduced latency?
‣ Do I really mean “right-time data warehousing?”
• What can I use to reduce latency?
‣ What does OBIEE offer?
‣ What does the Oracle Database (and OWB)
offer?
• Can I blend these different offerings?
‣ Of course you can!
‣ It’s the name of the presentation
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 9
10. What do I mean by “real-time”?
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 10
11. • Traditional data warehouses
‣ Daily to weekly batch-load cycles
‣ Key-metrics are always delayed
‣ Information may not be “actionable”
• Real-time data warehouses
‣ Up to the minute or hour reporting
‣ Key-metrics are moving
‣ Information is actionable
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 11
12. An interval will always exist between the occurrence of a measurable
event and our ability to process that event as a “reportable fact”
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 12
13. L1&-6)&/(&)7#'.@@)&)2-#-"0)*#(@#.2$(/.23#'%-%#$%2#6%?)#'.@@)&.23#+%-)2$.)*=#L(&#)I%/0+)7#%#2)B#-&%2*%
+%-)2$"#-6%2#%#$6%23)#-(#%#&)3.(2%+#3&(10.23#(@#*-(&)*#-6%-#/%"#2))'#-(#>)#0&(0%3%-)'#-6&(136#/%2
%33&)3%-.(2#-%>+)*#4*))#L.31&)#M<=
K6)#'.?.*.(2#>)-B))2#'%-%#%$E1.*.-.(2#%2'#'%-%#0&)*)2-%-.(2#0&)*)2-*#%#D)"#E1)*-.(2#-6%-#%2"#$(/0%
$(2-)/0+%-.23#/(?.23#-(#%#&)%+,-./)#95#*"*-)/N#%&)#"(1#O1*-#.2-)&)*-)'#.2#0&(?.'.237#%*#&%0.'+"#%*#0(*
An interval will always exist between the occurrence of a measurable
(2#%+&)%'"#%$E1.&)'#'%-%7#(&#'(#"(1#%+*(#2))'#-(#&)0(&-#(2#2)B+"#%'')'#@%$-1%+#.2@(&/%-.(2C
event and our ability to process that event as a “reportable fact”
L(&#)I%/0+)7#%#B(&D)&#.2#%#/(>.+)
FIGURE 1: BI latency
$)2-&)#/%"#+.D)#-(#*))#.2@(&/%-.(2#
Event Report $1*-(/)&#.*P#%2'#-6)#+.D)+.6(('#-6%
$1*-(/)&P*#')/(3&%06.$#B(1+'#*B
*100+.)&=
K6.*#.2@(&/%-.(2#2))'*#-(#>)#0&)*)2
%&)#*0)%D.23#-(#-6)/<#%2'#0&)@)&%>
+.2),(@,>1*.2)**#%00+.$%-.(2#-6%-#/%
.2-)&%$-.(2=#R(B)?)&#.-#.*#12+.D)+"#-6
Recording Propagation Aggregation Query >"#-6)#$%++#$)2-&)#%3)2-#@(&#$1*-(/)
0&)'.$-)'#$61&2#B(1+'#>)#.2@+1)2$)
6(1&*=
S(2?)&*)+"7#%#*"*-)/#-6%-#.*#1*.23#
Real-time query latency -(#%+)&-#-(#0(-)2-.%+#@&%1'#4*-($D#/%&
$(/0%2.)*#%&)#0(**.>+)#)I%/0+)#1*
D2(B#%>(1-#10,-(,-6),/(/)2-#%$-.?
***+,-./0.12345,416,+537
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 12
14. Business Case
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 13
15. • We want to be able to segment our reporting
between “real-time” needs and static reports
‣ Most users will want real-time numbers
delivered with dashboards
‣ Some users (financial-minded) don’t want
numbers to change each time they run a report
• Use the standard Oracle BI Stack
‣ Oracle Database Enterprise Edition
‣ Oracle Business Intelligence Enterprise Edition
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 14
16. Latency versus Query Performance
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 15
17. Latency versus Query Performance
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 15
18. Latency versus Query Performance
Latency
Query Performance
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 15
19. Latency versus Query Performance
Latency
Query Performance
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 15
20. Oracle Next-Generation Reference Architecture
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 16
21. Requirement: Identifying and
Capturing Change
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 17
22. Requirement: Change Data Capture
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 18
23. Requirement: Change Data Capture
• Oracle CDC
• OWB Trickle-feed
mappings
• Trigger-based
capture
• Recording
modification dates
• MINUS Command
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 18
24. Requirement: Change Data Capture
• Oracle CDC • Oracle CDC
• OWB Trickle-feed • OWB Trickle-feed
mappings mappings
• Trigger-based • Message Queues
capture • Views over
• Recording database links
modification dates • File movements
• MINUS Command
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 18
25. Requirement: Change Data Capture
• Oracle CDC • OWB Mappings
• Oracle CDC
and Process Flows
• OWB Trickle-feed • OWB Trickle-feed
mappings • Other ETL Tools
mappings
• Message Queues • Custom ETL code
• Trigger-based
capture • Views over • Custom transaction
database links code
• Recording
modification dates • File movements
• MINUS Command
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 18
26. Requirement: Change Data Capture
• Oracle CDC • OWB Mappings
• Oracle CDC
and Process Flows
• OWB Trickle-feed • OWB Trickle-feed
mappings • Other ETL Tools
mappings
• Message Queues • Custom ETL code
• Trigger-based
capture • Views over • Custom transaction
database links code
• Recording
modification dates • File movements
• MINUS Command
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 18
27. Oracle Asynchronous Change Data Capture
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 19
28. Oracle Asynchronous Change Data Capture
Hot Log
Online redo logs mined locally
Change tables are updated and stored locally
Hot Log Distributed
Online redo logs are mined locally
Change tables are updated and stored remotely
Autolog Online
Online redo logs are mined remotely
Change tables are updated and stored remotely
Autolog Archive
Archived redo logs are mined remotely
Change tables are updated and stored remotely
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 19
29. Oracle Asynchronous Change Data Capture
Hot Log
Online redo logs mined locally
Change tables are updated and stored locally
Hot Log Distributed
Online redo logs are mined locally
Latency
Change tables are updated and stored remotely
Autolog Online
Online redo logs are mined remotely
Change tables are updated and stored remotely
Autolog Archive
Archived redo logs are mined remotely
Change tables are updated and stored remotely
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 19
30. Requirement: Change Data Capture
• Oracle CDC • OWB Mappings
• Oracle CDC
and Process Flows
• OWB Trickle-feed • OWB Trickle-feed
mappings • Other ETL Tools
mappings
• Message Queues • Custom ETL code
• Trigger-based
capture • Views over • Custom transaction
database links code
• Recording
modification dates • File movements
• MINUS Command
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 20
31. Requirement: Change Data Capture
• Oracle CDC • OWB Mappings
• Oracle CDC
and Process Flows
• OWB Trickle-feed • OWB Trickle-feed
mappings • Other ETL Tools
mappings
• Message Queues • Custom ETL code
• Trigger-based
capture • Views over • Custom transaction
database links code
• Recording
modification dates • File movements
• MINUS Command
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 20
32. Requirement: Change Data Capture
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 21
33. Requirement: Change Data Capture
• Trickle-feed mappings allow the development of Streams
Capture and Apply processes as continuously running
mappings.
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 21
34. Federated OLTP/EDW Reporting
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 22
35. • Layering in OLTP data to provide the real-time
component
• General approach is to use a BI tool such as
OBIEE
• No real database or modeling involvement
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 23
36. Classic Batch Load Star Schema: Process Flow
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 24
37. Classic Batch Load Star Schema: Fact Table Load
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 25
38. Federation with OBIEE
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 26
39. OBIEE Metadata and Data Layers
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 27
40. OBIEE Metadata and Data Layers
• Physical Data Sources, such
as relational databases, Relational, OLAP, File and
Application Data Sources
OLAP Cubes and application
source adapters
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 27
41. OBIEE Metadata and Data Layers
• A Semantic Model made up
of Metrics, dimensions,
Enterprise Semantic Model
hierarchies and calculations,
derived from...
• Physical Data Sources, such
as relational databases, Relational, OLAP, File and
Application Data Sources
OLAP Cubes and application
source adapters
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 27
42. OBIEE Metadata and Data Layers
• The Web Catalog contains
definitions of reports, alerts
Web Catalog
dashboards, KPIs,
based on...
• A Semantic Model made up
of Metrics, dimensions,
Enterprise Semantic Model
hierarchies and calculations,
derived from...
• Physical Data Sources, such
as relational databases, Relational, OLAP, File and
Application Data Sources
OLAP Cubes and application
source adapters
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 27
43. Flow of Data Through the Three-Layer Semantic Model
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 28
44. Flow of Data Through the Three-Layer Semantic Model
Simplification of the Data Model
Integration of Disparate DataSources
Addition of Business Logic and Calculations
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 28
45. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 29
46. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 29
47. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 29
48. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 29
49. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 30
50. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 30
51. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 30
52. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 31
53. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 31
54. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 31
55. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 31
56. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 32
57. Logical Table Sources Map to Physical Models
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 32
58. Logical Table Sources Respect the Physical Model
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 33
59. Logical Table Sources Respect the Physical Model
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 33
60. Logical Table Sources Respect the Physical Model
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 33
61. WITH
SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2,
T43901.CUST_LAST_NAME as c3,
T43971.SAL_AMT as c4
from
GCBC_CRM.CUSTOMERS T43901,
GCBC_POS.POS_TRANS T43971,
GCBC_POS.POS_TRANS_HEADER T43978
where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE
('2010-09-18 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T43978.TRANS_DATE )
union all
select T44042.CUSTOMER_FIRST_NAME as c2,
T44042.CUSTOMER_LAST_NAME as c3,
T44105.AMOUNT as c4
from
GCBC_EDW.CUSTOMER_DIM T44042,
GCBC_EDW.SALES_FACT T44105
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )),
SAWITH1 AS (select sum(D3.c4) as c1,
D3.c2 as c2,
D3.c3 as c3
from
SAWITH0 D3
group by D3.c2, D3.c3)
select distinct 0 as c1,
D2.c2 as c2,
D2.c3 as c3,
D2.c1 as c4
from
SAWITH1 D2
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 34
62. WITH
SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2,
T43901.CUST_LAST_NAME as c3,
T43971.SAL_AMT as c4
from
GCBC_CRM.CUSTOMERS T43901,
GCBC_POS.POS_TRANS T43971,
GCBC_POS.POS_TRANS_HEADER T43978
where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE
('2010-09-18 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T43978.TRANS_DATE )
union all
select T44042.CUSTOMER_FIRST_NAME as c2,
T44042.CUSTOMER_LAST_NAME as c3,
T44105.AMOUNT as c4
from
GCBC_EDW.CUSTOMER_DIM T44042,
GCBC_EDW.SALES_FACT T44105
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )),
SAWITH1 AS (select sum(D3.c4) as c1,
D3.c2 as c2,
D3.c3 as c3
from
SAWITH0 D3
group by D3.c2, D3.c3)
select distinct 0 as c1,
D2.c2 as c2,
D2.c3 as c3,
D2.c1 as c4
from
SAWITH1 D2
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 34
63. WITH
SAWITH0 AS ((select T43901.CUST_FIRST_NAME as c2,
T43901.CUST_LAST_NAME as c3,
T43971.SAL_AMT as c4
from
GCBC_CRM.CUSTOMERS T43901,
GCBC_POS.POS_TRANS T43971,
GCBC_POS.POS_TRANS_HEADER T43978
where ( T43901.CUST_ID = T43978.CUST_ID and T43971.TRANS_ID = T43978.TRANS_ID and TO_DATE
('2010-09-18 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') < T43978.TRANS_DATE )
union all
select T44042.CUSTOMER_FIRST_NAME as c2,
T44042.CUSTOMER_LAST_NAME as c3,
T44105.AMOUNT as c4
from
GCBC_EDW.CUSTOMER_DIM T44042,
GCBC_EDW.SALES_FACT T44105
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY ) )),
SAWITH1 AS (select sum(D3.c4) as c1,
D3.c2 as c2,
D3.c3 as c3
from
SAWITH0 D3
group by D3.c2, D3.c3)
select distinct 0 as c1,
D2.c2 as c2,
D2.c3 as c3,
D2.c1 as c4
from
SAWITH1 D2
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 34
64. WITH
SAWITH0 AS (select sum(T44105.AMOUNT) as c1,
T44042.CUSTOMER_FIRST_NAME as c2,
T44042.CUSTOMER_LAST_NAME as c3,
T45416.CALENDAR_YEAR as c4
from
GCBC_EDW.DATE_DIM T45416,
GCBC_EDW.CUSTOMER_DIM T44042,
GCBC_EDW.SALES_FACT T44105
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and
T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') )
group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)
select distinct 0 as c1,
D1.c2 as c2,
D1.c3 as c3,
D1.c4 as c4,
D1.c1 as c5
from
SAWITH0 D1
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 35
65. WITH
SAWITH0 AS (select sum(T44105.AMOUNT) as c1,
T44042.CUSTOMER_FIRST_NAME as c2,
T44042.CUSTOMER_LAST_NAME as c3,
T45416.CALENDAR_YEAR as c4
from
GCBC_EDW.DATE_DIM T45416,
GCBC_EDW.CUSTOMER_DIM T44042,
GCBC_EDW.SALES_FACT T44105
where ( T44042.CUSTOMER_KEY = T44105.CUSTOMER_KEY and T44105.SALES_DATE_KEY = T45416.DATE_KEY and
T45416.SQL_DATE = TO_DATE('2003-01-09 00:00:00' , 'YYYY-MM-DD HH24:MI:SS') )
group by T44042.CUSTOMER_FIRST_NAME, T44042.CUSTOMER_LAST_NAME, T45416.CALENDAR_YEAR)
select distinct 0 as c1,
D1.c2 as c2,
D1.c3 as c3,
D1.c4 as c4,
D1.c1 as c5
from
SAWITH0 D1
• The BI Server understands from the Fragmentation
content how to “optimize” the query
• Only one LTS is needed to provide the answer
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 35
66. Demonstration
Fragmentation with OLTP and EDW
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 36
67. • Strengths
‣ Low latency
‣ No complex modeling or ETL
‣ Incorporated into any standard dimensional
EDW
• Weaknesses
‣ Performance
‣ Difficult to conform multiple systems
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 37
68. EDW with a Real-Time Component
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 38
69. • Modeling complimentary real-time structures in an
EDW
• Use functionality of the Oracle Database and OWB
• Complex ETL and Database interaction
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 39
70. The Real-Time Partition
Fact Table
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 40
71. The Real-Time Partition
Fact Real-Time
Fact Table
Partition
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 40
72. The Real-Time Partition
Fact Real-Time
Fact Table
Partition
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 40
73. The Real-Time Partition
• All ETL processing of fact records are directed at
the Real-Time Partition using Micro-Batch
• No indexes, small data set
• Partition Exchange at the end of the day
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 41
74. The Real-Time Partition
• All ETL processing of fact records are directed at
the Real-Time Partition using Micro-Batch
• No indexes, small data set
• Partition Exchange at the end of the day
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 41
75. The Real-Time Partition
• All ETL processing of fact records are directed at
the Real-Time Partition using Micro-Batch
• No indexes, small data set
• Partition Exchange at the end of the day
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 41
76. The Real-Time Partition
• All ETL processing of fact records are directed at
the Real-Time Partition using Micro-Batch
• No indexes, small data set
• Partition Exchange at the end of the day
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 41
77. Early-Arriving Facts
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 42
78. What if Type 2 dimension changes haven’t arrived yet?
• Grab the most recent dimension record when that
entity already exists
• Use a default “Not Known Yet” record when they
don’t
• Prior to the partition exchange, we run our late-
arriving dimension loads
• We update the Real-Time Partition with the correct
surrogate keys
‣ OWB generates an efficient MERGE
‣ Small data sets in the Real-Time Partition
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 43
79. Real-Time Integration: Fact Table RT Load
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 44
80. Real-Time Integration: Fact Table RT Load
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 44
81. Real-Time Integration: Correct Dimensions
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 45
82. Real-Time Integration: “Closing the Books” on the Day
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 46
83. Plugging it together with OBIEE (Non-Fragmentation)
• Using standard database
functionality (UNION ALL)
• Could be done with a view
• Using Surrogate Keys in the
RT Partition
• OBIEE doesn’t have to
manage different LTS’s
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 47
84. Plugging it together with OBIEE (Non-Fragmentation)
• Using standard database
functionality (UNION ALL)
• Could be done with a view
• Using Surrogate Keys in the
RT Partition
• OBIEE doesn’t have to
manage different LTS’s
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 47
85. Plugging it together with OBIEE (Non-Fragmentation)
• Using standard database
functionality (UNION ALL)
• Could be done with a view
• Using Surrogate Keys in the
RT Partition
• OBIEE doesn’t have to
manage different LTS’s
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 47
86. Demonstration
EDW with a Real-Time Component
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 48
87. Summary
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 49
88. • Latency versus performance and scalability
• Latency and performance versus degree of
difficulty
• Different variations on the two test cases
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 50
89. Real-Time DW with OBIEE and Oracle Database
Stewart Bryson, Managing Director, Rittman Mead America
Collaborate, Orlando, April 2011
T : (888) 631 1410 or +44 (0) 8446 697 995 E : info@rittmanmead.com W: www.rittmanmead.com
Thursday, 21 April 2011 51