Oow2010 mead exadata

Exadata and Oracle Business Intelligence
Enterprise Edition Retail Case Study
Jon Mead
Managing Director - Rittman Mead Consulting
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com

Sunday, 26 September 2010

Overview



Overview

•Exadata

•OBIEE

•Agile development



Introductions

• Rittman Mead Consulting
‣ Oracle BI Consultancy (UK, US, India)
‣ Platinum Partner
‣ Long(est) running Oracle BI blog
‣ Annual BI Forum
‣ OBIEE Oracle Press book
‣ http://www.rittmanmead.com
‣ @rittmanmead
• Jon Mead
‣ Managing Director/Co-Founder
‣ BI Consultant
‣ @jonmead



Introductions

• On-line retailer
• Didn’t want to be named....
• Direct supplier
• Host market place



Background

T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.rittmanmead.com


Business Problem

It’s a simple case of supply and demand...

A retail organisation needs to be able to make
pricing and purchasing (supply) decisions
based on a near real time view of what is
happening in the its marketplace (demand).



Business Problem

Competitive
Advantage


Business Problem

BUT...

(Its’s not quite that simple)


Business Problem

Accounts
Fraud

Suppliers
Systems
Customers Buyers

Inventory
Competitors



What is needed

• Ability to process large volumes of data
‣ Volume of data
- 1000’s of transactions a minute at peak periods
‣ Complexity of the transformation
- Order lifecycle

• Ability to report on large volumes of data
‣ Year on year reporting
‣ Cross fact analysis
‣ Ad-hoc queries
‣ Deep historic analysis
‣ Near real time analysis during peak periods



What is needed

• Common and consistent business model
‣ Single abstraction of the business
‣ One unified tool to development and deliver

• Reduced development cycles
‣ Business is constantly evolving
‣ Agile development methodology
‣ Quick release cycles
‣ Early involvement from the business



In summary....

• Very fast changing market

• Key Drivers
‣ Optimise product line • Maximise profit
‣ Optimise pricing • Maximise customer satisfaction
‣ Optimise stock levels • Turn data into information quicker
than their competitors
• Coupled with
‣ Fraud analysis
‣ Customer tracking
‣ Order tracking



In summary....

Current Planned

Up to a monthly lag on
Data Near real time
certain data

Could take up to 24
Queries Near instant response
hours

Business Model Fragmented Consistent

Development Long time-frames Month long iterations



Typical Approach

• Multiple tools used to support reporting
‣ Each transactional system supplies its own reporting tool
‣ Multiple business models
• Data warehouse or marts for integrated reporting
• Complex aggregation model
‣ OLAP
• Potentially long development cycles
‣ Not visible to end-users
‣ High level of rework in the ETL development
• Static fact tables
‣ Difficult to re-load the data
• Focus on reduction of I/O during the design process
‣ Lots of performance tuning



New World Order

• New development methodology

• Real-time data access

• Data processing power

• Data querying power

• Single view of the business



New World Order

• New development methodology Agile approach

• Real-time data access Change Data Capture

• Data processing power Exadata

• Data querying power Exadata

• Single view of the business OBIEE



Solution

• Change Data Capture
‣ Asynchronous
• Oracle Warehouse Builder
‣ SCD Framework
• Oracle Reference Data Warehouse Architecture
‣ Staging
‣ ODS
‣ Performance layer
• OBIEE
‣ Dashboards
‣ Answers



What is Change Data Capture

• Asynchronous method of shipping log files
• Creates change tables
• Changed data windows can be extended, read and purged

• Replaced by Golden Gate?

• Ensure Supplemental Logging in enable (if you get the choice)



What is Exadata

• AKA The Oracle Database Machine
‣ Or Sun Oracle Database Machine
‣ Or Oracle Exadata Database Machine

• Combination of hardware and software

• First launched at Oracle Open World 2008
• Original (v1) machine was Oracle and HP...



Oracle and HP



What is Exadata?

• Oracle acquires Sun
• V2 release a year later
‣ OLTP
‣ Flash cache
‣ 11gR2 of the database
‣ Twice as fast for Data Warehousing

• Revised this year
‣ x2-2 - 8 2 core database servers
‣ x2-8 - 2 8 core database servers



Exadata V2 specification - hardware
• Database nodes
‣ Sun Fire x4170
‣ 2 quad-core CPU, 72GB RAM
‣ Infiniband cards
‣ Full rack has 8 database servers
• Infiniband switches
‣ Full rack has 3
‣ 2 for redundancy, 1 to connect to other machines
• Management switch
• Storage cells
‣ Sun Fire x4275
‣ 2 quad-core CPU, 24GB RAM
‣ From 12 x 600GB to 12 x 2 TB drives
‣ Infiniband cards
• Rack


Exadata V2 specification - software
• Database nodes
‣ Oracle Enterprise Linux 5.3
‣ Oracle Database 11g Release 2
- Oracle 11gR2 includes a substantial amount of code to support Exadata’s
features like smart scans and columnar compression, and this code is
simply unused in non-Exadata deployments.

• Storage cells
‣ This is where the magic happens....
‣ Set of software programs (CELLSRV, RS, MS etc) handling high-level I/O
processing



Exadata key features (V1 focused)

• Smart scan
‣ Reduce data transfer between database and storage servers
• Storage Indexes
‣ Partition pruning like effect
‣ Is Partitioning required?
• Hybrid Columnar Compression
‣ V2 feature (11gR2 feature, could upgrade V1)
• Flash Cache (V2)

• But also use Data Warehousing features



Smart Scan

• Data processing for Full Table Scan

Data Data Data Data Data Data

Memory Storage



Smart Scan

• Smart Scan for Full Table Scan

Data Data Data

Memory Storage



Compression

• Compressed data uses less I/O, but more CPU cycles

Data Data Data Data Data Data

Memory Storage



What is OBIEE?



Oracle Business Intelligence Enterprise Edition Plus

• Dashboards, Ad-hoc Reporting, Alerts, Microsoft Office Integration
‣ High quality graphical, role/user based views
‣ Multiple views of same data
• Point and click ease of use
• Heterogeneous database access - true Enterprise deployment
• Unified semantic/logical view of data from multiple sources
• Alerts, scheduling and distribution
• Powerful/visual metadata design tool
‣ Common Enterprise Information Model



Architecture



Next-Generation Reference DW Architecture



Staging Layer

Exadata

OWB
Foundation Layer

Performance Layer

Reporting Solution



Staging Layer



• Change Tables for CDC
‣ Temporary holding area for incremental data
‣ Asynchronous Autolog Online Change Data
Capture
• Reject Tables
‣ Foundation layer
‣ Performance layer
• Staging tables for Event Syslogs
• Elimination of “noise” data



Foundation Layer



Foundation Logical Model

• Process-neutral
• Devoid of business requirements
• Tables will be represented as they are in the
source systems
‣ Normalized structure maintained
‣ Additional elements added by CDC
‣ Primary key plus Transaction Date
‣ Transaction Source
• Event table will be modeled away from key value
pairs to represent true event data



Foundation Physical Model

• Partition tables on Transaction Date
‣ Segmentation of data
‣ Improved query performance
‣ Local indexes
• Aggregation in the foundation layer
‣ Materialized views for Query Rewrite
‣ Partition Change Tracking (PCT)
‣ Materialized views on Prebuilt Tables
‣ “Sessionizing” aggregation if desired



Performance Layer



Logical Model

• Data Bus Architecture
‣ Conformed dimensions
‣ Fact tables at the lowest level of granularity
• Hierarchies maintained in a single table whenever
possible
‣ Lowest-common denominator reporting tool
support
‣ Removal of complexity for self service model
‣ Better performance
• No aggregation in the logical model



Physical Model

• Partitioning
‣ Hash-partitioned dimension tables
‣ Range-partitioned fact tables
• Star transformation
• Aggregation
‣ Materialized views for Query Rewrite
‣ Partition Change Tracking (PCT)
‣ Materialized views on Prebuilt Tables
• New 11gR2 Parallelism
• Exadata Smart Scan
• Database Machine Compression



Approach



Key approaches

• SQL Modelling
• Early visibility of data to
• Re-aligning OBIEE end users
physical layer



High level project approach

Month 1 Month 2 Month 3

Discovery Analysis

Review 1

Iteration 1 Development

Review 2

Cycle 1 rework
Iteration 2 +
Development

Review 3



What did we not do?

• Typical steps to ‘create’ an Exadata environment

‣ Operating System installation on all four nodes
‣ Setting it up network-wise (including the IB switches)
‣ Setting it up Oracle-wise
‣ Disks config on each of the 7 Exadata Storage Servers
‣ Grid Infrastructure installation on all 4 nodes
‣ RDBMS Installation on all 4 nodes
‣ ASM config upon the grid disks
‣ Smart Flash Cache config
‣ RAC enabled database creation on the ASM diskgroups
‣ Testing and Documenting



High level project approach - analysis
• Requirements analysis
‣ User interviews
‣ Email forms
‣ Reviewing existing reports
• In parallel data analysis
‣ SQL modelling
‣ Defining test cases
‣ Defining acceptance criteria
• Architecture
‣ System architecture
‣ Environments
‣ Best practices
• Models
‣ Logical
‣ Physical


Analysis

Month 1

Requirements Analysis

Data Analysis

Discovery Architecture

High level design

Planning

Review 1



Approach - iterations

• Selection and prioritisation
• Start to build ETL
‣ Use existing or new SQL models if required
• Design logical layer and implement in OBIEE
• Implement physical layer to support logical layer
‣ Good enough approach
• Give end users visibility
‣ Test acceptance criteria
• Re-aligning OBIEE physical layer
• Fact tables may be reloaded
• Work more advanced calculation back into logical, physical and potentially
performance layer



Re-aligning OBIEE physical layer

• Swapping out physical tables

Presentation Layer

Common Enterprise Information Layer

Prototype Prototype Prototype
Customer Customer Product
Dimension Dimension Dimension





Presentation Layer


Prototype
Actual Prototype Prototype





Presentation Layer


Prototype
Actual Prototype
Actual Prototype
Actual



Iteration 1

Month 2

Selection

Detailed Design

Iteration 1 ETL Development

OBIEE Development

End User Testing

Review 2



Design



Design considerations

• Do we change the design for Exadata?
‣ Best practices
‣ Performance
‣ Where is processing done
• Preconceptions
‣ Don’t use indexes
‣ Use HASH partitioning



Design considerations

• Do we need a Data Warehouse?
‣ Data quality
‣ Tracking dimensional change
‣ Combining sources in OBIEE with high volumes of data can lead to
performance problems WHATEVER the data source



Design considerations - principals

• Use an atomic dimensional data model
• Minimise the number of downstream facts
• Don’t have an aggregation strategy
• Use the logical separation in OBIEE
• Use Data Warehousing features to compliment Exadata features



Combining Exadata features and Data Warehousing features

• Full Table Scan (Smart Scan) with Partitioning

Data Data Data

Memory Storage



Combining Exadata features and Data Warehousing features

• Full Table Scan (Smart Scan) with Partitioning



Observations



Observations - data processing

• I/O intensive operations are lightning fast.
‣ A pivot operation on 80M rows completed in a < 10s

• It seems as the memory-intensive processes such as
SQL analytics perform roughly the same
‣ However Exadata does have a lot of RAM so PGA
should be large

• V2 note: on a subsequent project we saw Bloom
Filtering with SQL Analytics statements



Observations - data querying

• Parallelism and hence performance works best with partition-wise operations
‣ HASH partition on surrogate keys for dimensions (may work for business key)
‣ RANGE partitioning reduces I/O on date related queries
‣ Would have been good to HASH sub-partition fact on product dimension
• Indexes
‣ Still useful when a specific rows required, e.g. tree-walking with CONNECT BY
PRIOR
‣ Not required for bulk operations
‣ Bitmap indexes on fact are still good for STAR transformation

• V2 note: we have seen STAR transformation out perform Smart Scan in some
queries



Observations - environment

• There was only one Exadata box
• Develop in production?
‣ Green field project
• Resulted in testing being very difficult if not on Exadata
‣ No surprise: it didn’t perform

• Became much harder for future iterations
‣ However particularly for the ETL the historic data was already in the
Foundation layer



Observations

• Advanced compression
‣ Big gain was disk space
• Sequences
‣ Must increase size of Sequence pool
‣ Common to all RAC instances
• Patching
‣ There is still some complexity in the patching process
• Not a magic wand
‣ It is still possible to produce badly designed star schemas
‣ It is still possible to produce badly designed OBIEE repositories
‣ It is still possible to run a query requesting every cell of data in the database
‣ It can still run slow



Wins - the numbers...

• ETL
‣ 10 facts; 15 dimensions; 300 tables; 157 OWB mappings
- Daily load: 1 hour 30 minutes
- Daily load : approximately 130 million rows pushed
through staging area and into the ODS and the Data
Warehouse
‣ One of the facts is refreshed with approximately 100
million rows and only takes 10 minutes

• Reporting
‣ Reports could be run over multiple fact over multiple
years
‣ Report that took 24 hours to run now completes in 15
minutes



Summary

• Benefits
‣ Organisation now has near real time supply and demand data
‣ Ad-hoc reporting capability
‣ Reduced turnaround of generating reports from 2 to 3 days of spreadsheet
crunching to instant available dashboard
‣ One source of truth
• Made possible by
‣ Extreme performance in the ETL layer
‣ Extreme performance in the Reporting layer
‣ Agile development approach



Summary

• Exadata and OBIEE provided means to successfully deliver project
‣ Fast data processing
‣ Support for agile approach

• Combination of software, hardware and approach (methodology)
‣ Exadata
‣ OBIEE
‣ Agile approach



Questions



Oow2010 mead exadata

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (11)

Similaire à Oow2010 mead exadata

Similaire à Oow2010 mead exadata (20)

Dernier

Dernier (20)

Oow2010 mead exadata