Summit 2011 ods edw technical

Business Intelligence and Data Warehouse

Goals, Trends, and Complexities of Supporting
an Integrated SGHE ODS/EDW
Data and Date Driven
Knowledge System
BAS S118

Greg Turmel
Sr. Database Administrator
Tennessee Board of Regents

Date/Time: Monday, October 17 10 am ‐ 12:30 pm
1


ODS foundations:

 Project Vision
 Initial server configuration
 ODS Metadata
 STREAMS concepts
 Feeding and Care of the data stream:
A total commitment to quality (from everybody)
 Technical Challenges

Open discussion:
2


Project baselines: A vision & sponsorship

Vision
First Stage
Sponsorship
Executive Management

Implementation Plans
Second Stage
Resource Pool Defined
Operational Management

Infrastructure and Technical Support
Third Stage
Commitment to Quality
Project Resources – Integration Systems Management

3


Additional questions to ask:
 What is the project scope?
 What business requirement is driving the effort?
 What is the source system for record retrieval?
 What resources exist for the datastore?
 What resources exist to support a Data
Warehouse?
 What resource does the organization have in
supporting data modeling / design?
 What experience does the resources have?
(Source/Target – Logical/Physical designs)
 What Extraction methods (tools/technologies)
will be used?
Can’t answer these basic questions then the project will have questionable results…… 4


Server Preparation: Key things to remember
 Oracle Solaris 10 update 6 or above
 Oracle 11gR2 (11.2.0.1) Binaries installed/patched
 Oracle account – Modified .profile
ulimit – n 65536
ulimit – s 16384
Note: [OWB JAVA errors]
 Oracle 11g memory management uses ‘projects’
 Oracle 11g no longer uses the companion cd
 Oracle 11gR2 drops 32 bit libraries – binary
support
 Oracle OWB is now a default / part of the database

5


Configuration issues
Server Preparation:
 Oracle user account creation: cat /etc/password
oracle:x:101:101::/u01/home/oracle:/bin/ksh

 Oracle Project creation: cat /etc/project
user.oracle:100:Oracle:::process.max‐sem‐
nsems=(priv,256,deny);project.max‐sem‐
ids=(priv,100,deny);project.max‐shm‐
ids=(priv,100,deny);project.max‐shm‐
memory=(priv,7516192768,deny)

 Assign memory allocation to the project: limits
6


Note:

Banner
vs.
Oracle

7


ODS metadata
 During deployment a static webpage is delivered
for user/developer and architectural purposes

 The metadata can be updated by your IT staff
 [e.g.] Feeding the database with Campus data 8


ODS metadata

 http://tbrtap01.tbr.edu:9250/metadata/ODS_index.html 9


STREAMS concepts:
1st – You have to get it integrated and running

 DBMS_CAPTURE_ADM
 DBMS_PROPAGATION_ADM
 DBMS_APPLY_ADM

These Oracle packages are utilized by the ODS
installation to create the Oracle® Streams queues.

These packages are called from the ODS package
ODSSTG.MGKSTRC during the installation of ODS 8.2.
SGHE:
FAQ 1‐81W3F1 ‐‐ ODS 8.2 and Oracle® Streams
FAQ 1‐81W3A9 ‐‐ Oracle Doc Id 290605.1 Oracle Streams STRMMON Monitoring Utility 10
FAQ 1‐81W38Z ‐‐ Oracle Doc Id 273674.1 Streams Configuration Report and Health Check Script


STREAMS concepts:

Streams Capture: Reads the database redo logs. Collects DML
and DDL changes that have been made on
the Banner source table

Streams Propagation: The process moves changes (LCR) from the
Banner source DB to the ODS target DB

Streams Apply: Takes the changes into the ODS stage tables.
Matches and updates the modified record in
the ODS target table

Error Queue: Low level streams data errors encountered
can be reviewed and reprocessed.
11


STREAMS Validation: Source systems
select CAPTURE_NAME, STATUS from DBA_CAPTURE;

BPRA_BANNER$CAP ENABLED

select PROPAGATION_NAME, DESTINATION_DBLINK,
STATUS from DBA_PROPAGATION;

BPRA_BANNER$PROP
ODSD.TBR.EDU
ENABLED

Validate streams configuration: No duplication
12


STREAMS Validation:
SELECT MTVPARM_EXTERNAL_CODE,
MTVPARM_INTERNAL_CODE_2 from MTVPARM
WHERE MTVPARM_INTERNAL_CODE_GROUP = 'STAGE
CONFIGURATION' AND MTVPARM_INTERNAL_CODE =
'SOURCE ALIAS';

 TEST.TBR.EDU BPRA_BANNER

Populating MGRSDAX from Banner GTVSDAX

 On ODS: desc MGRSDAX; (describe)
 On Banner: desc GTVSDAX; (describe) 13


STREAMS Validation
 Ensure that aq_tm_processes is not explicitly set
to 0 (Oracle Support recommends – unset the parameter):

‐ FAQ 1‐2USESE Oracle Metalink Note 428441.1
‐ "Warning: aq_tm_processes Set To 0" – Alert Log

 ALTER DATABASE RENAME GLOBAL_NAME TO
fully_qualified_database_name;

 DATA_PUMP_DIR – setting it is a requirement.
select DIRECTORY_PATH from dba_directories where
DIRECTORY_NAME = 'DATA_PUMP_DIR';
14


Database Server # A Database Server # B
User or process input

Banner Transactional System

Operational Data Store
Changes

Complex Join
Query
Queue Queue
Streams Capture Propagation Streams Apply

Write
15


16


Feeding and Care of the data stream:
 Data security: Are the feeds behind fire walls?

 Masking: Are you using production quality
data in TEST?
 Compliance: Are you capturing and transmitting
SSN, Health, or transactions?
 Exploits: Are you blocking SQL Injection?

 Audit: Are you regularly scheduling a
review?
 Monitor: Are you proactively monitoring
systems?
17


Technical challenges:
Impact analysis – testing change management?

Dataload Dataload Dataload
Change Management Banner

Change Management Datamart
Change Management EDW
Change Management ODS


18



Demand Management: A Lesson in what not to allow…

8* Methods MTP (Move to production process)
Change Management Banner

PPRD DEVL
PROT PROTROY

CSPPRD TSTH
19


Technical challenges: Change management

20


Technical challenges: Data Quality
Access
data /odbc
uploads

EDW

Data

Data

Data

Data Oracle 11g
datamart

Data Oracle 10g
datamart

Data Oracle 9i
datamart

Data
21


Design/Analyze Major System changes:
 Require an understanding in how changes affect down line systems: Reporting – [e.g.] KPI’s

EDW 8.2 GOAL:
testing

EDW 8.2 workshops
Oracle Fusion April: End of Product life: April: Installing EDW
Middle Tier WLB 11gR1 SGHE Banner 7.x May: Installing EDW
TN Mods: SMO
Banner – June 15: Gen 8.4 June: Installing EDW
July 15: Student 8.51 - delivered
Luminis 4.3 - Solaris 10

HR 8.5: IPEDS
Note – No 8.2

EDW 8.2 GOAL:
production
Oracle 11gR1 Banner 9: Released
Banner Database Sept 2011
Comm Framework
Stu Class Schedule
Stu Course Catalog April: Prepare and
begin delivering SGHE
EDW Workshops
- Enhancing BI
efforts across the State

22


Technical challenges: Defects and Config
FAQ 1‐BFL58F ‐ ODS 8.2 install stop responding when ETL scripts (section 3.7.3) are
executed in a production environment.

Defect 1‐BNA5XL student_etl_install.sql SATURN.SOTVCUR ERROR: ORA‐20000: Adding
supplemental log data terminated with the following error: ORA‐14450: attempt to
access a transactional temp table already in use

Related FAQs:
FAQ 1‐BPCU06 ODS 8.2 Report Staging Area Status DBA_APPLY_ERROR ORA‐01403
FAQ 1‐DKNNNB ODS 8.2 Streams error ORA‐01435: user does not exist.
FAQ 1‐GBP75K ODS 8.2 ORA‐04031 unable to allocate bytes of shared memory
FAQ 1‐AXHXP3 ‐ How to restage the ODS 8.2 / 8.2.1 staging tables
FAQ 1‐C8Y1U1 ‐ How to restage one specific ODS 8.2 stage table.
FAQ 1‐CHWKO1 ODS 8.2 Streams Banner Source data not propagating to ODS target
FAQ 1‐IEH1Z4 ODS 8.2 Oracle Streams Spilled messages and BPRA_BANNER$CAP queue
FAQ 1‐IGVVDY ODS 8.2 and Oracle Streams and Init.ora Parameter GLOBAL_NAMES
FAQ 1‐JF68GU ODS 8.2 and How to reload the data only for a staged table in BPRA
FAQ 1‐I4UP5C ‐ Is my Oracle environment ready for ODS 8.2 and Oracle Streams?
FAQ 1‐BU6KRN ‐ ODS8.2 how to reconcile stage tables
23


Technical challenges: Oracle support
Note 273674.1 Streams Configuration Report and Health Check
Note 437838.1 Streams Specific Patches
Note 238455.1 Streams Supported and Unsupported Datatypes
Note 782541.1 Streams Replication Supplemental Logging
Note 290605.1 Oracle Streams STRMMON Monitoring Utility
Note 365648.1 Explain TXN_LCR_SPILL_THRESHOLD in Oracle
Note 265201.1 Troubleshooting Streams Error ORA‐1403 No Data
Note 779801.1 Streams Conflict Resolution
Note 461278.1 Example of a Streams Heartbeat Table
Note 313478.1 Performing Manual DDL in a Streams Environment
Note 335516.1 Streams Performance Recommendations
Note 730036.1 Overview /Troubleshooting Streams Performance

A complete list of streams articles on Metalink: Knowledge
tab ‐> Database ‐> Information Integration ‐> Streams 24


Open Discussion

25


10 Minute Break

26


EDW foundations: In this section we discuss

 Establishing project timelines

 What STREAMS baselines are needed

 Technical challenges

 EDW Metadata

 Argos delivered cubes for the reporting community

Open discussion: 27


Building Time lines supporting your projects
ODS/EDW 8.2 implementation Timeline as of September 2011
*Refres her Tra i ni ng
2011 2012 2013
Task or Effort SEPT OCT NOV DEC JAN FEB MAR APR MAY JUN JUL AUG SEPT OCT NOV DEC JAN FEB MAR APR

ODS 8.2 Tra i ni ng *
ODS 8.2 Tes t Impl ement ODST
ODS 8.2 Prod Impl ement ODSP

EDW 8.2 Tra i ni ng
EDW 8.2 Tes t Impl ement EDWT
EDW 8.2 Prod Impl ement EDWP

ODS 8.3 Pa tchi ng Tenta ti ve Schedule
EDW 8.3 Pa tchi ng Tenta ti ve Schedule

28


EDW work is:
Building Patterns out of Random Chaos/Mashups?
‐ Not!

 It is highly refined with specific deliverables:

 Goal orientated – Subject studied over time
‐ (Trending, Strategic, Decisions)

 Non‐Volatile – data is historical based on
criteria
‐ (Month, Year)
‐ not dynamic (Do not edit data) 29


Organization: structured or a mashup?

Business Requirements Business Requirements

Analysis
Mashup Ideas
Design
Discovery
Implementation
Testing & Evaluation
Evolution Selection

Retirement Deployment
Composition

Usage
Deployment

Retirement Application
Usage 30
Resource: IEEE computer.org/itpro


EDW metadata:

http://tbrtap01.tbr.edu:9250/metadata/EDW_index.html 31


EDW work supports predictive analysis:
 [e.g.] Profiling at risk students (goal focused)

‐ behavior in their high school, freshman,
sophomore, junior years…

‐ Financial assistance, parental capacity to help,
scholarships available…

‐ Leadership roles, involvement in the community

 [e.g.] Cost associated with lost opportunities with
graduating a student across the system
32


Analytics: Predictive Analysis methods using
Longitudinal Studies on Students
What if:
 We identified the Cost associated with losing students?
 Revenue over time lost associated with these students?
 Educated to a certain level and they disappear?
 What was the impact to the University or College system?
 The economic area impacted? The State education results?

Student Population Avg # of Class hours Sum total of all students
Lost Cost per class taken Avg cost of tuition lost Campus level

5367 $166.00 24 3984 $21,382,128.00 CC Sophomore

11629 $166.00 24 3984 $46,329,936.00 CC Freshman

853 $366.00 24 8784 $7,492,752.00 University Junior

5118 $366.00 24 8784 $44,956,512.00 University Sophomore

11089 $366.00 24 8784 $97,405,776.00 University Freshman

$217,567,104.00
33


Storage Requirements: Partitioning data
Schemes such as: Single or Composite
Why: Creates faster queries in reporting (tuning)

Single: Range, Hash, or List

Composite: Two dimensions [e.g.] Order/Shipped
Range – (Range) (Hash) (List)
List – (Range) (Hash) (List)

1. Manageability
2. Availability
3. Pruning
34


Range – maps data to a partition based on a range
of values.

01.01.2010 until 12.31.2010
01.01.2011 until 12.31.2011

Hash – maps data to a partition based on an algorithm
key that evenly distributes across devices.

List – maps data specifically to lists of discrete values
such as West Tennessee Region, Middle
Tennessee Region, or East Tennessee Region.
35


Single – Level (Range, Hash, or List)

http://download.oracle.com/docs/cd/E11882_01/server.112/e16541.pdf
36


Composite – Level (Range, Hash, or List)

http://download.oracle.com/docs/cd/E11882_01/server.112/e16541.pdf

37


Schemes such as: Composite
Composite – Level (Range, Hash, or List)
 Fiscal Year – July 1st ‘11 until June 30th ‘12
 Calendar Year – Jan 1st ‘11 until Dec 31st ‘11
 Academic Year – August 2011 until May ‘12

Joined with regional designation:

 West Tennessee
 Middle Tennessee
 East Tennessee
38


ETL: Oracle 11g Warehouse Builder
(extract, transform, & load)

Install and Admin Guide ‐ b31280

Lesson 1: Creating OBIEE Meta for OLAP 11g Cubes
http://st‐curriculum.oracle.com/obe/db/11g/r1/olap/biee/createbieemetadata.htm

Lesson 2: Building OLAP 11g Cubes
http://st‐curriculum.oracle.com/obe/db/11g/r1/olap/cube/buildicubes.htm

Lesson 3: Querying OLAP 11g Cubes
http://st‐curriculum.oracle.com/obe/db/11g/r1/olap/cube/querycubes.htm 39


Reporting:
 Snapshots of the system:
 FREEZE dates for Analytics

 Trend Analysis:
 Focus points
 Scatter plots
 Dealing with outliers

 Define collection:
 Build cube based on need, not everything

40


Reporting:
 Impact of load:
 Scheduling and heavy resource use

 Adding Legacy Data:
 Criteria to set – Integration of archival data

 Updates:
 No update processes / no data tweaking
activities

41


Argos Co‐op datablock code available:

Admissions Dashboard:
 Admissions Dashboard with multiple forms,
enrollment OLAP, KPI, Demographic and test scores.

Course Registration EDW:
 The Course Registration datablock displays the
generated credits and number of seats (and all
available measures) by academic year and academic
period.
42


Course Enrollment Summary Report:
 Course enrollment by academic period, college, and
department.

Enrollment Ethnicity:
 Shows ethnicity of enrolled students based on term.

Enrollment Trend Analysis:
 Displays enrolled student name and Banner ID based
on Academic year and term.
43


Financial Analytics ‐ Departmental Expense:
 Uses OLAP cube for financial expenses by
department with several dimensions

Financial Aid Pre‐Student:
 Uses the Financial Aid Pre‐Student datablock to
understand the trends in pre‐student acceptance
and enrollment based on how financial aid
amounts are allocated.

44


Financial Aid Student:
 The Financial Aid Student datablock can be
used to understand the trends in packaging
financial aid awards, and to support
improved allocation of financial aid amounts

General Ledger EDW:
 Use the General Ledger datablock to
understand the trends in the general ledger
activity, and to better manage overall
financial health of the institution.
45


Grant and Project:
 Use the Grant and Project datablock to understand
the trends in grants and other sponsored research
projects, and to better understand and manage
research funding and spending.

Recruiting and Admission:
 Recruiting and Admission datablock trends in
recruiting and admissions. To better manage the
enrollment funnel, and to understand trends in
financial aid awarding to new students to better
manage financial aid funds.
46


Workforce Turnover Summary Report:
 Work force turnover by start/end date, gender,
department, employer, class and ethnicity.

Workforce Headcount Analytics:
 Uses OLAP cube to present Workforce analytics.
Comes with several dimensions.

47


Graduation Completion:
 The Graduation Completion datablock can be
used to understand graduation trends, and to
monitor and improve graduation rates.

Operating Ledger:
 The Operating Ledger datablock can be used to
understand the trends in operating expenses and
revenue to help you to better plan and forecast.

48


Program Enrollment Trends:
 Uses OLAP cube to present program enrollment
trends. Several dimensions are included.

49


Twelve project ‘success’ factors to remember:
 Build a system the customer can use ‐ Think:
Customer oriented results ‐ quick, meaningful,
clear

 Answer problems that keep coming up during
conversations... [e.g.] Student Retention &
Graduation Rates

 Big picture is Important, but... create value now.

 Don't push people into a project, create a project
customers find value in
50


 Analytical efforts provide insight into solving the
problems – knowing the difference in what is
pushing/pulling a project (Discovery vs. Economic)

 Did you hear the customer? Did you arrive at a
consensus? Did you use the KIS (keep it simple)
method? Did you engage the customer in a review?
Did you tell friends (expand its value)

 Feedback... user experience

 If you lost your audience... why, then fix (change)
the direction 51


 Truth and Transparency in the data, the collection
methods, the logic in any transformation =
competence and value to the customer

 No one cares about a personal ego, the technology,
or commitment ‐ What is the customers viewpoint =
What's the product and What's in it for them?

 Measure and cost out the project including human
resources (time & energy)

 Never give up. One failure does not equal the
abandonment of a project 52


Q&A ?

53


 Master Note for Streams Recommended Configuration [ID 418755.1]

 Streams Secure Queues & Using DBMS_STREAMS_ADM.SET_UP_QUEUE to Setup
the Secure Queue [ID 230902.1]

 Streams Support for Compression [ID 763997.1]

 How To Exclude A Table From Schema Capture And Replication When Using
Schema Level Streams Replication [ID 239623.1]

 How To Configure Streams Real‐Time Downstream Environment [ID 753158.1]

 How To Setup Schema Level Streams Replication with a Downstream Capture
Process with Implicit Log Assignment [ID 733691.1]

 How To Access Streams Or Advanced Queuing Information From The 10.2
Dbconsole [ID 336061.1]

 Banner Operational Data Store 8.2 Handbook / Architecture

54


 11gR2 Streams Concepts and Administration ‐ e17069.pdf

 11gR2 Streams Replication Administrators guide ‐ e10705.pdf

 11gR2 Streams Advanced Queuing users guide ‐ e11013.pdf

 11gR2 Oracle Database Warehouse Builder ‐ Concepts ‐ e10581.pdf

 11gR2 Oracle Database Warehouse Builder Release Notes ‐ e10585.pdf

 11gR2 Oracle Warehouse Builder Data Modeling ‐ ETL ‐ Data Quality guide ‐ e10935.pdf

 BPRA Upgrades ‐ Oracle 11g/OWB 11g, PM App Dependencies, Jim Carter, Brian Large:
http://www.edu1world.org/CommonsBI/wiki/document/4754

 Post‐Installation Steps for Oracle Warehouse Builder 11g Release 2 (11.2.0.2)
http://download.oracle.com/docs/cd/E11882_01/relnotes.112/e10585/toc.htm

55


Contact Information:
Greg Turmel
Sr. Database Administrator
Office of Information Technology
Tennessee Board of Regents
1415 Murfreesboro Rd. #358
Nashville, TN. 37217

615.366.4467 (Office)
615.365.1598 (Fax)
http://itinfo.tbr.edu (IT website)
http://twitter.com/datahaulr
http://www.linkedin.com/in/gturmel
http://www.slideshare.net/gturmel
http://itinfo.tbr.edu/itinfo/tiki‐download_file.php?fileId=515
56

Summit 2011 ods edw technical

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (15)

Similaire à Summit 2011 ods edw technical

Similaire à Summit 2011 ods edw technical (20)

Plus de Greg Turmel

Plus de Greg Turmel (8)

Dernier

Dernier (20)

Summit 2011 ods edw technical