Extending the Admin UI: Proactively Preventing Poor BPRA Load/Refresh Performance via Custom Utilities

1© 2015 ELLUCIAN. CONFIDENTIAL & PROPRIETARY | Session ID
Extending the Admin UI: Proactively Preventing Poor
BPRA Load/Refresh Performance via Custom Utilities
Eric Boyce - DBA
Bryan Mack – BI Development Team Lead
April 14, 2015
Session 11951

Session Rules of Etiquette
• Please silence your cell phone/pager/purse dog
• If you must leave the session early, please do so as discreetly as possible
• Please avoid side conversation during the session
• Please hold questions for the end of the presentation
Thank you for your cooperation!

Session rules of etiquette
Bryan L. Mack
bryan.mack@cccs.edu
Higher-Ed Employment History
• Colorado School of Mines, Asst. Director of Advancement Services
2002-2012
• Colorado Community College System, Team Lead/Data Warehouse Developer,
2012-2015

Session rules of etiquette
Eric Boyce
eric.boyce@cccs.edu
Higher-Ed Employment History
• University of Northern Colorado, BPRA Database Administrator
2009-2012
• Colorado Community College System, BI Sr. Oracle Database Administrator,
2012-2015

Purpose
• To create an understanding of how simple it can be to add desired
functionality to the BPRA Administration Interface. Further, we’ll step
through two scenarios whereby the Colorado Community College System
benefited from augmenting the BPRA Administration Interface’s
functionality.

1 Problem I – Inconsistent ODS Refresh Times
2 Potential Fixes – Baselines, Profiles, Statistics
3 Implementing a Solution – Stats Gathering
4 Problem II – BPRA Batch Window Utilization
5 Implementing a Solution – Chain Builder
Agenda

Problem I – Inconsistent ODS Refresh Times

Inconsistent ODS Refresh Times
• Greeted with this in the Morning:

Why the Poor Performance
Root Cause: Stale Statistics
● Over 4,300 daily batch jobs in CCCS’s Banner environment (AppWorx)
● Due to process ordering and contingencies batch execution times are
dynamic
● Changing Maintenance Window for statistical collection ineffective

Bad Stats Example
NUM_ROWS – Updated by stats collection

Potential Fixes – Baselines, Profiles,
Statistics

Baselines, Profiles, Statistics
● There is nothing wrong with using baselines, profiles, or dynamic
sampling to achieve performance improvements
● We wanted a tool that was simple, yet effective, in improving/maintaining
ODS performance
● We needed something that could handle Ellucian upgrades without
adversely affecting performance
i.e. Not reliant upon sql_id values
● We know the problem is poor statistics. No need to put cart in front of the
horse by creating baselines or profiles

When Should Statistics Be Collected?
According to Oracle
Source: http://docs.oracle.com/cd/E25054_01/server.1111/e16638/stats.htm#i41884

Requirements
● Collect statistics on change tables, composite tables, and indexes just
prior to the execution of a mapping
● Must be simple interface to understand and utilize… otherwise nobody
will use it
● Must have ability to ascertain mappings’ objects dependencies
● Should have the ability to handle various DBMS_STATS parameters

Implementing a Solution – Stats
Gathering

Identification
Identify exact locations where mappings are executed:

Digging Deeper
● Matching String Literals from Control Report to MGKMAP Body

Verification
Query against all_source

Before we go further…
Ellucian’s stance on the modification of baseline code:
-Any customizations to baseline code are NOT supported-

Customizing Baseline Code
IMPORTANT: Assure that the baseline scripts are saved off and
available. If an error is encountered you can always revert to the
functional baseline version.

Testing New MGKMAP
● It worked! We now know where the stats collection should take
place
● We also have the mapping object name

Creating a Package to Gather Statistics
PACKAGE:
IA_ADMIN.CCCS_GATHER_ODS_STATS
PROCEDURES:
p_gather_mapping_stats (mapping_name, estimate %)
p_gather_composite_tab_stats(mapping_name, estimate %)
p_gather_srce_and_change_stats(composite_view, estimate %)
FUNCTIONS:
f_get_stats_check (mapping_name)

p_gather_mapping_stats
1ST – Gather Composite Table Stats
- Determines Composite Table using DBA_DEPENDENCIES

2nd – Evaluate Mapping Type
- If it is a DELETE mapping, gather statistics on change table
- Determines Change Table using DBA_DEPENDENCIES

3rd – If mapping is not a DELETE, then it must be an UPDATE or LOAD
mapping.
- Determines Composite View and dependent tables

3rd (cont.)– If composite view found, call p_gather_srce_and_change_stats
- Loops through source tables and collects stats

f_get_stats_check
● Called from MGKMAP prior to any collection of stats
● Function checks to see if statistics should be collected for a
particular mapping
● Allows for control as to which mappings collect statistics
● Uses a custom MTVPARM record

Remember when…
● We want to collect stats on this mapping prior to execution to prevent this
from happening in the future

Create MTVPARM Parameter
● Beneficial to store the parameter values in MTVPARM as they’re easily
accessible via the Admin UI

Once again…
● Once again, we have bad statistics.

Control Report – With Stats Collection

Control Report – No Stats Collection
● If there is no MTVPARM record for a mapping, the f_get_stats_check will
evaluate to FALSE and statistical collection is simply ignored.

Control Report – Stats Collection on UPDATE

Problem II - BPRA Batch Window
Utilization

Batch Window
● CCCS’s BPRA environment has a batch window from 00:00 – 07:30.
● ODS refreshes are executed concurrently as REFRESH_ALL takes too
long.
● Implementing EDW and SRP caused overruns
(52 Hours initially, cut down to 3 hours after in-house tuning)
● Varied ODS refresh times made it difficult to predict when to schedule
the EDW Operational Refresh Job

Need for Contingency
● SRP reliant upon ODS Student, ODS Finaid, and ODS General
refresh completion
● Wanted a way to maximize our batch window by not allowing time
between jobs.
● Also wanted the ability to run concurrent jobs

Oracle Chains
“A chain is a named series of tasks that are linked together for a
combined objective. Chains are the means by which you can implement
dependency based scheduling, in which jobs are started depending on
the outcomes of one or more previous jobs.”
Source: http://docs.oracle.com/cd/B28359_01/server.111/b28310/scheduse009.htm#ADMIN10021

Oracle Chains
Chains can be thought of as a series of steps with defined rules as to when a
step should be performed. A step is associated with a task or action.
Step Action When to start
-------------------------------------------------------------------------------------------------------
Step 1 - Choose Bread TRUE (Immediately)
Step 2 - Add Condiments Step 1 Completed
Step 3 - Add Meat/Veggies Step 2 Completed
Step 4 - Assemble Step 3 Completed
Step 5 - Eat Sandwich Step 4 Completed
Step 6 - Go to Sleep (End) Step 5 Completed

In our Case
Step Action - Program When to start - Rule
-------------------------------------------------------------------------------------------------------
Step 1 - Refresh Student TRUE (Immediately)
Step 2 - Refresh Finaid TRUE (Immediately)
Step 3 - Refresh General TRUE (Immediately)
Step 4 - Refresh Custom MViews Step 1 Completed
Step 5 - Refresh EDW Operational Steps 1, 2, 3, 4 Succeeded

Defining Actions via Programs
● Oracle Scheduler Program Objects define an action for a Chain step
● Can be used to abstract a PL/SQL Block, Stored Procedure, or an
executable

Building Programs
● So couldn’t we just put these Refreshes into Programs using PL/SQL blocks?
DECLARE
JOB_NUM NUMBER;
BEGIN
SELECT sys.jobseq.nextval INTO JOB_NUM from dual;
mgkmap.P_RunETLMapSlots(9,JOB_NUM,'REFRESH_FINAID',NULL, '');
END;

Creation of a Program
● Could be done in Enterprise Manager but can be tedious when doing a
large quantity.

Implementing a Solution – Chain
Builder
•

Sample Admin UI Job Script
● Ellucian delivers a sample Admin UI job script in the ODS installation
code tree:
odsodsdevlodsia_admindbscriptsutility_scriptssample_job.sql
● Serves as a good reference for the necessary pieces when creating a
custom job.

INSTALLED PROCESS Parameter
● Creation of an INSTALLED PROCESS Parameter creates a link in the
Admin UI’s ‘Schedule a Process’ Menu

JOB Parameter
● Creation of a JOB Parameter links the Installed Process to a Packaged
Procedure

Basic Packaged Procedure Header
● Main procedure of sample job noted earlier
● Main procedure of CCCS_CHAIN_ADMIN

Questions…Questions…Questions…
● Before we go any further we have to ask ourselves:
1. How many steps should the chain builder handle?
2. How many rules will we need?
3. How are the rules created?
4. How can we monitor progress as chain is executed?
5. How do we avoid having objects of the same name?

First Question
1. How many steps should the chain builder handle?
A: CCCS chose to start with 7. This was the fewest number of steps we
would need to create a chain that could handle our EDW/SRP
requirements.
Note: It was decided that although CCCS would start with 7 steps, the utility
needed to be flexible enough to easily handle more steps in the future.

Second Question
2. How many rules will we need?
A: We need the number of steps decided upon +1
- This is because there is a final rule that states when all steps are finished,
the chain is has completed–
- Additionally, the final rule need not be a parameter for a user to control as
this could cause the chain to stall.

Creating Parameters

Rules and Steps
7 Rules and 7 Steps

Additional Parameter
● We also wanted to allow the user to specify their own Chain
Name

So How’s it Look

Parameter Handling
● We use tables to store the values we’d like to display for parameter
values. The ‘SELECT’ parameter type simply displays the results of
a query.

Parameter Handling (cont.)
Example.

Translation Table
● We use the ia_admin.cccs_chain_prog_xwalk table to map Display
names to actual object names. Further, the SEQUENCE_NUMBER gives
us a means of easily controlling, which processes are displayed in Admin
UI.

Rules Table
Rules possibilities are also stored in
a table.

Admin UI Submission

How Parameters are Passed
● The job generated contains all of the parameters inputted, in a pipe
delimited format.
● Knowing all the parameters, we can now build a chain

Parsing Parameters
● All parameters come into the parms
input parameter
● Use mgkutil.F_GetParameterValue to
determine what the inputted values are

Building Chain
● After validation that the job has basic parameters in place, we verify that
no other chains of the same name exist. If they do, they are removed
and an entry noting this is made in the control report. Following this step,
an empty chain is created

Building Chain Steps
● Once the chain is built, we
add the steps for it
● Display names for programs
are parsed to their actual
object names
● Using a counter variable we
loop through all the
programs creating chain
steps along the way.

Building Chain Rules
● Chain Rules are built in a similar manner as the Chain Steps with
a loop and a counter variable.
● There is one final rule that is built. It is the summation of all steps
and informs Oracle that the chain is complete.

Creating Scheduler Job
● We’ve now created a chain, added steps to it, and added rules to it.
● The only thing left to do now is to submit it.

Checking Mechanism
● As we wanted to know the status of the scheduler job, we created a self
checking mechanism
● Every five minutes the chain run is evaluated and it’s status is reported in
the control report

Final Control Report

Adding More Steps in the Future
● To add additional steps, all one needs to do is create additional
PROGRAM and RULE parameters following the naming
convention laid forth
● Additionally, rules must be added to cccs_rule_possibilities or
whatever table is chosen to store the RULE literals

Summary
With an understanding of how the Admin UI works, a developer can
easily create utilities/functionality to augment what is already a great
product delivered by Ellucian

Questions & Answers

Thank you!
Eric Boyce - DBA
Bryan Mack – BI Development Team Lead
Please complete the online session evaluation form.
Session 11951

Extending the Admin UI: Proactively Preventing Poor BPRA Load/Refresh Performance via Custom Utilities

Recommended

Recommended

More Related Content

Similar to Extending the Admin UI: Proactively Preventing Poor BPRA Load/Refresh Performance via Custom Utilities

Similar to Extending the Admin UI: Proactively Preventing Poor BPRA Load/Refresh Performance via Custom Utilities (20)

More from Bryan L. Mack

More from Bryan L. Mack (8)

Recently uploaded

Recently uploaded (20)

Extending the Admin UI: Proactively Preventing Poor BPRA Load/Refresh Performance via Custom Utilities