Fully Automated SOA ETL Metadata Capture Soln

Copyright © 2007 CA. All rights reserved. All trademarks, trade names, service marks and logos referenced herein belong to their respective companies.
A Fully Automated
Metadata Capture Solution
Session # DG105SN
Focus Area DADIM

Abstract
The Harris Information Warehouse Environment
(HIWE) Reporting environment in cooperation with
the Allfusion Repository for Distributed Systems has
delivered a fully automated metadata capturing
solution, a first of its kind at BMO. The technical
foundation of this solution is based on Service
Oriented Architecture (SOA) implemented using
Simple Object Access Protocol (SOAP) as the
communication framework. This session will provide
an architecture / technical overview of the
automation of the end-to-end processes involved;
from the initial creation of metadata in reports hosted
on Microsoft Reporting Service platform; to the final
Enterprise Metadata Repository website.

Company
BMO Financial Group (NYSE, TSX: BMO), Established in
1817 as Bank of Montreal, BMO Financial Group is a
highly diversified North American financial services
organization. With total assets of $320 billion at October
31, 2006 and almost 35,000 employees, BMO provides
a broad range of retail banking, wealth management and
investment banking products and solutions. Canadian
clients are served through BMO Bank of Montreal, BMO
Nesbitt Burns, BMO InvestorLine and BMO Harris
Private Banking. In the United States, clients are served
through Chicago-based Harris Bank, an integrated
financial services organization that provides banking and
investment services. BMO’s North American investment
and corporate banking division, BMO Capital Markets,
provides a full suite of financial products and services to
North American and international clients.

Biography
Mark Pahulje
Bank of Montreal
 7 yrs. in Metadata Management Group
 Winners of Wilshire Award for Best Practices in Metadata
Management 2006
 In previous life, 5yrs experience a BI Consultant, during
which time developed MIND and won award as profiled in
"DB2's Bank of Montreal to save $270M with Web-enabled
business intelligence" using Microstrategy’s Business
Intelligence Platform.
 Winner of User Interface Engineering’s Advanced Design &
Usability Techniques team challenge at BMO, involving 250
participants.
 Aspire to ride “Le Tour de France”
mark.pahulje@bmo.com

Agenda
 Background Info (5mins)
 Presentation (45mins)
 Metadata Management overview
 HIWE project overview
 Metadata Architecture overview
 Walkthrough of a Capture Process
 Parsing Problem
 Walkthrough of Metadata Website
 QA (10mins)

BMO’s Metadata Team
 The Metadata Management group is part of the
Information Management department within the
Enterprise Systems organization of BMO FG’s
Technology and Solutions.
 embedded within the Bank’s systems development
life cycle as part of the CMMI program
 part of standards process for ensuring the integrity
of customer information
 engaged in deliver specific types of metadata
through external projects and business initiatives.
 Our involvement in projects ranges from managing
metadata within traditional IT systems development,
data marts or vendor solutions to implementing data
quality initiatives, or delivering information
strategies and architectures.
 In spite of these consulting services, Metadata
Management is not considered to be a standalone
program.

We embed governance and quality as
part of an overall information strategy
aligned to business goals
Value Creation
Business Intelligence
Centre of Competency
Architecture
Renewal
Data Mart
Consolidation
Access
Technology Strategy
(Metadata)
Governance &
Quality
Policy & Standards
Governance Process
Info Stewardship
Integrated Change
Management
Best in Class
Delivery
Global Resourcing
CMMI
Key enterprise
initiatives
Information
Into Action:
Competitive advantage
Real Outcomes
Trusted Information
Best in Class
People
High Performance
Culture
Resource Management

BMO Financial Group honored with
2007 WILSHIRE AWARD for Metadata
Best Practices
LOS ANGELES—January 18, 2007—Wilshire Conferences, Inc.
and The Data Administration Newsletter (TDAN.com) have
announced the winner of the 2007 Wilshire Award for
Metadata Best Practices. From among a pool of exceptionally
accomplished nominees, the panel of eight judges has chosen
the BMO Financial Group (BMO) as this year’s recipient for its
implementation of an outstanding, multi-faceted metadata
initiative.
Candidates are evaluated on numerous best-practice criteria
including (though not limited to) the following:
 Overall business impact/benefits
 Return on objective
 Measurable return on investment
 Scale of the solution
 Number and breadth of applications
served by the solution
 Length of time the solution has been in place
 Innovative problem solving
 How well the submission addresses
the questions asked above

Harris Information Warehouse
Environment (HIWE) Project
Mission
 Implement a new Data Mart and supporting
infrastructure - replace Alltel Information
Warehouse not renewed in 3ys
 Enhance Marketing & Information Decision
Support (MIDS)
 Reduce Operational Costs and time to
produce information
 Support aggressive growth strategy
 Support Enterprise-wide information needs
 As much as 30% of MIDS resources has to
be dedicated to supporting the environment
to compensate for current limitations
 Time to meet information delivery and
performance of the system is inadequate to
meet client expectations and demands.

HIWE Project Timeline
Metadata
Completed
HIWE Project is
initiated
HIWE planning
phase is
completed and
Stage 1 Started
MIDS writes high level and
Detailed Business Requirements
Harris Standard Staging development
Harris Standard
Staging History
Load and
MIDS/Basel UAT
Starts
HSS load
complete
d except
MLS and
Pershing
Harris Marketing Datamart development
MLS and Pershing
load complete
Stage 2 – Report Environment, Report
Replacement, Data Mining work space,
Additional Interfaces, Change Records
UNICA Implementation
Stage 2
Funding
Datamart
Loading
Starts
SRA #1 signed for
Stage 1
Datamart
Loading
Complete
O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J
| | | | | | | | | | | | | | | | | | | | | | | | | | | c | | | | | | | | | | | | | | | | | |
2002 2003 2004 2005 2006

Harris Information Warehouse Environment –
High Level Application Architecture

Enterprise Repository Architecture
Population
Application
Oracle
DBMS
Data SourcesData Sources
AdminAdmin
UserUser
OCIF, BIW,
Basel…
Repository
Database
contains 500
tables to
model date
types. During
a load any
number of
tables
maybe
appended
too.
When
completed,
results are
immediately
available to
web users
and
repository
clients.
Windows Clients
Data Tier
Metadata Management - The Roadmap to your Information Assets
ActiveX/COM
* Assembler
* C/C++
* Cobol
COOL:DBA
* Excel Spreadsheets
* DB2-MVS
* Word Documents
* Erwin
ETI
IDMS
* Glossary
Informix
* JCL
Java
MS SQL
* ODBC
* Oracle
Paradigm Plus
PeopleSoft
PowerBuilder
Sybase
Visual Basic
Visual C/C++
* LDAP
* currently owned
Data Source
Types
WebWeb
ClientClient
RepoRepo
ClientClient
Corporate Data
Artefacts
Presentation
Tier
BMO
Corporate
Repository -
Custom Web
Application
Advantage
Repository Engine
processes request
and loads the data
into the database.
Basel
ETL….
Basel
Erwin
Model
…
Web Server
End
Users
Advantage
Repository
Engine
Repo ClientRepo Client

Standard Scanning Process
Diagram

HIWE Meta Model Extensions

Web Service Oriented
Architecture 101
Web Services
Description Language
Universal Description
Discovery and Integration
Simple Object
Access Protocol
HIWE Reporting
Web Service Provider
HIWE Reporting
Web Service Consumer

Wide Screen Shot With a Header
Automated Metadata Scan Architecture
HIWE Reporting
Web Service Provider
Metadata
Repository
Database
1. HIWE Web Consumer App
requests and consumes soap message from
Microsoft Reporting Web Service provider &
parses & matches meta-model relationship
chain, if found in repository db.
}}} Output: A file in .csv format text file.
PCC &
PCD
2. General Custom
Scan App
reads a CSV file and produces
proprietary CA Allfusion
Repository for Distributed
Systems load files.
}} } Output: Load files (.pcc,.pcd)
Advantage Repository
Engine (ARE)
Windows
Scheduler
3. PCAFLDC
is a generic out-of-the-box vendor cmd
line loader utility that reads repository
files (.pcc, .pcd, .pcr) into the Metadata
Repository database processed by
ARE service.
}}} Output: Log file (.log)
CSV
Email
Alert
log
4. HIWE
Sweeper App
alerts Metadata team
via email to any errors
or completions of
previous apps.
PCR
XSD
XML
New
XML

BA User’s Report 2590

HIWE Report XSD

Web Consumer Consumes Sample
Reports XML file
* Footnote; Verdana 9pt, Bold

HIWE Web Consumer Email

HIWE Web Consumer CSV

HIWE GCS Mapping File XML

2. General Custom Scanner
Email

Sample PCD Generated File
PLATINUM Compressed Aggregate Format Data File v.2.0.0
//
E,ELEMENT,12,3,C11,Region Name,C10,PRODUCTION,S0
T,0,12,1,DESCRIPTION,11,Region Name
E,REL_DT,13,3,C8,VARCHAR2,C10,PRODUCTION,S0
R,RPT_FLD,14,12,2,2,C55,Secured Loans Report (For Metadata testing
2).DERIVED_3,C9,DERIVED_3
T,0,14,1,DERIVATION,37,=Trim(Fields!RGN_NM.Value) ' Total'
T,0,14,1,WHERE_CLAUSE,37,=Trim(Fields!RGN_NM.Value) ' Total'
R,RP_HS_FL,15,4,14,0
E,TABLE,16,3,C5,RGN_C,C10,PRODUCTION,S1
R,REL_COL,17,12,13,4,C6,RGN_NM,S80,S0,S0
R,TBL_COL,18,16,17,0
R,FLD_SRC,19,14,18,0
2).DERIVED_4,C18,Cur Bal (Subtotal)
T,0,20,1,DERIVATION,30,=Sum(Fields!CUR_BAL_AMT.Value)
T,0,20,1,WHERE_CLAUSE,30,=Sum(Fields!CUR_BAL_AMT.Value)
R,RP_HS_FL,21,4,20,0
2).DERIVED_5,C18,Cmt Bal (Subtotal)
T,0,22,1,DERIVATION,37,=Sum(Fields!CUR_CMTMNT_BAL_AMT.Value)
T,0,22,1,WHERE_CLAUSE,37,=Sum(Fields!CUR_CMTMNT_BAL_AMT.Value)
R,RP_HS_FL,23,4,22,0
2).RGN_NM,C6,RGN_NM
T,0,24,1,DERIVATION,13,Straight move
R,RP_HS_FL,25,4,24,0
R,FLD_SRC,26,24,18,0

HIWE PCC File

HIWE Report Reuse Rules .PCD

3. PCAFLDC.exe Scheduled
Batch File
HIWEParmDaily.txt
-cHIWEWCOutputDailyFile.pcc
-fHIWEWCOutputDailyFile.PCD
-rHiweDailyPrscan.log
-l
-uirm_prod
-prepadmin
-REndeavorCA
-tPRODUCTION
-e0
-*BMO_REPORT3.pcr
-nHMDM Reporting
-yCATEGORY
timeit PCAFLDC "PCAFParmDaily.txt" > "PCAFParmDailyCmdLE.err“
tail /28 “HIWEDailyPrScan.log" > "PCAFImportReportSummary_Tail.log"

HIWEDailyPrScan.log
PCAF Import Report
Control file: HIWEWCOutputDailyFile.pcc
Data file : HIWEWCOutputDailyFile.PCD
Parsing Reuse Rules File ==>BMO_REPORT3.pcr
HMDM Reporting - Populating Container
Dangling relationships:
None
Forcing database COMMIT
Object Reuse Testing Begins
...
Perfect: ELEMENT Name: Unknown HIWE element
...
Adding: REL COL Name: BUC_CD
...
Perfect: RPT FLD Name: 2590 Privacy Law States - Counts and Listing.DERIVED_1
....
Importing...
Using container with name: HMDM Reporting id: 412682690.
0 seconds elapsed
===============================================================
PCAF Import Report - summary
===============================================================
Control file: HIWEWCOutputDailyFile.pcc
Data file : HIWEWCOutputDailyFile.PCD
Entity Relate
Parsed: 43 295
Fail to parse: 0 0
Duplicate tag: 0 0
To hookup: 43 295
Dangling: 0 0
Perfect: 43 291
Updated: 0 0
To add: 0 0
Fail to add: 0 0
------------------------------------------
Total added: 0 0
Container relates added: 0
Container relates reused: 342
Container relates skipped: 0
Container relates fail: 0
4.PCAFImportReportSummary_Tail.log

4. HIWE Scan Sweeper Email

Parsing Problem
- Stored Procedure RDL
<DataSets>
<DataSet Name="dsetSecuredLoans">
<Query>
<CommandType>StoredProcedure</CommandType>
<CommandText>BPAI.PSECUREDLOANS</CommandText>
<QueryParameters>
<QueryParameter Name="PROFILEENDDATE">
<Value>=Parameters!PROFILEENDDATE.Value</Value>
</QueryParameter>
</QueryParameters>
<DataSourceName>dsHIWE</DataSourceName>
</Query>
<Fields>
<Field Name="LOAN_GRP_CD">
<rd:TypeName>System.Int32</rd:TypeName>
<DataField>LOAN_GRP_CD</DataField>
</Field>
<Field Name="RGN_NM">
<rd:TypeName>System.String</rd:TypeName>
<DataField>RGN_NM</DataField>
</Field>

Sample Stored Procedure
CREATE PROCEDURE PSECUREDLOANS (IN PROFILEENDDATE DATE)
SPECIFIC PSECUREDLOANS
READS SQL DATA
NOT DETERMINISTIC CALLED ON NULL INPUT
LANGUAGE SQL
DYNAMIC RESULT SETS 1
====================================================================================
====================================================
P1: BEGIN
DECLARE currSQLState CHAR(5);
DECLARE SQLCODE INTEGER DEFAULT 0;
DECLARE SQLSTATE CHAR(5) DEFAULT '00000';
DECLARE DT_FROM_CURSOR DATE;
DECLARE EXIT HANDLER FOR SQLEXCEPTION ErrHandler:
LOOP SET currSQLState = SQLSTATE;
LEAVE ErrHandler;
END LOOP;
IF (PROFILEENDDATE IS NULL) THEN
-- SET PROFILEENDDATE = '12/31/9999';
SET PROFILEENDDATE = (SELECT CUR_DT FROM HMDM.CUR_DT_CNTL_D WHERE SRC_SYS_ID = 'AGGR');
END IF;
-- CALL BPAI.FIND_EOM_DT(PROFILEENDDATE);
BEGIN
DECLARE C1 CURSOR WITH RETURN FOR
select 1 as Loan_Grp_CD, b.rgn_nm, b.rgn_cd, a.acct_nbr, a.area_cd, a.buc_cd, a.acct_typ_cd, af.cur_bal_amt,
lf.cur_cmtmnt_bal_amt, a.opn_dt, a.status_cd, a.prfl_end_dt
from hmdm.cons_acct_d a
inner join hmdm.buc_org_c b on a.buc_cd = b.buc_cd
inner join hmdm.cons_acct_f af on a.sk_acct_id = af.sk_acct_id
inner join hmdm.cons_ln_f lf on a.sk_acct_id = lf.sk_acct_id
inner join hmdm.mrp_iw_prod_xref m on a.prod_id = m.prod_id
where a.src_sys_id in ('XAM', 'XIM') and m.iw_prod_id in (700000000031, 700000000134)
and a.status_cd <> '050' and a.prfl_start_dt <= PROFILEENDDATE and PROFILEENDDATE >= a.prfl_end_dt
and a.prfl_actv_ind = 1
UNION…

Sample DAD File
What is a Document Access Definition (DAD) ?
A DAD (Document Access Definition) file is an
XML-formatted file that associates XML document
structure
to a DB2 database. It is used by the DB2 XML
Extender to compose and decompose XML data
into relational
data. For more information on DAD files and their
particular syntax, refer to the DB2 XML Extender
Library

RDB to XML Mapping Editor

Walkthrough of Metadata Website

Wide Screen Shot With a Header

Analysis of Report Privacy
Law States [2590]
1.
2.
3.
4.

1. Which Application?

2. Fields in this Report?

Report Field TID Details
TID
TID

Table Column Details for TID

Partial Physical Table Listing

3. Sourcing?

4. What FileName?

Recommended Sites
http://www-128.ibm.com/
developerworks/websphere/
techjournal/0111_lau/lau.html

Summary
A few words to review
 If you can influence the Reporting
Tool choice
 Choose a parsing strategy early
 Generalize your architecture to be
repeatable for any reporting tool

Questions & Answers
Mark Pahulje
Metadata Specialist
416 513-5636
mark.pahulje@bmo.com
BMO Financial Group
Technology and Operations
Information Management
120 Bloor Street East, 5th Floor
Toronto, Ontario
Canada M4W 3X1
www.bmo.com

Related Sessions
Session # Insert title of
session related to
your session topic
EJM01SN Enterprise Job
Management
Status and Plans

Exhibition Center
Related CA and Partner Technology
 Computer Associates
 Booth 123 - Product X
 Booth 456 - Product Y
 Partner
 Booth 998 – Company / Product X
 Booth 999 – Company / Services Y
 Exhibition Center Tours
 Sign up at Information Desk Booth #xxx

CA Technology Services
and Education
Hear how CA’s learning solutions
can help you meet your business
objectives
 Visit CA Education in the Exhibition Center
in Booth xxx, visit ca.com/education or call
us at 1-800-237-9273
Learn how CA Technology Services
can help your business
 Visit the CA Technology Services stations in
the exhibition center or on the web at
ca.com/services

Session Evaluation Form
 After completing
your session
evaluation form ...
 ... place it in the
basket at the back of
the room.
 Please left justify the
session number

Fully Automated SOA ETL Metadata Capture Soln

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Fully Automated SOA ETL Metadata Capture Soln

Similaire à Fully Automated SOA ETL Metadata Capture Soln (20)

Dernier

Dernier (20)

Fully Automated SOA ETL Metadata Capture Soln

Notes de l'éditeur