Overview of Data Governance requirements, techniques and outcomes. Presented at 5th Annual Records & Information Officers' Forum, Melbourne 19-20 Feb 2014.
Best VIP Call Girls Noida Sector 40 Call Me: 8448380779
The ABC of Data Governance: driving Information Excellence
1. The ABC of Data Governance
and its role in driving Information Management Excellence
Alan D. Duncan February 2014
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
2. A bit about me....
•
•
Alan Duncan, Director of Data Governance, UNSW
21 years Information Management & Business
Consulting
– EDS, KPMG, CPW, Acuma, Pelion, SMS
– Scottish Power, United Distillers, O2, Astra Zeneca,
Carphone Warehouse, Vodafone, Riyad Bank
– Commonwealth Bank, NSW Roads & Maritime
Services, Centrelink, OATSIH, NSW Family &
Community Services, CASA, AMSA, FaHCSIA, DAFF,
Navy…
•
•
Information-Management.com “Top 12 on Twitter”
Best supporting Actor, 2005 Barnet Drama Festival
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
3. …and a bit about UNSW.
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
4. Agenda
• Why we need to rethink our Information
Management Strategies
• Techniques and capabilities we need to focus on
• What data should be stored, and how long should
we keep it?
• Information Management Excellence
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
5. “The value of an idea
lies in the using of it.”
Why we need to rethink our information
Management Strategies,
Sponsored by Thomas Edison
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
6. “A” is for “Action”
Identify measurable and targeted Business Outcomes
Why do we need information? For whom? What will we do
differently?
Confirm the Information Holdings & Gaps
What do we need to provide? (Content + Context)
Establish DG Operating Model
Who is accountable? By what
processes?
Implement DG/IMCC Services
Catalogue:
“When it is obvious that the What core capabilities do we need?
Execute Activities & Tasks
goals cannot be reached,
How do we deliver? Who does the
don't adjust the goals,
work?
adjust the action steps.”
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
7. “A” is for “Access”
•
•
•
•
•
•
Stimulus to improve data quality
Consistency of data definitions
Openness and trust
Transparency & accountability
Opportunity value
Proactive publication and Open
Data vs. “Need to know”
“Publish and be damned!”
http://www.ted.com/talks/tim_berners_lee_the_year_open_data_went_worldwide.html
7
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
8. “A” is for Analytics
•
•
•
•
•
“Traditional” BI (reporting & ad-hoc analysis)
Data Mining
Statistical modelling
Data visualisation
Textual analytics
• What questions do we want to answer?
• What questions can we answer with the data
we’ve got?
• What other data would we need?
• What does the data tell us we should be
asking?
“The alchemists in their
search for gold discovered
many other things of
greater value.”
Alan Duncan, Director of Data Governance, UNSW
Uncontrolled when printed
8
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
9. Summary: Information Use Cases
•
•
•
•
Based on our current understanding of business needs, the following classes of
Information Use Case are identified
Detailed Requirements Analysis should be conducted on a project-by-project basis
to explore any detailed Use Cases within each class
Not all detailed Use Cases need to be defined ahead of time
Solutions should be flexible to accommodate new and changing Use Cases
Structured
data
reporting
Strategic
Intelligence
and Data
Mining
Publish
content to a
community
Executive
briefings
Education,
Training,
Learning
Search for
content
previously
created
Records
Management,
Compliance &
Audit
GIPA &
Privacy
Responses
Ability to publish
Filtering/screening/validation of what gets published
Feedback loop, measure of usefulness & continuous improvement
Shared understanding (IT & Business)
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
10. “To succeed in life,
you need two things:
ignorance and
confidence.”
Techniques and capabilities,
sponsored by Mark Twain
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
11. “A” is for “Asset”
Governance
Owners
Owners
Owners
Information
Asset
User
Community
Steward
Experts
Admin
Asset
Management
Tools
The “Information Asset Community”
Audit Data
Symbion
Health
System
Individual Flight Data
Example High Level Data
Systems & Flows
Human Resources – Time
AOD Case
Mgmt
System
Timelog
Medical
FTNS
Organisational
Flight Data
Version 1.0
Human Resources – Time
FTTO
Alcohol and Other Drugs
AOD Audit
Contacts – Ind, Org’s
Service Log
Contacts – Ind, Org’s
APEX
Workflow
Dangerous
Goods
Retain
PAWS
HRMS
Work Orders
API Upload
System
Dangerous Goods
DTAR/OTAR
Permissions / Change
of Status
MAAT
AOC
Surveys
ATOG Job
Register
Airports
Web Control
Mgmt
System
(WCMS)
FRLI
ADMS
Surveys/
Certifications
Human Resources - Travel
Events/Occurrences,
Aircraft, Aerodromes
Tracker
ASIR
Alternative
Means of
Compliance
(AMOC)
AMOC / Exemptions
Aircraft
AFD
Human
Resources – Flex
Time
ChangePoint
Testing
System
Aerodromes
Service Delivery
Surveillance/
Audit/
Reporting/
Tracking
Payroll (Salary)
TRIM
Aircraft
Finance
Human Resources
Alternative Means of Compliance
(AMOC)
Surveillance/Audit/
Reporting/Tracking
Service Delivery, HR & Finance, Agreements, Permissions, Aerodromes,
Participants, Aircraft
Events/Occurrences, Aircraft,
Aerodromes
AWS
workflow / service delivery
WMS
AIRS
workflow / service
delivery
STI
Medical Exams
Exemptions
Database
Payments
ESIR
Surveys /
Surveillance
Service Delivery
Directives
Incidents
Surveillance /Audits/
Reporting/ Tracking
eRooms
Superannuation Contributions
“Science is organized
knowledge. Wisdom is
organized life.”
Exemptions
Surveys / Surveillance
SM7
Details of
Operators
Comweb
CASA
Internet
MRS
Financial Actuals
Individuals/ARNS
Compensation Payments
CBMS
(DoFD)
AD
Enterprise Data
Warehouse
ASSP
Workflow and Online Collaboration/
Service Delivery
COMCARE
Content
Permissions / Change of
Status
Surveillance /Audits
Landings/ Take Offs
HRFlex
Surveys
Industry Payments
Cash Payments / Organisation Info
Financial Actuals
Calumo
AUSSAR
AIRS
Interchange
FMIS
Reconcile
Invoice against
Flown Hours
Employee
Expenses /
Adjustment
Journals
Search and Rescue
ATSL
Examinations
Cash Payments
ProMaster
Medical
Examinations
ATO
Business
Portal
Business Process
Medical Examinations
Remittance
Advice
Mandatory
AME
Examinations
Cyber Exams
External
Superannuation
Companies
FCAT
PAYG
payments, Salary
payments, and
Superannuation
payments.
Core
Suppliers
Corporate Support
CLIC
Inventory
Cash Payments
External
Bank Data
File
Data
Salary Payments
Mandatory
Inventory
Mgmt
System (IMS)
Defects/Events/Occurrences, Aircraft,
Aerodromes
Trending
Core
SDR
Physical Inventory
Human Resources
Aircraft
300+ Access
Databases
Contacts
ComBIZ
Online
Cash Receipts
Corporate Support
Publishing
System
HR - Time
External
Airspace
Thomas
Logistics
Applications / Permissions
General
System Interfaces map
EPK
Permissions
Aircraft Equipment Finances
Organisational
GMEL
External System
Baseline/Minimal Equipment
MMEL
Automatic One-way Relationship
Automatic Two-way Relationship
Manual One-way Relationship
Manual Two-way Relationship
Uncontrolled when printed
Info Asset Register
(inventory)
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
12. “B” is for “Benchmark”
STRAT002
STRAT005
STRAT007
STRAT015
CUL002
CUL003
ORG002
ORG006
ORG003
ORG007
ORG008
ORG009
ORG010
POL005
POL001
POL003
POL002
POL004
PROC011
PROC014
PROC003
PROC002
PROC001
PROC004
PROC007
PROC005
PROC008
PROC009
PROC010
“One day Alice came to a fork in the
road and saw a Cheshire cat in a
tree. Which road do I take? she
asked. Where do you want to go?
was his response. I don't know,
Alice answered. Then, said the cat, it
doesn't matter.”
Uncontrolled when printed
PROC013
SYS004
SYS001
SYS006
SYS007
SYS008
SYS002
SYS003
SYS009
QW001
SYS010
QW007
QW002
QW004
QW003
QW005
QW006
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
13. “B” is for “Big Data”
•
•
•
•
Three, four, five, six “Vees”?!
A lot of data (Tb/day)
Streaming data (monitoring, flow-of-control and
alerting analytics)
Inference from semi-structured data (Twitter,
Facebook)
Synthesise insight from millions of pages of text
Programmatic analysis for specific scenarios (hard in
SQL)
A disruptive catalyst to put information at the top of
the organisational agenda
Not just about the data! Business scenarios are key
•
Beware the Vendors!
•
•
•
•
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
14. “C” is for “Cloud”
In principle, it’s just another place to store data….
… but security, privacy considerations are heightened.
14
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
15. “C” is for compliance
•
•
•
•
•
•
•
•
•
“All I want is compliance with
my wishes, after reasonable
discussion.”
15
Uncontrolled when printed
•
•
Freedom of Information Act 1982 (Cth)
Freedom of Information Amendment (Reform) Act 2010
(Cth)
Privacy Act 1988 (Cth)
Privacy Amendment (Private Sector) Act 2000
Privacy Amendment Act 2012 (Cth)
Privacy Amendments (Privacy Alerts) Bill 2013 (Cth)
State Records Act 1998 (NSW)
Government Information (Public Access) Act 2009 (NSW)
Privacy & Personal Information Protection Act 1998
(NSW)
Health Records & Information Privacy Act 2002 (NSW)
NSW Government Guide To Labelling Sensitive
Information 2011 (NSW Financial & Services)
But is “compliance” a motivator?
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
16. “C” is for Collaboration Culture
• A general willingness to share
information
• Co-operative, communicative &
collegiate OR control, coercion
& criticism?
• The “whose data is this?” cue
• Call-to-action?
• Accountability &
measurement?
16
Uncontrolled when printed
“Respond intelligently even
to unintelligent treatment.”
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
17. “D” is for Data Models
Enterprise Information Model
Related Data
Governance Processes
Conceptual Layer (Business)
Business Glossary
Business Content
Data Domains
Data Elements, Definitions,
Aliases, and Security
Business Rules Data
Business Constraints
Information
Asset
Register
Business Measures
Endorsed Standards
for Content
Conceptual Model:
Groupings & Relationships
Master data models
Information
Concepts
Data Subject Areas
Business Rules
Definitions
Classification Entity
Hierarchies
Business Constraints
Business Measures
Data Entities
Mappings
Data Attributes
Physical Layer
Logical
Logical
Message
Message
Schemas
Schemas
Physical
Physical
Messages
Messages
Formats
Formats
Application
Application
Logical Data
Logical Data
Models
Models
Master Data Repositories
Operational
Operational
DB Models
DB Models
Analytical
Analytical
DB Models
DB Models
UNSW Core Systems
Systems Data
Models
MDM Data
Model
Core
Systems
MDM
Information Model Management Process
Reference models
Metadata Management Process
Logical Layer (Transition)
Information Alliances: Data Ownership & Stewardship Process
Business Data
Element
Domain Values
Metadata
Repository
Cubes
Cubes
Physical Instantiations
DWH
DB
HR
DB
G/L
Uncontrolled when printed
Student
Admin
etc...
SOA/EP
Messages
MDM Processes
“Do not quench your
inspiration and your
imagination; do not
become the slave of your
model.”
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
18. “D” is for Data Quality
Careless data entry
& lack of validation
Data is in different
Formats
Multiple codes exist
for the same thing
Overlapping subsets
in different places
Organisations
change rapidly
“Get your facts first,
then you can distort
them as you please.”
Uncontrolled when printed
Information
Disconnect
Teams have different ways
of reporting data
Region
Multiple, inconsistent
master data
Teams use different
IT systems
Month
?
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
19. Summary: Data Governance increases utility
& value of information
Information Asset Register
(Catalogue of data holdings)
Data Quality Management
(Profiling, root-case analysis,
issues tracking & resolution)
Information System(s)
Information System
Information System
Data Set
Information System
Information Requirements &
Business Analysis
(Identification & traceability of
business definitions & rules)
Uncontrolled when printed
Data Modelling
(Consistent, inter-operable data
structures & semantic meaning)
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
20. “The bigger the better;
in everything.”
What data should be stored, and how
long should we keep it?
Sponsored by Freddie Mercury
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
21. “E” is for “Everything”
• All of the data, all of the time?
• Granular, forensic history
• Modern data management & analytics solutions can make “all
of the data, all of the time” a reality
• The bigger challenge is that the business community is not
analytically skilled enough to navigate the data and draw
meaning from it…
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
22. “E” is for “Expectations”
• Finding data that makes an impact
• Having data for the problem at hand
• Trusting the data to guide your
decision
• Justifying pre-determined answers
• Setting inappropriate goals
• Not having the right data tools
• Not thinking about value
Uncontrolled when printed
“Two things are infinite.
The universe and human
stupidity. …and I’m not so
sure about the universe.”
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
23. “F” is for “Forethought”
•
Think about both current and future demand
– Cf. Google philosophy to “keep everything”
– Every click, every font change…
•
Tiered storage technology for different scenarios
– Fast-access disk for computational processing (TB)
– Drill-through near-line disk (TB/PB)
– Off-line active store (optical, tape) for archive (PB/EB)
•
Information Lifecycle Management
–
–
–
–
Copy/replication management
Test data management
Auditability of data movements between environments
e.g. IBM Optim, Oracle ILM, Informatica ILM, Hadoop Project
Falcon
“Forethought we may have,
undoubtedly, but not
• Integration/coupling of HDFS & EDW/BI?
foresight.”
•
Uncontrolled when printed
Data Requirements Management…
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
24. Summary: Data Requirements Management two modes of operation
Need both “push” and “pull” modes
for evidence-based decision-making
Data Factory
(“push”)
Source new data;
Collection &
Integration;
Preparation & Quality.
Demand-oriented
Inbound requests for
specific requirements
Product-based
delivery (“pull”)
Plus metadata management, info asset management,
data management, process management practices..
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
25. “I try not to think with my
gut. If I‘m serious about
understanding the world,
thinking with anything
besides my brain, as
tempting as that might be,
is likely to get me into
trouble.”
Information Management Excellence
sponsored by Carl Sagan
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
26. “G” is for Governance
•
•
•
•
•
•
•
•
•
“I don’t know what you’re going to do with my data once
you have it.”
“If I give you my data, you might then ask me to do some
extra work to meet your additional requirements.”
“You may not interpret the data in the same way that I do.”
“I’m an expert in this area, you’re not. The data is too
complex for you to understand.”
“It’s too difficult to get the data out of the system and I’d
need help from I.T.”
“I don’t have the budget to pay for your requirements.”
“I’d like to help but I’m just far too busy.”
“I know there are flaws in the data, but it’s good enough
for my needs. You might criticize me for the errors.”
“Management may ask additional questions and hold me
to account for the work I’m doing”.
26
Uncontrolled when printed
“I’m not interested in
preserving the status quo;
I want to overthrow it.”
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
27. Data Governance: overcoming “I object!”
• Control, structure, discipline &
compliance? OR Advocacy service &
information broker?
• Intimate understanding of business goals
& processes
• Engagement, diagnosis & facilitation
• Understand & articulate the meaning of
data, in context
• Coach, mentor and advocate
• Highly visible point-of-access
• Self-service Information Portal
• Conduit, communicate & co-ordinate
• Leadership & direction
• “Info as a Product”
27
Uncontrolled when printed
“The art of government is to
make two-thirds of a nation
pay all it possibly can for the
benefit of the other third.”
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
28. Conclusion: Information Excellence
EIM Framework:
Enterprise Information Management Framework describes each aspect of an organisations
information management state, provides a baseline of maturity against best practice and a
framework of business transformation to your aspirational information management state.
Provides linkage and balance between business,/IT, and human/technical aspects of EIM.
Information Culture
Enterprise Information Model
Data Governance
Organisation and People
IM Processes
IM Policies
IM Solutions and Technology
Conceptual Model
Information
Governance
Capture & Formalise
Requirements
& Rules
Business Processes
IM Governance Process
Impact
Assessment
& Implementation
Definitions, Derivations, Decision Rules, Execution Rules
Information and IM
Risk Management
Logical Model
Architecture
Mgt
IM Stewardship Process
DB Models
Semantic Specs
ETL Specs
Report
Definitions
Metadata
Lineage
Impact
Etc.
Information and IM
Quality Mgmt
Information Security
Data Mining
Reporting design
Analytics
Business Intelligence
Data Warehousing
Knowledge Transfer
Information Asset
Mgmt
Metadata
Information Asset Access and Use
Management
Assess and
Accessibility
Intellectual Property
Copyright
Ownership &
Stewardship
Licensing and
Rights
Management
Pricing
Exchange
Registration
Information and IM
Governance
Processes
Privacy
Publishing
Record Keeping
ETL
Staging
ETL
Warehouse
Cubes
Standard Report Library
Operational
System
Archiving
Conservation and
Preservation
Retrieval and Access
Digital Continuity
Record Creation
and Capture
Collection Management
Retention and Disposal
Record Management
Data
Marts
Human Resources
Security
Management
Physical and
Environmental
Management
Search and Discovery
Information Asset
Classification
Asset Alignment/Mgt
Physical Model
Information
Security Policy
and Governance
Asset
Management
Meta Knowledge
Information and IM
Policy, Principles
and Architecture
Information IM
Workforce
Management
Architecture Changes
Logical Model
Information and IM
Strategy and
Planning
Knowledge Management
Technical MetaData Management
Data management
Communications
and Operations
Management
Information
Security Incident
Management
Access
Management
Information
system
acquisition,
development and
maintenance
management
Metadata
Collection
Redress Mechanisms
Data Quality and
Integrity
Data Conversion
& Transformation
Business
Continuity
Data Modelling
Data Cleansing
Data Migration
Compliance
Management
Data Integration
Semantic
Layer
Data Capture
Data De-duplication
A holistic, data-centric approach to Information Management & Data Governance,
addressing both human and technical factors in both Business and IT domains
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan
29. Intellectual curiosity
Skeptical scrutiny
Critical thinking
http://www.informationaction.blogspot.com.au/
@Alan_D_Duncan
http://www.linkedin.com/in/alandduncan
Uncontrolled when printed
Alan Duncan, Director of Data Governance, UNSW
E: Alan.Duncan@unsw.edu.au Tw: @Alan_D_Duncan LinkedIn: http://www.linkedin.com/in/alandduncan