Zen and the Art of Datanauting

1. Exploration of large and complex data estates to gain an accurate understanding of the data structures and data quality Zen, and the art of Datanauting Carl Bray Product Manager, Ontology Systems Matt Clark Design Authority, BSkyB

2. Datanauting Boldly going where no data integrator has gone before… 2

3. 3 15 years of transaction data 10 million+ customers 900 engineers making changes 30 TB of data 20+ Applications Q) How do you start to understand this data estate?

4. The company • UK subsidiary of a global media organisation • Provides fixed line telephone, Internet and television entertainment services to UK residents • 10 million+ customers, trading for 15 years Business drivers: • Driven by marketing innovation • Extend and upsell to customer base • React to competitive threats • Technical infrastructure impacting commercial agility The motivation behind the project Background and Business Drivers 4

5. Objective • Significantly reduce the time to capture new business strategies in IT systems Significant change in IT delivery • Embrace Agile delivery of new functionality • Develop new payment and sales systems • Access and extend existing data • Multiple SCRUM teams using test-driven development • Phased delivery Short-term technical drivers • Quickly understand the structure, nature and consistency of the existing data Longer term technical drivers • Introduce a service-based semantic agent to access software services Fundamentally changing the way IT functionality is delivered A new IT Strategy 5

6. Subject matter experts (SMEs) • Understanding the data means interfacing with SMEs • Multiple SCRUM teams need access to SMEs • Knowledge is in Silos and not co-located with SCRUM teams • SMEs may not know the answers Bottleneck / Choke point • SCRUM teams need quick answers to data / process questions • SME bandwidth stifles SCRUM agility • Introduces a single project bottleneck/choke point Overwhelming the SMEs • Free and unfettered access to the SMEs would create chaos • Need to filter questions to the SMEs Challenges Many technical challenges stood in their way 6 CRM Billing Ref Data Debt Orders Ticketing Content Product SME SCRUM SCRUM SCRUM SCRUM SME SME SME SME SME SME SME

7. Many systems with complex interdependencies • CRM • Billing • Reference Data • Debt processing • Order handling • Trouble ticketing systems • Subscriber card management systems • Content access entitlements • Product catalogue Fragmentation • Business entities fragmented • “Customer” properties in many systems The Scope and Scale of the Problem Payments and sales system involving 20+ systems and legacy data 7

8. Data estate problems • Data quality isn’t consistent • Data fragmentation is high • Understanding the data is complex • How are business entities stored in different applications and data sources? • What impact should processes have on the data – flags, statuses, etc. • When data is duplicated, which data sources should take preference? • Scale of data • 30+ TB of historic trading data • 3 Vs - The Variety and Volume of data are very high The Data 30TB of transactional data over 15 years of system changes 8 ?

9. Non-semantic alternatives • Train more SMEs • Work around SME’s other priorities • Educational workshops • Take time to document systems Data-profiling alternatives • Reverse engineering schemas • ETL Tooling • Didn’t want to create yet another data warehouse Chose a datanauting approach • Supports their commitment to Agile development • Allows SCRUM teams to explore and ask questions of the data without overloading SMEs Alternatives Alternative approaches to solving the problem were considered 9

10. What we do, and why we’re different • Ontology leverages graph and semantic search technologies to address enterprise data issues • We address complex data integration problems • Data Acquisition • Data Correlation • Data Migration • We produce fully fledged operational applications that use semantic search in • Telecommunications • Media • Financial services • The Ontology Difference • Inherently agile – no schema • Datanauting: data-first, structure later • Just enough modelling • Structured and unstructured data How we approached the problem The Ontology Approach 10

11. Exploration of data sources… The Ontology Approach - Datanaughting Identify sources Connect to sources •Index source Search for entities •Refactor entities •Create URI pattern matching •Map entities to RDF Search for linked entities •Add references Search for equivalent entities •Create matching URIs •Map entities to RDF

12. • DBs • SPARQL Endpoints • Structured files • MS Excel, CSV, XML, RDF • CISCO and other device configurations • Propriety formats • Unstructured files • MS Word, PDFs, etc. The Ontology Approach - Datanaughting Identify sources Identify sources Connect to sources Search for entities Search for linked entities Search for equivalent entities

13. • Setup the connection • Index sources • Add search facets • Tokenise compound values e.g. • Service names are concatenated “Service-LON/01” • Product names use “CamelCase” The Ontology Approach - Datanaughting Connect to sources Identify sources Connect to sources Search for entities Search for linked entities Search for equivalent entities

14. • Search for business entities • Refactor “denormalised” data • Choose a URI pattern to represent instances • Set a type for the entity • Map properties to owl:DatatypeProperty The Ontology Approach - Datanaughting Search for entities Identify sources Connect to sources Search for entities Search for linked entities Search for equivalent entities

15. • Search for entities that should be linked • Add references (owl:ObjectProperty) between entities that are to be linked The Ontology Approach - Datanaughting Search for linked entities Identify sources Connect to sources Search for entities Search for linked entities Search for equivalent entities

16. • Search for semantically equivalent entities in other data sources • Search based on property names • Search based on strict value matching/weighting • Search based on sub-string matching/weighting • Reuse the URI pattern • Create references The Ontology Approach - Datanaughting Search for equivalent entities Identify sources Connect to sources Search for entities Search for linked entities Search for equivalent entities

17. High-level solution to the problems the organisation faced • Removed the SME bottleneck - a key enabler for the Agile / SCRUM approach • Creates a searchable domain model, breaking the data into discrete “chunks” • Ontology allows the SCRUM teams to understand the legacy data through ad-hoc queries • Can understand how business concepts are mapped across multiple contradictory data repositories • The quality and suitability of data can more easily be assessed • Provides a definitive view of the commercial position for a given subscriber or set of subscribers • Backlog and sprint priorities are based on a complete understanding of the complexity of the task • Provide data to facilitate mock ups and test harnesses Ontology provides SCRUM members with insight into the data Project Results 17

18. Project Results SCRUM teams gain insight into data 18 CRM Billing Ref Data Debt Orders Ticketing Content Product SME SCRUM SCRUM SCRUM SCRUM SME SME SME SME SME SME SME

19. Project Results Product Architecture 19 Modeller External Event Sources Web UI Ontology Intelligent 360 Ontology Integrity Manager Semantic Graph Store Query API Universal Search Core Semantic Processing Core Universal Search Core Authenticationand Notification LDAP Server (optional) Mail Server (optional) HTTPS RTIA Fully Modelled Data Sources CSV RDBMS XML JDBC XLS Other Data Sources DOC PDF XLS MAIL XML Ontology 4 Modeller Ontology 4 RuntimeHTTPS End Users (Browser Access)

20. Variety • Ability to access data in a variety of formats • Avoid integration to live systems • Possible to work from database - dumps avoids politics • Embracing change – inherently agile Volume • Ontology techniques for managing data scale • Partial index of data • Partial modelling • Semantic search with SQL query to live systems Velocity VarietyVolume Project Results Dealing with two large Vees 20

21. Why Ontology? • Agile response through inherently agile technology • Datanauting provides agile response to SCRUM teams • SME time can now be used for valuable queries Technical advantages • No Schema, No Integration, No Big Bang, No Search Restrictions, No Upfront Risk Benefits delivered • Speed – Greatly accelerated the analysis phase of the project • Risk – Project is not viable without an understanding of the data No Upfront Risk No Schema No Integration No Big Bang No Search Restrictions Zen, and the art of Datanauting Advantages of the Ontology approach to Data Integration 21

22. Learn More To learn more about Ontology Systems, or to access more detailed information about our products and services, please either: Call +44 20 7239 4949 Visit ontology.com Email info@ontology.com Subject to change. All rights reserved. © 2013 No part of this document may be reproduced in any form or by any means for any purpose without our written permission. All other trademarks appearing in this document are acknowledged as the trademarks of their respective owners. Ontology-Partners Limited trading as Ontology Systems Ontology Systems Phoenix Yard, 65 Kings Cross Road, London WC1X 9LW, UNITED KINGDOM Registered in England No. 5794201. Registered Office. Dalton House, 60 Windsor Avenue, London SW19 2RR UNITED KINGDOM

Zen and the Art of Datanauting

Zen and the Art of Datanauting

Recommandé

Contenu connexe

Tendances

Tendances(20)

Similaire à Zen and the Art of Datanauting

Similaire à Zen and the Art of Datanauting(20)

Dernier

Dernier(20)

Zen and the Art of Datanauting