Data vault seminar May 5-6 Dommel - The factory and the workshop

•

3 j'aime•1,468 vues

This document discusses an open source metadata driven data warehousing project at a Dutch waterboard. It describes the basic architecture of using open source tools like Pentaho and Quipu to build a source data vault, business data vault, and generate data marts. It also covers lessons learned, including that it is possible to quickly build an EDW with open source software, but some challenges around automation and performance still remain. The goal is to deliver added business value in a cost effective way.

Technologie

The factory and the workshopOpen source metadata driven data warehousing Johannes van den Bosch @johannesvdb

Agenda Background Organization Project Basicarchitecture The factory …and the workshop Topics fromyesterday Real time Business keyintegration Staging out Hierarchies and supertypes Self service BI and writeback Lessonslearnt

Waterschap De Dommel Dutch Waterboard in south of Netherlands Managing water quality and quantity for 900.000 citizens and 150.000 hectares 375 employees of which 2 BI Projectsmanagedon time, money and goals Demandforintegrated management information

Project Current BI architecturereachedlimits New greenfieldarchitecture Open source advocate Passionatebeliever in costeffectivesolutionsforgovernment It’s our money! Convinced management No software cost Internalhours (2x 0.3 FTE) 1 year

Open source software ETL Pentaho Data Integration (Kettle) Data warehouse management Quipu Documentation MediaWiki Modeling Power*Architect

EDW architecture Reporting Analysis Dashboards Data mart 1 Data mart n Business Data Vault Source Data Vault 1 Source Data Vault 2 Source Data Vaultn Supplydriven Demanddriven Generated and automated Staging 1 Staging 2 Stagingn Source 1 Source 2 Sourcen

Developmentapproach supply driven effort demand driven time

The factory Source Staging Source Data vault ETL ETL

HowQuipuworks Source model Target model Template Load code (ETL)

Notcross-platform! …I want my ETL tool..! 3. Generatestaging ETL Default ETL: INSERT INTO staging_table SELECT fields FROM source_table

PoC Decided to try and build the bDV and Data Marts 100% virtual bDV = views on top of sDV Data marts = views on top of bDV Conclusion: it is possible

Functionalcomponents History Integration Transformation applyingsemantics, filtering, etc.

bDV design decisions: full bDV Source data vault Business data vault H T H I H

Integrationstrategies 1) same-as link H L H S S 2) integratedhub S H S 3) integratedhub + integratedsatellite H S

Integration: hubs Source Source data vault Business data vault person employee_h employee_h_s person_h person_h_s System x Users Users_h Users_h_s System y

Hubintegration – virtual System x BK1 System y BK2 New BK Integration business rule

Transformation: supertype example sDV Business rule bDV 5___ P______ 4____

Transformation: hierarchyexample sDV bDV

Data marts: virtual (simpleexample) Dimensionfromhub + sat Factfrom link + sat SCD type 1

bDV design decisions: partialbDV Source data vault Business data vault T H I H T H T

Full bDVvspartialbDV Full Lots of elements to define Easy data marts Partial Lesswork More T between data vault and data marts Multiple versions of the truth

VirtualvsPhysical Virtual (views) No physicalmaintenance Easy to adapt Performance limitations Platform defines performance Lineage (dependingon platform) Real time Auditability? Physical Scalability / performance Manualtweaking (indexes, etc.) Surrogatekeys easy More intuitive to develop (ETL in stead of SQL) More complex transformations (ie. aggregations)

Self service BI and write back Palofor Excel Open source MOLAP Everycellpoints to location in the cube Writeback to cubepossible EDW cube excel

Lessonslearnt Itispossible to quicklybuildan EDW with open source software Somereallycooldevelopments (ie. data mart generation) Automationonlygoessofar Somechallengesstillneed to beaddressed …it is business intelligenceafter all. Automate, ifitsavesyou money Itcan save you time to focus on the important stuff The end product counts: does itdeliveraddedvalue? What’s the best EDW architecture? Itdepends!™

Contenu connexe

En vedette

Bi Themadag 2011

johannesvdb

Data Vault ReConnect Speed Presenting PM Part Four

Hans Hultgren

Data Warehouse Agility Array Conference2011

Hans Hultgren

Data Vault ReConnect Speed Presenting AM Part One

Hans Hultgren

A Lean Data Warehouse, compared with a traditional one (with a dimensional or 3rd normal form model), is faster to deliver, freer of waste, and inherently more adaptable to change. From my experience in the trenches, each of these benefits fit squarely in the 'must have' category. Data Vault is an excellent logical architecture with which to design a Lean Data Warehouse. This article describes the priorities of a Lean Data Warehouse, and compares the two traditional modeling methods with Data Vault, concluding that Data Vault is more suited to deliver on those Lean priorities.

Lean Data Warehouse via Data Vault

Daniel Upton

Guru4Pro Data Vault Best Practices

CGI

Präsentation auf der DOAG Konferenz Metadaten sind ein häufig vernachlässigtes Thema, da Metadaten als langweilig betrachtet oder auch nicht bewusst wahr genommen werden. Auch die eher abstrakten Beschreibungen wie "Metadaten sind Daten über Daten" sind nicht gerade hilfreich. In der Präsentation werden die verschiedenen Arten von Metadaten (fachlich, technisch, prozessual) besprochen. Es wird darauf eingegangen, wie diese in einem Data Vault Projekt genutzt wurden, um z.B. Vorgaben festzulegen oder Code zu generieren.

Metadaten und Data Vault (Meta Vault)

Andreas Buckenhofer

Data Warehouse automation in de praktijk (Heliview 30 1-2014)

johannesvdb

Tectrade Magazine 4

Dennis Molenaar

Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Andreas Buckenhofer

Data Vault and DW2.0

Empowered Holdings, LLC

Data Vault ReConnect Speed Presenting AM Part Two

Hans Hultgren

Data Vault ReConnect Speed Presenting PM Part Three

Hans Hultgren

Agile Methods and Data Warehousing (2016 update)

Kent Graziano

Agile BI via Data Vault and Modelstorming

Daniel Upton

Part 2 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Andreas Buckenhofer

CDC und Data Vault für den Aufbau eines DWH in der Automobilindustrie

Andreas Buckenhofer

Data Vault: Data Warehouse Design Goes Agile

Daniel Upton

Big Data Modeling

Hans Hultgren

Synopsis: [Video link: http://www.youtube.com/watch?v=ZNrTxSU5IQ0 ] Jim Stagnitto and John DiPietro of consulting firm a2c) will discuss Agile Data Warehouse Design - a step-by-step method for data warehousing / business intelligence (DW/BI) professionals to better collect and translate business intelligence requirements into successful dimensional data warehouse designs. The method utilizes BEAM✲ (Business Event Analysis and Modeling) - an agile approach to dimensional data modeling that can be used throughout analysis and design to improve productivity and communication between DW designers and BI stakeholders. BEAM✲ builds upon the body of mature "best practice" dimensional DW design techniques, and collects "just enough" non-technical business process information from BI stakeholders to allow the modeler to slot their business needs directly and simply into proven DW design patterns. BEAM✲ encourages DW/BI designers to move away from the keyboard and their entity relationship modeling tools and begin "white board" modeling interactively with BI stakeholders. With the right guidance, BI stakeholders can and should model their own BI data requirements, so that they can fully understand and govern what they will be able to report on and analyze. The BEAM✲ method is fully described in Agile Data Warehouse Design - a text co-written by Lawrence Corr and Jim Stagnitto. About the speaker: Jim Stagnitto Director of a2c Data Services Practice Data Warehouse Architect: specializing in powerful designs that extract the maximum business benefit from Intelligence and Insight investments. Master Data Management (MDM) and Customer Data Integration (CDI) strategist and architect. Data Warehousing, Data Quality, and Data Integration thought-leader: co-author with Lawrence Corr of "Agile Data Warehouse Design", guest author of Ralph Kimball’s “Data Warehouse Designer” column, and contributing author to Ralph and Joe Caserta's latest book: “The DW ETL Toolkit”. John DiPietro Chief Technology Officer at A2C IT Consulting John DiPietro is the Chief Technology Officer for a2c. Mr. DiPietro is responsible for setting the vision, strategy, delivery, and methodologies for a2c’s Solution Practice Offerings for all national accounts. The a2c CTO brings with him an expansive depth and breadth of specialized skills in his field. Sponsor Note: Thanks to: Microsoft NERD for providing awesome venue for the event. http://A2C.com IT Consulting for providing the food/drinks. http://Cognizeus.com for providing book to give away as raffle.

Agile Data Warehouse Design for Big Data Presentation

Vishal Kumar

En vedette (20)

Bi Themadag 2011

Data Vault ReConnect Speed Presenting PM Part Four

Data Warehouse Agility Array Conference2011

Data Vault ReConnect Speed Presenting AM Part One

Lean Data Warehouse via Data Vault

Guru4Pro Data Vault Best Practices

Metadaten und Data Vault (Meta Vault)

Data Warehouse automation in de praktijk (Heliview 30 1-2014)

Tectrade Magazine 4

Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Data Vault and DW2.0

Data Vault ReConnect Speed Presenting AM Part Two

Data Vault ReConnect Speed Presenting PM Part Three

Agile Methods and Data Warehousing (2016 update)

Agile BI via Data Vault and Modelstorming

Part 2 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

CDC und Data Vault für den Aufbau eines DWH in der Automobilindustrie

Data Vault: Data Warehouse Design Goes Agile

Big Data Modeling

Agile Data Warehouse Design for Big Data Presentation

Similaire à Data vault seminar May 5-6 Dommel - The factory and the workshop

Traditional data word

orcoxsm

Traditional databases and batch ETL operations have not been able to serve the growing data volumes and the need for fast and continuous data processing. How can modern enterprises provide their business users real-time access to the most up-to-date and complete data? In our upcoming webinar, our experts will talk about how real-time CDC improves data availability and fast data processing through incremental updates in the big data lake, without modifying or slowing down source systems. Join this session to learn: What is CDC and how it impacts business The various methods for CDC in the enterprise data warehouse The key factors to consider while building a next-gen CDC architecture: Batch vs. real-time approaches Moving from just capturing and storing, to capturing enriching, transforming, and storing Avoiding stopgap silos to state-through processing Implementation of CDC through a live demo and use-case You can view the webinar here - https://www.streamanalytix.com/webinar/planning-your-next-gen-change-data-capture-cdc-architecture-in-2019/ For more information visit - https://www.streamanalytix.com

Planning your Next-Gen Change Data Capture (CDC) Architecture in 2019 - Strea...

Impetus Technologies

"We now do Agile BI too” is often heard in todays BI community. But can you really "create" agile in Business Intelligence projects? This presentation shows that Agile BI doesn't necessarily start with the introduction of an iterative project approach. An organisation is well advised to establish first the necessary foundations in regards to organisation, business and technology in order to become capable of an iterative, incremental project approach in the BI domain. In this session you learn which building blocks you need to consider. In addition you will see what a meaningful sequence to these building blocks is. Selected aspects like test automation, BI specific design patterns as well as the Disciplined Agile Framework will be explained in more and practical details.

Agile Testing Days 2017 Intoducing AgileBI Sustainably - Excercises

Raphael Branger

IoT NY - Google Cloud Services for IoT

James Chittenden

Agile Testing Days 2017 Introducing AgileBI Sustainably

Raphael Branger

By managing Data in Motion, Data at Rest, and Data in Use differently, modern Information Management Solutions are enabling a whole range of architecture and design patterns that allow enterprises to fully harness the value in data flowing through their systems. In this session we explored some of the patterns (e.g. operational data lakes, CQRS, microservices and containerisation) that enable CIOs, CDOs and senior architects to tame the data challenge, and start to use data as a cross-enterprise asset.

Creating a Modern Data Architecture for Digital Transformation

MongoDB

Leer hoe Pentaho kan helpen om zowel legacy data en ongestructureerde (Big) data van verschillende bronnen te blenden en te verrijken om zo waarde te creeeren voor uw organisatie. Praktische voorbeelden illustreren hoe Pentaho dit al bij vele organisaties heeft weten te bereiken. Zie hoe organisaties Pentaho onder andere inzetten om: • problemen met te lange ETL jobs op te lossen waardoor Data Warehouse loads weer doorgaan, • de kosten van data-integratie te verlagen, • het overlopen van traditionele Data Warehouses en bijkomende kosten doet voorkomen, • Data Quality en Data Governence in uw process inbrengt en • hoe dit vervolgens embedded in uw applicaties kan worden geanalyseerd.

Big Data Expo 2015 - Pentaho The Future of Analytics

BigDataExpo

Modernizing Your Data Warehouse using APS

Stéphane Fréchette

Big Data Analytics in the Cloud with Microsoft Azure

Mark Kromer

Skilwise Big data

Skillwise Group

Skillwise Big Data part 2

Skillwise Group

Many of the Big Data and IoT use cases are based on combining data from multiple data sources and to make them available on a Big Data platform for analysis. The data sources are often very heterogeneous, from simple files, databases to high-volume event streams from sensors (IoT devices). It’s important to retrieve this data in a secure and reliable manner and integrate it with the Big Data platform so that it is available for analysis in real-time (stream processing) as well as in batch (typical big data processing). In past some new tools have emerged, which are especially capable of handling the process of integrating data from outside, often called Data Ingestion. From an outside perspective, they are very similar to a traditional Enterprise Service Bus infrastructures, which in larger organization are often in use to handle message-driven and service-oriented systems. But there are also important differences, they are typically easier to scale in a horizontal fashion, offer a more distributed setup, are capable of handling high-volumes of data/messages, provide a very detailed monitoring on message level and integrate very well with the Hadoop ecosystem. This session will present and compare Apache NiFi, StreamSets and the Kafka Ecosystem and show how they handle the data ingestion in a Big Data solution architecture.

Data Ingestion in Big Data and IoT platforms

Guido Schmutz

Independent of the source of data, the integration of event streams into an Enterprise Architecture gets more and more important in the world of sensors, social media streams and Internet of Things. Events have to be accepted quickly and reliably, they have to be distributed and analyzed, often with many consumers or systems interested in all or part of the events. Storing such huge event streams into HDFS or a NoSQL datastore is feasible and not such a challenge anymore. But if you want to be able to react fast, with minimal latency, you can not afford to first store the data and doing the analysis/analytics later. You have to be able to include part of your analytics right after you consume the data streams. Products for doing event processing, such as Oracle Event Processing or Esper, are available for quite a long time and used to be called Complex Event Processing (CEP). In the past few years, another family of products appeared, mostly out of the Big Data Technology space, called Stream Processing or Streaming Analytics. These are mostly open source products/frameworks such as Apache Storm, Spark Streaming, Flink, Kafka Streams as well as supporting infrastructures such as Apache Kafka. In this talk I will present the theoretical foundations for Stream Processing, discuss the core properties a Stream Processing platform should provide and highlight what differences you might find between the more traditional CEP and the more modern Stream Processing solutions.

Introduction to Stream Processing

Guido Schmutz

Watch full webinar here: https://bit.ly/34iCruM Many organizations are embarking on strategically important journeys to embrace data and analytics. The goal can be to improve internal efficiencies, improve the customer experience, drive new business models and revenue streams, or – in the public sector – provide better services. All of these goals require empowering employees to act on data and analytics and to make data-driven decisions. However, getting data – the right data at the right time – to these employees is a huge challenge and traditional technologies and data architectures are simply not up to this task. This webinar will look at how organizations are using Data Virtualization to quickly and efficiently get data to the people that need it. Attend this session to learn: - The challenges organizations face when trying to get data to the business users in a timely manner - How Data Virtualization can accelerate time-to-value for an organization’s data assets - Examples of leading companies that used data virtualization to get the right data to the users at the right time

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Denodo

Presentation by Erik van der Hoeven (Wisdom as a Service) at the Data Vault M...

Patrick Van Renterghem

3dw

Kumanan Kadhirvelu

HIPAS UCP HSP Openstack Sascha Oehl

Sascha Oehl

Hadoop from Hive with Stinger to Tez

Jan Pieter Posthuma

"Building Data Warehouse with Google Cloud Platform", Artem Nikulchenko

Fwdays

3dw

umavipplow

Similaire à Data vault seminar May 5-6 Dommel - The factory and the workshop (20)

Traditional data word

Planning your Next-Gen Change Data Capture (CDC) Architecture in 2019 - Strea...

Agile Testing Days 2017 Intoducing AgileBI Sustainably - Excercises

IoT NY - Google Cloud Services for IoT

Agile Testing Days 2017 Introducing AgileBI Sustainably

Creating a Modern Data Architecture for Digital Transformation

Big Data Expo 2015 - Pentaho The Future of Analytics

Modernizing Your Data Warehouse using APS

Big Data Analytics in the Cloud with Microsoft Azure

Skilwise Big data

Skillwise Big Data part 2

Data Ingestion in Big Data and IoT platforms

Introduction to Stream Processing

Bridging the Last Mile: Getting Data to the People Who Need It (APAC)

Presentation by Erik van der Hoeven (Wisdom as a Service) at the Data Vault M...

3dw

HIPAS UCP HSP Openstack Sascha Oehl

Hadoop from Hive with Stinger to Tez

"Building Data Warehouse with Google Cloud Platform", Artem Nikulchenko

3dw

Dernier

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows. We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases. This video focuses on the deployment of external web forms using Jotform for Bonterra Impact Management. This solution can be customized to your organization’s needs and deployed to support the common use cases below: - Intake and consent - Assessments - Surveys - Applications - Program registration Interested in deploying web form automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Jeffrey Haguewood

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

MadyBayot

In this keynote, Asanka Abeysinghe, CTO,WSO2 will explore the shift towards platformless technology ecosystems and their importance in driving digital adaptability and innovation. We will discuss strategies for leveraging decentralized architectures and integrating diverse technologies, with a focus on building resilient, flexible, and future-ready IT infrastructures. We will also highlight WSO2's roadmap, emphasizing our commitment to supporting this transformative journey with our evolving product suite.

Platformless Horizons for Digital Adaptability

WSO2

Architecting Cloud Native Applications

WSO2

DBX First Quarter 2024 Investor Presentation

Dropbox

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Zilliz

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

💥 You’re lucky! We’ve found two different (lead) developers that are willing to share their valuable lessons learned about using UiPath Document Understanding! Based on recent implementations in appealing use cases at Partou and SPIE. Don’t expect fancy videos or slide decks, but real and practical experiences that will help you with your own implementations. 📕 Topics that will be addressed: • Training the ML-model by humans: do or don't? • Rule-based versus AI extractors • Tips for finding use cases • How to start 👨‍🏫👨‍💻 Speakers: o Dion Morskieft, RPA Product Owner @Partou o Jack Klein-Schiphorst, Automation Developer @Tacstone Technology

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

UiPathCommunity

Dubai, known for its towering skyscrapers, luxurious lifestyle, and relentless pursuit of innovation, often finds itself in the global spotlight. However, amidst the glitz and glamour, the emirate faces its own set of challenges, including the occasional threat of flooding. In recent years, Dubai has experienced sporadic but significant floods, disrupting normalcy and posing unique challenges to its infrastructure. Among the critical nodes in this bustling metropolis is the Dubai International Airport, a vital hub connecting the world. This article delves into the intersection of Dubai flood events and the resilience demonstrated by the Dubai International Airport in the face of such challenges.

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

Orbitshub

MINDCTI Revenue Release Quarter One 2024

MIND CTI

CNIC Information System with Pakdata Cf In Pakistan

danishmna97

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Understanding the FAA Part 107 License ..

Christopher Logan Kennedy

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Juan lago vázquez

Corporate and higher education. Two industries that, in the past, have had a clear divide with very little crossover. The difference in goals, learning styles and objectives paved the way for differing learning technologies platforms to evolve. Now, those stark lines are blurring as both sides are discovering they have content that’s relevant to the other. Join Tammy Rutherford as she walks through the pros and cons of corporate and higher ed collaborating. And the challenges of these different technology platforms working together for a brighter future.

Corporate and higher education May webinar.pptx

Rustici Software

Dernier (20)

Apidays New York 2024 - The value of a flexible API Management solution for O...

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

presentation ICT roal in 21st century education

AWS Community Day CPH - Three problems of Terraform

EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER

Platformless Horizons for Digital Adaptability

Architecting Cloud Native Applications

DBX First Quarter 2024 Investor Presentation

How to Troubleshoot Apps for the Modern Connected Worker

"I see eyes in my soup": How Delivery Hero implemented the safety system for ...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf

MINDCTI Revenue Release Quarter One 2024

CNIC Information System with Pakdata Cf In Pakistan

Boost Fertility New Invention Ups Success Rates.pdf

Understanding the FAA Part 107 License ..

Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood

Corporate and higher education May webinar.pptx

Data vault seminar May 5-6 Dommel - The factory and the workshop

1. The factory and the workshopOpen source metadata driven data warehousing Johannes van den Bosch @johannesvdb

2. Agenda Background Organization Project Basicarchitecture The factory …and the workshop Topics fromyesterday Real time Business keyintegration Staging out Hierarchies and supertypes Self service BI and writeback Lessonslearnt

3. Waterschap De Dommel Dutch Waterboard in south of Netherlands Managing water quality and quantity for 900.000 citizens and 150.000 hectares 375 employees of which 2 BI Projectsmanagedon time, money and goals Demandforintegrated management information

4. Project Current BI architecturereachedlimits New greenfieldarchitecture Open source advocate Passionatebeliever in costeffectivesolutionsforgovernment It’s our money! Convinced management No software cost Internalhours (2x 0.3 FTE) 1 year

5. Open source software ETL Pentaho Data Integration (Kettle) Data warehouse management Quipu Documentation MediaWiki Modeling Power*Architect

6. EDW architecture Reporting Analysis Dashboards Data mart 1 Data mart n Business Data Vault Source Data Vault 1 Source Data Vault 2 Source Data Vaultn Supplydriven Demanddriven Generated and automated Staging 1 Staging 2 Stagingn Source 1 Source 2 Sourcen

7. Developmentapproach supply driven effort demand driven time

8. Plant, productionlines

9. The factory

10. The factory Source Staging Source Data vault ETL ETL

11. HowQuipuworks Source model Target model Template Load code (ETL)

12. 1. Loadsource model

13. 2. Generatestaging DDL

14. Notcross-platform! …I want my ETL tool..! 3. Generatestaging ETL Default ETL: INSERT INTO staging_table SELECT fields FROM source_table

15. 4. Generatesource data vault model

16. 4. Generatesource data vault DDL

17. 5. Generatesource data vault ETL

18. Starting up the factory

19. The workshop

20. PoC Decided to try and build the bDV and Data Marts 100% virtual bDV = views on top of sDV Data marts = views on top of bDV Conclusion: it is possible

21. Functionalcomponents History Integration Transformation applyingsemantics, filtering, etc.

22. bDV design decisions: full bDV Source data vault Business data vault H T H I H

23. Integrationstrategies 1) same-as link H L H S S 2) integratedhub S H S 3) integratedhub + integratedsatellite H S

24. Integration: hubs Source Source data vault Business data vault person employee_h employee_h_s person_h person_h_s System x Users Users_h Users_h_s System y

25. Hubintegration – virtual System x BK1 System y BK2 New BK Integration business rule

26. Transformation: supertype example sDV Business rule bDV 5___ P______ 4____

27. Transformation: hierarchyexample sDV bDV

28. Data marts: virtual (simpleexample) Dimensionfromhub + sat Factfrom link + sat SCD type 1

29. Virtual: lineage

30. bDV design decisions: partialbDV Source data vault Business data vault T H I H T H T

31. Full bDVvspartialbDV Full Lots of elements to define Easy data marts Partial Lesswork More T between data vault and data marts Multiple versions of the truth

32. VirtualvsPhysical Virtual (views) No physicalmaintenance Easy to adapt Performance limitations Platform defines performance Lineage (dependingon platform) Real time Auditability? Physical Scalability / performance Manualtweaking (indexes, etc.) Surrogatekeys easy More intuitive to develop (ETL in stead of SQL) More complex transformations (ie. aggregations)

33. Self service BI and write back Palofor Excel Open source MOLAP Everycellpoints to location in the cube Writeback to cubepossible EDW cube excel

34. Lessonslearnt Itispossible to quicklybuildan EDW with open source software Somereallycooldevelopments (ie. data mart generation) Automationonlygoessofar Somechallengesstillneed to beaddressed …it is business intelligenceafter all. Automate, ifitsavesyou money Itcan save you time to focus on the important stuff The end product counts: does itdeliveraddedvalue? What’s the best EDW architecture? Itdepends!™

Data vault seminar May 5-6 Dommel - The factory and the workshop

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (20)

Similaire à Data vault seminar May 5-6 Dommel - The factory and the workshop

Similaire à Data vault seminar May 5-6 Dommel - The factory and the workshop (20)

Dernier

Dernier (20)

Data vault seminar May 5-6 Dommel - The factory and the workshop