Presentation held on may 6th 2011 on the Advanced Dat Vault seminar in the Netherlands: Abstract: Data Warehousing is still being ridiculized by popular literature and opportunistic vendors (and sometimes analysts) in the Business Intelligence domain - they tend to call it 'traditional' as opposed to their 'silver bullet technology'. However, data warehousing has evolved and Data Vault - although undervalued by many - is fueling this evolution. Data Vault methodology enables architects to (finally) embed data warehousing into the Enterprise Architecture, something they struggled with the last 15 years. In the Netherlands, Data Vault sky rocketed innovation in the data warehousing scene. Accelerators in terms of frameworks and software are being build by experts in the field and several product vendors picked up on it. The presentation of Ronald Damhof will discuss briefly the evolution of data warehousing as it stands today, the position of Data Vault in the Enterprise Architecture, the different species/forks that exist in Data Vault and the automation that comes with it.

1. “It is not the strongest of the species that survives, nor the most intelligent that survives. It is the one that is the most adaptable to change” Charles Darwin Data Vault is anevolutionary step Data Vaultfirmlypositioned data warehousing in EA Data Vaultforksinto different species Andyes, we can speed up evolution

4. 3:1 mileage Expensive to buy Slow…. Inflexible Carbon based, polluting! Need specialists to drive Need specialists to repair

5. It can go anywhere It can go anytime It can be used by anyone Very fast!! Of course, extremely energy efficient However…. It won’t fly

6. Sorry, not more than 100 miles Oh, charging takes like 5 hours Batteries are somewhat expensive Prone to (electric) failure No infrastructure yet

7. Utilize new methods and technologies Make it effective with todays legacy Utilizing todays infrastructure As well as adapting to new ones Retain the good, get rid of the bad Make it efficient, better mileage Make it durable Make it repeatable Make it affordable Make it Agile Make it fit in the environment

8. What products are asked? What are the quality characteristics? How are these products made?

9. What

10. Product & Services What

11. What Compliant Adaptible Sustainable Decoupled Centralized Standardized & Industrialized Effective

12. How ‘Calculating risk’ Source ‘Yield modules’ Source ‘Customer segmentation’ Semantic gap

13. How 4. Generate (BI) products 3. Enrich and cleanse data 2. Register & (anchorize data) 1. Get the raw (uncut) data Information Delivery Proces

14. Recipient End-user (Local) 4 4 4 4 4 Data & function service 3 3 3 3 3 Information Delivery process 2 2 2 2 2 1 1 1 1 1 Generic BI proces (Central) Data sources(internal & external)

15. Key Design Decisions Adaptable Sustainable Compliant Centralized

16. Adaptable Effectiveness Sustainable Decoupled Compliant Centralized

17. Key Design Decisions Compliant Adaptable

18. View: Component view 1 2 3 4 Company xxx data warehouse & Business Intelligence Domain 4 Sources BI apps Reports 3 2 Source store 1”, 2” Business View, Data feeds BI AppsAnalysis 1 Enterprise Data Warehouse BI Apps Ad-hoc Function, ‘How’ External sources Data, ‘What’ ‘Where’, ‘Whom’

19. Sourcestore to BV Sourcestore to product Source to product EDW (DV) Adaptable Sustainable Compliant Decoupled Effective Standardized Centralized

20. View: Component view 1 2 3 4 Company xxx data warehouse & Business Intelligence Domain 4 Sources BI apps Reports 3 2 Source store 1”, 2” Business View, Data feeds BI AppsAnalysis 1 Enterprise Data Warehouse BI Apps Ad-hoc Function, ‘How’ External sources Data, ‘What’ ‘Where’, ‘Whom’

21. Administrative process Information Delivery Process Decision- & control Data & Information recipients Generate& Distribute Enrich Register (& anchorize) Attain Proces PDCA DV basedData Warehouse Systems(internal &external) Information products Compliance reporting Risk Management Supply/Data Demand/Function Performance Management Data products Businessrules Supply chain optimization Staging Fraud detection Market basket analysis Control / Metadata

23. Performance of the systems has direct effect on efficiency of users

24. Most core business activities are ‘one line’

25. Systems work is mostly maintenance

27. Performance of the systems has direct effect on efficiency of users

28. New systems promise major processand service improvement

29. New systems promise major costreduction

31. Performance of the system has no direct effect on efficiency of users

32. Company canquicklyrevertto manual procedures

34. New systems promise major costreductions

35. New systems will close significant cost, service or process performance withcompetiters

36. IT constitutes more than 50% of capitalspending

37. IT makes up more than 15% of total corporate expensesData Vault & Governance Strategy Need for reliability in IT Need for innovation with IT

39. Externalfocused

40. Competitative intelligence

42. Control of assets

43. Relaiabilityand availability of:

44. Information

45. IT Services

46. Efficiency / Cost Control

47. Security

48. Avoid surprises

49. Manage risks

50. Project Portfolio

51. Stable ArchitectureData Function Need for reliability in IT Need for innovation with IT

52. Data Vault & Self ServiceThe development model Central functiondevelopment CentrallycoordinatedInfrastructuredevelopment Gedelegeerde Ontwikkeling Localfunctiondevelopment Localfunctiondevelopment Selfservice development Delegateddevelopment Selfservice development Delegateddevelopment Function (Opportunisticdevelopment) Data (Systematic development) Data CentrallycoordinatedICT development

53. Data Vault Species

54. 1 - Classic Data Vault Business Transaction System Data Vault Data Marts Staging Out Business Transaction System Generic Business Rules Rule Vault Structure transformation Hub = business keys Business rule execution Structure and value transformation Standardized Centralized Adaptable Effectiveness Sustainable Decoupled Compliant ? ?

55. 2 - Source Data Vault Business Data Vault Staging Vault Business Transaction System Data Marts Business Transaction System Staging Vault Structure transformation No integration, Hub=surrogate keys Persisting staging in DV format Business rule execution Integration DV modelled Structure transformation Standardized Centralized Adaptable Effectiveness Sustainable Decoupled Compliant ? ? ?

56. Source Source 100% Semantic gap Business DV Source Staging DV Source Staging DV 100% Semantic gap Still the source Integration, cleansing, consolidation Business rule execution upstream ?? DV modelled

57. Source Source 100% Semantic gap Data Warehouse Business DV Source Source Staging DV Source Source Staging DV 100% Semantic gap Still the source Integration, cleansing, consolidation Business rule execution upstream ?? DV modelled

59. Information interdepence

60. Sponsorship level

61. Strategic View (decoupled)

62. Information maturity; conceptual business process knowledge present

63. Scale and complexity of data and/or organization

64. Relatively big semantic gap between Source  Requirement

65. Factors that contribute to (2)

66. Scale and complexity of data and/or organization

67. Bad data quality

68. Business keys hard to identify

69. Information maturity; no conceptual business process knowledge present

70. Urgency

71. Resource constraints

73. 1b – Classic Data Vault Business Transaction System Data Vault Data Marts Staging Out Business Transaction System Generic Business Rules Rule Vault Business Transaction System Data Vault Data Marts Staging Out Business Transaction System Generic Business Rules Rule Vault Structure transformation Light integration on the business keys Specific business rule execution Structure and value transformation Consolidation

74. Speeding up Data Vault Evolution?

75. Data Vault Innovation

78. 80-20 rule

79. Not everything can be generated, be real

80. Truly understand the mechanics

81. Do not underestimate the complexity

82. It is a pretty steep learning cycle

83. PoC, PoC, PoC

84. Make it yourself

85. Then “generate”

86. Generation software only in combo with modeling software and ETL software

88. Aim is to be 100% declarative

90. More advanced; generating XML code for 2nd gen. ETL tooling

93. Both DV species will live next to others

94. Technology will push the envelope

95. Automation typologies will be pushed even more towards metadata driven automation

96. WE – the people – need to evolve:

97. We need more talented engineers

98. We need more focused education

99. We need more knowledge sharing