Whats new in biGENiUS; Robert Kranabether - Trivadis TechEvent 2019

1. What‘s new in biGENiUS
Robert Kranabether

2. Agenda
• Application
• New Data Flow Set Management
• New Source Term Mapping Management
• Custom Property Types
• Target Objects
• Migration Framework
• Data Vault Generator
• Load Control Management Application

3. Demo
Data Flow Sets
Source Term Mappings
Custom Properties

4. Target Objects

5. Overview Generator Architecture
Cleansing
Core
Data Mart
Staging
Runtime Config
LCM
Architecture consists of
– DWH parts
– DWH part groups
– DWH componentsFactEntity
Entity Fact Dimen-
sion
Entity Fact Dimen.
Stage
SQL
Stage
Flat
File
Table
Loader
View
Action
Table
Loader
View
Action
Table
Loader
View
Table
Loader
View
Table
Loader
View
View View View
Table
Loader
View
Table
Loader
View

6. Applying a generator architecture to a
model
Model
Product
Sales
SubCategory
Category
Generator Configuration
Trivadis Best Practice
generate
• CA.Category_Cleansing
• CA.Product_Cleansing
• CA.Sales_Cleansing
• CA.SubCategory_Cleansing
• CA.CleansingActionLog
• DW.Category
• DW.Product
• DW.SubCategory
• DW.Sales
• CA.Category_Level
• CA.Product_Level
• CA.Sales_Fact
• CA.SubCategory_Level
• DW.Category_MergeView
• DW.Product _MergeView
• DW.SubCat _MergeView
• DW.Sales _MergeView
• DM.Category
• DM.Product
• DM.SubCategory
• DM.Sales
• CA.Category_CA1
• CA.Category_Level_Loader
• CA.Product_CA1
• CA.Product_Level_Loader
• CA.Sales_CA1
• CA.Sales_Fact_Loader
• CA.SubCategory_CA1
• CA.SubCategory_Level_Loader
• DW.Category_Core_Loader
• DW.Product _Core_Loader
• DW.SubCat _Core_Loader
• DW.Sales _Core_Loader
• CA.CA_FinalizeCleasning
• CA.CA_PrepareCleasning
• DM.DM_Finalize_Core
• DM.DM_Prepare_Core
• DW.DW_Finalize_Core
• DW.DW_Prepare_Core

7. Target Objects
DWH object „Person“
Cleansing Core Mart
CRM
ERP
Source
System
ResultSource Loader
1Source ResultLoader Action
Source ResultLoader Action
CA_EN_VI_SQL
CA_EN_LO_SQL
CA_EN_TA_SQL
1
2
Source
Result
Loader
Source Loader
1
11
22
Staging
ResultSource Loader
ResultSource Loader
1
1
Usage=Source
Type=View
SourceLayer=CA
Usage=Loader
Type=Procedure
DirectDependencies=CA_EN_AC3_SQL
RelationshipDependencies=DW_EN_LO_SQL
Usage=Result
Type=Table
Usage=Source
Type=View
Options=VirtualOnly
Usage=Result
Type=Table
Usage=Source
Type=View
Options=UseSqlCode,
BuildEmptySelect
Usage=Loader
Type=Procedure
DataFlowDependencies=SA_ST_LO_SQL,
SA_SC_LO_SQL
Usage=Result
Type=Table
Usage=Action
UsageType=Cleansing
Type=Procedure
Status=Activated
DirectDependencies=CA_EN_LO_SQL
RelationshipDependencies=CA_EN_LO_SQL
Usage=Loader
Type=Procedure
DirectDependencies=DW_EN_LO_SQL
SourceLayer=DW
Usage=Source
Type=View
Usage=Loader
Type=Procedure
Usage=Result
Type=Table

8. Target Objects in EM Viewer
foreach(var targetObject in Instance.GetTargetObjects(dwhPartName))
{
BuildCleanseView(targetObject);
}
foreach(var term in targetObject.BusinessKeys.Where(t => !t.IsForeignKey))
{
WriteLine(" {0,1}{1,-50}", delimiter,
ToSqlName("ca", term.ColumnName));
}

9. Migration Framework

10. So what‘s the point here?
biGENiUS offers you a wide range of customization and extensibility
Meanwhile, biGENiUS is constantly being developed over time.
Project start
Development
- customized rules
- templates
- settings
Release of new project version
Increasing complexity over
time
Development
- Adoption of standardizable customizations
- Introduction of new features
- Bug Fixes
Release of biGENiUS/generator version
biGENiUS
development

11. Why do you need this?
• You started some time ago using biGENiUS
• You want to…
• get rid of customizations that are now part of the standard
• use the latest generator features e.g. Data Vault 2.0, …
• benefit from the latest UI version e.g. improved data flow mappings, LCM Application

12. Which challenges do we have?
• Migration of UI and Meta DB is relativly easy, but that ain't true for the generator configuration!
• We have to ensure…
• to migrate all relevant nuts and bolts
• integrity of your modeled data
• integrity of your already loaded data and operation of your production data warehouse

13. What do we provide?
• Specialized meta database that supports your migration project
• Extensive reporting for changes between legacy and latest meta db version
• Identification of changes within customized and stock templates
• Automated way to import enties, facts, custom properties and settings

14. What does the upgrade process look like?
1. Setup
migration
environment
•Import legacy meta db
and create a new (up to
date) one
•Create the DWH in the
migration environment
2. Analyze
•Generate migration
report
•Identify changes to the
meta db and derive
required manual
adjustments
•Identify changes to the
templates and derive
required adjustments
•Generate migration
script
3. Migration
•Manual adjustments to
new meta db
•Execute the migration
scripts to import data
4. Testing
•Generate & deploy
migrated DWH
•Run initial &
incremental loads and
compare legacy and
new DWH
•Delta deployment to
migrate legacy DWH

15. Data Vault Generator

16. Data Vault
Dani Schnider explains: https://www.youtube.com/watch?v=Q1qj_LjEawc
▪ Areas:
▪ Raw Vault
▪ Business Vault
▪ Main Objects
▪ Hubs
▪ Links
▪ Satellites

17. BeerVault Architecture

18. Demo
Data Vault Generator

19. Demo
Load Control Management Application