Data Warehouse Design Considerations

Data WarehouseDesign Considerations
Ram Kedem

Copyright 2014 © Ram Kedem. All rights reserved. Not to be reproduced without written consent. ramkedem.com
Slowly Changing Dimensions
•Type 1 SCD
•OLTP updates are moved into the DW
•Any changes overwrites the current DW data
•Past actual data history is lost
•Historical data may be change if it doesn’t contain important business details (such as store location)

•Type 2 SCD
•Data is not overwritten in the DW
•A new row for the customer must be inserted
•Usually created Primary Key Issues
•For example –if customer details got changed, this approach suggest you insert another row in the Dimension for the same customer
•You must add a Surrogate Key (DWH Key)
•Incremented number for each update, same idea as Primary Key that consists from two columns.
•You must also add another column or two
•To flag the current value
•To provide date / time perspective

•Type 1 SCD

•Type 2 SCD

Understanding Indexing
•Indexing affects how data is stored and managed in SQL Server
•There are four main indexing options in SQL Server
•Clustered Index
•Non Clustered Index
•Filtered Non Clustered Index
•Columnstoreindex (include)

•Clustered Index
•Determines the physical storage order of the data
•There can be only one clustered index on a table
•Non Clustered Index
•Sorts data in a column or columns and stores pointers to the actual data row
•We can have up to 999 non clustered indexes on a table

•Filtered Non Clustered Index
•Creates a non clustered index on a subset of values in a column
•ColumnstoreIndex
•A non clustered index placed on a single column
•The column is stored and searched speratelyfrom the data row
•Adding a columnstoreindex to a column makes the column read- only
•https://www.simple-talk.com/sql/database- administration/columnstore-indexes-in-sql-server-2012/

ColumnstoreIndex
CREATE NONCLUSTERED COLUMNSTORE INDEX csi_products
ON dbo.products
(productName, UnitPrice, unitsinstock);
SELECT productName, UnitPrice, unitsinstock
FROM products ;

Indexing the Data Warehouse
•Indexing in the Data Warehouse can be tricky
•Too few indexes will allow data loads to be quick But query response time will be slow
•Too many indexes slow down load, and storage requirements go up But query response is good

Indexing the Data Warehouse
•General rule of thumb
•Dimension tables
•Place a clustered index on the surrogate key
•If the table has a lot of columns, create non-clustered indexes on the most popular columns
•Fact tables
•Place a non-clustered index on the single-column foreign keys to the dimension tables
•If the primary key is a composite of all the dimension foreign keys, make it a non-unique clustered index.

Understanding Index Views
•What is a view
•A result set of a query that is a virtual table
•The virtual table is not stored permanently in the database.
•The view can be referenced like a table in TSQL
•Indexing a view
•You can create a unique clustered index on a view
•The view result set get stored in the database, just like a regular table with a clustered index.

Understanding Index Views
•Advantages
•Improve the performance of joins and aggregations that process many rows

Understanding Data Compression
•SQL Server 2012 Supports data compression
•Data compression reduces the size of the database
•Packs more data onto few data pages
•Fewer data page reads are required to satisfy queries
•Lower IO means faster response; lower processing load on the server
•Extra CPU resource are required for data decompression / compression
•DWH usually doesn’t have much updates (other than Bulk Loading)

•SQL Server 2012 supports three compression types
•Page compression
•Focuses on duplicated values within the data page
•Stores one value, place a pointer at all other locations
•Row Compression
•Remove any unused bytes in a fixed data type
•CHAR(25)
•Unicode compression
•Reduces storage space for Unicode data that doesn’t require that space

•Which compression should you use
•Page compression
•It automatically uses row compression when page compression is used
•If you use row compression, you cant use page compression
•Facttables usually benefit the most from compression
•Compression is only available in SQL Server Enterprise Edition.

Data Lineage
•What is data lineage
•Data origination and flow details
•Where it is from, where it is going, how it is transformed in the process
•Same concept as comments in programming

Data Lineage
•Why do we need Data Lineage
•To provide meta-data context in the DWH
•Future business rules may change, affecting some data
•Making it invalid
•Making it suspect
•Making it more important
•Data lineage allows us to identify this data

Data Lineage
•Two main options for adding Data Lineage
•SSIS system variables
•TSQL System functions

Data Lineage using TSQL
SELECT
APP_NAME () ,
DATABASE_PRINCIPAL_ID (),
USER_NAME ()
SUSER_NAME (),
GETDATE () ,
CURRENT_TIMESTAMP () ,
CONNECTIONPROPERTY (‘Client_net_address’)

Using Partitions
•Fact tables become very large tables over time
•Very large database tables present serious challenges
•What if you need to delete large portion of the data ?
•TRUNCATE TABLE command performs deletion with minimal logging, but it deletes the entire table.
•Large data inserts become time consuming
•Index maintenance and storage can become problematic
•Table partitions deal with all these issues

Using Partitions
•What is a table partition
•A large table is stored in multiple files
•Divided by rows (based on condition)
•Usually date / time
•SQL SERVER 2012 allows up to 15,000 partitions on a single table
•Partitions and data are managed in the background

Using Partitions

Identifying our Dimensions / Fact Tables

Data Warehouse Design Considerations

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Data Warehouse Design Considerations

Similaire à Data Warehouse Design Considerations (20)

Plus de Ram Kedem

Plus de Ram Kedem (20)

Dernier

Dernier (20)

Data Warehouse Design Considerations