Change Data Capture for Business Intelligence

1. The foundation of Business Intelligence
Jim Deppen
deppenjim@gmail.com
1

2.  Jim Deppen, Technology Specialist with HP
Enterprise Services.
 Servicing as a Data Architect for the US Army
Human Resources Command.
 20+ years Experience in Database
Technologies.
 Email: deppenjim@gmail.com
 Twitter: @jimdeppen
2

3.  Isn’t the Development Framework the most
important thing for successful a BI
application?
 How does this align with Kimball or Inmon?
 How does this support the Star Schema?
 Isn’t BI just about creating great Dashboards?
3

4. 4

5. 5

6.  BI is more than turning a traditional report
into a dashboard.
 BI is about building a structure of data that
allows for new insights to be discovered.
6

7.  Change Data Capture is a process to capture
insert, update, and delete activity from a
data source.
 Change Data Capture is the most accurate
technique to detect a business event.
7

8. “Business events are individual
actions performed by people or
organizations during the
execution of business
processes”.1
1 Agile Data Warehouse Design by Lawrence Corr
8

9.  A strategy of capturing business event data
as it occurs is the most accurate technique to
build consistent business metrics.
 Important business metrics are many times
lost due to poor Operational Application
Database Design.
 Many Operational Applications have a lack of
analytic foresight.
9

10.  Knowing the factors that led to a positive
outcome will lead to insights of more positive
outcomes in the future.
 In the end, knowing the what factors led to an
outcome, is more important than the
outcome itself.
10

11.  Operational Applications were not designed
to capture events. The intermediate steps
are not important.
 An Operation Application may overwrite a
number of transitional steps before
completing the transaction.
11

12.  The business process may cross multiple
internal applications.
 Key data is coming from external Internet
sources.
12

13. Question:
How many of you have ever had a customer
ask: "Hey, can you help us track some Business
Events?”
13

14.  Can you help us duplicate some reports?
(Bob quit. Joe accidentally deleted our
Access Database. )
 Can you help us turn these reports into an
Executive Dashboard?
 We are having trouble manipulating the 10
million rows in our Excel Spreadsheet. Can
you us help make it faster?
14

15.  Understanding the customer’s business
process should be your top priority.
 This is more important than understanding
their computer systems and applications.
15

16.  Conduct meetings with business stakeholders
to discover the business process time lines.
 Brainstorm business events that lead to
positive outcomes.
 Understand the transitions from one event to
the next.
 Understand the whole life cycle.
16

17. 17-24 Year Old Population (QMA) 17-24 Year Old Population
Target Prospect Population (Army Qualified) College Bound
College Bound Qualified
Lead (Refined)
Interest
Appointments Made
MSL I
Appointments Conducted
MSL II
MSL III
Test Passed
MSL IV
Contract CMSN
Access BOLC II
IET BOLC III
First Unit First Unit
First Term First Term
17

18. Great Reference on the subject:
Agile Data Warehouse Design
by Lawrence Corr
18

19.  Business Events are the Facts in the
Dimensional Model.
19

20.  Database Triggers
 Database detection of table Insert,
Update, and Delete events.
 Supported by all major databases.
 Can be added to COTS applications with
little or no impact.
20

21.  Data Modification Stamping (also known as
Change Tracking). Involves the addition of
tracking columns to a table.
 Time Stamping
 Status Indicator
 Version Number
21

22.  Database transaction log scanning. Decode
database transaction logs to detect
database Insert, Update and Delete events.
 Examples
 Microsoft SQL Server Change Data Capture
 IBM InfoSphere Change Data Capture
 Oracle Goldengate
 Can be added to COTS applications with no
impact.
22

23.  Sales prospect becomes a customer
by purchasing a product.
 Sales prospect clicks online ad link.
 Sales prospect calls TV ad 800 number to inquire
about your product.
 Salesperson sets a follow-up date with prospect.
 Customer’s purchase depletes the last product.
 Gross Margin of a product falls below a specified
level.
23

24. 24

25. 25

26. 26

27. 27

28. 28

29. 29

30. Sales Prospect
becomes a customer
by purchasing a
product
30

31. Gross Margin of a
Product falls below a
specified level.
31

32. Sales Prospect clicks
on an online ad link
32

33. Sales Prospect clicks
on an online ad link
Sales Prospect calls
TV 800 Number
33

34. Sales Prospect clicks
on an online ad link
Sales Prospect calls
TV 800 Number
Sales Person sets
follow-up date with
prospect
34

35. Customer purchase
depletes the last
product from stock
35

36.  Sales prospect becomes customer by
purchasing a product.
 A salesperson changes the Person Type from
“Prospect” to “Customer” in the Sales Application.
 This changes the Person_Type_Code from ‘P’ to
‘C’ on the “Person” Table
 This update action overwrites the information
about a persons prior “Prospect” status.
36

37.  You no longer know the time it took to
convert this person into a customer.
 The time this conversion takes may be key to
increasing sales, especially if this could be
correlated to other business events.
37

38.  Build an infrastructure and a process to
capture business events.
 Choose a method to detect Business Events on
your database. (Database Triggers, Change
Columns, or Transaction Log Scanning).
 Build a table to store your Business Events.
 Use the Business Events as the seeds to populate
your BI Fact Table.
38

39. 39

40. CREATE TRIGGER NewProspect ON Person
FOR INSERT
AS
INSERT INTO Business_Event
(Business_Event_Type_ID, Table, Column, Key_Value, Data_Value)
SELECT
(select Business_Event_ID from Business_Event_Type where Business_Event_Desc = 'NewProspect'),
'Person', 'Person_Type_Code', convert(varchar, Person_ID), convert(varchar, Person_Type_Code)
FROM inserted
40

41. Event Business Event Date Table Column Key Data Value
ID Event Stamp Value
Type ID
1 1 2012-07-05 11:09 Person Person_Type_Code 100 P
AM
2 1 2012-07-05 12:15 Person Person_Type_Code 102 P
PM
3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C
4 5 2012-07-07 9:23 AM Product Current_Stock 9519 0
5 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com
Contact /julyoffer
6 7 2012-07-07 10:12 Order_Ite Calculated 9519 -2.76
AM ms
7 12 2012-07-07 11: 11 Customer Followup_Date 960 2012-07-21
AM Contact
8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122
Contact
41

42. Event Business Event Date Table Column Key Data Value
ID Event Stamp Value
Type ID
1 1 2012-07-05 11:09 Person Person_Type_Code 100 P
AM
2 1 2012-07-05 12:15 Person Person_Type_Code 102 P
PM
3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C
4 5 2012-07-07 9:23 AM Product Current_Stock 9519 0
5 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com
Contact /julyoffer
6 7 2012-07-07 10:12 Order_Ite Calculated 9519 -2.76
AM ms
7 12 2012-07-07 11: 11 Customer Followup_Date 960 2012-07-21
AM Contact
8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122
Contact
42

43. Event Business Event Date Table Column Key Data Value
ID Event Stamp Value
Type ID
1 1 2012-07-05 11:09 Person Person_Type_Code 100 P
AM
2 1 2012-07-05 12:15 Person Person_Type_Code 102 P
PM
3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C
4 5 2012-07-07 9:23 AM Product Current_Stock 9519 0
5 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com
Contact /julyoffer
6 7 2012-07-07 10:12 Order_Ite Calculated 9519 -2.76
AM ms
7 12 2012-07-07 11: 11 Customer Followup_Date 960 2012-07-21
AM Contact
8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122
Contact
43

44. Event Business Event Date Table Column Key Data Value
ID Event Stamp Value
Type ID
1 1 2012-07-05 11:09 Person Person_Type_Code 100 P
AM
2 1 2012-07-05 12:15 Person Person_Type_Code 102 P
PM
3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C
4 5 2012-07-07 9:23 AM Product Current_Stock 9519 0
5 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com
Contact /julyoffer
6 7 2012-07-07 10:12 Order_Ite Calculated 9519 -2.76
AM ms
7 12 2012-07-07 11: 11 Customer Followup_Date 960 2012-07-21
AM Contact
8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122
Contact
44

45. Event Business Event Date Table Column Key Data Value
ID Event Stamp Value
Type ID
1 1 2012-07-05 11:09 Person Person_Type_Code 100 P
AM
2 1 2012-07-05 12:15 Person Person_Type_Code 102 P
PM
3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C
4 5 2012-07-07 9:23 AM Product Current_Stock 9519 0
5 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com
Contact /julyoffer
6 7 2012-07-07 10:12 Order_Ite Calculated 9519 -2.76
AM ms
7 12 2012-07-07 11: 11 Customer Followup_Date 960 2012-07-21
AM Contact
8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122
Contact
45

46. Event Business Event Date Table Column Key Data Value
ID Event Stamp Value
Type ID
1 1 2012-07-05 11:09 Person Person_Type_Code 100 P
AM
2 1 2012-07-05 12:15 Person Person_Type_Code 102 P
PM
3 2 2012-07-07 9:23 AM Person Person_Type_Code 102 C
4 5 2012-07-07 9:23 AM Product Current_Stock 9519 0
5 10 2012-07-07 9:45 AM Customer_ Reference_URL 100 http://www.acme.com
Contact /julyoffer
6 7 2012-07-07 10:12 Order Calculated 9519 -2.76
AM Items
7 12 2012-07-07 11: 11 Customer Follow-up Date 960 2012-07-21
AM Contact
8 11 2012-07-07 1:29 PM Customer Sales_Person_ID 987 122
Contact
46

47.  A central location to drive the population of
all Fact and Slowly Changing Dimension
tables.
 Elimination for the need to flush and reload
Fact and Dimension tables.
 Potential to process data in real-time.
 Ability to capture business event from status
columns.
 Capability to retain history of business
events over time.
47

48.  BI’s most important function is the analysis of
Business Events.
 Learn the business process first.
 Help customer to look beyond traditional
reporting.
 Data about intermediate steps in a Business
Process are usually lost.
 CDC is the most accurate way to capture this
business event data.
48

49. Change Data Capture
The foundation of Business Intelligence
deppenjim@gmail.com
@jimdeppen
49

50. The foundation of Business Intelligence
Jim Deppen
deppenjim@gmail.com
50