Presentation performed in IM 2011.

1. A Solution for Identifying the Root Cause of
Problems in IT Change Management
Ricardo L. dos Santos, Juliano A. Wickboldt, Roben C. Lunardi, Bruno L. Dalmazo,
Lisandro Z. Granville and Luciano P. Gaspary
Federal University of Rio Grande do Sul, Brazil
Claudio Bartolini
Hewlett Packard Laboratories, Palo Alto, USA
Marianne Hickey
Hewlett Packard Laboratories, Bristol, UK
© 2009 Hewlett-Packard Development Company, L.P.
The information contained herein is subject to change without notice Labs & Institute of Informatics – UFRGS
Malia Project – HP

2. Outline
• Introduction
• Related Work
• Proposed Solution
• Diagnosis Process
• Information Model
• Conceptual Architecture
• Root Cause Analyzer
• Conclusion and Future Work
Malia Project – HP Labs & Institute of Informatics – UFRGS
2
Wednesday, March 06, 2013

3. Introduction
• Context
• The complexity of IT infrastructure makes the IT
processes a critical mission
• ITIL has become important for IT processes
• IT Change Management
• Defines how the IT infrastructure must evolve in a
consistent and safe fashion
• Defines how changes should be conducted
Malia Project – HP Labs & Institute of Informatics – UFRGS
3
Wednesday, March 06, 2013

4. Introduction
• IT Problem Management
• Defines how lifecycle of IT problems is managed
• Its main goals are
• Preventing the occurrence of IT related problems
• Eliminating recurring problems
• Minimizing the impact of problems that cannot be prevented
• Allows to simplify procedures for detecting the root
cause of IT problems and reducing the associated
costs
Malia Project – HP Labs & Institute of Informatics – UFRGS
4
Wednesday, March 06, 2013

5. Introduction
• Current Scenario
• Organizations use tools for to help in analysis and
identification of the root cause of failures
• When they do not point directly into the root cause, is able to
provide important information to the responsible technical staff
• Enables the reuse of knowledge, which results in a reduction of
associated costs
• However, these tools have serious limitations, such as
• The description of cases in diagnostic tools is performed in ad
hoc fashion
• Poor reusability of cases
• Low maintainability
Malia Project – HP Labs & Institute of Informatics – UFRGS
5
Wednesday, March 06, 2013

6. Introduction
• Problems
• Often diagnoses are not completed correctly due to
outdated cases
• The complexity of IT infrastructure where the failure
occurs is ignored
• Data of previous diagnoses are not considered
Malia Project – HP Labs & Institute of Informatics – UFRGS
6
Wednesday, March 06, 2013

7. Introduction
• Objectives
• Propose an interactive method for identifying causes of
failure in change processes
• Based on information of the diagnosis of previous failures in
change activities
• Considering the affected IT infrastructure
• Optimize the root cause identification process of
failures in IT changes
• Reuse the knowledge acquired in previous diagnoses
Malia Project – HP Labs & Institute of Informatics – UFRGS
7
Wednesday, March 06, 2013

8. Proposed Solution
Diagnosis Process
Help Desk Root Cause Operator
Analyzer
IR RC
Iterative Diagnosis
Incident Report Question Answered Root Cause
Selection Question
Malia Project – HP Labs & Institute of Informatics – UFRGS
9
Wednesday, March 06, 2013

9. Proposed Solution
Conceptual Architecture
Change Change Deployment
CI
CI
Designer Planner System CI
Change Management System
Root Cause
Analyzer
Config. Mgmt. RC
Database Diagnosis Log
Operator
Recorder
Diagnosis System
CI
L
CI CI
CI
CI
Input Weight Question Question
IR RC
Processor Calculator Selector Verifier
RC
RC
RC
Root Cause Analyzer
Malia Project – HP Labs & Institute of Informatics – UFRGS
10
Wednesday, March 06, 2013

10. Proposed Solution
Information Model
ManagedElement
ManagedSystem
ExchangeElement
Element
SolutionCategory PRS_ServiceInciden
Logical Element SolutionElement Category
1..* * t
* * 1 1..*
CategoryParentChild
0..1 1 QuestionCategory
EnabledLogical
PRS_Problem
Element
0..1
RecordedProblem 0..1 0..1 RootCause
1 ▲ ServiceProblem
MessageLog
determinesProblem
determinesOthersQuestions
1..* 1..* * 0..*
1..* possiblesAnswers 1
RecordLog Answer Question
1 1 1 1
RecordedAnswers
RecordedQuestions
Malia Project – HP Labs & Institute of Informatics – UFRGS
11
Wednesday, March 06, 2013

11. Proposed Solution
Root Cause Analyzer
Operator
Reports the
incident
Log
Log CI
Log CICI
CI
Priority
…
Malia Project – HP Labs & Institute of Informatics – UFRGS
14

12. Proposed Solution
Root Cause Analyzer
Category that
Identif ication Calculates the
has the Obvious?
based on IR weight of RCs
greatest weight
RC Calculates the
RC
RC Identif ication
weight of
Question that
based on has the 80% with the
Questions and
categories greatest weight same answer
Categories
Calculates the Threshold
weight of
CI
L
CI
Identif ication
based on RCs possible
Question with
higher level
Answers
Input Processor Weight Calculator Question Selector Question Verifier
Root Cause Analyzer
Malia Project – HP Labs & Institute of Informatics – UFRGS
15
Wednesday, March 06, 2013

13. Proposed Solution
Root Cause Analyzer
Operator
Log
Log CI
Log CICI
IDENTIFIES:
•Logs of previous diagnoses
•RCs, Questions and Answers (categories)
•Affected CIs
Malia Project – HP Labs & Institute of Informatics – UFRGS
16

14. Proposed Solution
Root Cause Analyzer
• An example of the affected IT infrastructure
Malia Project – HP Labs & Institute of Informatics – UFRGS

15. Proposed Solution
Root Cause Analyzer
Log
Log CI
Log CICI
CI Category
CI Category
Internet Third Party Services
WEB Server (215) System Computer System Server Web Server
DNS Server (423) System Computer System Server DNS Server
Apache 2.2 Software Web Server
PHP 5.0.2 Software Programming Language
Hard Disk Device Storage Devices
Network Card Device Network Cards
Malia Project – HP Labs & Institute of Informatics – UFRGS
18

16. Proposed Solution
Root Cause Analyzer
Category that
Identif ication Calculates the
has the Obvious?
based on IR weight of RCs
greatest weight
RC Calculates the
RC
RC Identif ication
weight of
Question that
based on has the 80% with the
Questions and
categories greatest weight same answer
Categories
Calculates the Threshold
weight of
CI
L
CI
Identif ication
based on RCs possible
Question with
higher level
Answers
Input Processor Weight Calculator Question Selector Question Verifier
Root Cause Analyzer
Malia Project – HP Labs & Institute of Informatics – UFRGS
19
Wednesday, March 06, 2013

17. Proposed Solution
Root Cause Analyzer
Log
Log
Log CI
CICI
Select questions and
answers based on the Set of RCs, Answers, and Questions
categories of CIs
identified
Configuration
Mgmt. System
RC RC RC RC RC
Malia Project – HP Labs & Institute of Informatics – UFRGS
20

18. Proposed Solution
Root Cause Analyzer
RC RC RC
System Computer System Server Web Server RC RC RC
RC RC RC
Application Web Server RC RC RC
RC RC RC
Device Network Cards RC
RC RC
RC
RC RC
Third Party Services
RC
Application
System
Device
74 95 35
Computer System
Network Card
Web Server
30
30 70
Server
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
21

19. Proposed Solution
Root Cause Analyzer
Software
System
Device
Computer System
Network Card
Web Server
Server
Third Party Services
Web Server
Malia Project – HP Labs & Institute of Informatics – UFRGS
22

20. Proposed Solution
Root Cause Analyzer
Category that
Identif ication Calculates the
has the Obvious?
based on IR weight of RCs
greatest weight
RC Calculates the
RC
RC Identif ication
weight of
Question that
based on has the 80% with the
Questions and
categories greatest weight same answer
Categories
Calculates the Threshold
weight of
CI
L
CI
Identif ication
based on RCs possible
Question with
higher level
Answers
Input Processor Weight Calculator Question Selector Question Verifier
Root Cause Analyzer
Malia Project – HP Labs & Institute of Informatics – UFRGS
23
Wednesday, March 06, 2013

21. Solução Proposta
Root Cause Analyzer
Software
System
Device
74 95 35
Computer System
Network Card
Web Server
30 70 30
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
24
6 March, 2013

22. Proposed Solution
Root Cause Analyzer
Category that
Identif ication Calculates the
has the Obvious?
based on IR weight of RCs
greatest weight
RC Calculates the
RC
RC Identif ication
weight of
Question that
based on has the 80% with the
Questions and
categories greatest weight same answer
Categories
Calculates the Threshold
weight of
CI
L
CI
Identif ication
based on RCs possible
Question with
higher level
Answers
Input Processor Weight Calculator Question Selector Question Verifier
Root Cause Analyzer
Malia Project – HP Labs & Institute of Informatics – UFRGS
25
Wednesday, March 06, 2013

23. Example Selection of Questions
Current Version
Application
System
Device
74 95 35
Computer System
Network Card
Web Server
30 70 30
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
26
6 March, 2013

24. Example Selection of Questions
Current Version
Software
System
Device
74 95 35
Computer System
Network Card
Web Server
30 70 30
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
27
6 March, 2013

25. Proposed Solution
Root Cause Analyzer
Category that
Identif ication Calculates the
has the Obvious?
based on IR weight of RCs
greatest weight
RC Calculates the
RC
RC Identif ication
weight of
Question that
based on has the 80% with the
Questions and
categories greatest weight same answer
Categories
Calculates the Threshold
weight of
CI
L
CI
Identif ication
based on RCs possible
Question with
higher level
Answers
Input Processor Weight Calculator Question Selector Question Verifier
Root Cause Analyzer
Malia Project – HP Labs & Institute of Informatics – UFRGS
28
Wednesday, March 06, 2013

26. Example Selection of Questions
Current Version
Application
Questions RC 1 (10)
System
Device
74 95 35
Questions RC 2 (20)
Computer System
Questions RC 3 (40)
Network Card
Web Server Questions RC 4 (25)
30 70 30
Questions RC 1,3 (50)
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
29
6 March, 2013

27. Proposed Solution
Root Cause Analyzer
Category that
Identif ication Calculates the
has the Obvious?
based on IR weight of RCs
greatest weight
RC Calculates the
RC
RC Identif ication Question that
weight of
based on has the 80% with the
Questions and
categories greatest weight same answer
Categories
Calculates the Threshold
weight of
CI
L
CI
Identif ication
based on RCs possible
Question with
higher level
Answers
Input Processor Weight Calculator Question Selector Question Verifier
Root Cause Analyzer
Malia Project – HP Labs & Institute of Informatics – UFRGS
30
Wednesday, March 06, 2013

28. Example Selection of Questions
Current Version
Software
Questions RC 1 (10)
System
Device
74 95 35
Questions RC 2 (20)
Computer System
Questions RC 3 (40)
Network Card
Web Server Questions RC 4 (25)
30 70 30
Questions RC 1,3 (50)
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
31
6 March, 2013

29. Proposed Solution
Root Cause Analyzer
Category that
Identif ication Calculates the
has the Obvious?
based on IR weight of RCs
greatest weight
RC Calculates the
RC
RC Identif ication Question that
weight of
based on has the 80% with the
Questions and
categories greatest weight same answer
Categories
Calculates the Threshold
weight of
CI
L
CI
Identif ication
based on RCs possible
Question with
higher level
Answers
Input Processor Weight Calculator Question Selector Question Verifier
Root Cause Analyzer
Malia Project – HP Labs & Institute of Informatics – UFRGS
32
Wednesday, March 06, 2013

30. Proposed Solution
Root Cause Analyzer
Category that
Identif ication Calculates the
has the Obvious?
based on IR weight of RCs
greatest weight
RC Calculates the
RC
RC Identif ication
weight of
Question that
based on has the 80% with the
Questions and
categories greatest weight same answer
Categories
Calculates the Threshold
weight of
CI
L
CI
Identif ication
based on RCs possible
Question with
higher level
Answers
Input Processor Weight Calculator Question Selector Question Verifier
Root Cause Analyzer
Malia Project – HP Labs & Institute of Informatics – UFRGS
33
Wednesday, March 06, 2013

31. Results
Question Question Question Question Question Question
Exim is installed? POP3 is running? Switch is up? Server is Switch is Apache is
reachable by reachable by installed?
ping? ping?
Executions: 25 Executions: 22 Executions: 48 Executions: 41 Executions: 36 Executions: 16
Answers: 88% YES Answers: 63,6% NO Answers: 75% NO Answers: 73,2% NO Answers: 77,8% NO Answers: 93,7% YES
Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: NO Operator’s Answer: YES Operator’s Answer: YES
Question Question Question
Path of Apache Network adapter Switch ports are
cfg is Ok? is configured? configured?
Executions: 15 Executions: 18 Executions: 28
Answers: 66,6% NO Answers: 50% YES Answers: 71,4% YES
Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: NO
Question Question Question Question Question Question
Exim is installed? POP3 is running? Apache is Path of Apache Exim is running? Path of Exim cfg is
installed? cfg is Ok? Ok?
Executions: 25 Executions: 22 Executions: 16 Executions: 15 Executions: 15 Executions: 8
Answers: 88% YES Answers: 63,6% NO Answers: 93,7% YES Answers: 66,6% NO Answers: 86,6% YES Answers: 100% YES
Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: YES Operator’s Answer: NO
Malia Project – HP Labs & Institute of Informatics – UFRGS
34
Wednesday, March 06, 2013

32. Conclusion and Future Work
• The proposed solution proved to be feasible with
the following features
• Interactive
• Considering the affected IT infrastructure
• Based on information of the previous diagnoses
• Reuse the knowledge acquired
• Optimize the root cause identification process of
failures in IT changes
Malia Project – HP Labs & Institute of Informatics – UFRGS
35
Wednesday, March 06, 2013

33. Conclusion and Future Work
• Melhorar o bootstrpping do sistema
• Improve the selection algorithms
• Consider other metrics (age)
• Consider other selection methods (popularity)
Malia Project – HP Labs & Institute of Informatics – UFRGS
36
Wednesday, March 06, 2013

34. Thanks for your attention!
Questions?

35. Proposed Solution
Root Cause Analyzer
Category that
Identif ication Calculates the
has the Obvious?
based on IR weight of RCs
greatest weight
RC Calculates the
RC
RC Identif ication
weight of
Question that
based on has the 80% with the
Questions and
categories greatest weight same answer
Categories
Calculates the Threshold
weight of
CI
L
CI
Identif ication
based on RCs possible
Question with
higher level
Answers
Input Processor Weight Calculator Question Selector Question Verifier
Root Cause Analyzer
Malia Project – HP Labs & Institute of Informatics – UFRGS
38
Wednesday, March 06, 2013

36. Example Selection of Questions
Current Version
Software
System
Device
74 95 35
Computer System
Network Card
Web Server
30 70 30
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
39
6 March, 2013

37. Example Selection of Questions
Current Version
Application
System
Device
74 95 35
Computer System
Network Card
Web Server
30 70 30
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
40
6 March, 2013

38. Example Selection of Questions
Current Version
Software
System
Device
74 95 35
Computer System
Network Card
Web Server
30 70 30
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
41
6 March, 2013

39. Example Selection of Questions
Current Version
Software
Questions RC 1 (10)
System
Device
74 95 35
Questions RC 2 (20)
Computer System
Questions RC 3 (40)
Network Card
Web Server Questions RC 4 (25)
30 70 30
Qestions RC 1, 3 (50)
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
42
6 March, 2013

40. Example Selection of Questions
Current Version
Software
Questions RC 1 (10)
System
Device
74 95 35
Questions RC 2 (20)
Computer System
Questions RC 3 (40)
Network Card
Web Server Questions RC 4 (25)
30 70 30
Common Questions
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
43
6 March, 2013

41. Example Selection of Questions
Current Version
16 bits How many 32 bits
bits is the
application?
64 bits
Malia Project – HP Labs & Institute of Informatics – UFRGS
44
6 March, 2013

42. Example Selection of Questions
Current Version
Software
Questions RC 1 (10)
System
Device
74 95 35
Questions RC 2 (20)
Computer System
Questions RC 3 (40)
Network Card
Web Server Questions RC 4 (25)
30 70 30
Common Questions
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
45
6 March, 2013

43. Example Selection of Questions
Current Version
Software
Questions RC 1 (10)
System
Device
74 95 35
Questions RC 2 (20)
Computer System
Questions RC 3 (40)
Network Card
Web Server Questions RC 4 (25)
30 70 30
Common Questions
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
46
6 March, 2013

44. Example Selection of Questions
Current Version
Software
Questions RC 1 (10)
System
Device
74 95 35
Questions RC 2 (20)
Computer System
Questions RC 3 (40)
Network Card
Web Server Questions RC 4 (25)
30 95 – 70 = 25 70 30
Common Questions
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
47
6 March, 2013

45. Example Selection of Questions
Current Version
Software
System
Device
74 25 35
Computer System
Network Card
Web Server
30 70 30
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
48
6 March, 2013

46. Example Selection of Questions
Current Version
Software
System
Device
74 25 35
Computer System
Network Card
Web Server
30 70 30
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
49
6 March, 2013

47. Example Selection of Questions
Current Version
Software
System
Device
74 25 35
Computer System
Network Card
Web Server
30 70 30
Server
26
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
50
6 March, 2013

48. Example Selection of Questions
Current Version
Software
Questions RC 5 (15)
System
Device
74 Questions RC 6 (5) 25 35
Questions RC 7 (12)
Computer System
Network Card
Web Server
Questions RC 8 (12)
30
30 Questions RC 9 (2)70
Questions RC 10 (3)
Server
Questions RC 11 (25)
26
Common Questions
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
51
6 March, 2013

49. Example Selection of Questions
Current Version
Software
Questions RC 5 (15)
System
Device
74 Questions RC 6 (5) 25 35
Questions RC 7 (12)
Computer System
Network Card
Web Server
Questions RC 8 (12)
30
30 Questions RC 9 (2)70
Questions RC 10 (3)
Server
Questions RC 11 (25)
26
Common Questions
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
52
6 March, 2013

50. Example Selection of Questions
Current Version
16 bits How many 32 bits
bits is the
application?
64 bits
The devices
No Yes
are well
plugged?
Malia Project – HP Labs & Institute of Informatics – UFRGS
53
6 March, 2013

51. Example Selection of Questions
Current Version
Software
Questions RC 5 (15)
System
Device
74 Questions RC 6 (5) 25 35
Questions RC 7 (12)
Computer System
Network Card
Web Server
Questions RC 8 (12)
30
30 Questions RC 9 (2)70
Questions RC 10 (3)
Server
Questions RC 11 (25)
26
Common Questions
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
54
6 March, 2013

52. Example Selection of Questions
Current Version
Software
Questions RC 5 (15)
System
Device
74 Questions RC 6 (5) 25 35
Questions RC 7 (12)
Computer System
Network Card
Web Server
Questions RC 8 (12)
30
30 Questions RC 9 (2)70
Questions RC 10 (3)
Server
Questions RC 11 (25)
26
Common Questions
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
55
6 March, 2013

53. Example Selection of Questions
Current Version
16 bits How many 32 bits
bits is the
application?
64 bits
The devices
No Yes
are well
plugged?
No Network Yes
card
detected?
Malia Project – HP Labs & Institute of Informatics – UFRGS
56
6 March, 2013

54. Example Selection of Questions
Current Version
Software
Questions RC 5 (15)
System
Device
74 Questions RC 6 (5) 25 35
Questions RC 7 (12)
Computer System
Network Card
Web Server
Questions RC 8 (12)
30
30 Questions RC 9 (2)70
Questions RC 10 (3)
Server
Questions RC 11 (25)
26
Common Questions
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
57
6 March, 2013

55. Example Selection of Questions
Current Version
Software
Questions RC 5 (15)
System
Device
74 Questions RC 6 (5) 25 35
Questions RC 7 (12)
Computer System
Network Card
Web Server
Questions RC 8 (12)
30
30 Questions RC 9 (2)70
Questions RC 10 (3)
Server
Questions RC 11 (25)
26
Common Questions
Third Party Services
Web Server
5
20
Malia Project – HP Labs & Institute of Informatics – UFRGS
58
6 March, 2013

56. Example Selection of Questions
Current Version
16 bits How many 32 bits
bits is the
application?
64 bits
The devices
No Yes
are well
plugged?
No Network Yes
card
detected?
Network
No card is
configured
correctly?
Network card Yes
not
configured
Malia Project – HP Labs & Institute of Informatics – UFRGS
59
6 March, 2013