1. BUG BEST PRACTICE
LIANG GAO (LIANGG@GMAIL.COM)

2. AGENDA
Bug filing best practice
You should learn….
Value added for bug management
Defect management
Bug follow up best practice

3. BUG IS IMPORTANT
Your performance is partially judged by the quality and
amount of the bug you filed.
As an engineer, reputation is the most important thing. A
tester’s reputation is as good as the quality of your bug can
get.
Most of the time, it is the only way you communicate with the
developer.
Who will read your bug report? – developer, your managers,
developer’s manager, product manager, marketing folks,
customers……

4. WHERE ARE BUGS
COME FROM?
Laziness (does not perform error handling)
Inexperience (does not know the impact of code changes)
Carelessness (blind cut-n-paste)
Bad design/infrastructure (bad module design  un-
maintainable code)
Incomplete documentation (misunderstood customer
requirements, missing information in func spec)
Complexity of software (hard to maintain/enhance/test)
Bad tools (compiler, CM tool, merge tool, etc)
Lack unit-level testing (misses lots of basic bugs)
Time pressure (fatigue)  Laziness and carelessness
 Code scales linearly, the complexity scales exponentially

5. THE QUALITY OF A
TESTERScript Quality
Test Case Quality
Bug Quality
Case Execution
Quality

6. HOW TO FIND A BUG?
You think it is a bug because?
• Understand the product/function spec correctly
• Understand the use case scenarios correctly
• Put yourself in a user’s perspective.
• Your experience counts a lot, 上 兵纸 谈 is harmful
Bad bugs cause you trouble
• Junk bug is counter productivity, and ruin your
reputation
• Bad documented bugs cause you a lot of time and
energy, like…. living in a nightmare.

7. HOW TO FIND A BUG?
Debugging is like watching Magic Tricks
(Michael Amma17:40 - 19:00)
James Brown 5:10

8. WHY I CAN FIND HIGH QUALITY BUGS BUT
YOU CAN’T?
High quality bug
• Affect the major functionality
QoS still work after delete the configuration
• Not Cosmetic
Bug 3744 - WebUI 的导航栏没有滚动条 , 不便于用户浏览操
作
Because I understand the functionality deeper than you
Because I use more testing types than you.
Because I design more use case than you
Because I carefully execute the cases than you
Because I pay more attention to the details than you.
Because once I found a crack, I’ll keep on going, but you didn’t

9. NONREPRODUCIBLE BUGS –
HEADACHE?
Random is not equal to irreproducible

10. NONREPRODUCIBLE BUGS –
HEADACHE?
Always report non-reproducible bugs, they might be time
bombs.
Non-reproducible bugs are always reproducible.
• Timing and delay?
• Only happen during installation
• Depend on some data like corrupted database?
• Sequence of the execution?
• Interaction with other programs in the background?

11. NONREPRODUCIBLE BUGS –
ELABORATE
a. I saw this one, but was never able to make it happen again.
b. I see this when I run with this setup, but sometimes I don't.
c. This is the same anomaly I see under several setups, but not
all the time.
d. Under X setup, this happens sometimes and not other times.
There may be something different from one X to the other, but
I' not seeing it.
What is the impact if this happens all the time?....

12. NONREPRODUCIBLE BUGS –
DEBUGGING STORY
²»¿É¸´ÏÖBug
ʵս
ר¼Ò̸֮
ר¼Ò̸֮ - 2

13. AVOID CRITICAL BUG IN THE LATE
DEVELOPMENT
Execute test cases based on priorities
Discover critical bug in the early development stage
• It gets fixed early, less resource waste
• It gives developer more lead time – critical bugs are hard to fix

14. MOST IMPORTANT
THINGS ON A BUG
Subject, Subject, Subject
• One line
• Exactly pinpoint the problem.
• Accurately describe the problem
Bug 3747 - WebUI 界面上的字体、图标等在 Linux 系统下的显示不
正常
Bug 3774 - 客户端能够通过 Admin 端口与 DUT 建立 PPTP 连接 , 不符
合设计规范
Bug 3768 - admin route: 不能为同一个服务创建多条 admin 路由
Bug 3788 - 在配置 DUT 的时候， DUT 死机
Which one is better?

15. WHAT IS THE BUG
QUALITY
Reproducible
• Testers filed bug need to be reproducible in developer’s
environment.
• Customer found bugs sometime is hard to reproduce, but that
not tester’s concern.
• Test case is not the step to reproduce.
• Need to find the main cause by ruling our the unrelated
factors. Need to pinpoint the root cause (Strong Analytical
skills).
• 蛛 迹只是 象，需要 一 追踪，找到 的本丝马 现 进 步 问题 质。
ISIC crash the box, can you find out which packet, and why?

16. HOW TO WRITE A
REPRODUCIBLE BUG
REPORT
A short paragraph for detail description.
A topology file with detail annotation. (IP address, DUT type
etc)
Step to reproduce
• Step 1. execution step with detail output.
• Step 2… execution step with detail output
• Seep 3: where the problem is. What is the expected output,
what is the actual output.
Detail DUT configuration dump,
Anything else (document) you think can claim the
validation of your bug?
Review each others bug reports

17. AGENDA
Bug filing best practice
Bug follow up best practice
Defect management
Integration of bug tracking system with other engineering
tools

18. UNDERSTAND BUG
STATE FIRST!
N:
A:
I: Information needed: Take action ASAP
R: Need Tester’s verify. Take action ASAP
V:
Monitor your bug daily: usually you will get an email reminder
whenever a bug state is changed. Pay special attention to “I”
and “R”
Best practice: I and R state bug need to be updated within 24
hours.
Ask your team lead for help, monitoring bug every day is hard.

19. INTERACT WITH
DEVELOPERS -1
Understand the developer’s response.
It is like writing an email, need to be polite. Need to have start
paragraph and end paragraph.
It needs to be thorough and accurate (provide as many
useful info as possible).
A quick and good response help you gain the respect from
developer.
Ask your team lead for help is a good idea.

20. INTERACT WITH
DEVELOPERS -2
Meet the programmers who will read your report
The best approach may be to demonstrate your bugs to the
programmers.
When the programmer say it is fixed, make sure it isn’t still
broken
Verify bug fixed promptly
When fixes fail, talk with the programmer.
Bug report should be closed by testers..

21. AN EXAMPLE OF
QUERY
State-Changed-From-To: open->info
State-Changed-By: murphyw
State-Changed-When: Sun, 24 Jun 2007 23:14:08 -0700
State-Changed-Why:
1. ‘set policy id 1 attack "CS:ftp" action close’, does this CLI mean that
we should drop FTP traffic?
2. Please help to provide debug information on successful case.
3. Please turn on snoop and snoop detail during debugging.
Thanks!
How Would you response?

22. AGENDA
Bug filing best practice
Bug follow up best practice
Defect management
Integration of bug tracking system with other engineering
tools

23. DEFECT
MANAGEMENT
How to know how many bugs filed per person, per day – bug search
is your best friend.
How to know how many bugs filed for certain type of testing
• Search for testing type – regression; performance;
functionality;…
• Add key word in front of the bug subject field then search the
keyword
REG: extended access list can not block traffic.
Pre-Test:-Pascal-when the packets can't pass through VPN
tunnel with deep inspect
• Be careful when add the interest field: those are the person
will be notified every time this bug state is changed.
• If there is a field called attribute, use it. It helps to categorize
and manage bugs we filed.

24. FILING BUG REPORTS
File every bug found (except duplicate bugs), annotate if a
bug:
• Blocker: High priority bug preventing further testing
• Regression: Bugs that introduce in new build
• Unreproducable: Bugs that cannot be reproduced
• Customer found: Bugs found by customers (Beta/FCS)
Pick right Priority (usage) & Severity (impact)
• P3/S1 : Login 200 times with 200 characters user name will
crash the device with exception core dump
• P1/S3 : GUI splash screen misspelled company name
Bug report is a statistics tool, use to predict readiness of
software and future releases

25. DEFECTS QUALITY
CONTROL PROCESS

26. BUG TREND
CONVERGENCE
Bug Filed
Release Date

27. BUG TREND
CONVERGENCE
Bug Fixed
Release Date
Bug Filed

28. BUG TREND
CONVERGENCE
发现 Bug
正常测试 分散测试
发布日期
Bug 修复
Bug 严重程度
日期
Bug 数目

29. BUG TREND
CONVERGENCE
发现 Bug
正常测试 分散测试
发布日期
Bug 修复
Bug 严重程度 日期
Bug 数目

30. BUG TREND
CONVERGENCE
发现 Bug
正常测试 分散测试
发布日期
Bug 修复 Bug 严重程度 日期
Bug 数目

31. USE BUG TREND TO
FIND QUALITY PROBLEM

32. USE BUG TREND TO
FIND QUALITY PROBLEM

33. BUG FIELDS FOR
METRICS
Status
Date Reported
Who Found
Assigned To
Program Area
Severity
Priority
Resolution

34. INTERESTING BUG
METRICS
Priority 1 Find Rate
− This is the find rate of bugs that at any point in time
become priority 1. It is a retrospective analysis.
Open Bug Aging Report − A histogram of how long bugs
have been open. This is most useful early in long projects.
Deferred Bug Aging Report − A histogram of how long
deferred bugs have been waiting on the list. It helps spot
chronically deferred bugs.
Promotions/Demotions/Deferrals Chart − Three lines that
help us see the triage process at work.

35. AVOID CONTROL METRICS;
EMBRACE INQUIRY METRICS
A control metric is any metric that drives decisions. − Any
metric you use to control a self-aware system will be used by
that system to control YOU.
An inquiry metric is any metric that helps you ask the right
questions at the right time. − An inquiry metric might look
like a control metric. The difference is how you use it and
what you infer from it.

36. EXAMPLE CONTROL
METRICS
“The developer who creates the fewest bugs will receive a
bonus.”
“Testers must average at least three bugs found per day.”
“The product may not be released unless we’ve gone at least
one week without finding a bug.”
“The product may not be released with more than 10 high
severity bugs.”

37. EXAMPLE INQUIRY
METRICS
Why do some developers have more bugs reported on their
code than others? How might it be good to have more bugs
reported? How might it be bad?”
“How do find rates differ among testers? What makes them
differ?”
“What does the bug find rate tell us about the readiness of
the product? Could the find rate be falling because testing is
slack, and not because the product is good?”
“What are the high severity bugs? Should we fix them?”

38. INQUIRY METRICS
INVITE LEARNING
Observe: You try to see what's happening.
Inquire: You conjecture about the meaning and significance
of the observations. You collect additional observations to
corroborate or refute our conjectures.
Model: You form and improve theories about why the system
behaves as it does, how you know that it behaves that way,
and what you can do about it.

39. QUESTIONS BUG METRICS MIGHT HELP
WITH TEST PROCESS
Productivity
• − Are the testers working at full speed?
• − Are they speeding up or slowing down?
• − Is anything blocking their work?
• − Are the testers burning out or staying sharp?
• − Are testers and developers getting along together?
Status
• − How close is the test effort to reaching a point of diminishing
returns?
• − What kinds of problems are being found?
• − Is the bug tracking system being used properly?
Focus
• − Are the testers focused on the areas of greatest risk or need?
• − Could the testers use some outside help?

40. QUESTIONS BUG METRICS MIGHT HELP
WITHQUALITY IMPROVEMENT
Productivity
• − Are the developers working at full speed?
• − Are they speeding up or slowing down?
• − Is anything blocking their work?
Status
• − Is the product improving?
• − How close is the product to being good enough?
• − What must happen in order to meet the schedule?
Focus
• − Is the work distributed effectively among the developers?
• − Do any areas of the product need more attention to improve
more quickly?
• − Is the triage process working well?

41. DAILY METRICS!
Consider producing a bug metric summary:
• − Daily
• − Automatic
• − Archived
• − Web Based
This becomes a resource for retrospective analysis.
Additionally, daily graphs can help you notice fast breaking
dynamics.
Include weekends in the graphs.

42. EXAMPLE: WHY WAS CONVERGENCE
HARDER DURING THE SECOND CYCLE?

43. CYCLE 1 CRITICAL BUGS

44. CYCLE 2 CRITICAL BUGS

45. DIFFERENT SLOPS

46. WHY SLOW CONVERGENCE?

47. OPEN BUGS BEFORE ENTER
CYCLE 2

48. LOW PRIORITY BUG WAIT FOR
THE FIX

49. AGENDA
Bug filing best practice
Bug follow up best practice
Defect management
Integration of bug tracking system with other engineering
tools

50. VALUE ADDED
DEFECT/CODE
INTEGRATION
Bug system can be integrated with CVS, Subversion etc code
control tool
When check in code, developer can specify the bug id, the
bug will be automatically updated with code diff.
When check in code, developer can specify the bug id, and
bug’s state will be automatically changed to “R”, and trigger
tester to verify it.
Other scenarios you can think of?
This is the area that we can provide the value added for our
customer

51. THINGS YOU SHOULD
REMEMBER AFTER THIS
Bug quality means it can be reproduced by developers in
their environment.
Most important thing on a bug is its subject
Junk bug is counter productive, and ruin your reputation.
I, R state bug need to be updated ASAP.
Use Sigma Bug Template to file bugs.

52. BUG REVIEW CHECK
LIST - 1
Is the summary short (about 50-70 characters) and descriptive?
Can you understand the report?
• As you read the description, do you understand what the reporter
did?
• Can you envision what the program did in response?
• Do you understand what the failure was?
Is it obvious where to start (what state to bring the program to)
to replicate the bug?
Is it obvious what files to use (if any)? Is it obvious what you
would type?
Is the replication sequence provided as a numbered set of steps,
which tell you exactly what to do and, when useful, what you
will see?

53. BUG REVIEW CHECK
LIST - 2
Does the report include unnecessary information, personal
opinions or anecdotes that seem out of place?
Is the tone of the report insulting? Are any words in the
report potentially insulting?
Does the report seem too long? Too short? Does it seem to
have a lot of unnecessary steps? (This is your first
impression—you might be mistaken. After all, you haven’t
replicated it yet. But does it LOOK like there’s a lot of excess
in the report?)
Does the report seem overly general (“Insert a file and you
will see” – what file? What kind of file? Is there an example,
like “Insert a file like blah.foo or blah2.fee”?)

54. BUG REVIEW CHECK
LIST – 3Can you replicate the bug?
Did you need additional information or steps?
Did you get lost or wonder whether you had done a step
correctly? Would additional feedback (like, “the program will
respond like this...”) have helped?
Did you have to guess about what to do next?
Did you have to change your configuration or environment in
any way that wasn’t specified in the report?
Did some steps appear unnecessary? Were they unnecessary?
Did the description accurately describe the failure?
Did the summary accurate describe the failure?
Does the description include non-factual information (such as
the tester’s guesses about the underlying fault) and if so, does
this information seem credible and useful or not?

55. BUG REVIEW CHECK
LIST – 4Does the description include non-factual information (such as
the tester’s guesses about the underlying fault) and if so, does
this information seem credible and useful or not?
Does the description include statements about why this bug
would be important to the customer or to someone else?

56. VERSION CONTROL
1st draft, Dec 7th, 2007, Liang Gao
2nd draft, Dec 27th, 2007, Liang Gao