At one time or another, every tester hears the dreaded question, “Why didn’t you guys catch these bugs?” We all have some standard responses, and they are most likely true). But what can we learn about our testing when we look beyond the easy answers? Pamela Gillaspie proposes that the key to improving your testing is determining the areas where bugs are slipping past your defenses. For her team, the practice is a lot like basketball. If you group the bugs into zones, you can devise a strategy to cover those zones more effectively. Some zones need a different testing approach than you’ve used; others might reveal a need for closer communication. Join Pamela as she shares her experience as defensive coordinator, addressing the developers’ playbook (What kinds of recurring problems do we see?), trick plays (The user is doing what?), and penalties (That wasn’t in the requirements!).

1. Testing with a Zone Defense
Pamela Gillaspie

2. People Who ❤ Bugs
Some people love bugs– like the happy testers in this office.

3. Person Who Does Not ❤ Bugs
Others, for whatever reason, don’t seem to share our enthusiasm for bugs. We’ll call them customers.
What I want to talk is how we, the people who appreciate a good bug, can use feedback from guys like this to figure out how we can beat him to the bugs next time.

4. CRUD
First, I’m going to take a moment to introduce a term, CRUD, because it’s probably familiar to a lot of you as something different than the way I’m using it. You might know it as Create Read Update Delete.

5. Customer-
Defects
Unique
Reported
As a quality metric, it stands for Customer-Reported Unique Defects. This basically means bugs reported by customers, and CRUD is an appropriate acronym for it. This is a meaningful metric to me– it’s a quick scorecard of how many (and what kinds of) bugs are
getting out the door.

6. Zone Defense!
So here’s the gist of the Zone Defense:
• We can’t test everything, so we need to be strategic about what we choose to test.
• To me, CRUDs are a good indication of what we can improve over our last release.
• What we do on my team is like a zone defense in football.
• We examine the CRUDs, find recurring themes (our zones) and then cover those as a group.
• I’m going to tell you about how we do this on my team, and give you an idea of some of the zones we’ve identified and what we’ve done about them.

7. Why are there so many bugs?
This isn’t something we’ve always done, at least not in a deliberate way. There’s a story behind it; and like many good stories, this one starts with something horrible.
One day my manager called me up to talk about our “release quality problem”. He wanted to know why there were so many bugs in the release.
I hadn’t thought that we had a release quality problem, and yet it wasn’t hard for me to think of reasons why we had one.

8. Why do we miss bugs?
What do you think? Why do bugs make it out into a release?
[Discuss].

9. Missed bugs
When I answered this question myself, here are the kind of things that came to mind:

10. Late
development
Missed bugs
Developers are still checking in code the day before (or even the day of the release). No one’s testing that code– possibly not even the developer– and every check-in that we don’t test is a roll of the dice.

11. Late
development
Missed bugs
Distractions
How many of you are testing more than one product at a time? How many of you help with support? How many of you get pulled into development or documentation? We are useful people! These things can’t always be helped, but they’re disruptive. If the
development calls for, say, 4 testers, and those testers can’t devote all of their time to the project, that’s going to be a problem.

12. Late
development
Bad
requirements
Distractions
Missed bugs
Sometimes you get vague requirements, such as “Improve the shopping cart UI”, and you don’t really know how to test those. Testers might even avoid testing against vague requirements at all.

13. Late
development
Bad
requirements
Not enough
staff
Distractions
Missed bugs
Sometimes there is just more work than people. Some aggravating factors:
• Business is not realistic about the testing that is needed.
• Uneven growth– development growing beyond QA.
• Especially on a small team, when people take time off, the deficit can be hard to work around, especially when it’s unexpected.

14. Late
development
Bad
requirements
Not enough
staff
Distractions
Missed bugs
These are the kinds of reasons I went through, and though they are often true for many of us, they’re not very satisfying, because none of them spoke to our responsibility.
I’m not so cynical as to say that none of these things ever change, but at the same time, my team needed to come up with answers that were under our control. So what did we do? We got very specific.

15. Someone Found These bugs: the Customers
Interception
Why did we miss bugs? Let’s look at the bugs. When we see what those bugs are, we’ll have a good shot at knowing how they got through.
This is where the football analogy comes in… We’ve got the development team like the quarterback firing bugs down the field, and users out in the field waving their arms. We need to get out there and intercept those bugs– which isn’t as hard as it sounds, because
we have their playbook. It’s in our bug database!

16. Man-to-Man Coverage
A quick note on what we didn’t do… By going through the bug list, it wasn’t really about addressing the specific bugs we had in front of us (what I call man-to-man coverage.) It’s a good thing to test for bug fixes– and some bugs do have a habit of recurring– but what
we were really focused on was finding ways to predict whole classes of bugs.

17. Traits of a bug
So… we started going through our CRUDs one-by-one and asking ourselves, what do we know about this bug? What are the contributing factors? If you’re examining a bug, what kinds of highlevel categorizing would you do?

18. Platform
Traits of a bug
Does the bug only occur on a specific platform, OS version, or hardware?

19. Platform
Traits of a bug
Frequency
Does this bug present all the time, or sporadically?

20. Platform
Application
layer
Frequency
Traits of a bug
Where is the problem:
• UI– controls not appearing on the screen?
• Business logic– bad calculations?
• Lower level problems such as memory leaks?

21. Platform
Happy/sad
Application
layer
Traits of a bug
Frequency
Is it a happy or sad path bug? Was the user doing something you expected? Something unexpected, but in theory, valid? Something dumb? Something dumb that you could have foreseen?

22. Platform
Happy/sad
Application
layer
Frequency
Aggravating
circumstances
Traits of a bug
Was the user on a poor-quality network or a crappy machine? Was the user in a restricted-security environment?
Were there third party products involved?

23. Case Study
Here’s the whiteboard where we started breaking things down in our conference room. We wrote areas of the product across the top of the table, and then we started filling in descriptors down the left.
As a case study, I’m going to tell you about some of the zones we identified through this. Some of them called for more or different testing than we’d been doing, and some of them revealed weaknesses in our larger development process.

24. Hail Marys
One of the zones we identified was “Hail Marys”, the long plays.
Examining the CRUDs, we found that we had customers who were using our product in a much larger way than we do. Some of the CRUDS we found were:
• Scripts that failed to colorize with 52,000 lines of code.
• Slow searching behavior in suites with thousands of resources.
• A crash after a test had been running for 6 days.
How did we miss these bugs? As users, we don’t run like this. We modularize the heck out of stuff, so we don’t have large scripts. We store data outside our suites. We also break things down into lots of suites. Our tests were designed to run at, for most, hours at a
time.
Solution: This was actually the easiest problem we solved. What we did was take our nightly tests and copy them into huge, monstrous suites that we repeat for days at a time. This has helped us identify some memory leaks and other performance issues.

25. Penalties
This zone was a little trickier to solve from a technical standpoint: the super-sad path bugs. It’s not just that the user did something “wrong”. It’s that the user did something wrong an hour ago, kept on going, and then encountered a seemingly “random” problem later.
An example of this is a user installing an application as one user, and then checking out his tests from a different user account, and the checkout is not resolving. Or the user saw an unknown process in task manager and didn’t know what it was, so he killed it.
In this case our solution was more targeted breakage. Instead of thinking about things the user might do, we tried to think of what the user could possibly do that would cause a problem. When I worked for a smaller company, I wasn’t able to spend time on issues like
this, but now in my current position, these kinds of issues make up a significant portion of our CRUDs. Part of that is product maturity, and part of that is that our enterprise customers are running in more and more complex environments.

26. Missed Tackles
• Baring my soul… Emotionally, the toughest zone was the Missed Tackle zone.
• We coulda/shoulda/woulda found the bugs, and there is no good reason why we didn’t.
• We assumed that if 3 of us were banging on a new feature, we would find all of its issues, but it turned out that we were all banging on the same thing.
• To make matters worse, the first of the testers learned from me, the subsequent testers learned from the from us– so we probably all have a similar style.
• What we needed was a little more guidance in our exploratory testing and our automation.
Of all of the zones, the toughest one to look in the eye was the category we called “Simple”. These were bugs that we felt like we just should have caught, but didn’t.
We realized that sometimes we would all be hammering on a new feature, but we would be hammering in the same places. This was probably made worse by the fact that every new tester is taught by the existing testers– so if we have blind spots, we probably have
the same blind spots.
What we needed, in this case, was just more guidance, both in our exploratory testing and in defining our automated tests, so we started using mind maps.

27. Here’s an example of a mind map we made for a new feature going into one of our products, a variable watcher. It shows all of the facets of the variable watcher that we need to consider, and we wrote this in about 10 minutes.

28. Audibles
So far, everything we’ve talked about has been strictly testing. This one is about the larger development process.
Looking through the CRUDs, we found that there were some bugs in functionality we didn’t know existed.
In a couple of the cases, there was an issue opened to track a feature, with a reasonable description, followed by many comments. Somewhere buried in there was a “while I’m in there, I’ll make it do this”.
I’ve also been around long enough to have seen the odd “stealth check-in” that’s like a congressional bill. The developer is checking in something reasonable and valid, and then they tack on their pet code at the end.
To deal with these issues, we had to talk to the development team, and they got it. If yours don’t, make sure they know it’s not an indictment of their code; it’s about you having the information you need to do your job.

29. Back to the Drawing Board…
The first time we held one of these sessions, we wrote up a table on a whiteboard in the conference room, where it actually garnered a lot of attention. Our developers were quite interested in these observations– and we had some good conversations about them.
It’s worth noting, however, that originally it was just the test team writing this, so it was all coming from a strictly testing perspective. Everybody who walked into the conference room wanted to know what wasn’t whose fault, and whose fault was it. (In subsequent
iterations, I’ve labeled that row “Communication”.)

30. Trick Plays
We have some unique and fascinating customers, who sometimes do things we just couldn’t have predicted. Just as you might have a will-not-fix in your bug database, these issues would get a will-not-test status. For example, we had a customer having keyboard
problems running a Chinese VM using a German keyboard on and American computer.
Keeping track of these issues has a couple of benefits: First of all, it gives us the peace of mind that, yes, we have 15 outstanding issues, but 5 of them are things we were never going to test for. On the other hand, if we started seeing multiple reports of some of
these issues, we would adapt and focus more testing there.

31. Defensive Coordination
We’ve talked about some of the kinds of recurring issues we see in testing, and we’ve talked about ways to address those issues.
I want to talk about the possibility that you have issues that you just don’t have the capacity to address. The problem might be staff, tools, hardware… and you feel pretty certain that you’re not going to get it.
My advice is to go through this process anyway. It helps to be able to go to your manager and say, “These are issues that our customers are having, here’s what we’d like to do about it, but it’s not realistic with a team our size.” Or “See, we’re going to need some
automated testing if we want to keep up with the variety of devices we need to test.”
I have gotten an extra hire by having this conversation– but even if that’s not possible, you’ve laid the groundwork for when resources become available. In the worst-case scenario, you’ve done some expectation setting about what you can and can’t test.

32. Check the Scoreboard!
Before we wrap up, I want to circle back to the beginning and talk about the perception of quality… The time I was talking about, when I was asked why the last release was so buggy, it turned out that the product in question had nothing like a quality problem. Two
weeks into the release, we’d had one CRUD– but it happened to be a CRUD reported by a very vocal customer. So… someone at my company had heard that our release was “unusably buggy”, and he took that at face value– and when he asked me why the release
was buggy, I also took that at face value, and assumed that we’d dropped the ball. We hadn’t. But now, I really know how we’re doing– and if I don’t, I know to find out before I roll with an assumption.

33. Post-Game
• Use your CRUDs to see what you’re missing.
• Share your data to improve development.
• Find your zones– test to the zones!
• Acknowledge your scope.
• Use your CRUDs to show the needs of your
team.
Today we talked about actions that you, as a tester, can take to improve your release quality:
1. Use your CRUDs to find out which bugs you’re missing.
2. Find patterns– test to the patterns!
3. Acknowledge your scope. Be open to discussing scope.
4. Use your CRUD analysis to demonstrate the needs of your team.
5. Don’t rely on the myth of quality that builds up around a release.

34. Testing with a Zone Defense
Pamela Gillaspie