Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (16)
Similaire à Maybe We Don’t Have to Test It
Similaire à Maybe We Don’t Have to Test It (20)
Plus de TechWell
Plus de TechWell (20)
Dernier
Dernier (20)
Maybe We Don’t Have to Test It
- 1. (Maybe We Don’t Have to) Test It
- 3. The Problem
- 7. 1 1 0 01 01 1 0 0 0 1 0 1 01 1 01 1 1 01 0 0 0 1 0 0 1 101 0 0 10 1 01 0 0 1 0 1 0 Tester
- 8. Video of arrogant tester.
- 10. “If it exists, I want to test it” James Bach
- 11. “If it exists, I want to test it. (The only exception is if I have something more important to do.)” James Bach
- 14. 0 1 0 101 0 0 0 1 0 1 01 1 01 1 1 1 01 0 0 0 1 0 0 0 0 01 1 1 0 1 0 0 1 0 01 0 1 01 0 0 0 1 01 1 1 0 1 0 1 0 1 01 1 01 1 1 01 0 0 0 1 0 0 0 1 0 11
- 15. Patterns
- 16. No repro steps.
- 18. How can I repro this? Beats me.
- 19. I’ll play around in QA. Have fun.
- 20. Um, no luck yet. You’re wasting time.
- 21. Can’t you debug it or something? No.
- 22. How will we know if it’s fixed before we release to prod? We won’t.
- 27. Things that don’t go to production.
- 33. Patches for critical production problems that can’t get worse.
- 34. Bug 102927: Users unable to rework today’s schedule with empty alternate.
- 36. Cosmetic changes with timely test setup.
- 38. Would you bother to test this? 40 30 20 10 0 Yes No
- 43. Maximum number of items that can be serialized or deserialized in an object graph is '65536'
- 44. Trust but verify
- 45. Too technical for a non-programmer to test.
- 51. Inadequate test data or hardware.
- 55. Bugs not logged by you.
- 56. Techniques
- 58. Are you crazy?
- 59. Only a fool would bother testing this.
- 60. I’ll just mark it “Tested” without doing much.
- 61. An endorsement without careful thought or personal investment in the outcome.
- 62. 1. Some degree of risk analysis took place. 2.The impact to production would be low.
- 64. 3 Questions to Start Asking: 1. Does this need to ever be tested? 2. Does it need to be tested by me? 3. Who cares if it doesn’t work?
- 65. “NOT testing” may still leave room for testing by a broader definition
- 66. We may be… • Regression testing/checking. • Questioning our team. • Learning product architecture. • Evaluating others’ tests.
- 67. Instead of saying “NOT testing”, try saying “testing something more important”.
- 68. 1. Spend more time on valuable testing. 2. Look for “not-test” opportunities. 3. Use my 9 patterns and find your own. 4. Introduce a code word.
- 69. Video of revised arrogant tester.
Notes de l'éditeur
- I would like to start by telling you a story…(each of the 8 photographs will have a number that appears on it)My wife and I had our first baby last year, so we’ve been getting frequent house guests to visit with Josie. Many of these guest are from out of town so they spend the night. There can be a lot of preparation that goes into ensuring your house guests enjoy their visit. You want the house to be clean, you want fresh sheets on the guest bed, clean towels out, etc. I’d like to tell you a story about preparing for my parents’ visit to our house. About two days before my parents were scheduled to arrive, my wife and I began preparing the house…#1 – No Repro StepsHouse cleaning metaphor. The last time they visited, my mom found a bug. It was crawling across the bedroom floor. My wife said to put spread some borax around but I wanted to be positive it would solve the problem. I became obsessed with trying to recreate this situation to see the bug crawl across the floor. I just never saw it. Very frustrating. What if the borax doesn’t work?…it’s a time waster and #2 Things that don’t go to production: House cleaning metaphor. Cleaned master bedroom but the guests never see it. #3 – Patches for critical prod problems that can’t get worse:House cleaning metaphor. Guest toilet won’t flush. I am interested in making sure the plunger is the right solution before just giving it to themGuest toilet won’t flush. Guest needs help. Wife is like programmer. Wife says to just give them the plunger. I want to investigate. What if the plunger doesn’t work. We also have a drain snake. Or maybe they just aren’t holding down the handle long enough.#4 - Cosmetic changes with log setup:House cleaning metaphor. Wife put magazines on the coffee table, but I wanted to check myself to make sure she didn’t mess it up. I'm not sure how she could have messed it up but… then I wasted a bunch of time thinking about which magazines should be on display.#5 – straight forward config changes:House cleaning metaphor. She put extra chairs around the table but I insisted upon experiencing the problem with only two chairs, then comparing that experience to having 4 chairs. Meanwhile, I was using time I could have used to clean the dining room.#6 –Too technical - House cleaning metaphor. Mopping. I needed help every step of the way. I used all her time. I’ll probably never mop again.#7 – Non-tester on loan:House cleaning metaphor. My neighbor is meticulous about his yard. He offered to blow away my leaves. I declined. I’m the homeowner. It’s my job. What if he misses part of my lawn#8 – inadequate data/hardware:House cleaning metaphor. My mom is allergic to cats. Melissa said to keep the cats in the basement during the visit. I wanted to make sure it would solve the problem so I put the cats in the basement before they arrived and tried to see if it made a difference. M pointed out that my efforts are useless because I’m not allergic to cats. We would have to wait until Mom gets here to see if it works.Backup idea: House cleaning metaphor. I know the guests won’t want to play board games. But I’m going to get some out and play them to freshen up on the rules. This is really not helping us prepare at all…still, I like that it makes me look like I’m carefully preparing for their visit.
- We’re gonna tie that story into my topic in a few minutes. But first, I want to tell you about a problem I’ve noticed. Maybe you’ve noticed it as well.
- We all kind of have a full plate with this job. There is always more to do.http://www.flickr.com/photos/peasap/3016306391/
- Some of us feel outnumbered. Like there are too many programmers.http://www.flickr.com/photos/kenneth-/7421336698/
- But then we make it worse…many testers have a big chip on their shoulder. They have an attitude like this…“I am the tester. Therefore, everything must be tested...by me...even if a non-tester already tested it...even if I already know it will pass...even if a programmer needs to tell me how to test it...I must test it, no exceptions!”How many of you think like that? ….(put your hands down) My goal for this talk is to convince you not to think like that.
- I think most testers picture a model like this in their head. Every bit their programmers generate, must pass through their expert tester brain.
- How many of you have ever executed a bunch of tests that you knew were going to pass? But you did it anyway. It took an hour. And you were right. They passed.How many of you have ever predicted a bunch of tests would fail, and you ended up being right?That is what I’m talking about. Maybe we are doing too much routine testing.Who taught us this? Picture of manager, process hawk, other testers
- Basic heuristic…as seen on a testing blog. I disagree with this. Until I looked up the full context on James’s blog.
- Add picture of Bach7/8/2006 blog post“If it exists, I want to test it. (The only exception is if I have something more important to do.)”Quick show of hands. How many of you have more to test than you have time for? So determining where to spend your time is important.
- This is what I’m not going to show you.
- IT consulting companyI’m not going to show you show to prioritize your tests. You already know that. Instead, I’m suggesting you may want to actually decide not to test certain things early enough not to spend any time on them…these tests don’t even get on your list of possible tests.
- An easier approach, for me, has been to look for patterns at a high level and admit to myself and the team, that it is not the best use of my time to test those things.
- As we talk about these 9 patterns, the assumption will be, that Our team has developed cadence.we have something more important to test.if something should go wrong with the item we are not testing, its impact to production will be low.We are executing our standard critical regression tests before deploying anything to prod.
- Bug106621DimMedia returns error intermittently, “Warning: Null value is eliminated by an aggregate or other SET operation” Nobody knows the repro steps. Nothing bad appears to happen that affects the users. We just don’t want to waste time evaluating the error in the future. So the programmer determines “some people on the internet” ameliorate it by adding a coalesce to a select statement.A non-inquisitive tester pulls this bug report from their list of “Ready To Test” bugs reports, and gets busy trying to repro the error. Time wasters: “How can I repro this?” “How will we know if it’s fixed before we release it to prod?”
- Time wasters!
- Time wasters!
- Time wasters!
- Time wasters!
- Time wasters!
- …Tester Nightmareshttp://www.flickr.com/photos/mr_salvis_are/3330230315/
- There are big risks
- And there are little risks
- Sure, this is a risk. But so is all that other important testing you should get back to.
- Feature106218
- The SM data contains information on actual service calls made in the call hierarchy. This data can be used to construct a visualization of dependencies in the form of a graph
- Here’s another example (this was not in Kanban screen capture)
- If there’s nothing else on your plate, you may want to test it.
- 10292797258Need something else interesting for this section
- Programmer sees existing code that expects shows on the alternate. Changes code to not expect shows on the alternate. Needs it fixed within the hour. It would take more than an hour to recreate the scenario in a test environment. Users cannot do anything else with product until this is fixed.
- 103789 - untested scenario occurs in prod. Kills everything. (The fix was more than a config change.) In order to test the fix in downstream environment, one would have to spend significant time recreating prod scenario or copying prod data to a QA environment.
- It takes about 30 minutes to trigger this message. The prog found the misspelling and corrected it. This misspelling has been in production for 3 years.
- When we originally added this feature, we tested with SCTE events = True and with SCTE event = false. We trust those settings to some degree. We can open the changeset and compare with the previous version to make sure it made it into the QA build. To actually test the change, we would have to spend about 2 hours setting up the example.
- This is a WPF default. Our fix was to increase the MaxItemsInObjectGraph setting to 2 million, the max. The programmer was very confident. It would take the tester a day to create a job big enough to get that error. No feasible way to put prod data in QA without disrupting current testers. Test it in prod.
- 96659
- Doing fake testing and marking it Verified is dangerous. Exclaiming that you are not going to test it because your test would only provide misleading data, set up the team for vigilance.
- Orphaned comment92493
- I used to set up prod data in QA to test a data update. Now I read the SQL.
- If they were able to log the bug report, and it has good repro steps. Chances are:They know how to tell if it’s fixed.They WANT to test it’s fix.Warning: Non-testers probably won’t regression test the fixed area.
- Okay, it’s all fine and dandy to not test things but my process forces QA to stamp the word “Verified” on everything to get it off the Kanban board.Declare it!Come up with a code word.Satisfy an Audit.
- First, tell team you are not going to test this because…
- At which point you should say, okay, I’ll test it.Or the team would say…
- Don’t do this!
- You may want a term to describe this testing maneuver. So you can communicate efficiently with your team. On my project teams we use “Rubber Stamp”.Wikipedia’s political definition is…Which fits fairly closely. Used in context we’ll say, “I’m going to rubber stamp it”We didn’t test it, we just rubber stamped it.The danger in not declaring said test maneuver, is someone on the team may think it was tested.
- Picture of sox.One of my project teams is a SOX Compliant application. Our auditors are okay with NOT testing as long as we provide an artifact that shows two things:some degree of risk analysis took place.the impact to production would be low.We do this by either:attaching an email with a discussion thread documenting the decision (a natural artifact. It already exists)leaving a comment on the work item.
- This comment was added to a work item.
- …
- But seriously…when I say, “let’s not test it”. I refer to testing in its narrow, traditional use; exercising the application to evaluate a specific function. So even though, by Rubber Stamping something, we may not be testing by that definition, we may be testing by a broader definition…
- “NOT testing” is a provocative way of starting a discussion. But there may be a more effective way of describing it to your team.(Need picture of more important vs. less important) But even more importantly, when we’re not testing something…we’re testing something else.
- (Need picture of happy tester) When you get back to work, and you schedule that meeting you promised, the one where you’ll discuss all the ideas you got at STARCanada. Consider the following:Tell your team you would like to spend more time testing things you are good at testing.To get this extra time, you will experiment with NOT testing certain work items yourself. Delegate testing to other willing victims on your team. Share my 8 patterns and swap them out with others that better fit your team.Suggest a code word like “Rubber Stamping” and define it for the team.