TestWorksConf: Experience exploratory testing
•Télécharger en tant que PPTX, PDF•
0 j'aime•534 vues
Session by Maaike Brinkhoff & Maaret Pyhäjärvi at TestWorks Conference in Amsterdam.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (10)
Similaire à TestWorksConf: Experience exploratory testing
Similaire à TestWorksConf: Experience exploratory testing (20)
Plus de Maaret Pyhäjärvi
Plus de Maaret Pyhäjärvi (20)
Dernier
Dernier (20)
TestWorksConf: Experience exploratory testing
- 1. Experience Exploratory Testing 1 Maaike Brinkhof (@maaikees) Maaret Pyhäjärvi (@maaretp)
- 2. Exploratory testing is a systematic approach for discovering risks using rigorous analysis techniques coupled with testing heuristics.
- 3. “[Exploratory Testing is] simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next” James Bach
- 4. Realizations on Nature of Testing 4 20 16 1639 5±2
- 5. 20 5 DYNAMICALLY ADAPT FOR LIMITED BUDGET OPPORTUNITY COST
- 7. 1639 7 ROUTES ARE RELEVANT NOT ALL BUGS ARE EQUAL
- 9. 9 Testing as Performance (Exploring) vs. Testing as Artifact Creation
- 10. 10 What testing gives us UnitTesting ExploratoryTesting SPEC FEEDBACK REGRESSION GRANULARITY GUIDANCE UNDERSTANDING MODELS SERENDIPITYTesting as artifact creation Testing as performance
- 12. 12 We’re at a stage in software industry where you can find 76 problems including a showstopper in your first 4 hours of testing, ever. Imagine how the testing profession is changing when that is no longer true?
- 13. Learning in Layers 13 A Demonstration of Exploratory Testing
- 14. 14 Teaching Exploratory Testing in a Mob
- 15. Mobbing Setup and Roles Driver Navigators Designated Navigator Facilitator
- 16. Mob Testing – Rules for the Exercise No thinking as the driver Yes, and… Kindness, consideration and respect
- 18. 18 1st Charter: from nothing to something How would you learn something you’ve never seen before?
- 20. Bring structure to ET: Charters Form a mission before you test Think about the tools you want to use Think about what you’re hoping to find out Example Charter: Explore ApprovalTests By putting different variables in the verifyObject method To discover if you get any unexpected behaviour
- 21. Examples from Elisabeth Hendrickson: Explore IT
- 22. 22 2nd Charter: Get Specific Explore ApprovalTests By putting different contents in the verifyObject method To discover unexpected behaviour
- 24. ET approaches, suggestions: SFDPO (James Bach) Sequences and interactions (Explore It!) Entities: CRUD, follow the data (Explore It!) States & transitions (Explore It!) Tours: super model tour, clubbing tour, couch potato, landmark tour, intellectual tour, FedEx tour Test Heuristics cheat sheet More suggestions?
- 25. Suggested Tools: Browser: inspector, Postman, Bug Magnet, color picker, empty cache/cookies, Spectrum Logging/Proxy: Charles, Fiddler Security: OWASP ZAP Performance/load: Gatling, Jmeter Git Any other suggestions?
- 26. Tools to support documenting Rapid Reporter (mac/windows) Post It’s Microsoft Exploratory Testing add-on TextEdit Screen recorders A second (or 3rd, 4th…) pair of eyes Charles/Fiddler to record you API calls More ideas?
- 27. 27 Disposable test automation? Programming tests for beyond regression?
- 28. Automation ideas beyond regression Fuzzing Model-based Testing Monitoring/analytics Step to –tools Chaos monkey State control with mocks/stubs Pairing with automation ‘tour bus’ Data generation Mutation testing 28
- 29. Main characteristics of ET Self-directedness Freedom Responsibility Learning Working with your own biases Improving yourself as a tester Contextual
- 30. 30 Maaret Pyhäjärvi Email: maaret@iki.fi Twitter: @maaretp Web: maaretp.com Blog: visible-quality.blogspot.fi (please connect with me through Twitter or LinkedIn) Maaike Brinkhof Email: mbrinkhof@xebia.com Twitter: @maaikees Web: maaikebrinkhof.nl Blog: maaikebrinkhof.nl
- 31. Extra slides beyond this one 32
- 32. Example Charters Explore input fields With injection attacks To discover security vulnerabilities Explore editing profiles With invalid usernames To discover if there are any instances where username constraints are not enforced
- 33. Example Charters (2) Explore editing profiles With invalid usernames To discover if there are any instances where username constraints are not enforced
- 34. ET Definition Cem Kaner: “Exploratory testing is a style of software testing that emphasizes the personal freedom and responsib of the individual tester to continually optimize the value of her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.” James Bach: “Simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next”
- 35. Testing vs Checking debate Checking: tests you design in advance, can be automated, can be scaled, modeled, etc. Testing: only a human can do testing, it involves judgement on the fly, skill, sharp decision making. You cannot test everything! So test wisely
- 36. Context Driven Principles The value of any practice depends on its context. There are good practices in context, but there are no best practices. People, working together, are the most important part of any project’s context. Projects unfold over time in ways that are often not predictable. The product is a solution. If the problem isn’t solved, the product doesn’t work. Good software testing is a challenging intellectual process. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.
- 37. Agile Manifesto Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more.
- 38. That's where the power of exploratory testing comes in: the richness of this process is only limited by the breadth and depth of our imagination and our emerging insights into the nature of the product under test.
Notes de l'éditeur
- Question: Which title do you want? Exploratory Testing Experienced and Explained? Or this one?
- asking all the questions beforehand? OR, asking all the questions at the time you need them and learning from the answer one by one
- expecting results beforehand bias testing is about trying to find the unexpected problems too
- variables. Testing isn’t about coverage/covering every variable.
- human memory is limited *room for discussion*
- There’s a group of people who think of testing as creating artifacts: test cases, test automation. Tests are something that is used later for testing to happen again. The modern way of thinking leads this group of people to mostly consider test automation, recognizing that making people run mundane repeatable scripts isn’t the goal. With the automation artifacts, we get four types of benefits. The tests work as specification, giving us a detailed example of something. The tests give us feedback when we’ve implemented something, to see if what we have is what we intended. The tests stay around to guard us against regression, us forgetting details when paying attention elsewhere. And the tests give us granularity, helping us see where the problem is when the tests fail. Another group of people thinks of testing as performance. It’s learning in layers, to prepare for the ultimate show with all the rehearsals where continuous learning enables us to put our best performance forward towards the end. Artifacts might be useful, but they are often more of an output to document all the layers of learning rather than driving the thinking. With testing as performance, we also get four types of benefits. While this style of testing does not give us a spec, it gives us guidance. It’s often not a binary Yes/No, but more of a consideration if there could be a problem here. The guidance enabling us to learn about the unknown unknowns turn into specs as we make our decisions on how to react to feedback. This testing gives us understanding, in the wider context of use and value. This testing gives us models of what the world around the application looks like, enabling us to go back to lessons we’ve learned and fast-track future learning. Finally, this testing gives us serendipity, lucky accidents to find many of the things that we should know of, but turn blind to assuming we know what we’re looking for. While exploratory testing starts with the testing as performance in mind, it also generates artifacts whenever deemed useful. Testing of an API is a great way to get to the core of this difference in how we think about testing. All testing may be exploratory, but some of it is focused on creating artifacts. Saying “scripting is just an approach” is belittling when scripting can be the main approach to use the powers when testing. “there’s a process of knowing” – learning Does not give as regression; serendipity (safety against things happening randomly) / unwanted serendipity events. This is what it is and what it could be. There’s a direction to it, not just statement of what it is. Coaching is not just feedback, it’s pointing them to the right way. Safety. EXPERIENCE (the verb) rather than facts ; emotions over facts. REACTIONS. HISTORY, Lessons learned, checklists. Modeling. UNDERSTANDING – where you start (knowing the thing (code & environment), knowing the user, knowing the problems, knowing the developers (how to help them and what they do so that you can efficiently test), knowing the hackers (weird use cases outside common ‘have you tried reading it upside down’) , knowing all stakeholders, knowing the business priorities) Uncovering things I cannot know, giving the application a change to reveal information for me. This allows you to know things.
- EACH MOUSE CLICK INTRODUCES THE NEXT PERSONS POSITION. ORDER: DRIVER, DESIGNATED NAVIGATOR, NAVIGATORS
- Notice how easier the left side is to read and look at than the right side
- Nog verder uitwerken, huiswerk voor Maaike Deelnemers zelf laten kiezen welke ze willen doen! http://searchsoftwarequality.techtarget.com/tip/Six-tours-for-exploratory-testing-the-business-district-of-your-application http://reqtest.com/testing-blog/what-is-tour-testing-a-structured-approach-to-exploratory-testing/
- Ik ga niet weer een definitie oplepelen hier, tenzij jullie vinden dat ik dat wel moet doen. Wel definitie voorbereiden, op papier zelf. Mensen kunnen er om vragen. Ik wil vooral benadrukken wat de mooie kanten van ET zijn t.o.v en in samenwerking met de rol die test automation heeft.
- Documenteren in ET: vroeger was het ‘alleen bugs zijn boeiende dingen om te documenteren’. Ik denk daar tegenwoordig anders over: als je heel erg leunt op scripted automated tests en daarnaast alleen dit doet moet je het slim aanpakken. Zorg dat zo min mogelijk kennis verloren gaat en dat de leercurve in je tests omhoog blijft gaan. Kortom: leer van jezelf. Dit kan je alleen voor elkaar krijgen als je tijdens AL je test sessies wijselijk documenteert. Schrijf op WAT je hebt getest. Waarom is dat boeiend?? Omdat je dan weet wanneer je te weinig variatie in je tests hebt gebracht. Blijf de invoer en stijl veranderen, anders worden je test sessies minder waard.
- “That's where the power of exploratory testing comes in: the richness of this process is only limited by the breadth and depth of our imagination and our emerging insights into the nature of the product under test.” http://www.satisfice.com/articles/what_is_et.shtml
- http://context-driven-testing.com/