Contenu connexe Similaire à Pivotal Labs Open View Presentation Quality Assurance And Developer Testing Similaire à Pivotal Labs Open View Presentation Quality Assurance And Developer Testing (20) Pivotal Labs Open View Presentation Quality Assurance And Developer Testing1. Quality Assurance
and Developer Testing
Edward Hieatt, VP Engineering
Ian McFarland, VP Technology
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
2. What’s the goal?
• To develop software quickly, cheaply, with
the lowest defect rates possible
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
3. In practice, we want...
• Ability to deploy new releases frequently
o at a bare minimum, once per iteration/sprint
o ideally, as each new feature is finished
• Confidence that everything we build has a very low defect
rate
• Confidence that adding features/performing refactorings -
won't introduce defects
• Minimal firefighting/bug fixing - focus on new features
• Keep code flexible enough to keep feature cost constant
• Sleep well at night
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
4. The Old Way
• Write code
• Deploy to QA
• QA finds and reports bugs
• Development fixes bugs
• Deploy to QA
• QA finds bugs
• Development fixes bugs, repeat
• QA doesn't find anymore bugs
• Deploy to production
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
5. Who’s responsible for
Quality?
•QA’ers are responsible
•Quality tested after development is
complete
•Developer-QA team relationship is
loose & adversarial
•Automated tests are an afterthought
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
6. The Result
• Ever-increasing percentage of development time
spent on bug fixing and firefighting
• Ever-decreasing percentage available for feature
development
• Increasing fear that new features will inject bugs
• Architecture changes become prohibitively
expensive
• Performance concerns conflict with stability concerns
• Eventually, feature production grinds to a halt
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
7. Rethinking our attitudes
torward testing
• Everyone is responsible for quality
o Developers responsible for the quality of their code
o 'Customer' responsible for accepting work
o QA operates at a higher level of abstraction
• Reduce the cost of defects
o Move defect detection as early in lifecycle as
possible
• Make quality transparent
o Make bug detection easy
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
8. Quality as the focal point
• Tests are the core of the development process
• Test all the time, not just at the end
• Automate everything
• Invest in making it fast and easy to run tests
• Write automated tests before we write the code
• Make build status visible to the entire team
• Run regression suites at every check-in
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
10. Kind of Tests
• Unit Tests
• Functional Tests
• Acceptance Tests
• Smoke Tests
• Integration Tests
• End-to-End tests
• Scenario Tests
• System Tests
• UI Tests
• Manual tests
• Exploratory Tests
• Regression Tests
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
11. Kind of Tests
• Unit Tests
• Functional Tests
• Acceptance Tests
• Smoke Tests
• Integration Tests
• End-to-End tests
• Scenario Tests
• System Tests
• UI Tests
• Manual tests
• Exploratory Tests
• Regression Tests
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
12. Testing Frameworks
•Domain Specific Languages (DSLs)
•FIT and FITnesse
•DSLs and FIT let PMs, analysts and QA
write more and better tests
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
16. Where’s the Bug?
Test Fixture Client
View
Controller
Test Fixture Model
Database
17. Isolation
•Tests at different granularities help you
more quickly identify the source of the
problem
•Good test coverage tells you what’s wrong
as quickly as possible
18. The Food Pyramid
Confidence in System as a Whole Increases
Test Isolation, Focus, Speed Increases
Scenario
Tests
Functional
Tests
Integration Tests
Unit Tests
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
19. Testing as part of
Development
•Test-Driven Development
•Red-Green-Refactor
•No code before a red test demands it
•Outside-in vs. Inside-out
•Fast test run frequently
•Continuous Integration
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
21. A Typical Day at Pivotal
• Standup - team communication
• Pair-up
• Pair pops next story from stack in Tracker
• Update from RCS
• Test-drive story [repeat]
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
22. Test-Driving a Story
• Write functional test (test is red)
• Write unit tests [repeat]:
• Write unit test (test is red)
• Fix unit test by writing code (test is green)
• Refactor (tests are green)
• Rerun functional test (test is green)
• Run all tests (tests are green)
• Update from RCS
• Run all tests (tests are green)
• Check in
• Mark story finished
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
23. How do we get there?
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
24. Action Plan
•Stop the bleeding
•Transition to testability
•Retrofit coverage where needed
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
25. Stop the bleeding
•Write a test every time a new defect is
detected, to isolate the test
•Test all new code
•Refactor touch points with old code for
testability
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
26. Transition to
Testability
•Characterization/black box tests first
•Refactor for testability
•Gradually work down to unit tests in
legacy code
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
27. Retrofit coverage
•Make retrofitting tests part of everyday
work
•Don’t “stop” and cover all your old code
at once
•Measure coverage as you go
•Have goals and measure progress
•Understand what coverage you have
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
28. Steady State
•Maintain Testability
•Continuously improve test suite
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
29. Maintaining Testability
•...requires vigilance
•It’s easy to slip into old habits...
•when there’s time pressure
•when you’re tired
•when you’re soloing
•When it’s hard to write tests, do it more
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
30. Maintaining Testability
•Keep code amenable to testing
•Keep coverage high, and focused
•Keep the test corpus small
•Less is more, assuming your coverage is
complete
•Don’t punt on hard testing problems
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
31. Continuous
Improvement
•Test Speed
•Test Clarity
•Test Quality
•Test Readability
•Remove stale tests
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.
32. The role of QA
on an Agile team
• Different pair of eyes
• Can focus on exploratory testing and test
planning
• Catch gaps in requirements and coverage
• Should write additional coverage
• Less monkey work, more brain work
• Any bug discovered by QA must have an
automated regression test case written
Copyright © 2008 Pivotal Labs, Inc. All rights reserved.