The Lies We Tell About Software Testing
•
3 likes•3,176 views
This talk is more about half-truths than lies, but the concepts are important. There are several areas in which we think we have more confidence in our information than we actually do. This presentation covers both areas in which the evidence is against our beliefs and areas in which we simply need more information.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (20)
Similar to The Lies We Tell About Software Testing
Similar to The Lies We Tell About Software Testing (8)
More from Curtis Poe
More from Curtis Poe (13)
Recently uploaded
Recently uploaded (20)
The Lies We Tell About Software Testing
- 1. Tes$ng Lies (The &tle is the first lie) Hannover, Germany, 2014 Cur$s “Ovid” Poe h5p://www.allaroundtheworld.fr/ Photo CC license: h5p://commons.wikimedia.org/wiki/File:Hannover_Altstadt_128-‐h.jpg Copyright 2014, All Around The World March 26, 2014
- 2. But first … March 26, 2014 Copyright 2014, All Around The World h5p://www.slideshare.net/gvwilson/bits-‐of-‐evidence-‐2338367
- 3. Bits of Evidence by Dr. Greg Wilson March 26, 2014 Copyright 2014, All Around The World • Overwhelming evidence that evidence works • Science demands evidence • Programmers provide anecdotes • (… and defend them vigorously)
- 4. Evidence • Absence of evidence is not evidence • … but evidence of absence is March 26, 2014 Copyright 2014, All Around The World
- 5. Absence of Evidence • There is no evidence that gods exist • There is no evidence that gods don’t exist March 26, 2014 Copyright 2014, All Around The World
- 6. Evidence of Absence • Belief: small subs more maintainable/correct • Reality: these proper$es are absent March 26, 2014 Copyright 2014, All Around The World
- 7. Evidence of Absence • Belief: small subrou$nes/methods “be5er” • Inverse correla$on (Bell, Perricone 1984) • No correla$on (Shen, et al. 1985) • No correla$on and larger is cheaper (two studies) • Small rou$nes had more errors (Selbi, Basil 1991) • Less maintenance for larger rou$nes (Lind, Vairavan 1989) • 500+ lines had more errors (Jones 1986) (cita$ons via: Code Complete, Steve McConnell, Microsoh Press, 2nd Edi$on) March 26, 2014 Copyright 2014, All Around The World
- 8. P.O.P. Copyright 2013, Cur$s "Ovid" Poe Reality trumps opinion 3/26/14
- 9. Yeah, yeah, let’s get started March 26, 2014 Copyright 2014, All Around The World
- 10. Caveats March 26, 2014 Copyright 2014, All Around The World • Some points self-‐evident • Some points maddening • Not many studies • Always be skep$cal Photo CC BY-‐SA 2.0 h5p://www.flickr.com/photos/busbeytheelder/515542705/
- 11. Code Coverage March 26, 2014 Copyright 2014, All Around The World
- 12. 100% Code Coverage March 26, 2014 Copyright 2014, All Around The World
- 13. 100% Code Coverage • What if we pass zero? • … or a string? • … or no argument? • … or a filehandle? • … or type when reciprocals don't make sense? • … or, or, or … March 26, 2014 Copyright 2014, All Around The World
- 14. Expecta$ons • Tes$ng should be an act of aggression • Programmers tend to test against expecta$ons • False confidence March 26, 2014 Copyright 2014, All Around The World
- 15. Code Coverage of Dead Code March 26, 2014 Copyright 2014, All Around The World
- 16. Unit Versus Integra$on Tests • Unit tested subs don’t talk to other subs • But your code certainly does! • Dead code is cogni$ve overhead • Dead code can’t be tested via your interface March 26, 2014 Copyright 2014, All Around The World
- 17. Code Coverage of Dead Code March 26, 2014 Copyright 2014, All Around The World
- 18. March 26, 2014 Copyright 2014, All Around The World
- 19. March 26, 2014 Copyright 2014, All Around The World
- 20. Covered but not tested March 26, 2014 Copyright 2014, All Around The World
- 21. Devel::Cover Recap • 100% code coverage doesn’t handle data • Don’t cover dead code • Covered but not tested • Don't blindly trust your tools March 26, 2014 Copyright 2014, All Around The World
- 22. Kwalitee • Lie: Perl community is the most test-‐infected • Truth: Quan$ty ain’t quality March 26, 2014 Copyright 2014, All Around The World
- 23. CPAN Testers (March 6, 2014) March 26, 2014 Copyright 2014, All Around The World
- 24. Versions and Plauorms March 26, 2014 Copyright 2014, All Around The World
- 25. CPAN Testers March 26, 2014 Copyright 2014, All Around The World
- 26. Perl Isn’t Stealing Enough We need quality to go with quan$ty March 26, 2014 Copyright 2014, All Around The World
- 27. All Pairs Tes$ng March 26, 2014 Copyright 2014, All Around The World
- 28. The Coupling Effect • Inves$ga$ons of the Sohware Tes$ng Coupling Effect — A. Jefferson Offu5 • Competent programmers write nearly correct programs • Most faults are simple faults • Simple faults can be easily detected March 26, 2014 Copyright 2014, All Around The World
- 29. All-‐Pairs Tes$ng • 10 arguments == 10 billion combina$ons • Combined pairs == 54 combina$ons • Extend each argument with reasonable values March 26, 2014 Copyright 2014, All Around The World
- 30. All-‐Pairs Tes$ng First pairs Second Pairs Third Pairs Fourth Pairs And so on arg1, arg2 — — — … arg1, arg3 arg2, arg3 — — … arg1, arg4 arg2, arg4 arg3, arg4 — … … … … arg4, arg5 … arg1, arg0 arg2, arg0 arg3, arg0 … … March 26, 2014 Copyright 2014, All Around The World
- 31. All-‐Pairs Tes$ng March 26, 2014 Copyright 2014, All Around The World • Fast genera$on of tests • Likely to catch many more bugs • Built in to many tools (for sta$c languages) • For Perl: ALLPAIRS Test Case Genera$on Tool
- 32. Documenta$on • Lie: Tests are documenta$on • Truth: I wish March 26, 2014 Copyright 2014, All Around The World
- 33. March 26, 2014 Copyright 2014, All Around The World
- 34. March 26, 2014 Copyright 2014, All Around The World
- 35. March 26, 2014 Copyright 2014, All Around The World
- 36. Tests As Documenta$on • Tests are ohen obfuscated • Ohen wri5en as an aherthought • Missing or non-‐descrip$ve test names • Describe code logic instead of business logic March 26, 2014 Copyright 2014, All Around The World
- 37. Making Tests Documenta$on • Write them to be read • Describe use cases with: – test fixtures names – subtest names – test method names March 26, 2014 Copyright 2014, All Around The World
- 38. March 26, 2014 Copyright 2014, All Around The World
- 39. March 26, 2014 Copyright 2014, All Around The World
- 40. March 26, 2014 Copyright 2014, All Around The World
- 41. BDD for Test Documentaiton • Test::BDD::Cucumber • Perl Behavior Driven Development (slideshare) March 26, 2014 Copyright 2014, All Around The World h5p://commons.wikimedia.org/wiki/File:Og%C3%B3rki...jpg
- 42. Test Driven Development • Lie: TDD is a far superior tes$ng methodology • Truth: Prove it. March 26, 2014 Copyright 2014, All Around The World
- 43. A Typical TDD Study • Evalua$ng the Efficacy of Test-‐Driven Development: Industrial Case Studies • Compared to non-‐TDD • TDD increased development $me ~15% • TDD had far fewer defects than non-‐TDD March 26, 2014 Copyright 2014, All Around The World
- 44. What is “non-‐TDD”? • No tes$ng? • Test last? • Test as you go? • QA-‐only tes$ng? • It doesn’t say! March 26, 2014 Copyright 2014, All Around The World
- 45. Same Study “An (sic) year-‐long empirical study performed at IBM using professional programmers found that the TDD prac?ce helps programmers produce higher quality code.” March 26, 2014 Copyright 2014, All Around The World
- 46. Same Study “An (sic) year-‐long empirical study performed at IBM using professional programmers found that the TDD prac?ce helps programmers produce higher quality code.” Higher quality code than what? And what does “quality” mean? March 26, 2014 Copyright 2014, All Around The World
- 47. Pro-‐TDD Studies • Comparisons against straw men common • Ohen use students • Some$mes use single developers • Some$mes aware they’re being studied • Or new to TDD (may be more mo$vated) March 26, 2014 Copyright 2014, All Around The World
- 48. A TDD Enthusiast Looks At Evidence • The benefits of TDD are neither clear nor are they apparent — Kane Mar – Part 1 – Part 2 – Part 3 March 26, 2014 Copyright 2014, All Around The World
- 49. Kane Mar – Aher Reading Studies “[Claims about TDD] such as beHer design, beHer APIs, simpler design, lower complexity, increased produc?vity, more maintainable code etc., are simply not supported.”1 1. A cherry-‐picked, out-‐of-‐context quote. March 26, 2014 Copyright 2014, All Around The World
- 50. The Reality (vis-‐à-‐vis other tes$ng strategies) • TDD tends to generate more tests • Higher quality not proven • Absence of evidence, not evidence of absence • Don’t stress about it March 26, 2014 Copyright 2014, All Around The World
- 51. Summary • Evidence trumps anecdotes • We rely on anecdotes • That’s sort of OK if we don’t have evidence˜ • Code coverage is only part of the solu$on • We need to escape our bubble • Be skep$cal of hype • Be skep$cal of me March 26, 2014 Copyright 2014, All Around The World
- 52. Summary for TDD Fana$cs • Don't sweat anecdotes for casual ideas • Don’t accept anecdotes from dogma$sts March 26, 2014 Copyright 2014, All Around The World
- 53. March 26, 2014 Copyright 2014, All Around The World Image by Hay Kranen
- 54. Bonus Slides! March 26, 2014 Copyright 2014, All Around The World
- 55. Basis Path Tes$ng March 26, 2014 Copyright 2014, All Around The World
- 56. Cycloma$c Complexity March 26, 2014 Copyright 2014, All Around The World Cycloma$c Complexity (McCabe) = (edges – nodes) + ( 2 * exit nodes ) = ( 19 – 14 ) + ( 2 * 1 ) = 7 = minimum number tests
- 57. Calcula$ng Cycloma$c Complexity March 26, 2014 Copyright 2014, All Around The World
- 58. Cycloma$c Complexity and Bugs • Some studies show a posi$ve correla$on • Other studies are not conclusive • May make bugs harder to fix • May lead to side effects when making changes • A Complexity Measure — McCabe • A Cri$que of Sohware Defect Predic$on Models — Fenton, Neil • Cycloma$c Complexity Metrics Revisited — Gill, Kemerer March 26, 2014 Copyright 2014, All Around The World
- 59. Perl March 26, 2014 Copyright 2014, All Around The World
Editor's Notes
- No evidence for or against the existence of a god or godsStrong evidence of absence of effect from astrology
- LorenzoTorrentino 1550 1st edition. Ptolemaic view of the solar system.http://en.wikipedia.org/wiki/File:1550_SACROBOSCO_Tractatus_de_Sphaera_-_(16)_Ex_Libris_rare_-_Mario_Taddei_.JPG
- Caveat: integration tests may have several pieces of dead code talking to each other!
- Simple faults can be corrected by a single change