Ten Simple Rules for Writing Great Test Cases (JavaOne 2017)
•Download as PPTX, PDF•
1 like•1,283 views
We’ve all been there: debugging problems in a test and silently screaming into the dark. Poor test cases can cost you time and effort and degrade the quality of your application or product. Testing is vitally important, but so is having a test suite you can use effectively and can rely on. This session takes you through the top 10 rules for writing effective and reliable test cases. The new kids on the block such as Cloud and Docker and general “infrastructure as code”-style solutions may make you believe that old rules are just old. This presentation will make you think again. Knowledge gained from personal experience is always best. Learn from two old masters how to design great test cases, and maybe you’ll never have to visit the dark side again.
Recommended
Recommended
More Related Content
More from Steve Poole
More from Steve Poole (20)
Recently uploaded
Recently uploaded (20)
Ten Simple Rules for Writing Great Test Cases (JavaOne 2017)
- 1. #JavaOne @spoole167 @stuartmarks TEN SIMPLE RULES FOR WRITING GREAT TESTCASES
- 2. #JavaOne @spoole167 @stuartmarks Steve Poole : IBM JVM Developer DevOps practitioner Developer Advocate Stuart Marks : Oracle Principal MTS Java / OpenJDK Core Libraries
- 3. #JavaOne @spoole167 @stuartmarks “I get paid for code that works, not for tests” How to maximize your effort and protect your investment in tests and testing now with added Cloud
- 4. #JavaOne @spoole167 @stuartmarks 1. Think before you act 2. Make your tests understandable 3. Keep your tests “small and simple” 4. Test one thing only 5. Fast tests only 6. Absolute repeatability 7. Independent tests only 8. Provide diagnostic data on failure 9. No hard-coding of your environment 10.No extraneous output
- 5. #JavaOne @spoole167 @stuartmarks 1. Think before you act What are you testing? Why are you testing? Plan https://www.flickr.com/photos/ayurvedicmedicines/
- 6. #JavaOne @spoole167 @stuartmarks Unit tests Function tests System tests Developer QA Easy to understand Long runningFast running Complex Domain Knowledge required Dev QA Don’t think like this Think like this
- 7. #JavaOne @spoole167 @stuartmarks 2. Make your tests understandable Comments Expected behaviour Aid debug Diagnostics https://www.flickr.com/photos/83633410@N07/
- 8. #JavaOne @spoole167 @stuartmarks 3. Keep your tests “small and simple” Separate test logic / setup Much easier to debug Use setup / teardown https://www.flickr.com/photos/9266144@N02/
- 9. #JavaOne @spoole167 @stuartmarks 4. Test one thing only One scenario per test Enables fast debug Obvious why test failed https://www.flickr.com/photos/ryantron/
- 10. #JavaOne @spoole167 @stuartmarks 5. Fast tests only Run unit tests often as possible Maintain quality bar Quick results https://www.flickr.com/photos/mcleod/
- 11. #JavaOne @spoole167 @stuartmarks 6. Absolute repeatability Non-deterministic tests are a headache Fix intermittent tests immediately No value, waste of resource Must trust all tests https://www.flickr.com/photos/fdecomite/
- 12. #JavaOne @spoole167 @stuartmarks 7. Independent tests only Must run in any order Run subset, faster results No dependencies https://www.flickr.com/photos/sheila_sund/
- 13. #JavaOne @spoole167 @stuartmarks 8. Provide diagnostic data on failure Use message in asserts Make it simple to debug Reference input data Record test environment info https://www.flickr.com/photos/davidbaker/
- 14. #JavaOne @spoole167 @stuartmarks 9. No hard-coding of your environment No ports, IP addresses, data files, databases Use config files, system properties or mock objects Portable tests https://www.flickr.com/photos/pug50/
- 15. #JavaOne @spoole167 @stuartmarks 10. No extraneous output A passing test is a silent test Too much output = confusion Use option, config file to turn on debug, save output https://www.flickr.com/photos/3-bs/
- 16. #JavaOne @spoole167 @stuartmarks Pirate rules Guidelines only Some may seen obvious. Some may seem pedantic Until you inherit a testsuite… Thank you. https://www.flickr.com/photos/gapic/
Editor's Notes
- Questions for the end or on twitter..
- Steve: People say “ I don’t need to write tests my code is good enough.” “I only get paid for code” Your brain is not big enough to keep even simple code in your head Good tests are essential – poor tests are worse than useless. This talk is about showing why you need to write tests and why in the world of cloud and containers its becoming even more important This talk is about getting the best out of the time you spend writing tests.
- Pirate Rules (guidelines only) Can we agree on rules vs guidelines ? This list is the top ish(10) from professional testers and developers. Most of our pain has come from from personal experience in breaking these rules.. Our experiences only. Valueable. Its not exclusive, just our opnion. And we don’t have time for more.. Non of these are unimportant
- Steve – intro and substance --- Your first job is to capture the expected behavior of your system so you can continue to show you have not broken behavior. Old platitudes: “You cannot test quality in” But without tests you cannot assert ‘quality’ you only have so much time to capture your systems behavior in tests - don’t waste it We’ve all seen large test suites that mostly don’t test anything useful : simple getters and setters please?
- What happens if your tests are not understandable? Should tests be easier to understand than the code their testing? Tests should be coded and commented well. A test suite is the inverse of the code. Its codifing expected behaviour so write it down what your think the behavior is. Don’t assume your testcode is perfect – write comments to explain your intent When an test fails you don’t want to be spending time figuring out if it’s the test. How would you report a bug if you can’t work out what the test is testing? They say debugging is twice as hard as writing code - its easy to write tests that are much harder to understand than the code their testing. Debugging failing tests like that takes more brain power than anyone possesses. Be nice to you future self trying to figure out whats wrong when your trying to deploy code at midnight
- Steve: tests should have a well defined setup Can you find the actual line of testing in a large test suite Why are we saying this - how often have you had to step a 1000times Or debug a nested loop inside a nested loop? Or inserted code to allow you to set a break point. If you have to hack the test its too big clean separation between getting to to the actual code
- Number of times a test has multiple asserts to ’help’ you work out what was wrong before running the test. But pre-failures mean your not getting close to your actual code. Let other tests find those problems. - add help info into the logs not as asserts Tests that test multiple things complicate how your measure progress. Soon you find you have blocking defects that are artificial – they’re not decessarly important to your users but they stop lots of tests running. And of course its so much easier to run simple tests and understand the point of the test if its only got one assert
- Steve: Testing is domain / time context We’d all like all our tests to run all the time. We can’t do that so when backing off chose wisely. Running the whole testsuite is useful. Long tests encourage you to not run tests. You let more bugs through Even if part of a CI run.. Long running test suites are evil. They encourage corner cutting. The make you think about not running tests… They make your developers less productive. I’ll just submit this one line code change for building over night… Or I’ll have my private set of tests - I’ll just run the tests that I think are the right ones. Things fail later (and cost more) when you could have got the developer to run the test – if only it ran faster..
- Q: “Steve how does this work in a cloud environment?” “Everything fails all the time in the cloud” Cloud: reliable system built from unreliable components. This manifests itself in intermittent test failures. How to distinguish "valid" failure (failure that can be handled by the larger system) vs an "invalid" failure? Do we care? "Offensive programming" – deliberate injection of failures. Useful for system tests. --- Random without a seed. Sleeping to wait for stuff to happen. Repeatability with timeouts is difficult. “Chaos Monkeys”
- Steve: War story - junit test ordering. Broke then jit… Paralleise using clusters Exposed unexpected test interdependences… State, setup database , add records etc, then run tests. Later tests assumed state based on those . I want to one day run one of these tests singualry for debug purposes. I don’t want to run 50 others first! Run your tests in random order?
- Steve : War story FFDC (any part of the code could have raised the error. Customer wanted it fixed…! Clear simple information at failure to find code and culprit data Note that you need to test that failure generation too. Log(foo) Log(foo.bar) If(foo==null) … works for me syndrome + Cloud: Remote tests are difficult remove env is always different and un reproducable locally..
- Steve: War story “the App server test env and 1yr” + Cloud needs you not to do this.. Every instance could be on a different box, somewhere else in the world Reduce opportunities to be effected by using mock objects.
- Q: You like verbose output don’t you Steve? Steve: if you have too much output - > you invest in systems to analyse the log and look for the real ‘truth’
- Steve: Just like the movie says – these are all guidelines. We think you should think before you ignore the rules. Wait until you inherit your first test suite.. Q: Stuart –any closing thoughts?