1.
1
C++ Testing Techniques, Tips and Tricks
Clare Macrae (She/her)
clare@claremacrae.co.uk
21 November 2019
C++ London

2.
2
About Me
• Scientific C++ and Qt developer since 1999
• My mission: Sustainable and efficient testing and refactoring of legacy code
– Co-author of “Approval Tests for C++”
• Consulting & training via “Clare Macrae Consulting Ltd”
– https://claremacrae.co.uk
• All links from this talk via:
– github.com/claremacrae/talks

3.
3

4.
4
Why?
• Share things I wish I knew earlier
– Some well known
– Some less so…
• High-level overview, with links
– Avoiding detail
• Anything unclear?
– Please ask!

5.
5
Getting Started Testing C++

6.
6
Tip: Pick a Test Framework
• Lots to choose from, including:
• Get to know it well!

7.
7
Tip: Practice!

8.
8
Practice!

9.
9
Scenario: Writing your first test
• Getting from 0 to 1 tests is always the hardest – that’s normal
• Getting from 1 to 2, and then 2 to 3, always much easier
• Step 1: Make the first test as easy as possible – it’s going to be hard!
• Step 2: Write at least 3 tests before deciding if it’s a good idea!

10.
10
Techniques for Testing

11.
11
Traditional unit tests
// Let Catch provide main():
#define CATCH_CONFIG_MAIN
#include <catch2/catch.hpp>
int Factorial( int number ) {
return number <= 1 ? number : Factorial( number - 1 ) * number; // fail
// return number <= 1 ? 1 : Factorial( number - 1 ) * number; // pass
}
TEST_CASE( "Factorial of 0 is 1 (fail)", "[single-file]" ) {
REQUIRE( Factorial(0) == 1 );
}
TEST_CASE( "Factorials of 1 and higher are computed (pass)", "[single-file]" ) {
REQUIRE( Factorial(1) == 1 );
REQUIRE( Factorial(2) == 2 );
REQUIRE( Factorial(3) == 6 );
REQUIRE( Factorial(10) == 3628800 );
}

12.
12
Property Based Testing
• State the Invariants – what is required to be true

13.
13
Kevlin Henney: FizzBuzz Trek HD

14.
14
C++ Property Based Testing
• Can roll your own, in helper functions
• RapidCheck:
– github.com/emil-e/rapidcheck

15.
15
Summary: Techniques for Testing
• Traditional Unit Tests
• Property Based Testing

16.
16
Inheriting Legacy/Existing Code

17.
17
Typical Scenario
• I've inherited some
legacy code
• It's valuable
• I need to add feature
• Or fix bug
• How can I ever break
out of this loop?
Need to
change
the code
No tests
Not
designed
for testing
Needs
refactoring
to add
tests
Can’t
refactor
without
tests

18.
18
Typical Scenario
•
•
•
•
•
Need to
change
the code
No tests
Not
designed
for testing
Needs
refactoring
to add
tests
Can’t
refactor
without
tests
Topics of
this part

19.
19
How good are your existing tests?

20.
20
Measure Code Coverage
• Caution!
• Useful to see unexecuted code
• Use a code coverage tool
– What to measure?

21.
21
Line Coverage: CLion 2019.3

22.
22
Branch Coverage

23.
23
Reference: Set-up on my Mac
• Slightly hard-won:
• g++-8
• CodeCoverage.cmake in github.com/bilke/cmake-modules
• lcov
• ~/.lcovrc
genhtml_branch_coverage = 1
lcov_branch_coverage = 1
lcov_excl_br_line = LCOV_EXCL_BR_LINE|CHECK|REQUIRE

24.
24
Mutation testing: Sabotage the code!
• Test the tests
• Small changes
• Re-run tests
• Fail: 
• Pass: 

25.
26
Golden Master Test Setup
Input
Data
Existing
Code
Save
“Golden
Master”

26.
27
Golden Master Tests In Use
Input
Data
Updated
Code
Pass
Fail
Output
same?
Yes
No

27.
28
Approval Tests
TEST_CASE( "Factorials up to 10" )
{
// Create a container with values 0 ... 10 inclusive
std::vector<int> inputs(11);
std::iota(inputs.begin(), inputs.end(), 0);
// Act on all values in inputs container:
Approvals::verifyAll("Factorial", inputs,
[](auto i, auto& os)
{
os << i << "! => " << Factorial(i);
});
}

28.
29
Quickly Testing Legacy C++ Code
with Approval Tests

29.
30
Scenario: Golden Master is a log file
• Dates and times?
• Object addresses?
2019-01-29 15:27

30.
31
Options for unstable output
• Introduce date-time abstraction?
• Customised comparison function?
• Or: strip dates from the log file

31.
32
Tip: Rewrite output file, before testing it
2019-01-29 15:27 [date-time-stamp]

32.
33
Summary: Legacy Code
• Validating existing tests
– Consider Code coverage to validate initial tests
• Prefer Branch coverage to Line coverage
– Consider Mutation testing to break tests
• Consider Fuzz testing to find extra test cases
• Approval Tests
• Be creative to make your tests work for you!

33.
34
Maintaining Tests

34.
35
Scenario: Tests too slow for CI
• “Builds take 2 to 3 days!” => No hope of Continuous Integration
• Most of that was running tests
• No prospect of speeding up some very slow tests

35.
36
Solution: Don’t run slow tests during day
• Mark some tests as Slow
– Filtering built in to Google Test Framework
• Teach CI system to run sub-set of tests
– Only run the Slow tests at night
– Everything else run on every commit
• Thanks to Michael Platings for making this happen!

36.
37
Speeding up tests
• Incredible value of powerful tools for seeing where time goes

37.
38
Tip: Never tolerate Flickering tests
• Tests that fail randomly
• Rapidly devalue other tests
• Fix them, or delete them!
• Especially don’t mix performance tests with unit tests

38.
39
Summary: Maintaining Tests
• Fast feedback from Continuous Integration
• Stomp on flickering tests

39.
40
Testing Qt User Interfaces

40.
41

41.
42
Qt’s GUI powers make
Automated Testing Harder

42.
43
Introducing ApprovalTests.cpp.Qt
•Goals
• Rapid start testing Qt code
– Useful even if you don’t use Approval Tests!
• Approval Tests support for Qt types
– Easy saving of state in Golden Master files
• https://github.com/approvals/ApprovalTests.cpp.Qt
• https://github.com/approvals/ApprovalTests.cpp.Qt.StarterProject
• v.0.0.1

43.
44
Example: Checking Table Contents
• Inherited complex code to set up a table
• Want to add at least a first test – of the text in the cells

44.
45
Verifying a QTableWidget
TEST_CASE("It approves a QTableWidget")
{
// A note on naming: QTableWidget is a concrete class that implements
// the more general QTableView. Here we create a QTableWidget,
// for convenience.
QTableWidget tableWidget;
populateTable(tableWidget);
ApprovalTestsQt::verifyQTableView(tableWidget);
}

45.
46
Approval file: .tsv

46.
47
More Info…
• Key point: Use Fixtures to write Expressive, Maintainable Tests
– slideshare.net/ClareMacrae/quickly-testing-qt-desktop-applications

47.
48
Summary
• It’s never too late to start testing!
– Pick a framework and practice it
– Explore different types of testing
• You can test legacy code too!
• Keep maintaining your tests
• Even GUIs can be tested

48.
49
C++ Testing Techniques, Tips and Tricks
• All links from this talk, and more, via:
– github.com/claremacrae/talks
• Sustainable and efficient testing and refactoring of legacy code
• Consulting & training via “Clare Macrae Consulting Ltd”
– https://claremacrae.co.uk
– clare@claremacrae.co.uk
• Any Questions?
• Any More Tips?