Static analysis tools are used for improving software quality and reliability. Since these tools can be time consuming when used for analysis of big codebases, they are normally run during scheduled (e.g. nightly) builds. However, the sooner a defect is found, the easier it is to fix efficiently. In order to detect defects faster, some analysis tools offer an integration with the integrated development environment of the developers at the cost of not always detecting all the issues. To detect defects earlier and still provide a reliable solution, one could think of running an analysis tool at every build of a continuous integration system. In this paper, we share the lessons learned during the integration of the static analysis tool Klocwork (that we are developing) with our continuous integration system. We think that the lessons learned will be beneficial for most companies developing safety-critical software (or less critical systems) that wish to run their analysis tool more often in their build system. We report these lessons learned along with examples of our successes and failures.

1. 1© 2016 Rogue Wave Software, Inc. All Rights Reserved. 1
Using static analysis tools
within continuous
integration systems
Claude Bolduc, Software architect

2. 2© 2016 Rogue Wave Software, Inc. All Rights Reserved. 2
Clang Tidy
Clang Static
Analyzer Krakatoa
AST matching
Dataflow
analysis (PATH)
Formal
methods

3. 3© 2016 Rogue Wave Software, Inc. All Rights Reserved. 3
SCA tool for security
and reliability
ISO
26262
AST and
PATH
C/C++,
java, C#

4. 4© 2016 Rogue Wave Software, Inc. All Rights Reserved. 4
Scheduled nightly analysis
of our source code
(integration)
“Desktop analyzer”
for individual
developers
(verify local changes)

5. 5© 2016 Rogue Wave Software, Inc. All Rights Reserved. 5
More expensive context
switching when a defect is
discovered.
Who introduced the defect
in a sea of commits?
Desktop analyzers
might miss
defects!
• Missing code
• Different
configurations
Difficult to track
and manage usage
on developers’
machines
This process does NOT mimic
our complete process!

6. 6© 2016 Rogue Wave Software, Inc. All Rights Reserved. 6
Poll
repository
Build
Run
SCA
Test
Results
Continuous
integration
workflow
(CI)

7. 7© 2016 Rogue Wave Software, Inc. All Rights Reserved. 7
Could work
on small
codebases
Does not scale
well for bigger
codebases
Analysis time increase!

8. 8© 2016 Rogue Wave Software, Inc. All Rights Reserved. 8
• Goal: Use deep semantic analysis
(dataflow analysis) in our CI builds
without impacting too much the time
for our builds
• Ongoing for more than 2 years
• 14 months of collected data in our
source code

9. 9© 2016 Rogue Wave Software, Inc. All Rights Reserved. 9
Builds done in 14 months: 2863 (around 10 builds/day)

10. 10© 2016 Rogue Wave Software, Inc. All Rights Reserved. 10
0
10
20
30
40
50
60
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49
Newdefectsintroduced
Build
New third party library
1 file, same issue
Merge big feature
> ½ of builds
Introduced 1 or 2 defects

11. 11© 2016 Rogue Wave Software, Inc. All Rights Reserved. 11
Lessons
learned

12. 12© 2016 Rogue Wave Software, Inc. All Rights Reserved. 12
• Klocwork already has an incremental analysis for:
– AST matching
– dataflow analyses
Call graph
No need if side effects did not change!

13. 13© 2016 Rogue Wave Software, Inc. All Rights Reserved. 13
0
20
40
60
80
100
120
140
160
180
CI build (no analysis) Nightly build (no analysis) Scheduled nightly analysis with
incremental mode
Timeinminutes
Time to run

14. 14© 2016 Rogue Wave Software, Inc. All Rights Reserved. 14
• Which information do we need in the context of CI?
• Only information about new defects found and fixed!
• But Klocwork is generating data for:
– Metrics
– Symbols and relations for code navigation
– Architecture
– Historical data (relation between builds)
– …
• In CI, we should only focus our process on new defects!

15. 15© 2016 Rogue Wave Software, Inc. All Rights Reserved. 15
0
20
40
60
80
100
120
140
160
180
CI build (no analysis) CI analysis Nightly build (no analysis) Scheduled nightly
analysis with incremental
mode
Timeinminutes
Time to run
Quick analysis
Full-fledged analysis

16. 16© 2016 Rogue Wave Software, Inc. All Rights Reserved. 16
Educate and
engage people
in the new
process

17. 17© 2016 Rogue Wave Software, Inc. All Rights Reserved. 17
Fail the build?
Workflow for recovery?
Pass the build?
• Coding standard like MISRA
• Build workflow centered
around pull requests
• Defects that involves deep
dataflow analysis:
• Long investigation
• Long resolution
• Potential FP
Should be driven by the kinds of defects AND the development process!
Defects are not “lost”:
Safeguard of the
scheduled analysis

18. 18© 2016 Rogue Wave Software, Inc. All Rights Reserved. 18
• More precise initial data enables easier automation
Before Now
Algorithm:
Mix of “who change
what” with “expert”
Algorithm:
Who introduced the
bad commit?
Advantages:
• Fix defects earlier
• “Right” person gets notified
• Memories are still fresh

19. 19© 2016 Rogue Wave Software, Inc. All Rights Reserved. 19

20. 20© 2016 Rogue Wave Software, Inc. All Rights Reserved. 20