Florian Villoing presents the infrastructure AdaCore put in place to build and tests its compilation tool chains and add-on technology on a daily basis. He also present the "qualification machine" that AdaCore created to ease the DO-178B tool qualification process. These slides were used to support the talks Florian gave at the Agile Tour 2009 conferences in Grenoble (October 20, 2009) and Valence (October 22, 2009).

1. Lean Software
Production and Qualification
Infrastructures
Florian Villoing – AdaCore
villoing@adacore.com
Grenoble – October 20, 2009
Valence – October 22, 2009

2. What is it about?
● A software production infrastructure
● A software qualification infrastructure
In a lean fashion!
2

3. Lean Software Production
3

4. About AdaCore's products
● Ada compilation toolchain
➔ Compiler (based on GCC), debugger
➔ Interfacing with C, C++ and Java
➔ 2 IDEs: GPS + GNATbench (Eclipse plug-in)
➔ Coverage analysis
● Add-ons: ASIS, XML/Ada, GtkAda, AWS, PolyORB
● Static analysis tools
➔ Metrics computation
➔ Stack analyzer
➔ Coding standard checker
➔ Automatic peer review
4

5. What is a compiler?
“A compiler is a computer program (or set of programs)
that transforms source code written in a computer
language (the source language) into another computer
language (the target language, often having a binary
form known as object code).” Wikipedia
5

6. Native and cross compilers
● A native compiler generates code for the same
machine (the host)
● A cross compiler generates code for another
machine (the target)
native
cross
Host Target
6

7. Native compilers
● Different operating systems
● Windows, Linux, Solaris, Mac OS X, HP-UX, VMS,
LynxOS, Tru64, AIX, IRIX, etc.
● Different Versions
● Windows XP, 2000, Vista, etc.
● Linux Red Hat 3/4/5, SuSe 8/9/10, etc.
● Different processors
● x86, x86 64bits, Itanium, SPARC, PowerPC, Alpha,
PA-RISC, etc.
7

8. Cross compilers
● Host environment (OS, OS version, processor)
➔ Linux, Windows, Solaris
● Target environment (Target OS, Target OS Version,
processor)
➔ VxWorks, PikeOS, ElinOS, Lynx OS, Nucleus OS, etc.
➔ VxWorks 5, VxWorks 6, VxWorks 653, etc.
➔ PowerPC, ARM, AVR, LEON, ERC32, etc.
● Run-time system
➔ ZFP, Ravenscar, Full, etc.
8

9. Combinatory explosion
● Native toolchains
● OS x OS version x processor = 46
● Cross toolchains
● Host OS x host OS version x host processor x
Target OS x target OS version x target processor x
run-time system = 49
9

10. And things are getting more
complex...
● New platforms are added each year
● Few are removed
● Long-term projects need to be supported for up
10-15 years
How do I deal with that?
10

11. Some figures
● 95 different platforms (native and cross)
● 15 independent test suites
➢ 15000 tests and 10 million lines of code
➢ 1.4 million tests run each night
● 96 scripts to control the infrastructure
➢ About 2300 scripts run each day
11

12. How to solve the equation?
● Adopting a lean strategy
● Introducing agile tactics
Let's move on!
12

13. The Toyota Production System
TPS
Jidoka Just-in-Time
● Automatic defects ● Making only what is
detection needed, when it's needed,
in the amount needed
+
● Stop the line
● Limited stocks with all
● Identify the problem parts
● Andon (problem display ● Replace used parts from
board) the preceding process
● An operator for several ● Limited stocks for the
machines preceding process
13

14. Adopting a lean strategy
● Identify the value
● Identify and remove waste (muda)
● Automate
● Detect defects early
● Fix the cause of defects
14

15. The value
● Provide high quality software
● Fixed release schedule
● Provide pre-release versions as soon as
possible
● Support new platforms
● Add new features
● Add new tools
15

16. Quality & Open-Source
● All AdaCore's products are Open-Source
● It's possible to achieve high quality
● AdaCore is an active member of the GCC
community
● We identify a reasonably stable version of GCC
● We run it through our QA process
● We contribute our changes back to the community
16

17. Lean testing strategy
● A fundamental piece toward achieving high
quality
● Goals:
● Detect defects as early as possible
● Adopt different testing tactics to minimize risk
● Keep high level of productivity
● Balance the machine load
17

18. How to detect defects?
1. Local testing
2. Mailserver
3. Check-in
5. Nightly builds
SVN
4. Continuous Builder
18

19. The mailserver system
● Email-based interface Less than 20 minutes!
● To test patches
● On a remote machine
● Incremental build
● Full testing even on exotic platforms
● Clean result mandatory before check-in
19

20. The continuous builder
● Every check-in triggers an automatic build
● Fast machines
➔ quick feedback
● Stop-the-line: should an error be reported,
immediate attention/action is required
● Limit the risk of nightly builds failures
➔ Delays in intermediate releases delivery
➔ More waste, less value
➔ Customer satisfaction decreases
20

21. Nightly builds
SVN
Packaging
Linux Windows Building
Testing
Red Hat 5 SuSe 9 XP Server Vista 21

22. Nightly builds
● For cross platforms
➔ We test a different version of the target OS
everyday
➔ Use of simulators
➔ More stability
➔ Better performances
● Heavy use of virtual machines
➔ Ease maintenance of obsolescent platforms
22

23. Lean production infrastructure
● Shippable software is produced every day
● Defects are detected early
● Allows to achieve high quality
● Allows to deliver intermediate releases quickly
23

24. Official releases
● Only 2 releases a year
➢ One major release (January/February)
➢ One corrective release (June/July)
● Customers can plan their own release cycle in
advance
● Intermediate releases are still available
➢ Provide quick fixes
➢ Allows to experiment with new features
24

25. Official releases
● Failed builds are relaunched
● Manual review of each test suite report
● Human sanity check of each package
● Cross toolchains are tested on real boards
➢ 48h to run the ACATS on some cross platforms!
25

26. The release schedule
6.2 branch 6.3 branch
6.2.2 Release 6.3.2 Release
6.2.1 Release 6.3.1 Release
6.2.0 Beta 6.3.0 Beta Development branch
26
2009 2010

27. New features and known problems
● New features are advertised to everybody on
our web site
● Known problems are listed on the customer
web interface
● Improve traceability
● Provide easy access to workarounds
● Raises interest in evolution of the technology
27

28. Visual management with GAIA
● Monitor scripts ● Everything is stored
● Monitor machines in a database
● Monitor test suites
● Human-readable
weather reports
● Submit analysis ● Use of the django
framework
28

29. GAIA: How does it work?
Test machines GAIA Server GAIA User's Interface
DB
29

30. GAIA: Test suite overview
30

31. GAIA: Weather reports
31

32. Lean Software Qualification
32

33. Critical software certification
● Aerospace
● ECSS-E-ST-40C/ECSS-Q-ST-80C
● Civil Avionics
● DO-178B/ED-12B
● Air Traffic Management
● DO-278B
● Security
● Common Criteria
● Railway
● EN-50126/EN-50128/EN-50129
● Others...
33

34. DO-178B/ED-12B
● “DO-178B, Software Considerations in Airborne
Systems and Equipment Certification”
● Last revision: December 1, 1992
● DO-178C is in the pipeline
● 5 Design Assurance Levels (DAL)
● A => Catastrophic
● B => Hazardous
● C => Major
● D => Minor
● E => No effect
34

35. What is in DO-178B?
● Process oriented:
● Planning
● Development
● Verification
● Configuration management
● Quality assurance
● Certification liaison
35

36. DO-178B: The Waterfall Model
Not very agile...
Source: Wikipedia 36

37. DO-178B: The V-Model
Not agile either...
Source: Wikipedia
37

38. Certification and Qualification
● Embedded software is certified
● A tool is qualified
● As a development tool
– Output is part of the airborne software
– Can introduce errors
– Certification-like process
● As a verification tool
– May fail to detect an error
– Lightweight process
– Tool Operational Requirements
– Requirements based testing
38

39. Qualification of verification tools
● A tool may be reused in different contexts
● Operational requirements may change
● Different part of the tool may be used in
different context
● The tool may evolve
Let's do agile qualification!
39

40. The qualification machine
● Based on FitNesse
● Centralize all qualification artifacts
● Ensure consistency between requirements, test
cases and test data
● Automatic generation of qualification
documents
It's now easy
to add new requirements
and associated tests!
40

41. The qualification machine
Let's see it in action!
41

42. References
● Lean Software Development, An Agile Toolkit by
Mary and Tom Poppendieck
● Implementing Lean Software Development by
Mary and Tom Poppendieck
● Lean Software Strategies, Proven Techniques
for Managers and developpers by and Peter
Middelton and Jim Sutton
● http://www2.toyota.co.jp/en/vision/production_s
ystem/
42