Yannick Moy's presentation on the Hi-Lite project at the ERTS 2012 event in Toulouse France. The paper "Integrating Formal Program Verification with Testing" can be found at http://www.erts2012.org/Site/0P2RUC89/7A-1.pdf
5. Cost of testing
• Cost of testing greater than cost of development
• 10% increase each year for avionics software (Boeing META Project)
• Uneven repartition:
20% 80% of effort!
80%
• Uneven quality: 80% of errors traced to 20% of code
(NASA Software Safety Guidebook)
• Need to reduce and focus the cost of testing
6. DO-178C: formal methods can replace testing
Formal methods […] might be the
primary source of evidence for
the satisfaction of many of the
objectives concerned with
development and verification.
2011: Formal Methods Supplement (DO-333)
7. Myths of formal methods
• Myth 4: Formal methods require highly trained mathematicians
• Myth 5: Formal methods increase the cost of development
• Myth 6: Formal methods are unacceptable to users
• Myth 7: Formal methods are not used on real, large-scale software
(Anthony Hall, Praxis Systems, 1990)
8. Practice of formal methods
Since 2001, Airbus has been
integrating several tool supported
formal verification techniques into
the development process of
avionics software products.
2009: Formal Verification of Avionics Software Products
(Souyris, Wiels, Delmas, Delseny)
9. Cost of verification
20% 80% of 20% 80% of
80% testing effort 80% formal effort
Hi-Lite goal: using formal verification first, then testing…
4%
16%
testing
80%
formal
… to reduce and focus the cost of verification
14. Testing vs. Formal Verification
prove pre of Q
use Q code
assume post of Q
cover P constructs P calls Q
P calls Q
P P
Q Q
assume pre of Q
actual body of Q
prove post of Q
or stub…
local exhaustivity argument: global soundness argument:
each function covered P all functions proved
enough behaviors all assumptions justified
explored
Q
R
15. Combining tests and proofs
P is tested
P calls Q
How so we justify
assumptions made
P during proof?
Q
Q calls P Q is proved
verification combining tests and proofs should be
AT LEAST AS GOOD AS
verification based on tests only
16. Caution: contracts are not only pre/post!
strong typing parameters
not aliased
)…
parameters
initialized
data dependences
17. Combination 1: tested calls proved
P is tested
P calls Q
during testing:
check that P
precondition of Q Q
is respected Q is proved
assumption for proof:
precondition of Q
is respected
18. Combination 2: proved calls tested
P is tested
during testing:
check that P
postcondition of P Q
is respected Q calls P Q is proved
assumption for proof:
postcondition of P
is respected
19. Testing + Formal Verification
tested
P proved
Q
R
proved
local exhaustivity argument: global soundness argument:
- test: function covered - proof: assumptions proved
- proof: by nature of proof - test: assumptions tested
Testing must check additional properties
Done by compiler instrumentation
20. GNAT toolsuite
executable
GNAT GNATtest
compiler unit testing aggregated
verification
results
GNATprove
unit proof
22. Airbus 5 “must-have” of formal methods
• Soundness
• Applicability to the code
• Usability by normal engineers on normal computers
• Improve on classical methods
current work
• Certifiability
23. Benefits of openness
.org
• announcements • public: • all code
• meeting slides meeting minutes • dev docs
• articles / docs technical work • user docs
69 members
• private:
management
partner code
external collaborations with industry and academia