1.
AGILE METHODS AND SAFETY-CRITICAL SOFTWARE
Are They Compatible ?
Peter Gardner
Technical Consultant

2.
SILVER-ATENA’s Business
• Software development principally in the rail and
avionics sectors
• Consultancy services
• Provides skilled software engineering staff to work
on clients’ premises
• Full system development capability through other
partner companies in the same group
• Group offices in Malmesbury, Bangalore, Madrid
and Munich
Page 2

3.
SILVER-ATENA Agile Club
• Established August 2009
• Purpose : investigate the use of Agile
methods in safety-critical software
development
• “Client” is the Open-DO wider community
Page 3

4.
SILVER-ATENA Agile Club
• Average attendance 12 people, internal and
external
• Total attendance 32 : consultants, software
engineers, project managers, quality
department
• Videos, Powerpoint, discussion
Page 4

5.
What the Presentation is About
• (1) Can Agile methods be applied to
safety-critical software and the software still
be rigorously built and meet certification
criteria?
[compatibility].
• (2) What evidence is there for the benefits
of Agile methods, especially as regards
safety-critical software ?
[cost, quality, time etc]
Page 5

6.
What the Presentation is About
• For (1), this paper will look at, in
particular, DO-178B and EN 50128, for the
aviation and rail sectors respectively
(level A/B, SIL 3/4)
• For (2), this paper will summarise the
results of a websearch looking for evidence of
the benefits of Agile
• I will also look at some opinions from
people who work in the sector
Page 6

7.
Categories of Source Papers
• Papers on Agile methods and non-safety-
critical software (10)
• Papers on Agile methods and safety-critical
software (7)
Page 7

8.
Different Variants of Agile
• XP
• Adaptive Software Development
• Crystal (family)
• Dynamic Systems Development Method
• Scrum
• Agile Unified Process
• Feature-Driven Development
Page 8

9.
Features of Agile Methods
• Short release cycles
• Incremental requirements development
• Customer presence in project
• Continuous integration
• Customer determines functions in each release cycle
• Generally for small teams (up to about 12)
• More difficult to apply to larger teams
• Adapt for each project
• Multi-skilled team members (possibly)
• Any team member can change anything
Page 9

10.
Features of Agile Methods
• Pair-programming (XP)
• Test-driven development
• Retrospectives
• Larger teams should have shorter cycles (!)
• Refactoring
• Less emphasis on design and
documentation
• Pair-programming (XP)
• Self-organising teams
• Project lasts as long as customer gives
high-value requirements
• Departments as service providers
Page 10

11.
Yes, mostly
• Pass on 15 out of 20
• Maybe on 3 out of 20
• No, but not serious, for 1 out of 20 :
anyone can change anything
• But there is a real problem with one item :
reduced emphasis on design and documentation
Page 11

12.
Features of Agile Methods
• Short release cycles
• Incremental requirements development
• Customer presence in project
• Continuous integration
• Customer determines functions in each release cycle
• Generally for small teams (up to about 12)
• More difficult to apply to larger teams
• Adapt for each project
• Multi-skilled team members (possibly)
• Any team member can change anything
Page 12

13.
Features of Agile Methods
• Pair-programming (XP)
• Test-driven development
• Retrospectives
• Larger teams should have shorter cycles
• Refactoring
• Less emphasis on design and
documentation
• Change company structure (ideally)
(departmental to project structure)
• Self-organising teams
• Project lasts as long as customer gives
high-value requirements
• Departments as service providers
Page 13

14.
DO-178B Outputs(1)
• Plan for Software Aspects of Certification
• Software Development Plan
• Software Verification Plan
• Software Configuration Management Plan
• Software Quality Assurance Plan
• Software Requirements Standards
• Software Design Standards
• Software Code Standards
• Software Requirements Data
Page 14

15.
DO-178B Outputs(2)
• Software Design Description
• Source Code
• Executable Object Code
• Software Verification Cases and Procedures
• Software Verification Results
• Problem Reports
• Software Configuration Management Records
• Software Quality Assurance Records
• Software Accomplishment Summary
Page 15

16.
EN 50128 Outputs (part 1)
• System safety plan
• Software Quality Assurance Plan
• Software Configuration Management Plan
• Software Verification Plan
• Software Integration Test Plan
• Software /Hardware Integration Test Plan
• Software Validation Plan
• Software Maintenance Plan
• Data Preparation Plan
• Data Test Plan
Page 16

17.
EN 50128 Outputs (part 2)
• Software Requirements Specification
• System Safety Requirements Specification
• Software Requirements Test Specification
• Software Architecture Specification
• Software Design Specification
• Software Module Design Specification
• Software Module Test Specification
• Software Source Code
Page 17

18.
EN 50128 Outputs (part 3)
• Software Module Test Report
• Software Integration Test Report
• Software Requirements Verification Report
• Software Module Verification Report
• Software Architecture and Design Verification Report
• Software Source Code Verification Report
• Software/Hardware Integration Test Report
• Software Validation Report
• Software Assessment Report
• Software Change Records
Page 18

19.
Documents and Contents
• Its not just a case of producing documentation
• Much more importantly, one has to produce the
contents of the documents i.e software
development artefacts
• For example, architectural design,
low-level design, test cases etc
Page 19

20.
Traceability
• Both DO-178B and EN 50128 require
traceability through the lifecycle, yet :
• Agile methods never even consider
traceability
Page 20

21.
Independence of Roles
• DO-178B and EN 50128 also require (for higher
levels) some of the roles to be independent eg.
implementor and tester
• This constraint also has to be factored into Agile
methods
Page 21

22.
Views on Design,
Documentation, Traceability
• Stojanovic et. al. : “[According to] their proponents software code is the main
deliverable, while the role of system analysis, design and documentation in
software development and maintenance is de-emphasised and to some extent
ignored”.
• Chenu : “without any adaptation, XP’s set of by-the-book practices does not
provide the formalism and the proofs required for certification
inspections….documentation must be considered as part of the product”
• Chenu : “we’ve added the continuous traceability practice”
• ITEA Deliverable D.1 :“Agile development methods and practices as applied
for classical software development cannot identically be copied for embedded
real-time software development”.
• ITEA Deliverable D.1 : in large, complex systems “a predefined architecture is
needed…refactoring is not enough, a design is needed”.
• ITEA Deliverable D.1 : in safety-critical systems“emphasis is placed [on]
requirements traceability and thus extra documentation is required”.
Page 22

23.
Views on Design,
Documentation, Traceability
• ITEA Deliverable D.2.12 : “developers of mission-critical airborne software are
heavily constrained by the RTCA DO-178B regulations. These regulations impose
strict rules regarding traceability and documentation that make it extremely hard
to employ an iterative software development process” [did they really mean
this?]
• ITEA Deliverable D.2.12 : “the mission-critical nature of this software has lead
to stringent procedures and plans that could specifically exclude the use of Agile
methods”
• Wils and Van Baelen : “Agility potential is inherently lower for DO-178B
compliant projects”
• Thomas : “beware Agile methods…dangerous where…the system is safety-
critical” [says use Z and SPARK etc]
• Ambler : would be “leery of applying Agile modelling to life-critical systems”.
• Somerville : “critical system engineering needs Agile approaches to
development”
• Somerville : “XP includes a set of good practices that have the potential to
contribute to critical systems engineering” but “some of these need to change
and new practices need to be included in the XP process”
Page 23

24.
Views on Design,
Documentation, Traceability
• Van Schooenderwoert : “Agile doesn’t treat safety-critical software differently;
all is maximum quality”
• Turk et.al. : “formal specification, rigorous test coverage, and other formal
analysis and evaluation techniques included in software engineering approaches
provide better, but also more expensive, mechanisms to tackle the development
of safety- or business-critical software”
• Rakitin : “it seems to me that this is nothing more than an attempt to
legitimise hacker behaviour”
Page 24

25.
Empirical Evidence(1)
• We have seen some opinions
• Is there any actual data ?
Page 25

26.
Empirical Evidence(2)
• My web search found two papers (which
were themselves surveys) :
• Abrahamsson [7] found 36 papers with
some evidence
• ITEA (Information Technology for European
Advancement) project deliverable D.1
contained three studies
Page 26

27.
Empirical Evidence(3)
• Abrahamsson : online survey, 36 papers with
empirical evidence :
- a reduction in release time of 20% to 75%
- a pre-release reduction in defects of 40% to 65%
- a post-release reduction in defects of 30% to 40%
- a productivity improvement of 15% to 70%
Not known if any of these were safety-critical projects
Page 27

28.
Empirical Evidence(4)
• ITEA
(1) “has the adoption of Agile processes altered the quality of
applications?”
significantly better 36%, better 53%, unchanged 11%, somewhat worse 1%
(2) “has the adoption of Agile processes altered the cost of development?”
much less expensive 5%, less expensive 44%, unchanged 46%, more
expensive 5%
(3) “has the adoption of Agile processes altered the level of business
satisfaction with the software ?”
significantly better 26%, better 57%, unchanged 16%, somewhat worse
0%, much worse 1%.
(4) “what proportion of projects are appropriate for Agile processes?”
all 16%, most 50%, half 22%, some 7%, none 5%
• Only two out of seven specific projects reported in the literature were safety-
critical projects
Page 28

29.
Summary
• I have examined whether Agile methods
are compatible with the production of safety-
critical software
• I have presented evidence from the
literature of the benefits of Agile methods,
particularly as regards safety-critical software
Page 29

30.
Conclusions
• Agile methods, in general, need to be adapted for use in the
area of safety-critical software development
• Certification requirements mean that necessary
documentation must be incorporated into any
Agile process selected.
• Traceability and role independence must also be considered
• The logical extension of Agile methods to the safety-critical
world requires continuous certification and a colocated
certification authority
• Release cycles are likely to be longer in safety-critical projects but
there is no reason not to move towards the concept of frequent
release cycles with an onsite customer
• We need to gather more evidence, using reliable, standard metrics
• Silver-Atena’s Agile club would be willing to assist in this work
Page 30

31.
References (non-safety-critical)
[1] Modeling and Architectural Design in Agile Development Methodologies, Stojanovic, Dahanayake, Sol
http://www.emmsad.org/2003/Final%20Copy/27.pdf
[2] Agile in Embedded Software Development :
State-of the-Art Review in Literature and Practice ITEA Deliverable D.1
http://www.agile-itea.org/public/deliverables.php
[3] A Practical Guide to Seven Agile Methodologies, part 1, Rod Coffin and Derek Lane
http://www.devx.com/architect/Article/32761/1954
[4] A Practical Guide to Seven Agile Methodologies, part 2, Rod Coffin and Derek Lane
http://www.devx.com/architect/Article/32836/1954
[5] Empirical Findings in Agile Methods, Lindvall et. al.
http://www.cs.umd.edu/~mvz/pub/agile.pdf
[6] The Agile Unified Process, Scott Ambler
http://www.ambysoft.com/unifiedprocess/agileUP.html
[7] Pealing the Hype into Pieces : What do we Really Know About Agile in Research and Practice ?
Pekka Abrahamsson,VTT Technical Research Centre of Finland
http://www.agile-itea.org/public/papers/OLIO_abrahamsson.pdf
[8] Limitations of Agile Software Processes, Turk, France, Rump
http://www.agilealliance.org/system/article/file/1096/file.pdf
[9] The Cockburn Scale
http://en.wikipedia.org/wiki/Cockburn_Scale
[10] Manifesto Elicits Cynicism, Steven Rakitin Computer, December 2001
http://sunset.usc.edu/events/2002/arr/letters.pdf
Page 31

32.
References (safety-critical)
[11] Agility and Lean for Avionics, Emmanuel Chenu
http://manu40k.free.fr/AgilityAndLeanForAvionics1.pdf
[12] ITEA Deliverable D.2.12 Towards An Agile Avionics Process, Wils and Van Baelen
http://www.agile-itea.org/public/deliverables/ITEA-AGILE-D2.12_v1.0.pdf
[13] Agility in the Avionics World, Wils and Van Baelen
https://lirias.kuleuven.be/bitstream/123456789/133945/1/2006-Sofia1.pdf
[14] Critical Software : The Need For A Radical Solution, Martyn Thomas
http://www.thomas-associates.co.uk/Critical%20Software.pdf
[15] Get Ready for Agile Methods, With Care, Barry Boehm, IEEE Computer 2002
http://www2.umassd.edu/swpi/xp/papers/r1064.pdf
[16] Extreme Programming for Critical Systems, Ian Sommerville
http://www.cs.st-andrews.ac.uk/~ifs/Talks/XPForCritSys.pdf
[17] Safety-Critical Applications Built via Agile Discipline, Nancy van Schooenderwoert
http://www.boston-spin.org/slides/boston_spin_slides_2008_09.pdf
Page 32

33.
Thank you for your attention !