The next talk in our series from the recent Open-DO Conference is from Neil White,Principal Engineer with Altran Praxis. His talk provides an overview of the formal methods used on the iFACTS project. iFACTS is delivering increased Air Traffic Control capability to the UK.
25. The Challenge of Test Design How many potential tests for this fragment?
26.
27.
28.
29.
30.
Notes de l'éditeur
Document reference: S.P9999.99.99, issue 1.0 Page
Document reference: S.P9999.99.99, issue 1.0 Page Good morning If you were expecting Rod Chapman, and are thinking “gosh Rod’s let himself go”, then the key news is that I’m not Rod. I’m Neil White. I work as a principal engineer for Altran Praxis. More significantly, I’m the engineering manager for several projects, including the large formal methods project I’m talking about today. I also run the software practice. For those of you who don’t yet know, Altran Praxis was recently formed by the merger of Praxis and SC2. Altran has been Praxis’ parent for over 10 years, and SC2 was a sister company based in the south of France. The two companies have been working closely for a while, and bring different technologies together but with a very similar ethos. It’s a great mix. Public marriage of existing private relationship. In today’s context, Praxis brings a long history of formal methods from the UK base, and a long history of Agile methods from the French base.
Document reference: S.P9999.99.99, issue 1.0 Page In 30 mins I can deal with one topic in depth, or go for a rapid canter across a wider spectrum. I’m going for the latter, not least to ward off the snooze effect of that food! I’m going to talk generally about a project called iFACTS to set the context, and then I’m going to talk about formal methods through the project lifecycle and give a personal opinion of the pro’s and con’s. I ought to fess up: I’m a long term formalist with a passion for mathematics and rigor, but I also like success, and delivery, and especially profit, so formal methods have to work for me in industry.
Document reference: S.P9999.99.99, issue 1.0 Page iFACTS is an ATC system being procured by NATS – the UK air traffic provider. NATS are world leaders in ATC innovation. Partly through necessity; UK ATC is very very busy because of the dense population and our geographic position below transatlantic air traffic. I can only scratch the surface of the project. There is loads more detail on the NATS website
Document reference: SPARK User Group, issue 1.1 Page UK airspace is divided into sectors.
Document reference: SPARK User Group, issue 1.1 Page Each sector has a team of 3 (currently) looking after the traffic in that sector. The planner accepts aircraft into the sector, and arranges for them to be handed off to the next sector. Arrangements include height, speed, heading, etc. He talks to other planning controllers. Ne deals with the boundary. The tactical is the controller in comms with the aircraft. She basically gets them from the in to the out of this sector. She deals with the interier. The assistant prints paper strips and generally helps out. Increased capacity comes through more sectors and thus more controllers. Except we have hit the limit. The hand-over burden now outweighs getting a new sector. We need tools to help.
Document reference: S.P9999.99.99, issue 1.0 Page iFACTS will allow greater capacity in the existing sectors through the provision of new tools.
Document reference: S.P9999.99.99, issue 1.0 Page We replace the paper flight strips with electronic ones. Not a great computer challenge. (Big usage challenge though.) An enabler. Enabler because once we have all the data that’s currently on paper into the system, we can do things with it. We create a trajectory through space and time for each flight. We add uncertainty as a cone along the trajectory. The closer to “now”, the more certain you can be. Some maneuvers increase or decrease uncertainty. We can then compare every trajectory with every other trajectory to identify possible conflicts up to 15 mins in advance. Currently controllers work with a much shorter look-ahead of only a few minutes. This also gives the controllers a “what if” capability so that we don’t maneuver aircraft into annoying places in the first place! So as you can see, we augmenting – not replacing – the current system. “ Biggest advance in ATC since the introduction of Radar”
Page This is an example of part of the HMI. Each symbol is a pair of aircraft. This is time to the closest approach between a pair of aircraft. This is the distance at closest approach. Note this says nothing about current gap. Symbols tell you the attitude of the approach. Colors tell you something about severity. White is a deviation – an aircraft not doing what it’s told. Document reference: SPARK User Group, issue 1.1
Document reference: S.P9999.99.99, issue 1.0 Page So lets’ start looking at formal methods.
Document reference: S.P9999.99.99, issue 1.0 Page The specification is large, and split into a couple of technologies. The dominant part is a formal Z specification. There is some inherited mathematics defining algorithms. We could re-write in Z, but it costs, and can only add defects. We don’t! It’s already unambiguous. We just tie functions to Z. The HMI specification is in state tables. A small amount – eg stating non-functional requirements on performance or resource usage is in English commentary.
Random bit of Z. Not expected to read this! English description and (more detailed) mathematical description. This is a schema. These are variables with types. This is a mathematical relationship between the variables. We can generate the document with and without the mathematics for distinct readerships. The English needs to work in both. The maths has more detail. Extends, doesn’t contradict, the English. 4250 pages. All customer reviewed. Everything flows from this: design, code, test, everything.
Document reference: S.P9999.99.99, issue 1.0 Page How do we get a body of Z engineers? Reading and writing are different skills. Teaching reading is easy, and we have a lot of data to support that. People are up to speed fast. We can teach almost anyone who si not scared of basic maths. Teaching writing is harder. We pre-select harder, and the learning curve is longer and steeper. Some people don’t make it. Not a surprise – not everyone can do anything. There are people who will never write good code, or write good tests.
Document reference: S.P9999.99.99, issue 1.0 Page Tools support is a key issues. We use Word. We don’t love it! However, when teaching people Z you really don’t want to simultaneously teach them other tools too. Fight selective battles! The template includes a Z font, an ability to kick off the FuZZ type checker, and the ability to launch a graphical analysis too that shows you the linkage and structure of the specification.
Document reference: S.P9999.99.99, issue 1.0 Page Word has made for an easier environment for new users. But it retains all the usual problems of large word documents. In particular, when developing a branch the merge can be tortuous. Binary word files means that you have little option but to use Word to do the merge. Going forward, a Z-aware merge tool for the underlying OO XML might be one option to help merge.
Document reference: S.P9999.99.99, issue 1.0 Page The HMI spec is a simple state machine Describe… We could clearly draw this, but we get more material on the page in tables. Leads to a clear mapping to code if we want. Under the look-and-feel, we tie operations into the Z. So a button press is the trigger for a Z operation. A text field is the output from a Z operation.
Document reference: S.P9999.99.99, issue 1.0 Page Beauty of this is the sheer simplicity.
Document reference: S.P9999.99.99, issue 1.0 Page
Document reference: S.P9999.99.99, issue 1.0 Page I don’t want to shock, but I’m actually going to pass up the opportunity to extol the virtues of SPARK. I think this audience is pretty SPARK-aware, but please grab me later if you want to talk about the language. In summary, it’s an annotated subset of Ada which is designed for people who want their programs to be safe, secure, or frankly just correct. We have 150KLSLOC, all of which has a proof of the absence of any possible Ada exception.
Document reference: S.P9999.99.99, issue 1.0 Page Again, we have trained a lot of people with a diverse background. All our SPARK coders read Z and do proof. Note that we are not doing a correctness proof. It’s not cost-effective for this project. The level of integrity doesn’t warrant the work. (Remember my comment on profit!)
Document reference: S.P9999.99.99, issue 1.0 Page In comparison to Z, the SPARK toolset is mature and excellent. And again, please see me for details or your AdaCore rep for a very reasonable quotation!
Document reference: S.P9999.99.99, issue 1.0 Page
Our testing is driven by the Z. We require specification and code coverage. We devise possible conditions by analysis of the mathematics. Partition analysis and equivalence classes. We write these in a Z-like notation.
Keeping test under control is – however – a big challenge. Just because you can devise a test case, doesn’t mean that you can afford to generate the test, or that it’s a good test. How many in this small example?
Document reference: S.P9999.99.99, issue 1.0 Page Far too many! We triage out the low-value test conditions. Drop duplicates or contradictions. Then by carefully crafting scripts we can knock off a large number of conditions in one go. This is an activity that takes skill and domain knowledge. If you take the easy option “we will test that” then out test program will grow to commercially un-viable proportions. Too long, and too costly. But of course, we do need to be sure of the verification – we need a safety argument at the end of the day! In summary: Maths tells you all possible tests, but so many that we could test for years. Trick is to pick high value tests
Document reference: S.P9999.99.99, issue 1.0 Page We use an independent implementation to help with the detailed trajectory algorithm testing. We use the “reference model” to generate the expected outcomes. Diverse implementation notation and programmer, so risk of common failure. (Outside of spec error.)
Document reference: S.P9999.99.99, issue 1.0 Page Interestingly, the reference model has proven very accurate. (Although not not eg fast enough for real use.) Need to be careful not to draw too many conclusions from a small study. Worthy of further evaluation. We could throw loads of tests at this automatically!
Document reference: S.P9999.99.99, issue 1.0 Page Conclusion Formal methods – being unambiguous – help us throughout the life cycle. We have a project of over 100 engineers, and have had precisely zero scaling problems in the technologies. In particular, the oft-cited problem reasons of “training” and “learning” doesn’t hold true. (Which of course argues that we can train any number of Ada programmers too.) The achilles heel – if there is one – is the tool support. We don’t have enough tool support, and enough integrated tooling. There are exceptions – like the Examiner – but for Z etc. we need more.