The document discusses testing embedded software and highlights several challenges and opportunities. It notes that while embedded systems are diverse, there are now many open hardware platforms like Arduino, TinyOS, Android and ROS that enable more academic research. However, properties to test for are difficult to define as embedded systems interact continuously with the physical world. The document advocates using tools like static analysis, contracts and model checking to test for stack overflow, interface correctness, memory safety and application-level distributed properties. Many bugs were found this way in existing systems like TinyOS.

1. Testing EmbeddedSoftware John Regehr University of Utah

2. “Over 15 billion ARM based chips shipped to date” [ARM web site, 2011] “The microcontroller market is forecast to reach over $16 billion worldwide in 2011” [Microcontroller Market Tracker, 2011] 2

3. 3

4. Diverse! 4

5. Diverse! I have 6 pins and 32 bytes of RAM 5

6. Diverse! 6

7. Diverse! 7

8. Diverse! I am quad core @ 1.5 GHz and have a GPU 8

9. Usually there are multiple processors On-chip networks In-device networks Distributed systems Resource constraints are… Severe – to minimize unit cost Hard – failure if system runs out of… Time RAM – stack or heap Energy 9

10. Continuously interact with the world through I/O devices May be little abstraction of HW Probably using both interrupt handlers and threads Often there are fault tolerance and security requirements 10

11. Sensor network -> 103–105 LOC Modern airplane -> 106–107 LOC Hybrid vehicle -> 107–108 LOC How do we get these right? Mostly testing 11

12. Software on many individual processors is small Permits aggressive analysis and testing Constrained domain simplifies testing Embedded systems are (by definition) special-purpose devices 12

13. The “Real System Problem” Many interesting embedded codes are proprietary Necessary tools may be expensive or nonexistent Compilers, debuggers, simulators May not be able to run it in the lab Often lacks specifications and oracles 13

14. Consequently, academic embedded work may be… Forced to use small, contrived examples Out of tune with industry 14

15. Consequently, academic embedded work may be… Forced to use small, contrived examples Out of tune with industry 15 Solution: Ubiquitous open embedded platforms

16. Arduino Arduino Uno: 8-bit AVR processor @ 16 MHz 2 KB RAM ~$30 Emphasis is on interfacing 16

17. Arduino Nice IDE + libraries + C/C++ Minimal abstraction of the embedded processor 18 new books in 2011 17

19. Very few Arduino tool papers exist

20. This is a big opportunityArduino Nice IDE + libraries + C/C++ Minimal abstraction of the embedded processor 18 new books in 2011 18

21. TinyOS OS and middleware support for sensor networks Sensing Collection and dissemination Localization Applications are in nesC, a C dialect 19

22. TinyOS “Motes” based on a variety of MCUs Cost $50 – $200 Good simulators exist There are a few books 20

24. Many open problems“Motes” based on a variety of MCUs Cost $50 – $200 Good simulators exist There are a few books 21

25. Android OS + middleware for smart phones / tablets ARM based hardware running Linux Much less constrained than motes and Arduino 22

26. Android Application code in Java Great tools Tons of books 23

28. Most are very recent

29. This is not a scary platformApplication code in Java Great tools Tons of books 24

30. ROS – Robot Operating System Linux-based infrastructure for programming robots Primary abstraction is graph of communicating processes Local and distributed 25

32. Plenty of other open embedded platforms exist FreeRTOS Contiki Pacemaker Challenge Etc. Embarrassment of riches Still, huge room for improvement Where’s the open automobile? 27

33. So, let’s test some embedded software But what are we testing for? 28

34. Properties / Oracles Temporal safety Deadlines Or just responsiveness Memory safety Contracts / assertions Reference implementation 29

35. Worst-Case Execution Time What is the upper bound on execution time for a piece of code? We care because the world has deadlines Static analysis of WCET is extremely difficult if there is… A cache Preemption An aggressive processor 30

36. True WCET Number of executions Execution time Conservative WCET Longest observed ET #2 Longest observed ET #1 31

37. 32 + = printf() pthread_attr_setstacksize (&attr, &mystacksize); HANDLE WINAPI CreateThread( LPSECURITY_ATTRIBUTES lpThreadAttributes, SIZE_T dwStackSize, LPTHREAD_START_ROUTINE lpStartAddress, LPVOID lpParameter, DWORD dwCreationFlags, LPDWORD lpThreadId );

38. Stack Overflow in TinyOS 33 4 KB

39. Stack Overflow in TinyOS 34 main() 4 KB

40. Stack Overflow in TinyOS 35 main() irq 0 4 KB

41. Stack Overflow in TinyOS 36 main() irq 0 4 KB irq 1

42. Stack Overflow in TinyOS 37 main() irq 0 4 KB irq 1

43. Stack Overflow in TinyOS 38 Not the same thing as buffer overflow! Type safe language doesn’t solve this problem main() irq 0 4 KB irq 1

44. Eliminating Stack Overflow Testing is hard Need to drive code to its WC stack depth Interrupt coincidences are rare Approach: Static analysis of compiled code Can’t estimate stack depth of source 39

45. Estimate WC stack depth of each sequential flow, handling Indirect branches Recursion Loads into the stack pointer Compute “interrupt preemption graph” Find longest cycle in this graph 40

46. 41 in r24, 0x3f ; r24 <- CPU status register cli ; disable interrupts adc r24, r24 ; carry bit <- prev interrupt status eor r24, r24 ; r24 <- 0 adc r24, r24 ; r24 <- carry bit mov r18, r24 ; r18 <- r24 ... critical section ... and r18, r18 ; test r18 for zero breq .+2 ; if zero, skip next instruction sei ; enable interrupts ret ; return from function

47. Stack analysis tool deployed in the TinyOS distribution Results are typically much larger than worst observed stack depths But, we validated its results by randomly firing interrupts 42

48. Need… more… oracles… 43

49. TinyOS applications are built using components Interface requirements documented but not checked Interface misuse often silent 44

50. We augmented nesC with contracts Dynamic checking reasonable efficient Found some long-standing bugs 45

51. nesC is not type safe Memory safety bugs in TinyOS are difficult We ported an existing safe C dialect Found some otherwise-impossible bugs Main problem was getting overhead under control Whole-program optimization 46

52. 47 Code size 35% 13% -11%

53. 48 Increasing Availability Normal TinyOS: 0% average availability Array Out-of-bounds Normal TinyOS Safe TinyOS: 95% average availability Array Out-of-bounds Rebuild Soft state Safe TinyOS Reboot

54. What about application-level sensornet properties? All the interesting ones are distributed We adapted TOSSIM, a non-cycle-accurate simulator, to be… A random tester A depth-bounded model checker Oracles: Type safety checks Application-level properties 49

55. Application-Level Properties Eventually… Each send buffer is unlocked No cycles in the routing tree All nodes become part of the collection tree All nodes have consistent values 6 out of 8 of these properties require global knowledge 50

56. Found 12 previously unknown bugs in TinyOS 2.0 10 safety, 2 liveness Random testing outperformed depth-bounded model checking Even after a lot of work on POR But required work to shorten long error traces 51

57. Conclusions Open embedded platforms exist Some have steep learning curves Finding oracles is hard Generating valid input is hard Embedded systems are fun and important and rewarding 52