In this presentation, delivered at Embedded World 2015, we will discuss and compare the ability of unit testing, flow analysis, and static analysis to address software quality, safety, and security issues for embedded systems. In addition to a deep and methodical exploration of how each technology addresses these concerns, we will discuss real-world feedback from organizations that have experience using the technologies in the context of a complete development cycle. The presentation will include the application of unit testing, flow analysis, and static analysis to sample software embedded on an IAR board.

1. Paraso&
Proprietary
and
Confiden1al
1
2/26/15
A
Comparison
of
Three
Bug-­‐finding
Techniques
and
Their
Rela:ve
Effec:veness
Mark
Lambert
-­‐
Paraso&
Corp

2. Paraso&
Proprietary
and
Confiden1al
2
Paraso&
Proprietary
and
Confiden1al
2
Agenda
§ Overview
and
Comparison
1. PaDern-­‐based
Analysis
2. Flow
Analysis
3. Unit
Tes1ng
§ Applica1on
and
Demonstra1on
§ Desktop
analysis/tes1ng
vs.
on
target
§ Using
Applica1on
Monitoring
to
uncover
run1me
problems
§ Combining
Coverage

3. Paraso&
Proprietary
and
Confiden1al
3
Paraso&
Proprietary
and
Confiden1al
3
Sta1c
Code
Analysis
Pa?ern-­‐Based
Sta:c
Analysis
Preven1on
technique
Analyzes
code
structure
(parse
tree)
to
apply
best
prac1ces
Flow-­‐Based
Analysis
Detec1on
technique
Analyzes
code
flow
to
determine
“dangerous
paths”
Metric
Threshold
Analysis
Advisory
technique
Finds
complex/
hard-­‐to-­‐test
code
prone
to
errors

4. Paraso&
Proprietary
and
Confiden1al
4
Paraso&
Proprietary
and
Confiden1al
4
Code
Analysis
§ Well
understood
o&en
under
valued
§ Define
the
goal
of
the
analysis
and
the
Policy
for
compliance
§ Policy
defines
reduc1on
of
business
risk
not
pursuit
of
perfec1on
§ E.g.
Security
(CWE,
CERT,
…)
,
Safety
Cri1cal
(ISO26262,
DO178B,
…)
§ Start
small
to
promote
adop1on
and
monitor
for
areas
of
improvement
§ Apply
con1nuously
through
the
SDLC
§ Balance
desktop
interac1ve
feedback
(pre-­‐check-­‐in)
with
server-­‐side
depth
of
analysis
(post-­‐check-­‐in)

5. Paraso&
Proprietary
and
Confiden1al
5
Paraso&
Proprietary
and
Confiden1al
5
Peer
Code
Reviews
§ Highly
valuable
in
finding
REAL
bugs;
Algorithms/Design
§ Use
carefully
§ Only
apply
a&er
Sta1c
Code
Analysis
§ Only
apply
where
there
is
RISK
Image: http://www.jasonawesome.com/2010/06/01/executing-a-php-code-review/

6. Paraso&
Proprietary
and
Confiden1al
6
Paraso&
Proprietary
and
Confiden1al
6
Unit
Tes1ng
§ Unit
Tes1ng
§ Code
focused
valida1on
§ Test
components
of
the
system
in
isola1on
(stubs)
§ Code
needs
to
be
built
to
be
testable
§ Host-­‐based
and
on-­‐target
§ Test
before
hardware
available
§ Where
is
the
ROI?
§ Did
we
design
it
properly
§ Does
it
func1on
correctly?
§ Have
we
mi1gated
the
business
risk?
§ How
much
is
enough?
§ Code
Coverage
+
Peer
Review

7. Paraso&
Proprietary
and
Confiden1al
7
Paraso&
Proprietary
and
Confiden1al
7
Explora1ve
Tes1ng
§ Ad-­‐hock/Unstructured
Tes1ng
of
func1onal
areas
§ Important
part
of
Agile
QA/
feedback
process
§ Requires
traceability
to
user-­‐stories
and
code
§ Should
be
‘reinforced’
with
automated
tests

8. Paraso&
Proprietary
and
Confiden1al
8
Paraso&
Proprietary
and
Confiden1al
8
Applica1on
and
Demonstra1on
§ Code
Analysis
from
within
IDE
§ PaDern-­‐Based
Analysis
§ Use
Flow
Analysis
to
find
poten1al
memory
issues
§ Dynamic
Analysis
§ Using
Applica1on
Monitoring
to
uncover
run1me
problems
§ Using
Coverage
to
determine
what
was
tested
§ Using
Unit
Tests
to
increase
coverage
§ Combining
Coverage
from
Manual
and
Unit
Tes1ng

9. Paraso&
Proprietary
and
Confiden1al
9
Paraso&
Proprietary
and
Confiden1al
9
Host-­‐based
and
On-­‐Target
Unit
Tes1ng
Host
Development
Environment
Simulator
or
Target
Device
Host
Based
Flow
Create/Extend
Tests
Execute
Tests
(Op1onal)
Review
results
and
coverage
Cross
compile
test
executable
Execute
Tests

10. Paraso&
Proprietary
and
Confiden1al
10
Paraso&
Proprietary
and
Confiden1al
10
Conclusions
1. Capture
Business
Expecta1ons
in
a
Policy
2. Apply
early
and
con1nuously
3. Use
a
workflow
for
remedia1on
4. Leverage
both
desktop
+
server
based
analysis
5. Translate
to
Business
Impact
and
Monitor
for
improvements

11. Paraso&
Proprietary
and
Confiden1al
11
2/26/15
Thank
you
Mark
Lambert,
Rich
Newman,
Miroslaw
Zielinski,
Adam
Trujillo
Mark.lambert@paraso&.com