Agile Testing for Embedded and IoT Software Development

W16

Agile
Testing

10/5/16
15:00

Agile
Testing
for
Embedded
and
IoT

Software
Development

Presented
by:

Thomas
Stiehm

Coveros,
Inc.

Brought
to
you
by:

350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073

888-‐-‐-‐268-‐-‐-‐8770
·∙·∙
904-‐-‐-‐278-‐-‐-‐0524
-‐
info@techwell.com
-‐
http://www.starwest.techwell.com/

Thomas
Stiehm

Thomas
Stiehm
has
been
developing
applications
and
managing
software

development
teams
for
eighteen
years.
As
CTO
of
Coveros,
he
is
responsible
for
the

oversight
of
all
technical
projects
and
integrating
new
technologies
and
application

security
practices
into
software
development
projects.
Most
recently,
Thomas
has

been
focusing
on
how
to
incorporate
DevOps
best
practices
into
distributed
agile

development
projects
using
cloud-‐based
solutions
and
how
to
achieve
a
balance

between
team
productivity
and
cost
while
mitigating
project
risks.
Previously,
as
a

managing
architect
involved
in
agile
development
at
Digital
Focus,
Thomas
found

that
agile
is
the
only
development
methodology
that
makes
the
business
reality
of

constant
change
central
to
the
development
process.

9/21/16
1
© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 1
Agility. Security. Delivered.
Agile Tes)ng for Embedded
So3ware Development
About Coveros
•  Coveros builds security-criIcal applicaIons using
agile methods.
•  Coveros Services
•  Agile transformaIons
•  Agile development and tesIng
•  DevOps and conInuous integraIon
•  ApplicaIon security analysis
•  Agile & Security training
•  Government qualiﬁcaIons
•  DCAA approved rates and accounIng
•  TS facility clearance
Areas of Exper8se

9/21/16
2
What is embedded so3ware
•  Embedded soTware is soTware that controls a device
•  A device is a combinaIon of soTware, ﬁrmware and hardware that
make up a system
•  The system has hardware components that are either sensors or
manipulators
•  The soTware and ﬁrmware control the hardware components, using
the sensors to discover and the manipulators to carry out acIons
•  Examples include:
•  Medical devices
•  Self driving cars
•  Fitness wearables
•  HVAC systems
What is agile tes)ng
•  Agile tesIng is the the set of pracIces used to build quality in and
make sure that soTware created by the team is of high quality
•  The whole team is responsible for quality
•  Test automaIon is integrated into the ConInuous IntegraIon and
ConInuous Delivery pipeline
•  TesIng of new funcIonality is conducted during the same sprint or
iteraIon in which the funcIonality is developed
•  Unit tesIng is used as part of the quality assurance process
•  Acceptance tesIng is conducted using acceptance criteria created
directly from the requirements

9/21/16
3
User Acceptance Tes)ng (UAT)
•  UAT is the process of ge]ng stakeholders to write down what results
they want to see from the soTware in order to determine it works as
the requirements intended
•  Acceptance criteria are wriên before development of the soTware
begins
•  Acceptance criteria are a form of requirements that are used to let
the team know what key aspects of the requirements need to be met
in order for the stakeholders to feel that they can accept the soTware
increment
•  TesIng of acceptance criteria happens at the end of an increment
and gives the team feedback on how well they have created soTware
that the stakeholders expected.
Behavior Driven Development (BDD)
•  BDD is a test automaIon process for tesIng soTware based on the
requirements
•  The tests are wriên before the code is wriên or at least
independently of the code
•  For some teams BDD is used to automate UAT tests
•  Many BDD tools exist such as Cucumber and Behave
•  Most tools use the Gherkin language, which uses the following
format:
•  Given ini8al condi8on
•  When some ac8on or trigger
•  Then resul8ng ac8on

9/21/16
4
Test Driven Development (TDD)
•  TDD, also called Test First Development, is the process of using unit
tests to help design the code for an applicaIon
•  The developer will write a test of what they intend the code to do
and then write the code to make the test pass
•  Red, Green, Refactor cycle
•  This has a number of posiIve results:
•  Makes the code more testable
•  Makes the code more modular or self contained
•  Makes the code more maintainable
•  Gives the code a regression test suite
•  Makes the intent of the code easier to understand
Con)nuous Integra)on (CI)
•  CI is the process of integraIng and tesIng code upon check in
•  In CI, integraIon is the process of integraIng a developer’s code with
the code already in the repository
•  The goal of CI is to make sure the code can build and pass an iniIal
set of regression tests before other tesIng and quality assurance is
applied to a speciﬁc build
•  CI uses a build server such as Jenkins to do a clean build of the code
and run a suite of unit tests
•  If the build breaks, the people that checked in since the last
successful build are informed and they are expected to get the build
back to a working state

9/21/16
5
Con)nuous Delivery (CD)
•  CD is the process of taking a build through a build pipeline that
consists of a CI process followed by several quality gates that verify
the code through various quality checks including:
•  Automated funcIonal tests
•  StaIc code analysis
•  Automated non-funcIonal tests like performance and security
•  Automated deployment into progressively higher environments
•  It could include on demand deployment to a manual test environment
•  On demand deployment to a UAT environment
•  On demand deployment to a demo environment
•  ConInuous Deployment is automaIcally progressing to producIon
including automaIcally deploying into producIon
Value Stream Mapping
•  Value stream mapping is the process of determining how a product is
created in your environment from incepIon to deployment and use
in producIon
•  By creaIng a value stream map you can ﬁgure out how many steps
you have in your delivery process and how long each step takes
•  Once you have a map of your process you can start to opImize and
automate the diﬀerent steps to streamline your process
•  The goal is to reduce the Ime it takes to move requirements through
your process and get new features into producIon

9/21/16
6
Build Pipeline
•  The build pipeline is the part of your value stream that takes checked
in code and makes it into high quality, tested soTware that is either in
producIon or ready to be distributed to customers
•  CI/CD are a subset of your build pipeline
•  AutomaIon, including test automaIon, is the only way to make a
build pipeline work in a fast and eﬃcient manner
•  Many of the quality gates in a build pipeline are tesIng focused and
require strong tesIng acumen in order to help deliver high quality
soTware
Version Control
•  Version control is the process of managing the code over Ime by
allowing a project to keep track of the state of the code as it
progresses
•  All code, including tes6ng code and test automa6on code needs to
be checked into version control
•  ApplicaIon code and tesIng code should be versioned together since
the test code acts on the applicaIon and changes over Ime to keep
in sync with changes to the applicaIon code
•  Version control tools include:
•  Subversion
•  Git
•  Others

9/21/16
7
Configura)on Management
•  ConfiguraIon Management are the tools and process used to
determine how soTware is setup in any given environment and how
to account for all of the different files needed to make the soTware
work
•  This includes binaries, configuraIon files and environment se]ngs
•  Most build pipelines have a component that manages configuraIon
elements for each deployment so that the team can know exactly
what is deployed and tested for every environment
Tes)ng So3ware vs. Hardware vs. System
•  With embedded systems you are tesIng three things:
•  SoTware – The code that manages the system
•  Hardware – The mechanical and/or electrical parts of the system
•  System – The combinaIon of hardware and soTware
•  SomeImes firmware is tested like soTware, someImes it is tested
like hardware (depending on how firm it is)
•  IsolaIng what you are tesIng can help you create test automaIon
that covers a much of the product as possible
•  Some amount of manual tesIng might be unavoidable

9/21/16
8
So3ware Tes)ng
•  During soTware tesIng you are tesIng only the soTware part of your
system
•  This means that all input (sensor data) should be supplied by a known
source, if possible by recording the data and playing it back, or
simulaIng it
•  Output can be ignored or simulated
•  Record and playback, or simulaIon can be less expensive and easier
to setup
•  You should be focused on tesIng the business logic of the soTware,
i.e. the calculaIons and determinaIons that soTware makes
•  You can also focus on tesIng the user interface or interacIve
elements of the system
Hardware Tes)ng
•  Hardware tesIng can be agile as well
•  Hardware can have a diﬀerent cadence than soTware
•  Agile hardware tesIng is about automaIng as much of the tesIng
process as possible
•  SoTware tools, both homegrown and third party, can be used to
automate hardware tesIng
•  Using the system soTware the controls that hardware to test the
hardware is acceptable but keep the focus on the hardware

9/21/16
9
System Tes)ng
•  System tesIng is the combinaIon of the soTware and hardware
working together
•  Depending on the nature of your embedded system, this can be hard
to automated or automate completely
•  Depending on the nature of your embedded system there may be a
good deal of tooling that can help you test the system or there may
be nothing at all
•  TesIng the system is criIcal because you don’t know how the
soTware and hardware will interact unIl you actually have them
interact
Test Automa)on vs. Manual Tes)ng
•  Agile tesIng focuses on test automaIon and in some types of
applicaIons it is possible to automate nearly all of the tesIng
•  With the focus on test automaton it is easy to think that manually
tesIng has no place in agile but that isn’t true
•  Manual tesIng is oTen a precursor to funcIonal test automaIon
•  Manual tesIng can be cheaper or more eﬀecIve for some features of
an embedded system
•  Exploratory tesIng can exercise the system in a way that test
automaIon will not and ﬁnd deeper issues

9/21/16
10
How much can you automate?
•  Look at this from a number of points of view:
•  What can you afford
•  What do you have the skills or abiliIes to automate
•  What is possible
•  What is the most beneficial
•  Finding a balance between test automaIon and manual tesIng will
be a challenge that all projects face
•  The answer will change over Ime as you learn and as your code
improves
Coding Standards (for tests too)
•  Using coding standards means:
•  Making the coding standards
•  Enforcing coding standards using code reviews
•  It doesn’t maêr what specific coding standards you use so much as
you have them and the team agrees to them
•  If possible, use an exisIng coding standard for your technology
•  Make coding standards compliance part of your code review
•  Beêr yet, use a code formaêr to apply the standard during code
commit
•  Apply coding standards to your tests as well

9/21/16
11
Code Inspec)ons
•  Code inspecIons include a number of things:
•  Code reviews
•  StaIc code analysis
•  Secure code reviews
•  Format and style reviews
•  Automate what you can, format and style are easy to automate
•  StaIc code analysis can be a good input into code reviews
•  Code review tools can help, things like Crucible and ReviewBoard
•  A secure code reviews is different than a code review and is a
different skill set, you should master both
Unit Tes)ng
•  Unit tests are developer tests that are used to make sure that the
code that they write behaves as the developer intended
•  Unit tests are wriên in the language and technology planorm of the
applicaIon code and oTen use a framework like Unity (see h^p://
www.throwtheswitch.org/unity/)
•  Unit tests might be run on the dev planorm or the target planorm,
this oTen depends on how the tooling works
•  Sample code:
void test_FuncIonWhichReturnsLocalVariable_ShouldReturnTheCurrentCounterValueAgain(void)
{
TEST_ASSERT_EQUAL(0, FindFuncIon_WhichIsBroken(78));
TEST_ASSERT_EQUAL_HEX(0x5a5a, FuncIonWhichReturnsLocalVariable());
}

9/21/16
12
BDD Sample Code
Scenario: Subtract two numbers
Given I have a calculator on
When I enter ”25" into the calculator
And I enter ”10" into the calculator
And I press subtract
Then the result should be "15" on the screen

Scenario: Add two numbers
Given I have a calculator on
When I enter ”25" into the calculator
And I enter ”45" into the calculator
And I press add
Then the result should be ”70" on the screen
Mock Objects
•  Mock objects are replacements for the dependencies of the code
under test
•  They are used to make it easier to test the code by responding the
same way all the Ime
•  There are a number of good mock object frameworks including
CMock (see h^p://www.throwtheswitch.org/cmock/)

9/21/16
13
Test Fixtures
•  Test fixtures are the code that creates a fixed state that is used to
baseline tests so that the test environment is the same each Ime the
tests are run
•  Test frameworks do this by exposing methods or funcIons that allow
for the configuraIon of dependent objects or for the creaIon of
mock objects
•  For a data related test a test fixture might load a specific set of data
into the database
•  Most unit tesIng frameworks and many test automaIon tools have
test fixtures build into them
Working with Simulators
•  A simulator is soTware or hardware used for tesIng that has the
same interface as the applicaIon or system it is replacing
•  Simulators provide a consistent interface for tesIng, much like test
fixtures or mock objects
•  Simulators are more full featured and can react to input, in more
varied ways that other tesIng tools
•  The idea is that a simulator will react more like the real component in
the real world
•  Simulators can be expensive to build and maintain but there may be
no other opIon

9/21/16
14
Tes)ng services for embedded so3ware
•  Many Internet of Things (IoT) devices use services
•  Services are Internet accessible bits of funcIonality that do things
like:
•  Collect data
•  Calculate results
•  Coordinate resources
•  TesIng these services can be easier to automate then the devices
themselves
•  Scale can be an issue for services
•  Network availability or reliability can be an issue for the devices
•  Performance tesIng is oTen important for tesIng services
What to take away from the presenta)on
•  Test automaIon is the key to agile tesIng
•  Do test automaIon planning while you do test planning
•  Create test suites that test your soTware separately from your
hardware and another suite that tests the system as a whole

9/21/16
15
Ques)ons?
•  Tom SIehm
•  tom.sIehm@coveros.com
•  @thomassIehm

Agile Testing for Embedded and IoT Software Development

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (12)

Similar to Agile Testing for Embedded and IoT Software Development

Similar to Agile Testing for Embedded and IoT Software Development (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Agile Testing for Embedded and IoT Software Development