Applying Agile Test to Complex Embedded Systems

“Applying agile test to complex embedded systems”

© 2012 Intune Networks Ltd. Confidential 1

How hard can it be?


Distributed Ethernet optical switch – initial release

 Up to 10 nodes per ring
 80Gbps input and output ports per node
– ≡ 7.2TBps of fully meshed capacity

 12Gbit/s INEX Peak
Internet traffic Use in 2011.

 500K Lines of code

 Distributed Architecture
– 6 Architecturally distinct Subsystems
– 3 Primary Software Loads iVX 8000

* In practice 500Gbps due to collision detection overhead

Sometimes it felt more like..


The $1,000,000 Question

• How do we test
increasingly
complex systems in
increasingly shorter
time intervals?

• Compounded..

Integration led Extensive
R&D strategy Automation
& VTE


Integration Order

VTE
System
Application

Application

Hardware Abstraction Layer

Proto 1W HW

Device Drivers
Device Drivers Device Drivers

Hardware
Demo Hardware


Integration methodology

Backlog

NMRC SW ___________________
TCS HW___________________
NPC SW___________________
NPC HW___________________
DSSE HW___________________
OFS SW___________________
OFS HW___________________
Integration
SPIKE ___________________

Spike 1 Spike 3

Integration Spikes: Identified high risk (complex, impact) interfaces
R&D backlog and teams aligned to achieve spikes


Programme Map with SPIKES

Integration CR-1 Test
Strategy

Q2 09 Q3 09 Q4 09 Q1 10
Mar/Apr June/July Sept/Oct Dec/Jan

HA1 HA1 HA1 HA1
TCSOptics B DSSE CME
24/3 24/3 22/5 HA1 ??? LA1
NPC
21/6
iNX 8000
Shelf
OAM &BI Rollout

BI modelling three BI Modelled in Real
nodes with real Shelf with IPsubnet BI Robusteness Shelf
TCS Restarting Port All Cards Ring Testing

SU_A1 SU_A2 SU_A3 SU_A4

Ring Protected
Shelf Comm Shelf Level Ring IP
Services
Prototype IP SUbnet SUBNET
IP IP IP IP
SU_A1 SU_A2 SU_A3 SU_A4
CC, Topology, IPSUBNET

Marvell DX167 CC GFP
1 Devel Board 2 Connect
4 X Y 3 Switches
CC Physical CC CC CC
GigE GigE Layer (optics) SU_A2 SU_A3
SU_A1
Freescale Freescale CC Scheduler
8548 TCS 8548 TCS & Robustness
QNX BSP Management
BT Simulated Discovery ?
Maintenance Adj
BT BT
SU_A1 SU_A2

APR
Ring Power Cal Ring Level
Lambda Assign
I/P
Port 90/10
Delay
C.A.
80/20
EDF C.L. WSS VOA =
Assembly (within WSS) 2x2
95/5
O/P
Port
Ring
OLC
OLC OLC
A switch
95/5
C.C.
PD_CC
FOA
Mon
Port
1st stage
VOA = 
VOA = 
C.C.
Bringup SU_A4
VOA_CC C.C.
SFP VOA = 
(within WSS)
PD_O
SU_A2 SU_A3 Scenarios
OLC

DCM (40 km)
C.D.
PD_D1
C.Off. CIRC EDF
A
VOA_O
within
EDFA
OLC Section
OLC
PD_A
SOA

GFF
PD_D2 4 Star
Coupler
VOA_A
2nd stage
CC , EDFA/WSS, SU_A4
PD_D3 Arm uP
Carrier
Gain Adaption
Sense PD_D4
I2C
Ethernet MII
I2C
CS, SFP Control
C.C.
SFP
Simple
Test
Card
1200 Laser
JTAG

Alarms/
Interrupts
Ethernet MII
Physical Shelf
RS232 TCS Module L2 Switch

TCS-DSSE Interconnect Ring Data Path
Cont Mode, Physical Shelf PAT
Cont Mode
AURA iNX 8000 Loopback Simple OPR
Aura IF & Mem
Shelf Burst Mode
Simple SCheduler
PHYSICAL/DD

TCS-TXP
DATA PATH

HS Control Loops DSSE DP DP DP
Line Loopback HA2a SU_A1 SU_A2 SU_A3
TXP
HA2 NPC
TCS-NPC Cont Mode HA2a iNX 8000
Ethernet Services
Aura IF & Mem SU_A2 Shelf
(No Marvel) Marvel, Eline
Eth
Serv


Milestone Tracking and Traceability to Requirements
AREA Pass/Fail Comments
Use of Development SPIKES Provided R&D focus, Investor Visable Milestones, and
to enable cross subsystem enabled key risks to be proven early
early integration


Agile methodology made it difficult to accurately
define. Initially only happy path focus.
Strengthen Q3, Q4 Test Regime
Clear Architectural definition of Hardware
Abstraction Layer. Logging Capability & Treating HW
and Device Drivers together

Prototype hardware using reference board to host
early software loads (pre-hardware availability)


Automation of early milestones & spikes. To enable
n-1 regression

All System UC and Subsystem UC Required for each milestone is maintained & tracked
within Database Virtual Test Environment and formalised
Creation
HAL interface to enable distributed system test
without hardware

Extensive Use Automation
 60% Integration team are automation primes.
 1000 Automated Tests nightly against mainline (trunk):
o Execution time 12 h, 207 HW, 700 VTE

 Significantly greater coverage than normal Sanity
 Tests explore both System functionality and Subsystem

 Daily Inspection of all test automation results in 9:30 Am
meeting
 Daily Bug discussion of critical issues preventing performance


Automation Example : int_nightly_hw_regr

 Tests are organised into Test Umbrella’s that test specific
areas.
• int_nightly_hw_regr:
SummaryHW - 20111031-104218

Campaign
No. Category Total Run % Run Pass Fail % Pass
ID

1 BI 85462 45 45 100% 42 3 93%

2 CCS 85462 22 22 100% 22 0 100%

3 ECS 85462 32 32 100% 32 0 100%

4 ECS_OPST 85462 32 32 100% 32 0 100%

5 OLC 85462 10 10 100% 10 0 100%

6 OPST_DP 85462 29 29 100% 28 1 97%

7 SANITY 85462 7 7 100% 6 1 86%

8 SYSTEM 85462 30 30 100% 27 3 90%

TOTAL 207 207 100% 199 8 96%

 Other packs include ECS_OPST, OLC_DP, OLC_Bringup etc
 Some Test Packs only suitable for Weekend runs

Automation Trending

 Automation allows specific test areas to be investigated or trended
over time. E.g OLC Sprint 32
olc_Sprint32
 Example for one Testpack, being extended for individual tests


Hardware Test Environment

• Processors
• 256/512M
• PowerQUICC III • 35 Processors
1.3Gbits • 60GBit RAM
• Total • 45GHz Processing
• 7GHz Processor
• 10 GB RAM


Virtual Test Environment

1 • 35 Processors
Ring
• 10GBit RAM
• 45GHz Processing

• vSphere Virtualises Server
• total 210 virtual processors
HP DL380 Server • 60Gbits RAM
Dual Hex Core 2.8GHz • Approx 0.5TB Storgae (HD Image
2Gbits/CPU)
64GBits RAM
• VTE Costs= 10-12K

• Hardware Cost £xxMillion
1 1 1 1 1 1


Utilisation NPC Card


Automation and Virtual Test Environment

AREA Comments
Virtual Test Environment Needs Controlled and as much attention as Hardware
Environment. Firewall server/resources and ensure source
control of VM machine/memory requirements


Clarify Ownership of Test Areas
Initial organisational separation made structured
re-use procedures, classes difficult
Prototype hardware using reference board to host
early software loads (pre-hardware availability)

Automation General

 Be prepared to Retire Tests
Maintenance early agile tests often not worth
lifetime cost. Allow to die & replace with better


Consider Lifetime costs
Focused initially on automation for device drivers
and FPGA streams, chose TCL language without
long term focus for software (moved Python)
All System UC and Subsystem UC Required Object Orientated, structured test environment.
for each milestone is maintained & tracked
within Database Classes. Design Authority as per Software.
Common
Clearly defined logging levels. Too much debug as
destructive as too little 17
© 2011 Intune Networks Ltd. Confidential

Conclusions

 Drive programme from clearly defined integration strategy, with
staged integration points between complex subsystems
 Treat as full milestone review and apply “waterfall”rigour to
these points
 Track programme overall according to milestones vs 4 week
agile sprints

 Invest in Virtual Test Environment
 Massively cost efficient
 Build into process and architecture, clean hardware
abstraction layer
 Apply source control to VM’s

 Next Steps: Automated Assistance Manual Testing
NBI Interface Compliance vs WADL

Applying Agile Test to Complex Embedded Systems

Recommandé

Recommandé

Contenu connexe

Similaire à Applying Agile Test to Complex Embedded Systems

Similaire à Applying Agile Test to Complex Embedded Systems (13)

Plus de David O'Dowd

Plus de David O'Dowd (20)

Dernier

Dernier (20)

Applying Agile Test to Complex Embedded Systems