This talk explained experiences and challenges implementing DevOps for a complex hardware/software-based network product with a large multi-site development team. DevOps was implemented across multiple organizations and system layers overcoming DevOps implementation challenges in a step-wise fashion. DevOps can be applied to complex multi-layer hardware/software based network products but be prepared for implementation challenges in the team and especially test orchestration and automation.

1. Case Study -
DevOps Cost-Effectively Secured Quality and
Time-to-Market for a Complex
Hardware/Software Network Product
Marc Hornbeek
Senior Solutions Architect

2. 2Spirent Communications
Before and After
BEFORE DevOps AFTER DevOps

3. 3Spirent Communications
Takeaways
5 Key Best Practices
(Continuous Testing)
Best Practices Assessment Tool

4. 4Spirent Communications
Network Test System
 Four-layered product:
 Use Cases
 Application
 Firmware
 Hardware
 36 million lines of code
 43 build targets
 93 deployment packages

5. 5Spirent Communications
Business Pains
 < 2 releases/year… LATE
 Rising escalations & defects
 Features deferred
 75% corrective work
 CapEx & Opex budget
 Lack of process visibility
 Intellectual property security

6. 6Spirent Communications
Technical Pains
 5 centers w/ separate branches / labs
 22 hour build + 24 hours test
 2 GB images 9 hours to transmit
 7 day deployment
 <50% tests automated
 Inconsistent metrics
 Culture Dev/QA/Delivery silos
 Credentials management issues

7. 7Spirent Communications
Business Goals
 Predictable monthly releases
 50% less corrective work
 25% more new features / year
 Stabilize spending
 Secure intellectual property

8. 8Spirent Communications
Vision and Architecture
Plan
 DevOps specialist team and Modular packages
 Security layers
Target Metrics
 95% automated build, test and release process
 One hour build/test (98% reduction from 46 hours)
 Nightly 8 hour regression (90% reduction from 2
weeks)
 24 hour release test (85% reduction from 4
weeks)
 One day deployment (85% reduction from 7 days)

9. 9Spirent Communications
Solution Approach
3. Accelerate
1. Infrastructure
2. Scale-Up

10. 10Spirent Communications
Integrate Processes
Common infrastructure
Process design
Controls

11. 11Spirent Communications
Stabilize and Secure
Process re-engineering
Tools
Fault tolerance
Intellectual Property admin

12. 12Spirent Communications
Metrics & Telemetry
Process times
Environment reliability
Intelligent dashboards
Security metrics

13. 13Spirent Communications
Remove Bottlenecks!
 Modular-ize
 Incremental-ize
 Remove dependencies
 Pipeline workflows
 Discount false failures

14. 14Spirent Communications
Scale It Up !
 Virtualize build and test
 Elastic build and test resources
 Multiple DevOps setups
 Workflow optimizations

15. 15Spirent Communications
Organize for Speed
 DevOps team expansion
 QA folded into development
 Customer-focused development
teams
 Fewer management layers

16. 16Spirent Communications
Optimize
 Distributed version management
with secure workgroups
 Target Pre-Flight for each
development team
 Development owns QA/testing
 Development owns training

17. 17Spirent Communications
Business Results
Phase 1
• Secure stable process
• Predictable dates
Phase 2
• Quality improvement
• Reduced corrective work
Phase 3
• CapEx stabilized
• OpEx reduced

18. 18Spirent Communications
Technical Results
 Stability: 99%
 Speed:
One hour build / test
8 hour package
24 hour release test
Security admin simplified

19. 19Spirent Communications
Lessons Learned
Unexpected OpEx reductions
Platforms stabilized faster
Manage culture proactively
Continuous testing
Best practices expertise

20. 20Spirent Communications
Continued Improvements
1) Pre-Flight to reduce
reverts
2) Hybrid cloud bursting
3) Smarter results analytics
4) Continuous culture
5) Automate security audits

21. 21Spirent Communications
Spirent CLEAR DevOps Solution Blueprint
Continuous
Deployment
(CD)
Continuous
Integration
(CI)
Plug-ins
Plug-ins
SUTTools
Lab Management
Physical, Virtual, Hybrid Lab
Analytics ALM
6. Expertise and professional services
1. Test orchestration & lab management
2. Comprehensive suite of test tools
3. Physical, virtual and mixed hybrid labs
4. CI/CT/CD/CCM tools integration (EVCI)
5. CT analytics, ALM integration
Orchestration
Continuous Test (CT)
CCM

22. 22Spirent Communications
CT Best Practice Assessment
Best Practice Description
(P)
Practice Level
Score To
what extent does the organization
practice this? 0=not sure, 1=Rarely, if
ever; 2= Sometimes; 3=Most of the
time; 4=Always; 5=We are really good
at this. Unsure enter NA.
(I)
Importance
Score How
important is this practice to the
organization? 0=not relevant,
1=not important, 2=nice to have,
3=important, 4=very important,
5=critical
(G)
GAP Priority
=f[(P),(I)]
Computed result is 1-15. A score
7.5 or higher indicates an
important GAP
CI builds are tested in a clone of the production environment. (Note:
"production environment" means "variations of customer configurations of a
product".)
2 4 8
Tests are selected automatically according to the specific software changes. CT is
orchestrated dynamically whereby the execution of portions of the CT test
suites may be accelerated or skipped entirely depending on how complex or
risky the software changes are.
2 4 8
Test resouces are scaled automatically according to the resource requirements of
specific tests selected and the available time for testing.
2 4 8
Release regression tests are automated. At least 85% of the tests are fully
automated and the remaining are auto-assisted if portions must be performed
manually.
2 4 8
The entire testing lifecycle including CI, nightly regression, weekend regression
and release acceptance phases are automatically orchestrated. The test suites
for each phase include a pre-defined set of tests that may be selected
automatically according predefined criterion.
2 4 8
Continuous Testing practices (CT) 2.0 4.0 8.0
Practice Level Average Importance Average Average GAP

23. 23Spirent Communications
Practice Area
(P)
Practice Level
Score
To what extent does the
organization practice this? 0=not
sure, 1=Rarely, if ever; 2=
Sometimes; 3=Most of the time;
4=Always; 5=We are really good
that this. Unsure enter NA.
(I)
Importance
Score
How important is this practice to
the organization? 0=not
relevant, 1=not important,
2=nice to have, 3=important,
4=very important, 5=critical
(G)
GAP Priority
=f[(P),(I)]
Computed result is 1-15. A score
7.5 or higher indicates an
important GAP
Pre-Flight Practices (Pre-Flight) 2.3 3.9 7.7
Continuous Integration practices (CI) 3.1 3.5 3.3
Continous Testing practices (CT) 2.0 4.0 8.0
Continuous Delivery practices (CD) 2.7 3.0 4.0
Continous Change Management practices (CCM) 2.3 3.0 5.3
DevOps System practices (System) 2.5 3.2 4.8
DevOps Team practices (Team) 3.0 3.8 3.8
Overall Assessment 2.5 3.5 5.3
Practice Level Average Importance Average Average GAP Level
DevOps Best Practice Assessment Offer
Email: marc.hornbeek@spirent.com

24. 24Spirent Communications
Summary
 DevOps works for platform
waterfall and Agile software
processes
 Best practices are critical
 Don't “do-CT-yourself”

25. 25Spirent Communications
spirent.com/devops

26. 26Spirent Communications
© Spirent Communications, Inc. All of the company names and/or brand names and/or product names and/or logos referred to in this document, in particular the name
“Spirent” and its logo device, are either registered trademarks or trademarks pending registration in accordance with relevant national laws. All rights reserved.
spirent.com
Thank you
Case Study -
DevOps Cost-Effectively Secured Quality and Time-to-Market for a Complex
Hardware/Software Network Product
Marc Hornbeek
Senior Solutions Architect
Marc.Hornbeek@spirent.com
Twitter: mhexcalibur