Talk at Iqnite conference for software quality and testing about how we have do continuous delivery at here.com Together with Siddharth Somasundaram http://www.iqnite-conferences.com/suisse/programme/programme.aspx

1. Testing in a
Continuous Delivery
Environment
Siddharth Somasundaram
Stefan Verhoeff

2. HERE
● Formerly known as Nokia / NAVTEQ Maps
● Powers Bing, Facebook and Yahoo Maps
● 4 out of every 5 cars with in-dash navigation are
powered by HERE

3. HERE
● Maps on the Web offering here.com

4. Testing Before Agile
● Testers are only involved in the later stage
● Long regression testing cycle before release
● Developers and Testers have separate work streams
○ Leads to conflicts between them
● Business pressure to release the software
○ Compromise on quality?

5. Importance of Agile
● Requirements change over time
● Testers are part of the planning process
● The entire team is responsible for Quality
● Faster release cycle
○ From once every 3 months to daily releases

6. Continuous Delivery
● Release early and often to production
● Automate testing and deployment
● Faster feedback, reduce risks and overhead for releasing
● Hot trend in the industry, especially web companies

7. But what about Quality Assurance?
● Lots of automation for regression testing
● Build enough trust in the test coverage
● Focus shifts from prevention of bugs to rapid discovery
and resolution
● Exploratory testing can be decoupled from releasing

8. How do we do it?
● Team Culture and Technical practices we found to be useful

9. Team Culture Practices
● One agile Team, One Product and One Goal
○ Product Owner
○ Scrum Master
○ Developers / Testers
○ Offshore teams
● Communication solves most of the problems

10. Collaborate using Work Agreements
The team formulates rules and hold each other
accountable. For example:
● Running tests before committing
● No commits on red build
● First priority is to keep the build green

11. Build Monitor visible in every room

12. Pair Programming

13. Share knowledge with Tech Talks

14. Combine developer and operation teams

15. Technical Practices

16. Continuous Integration

17. Test Driven Development
● Code and tests are developed at the same time

18. Build Pipeline
●

19. Test Coverage at all levels
© Alister Scott http://watirmelon.com/2011/06/10/yet-another-software-testing-pyramid/

20. Eliminate Flaky Tests
● Essential to keep the team
trusting the test results
● Mocking dependencies

21. Feature Flags
● No branching
○ Prevent integration problems
○ Quality built in
● Turn features on in production
when they are ready
● Quality boundaries for
different release stages

22. Logging & Monitoring
● What happens after we go live?
● Production monitoring ensures edge cases are covered
● Real-time metrics for: system, application, business

23. Logging & Monitoring
● Quickly responding to alerts
● Production is the ultimate proof the code works
Bug introduced We fixed it :-)

24. In Conclusion...

25. Is it Successful? The Metrics
2010 Now
Release quarterly Release daily
Mostly manual testing 7000+ unit tests, 400+
GUI tests
Preparing release takes
weeks
Release package ready
in 45 minutes
Deployment takes days Deploy 3 sites in 10
minutes
Management signoff
required
Team empowered to
make changes

26. Deployments 2012 vs 2013

27. Business Benefits
● Reduce risk involved in big changes
● Reduce time to market
● Smaller more frequent feature rollouts
● Quickly learn from feedback on product changes

28. Business Benefits
● Predictable releases
● Improved team morale
○ Less time spend in stressful firefighting
○ Less bug fixing, more feature delivery
○ Satisfaction of delivering to the customer

29. Challenges
● Everybody agreeing on the process
● Discipline comes over time only
● Feedback needs to be given quickly
● High investment in test automation and keeping it
running smoothly
● Need to react to incidents quickly

30. Future Plans
● Deploy every build to production
● Keep on investing in Automation and Monitoring
● Canary deployments to validate new releases before
global rollout
● Leverage quick cycle time for A/B testing driven feature
development

31. HERE Checklist
Team Culture
● One Team
● Work Agreements
● Build Monitor in the
room
● Pair Programming
● Tech Talks
● DevOps
Technical
● Continuous Integration
● Test Driven Development
● Build Pipeline
● Test Coverage on all
levels
● Feature Flags
● Logging & Monitoring

32. Merci Beaucoup
Thank You
Danke Schön
Grazie