This document provides an overview of continuous testing in DevOps presented by Michael Hackett of LogiGear Corporation. It discusses what DevOps is and how testing fits into the continuous delivery pipeline. It emphasizes the need for testing at every step of development and deployment through test automation. It also provides examples of challenges companies face in implementing continuous testing and DevOps practices.
What are the key points to focus on before starting to learn ETL Development....
Continuous Testing in DevOps
1. DT2
DevOps & Testing
6/8/2017 10:00:00 AM
DT2 Continuous Testing in DevOps
Presented by:
Michael Hackett
LogiGear Corp
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073
888-‐268-‐8770 ·∙ 904-‐278-‐0524 - info@techwell.com - https://www.techwell.com/
2. Michael Hackett
LogiGear Corp
The head of the training, consulting, and publishing division of LogiGear
Corporation, Michael Hackett has twenty-five years of experience with software
engineering and testing in banking, securities, healthcare, and consumer
electronics. He has trained teams in sixteen countries and runs a global
consulting practice for organizations wanting to increase quality and productivity.
Michael authored Testing Applications on the Web and Global Software Test
Automation. A Certified ScrumMaster, Michael is an instructor at UC Santa Cruz
and Berkeley Extension, and four universities outside the US.
10. 5/25/2017
8
What are some DevOps Culture changes?
People, Process, Product all collaborate and work together.
Operations is now part of the application development
Tools eliminate or reduce environment, data, installation,
provisioning and deployment challenges.
Move to cloud technologies
Everyone is responsible for deployment. Deployment is no
longer only Ops job.
Product is always at Release Quality. Eliminate technical
debt.
15
Jez Humble and David Farley
Continuous Delivery: Reliable Software
Releases through Build, Test, and
Deployment Automation
Read:
17. 5/25/2017
15
All this testing is to keep the pipeline moving along.
• Image from XebiaLabs
• https://xebia.github.io/cd-with-docker/img/continuous-deployment-pipeline.png
CD Pipeline
• But not “our” big concern
• Our Automation has to fit in, easily, run by anyone
The Tool Chain is important
18. 5/25/2017
16
CD is not only about tools!
It seems so these days.
Lean Practice of Quality at Every Step
• A poka-yoke is any mechanism in a lean manufacturing
process that helps an equipment operator avoid (yokeru)
mistakes (poka). Its purpose is to eliminate product defects
by preventing, correcting, or drawing attention to human
errors as they occur. (Wikipedia.org)
What is CD?
What is Testing’s role in this pipeline?
• At each step along the pipeline flow, some type of validation
or verification needs to happen.
• Automate as much as you can.
• Keep the pipeline moving! No bottlenecks from testing.
• Testing gives immediate feedback.
– A measure of how good testing is in CD- how immediate is
feedback?
What is CD?
19. 5/25/2017
17
All this testing is to keep the pipeline moving along.
• http://www.informit.com/articles/article.aspx?p=1621865&seqNum=2
Continuous Delivery: Anatomy of the Deployment Pipeline
By Jez Humble and David Farley
The Tool and Ops Revolution
Powering the Pipeline
Quality at Every Step
The Lean Principle of “Quality at Every Step”
Its the basis of DevOps as Continuous Testing
24. 5/25/2017
22
What has to be perfect?
• Same sprint automation
• Definition of Done
• Collaboration
• Estimating
• Testing skills
• Feedback
• Interaction with the business/PO
• Great User Stories
• Unit testing
…..no Scrumbutts!
Agile has to be perfect- already
Continuous Testing is in addition to and different from testing
individual user stories or testing in sprints. This testing will
continue as a mix of manual then automated.
Continuous Testing
26. 5/25/2017
24
This whole process is ONLY possible with big,
significant, high-volume, fast, low maintenance,
excellent automation…with immediate feedback.
Not a little automation. A LOT!
Why are we talking about this with testers?
Special Cases-
“Functional tests teams” may take over:
• Performance testing
• Security testing
• Production Monitoring
• API/Web Service testing
• Testing in production
• Capacity Testing
• Platform Testing
• Device Compatibility
Or
• Our automated suites may be taken over by another team
or tool.
Solve issues with other types of tests
27. 5/25/2017
25
Automated testing
Test Automation means Tool Suite
• UI tool/Functional
• API tool
• REST
• SOAP
• Security…
• Vulnerability
• Penetration
• Performance
• Cloud service for devices…whatever
• Monitoring in production
Not all tools work in different situations
The tool can’t get in the way.
A very popular browser tools has famously had problems
funning on multiple virtual machines at the same time.
The tool got in the way.
• If you are using a cloud solution- the tool has to work
• If you are using high numbers of VMs the tool has to
work.
• If you are testing in production, the tool cannot
interfere! Or worse, slow down or crash! (non-
intrusive)
29. 5/25/2017
27
SOA/MicroServices
Testing APIs,
Producing services
Using containers as service components on your
system.
Understanding how to test services- next technical
leap.
Your REST and SOAP testing, tools and automation
skills may need an upgrade.
MicroServices
How do you test?
Develop an API/Service Test Strategy
• Test in Isolation
• Test Integration
• Test in Scenario, E2E.
• Automate it.
MicroServices Test Strategy
30. 5/25/2017
28
Regression Automation
What is regression testing?
And by now, it has to be Regression Automation.
Regression Testing
Regression testing is a big important thing.
Yet in many companies, it’s value is unclear.
Your regression suites in DevOps must be fast and give excellent
feedback to the teams on risks of speedy product delivery.
We will talk about what this means to automation in the next
section- but many teams have to rethink how they choose tests for
the regression suite for different places in Continuous Testing.
32. 5/25/2017
30
Regression Testing
Re-Think your regression strategy!
Does your regression suite have the main customer
flows? Long end2end tests or small isolated function
tests?
How old are your regression tests?
Do they find bugs?
Do they have value?
Regression Testing
You will probably need a few regression suites
CI- Smoke- small
Test- Some kind of “full”- perhaps bloated and old
Staging- task, customer scenario, Workflow- lean and
mean
Production/Monitoring- transaction- better not break!
Limited goals.
37. 5/25/2017
35
1. Get rid of the name “QA” – Everyone is responsible for testing and
quality. If your team has not fully grasped this value- you can never do
DevOps. Call them testers, test engineers, quality engineers- SDET-
whatever. They do not own quality.
2. Not enough fast, nimble, meaningful, current, easy-to-run automation.
3. No easy way to convey coverage: smoke test, full regression: code
coverage, happy path tests, main functionality, changed area regression,
bug regression, high priority regression – what are you doing and what is
left out to be fast and nimble. Never say “test everything” because you
don’t.
4. Teams forget about traditional, during-the-sprint, ad hoc and scenario
and forced error tests that may never be automated but have to be run
to see the new code works like a customer wants it – not what a unit test
validates.
5. Test environments, staging environments, testing in production – all
environment problems must be fixed. Made simple, fast and leverage
virtualization. Ops should help.
What are the problems/issues for testing?
6. Some test teams have never done performance or security testing
before. Particularly performance testing, in DevOps often gets run
along with functional regression suites- you may need to start
maintaining these also.
7. Data problems must be fixed; masking, randomizing, mirroring, current,
live– whatever – fix it and make it fast.
8. Deal with the risks of testing in production. Communicate the risks.
Don’t ignore it.
9. Automate, automate, automate- do it smart, fix it fast, unbreak it, do it
right (framework, organize, well-designed, thoughtful, fast-fast)
10. Your team may not realize the amount of collaboration needed to make
this work. Open, friendly, clear, all-for-one team communication is
mandatory.
11. You need easy, fast smoke tests in CI. Maybe even take over running
Jenkins/AutoBuild/Bamboo/whatever.
12. Learn about working in the Cloud and API/Service Testing, if you are
not already, you will be soon.
What are the problems for testing?
38. 5/25/2017
36
1. Fix any Agile/Scrum problems
2. Get CI and Smoke Tests smooth, easy, great
3. Get the teams to better understand and agree to your use of “regression”
4. Pass all IT/Ops/ system problems over to other people- stop the
struggle. Get to know how the environments solution works.
5. Fill-in whatever knowledge gaps you find (containers, SOAP, Rest,
microservices)
6. Get fully knowledgeable of every testing task for your product regardless
of who does it- programmers, what testing happens in production?
– Gather info on security testing, performance testing, etc.
– On what platforms? Server? Mobile? Desktop?
7. Make sure you have an automation tool that very easily works in your
virtualized world and tool chain.
8. Make sure you have easy, fast, extensible automation framework.
9. Re-think how and what you automate: lean and mean.
10. Develop a deployment production line mentality: everything you do helps
or stops deployment.
Jumpstart
You have to know what you are talking about with DevOps;
what parts impact testing and what do not.
Most of DevOps is not about test teams.
Remember the Culture Shifts
Automate Lean and Mean
– Continuous testing does not mean continuously rerunning your
full regression.
– You may need MANY sets of regression suites
– These sets will be run at different times on different
environments.
– Your tools have to support this.
– Automated Testing keeps the pipeline moving
– Don’t be the bottleneck!
Summary