DevOps is a state of art describing each software development step as a repeatable, automatable and deterministic process excluding error-prone human factor first time in the history of software development. The model defines the entire value chance from concept to concrete product. It is an evolutionary end for the models of software development and agile movement. But, there is a problem with the concept; though described easily, implemented a little hard. DevOps Tactical Adoption Theory tries to make the transition process as smooth as possible. It hypothesis each step towards DevOps maturity should bring a visible business value empowering management and team commitment for the next step. The innovative idea here, it is not required to add the tools/processes to stack from sequential beginning to end, but seeking benefit. The reason behind the theory is to encourage practitioners to apply each step one-by-one and then having the benefits in projects. Consequently, each step is tested in terms of utility and proved method validity for the further steps. In contrast to previous adoption models, our model indicates concrete activities rather than general statements. Theory built on the claim that many DevOps transition projects considered problematic, impractical or even unsuccessful causing concept to become a goal more than a technique. Basically, theory consists of different areas of interest describing various actions on a schema. In the session, it is planned to demonstrate “DevOps Tactical Adoption Theory” with focus on Delivery Pipeline/Testing Practices sectioned "Continuous Testing in DevOps".

1. DevOps Tactical Adoption Theory:
Continuous Testing
Berk Dülger – Consultant, Account Manager

2. TEST CONSULTING TEST OUTSOURCING TEST TRAINING

3. Learning
Organizations
Chapter I

4. Learning Organizations
Learning organization is one
in which people at all levels,
individually and collectively,
are continually increasing
their capacity to produce
results they really care
about.
Peter Senge

5. Learning Organizations’ Five Discipline
Systems thinking: Ability to see whole
Personal mastery: The process of life long learning
Shared vision: A vision that shared & committed by everybody
Mental model: Generalized & assumed view of how the world works
Team learning: The process of developing the ability to create the
desired results as a team

6. Systems Thinking

7. State of Agile/DevOps
Adoption
Chapter II

8. Lean/Agile Adoption in Europe

9. Global DevOps Rates

10. Principles and Practices
Chapter III

11. Principles and Practices
Lean is the basis of Agile
Lean tells you to optimize the end-to-end process which creates value for your customer from
the initial idea to collecting cash. Lean principles focus on flow more than anything else:
bottlenecks in the process must be removed and wasteful activities need to be identified and
avoided.

12. DevOps is not a goal,
but a process of
continuous
improvement

13. Sustainable success requires
both bottom-up practices
and top-down management
support

14. DevOps Practices
Practices should address problems like,
• Manual Efforts
• Long feedback times
• Long MTTR
• Too much downtime
• Lead time
• Unrepetable work
…

15. Automation in DevOps
What can be automated?
• Dev environments
• Builds on pull requests and merge
• Static code analysis
• Code style checking
• Dynamic code analysis
• Verification
• Archiving artifacts
• Deployment

17. Tactical Adoption
Theory
Chapter IV

18. Tactical DevOps Adoption
DevOps Tactical Adoption Theory tries to make the transition process as
smooth as possible. It hypothesis each step towards DevOps maturity should
bring a visible business value empowering management and team
commitment for the next step.
The idea here, it is not required to add the tools/processes to stack from
sequential beginning to end, but seeking benefit.

19. Tactical DevOps Adoption
The reason behind the theory is to encourage practitioners to apply each
step one-by-one and then having the benefits in projects. Consequently, each
step is tested in terms of utility and proved method validity for the further
steps.

20. Large-Scale Adoption
• Begin with an end in mind
• Start with a pilot project
• Make someone/unit responsible
• Evangelize, build communities
• Gain executive buy-in
• Make people believe
• Drive tool standardization
• Automate, automate, automate: Build, Test, Deploy
• Demostrate the value!

21. Either two ways;
Choose to improve all categories for single project
Or, choose one category to improve across all projects
(i.e. Testing)

22. Continuous Testing
Chapter V

23. Testing in DevOps
There is no DevOps without Continuous Testing

24. Test Automation Pyramid
Business Facing
Technology Facing

25. Testing in DevOps
Checkforexpected
It’s not ideal to
automate
everything
Findtheunexpected

26. Unit
Integration
UI High
Medium
Low
Low
Medium
High
Medium
Long / High
Short
/ Low
Test Type
Business Logic Coverage
Code Coverage
Execution
Time / Costs

27. Testing in DevOps
Unit Testing aims to test small chunks of your code (seperate classes /
methods) in isolation from the rest of the world.
UI Testing, different name for system/acceptance testing, where you test the
entire system together to ensure it does what it is supposed to do under real life
conditions. (Unless by UI testing you mean usability / look & feel etc. testing,
that is typically constrained to items on the UI)

28. Testing in DevOps
You need both of these in most of projects, but at different times: unit testing
during development (ideally from the very beginning, TDD!), and UI testing then
later, once you actually have some complete end-to-end functionality to check.
If you already have a system running, but no tests, practically you have
legacy code. Strive to get the best test coverage achievable with the least
effort first, which means high level functional tests.
Adding unit tests is needed too, but it takes much more effort and starts to pay
back later.

29. Automating Tests: Continuous Testing
Compile Unit Tests
Functional
Acceptance Tests
Non-Functional
Acceptance TestsStatic Tests
Exploratory
Testing &
Business
Decision

30. Continuous Testing Anti-Patterns
Long and slow deployment pipelines
Test Data Management is not a big deal
We can skip non-functional tests
Can be done by anyone
Don’t need to refactor/maintain automated tests

31. Long and Slow Deployment Pipeline
Anti-Pattern
Tips:
• If next stage(Automated Acceptance Tests) takes a significant amount of time (e.g. More than 30
minutes), embed a small subset of them into commit stage. So, feedback interval will be
decreased to act fast on major incidents
• Run tests in parallel (TestNG for Java and MbUnit for .Net might be good choices)
• Focus on multi-threading for race conditions
• Design atomic scenarios

32. Long and Slow Deployment Pipeline
Anti-Pattern
Tip: Prefer wide and shallow architecture
rather than deep and narrow.

33. Test Data Management is not a Big Deal
Anti-Pattern
Four Design Techniques for Successful Test
Automation Data Management
A typical maturity level of data management for test automation
process is outlined here;
1. Fully Integrated Test Data
2. Partially Independent Test Data
3. Storing Test Data in an External Source
4. Dynamic Test Data Management (Micro Services, GUI ?)

34. Non-Functional Testing Anti-Pattern
Tips:
• Select most business critical cases (Either
widely used or critical for a business)
• Test against a production replica
environment, for example staging (As much as
possible)
• Do care about data. Effects computational
cost
• Focus on subject matter practices (Anything!)
• Use automated-acceptance tests with
counters (As a first step maybe)

35. Can be Done by Anyone Anti-Pattern
Reasons Behind The Idea
• Test automation is a development activity (Performance,
Security Testing etc. as well )
• Convincing people to have a carreer in the field
• Positioning the personnel and task in the right place
• …
May prefer a different job title, like ‘Software
Development Engineer in Test’ (SDET)

36. Automation Maintanance is not Required
Anti-Pattern
Automation code is passive, meaning effected by
any change in product code.
Even with a perfect automation architecture, many
times it is not that possible, you will need to
redesign against living product.
Sounds like Software Gardening!

37. Continuous Testing – E-commerce Case Study
Inflection Point
2-3 Test Cases per Man/Day
Nearly No Maintance Effort
3-5 Test Cases per Man/Day
Less Maintance Effort (%20)
2-3 Test Cases per Man/Day
Moderate Maintance Effort (%70)
3 Test Cases per Man/Day
Moderate Maintance Effort (%50)
~1 Test Cases per Man/Day
Heavy Maintance Effort (%90)
Maximum number of test cases
~350 – 500 depending of SUT
Based on metrics from 14 consultancy projects

38. QA Intelligence Survey 2015
Kristian Karl – TestIstanbul 2016
Mike Cohn Test Automation Pyramid
Google Search Trends - DevOps
searchsoftwarequality.techtarget.com/tip/Use-Agile-software-testing-principles-to-plan-your-tests
blog.martinfenner.org/images/Agile-vs-iterative-flow.jpg
www.slideshare.net/IBMDevOpsforEnterpriseSystems/lessons-learned-from-large-scale-adoption-of-devops-fori-bm-z-systems-
software
www.slideshare.net/ThoughtWorks/when-enterprise-meets-devops/15-
PRIORITIZE_PILLAR_OF_PRACTICES15ESSENTIALCollaborationBuild_for
www.slideshare.net/SkeltonThatcher/continuous-delivery-antipatterns-from-the-wild-matthew-skelton-continuous-lifecycle-
london-2016
confengine.com/agile-india-2016/proposal/1680/how-to-explore-the-learning-organization-within-the-agile-organization
References

39. Thank You
Berk Dülger
berk.dulger@keytorc.com
https://tr.linkedin.com/in/berkdulger