DevOps is growing in popularity in last years, particularly in (software) companies that want to reduce its lead time (time to business value from idea to production) measured in days/weeks instead of months/years? If you want your software to do the right things and do these things right, you need to test it implacably. The big problem is that companies see (and it is) the testing phase as the bottleneck of the process slowing down product release. To change that, we need a new way of testing our applications, making the release of an application a testing process as well, and involve QA since the beginning within the team. QAs are not a separate team anymore (DevTestOps). What is the role of QAs in this new approach? How is the testing pyramid affected? How you can fail on trying to speed up release frequency? In this session, we will not only describe but also actively demonstrate several techniques that you can use immediately following the session for testing applications like unicorns.

1. @alexsotob1
Testing in Production.
From DevTestOops to
DevTestOps
Alex Soto Bueno
@alexsotob

2. @alexsotob2
Alex Soto
Red Hat Engineer
www.lordofthejars.com
@alexsotob
Who Am I?

3. @alexsotob
Raffle Rules (applicable in the real)
3
1. Follow: @alexsotob
2. With picture of the session
3. Mention @alexsotob
4. With hashtag #devopsbcn19
ctwdevopsbcn19 40% discount

4. @alexsotob4
Questions

5. @alexsotob5
I think that is the
most stupidest thing
I ever heard it.
— Gideon Grey
“

6. @alexsotob6

7. @alexsotob7
https://puppet.com/resources/whitepaper/state-of-devops-report
High-performing
organizations are decisively
outperforming their lower
performing peers ...

8. @alexsotob8
https://puppet.com/resources/whitepaper/state-of-devops-report
46X46X More Deployments

9. @alexsotob9
https://puppet.com/resources/whitepaper/state-of-devops-report
440X440X Shorter Lead Times

10. @alexsotob10
https://puppet.com/resources/whitepaper/state-of-devops-report
96X96X Mean Time to Recover

11. @alexsotob11

12. @alexsotob12

13. @alexsotob13
Change is good.
— Rafiki
“

14. @alexsotob14
OPS PMO DBASEC
BIZ DEV
QA

15. @alexsotob15
Continuous
integration
and testing
Build
Test
Continuous
delivery and
deployment
Release
Deploy
Continuous
operations
Operate
Monitor
Continuous Improvement
Continuous Improvement

16. @alexsotob16
QA Bottleneck

17. @alexsotob17
I don’t want to survive.
I want to live.
— Captain B. McCrea
“

18. @alexsotob18
We are on Production

19. @alexsotob19
Shit! We break Production Environment

20. @alexsotob20
When you find out
you can live without it,
go along not thinking
about it.
— Baloo
“

21. @alexsotob21
Production is not sacrosanct anymore

22. @alexsotob22

23. @alexsotob23
- Unit Tests
- Component Tests
- Static Analysis
- Coverage Tests
- Benchmark Tests
- Contract Tests
- Acceptance Tests
- Mutation Tests
- Smoke Tests
- UI/UX Tests
- Penetration Tests
- Integration Tests
- Tap Compare
- Load Tests
- Shadowing
- Config Tests
- Canarying
- Dark Canaries
- Monitoring
- Feature Flagging
- Exception Tracking
- Feature Graduation
- Teeing
- Profiling
- Logs
- Chaos Testing
- Monitoring
- A/B Testing
- Tracing
- Auditing
- OnCall Experience
- Journey tests
Cindy Sridharan
Pre-Production
Testing In Production
Deploy Release Post Release
The New ¿Pyramid?

24. @alexsotob24
But wow!
Am I so ready
for this change.
— Anna
“

25. @alexsotob25
Regression-free versions (tap compare)

26. @alexsotob26
Diferencia

27. @alexsotob27
Simple Comparison

28. @alexsotob28
Noise Detection Comparison

29. @alexsotob29
Blue-Green Deployments

30. Blue/Green Deployment
CI/CD PRODUCTION ROUTER USERS
BUILD
SCM

31. Blue/Green Deployment
CI/CD PRODUCTION ROUTER USERS
BUILDSCM

32. Blue/Green Deployment
CI/CD PRODUCTION ROUTER USERS
SCM

33. Blue/Green Deployment
CD/CD PRODUCTION ROUTER USERS
SCM

34. @alexsotob34
Canary Release

35. @alexsotob35
CI/CD PRODUCTION ROUTER USERS
SCM

36. @alexsotob36
CI/CD PRODUCTION ROUTER USERS
SCM

37. @alexsotob37
CI/CD PRODUCTION ROUTER USERS
SCM

38. @alexsotob38
CI/CD PRODUCTION ROUTER USERS
SCM

39. @alexsotob39
CI/CD PRODUCTION ROUTER USERS
SCM

40. @alexsotob40
CI/CD PRODUCTION ROUTER USERS
SCM

41. @alexsotob41
CI/CD PRODUCTION ROUTER USERS
SCM

42. @alexsotob @AndyGeeDe42
Dark Launches

43. @alexsotob43
DEVELOPMENT QA STAGING PRODUCTION ROUTER USERS
SCM
INTERNAL USERS
Dark Canaries

44. @alexsotob44
DEVELOPMENT QA STAGING PRODUCTION ROUTER USERS
SCM
Shadowing Traffic

45. @alexsotob45
Feature Graduation

46. @alexsotob46

47. @alexsotob47
Service A
Feature Graduation
Service B v1
Service B v2
user-preference:”123*”
Service B v3
user-preference:”12*”
experimental
beta
production

48. @alexsotob48
Service A
Feature Graduation
Service B v2
Service B v3
user-preference:”12*”
beta
production
Both experimental and beta levels

49. @alexsotob49

50. @alexsotob50
This is the circle of sadness.
Your job is to make sure
that all sadness stays
inside of it.
— Joy
“

51. @alexsotob51

52. @alexsotob52

53. @alexsotob53

54. @alexsotob54
It's time to see
what I can do
To test the limits
and break through.
— Elsa
“

55. @alexsotob55
Istio — ‘Sail’
(Kubernetes — the ‘Helmsman’)

56. @alexsotob56
Put on your Sunday clothes
there's lots of world
out there.
— Wall-E
“
[http://bit.ly/istio-tutorial]
[http://bit.ly/istio-cheat-sheet-v1]

57. @alexsotob57
Every adventure
requires a first step.
— Alice
“

58. @alexsotob58
Start with the most sympathetic and innovative groups
MarketGrowth
Time
2.5% 13,5% 34% 34% 16%
The Chasm
Early
Adopters
Innovators
Early
Majority
Late
Majority
Laggards
The Technology Adoption Curve (Source: Moore and McKenna, Crossing The Chasm)

59. @alexsotob59
No chance, no way
I won't say it, no, no.
— Meg
“

60. @alexsotob60
And Now What?
Evangelism
Test Guardians
Bridge To Management
Define Strategies
Write Tests
No Manual Tests, No Maintenance Tasks

61. @alexsotob61
Today’s special moments are
tomorrow’s memories.
— Gene
“

62. @alexsotob62
Now,
you see, the world
is full of temptations .
— Jimmy Cricket
“

63. @alexsotob63
Venture outside
your comfort zone.
The rewards are
worth it
— Rapunzel
“

64. @alexsotob64
If I go there is just no telling
how far I’ll go.
— Moana
“

65. @alexsotob
Hay un amigo en mí,
cuando salgan a volar,
hay un amigo en mí
— Toy Story
“
@alexsotob
asotobue@redhat.com
http://www.lordofthejars.com/
lordofthejars

66. @alexsotob66
About Images
http://disney.wikia.com/wiki/File:TLK_Simba.png
http://disney.wikia.com/wiki/File:Tiana.12.png
http://insideout.wikia.com/wiki/File:FEAR_Fullbody_Render.png
http://disney.wikia.com/wiki/File:Judy_Hopps_Zootopia.png
https://mbtskoudsalg.com/explore/disney-up-png/#gal_post_3621_up-balloons-png.png
http://parody.wikia.com/wiki/File:Jiminy_cricket_disney.png
http://zootopia.wikia.com/wiki/File:Gideon_Adult_Transprent.png
http://lifeofheroesrp.wikia.com/wiki/File:Aladdin.png
http://disney.wikia.com/wiki/Captain_B._McCrea
http://pixar.wikia.com/wiki/Slim
http://pixar.wikia.com/wiki/Auguste_Gusteau
http://hero.wikia.com/wiki/File:Joy_(Inside_Out).png
http://disney.wikia.com/wiki/File:Diamond_baloo.png
https://www.flickr.com/photos/photogreuhphies/7383113142/in/photolist-cfqojb-pC8jJB-6hCtMr-a4hoxN-9gPWs9-6EHZzs-dV9aSt-81JzLb-85TxfK-qsw4px-81JAH3-9LBbLF-kAeQJ
qdoArQ-pC7bTE-cDWcPJ-doZAgW-aZhdma-oFEs5b-a1865x-23TdcqE-7oLqCo-8TWx33-9Aazdw-5hFSPs-pUCGJQ-oGPNnf-pCa8P9-bAdBVK-51ooTn-nsZnCK-bvV1nu-92f6Ys-p1rKvB-
http://www.stickpng.com/img/comics-and-fantasy/the-incredibles/edna-e-mode
https://www.kisspng.com/png-dory-nemo-palette-surgeonfish-disney-infinity-3-0-916908/
https://www.kisspng.com/png-frozen-olaf-s-quest-elsa-kristoff-anna-anna-frozen-825204/
https://www.kisspng.com/png-eve-wii-wallpaper-wall-e-png-photos-421264/
https://www.kisspng.com/png-rapunzel-free-png-image-63202/
https://www.kisspng.com/png-ratatouille-french-cuisine-film-animation-pixar-ra-631822/
https://www.kisspng.com/png-peter-pan-tinker-bell-wendy-darling-clip-art-fly-655477/
https://gallery.yopriceville.com/Free-Clipart-Pictures/Cartoons-PNG/Transparent_Snow_White_PNG_Clipart#.W_PigJNKgnU
https://www.kisspng.com/png-princess-jasmine-aladdin-genie-youtube-abu-aladdin-961044/
https://www.kisspng.com/png-a-bug-s-life-flik-youtube-film-streaming-media-bug-2538893/
https://www.kisspng.com/png-buzz-lightyear-jessie-toy-story-sheriff-woody-tim-103824/
https://www.kisspng.com/png-alice-s-adventures-in-wonderland-the-mad-hatter-wh-689107/
https://www.kisspng.com/png-russell-youtube-carl-fredricksen-pixar-film-ups-1441690/
http://disney.wikia.com/wiki/File:Bambi_-_Render.png
https://www.clipartmax.com/middle/m2H7H7Z5m2i8G6G6_joy-upside-down-transparent-png-stickpng-inside-out-inside-out-characters-sadness/
https://www.kisspng.com/png-bolt-ariel-the-walt-disney-company-walt-disney-ani-6179624/
https://www.clipartmax.com/middle/m2H7i8b1b1m2A0Z5_4-png-toy-story-slinky-png/
https://dlpng.com/png/115503
https://clipart.info/elsa-frozen-disney-png-6355
http://www.pngall.com/ariel-png
CC 3
Not Commercial
Usage
Not Commercial
Usage
CC 4

67. @alexsotob67
“You’re no one hero.” –Moana (showing simple
things)
“If I go there is just no telling how far i’ll go” –
Moana (showing simple things)