Sesión impartida en netcoreconf 2020 de Valencia

1. 2020
Netcoreconf
Advanced deployment scenarios:
Rings and Canaries
Sergio Navarro Pino
Tech lead at AnalyticAlways
@snavarropino

2. #netcoreconf
Sponsors

3. hello!
Sergio Navarro Pino
Tech Lead at Analytic Always
@snavarropino

4. Agenda
1. Canary releases
 How to implement canary releases?
 Considerations
 Routing users to the canary
 Release gates
2. Deployment rings
3. Dark launchings

5. 1.
Canary releases
Danilo Sato https://martinfowler.com/bliki/CanaryRelease.html

6. Motivation

7. Benefits
✘ Reduce the risk of introducing a new software versions
✘ Safe rollback strategy if issues are found
✘ The ability to do capacity testing of the new version in a production environment. Does my
new version require enhanced hardware capabilities?
✘ Collateral effect: you get zero downtime deployment !!

8. WHY?
✘ We cannot easily reproduce production traffic patterns
✘ We have no proper testing environment due to
○ Complex integration with third parties
○ Elevated hardware requirements
✘ We don’t have enough validation before going live
✘ You want to increase your confidence when a new release is deployed

9. 2.
How?

10. Similar to blue/green deployment we start deploying the new
version to a subset of our infrastructure. no request are routed
there yet
1. deploy new version

11. We start sending some request to the new version
2. start routing users

12. If no issues are found related to new version, we gradually
increase the percentage of users routed to the new version
3. Monitorize

13. Once all users have been routed to the new version, we remove
the old one
4. remove old version

14. Place your screenshot here
Let’s meet our heroes

15. Application monitoring
✘ A list of awesome APM (Application Performance Monitoring) tools & products
(commercial and OSS)
https://github.com/antonarhipov/awesome-apm

16. Place your screenshot here
Azure Application Insights

17. “Canary deployment also gives you a
rapid way to rollback - if anything goes
wrong you may route all users to the old
version

18. key points to be taken in account
✘ We are running two versions in parallel
✘ Be sure your software supports it
https://es.slideshare.net/sergio_pino/despliega-como-los-grandes-zero-downtime-deployment

19. key points to be taken in account
✘ Deliver to users a consistent experience
✘ How we are going to monitorize?
✘ Do we always have a rollback path?
✘ Don’t replace Devops practices

20. How we route users?
✘ Random pattern
✘ Geographic pattern
✘ IP range pattern
✘ Based on user type (freemium vs premium). Feature toggles
✘ Based on some application logic. Feature toggles

21. Place your screenshot here
How to finish the deployment?

22. What about release gates?
Gates allow automatic collection of health signals
from external services, and then promote the
release when all the signals are successful at the
same time or stop the deployment on timeout
https://docs.microsoft.com/en-
us/azure/devops/pipelines/release/approvals/gates?view=azure-devops

23. ✘ 10% canary users
✘ 25% canary users
✘ 100% canary users
Define several stages, and a gate to transition between them
Transition will only happen if
there are no alerts related to
Failed request, response time
and availability

24. How gates are evaluated?

25. Place your screenshot here
Azure DevOps Pipelines
Release Gates

26. 4.
Deployment rings

27. Deployment rings were first discussed in Jez Humble's
book. They support the production-first DevOps
mindset and limit impact on end users, while gradually
deploying and validating changes in production.
Impact (also called blast radius), is evaluated through
observation, testing, analysis of telemetry, and user
feedback.

29. Is it so easy?

30. 5.
Dark launching

31. The facebook case
During the two weeks prior to launch we began what we call a
"dark launch" of all the functionality on the backend. Essentially a
subset of user queries are routed to help us test, by making
"silent" queries to the code that, on launch night, will have to
absorb the traffic. This exposes pain points and areas of our
infrastructure that needs attention prior to the actual launch.
Increasing the demand on one subsystem may generate more
logs than anticipated and overwhelm analysis processes, or
unexpected network bottlenecks may appear.
https://www.facebook.com/note.php?note_id=96390263919

32. How the code looks?
Execute new code
but show the user
the result of old
code

33. Recap
Two
versions in
parallel
Testing in
production
Monitorize
your
changes

34. thanks!
Any questions?
You can find me at
@snavarropino

35. #netcoreconf
Sponsors

36. Más información:
info@netcoreconf.com
@Netcoreconf
Visítanos en:
netcoreconf.com