13. Fear is the path to the
dark side. Fear leads to
Inertia. Inertia Leads to
paralysis. Paralysis leads
to going out of Business.
Do not fear the Release!
21. How do you test performance?
Different apps have different profiles
Tests need to scale
Tests should be shareable
Tests should be straight forward.
15 years ago, seventeen people met to talk, ski, relax, and try to find common ground. What emerged was the Agile �Software Development� Manifesto. From that day on, our way of working has changed several times swinging between waterfall to agile back and forth trying to find a balance. Most of our teams are working now in Agile but many times it is only encapsulated to DEVS/QA or only to OPS. Finding it difficult to cover all the product life cycle with all the actors involved in agile methodology.
We have been talking about continuous delivery for a while, we want our products to arrive fast to our customers and fullfill their needs. We have created deployment pipelines, automating the release, automating the infrastructure. But many times we are still fighting against our fear to release.
We’ve found that the costs we incur–typically bugs or unpolished but functional features–are worthwhile in the name of getting feedback from our customers as quickly as possible about our product innovations. The sooner we have feedback from our customers, the sooner we know whether we guessed right about our product decisions and the sooner we can choose to either change course or double down on a winning idea.
Integration of Performance Testing in Continuous delivery
A continuous integration server to run and monitor our automated tests with every code commit
cluster immune system to monitor and alert on statistically significant regressions, and automatically revert the commit if an error occurs)
A source control commit check to stop new code being added to the codebase when something is broke
Real-time monitoring and alerting to inform the team immediately when a bug makes in through our deployment process
Root cause analysis (Five Whys) to drive incremental improvements in our deployment and product development processes ( HOSHIN )
Recognize this?
single Line of change takes a week to deploy
Slow feedback
Code Freeze? what is that?
Sleeples nights during deployments
about me
why Scrum teams usually require my skills?
well usually when there is a question in the air…
HOW MANY USERS can we handle? or how do the service scale?
but these questions could be answered easily with some planification during development.
We all know that performance is not just throwing money at more servers virtual or physical, but we are terrify by failing
what are the problems we usually face during our last phase?
To answer the question of how to scale the service we usually face the problems that
we don’t have a server similar to production.
we don’t know the behaviour of the users,
we don’t have a system with the volume of production.
we cannot have all services 3rd parties integrated and running at the same time.
You must unlearn,what you have learned, you need to leave your confort zone,
with no change there is no fail, but with no fail there is no way to improve.
How many of us still deploy with a blue green?
Pattern
Deploy software to a non-production environment (call it blue) while production continues to run. Once it’s deployed and “warmed up”, switch production (green) to non-production and blue to green simultaneously.
Anti-patterns
Production is taken down while the new release is applied to production instance(s).
So let me tell you a joke a really bad one but i bet some of us will feel represented by that.
Borat the devop says!!!
lets go back again we started talking about continuous deployment
what benefits can we have?
automate provisioning and deployment
ensure devs, testers and ops collaborate throughout → devops
incrementalism
decoupling deployment and release
Don’t be afraid of failing, inmune systems→ zero access policy ( phoenix or inmutable servers)
Monitoring graphite, munin, nagios Anomaly Detection!!!
Infraestructure as Code: Automate or Die! Ansible, Chef, Puppet…
Containers: Docker, Joyent Triton
don’t be afraid of failing we will fail! we are human
but what can we do to minimize the impact of our failing?
reducing release risk
we need to build our Contingency plan to know how to react
Significant Alarms
Anomaly
Build Environments mean time to restore service instead of mean time between failures
Developers need to talk to ops about the impact of the code:
What metrics will change and how?
what are the risks?
what are the signs that something is going wrong?
what are the contingencies?
Ops’ job is NOT to keep the site stable and fast
Ops’ job is to enable the business (this is dev’s job too)
The business requires change
But change is the root cause of most outages!
Discourage change in the interests of stability or Allow change to happen as often as it needs to
Ops need to talk to devs
what has changed?
what features entered?
how do we deploy the software
where do you expect
MTRS: mean time to restore service
MTBF: mean time between failures
Simian Army
Don’t look at me after BDD TDD and the rest of xDD we can add one more acronyms to our salad.
DDD is the tecnique that will help our collegues from Delivery to be sure about the products we create and develop. This tecniques will help to deploy ina more agile way
real user monitoring because not always the users are what we expect and we need to give service to everyone of our target
we need to be ready to find any anomaly detection and not only problems
we need to check the server health
Logs alarms significant and not raised all kind of alarms
and how does this benefit our performance tests?
why not testing a real environment
real servers
real traffic
real users
real workflow
https://www.facebook.com/note.php?note_id=96390263919
"Dark Launching": During the two weeks prior to launch we began what we call a "dark launch" of all the functionality on the backend. Essentially a subset of user queries are routed to help us test, by making "silent" queries to the code that, on launch night, will have to absorb the traffic.
Levers and the "Nuclear Option": We couldn't be sure of the exact level of traffic the launch would generate so we put together contingency plans to decrease the load on various parts of the site. This gave us overhead in core functionality at the expense of less essential services.
Pattern
Launch a new application or features when it affects the least amount of users.
Anti-patterns
Software is deployed regardless of number of active users.
config file with the status of the feature disabled or enabled
in this case there is just one front end that got the logic of the feature depending on its status
RISK: huge chance of having a huge config file with obsolete features
Diassociat deployment and feature activation
Helps you to have a production ready code all the time
Gives you the flexibility of trying out multiple features
QA can test individual features or group of features
Business can get fast feedback through feature previewing.
we can do progressing activation of a feature → Canary Testing
Measure impacts of a new version ( A/B testing)
Diffeer feature activation on production
Trunk Base Development
Python - Waffle
Golang - Flash
Node.js - feature-flags
C# Feature Switcher
2 or more versions of the same feature enabled
most of the times without a segmentation of the users so we can try randomly which of the features is the best.
An A/B test involves testing two versions of a web page — an A version (the control) and a B version (the variation) — with live traffic and measuring the effect each version has on your conversion rate. Start an A/B test by identifying a goal for your company then determine which pages on your site contribute to the successful completion of that goal.
Pattern
Release software to production for a small subset of users (e.g. , 10%) to get feedback prior to a complete rollout.
Anti-patterns
Software is released to all users at once.
es to use a strategy with multiple canaries, the first one being visible only to their internal employees and having all theFeatureToggles turned on so they can detect problems with new features early.
Canary releases can be used as a way to implement A/B testing due to similarities in the technical implementation. However, it is preferable to avoid conflating these two concerns: while canary releases are a good way to detect problems and regressions, A/B testing is a way to test a hypothesis using variant implementations. If you monitor business metrics to detect regressions with a canary [2], also using it for A/B testing could interfere with the results.