More and more companies are using Progressive Delivery to get all types of changes – new features, configuration changes, bug fixes, and experiments – into production in a safer, faster, and most importantly, a sustainable way.
Software companies that shift to Progressive Delivery benefit from low risk releases, faster time to market, higher quality, and in general happier teams. Sounds great right? But what happens when your system isn’t implemented correctly, or worse, tested properly?
This talk takes you on a journey of why teams use Progressive Delivery, and the path from basic to advanced feature flag usage.
Make sure you build the right product, and build the product right!
How to decide what testing strategies should be implemented (how to setup your unit test, end to end automation, etc)
What tactics are most effective for keeping your implementation healthy and effective (feature flag governance!).
Boost Fertility New Invention Ups Success Rates.pdf
Quality Leadership, Testing, and Governance Tactics that Make or Break Your Progressive Delivery System - Jeff Sing
1. Jeff Sing (Iterable)
Quality Leadership,Testing, and
Governance Tactics that Make or Break
Your Progressive Delivery System
2. Jeff Sing
● Sr Eng Manager - Quality & Operations
@Iterable
● 15 Years in QA in a variety of industries
● Progressive Delivery Advocate
jeff.sing@iterable.com
https://www.linkedin.com/in/jeffsing/
4. Progressive Delivery
“… a new new basket of skill and technologies concerned with modern
software development, testing and deployment. I am thinking of Canarying,
Feature Flags, A/B testing at scale.”
- James Monk, RedMonk
6. Continuous Delivery helps your team move faster by
making each change small and manageable.
Progressive Delivery helps your team move faster by
reducing the risk of each change by controlling the
audience exposed.
7. Data Driven Software Development
Did I build the right Product?
Did I build the Product right?
12. Software Development Life Cycle
Build & Test
● Features don’t need
to be completed to
get merged
● No wasted time
untangling messy
merge conflicts
Deploy
● Ship faster by
decoupling deploys
from releases to
shorten dev cycle
● Deploy frequently
and safety
Release
● Empower teams to
manage their own
releases
● Quickly rollback
production
incidents (way
faster MTTR)
13. Feature Flag Solutions
Companies home build their own progressive
delivery solution. Engineering usually builds
solution and iterates on functionality as the
system matures (Uber Piranha).
Vendor bought solution such as Optimizely or
LaunchDarkly. These vendors charge by seats
and unique visits. Typically they provide a SDK
to integrate with your system and custom
dashboards.
Homebuilt Solutions Vendor Solutions
19. Feature Flag Governance
Flag Info
General Info
- What does Flag do
when it’s On/Off
- What env is it
deployed?
Visibility
- Who needs to know if
this is On/Off?
- Who is monitoring
metrics?
Risk Level
How dangerous is this to
roll out?
What happens if something
catastrophic happens to
this flag?
Access Level
Who is allowed to make
changes to this feature flag?
How do we track this?
Exit Strategy
When is this feature flag no
longer useful?
When it is no longer useful,
who is removing it?
26. Testing Strategy
Test Type High Level Strategy
Manual Test Only most critical variations.
E2E Test Focus only on important variations
and test application still works if all
features are on/off
Integration Test Mock and stub to control feature
states. Focus on individual code paths
to ensure proper integration and
business logic
Unit Test Each code path should have its own
set of independent unit tests. Ensure
high code coverage, just as if you
didn’t have any feature flags in your
codebase.
28. General E2E Automation Test Strategy
Remove
INDETERMINISM
1. Special Test User
2. Test Query Parameter to
force a particular version
3. API Endpoint to set variation
HELENA
PATTERSON
“Despite being red, Mars is a
cold place. It’s full of iron
oxide dust, which gives the
planet its reddish cast”
33. Feature Flag Types
Feature Rollout
For deploying a new feature
that will be permanent.
Experimentation
Perform A/B/n tests against
our deployed features.
Short Term
34. enabled: True
header_color: #F829EA
header_sticky: True
header_height: 20
header_message: ‘We love
our users <3’
mobile_app_header
Experimentation
Does header “We
love our users <3”
drive more
conversions vs “Hi,
welcome?” enabled: True
header_color: #F829EA
header_sticky: True
header_height: 20
header_message: ‘Hi,
welcome?’
mobile_app_header
50% 50%
Customer A Customer B
35. Feature Flag Types
Feature Rollout
For deploying a new feature
that will be permanent.
Experimentation
Perform A/B/n tests against
our deployed features.
Permission
Change product experience
that certain users receive
Short Term
Long Term
37. Feature Flag Types
Feature Rollout
For deploying a new feature
that will be permanent.
Experimentation
Perform A/B/n tests against
our deployed features.
Permission
Change product experience
that certain users receive
Operation
Controls operational
aspects of your system;
allows control of certain
operational features in your
production environment
Short Term
Long Term
40. Feature Flag Types
Feature Rollout
For deploying a new feature
that will be permanent.
Experimentation
Perform A/B/n tests against
our deployed features.
Permission
Change product experience
that certain users receive
Operation
Controls operational
aspects of your system;
allows control of certain
operational features in your
production environment
Circuit Breaker
Turn on/off certain features
or conditions in your
production environment.
Short Term
Long Term
45. Feature Flag Governance
Flag Info
Access
Level
Risk Level
Exit
Strategy
Experiment
Hypothesis
driven
Product/Eng Low Experiment End
Permission
Business Tier
Provisioning Info
CS/AE/GTM
Restrictive
Access
High Rare
Operational Eng Runbook Eng High/Medium
If Feature
Deprecates
Circuit
Breaker
Eng Runbook Eng Low
If Feature
Deprecates
51. Permission: E2E Test Strategy
Permissions
1. Checking that
“provisioning” is correct
2. Automation should
check all paths
3. High sensitivity in
regression that we don’t
remove a customers
features
52. Operational: E2E Test Strategy
Operational
1. Validate “default”
condition
2. Strategy to validate
operational values may
vary...
3. Doesn’t always make
sense to automate all
paths...
enabled: True
checkout_version: 3
quick_checkout: True
header_height: 20
checkout_flow
53. Circuit Breaker: E2E Test Strategy
Circuit Breaker
1. Regression Automation:
Product still works if
flag is ON, and still
works if OFF
2. 3rd Party Integrations -
usually manually tested
54. TAKEAWAYS
POWERING MODERN
DEPLOYMENT
Feature Flags used correctly are
an incredibly powerful
engineering tool.
CONTROLLED BLAST
RADIUS
Quality champions rejoice! Plus
our MTTR should be factors
better!
01
03
02
04
05
06
THOUGHTFUL
IMPLEMENTATION
Ensure your organization has a
strong process for
releasing/managing/removing
your feature flag, or else
technical debt can punish you.
FEATURE FLAG
DIFFERENTIATION
Not all Feature Flags are alike,
why would you treat them so?
DETERMINISTIC
TESTING
Ensure you are properly testing
your feature flags and that you
have good feature coverage for
each path.
QUALITY LEADERSHIP
Every organization needs
leadership around governance
and testing.
55. Interested in learning more about Feature Flags? Or trying them?
VENDORS
■ Optimizely (Free Rollouts): https://www.optimizely.com/rollouts/
■ LaunchDarkly: https://launchdarkly.com/
■ Split: https://split.io/
FURTHER READING
■ Feature Toggles: https://martinfowler.com/articles/feature-toggles.html
■ Getting Started WIth Feature Flags: https://dzone.com/refcardz/getting-started-with-feature-flags
■ Unleash (open source Feature Flag): https://github.com/Unleash/unleash
RESOURCES
56. CREDITS: This presentation template was created by Slidesgo, including
icons by Flaticon, and infographics & images by Freepik.
Please keep this slide for attribution.
jeff.sing@iterable.com
https://www.linkedin.com/in/jeffsing
/
Stay in Touch
Notes de l'éditeur
Currently am leading the Quality Program and Engineering Operations at Iterable - a cross channel marketing platform that enables e-commerce customers to control their customer experience and understand and measure every interaction across their entire customer journey.
We serve customers like Doordash, MadisonReed, zillow, and asics.
15 Years
Web Security/Fraud Prevention
Medical Devices
Marketing Tech (most recently)
Everything from small medium to humongous
Continuous Delivery is a set of practices that ensure your code is always in a deployable state. You accomplish this by increasing the frequency at which code is committed, built, tested, and deployed—steps that in the past only occurred at the end of a project when it was ‘code complete’.
Because your team is delivering frequently, every change is smaller. Changes are more isolated and less risky. Problems are identified more readily, and bugs are fixed more easily because they are discovered immediately.
Progressive Delivery is the technique of delivering changes first to small, low-risk audiences, and then expanding to larger and riskier audiences, validating the results as you go. We can accomplish this by deploying first to a certain environment or certain user.
What does that mean? Well in QA, a lot of our orchestration is around to answer this question - Did I build the product right? Automation test, test planning, feature testing matrixes, etc is to answer did we release correctly? CI helps us with this because it makes it easier for us to do this, smaller changes, smaller releases...
On the other hand, all of this cannot help us determine if we built the right product. Using Progressive Delivery we can not only control our feature rollouts, but we can experiment on who and what our customers experience is, and determine if this is the right feature to release.
So let’s talk orchestration of Progressive Delivery: The primary method in the software development industry to orchestrate this are Feature Flags. Essentially you are wrapping your features behind a flag and orchestrating it so that when you deploy to production, it isn’t visible until you are ready to flip the switch. Now obviously their are more complexities to this, but this is the core concept.
You can build on top to have a method to deploy in a sliding scale (so 20% of your audience, 40%, 60% etc) or pass in certain values or configurations that only affect features behind that flag.
So let’s look at an example, here is what I would call a feature rollout which i just mentioned. Here my feature flag (mobile_app_header) is disabled. When my users come to my app, nothing is in the header. However when I’m ready to flip it on, it will appear in production.
Build & Test - no stale feature branches, smaller code reviews
Deploy - we actually deploy constantly at Iterable. This allows us to test certain features in production.
Release - SE can release certain features, teams can control betas. QA perspective MTTR has improved
For todays talk - I’ll talk around the 2 major ways companies are implementing FF - either as as homebuilt solution or vendors (in this case Launch Darkly). Each have their pros and cons but this talk easily applies to both.
So now that I’ve talked a bit about progressive delivery and the power of feature flags, let’s get to the meat of this talk… testing and governance tactics
What does this flag do: You have a feature flag out forever, the developer leaves, and now you have no idea what it does, I guess you can never turn it off safely…
Who turned off my flag: A new engineer comes, misreads some doc, and turns off a flag not realzing it causes a catastrophic outage
Who touched my flag: This one is more insidious, we didn’t remove a feature flag, and the other dev teams didn’t realize it was in your code base. Someone builds a feature flag upstream and turns it off, suddenly your flag no longer behaves the way you think it does. Who touched your flag? Why?
Knights capital - high frequency trading algorith - KNIGHT was largets trader in US equities market with a mrket share of 17.3% on NYSE and 16.9 on NASDAQ.
Developers reusing a flag that was no longer thought to be in use.
Dead code coming back to life.
One server started making as many trades as fast as possible without limit
Rolled back within 45 minutes. ½ billion dollars
Flag info x4
Access Level x2
Risk Level x2
Exit Strategy x2
Unit testing - your developers should be writing these with the mentality that you didn’t have any feature flags in your codebase
Integration tests - this is important to have orchestration to mock or stub your feature states. You need to make sure the code paths are tested
The top part of our pyramids - automation gets expensive, and you may have many feature paths, so for E2E you really want to make sure you are testing your critical variations - also don’t forget to test OFF state, as your flags may be in for abit and at any point you could be turning off your feature
Manual Testing - is super important. You should be checking your critical variations. What is important though is you need to expose your manual testers tooling to be able to control the flags to let them get into the right variation. … also make sure your tooling lets you know what variation you are in.
Experimentation - allows me to drive traffic to different features and measure conversions
Permission - change product experience - bucket your premium users into certain experiences
Operation - controls operational aspects - some examples, you can hook a feature variable to your features, so if the code base is activated it dynamically will pull that value in, meaning you can do something like point to where the source of an image is dynamically change it by modifying the resource location, or maybe a devops config change, where you can modify how many instances are being spun up, etc.
Since permission flags are acting as gates for a tested feature, testing for permission feature flags revolve around ensuring the right customer is correctly bucketed rather than if the feature is working. This is usually handled by the business systems team as we add users into the permission flag. In general they perform this by picking a customer and emulating their experience on our product and validating that they are getting the correct set of features purchased.
Since permission flags are acting as gates for a tested feature, testing for permission feature flags revolve around ensuring the right customer is correctly bucketed rather than if the feature is working. This is usually handled by the business systems team as we add users into the permission flag. In general they perform this by picking a customer and emulating their experience on our product and validating that they are getting the correct set of features purchased.