Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Supporting the Product through Continuous Delivery
Similar to Supporting the Product through Continuous Delivery (20)
More from UXDXConf
More from UXDXConf (20)
Recently uploaded
Recently uploaded (20)
Supporting the Product through Continuous Delivery
- 1. Supporting the Product through Continuous Delivery The Good, the Bad and the Ugly of Continuous Delivery
- 2. July 26 2018 Launching Dashlane 6
- 3. A refresher on Continuous Delivery • The ability to deliver to production, fast, reliably and on-demand, through an industrialized automated Release Pipeline.
- 4. High-Performing code practices • Centralized Code Repository • Strong orchestration through Continuous Integration • Code Quality through tooling and systematic mandatory code review
- 5. Continuous QA Automated Testing framework everywhere UNIT TESTS INTEGRATION TESTS FUNCTIONAL TESTS MANUAL TESTS AUTOMATED TESTS Full-Stack Testing: • Unit Tests • Integration Tests • Functional Regression Tests
- 6. Continuous Operations • Monitor everything • Morning Gymnastic practice • Alert System for notifications
- 7. The complexity of the many pipelines of delivery MySQL RedShift Mongo DB SQL Server iOSAndroid WindowsMac OS Web AppExtensions JS Semantic Engine (As a Service) Dashlane API (Web Services) Dashlane Web Site Native Clients Web Clients Server Hosting on Amazon Web Services User Data Computed Usage Logs AnalyticsRaw Usage Logs 3rd Party Providers Stripe PayPal Braze SendGrid AppStore PlayStore 3rd Party Partners Team Admin Console
- 9. Progressive Roll out Ship new code progressively • 1%, 5%, 20%… 100% • Monitor and increase
- 10. Feature-Flipping Activate features from the Back-Office for a segment of users
- 11. A/B Testing • Experimentation in production • Work in conjunction with Progressive Rollout and Feature Flip
- 12. Communication Failure • As you accelerate, communication and synchronization become critical • All teams should be aware of the release life cycle.
- 13. Output vs Outcome • Continuous Delivery is just an enabler • Drive to maximize outcome Maximize Value Not Output
- 15. Take-Aways 1. Continuous Delivery to accelerate your business. 2. Engineering Prerequisites • High-performance Code, full QA, continuous operations • Tooling: progressive rollout, feature flipping and A/B Test Engine. 3. Double down on communication and synchronization between teams. 4. Continuous Delivery is just about the plumbing. Make sure you have liquid gold flowing through it.
- 16. Questions Try Dashlane and get 6 months premium with the code UXDX2019
Editor's Notes
- July 26 2018. This date does not mean anything to you. It matters to me because this is the day we launched Dashlane 6, a massive release packed with new stuff for our users, supporting an important marketing campaign. On that day, we pressed the button in our Back-Office, and almost instantaneously cool new features became available to millions of Dashlane users. From an Engineering perspective, there was no rush hour, no scrambling to make sure everything was ready at the last minute. Code had been shipped progressively in the past weeks and months, hidden behind feature-flips. We had pre-tested with our beta users, we had A/B tested where needed ahead of time, so we could have a smooth ride on D-Day. This was the GOOD of Continuous Delivery, an enabler for our Product, our Marketing and our Business.
- But let me give you a refresher first. What is Continuous Delivery? My own definition is the following. It is an engineering practice that aims at having the ability to deliver to production, fast, reliably and on-demand, through an industrialized automated Release Pipeline.
- To achieve Continuous Delivery for an Engineering Team, you need 3 main prerequisites. First and foremost, high-performing code practices. Your code must be centralized in a single code repository, orchestrated thanks to a strong Continuous Integration platform (this is the core of your engineering system). It is important to support code quality through automated tools and systematic mandatory code reviews. The key metric here is how fast can a line of code move through your pipeline of delivery, meaning you need to aim for short build times, efficient notification mechanisms, high reaction time from the team. A healthy Code Review practice has a lot of implication for an engineering group, its efficiency and its morale. If developers don’t align on the expectations between them, you generate a lot of waiting time. One developer will commit some code and then be blocked by his peers as they review. We learnt the hard way, that it is key for developers to submit small amounts of code that can be reviewed quickly, to keep a smooth flow. Code review should be one of the top priority activities for a developer.
- Second in line is continuous QA. Ensure you have full-stack testing: unit testing of the code, integration testing of your various components and modules, as well as functional testing. This is called the pyramid of tests. The base, unit testing, should be fully automated with high code coverage, and then as you move up the pyramid, you should automate as much as possible, while balancing with the cost of maintaining stable test scenarios. Test Automation is a heavy investment with strong ROI on quality and stability. When I joined Dashlane, we had limited and inconsistent practices. We had to train everybody, developers, QA. We hired dedicated test automation engineers to build and maintain the test frameworks. We are way better today, but this keeps being an area of effort. As we progress, we are also able to extend test automation to more areas such as performance testing, security testing or load testing.
- Finally, when shipping to production, you must set up continuous operations, based on a monitoring platform that is as exhaustive as possible. It must include technical, product and business metrics, as well as an alerting system that will warn you in advance before your users realize a problem is happening. It is not as easy as it seems as if you don’t choose the right thresholds you will generate a lot of false positives. That’s the worse because then the team will start ignoring alerts. A good practice that we have is what we call “Morning Gymnastics”. In each team, someone is in charge of proactively checking dashboards and metrics to anticipate issues and warn the rest of the team. The goal here is to fix issues before they hit our customer support.
- Obviously when you have a complex platform, this is no easy feat. From our backend, our semantic engine, our native applications our web clients. At Dashlane we struggle to develop and maintain our pipeline across our many ecosystems. Continuous Delivery is costly, but worth the time and money.
- Now this will only give you a pipe, where code flows. You need to open the tap at the end.
- Progressive Rollout is what you need to deliver code safely to production. Most tech platforms can support progressive rollout today, so make use of it. You can open the tap slowly, shipping your code to a small percentage of your user base, monitor if things are ok before deploying to all. On a recent iOS release, code was being rolled out to 1% of our users and we identified a spike in crashes. That one had gone unnoticed through QA because it was related to production certificate generated by Apple. We stopped the deployment, found the source, hotfixed and could resume our rollout in confidence that this had impacted a reduced number of users. It never hit customer support.
- One key tool of Continuous Delivery is Feature Flipping. Feature Flipping gives you the ability to control the activation of a feature from the back-office for a certain segment of users. At Dashlane, this is in the hands of our Product Managers. They define the type of feature-flips they want to use for each new feature or experimentation. They are in charge of deciding when the moment is right to activate a new feature to which group of users. This is what we used for the Dashlane 6 launch. We had activated the features ahead of time for our beta users but we turned on the features fully only on the launch day. One challenge with feature flipping however is the little technical debt it generates on the code. It is important to clean up the feature flipping code after the fact to avoid clutter. Feature Flips need to remain a deployment tool, they should not be used for other means: we had teams that started using it as a way to configure the application for specific populations. Don’t. Create a real configuration mechanism for your application. Other teams were using it as an A/B mechanism. Don’t either. Use a real A/B Test engine for that.
- About A/B test engines, at Dashlane, we have 2 of them. One is a third-party tool used by the marketing team to test copy and design on our website. The other one is lower level and has been built in-house to allow us to test in every compartment of our applications. A/B Test Engines are necessary to experiment in production. You are probably all familiar with A/B Test practices, so I will only share a warning about synchronizing and dissociating A/B Tests when you scale. If you have many teams running A/B tests in parallel, make sure they communicate together and synchronize. Ideally they should be testing clearly separate parts of your app to avoid biases. With every tool, it is how you use them that will matter and make an impact on the business.
- Let me tell you a story. The BAD of Continuous Delivery. Earlier this year, we released an update to our VPN feature so that Dashlane users could select a specific country. The feature had gone smoothly through phases. The marketing team launched the campaigns of communication to announce the new features. Then we started getting hundreds of tickets in Customer Support of users complaining that the feature was not available to them. It appeared that the macOS release had been stuck in Apple validation and never reached production. It is critical that the whole team is synchronized, from engineering down to marketing, and aware of the side-effects associated to Continuous Delivery practices. Of course, it gives you a lot of control and flexibility, but it means you need to be even more strict on following what’s happening between progressive rollout, activation of feature-flips or running A/B Tests. Know your release life cycle and communicate. It is the team’s responsibility to track which version is in production on each platform, what feature has been activated for which users. It does not stop with shipping the code.
- Another challenge we have encountered is related to output vs outcome. Continuous Delivery will give you the ability to have intense throughput of delivery, but this is just an enabler. By investing in continuous delivery, you are just reducing a bottleneck. What you put in the pipe is what is important above all. Make sure your Product & Engineering organization maximizes for impact and outcome not for output. That technical capability and those tools are necessary but not sufficient to make an impact on the business.
- If you put crap in, you will get crap out.
- 4 Take-Aways as a conclusion: Investing into Continuous Delivery practices will give you the ability to accelerate your business. It requires deep engineering practices, around code, QA and operations, as well as appropriate tooling: progressive rollout, feature flipping and an A/B Test Engine. As you accelerate, make sure you double down on communication and synchronization between teams. Continuous Delivery is just about the plumbing. Make sure you have liquid gold flowing through it.
- Thank you.