Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (12)
Similaire à Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer
Similaire à Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer (20)
Plus de JAXLondon_Conference
Plus de JAXLondon_Conference (20)
Dernier
Dernier (20)
Smoothing the continuous delivery path – a tale of two teams - Lyndsay Prewer
- 1. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Smoothing the continuous delivery path A tale of two teams
- 2. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp What’s this all about? 2 Continuous Delivery Done Well
- 3. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp What’s this (Continuous Delivery) all about? 3 “... producing valuable software in short cycles and ensure that the software can be reliably released at any time”- https://en.wikipedia.org/wiki/Continuous_delivery
- 4. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Continuous doughnuts 4 Fast feedback Risk Reduction T: 0:00:000 T: 2:00:000
- 5. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp There’s no (CD) silver bullet 5 1 (php) monolith 50 deploys / day 1.5 billion page views / month http://goo.gl/XIB29i >600 java microservices >100 deploys / day ~2 billion requests per day http://goo.gl/ZJBiK6
- 6. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Prefer Continuous Improvement over Continuous Delivery ▪ .Net monolith ▪ 26 -> 24 -> 34 releases per year ▪ https://goo.gl/KF9xXw 6
- 7. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp 7 People first, practices/process second, products last
- 8. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp .Net monolith team - how they roll 8 ▪ Private sector ▪ Significant legacy code ▪ High daily traffic (~100 million visits) ▪ ~8 product teams ▪ Weekly deployments: – Pick release candidate – Regression test – Dog food – Deploy
- 9. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Scala microservices team - how they roll 9 ▪ .Gov - HMRC Digital ▪ 2 years old (greenfield) ▪ Calendar spikes (Visits peak is ~4 million) ▪ >30 product teams ▪ Multiple deployments per day – Teams “own” their own services – “You build it, you run it” - https://goo.gl/DyfeOf
- 10. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Continuous Delivery Done Well Patterns for successful practice 10
- 11. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Best practice #1 - Healthy CI - mind & body 11
- 12. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Best practice #1 - Healthy CI - mind & body 12 Almost green!
- 13. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Best practice #2 - Testing as an activity, not a phase 13 Product Owner Define Dev Build QA Test Web Ops Deploy
- 14. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Best practice #2 - Testing as an activity, not a phase 14 Product Owner Define Dev Build QA Test Web Ops Deploy Test
- 15. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Best practice #3 - Healthy automated tests 15 Unit Tests API UI
- 16. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Best practice #3 - (Un)healthy automated tests 16 API UI Unit Tests The “Hour Glass”
- 17. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp UI Best practice #3 - (Un)healthy automated tests 17 API The “Ice Cream Cone” Unit Test s
- 18. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Best practice #3 - Healthy automated tests 18 Unit Tests API UI The “Tear Drop”
- 19. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Best practice #4 - Low (manual) cost deployment (and rollback) 19 VS.
- 20. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Best practice #5 - Metrics, monitoring and alerting 20 http://seluxkanaur.deviantart.com/art/Sauron-s-Eye-6-420535318 “Big data” Social media User behaviour Business metrics Service performance Service errors Disk and Network Monitor all things!
- 21. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Metrics, monitoring and alerting - getting it right 21 Detect Alert Respond Display Analyse Time & Pain increasing!
- 22. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Metrics, monitoring and alerting - getting it wrong 22 Detect Respond Display Analyse Time & Pain increasing! Alert
- 23. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Metrics, monitoring and alerting - where and how? 23 Detect Alert Respond Display Analyse
- 24. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Metrics, monitoring and alerting - stories from the trenches 24 Detect Alert Respond Display Analyse
- 25. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Best practice summary 1. Healthy CI (mind and body) 2. Testing as an activty, not a phase 3. Tear drop shape automated tests (focus on behaviour, with loose coupling) 4. Low (manual) cost deployment and rollback 5. Metrics, monitoring and reporting 25 Product Owner Define Dev Build QA Test Web Ops Deplo y Test Unit Tests API U I Detect Alert Respond Display Analyse
- 26. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Shared Pain 26 Continuous Delivery Done Well
- 27. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Shared pain #1 - Zero down-time releases 27 https://en.wikipedia.org/wiki/VO2_max
- 28. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Shared pain #2 - Manual deployment steps 28 https://en.wikipedia.org/wiki/Tailor https://en.wikipedia.org/wiki/Zara_(retailer) VS.
- 29. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Shared pain summary 1. Zero down time releases 2. Manual deployment steps 29 VS.
- 30. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Team specific accelerators 30 Continuous Delivery Done Well
- 31. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Accelerators - 5 why’s 31 ● Create a timeline. ● Gather those involved. ● Walk through the timeline. ● Choose an entry point event: ○ Event ○ Why? ○ Process improvement ● Repeat x5 (ish)
- 32. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Accelerators - 5 why’s - example 32 Event Why? Improvement Error threshold alert triggered. Calls to service X were timing out. Implement failover / circuit breaker. Calls to service X were timing out. Queries run by Service X were taking too long. Alert on long running queries. Queries run by Service X were taking too long. Index was missing for a new field. ... ... ... ...
- 33. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Accelerators - automated release notes - the problem #1 33 Much confusion and pain
- 34. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Accelerators - automated release notes - the problem #2 34 Agile Project Management Tool Source control system Features Stories Defects Login Reporting Payments API’s
- 35. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Accelerators - automated release notes - the solution 35 Agile Project Management Tool Source control system Features Stories Defects Login 1. Identify source changes 2. Identify stories/defects 3. Identify areas 4. Group by story/defect 5. For each, summarise areas and authors 6. Group by: a. What’s done b. What’s in progress Reporting Payments API’s
- 36. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Team specific accelerators 36 Continuous Delivery Done Well
- 37. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Accelerators - teams “own” their own microservices 37 ▪ “You build it, you run it” - https://goo.gl/DyfeOf ▪ Prefer service over library dependencies
- 38. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Accelerators - backward compatible API’s (principle of least surprise) 38 https://en.wikipedia.org/wiki/VO2_max Production Staging QA Dev Everyone shares (and cares)
- 39. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp Team specific accelerators summary Monolith: 1. 5 -why’s 2. Automated release note Microservices: 1. Microservice dependencies, not library dependencies 2. Backward compatible API’s 39 Agile Project Management Tool Source control system Features Stories Defects Login Reporting Payments API’s
- 40. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp References and resources 40 ▪ Build Quality In (promo code - LP2015 - https://goo.gl/ZRuQKw) Continuous Delivery Implementing Lean Software Development Release It
- 41. © Equal Experts UK Ltd and lyndsayp ltd 2015@EqualExperts @lyndsp simple software solutions to big business problems. Making Software. Better. Thank You Twitter @EqualExperts @lyndsp LinkedIn linkedin.com/company/equal-experts uk.linkedin.com/in/lyndsp UNITED KINGDOM +44 203 603 7830 helloUK@equalexperts.com Equal Experts UK Ltd 30 Brock Street London NW1 3FG INDIA +91 20 6607 7763 helloIndia@equalexperts.com Equal Experts India Private Ltd Office No. 4-C Cerebrum IT Park No. B3 Kumar City, Kalyani Nagar Pune, 411006 Web www.equalexperts.com www.lyndsayp.com CANADA +1 403 775 4861 helloCanada@equalexperts.com Equal Experts Devices Inc 205 - 279 Midpark way S.E. T2X 1M2 Calgary, Alberta PORTUGAL + helloPortugal@equalexperts.com Equal Experts Portugal Rua Tomás da Fonseca - Torres de Lisboa Torre G, 5º Andar 1600-209 Lisboa Thank You
Notes de l'éditeur
- Introduce myself (lyndsayp Ltd and EE) Roles Over 30 teams working within a Scala microservices architecture, releasing into Production multiple times per day This session is an experience report of the best practices and pain points of two teams that both practice Continous Delivery. Establish: - what roles are present - how many deliver to web Show of hands - production deployment capability: one or more per year, per quarter, per month, per week, per day
- Two very different teams Government Scala microservices Private sector .Net Monolith Both teams are doing a great job of getting their doughnuts (of differing sizes and shapes) out to Production. This is an experience report will compare and contrast What best practices help these team go faster, faster What areas of pain slow them down
- Emphasise: Fast feedback Risk reduction
- Valuable software in short cycles reliably released at any time Fast feedback - shortest time from concept to cash Risk reduction - smallest value possible
- Whilst preparing this session, I read a fascinating blog post on how another highly successful company, called targetprocess, has evolved over the last 90 months. The article describes, in gory detail, their journey over the last 7 years, through various agile best practices. Over the last three years they’ve improved their delivery frequency, but still have some way to go. They’ve made lots of improvements in other areas though. The article reminds us that when improving one’s Continuous Delivery, it’s important not to do so at the expense of other practices. The overarching goal has to be Continuous Improvement, not Continuous Delivery.
- HMRC Digital (Public sector) Gov.uk tax interfaces for citizens and businesses (70% of all gov transactions, > 1 billion transactions online) Traffic varies across the year, with peaks at key tax calendar events 30+ Product Teams Tech Stack Scala Microservices Docker Mongo DB Deployment frequency You build it, you run it - Amazon CTO Werner Vogels
- CI that works Culture - Mind share
- CI that works - knowing what to automate - a story about .Net Monolith
- Cross functional teams Picture of team mix Picture of testing all the time Story about E
- Cross functional teams Picture of team mix Picture of testing all the time
- The test automation pyramid API and/or integration tests
- Picture different test mixes Common automated testing problems
- Picture different test mixes Common automated testing problems Picture different test mixes, highlighting problems
- Tear drop shape Behavioural focus
- Story about E, then .Net Monolith Story about HMRC (rollback)
- Eye of Sauron, looking at the whole stack (user behaviour, down to disk space) Highlight: What to focus on (behaviour instead of environment) Leading vs. trailing indicators
- Detect Alert Respond Display Analyse Diagram of right and wrong mix of the above
- Detect Alert Respond Display Analyse Diagram of right and wrong mix of the above
- Tell a story Mongo query defect at HMRC Kibana Change in success rate Kibana / splunk
- Making changes to Running machine Stories and Diagram of Cookie change Database change
- Story about no more manual scripts / steps
- Microservice architecture Amazon quote - Build It, Run It, Own it Separation of features and concerns Prefer service dependencies over library dependencies
- Principle of least surprise Service API's must always be backwardly compatible Story of when we were young, teams wanted to test against fixed versions of other services
- Promote Build Quality In Excellent chapter in ILSD on VSM’s