BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!
•Télécharger en tant que PPTX, PDF•
0 j'aime•1,058 vues
This is about leveling-up and REVOLUTIONIZING Testing as part of your Agile/DevOps Transformation. You can contribute more than testing functionality. You need to Level-Up your skill set by understanding the apps you are testing. # Images, # JS Files, # SQL Statements, Connection Pool Utilization and Garbage Collection Activity have to be added to your portfolio. Check these metrics when you do your functional testing and report regressions to your engineers even though the functionality is still good. But you just uncovered an Architectural regression that will lead to a scalabilty and performance problem. Finding these problems early will eliminate a lot of wasted and unplanned time later on in the lifecycle. that is your contribution to delivering software faster with better quality
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!
Similaire à BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs! (20)
Plus de Andreas Grabner
Plus de Andreas Grabner (17)
BTD2015 - Your Place In DevTOps is Finding Solutions - Not Just Bugs!
- 1. Andreas Grabner Your place in DevTOps is not about finding more bugs, but problem solutions
- 2. That’s why I ended up talking at BTD
- 6. The stuff we did when we were a Start Up and we All were Devs, Testers and Ops
- 8. Our Goal?
- 9. Utmost Goal: Minimize Cycle Time feature cycle time time Customer / Market Users
- 10. Utmost Goal: Minimize Cycle Time feature cycle time time minimize Users
- 11. Utmost Goal: Minimize Cycle Time feature cycle time time This is when you create value! minimize You
- 12. Release Acceptance Testing Unit Testing Performance Testing Coding Reduce Lead Time Automate & Improve Process
- 13. 700 Deployments / Year 50-60 Deployments / Day 10+ Deployments / Day Every 11.6 seconds
- 14. Inside the Amazon Numbers! 75% fewer outages since 2006 90% fewer outage minutes ~0.001% of deployments cause a problem Instantaneous automatic rollback Deploying every 11.6s
- 15. Challenges?
- 16. Deploy Faster!!
- 17. Fail Faster!!?
- 18. Right Focus?!
- 19. Failures happen!! Nobody likes it when …
- 20. Unless you work for Google or Microsoft
- 21. … or this …
- 23. .. as it leads to this …
- 24. The “War Room” Facebook – December 2012
- 25. … and potentially to this …
- 27. And this isn’t helping either …
- 32. Tester Developer
- 35. This ATTITUDE is the main PROBLEM as it LEADS to …
- 36. 80% $60B
- 39. 20% 80%
- 42. 4 Use Cases: WHY, HOW & METRICS
- 45. Mobile Web Site: This SHOULDN’T happen! 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB
- 47. Page Size Item Size # Objects
- 48. • Developers not using the browser built-in diagnostics tools • Testers not doing a sanity checks with the same tools • Some tools for you (Quick Demo) • Built-in Inspectors via Ctrl-Shift-I in Chrome and Firefox • YSlow, PageSpeed, SpeedTracer • Dynatrace • Level-Up: Automate Testing and Diagnostics Check Lessons Learned – NO Excuse for …
- 50. Not every Architect makes good decisions
- 51. • Symptoms • HTML takes between 60 and 120s to render • High GC Time • Developer Assumptions • Bad GC Tuning • Probably bad Database Performance as rendering was simple • Result: 2 Years of Finger pointing between Dev and DBA Project: Online Room Reservation System
- 52. Developers built own monitoring void roomreservationReport(int officeId) { long startTime = System.currentTimeMillis(); Object data = loadDataForOffice(officeId); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, officeId); } Result: Avg. Data Load Time: 45s! DB Tool says: Avg. SQL Query: <1ms!
- 53. #1: Loading too much data 24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory
- 54. #2: On individual connections 12444! individual connections Classical N+1 Query Problem Individual SQL really <1ms
- 55. #3: Putting all data in temp Hashtable Lots of time spent in Hashtable.get Called from their Entity Objects
- 56. # SQL Executions # of SAME SQLs Conn. Acquisition Time
- 57. • … you know what code is doing • Challenge the developers • Explore Tools that “might seem” out of your league! • Built-In Database Analysis Tools • “Logging” options of Frameworks such as Hibernate, … • JMX, Perf Counters, … of your Application Servers • Performance Tracing Tools: Dynatrace, NewRelic, AppDynamics, … Lessons Learned – Don’t Assume …
- 60. Test Environment Production Environment 8x slower 3x more SQL
- 61. Test Environment Production Environment Hibernate, Classloading, XML – The Key Hotspots Hibernate, Classloading, XML – The Key Hotspots I/O for Web Requests doesn’t even show up! That’s Normal: Having I/O for Web Request as main contributor
- 62. Top Contributor Class.getInterfaces Called from Hibernates FieldInterceptionHelper These calls all originate form thousands of calls to find item by code
- 63. Top Methods related to XML Processing Classloading is triggered through CustomMonnkey and the Xalan Parser Classloading is triggered through CustomMonkey and the Xalan Parser
- 64. Time Spent in API # Calls to API
- 65. • Plan enough time for proper testing • Anticipate changed user behavior during peak load • Only test what really ends up in Production Lessons Learned
- 68. Load Spike resulted in Unavailability Adonair
- 69. Alternative: “GoDaddy goes DevOps” 1h before SuperBowl KickOff 1h after Game ended
- 70. # of Domains Total Size
- 71. • Share your Performance Expertise with Developers • Implement & Test these “Feature Switch” Scenarios • Demand Live Production Data for Future Projects • Read Up & Educate yourself on more stories like this Lessons Learned
- 72. •http://blog.dynatrace.com •http://www.perfplanet.com/ •http://highscalability.com/ •http://blog.ruxit.com/ More of these Use Cases?
- 74. •# Images •# Redirects •Size of Resources •# SQL Executions •# of SAME SQLs •# Items per Page •# AJAX per Page Remember: New Metrics When Testing Apps •Time Spent in API •# Calls into API •# Functional Errors •3rd Party calls •# of Domains •Total Size •…
- 76. Level-Up Skills Browser Diagnostics Wireshark, Fiddler JBoss, Tomcat, JConsole IIS, ASP.NET, PerfLib Oracle, SQL Garbage Collection Thread Dumps
- 77. Architecture # of Services # of Databases # of Servers # of Calls between Components
- 78. Performance Page Load Time Render Time SQL Query Time Service Call Time
- 79. Scalability Memory Usage per User # Connections Cache Utilization Load Distribution Component Roundtrips
- 80. Collaboration Automation Sharing Measuring Level-Up Skills Performance Scalability Architecture 4 Pillars of DevOps + Quality Focus
- 82. Putting it into Test Automation 12 0 120ms 3 1 68ms Build 20 testPurchase OK testSearch OK Build 17 testPurchase OK testSearch OK Build 18 testPurchase FAILED testSearch OK Build 19 testPurchase OK testSearch OK Build # Test Case Status # SQL # Excep CPU 12 0 120ms 3 1 68ms 12 5 60ms 3 1 68ms 75 0 230ms 3 1 68ms Test Framework Results Architectural Data We identified a regresesion Problem solved Exceptions probably reason for failed tests Problem fixed but now we have an architectural regression Problem fixed but now we have an architectural regressionNow we have the functional and architectural confidence Let’s look behind the scenes
- 83. #1: Analyzing each Test #2: Metrics for each Test #3: Detecting Regression based on Measure
- 84. #1: Test Status Overview based on our new Metrics #2: Lets the build fail
- 85. Release Acceptance Testing Unit Testing Performance Testing Monitor Tests Analyze Results Quality Gate in your Build Tool Every 11.6 seconds
- 86. Deploy Faster!! With Better Quality
- 87. Andreas Grabner Your place in DevTOps is not about finding more bugs, but problem solutions Slides: slideshare.net/grabnerandi Get Tools: bit.ly/dttrial YouTube Tutorials: bit.ly/dttutorials Contact Me: agrabner@dynatrace.com Follow Me: @grabnerandi Read More: blog.dynatrace.com
- 90. Incorrect Sizing of Pools and Queues
- 91. Online Banking: Slow Balance Check 1.69m (=101s!) To Check Balance! 87% spent in IIS 600! SQL Executions
- 92. #1 Time really spent in IIS? Tip: Elapsed Time tells us WHEN a Method was executed! Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET Tip: Thread# gives us insight on Thread Queues / Switches
- 93. #2 What about these SQL Executions? Finding: EVERY SQL statement is executed on ITS OWN Connection! Tip: Look at “GetConnection”
- 94. #2 SQL Executions! continued … #1: Same SQL is executed 67! times #2: NO PREPARATION because everything executed on new Connection
- 95. Lessons Learned! ASP.NET Worker Thread Pool Sizing! DB Connection Pools More Efficient SQL
- 96. How to Monitor: Web Server Idle vs. Busy Threads
- 97. How to Monitor: Application Server App Server Threads
- 98. Idle vs. Busy Threads # SQLs / Request # GetConnection
- 100. Collaboration Automation Sharing Measuring 4 Pillars of DevOps + Quality Focus
Notes de l'éditeur
- And that’s my professional background
- Its not a hoax or hype – DevOps is a real thing. Especially driven by the release of books like “The Phoenix Project”, Velocity conference with a shift in focus to DevOps and many companies that jumped on the bandwaggon to build tools and offer services around DevOps
- So – some call it DevOps - …
- Some say it is simply the natural evolution of software engineering where we push stuff principles of Lean, Agile and Continuous Integration further to production
- And here is my definition of DevOps
- The other trend is the DevOps movements that tries to help us here. Heavily promoted and pushed by several people, organizations, books and conferences If you havent read the phoenix project please do so. Also make sure you are getting up to speed with concepts such as Continuous Delivery and doing this in an efficient way – as this is what we need to do in order to comply with the rapidly changing requirements
- Cycle time is the most relevant metric in the software delivery process. “How long would it take your organization to deploy a change that involves just one single line of code?” Mary Poppendieck
- Cycle time is the most relevant metric in the software delivery process. “How long would it take your organization to deploy a change that involves just one single line of code?” Mary Poppendieck
- Cycle time is the most relevant metric in the software delivery process. “How long would it take your organization to deploy a change that involves just one single line of code?” Mary Poppendieck
- The key goal that people want to achieve is to Reduce Lead Time. An automated build pipeline plays a huge role in it as we get rid of a lot of manual tasks that otherwise hold up the process When pushing out features faster it is important to also close the feedback loop to constantly improve the process and quality of the developed software
- Several companies changed their way they develop and deploy software over the years. Here are some examples (numbers from 2011 – 2014) Cars: from 2 deployments to 700 Flicks: 10+ per Day Etsy: lets every new employee on their first day of employment make a code change and push it through the pipeline in production: THAT’S the right approach towards required culture change Amazon: every 11.6s Remember: these are very small changes – which is also a key goal of continuous delivery. The smaller the change the easier it is to deploy, the less risk it has, the easier it is to test and the easier is it to take it out in case it has a problem.
- So – our goal is to deploy new features faster to get it in front of our paying end users or employees
- For many companies that tried this it may also meant that they fail faster
- Its also very important to keep the focus right – building and fixing those things that matter.
- Your app that you are responsible for crashes …
- The Fifa World Cup App one week before the worldcup. Crashed for the majority of Android users when refreshing the news section of the app caused by a memory leak introduced by an outdated library they used
- The classical war room – this is where most of the engineers trying to figure out what is wrong for hours or even days instead of building new stuff
- Bad press! Something you don’t like either if you want to sell your software or service
- Bad comments on the Google Play store for the Fifa App. If you invest a lot of time & money this is not going to help you harvest the investment as many people will probably just download an app from the competition
- But Why does this really happen?
- A lot of time developers and testers see the world from a DIFFERENT POINT of View as well as having different priorities: Developers want to build features and not fix bug that keeps them from bulding new features. Testers need to find bugs because that’s what they are measured on – so – this is a natural conflict in common goals. Also – the – “It works on my machine” attitude comes up a lot. Devs e.g: don’t trust the Testing Tools Same is true for Operations and Developers, Operations and Testers, Business and Operations, …. – there are always conflicts in interests and goals and there is always a trust issue with the tools used and the data captured
- As tester it often feels like you are the lonely ranger that is not scared of these bugs but the rest of the organization is as it jeopardizes the release schedule
- The close we get to release it seems like developers are really chased by testers. But they try to escape to finish the last user stories they committed to – which his also more fun than fixing bugs
- At release date – with not all bugs fixed it just happens that people simply deploy and let end users do the testing in production
- This attitude doesn’t help either
- Based on a recent study: 80% of Dev Team overall is spent in Bugfixing instead of building new cool features $60B annual costs of bad software instead of investing it in new cool features to spearhead competition
- We need to leave that status quo. And there are two numbers that tell us that it is not as hard to do as it may seem
- Based on my experience 80% of the problems are only caused by 20% problem patterns. And focusing on 20% of potential problems that take away 80% of the pain is a very good starting point
- Sounds super nice on paper – so – how do we get there?
- Now we are talking about several real life use cases from applications that crashed when deployed into production. For each use case we are looking into Why the app failed -> technical root cause which metrics to look at to identify the actual architectural, performance or scalability problem How you can level up to identify these problems and also automate that along the delivery pipeline
- So – we have seen a lot of metrics. The goal now is that you start with one metric. Pick a single metric and take it back to your engineering team (Dev, Test, Ops and Business). Sit down and agree on what this metric means for everyone, how to measure it and also how to report it Also remember that for most of these use cases discussed and metrics derived from it we only need a single user test. Even though we can identify performance, scalability and architectural issues – in most cases we don’t need a load test. Single user tests or unit tests are good enough
- Level-Up Skills by talking and exchanging ideas for your piers: developers, ops, business It is important that both sides start understanding the challenges of the other side. It is important that they speak the same language, e.g: what does this metric mean to you? How do you measure it? Sit down and level-up skills for everybody and agree on a common set of tools and metrics But what types of metrics?
- Metrics around Architecture, e.g: how many web service calls does it take to implement this new feature? How many AJAX calls do we make when people logon to our site? Is that smart?
- How fast is that piece of code? Is it efficient in its usage of CPU, Memory, Disk and Network?
- Is the application also going to scale? Which components perform better or worse with increasing load? Where is the breaking point? Where is the API that is the issue? Whats the architectural decision behind the app not scaling?
- If we add these 4 quality areas we convert DevOps into DevTOps We change testers from creating bugs to help finding solutions as a team!
- Once we figured out how to get these measures it is time to automate the capturing but also automate quality alerting in case these metrics are showing us that we ran into one of these well known use cases.
- Here is how we do this. In addition to looking at functional and unit test results which only tell us how functionality is we also look into these backed metrics for every test. With that we can immediately identify whether code changes result in any performance, scalability or architectural regressions. Knowing this allows us to stop that build early
- This is how this can look like in a real life example. Analyzing Key Performance, Scalability and Architectural Metrics for every single test
- This is how this could look like in Jenkins
- Now as we know which metrics we need to look at and how to automate the capturing and detect regressions from build to build we simply add it to the continuous delivery pipeline by letting these metrics act as quality gateways. We do not let a build move forward if we already know that it has a well known problem. Here are all the benefits Only good code reaches production We eliminate time spent in later stages if we already identify problems earlier We all level up our skills and become a better team We produce better software faster -> we don’t crash the car
- The problem is that there is not enough focus on code quality which leads lots of bugs ending up in production. We also know that it is more expensive and time consuming to fix a bug in production rather than earlier in the lifecycle This is exactly what leads to the costs and time spent in fixing issues
- So – if we focus on these well known problems early – if we even automated this process– we eliminate A lot of wasted time with testing of bad code that was now already eliminated earlier Better overall quality in production Fewer Bugs overall that have side-effects and a better general understanding of quality leads to lower costs on fixing bugs
- We start with CAMS – which is the base foundation of the initial Definition of DevOps But we add new areas around Quality Focus