Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Metrics Driven Development 
How to improve visibility, 
communication and feedback loop 
through metrics 
Erno Aapa @ernoa...
ERNO AAPA 
Developer / Team Leader 
DevOps-consultant 
Founder of DevOps-Finland 
Daily work as a developer. One of Avaus ...
Measurement is one of the keys 
Everyone emphasize the importance of it
Three Ways: Feedback 
Feedback loop 
4 11/09/14
M in C.A.L.M.S 
Measure everything 
Show the 
improvement 
5 11/09/14
Build-MEASURE-Learn 
Measure and learn 
6 11/09/14
… but in reality… 
we need it, but we don’t have it
What we have 
Business 
Monthly / Yearly 
reports 
Financial reports 
Google Analytics 
8 11/09/14 
Dev 
? 
Ops 
CPU, Mem,...
What that cause? 
• Bad visibility 
Ops-team tries to get the information from logs, load metrics, etc. 
Business doesn’t ...
Metrics Driven Development (MDD) 
“…is a practice where metrics are used 
to drive the entire application 
development” 
I...
Principles 
Define metrics before implementation 
of Metrics Driven Development (MDD) 
• Like in TDD implement test first,...
User story 
Define what to measure and what we expect to user story 
As a user I can order items in my shopping cart. 
We ...
Collect required metrics 
Developer adds required code to collect metrics 
# Somewhere in your where login happens... 
sta...
Share visibility 
Visualize metric and also what we expect 
14 11/09/14 
Visualize what we 
are expecting. Easy 
to everyo...
Single place of truth 
Provide single place for data to easily compare and analyze 
15 11/09/14 
Frontend Service Monitori...
Shared dashboard for key metrics 
Make most important metrics visible to everyone - all the time 
16 11/09/14
In the meeting 
Use the information when you’re making decisions 
• What features have went to live? 
• Did we achieve wha...
PRODUCT OWNER / STAKEHOLDERS 
- Get real time information from 
production 
- Can make data driven decisions 
- Support Le...
Communication 
19 11/09/14 
Ops 
Business 
Dev 
Metrics 
Shared view and 
goals helps 
communication 
Benefits of MDD
Imaginary case 
Simple Java web shop
My Webshop 
21 11/09/14 
Web 
shop 
UI 
User orders items 
through Web UI 
Server Graphite Grafana 
After processing the 
...
By developer 
22 11/09/14 
1. Create counter 
2. Increment counter 
THATS 
ALL!
By developer 
23 11/09/14 
SEE 
THE 
RESULTS! 
Type the metric 
name and…
… when you get started 
Gauges 
• Queue sizes 
Timers 
• Query times 
• Response times 
Counters 
• Execution counters 
Hi...
Pitfalls 
”Metrics are always powerful political ammunition, to be used for better or 
worse” 
Metrics-driven Enterprise S...
What tools to use? 
To collect, store, display? SaaS maybe?
27 11/09/14 
TOOLS WHY 
COLLECT • StatsD (multiple languages) 
• Code Hale Metrics (Java) 
• Easy to implement by self 
• ...
Graphite vs. InfluxDB 
Key / Value Send data Json 
echo “dc1.server2.cpuload 5.6" | nc graphite.com 
echo “app.visitors 1"...
Graphite vs. InfluxDB 
average(dc1.*.cpuload) 
sum(app.visitors) 
Not possible :( 
// Top 5 servers with highest load 
hig...
Graphite vs. InfluxDB 
Summary 
• Older project 
• Harder to install / configure 
• Over 100 aggregation functions 
sum, c...
What next? 
That’s cannot be all…
What next - Annotations 
What happen at 19:27? There were ad in the TV 
32 11/09/14 
Add annotations automatically when 
s...
What next - Alerts 
• Have a easy way to add new alerts 
• Don’t have to be a major alerts, small “reminders” are ok too 
...
Easy to get started, so why not? 
Try it today, create a demo to your team, convince them 
• Next time plan how you could ...
Questions? 
Thank you! 
Erno Aapa @ernoaapa 
Developer, DevOps-consultant 
Erno.aapa@avaus.fi 
10.09.2014
Prochain SlideShare
Chargement dans…5
×

Metrics driven development 10.09.2014

2 226 vues

Publié le

Presentation about Metrics Driven Development at DevOps-Finland meeting in Helsinki 10.09.2014.

Publié dans : Ingénierie
  • Identifiez-vous pour voir les commentaires

Metrics driven development 10.09.2014

  1. 1. Metrics Driven Development How to improve visibility, communication and feedback loop through metrics Erno Aapa @ernoaapa Developer, DevOps-consultant erno.aapa@avaus.fi DevOps-Finland meetup Helsinki 10.09.2014
  2. 2. ERNO AAPA Developer / Team Leader DevOps-consultant Founder of DevOps-Finland Daily work as a developer. One of Avaus tech team leaders and consult companies about DevOps. On free time organize DevOps-Finland meetings. Who am I? 2 11/09/14
  3. 3. Measurement is one of the keys Everyone emphasize the importance of it
  4. 4. Three Ways: Feedback Feedback loop 4 11/09/14
  5. 5. M in C.A.L.M.S Measure everything Show the improvement 5 11/09/14
  6. 6. Build-MEASURE-Learn Measure and learn 6 11/09/14
  7. 7. … but in reality… we need it, but we don’t have it
  8. 8. What we have Business Monthly / Yearly reports Financial reports Google Analytics 8 11/09/14 Dev ? Ops CPU, Mem, IO, Disk Zabbix/Ganglia/Nagios QA Test coverage reports CI server notifications Performance test results
  9. 9. What that cause? • Bad visibility Ops-team tries to get the information from logs, load metrics, etc. Business doesn’t have detailed visibility only high level reports • Dev “just do my job” They don’t feel the “heartbeat” of the service • Invalid conception of service state “No-one complains so it works… I guess :/ ” • “Business bug” – feature created but not used Service full of unused legacy features 9 11/09/14
  10. 10. Metrics Driven Development (MDD) “…is a practice where metrics are used to drive the entire application development” InfoQ / 2012
  11. 11. Principles Define metrics before implementation of Metrics Driven Development (MDD) • Like in TDD implement test first, implement metric first • For example: Include metrics to user stories Instrumentation-as-Code • Developers must be able to add new metric with minimal(=one line) effort Single Source Of Truth • Store common metrics from app, logs, monitoring agents and other tools to single place • Platform should be so timely, comprehensive, and intuitive to use that everyone instinctively relies on it Shared view for key metrics • Shared view to give all same vision • So simple that everyone can understand it Use metrics when making decisions • You have powerful information, use it when you’re making decisions Maintain and follow the metrics • Follow the metrics - now popular feature can be waste tomorrow • Remove unneeded metrics – it’s code, it need to be maintained 11 11/09/14 Erno Aapa / 2014 Librato Blog / 2014 InfoQ / 2012
  12. 12. User story Define what to measure and what we expect to user story As a user I can order items in my shopping cart. We measure visited users and count of orders. We expect to at least 20% of users to order items in shopping cart. 12 11/09/14 Can be defined in EPIC or FEATURE too
  13. 13. Collect required metrics Developer adds required code to collect metrics # Somewhere in your where login happens... statsd.increment("users.visited”) 13 11/09/14 # ...and where you handle the orders statsd.increment(”shopping.ordered”) Somefile.rb OtherFile.rb
  14. 14. Share visibility Visualize metric and also what we expect 14 11/09/14 Visualize what we are expecting. Easy to everyone understand what is good or bad. Use this information when making decisions
  15. 15. Single place of truth Provide single place for data to easily compare and analyze 15 11/09/14 Frontend Service Monitoring
  16. 16. Shared dashboard for key metrics Make most important metrics visible to everyone - all the time 16 11/09/14
  17. 17. In the meeting Use the information when you’re making decisions • What features have went to live? • Did we achieve what wanted? • Should we continue to improving them? • Or should we remove them? • What is our overall status? • What we do next to achieve our goal? 17 11/09/14
  18. 18. PRODUCT OWNER / STAKEHOLDERS - Get real time information from production - Can make data driven decisions - Support LeanStartup way DEVELOPER - Get the “feeling” and visibility to production - Drop unused features - Can focus on important parts TEST MANAGER - Get visibility to production - Can scale tests to match with production SYSADMIN - Get visibility to inside of the app - In case of problems easier to find the error Everyone benefit from it 18 11/09/14 just to name few…
  19. 19. Communication 19 11/09/14 Ops Business Dev Metrics Shared view and goals helps communication Benefits of MDD
  20. 20. Imaginary case Simple Java web shop
  21. 21. My Webshop 21 11/09/14 Web shop UI User orders items through Web UI Server Graphite Grafana After processing the order, server sends data to Graphite Use Grafana to visualize data.
  22. 22. By developer 22 11/09/14 1. Create counter 2. Increment counter THATS ALL!
  23. 23. By developer 23 11/09/14 SEE THE RESULTS! Type the metric name and…
  24. 24. … when you get started Gauges • Queue sizes Timers • Query times • Response times Counters • Execution counters Histograms • median, 75th, 90th, 95th, 98th, 99th 24 11/09/14 Measure time with single @annotation There is so many things what you can measure, if you have the capability
  25. 25. Pitfalls ”Metrics are always powerful political ammunition, to be used for better or worse” Metrics-driven Enterprise Software Development- book ”Tried a solution where OPS and DEV were sharing one metrics server… it didn't work” Mantas Klasavicius / Adform ”MONITOR ALL THE THINGS! No, you should not wrap every single method call in your application to increment a counter” Librato blog
  26. 26. What tools to use? To collect, store, display? SaaS maybe?
  27. 27. 27 11/09/14 TOOLS WHY COLLECT • StatsD (multiple languages) • Code Hale Metrics (Java) • Easy to implement by self • Provide ”one-line” –way to developer collect any metrics from application • Possible to change storage from configuration STORE • Graphite • InfluxDB • … and more • Store time-series data • Provides easy accessable API • Aggrigate data on ”on the fly” • Downsample old data DISPLAY • Grafana • Tesseo • … and more • Simple to use • Simple and clear graphs • Possible to create multiple dashboards SaaS • Librato.com • geckoboard.com • leftronic.com • HostedGraphite.com • Influxdb.com • …and many more • Easy to get started • Maintained • Free to test Give a demo and convince others!
  28. 28. Graphite vs. InfluxDB Key / Value Send data Json echo “dc1.server2.cpuload 5.6" | nc graphite.com echo “app.visitors 1" | nc graphite.com 2003 28 11/09/14 "name" : “cpu_load", "columns" : ["value", ”dc", ”server"], "points" : [ [5.6, ”dc1", ”server2”] ] "name" : “visitors", "columns" : ["value”, ”browser", ”version"], "points" : [ [1, ”Chrome” , 37.0] ]
  29. 29. Graphite vs. InfluxDB average(dc1.*.cpuload) sum(app.visitors) Not possible :( // Top 5 servers with highest load highestAverage(*. *.load,5) 29 11/09/14 Aggregate data select average(value) from cpu_load where dc = ‘dc1’ select sum(value) from visitors select sum(value) from visitors where browser = ‘Chrome’ and version > 35 Probably not possible :( Not full SQL!
  30. 30. Graphite vs. InfluxDB Summary • Older project • Harder to install / configure • Over 100 aggregation functions sum, cumulative, compare, highestAverage • Good for collecting metrics • Limited “WHERE” queries 30 11/09/14 • Really young project (Apr 2013) • Easy to get started • Only basic aggregation functions(17 total) sum, min, max, median, percentile • Good for collecting events • Limited possibilities to aggregate data Don’t pick one, use both! (and don’t forget to check SaaS services too!)
  31. 31. What next? That’s cannot be all…
  32. 32. What next - Annotations What happen at 19:27? There were ad in the TV 32 11/09/14 Add annotations automatically when something happens what can change the service state. Like deployments, other events, etc.
  33. 33. What next - Alerts • Have a easy way to add new alerts • Don’t have to be a major alerts, small “reminders” are ok too • Notify developers by email when any page response time goes higher than 300ms • Notify business and developers when any feature usage goes lower than 5% 33 11/09/14 Users not using it? Feature “favourite list” usage lower than 5% It’s waste… REMOVE IT! Cleaner codebase No complex UIs Less functionality = easier to maintain
  34. 34. Easy to get started, so why not? Try it today, create a demo to your team, convince them • Next time plan how you could measure it • Add required metrics to code • Use free SaaS services to get easily started • Create nice dashboard, add the most important graph and convince others! 34 11/09/14
  35. 35. Questions? Thank you! Erno Aapa @ernoaapa Developer, DevOps-consultant Erno.aapa@avaus.fi 10.09.2014

×