Nowadays, as the software industry is slowly becoming more mature, software measurement and performance measurement are becoming increasingly important. Organizations need to know their productivity and competitiveness in software development projects for various reasons. In many software development contracts, targets are set for the suppliers to reach. These targets are based on software metrics like productivity, speed of delivery and software quality. In order to check if the targets are reached, it is necessary to measure the functional size of the software product that is delivered and also the functional size of the software development project that is carried out, as there is usually a difference between these two sizes. To be able to use functional size in contracts, it must be measured in an objective, repeatable, verifiable and therefore defensible way. That being the case, the industry’s best practice is to use an ISO/IEC standard for functional size measurement, e.g. Nesma, COSMIC or IFPUG function points. However, these methods only measure the functional user requirements from the total software requirements to be delivered. In activities like project estimation and productivity measurement, the influence of the non-functional requirements is expressed in the Project Delivery Rate (PDR) which is expressed in effort hours per function point. If more than the average amount of non-functional requirements need to be realized in a project (or more severe non-functional requirements), the PDR used should also be higher. In the industry it is customary to set productivity targets based on an average (or calibrated) influence of non-functional requirements and this works quite fine in traditional software projects. In software development projects that are executed in an agile way, this is not always the case. When working agile, there are forces that influence the traditional way of performance measurement significantly, resulting in a number of serious issues. In this paper these issues are explained and a method to overcome these issues is proposed.

1. Performance
measurement of agile
teams

2. |
Performance measurement
of agile teams
continuous development of a product –
performance measured by sprint
Cracow Poland, October 7 2015
2Performance measurement of agile teams
Harold van Heeringen, METRI
Theo Prins, Sogeti Nederland
Edwin van Gorp, Sogeti Nederland

3. |
Sizing, Estimating & Control
 Sogeti department responsible for:
 Functional size measurement: Nesma, IFPUG and COSMIC;
 Estimating new projects, releases and contracts;
 Project control based on software metrics;
 Performance measurement;
 Benchmarking;
 Performance reporting.
 This presentation is about one engagement where the SEC department
was asked to help solving the issues with regard to the productivity of a
Sogeti agile team in a government project…. measured per sprint!
Performance measurement of agile teams 3

4. |
Background
 RfP of Dutch government agency in 2014
 Agile development of a product, taken over from another supplier;
 Performance measurement of the team per sprint;
 Based on ISO/IEC norm, in this case Nesma function points;
 Realistic targets set, based on Sogeti historical data, external
benchmark data and historical data of previous supplier!
 Sogeti offered to measure performance based on Project Delivery
Rate (PDR), measured in project function points delivered per sprint;
 Politics: Customer needs to prove the new supplier performs better!
 Customer selects Sogeti because of proved historical performance
and specialized department Sizing, Estimating & Control.
4Performance measurement of agile teams

5. |
The problem
 The application turned out to be very complex. Even the OTAP
environment was very complex to handle. The product backlog
consisted of many non-functional backlog items;
 Functional size measurement is a powerful way to objectively size
progress when it comes to functionality added, changed, or deleted,
but not for measuring non-functional sprint backlog items processed by
the team;
 The customer (product owner) decides which functional and non-
functional backlog items are put on the sprint backlog;
 Traditional data and performance measurement methods turned out to
be ineffective when measuring performance on sprint level when more
than average non-functional backlog items are put on the sprint
backlog.
5Performance measurement of agile teams

6. |
Two types of agile projects
1. Development of a set of specific requirements, prioritized on a backlog,
realized in a specific duration by a specific team in a specific amount of
effort hours and cost. At one point, the project, and the product, is
finished. (in traditional terms: new development or release).
2. Continued development (evolving) an existing application. No definite
end goal when the project or product is finished. A year divided into X
sprints of Y weeks and a fixed team working to deliver the sprint backlog
items. This only ends when the organization decides maintenance is no
longer needed. (in traditional terms: maintenance).
This industry paper investigates Type number 2.
6Performance measurement of agile teams

7. |
First supplier
 Big international system integrator;
 Application developed from scratch;
 About 15 sprints;
 Sprints with fluctuating functional sizes in Nesma FP;
 High complexity;
 Average PDR about 17 hours/FP.
 Almost all backlog items of functional nature!
 However, the customer was not happy and decided to select a new
supplier.
7Performance measurement of agile teams

8. |
After transition
 From Q3 2014 onwards, Sogeti took over;
 Sprints of 3 weeks;
 Sogeti team, product owner supplied by customer;
 Product backlog contains many non-functional items:
 Scrum team works hard, but hardly delivers function points;
 PDR (h/FP) relatively high, not in line with target PDR;
 Customer contract manager blames Sogeti for not being productive;
 Contract under pressure, media attention, pressure and politic issue;
 Sogeti department Sizing, Estimating & Control asked to analyze the
performance and to propose improvements.
Performance measurement of agile teams 8

9. |
Function points
 Important to understand, using a functional size measurement method
(an ISO/IEC standard) means measuring the size of the functional user
requirements that are implemented in the software;
 Non-functional requirements are not measured at all!
 More non-functional work in a sprint means that less functionality is
realized, and therefore a higher PDR (hours/FP)/lower productivity.
 In project estimation and benchmarking, the influence of NFR is
accounted for by the historical data used or the parametric model used,
or the peer group that is constructed based on projects with similar
characteristics.
Performance measurement of agile teams 9

10. |
Story points
 Usual way to estimate effort in agile teams;
 Team members assign story points to each backlog item, reflecting the
amount of work needed to realize the item;
 Subjective, not repeatable, not verifiable and not defensible, but
mainstream practice in agile teams because of ease of use;
 Story point-based metrics can not be compared with any measurements
or metrics outside the team;
 Story point measurement is not a standard, but SP do take into account
the effort spent on non-functional backlog items.
Performance measurement of agile teams 10

11. |
Non functional backlog items are important
11Performance measurement of agile teams
Sprint X FP SP Effort hours
Backlog item 1 4 4 90
Backlog item 2 0 6 120
Backlog item 3 0 2 45
Backlog item 4 5 3 65
Backlog item 5 4 3 80
Total 13 18 400
PDR = 400/13 = 30,8 Hours/FP
Ratio F/NF SP backlog items: 10/8

12. |
Sprint performance example
Performance measurement of agile teams 12
Sprint 1 2 3 4 5 6 7 8
Effort 345 389 367 412 365 375 390 401
Size (FP) 15 5 16 3 25 0 36 32
Sprint 1 2 3 4 5 6 7 8
PDR
(h/FP)
23,0 77,8 22,9 137,3 14,6 n/a 10,8 12,5
0,0
20,0
40,0
60,0
80,0
100,0
120,0
140,0
160,0
3 4 5 6 7 8 9 10
PDR(h/FP) Target PDR (h/FP)

13. |
Issue
 Team spends a lot of time on non-functional backlog items;
 PDR is not good enough to reach target;
 Stakeholders don’t understand this ‘technical issue’ and only see metrics
on the dashboard  PDR significant worse than expected;
 But… customer product owner decided on the product backlog items to
put on the sprint backlog!
 So, disappointing PDR mainly caused by the number of functional and
non-functional backlog items put in the sprint by the product owner (=
the customer!).
 Sogeti SEC wishes to address this issue and to come up with a proposal
for a more accurate performance measurement method.
13Performance measurement of agile teams

14. |
SEC proposal
Agile Normalized Size (ANS)
Functional size that could have been realized if the product owner only had
put functional backlog items in the sprint backlog.
Based on this ANS, a PDR (hour/FP) can be determined that can be
compared to the PDR’s in the databases with historical data.
14Performance measurement of agile teams

15. |
Method
1. Measure the functional size of the realized functional backlog items with
a standard method (Nesma/IFPUG FPA, COSMIC, …);
2. Determine whether the realized backlog items are functional or non-
functional;
3. Determine the number of story points of the functional backlog items
realized in the sprint;
4. Determine the total number of story points realized in the sprint;
5. Determine the agile normalized size:
(functional size / # functional story points)
* total # story points
15Performance measurement of agile teams

16. |
The example extended
16Performance measurement of agile teams
Sprint X FP SP Hours
Backlog item 1 4 4 90
Backlog item 2 0 6 120
Backlog item 3 0 2 45
Backlog item 4 5 3 65
Backlog item 5 4 3 80
Total 13 18 400
Agile normalized size = (13 / 10) * 18 = 23,4 nFP
PDR = 400/23,4 = 17,1 hours/nFP
Functional size: 13 FP
Functional SP: 10 SP
Total SP: 18 SP
Regular PDR: 400/13
= 30,7 hours/FP

17. |
The effect in multiple sprints
Sprint Size (FP) Functional SP Non-functional SP Total SP ANS (nFP)
16 20 32 12 44 27,5
17 25 28 16 44 39,3
18 18 24 20 44 33,0
19 29 35 4 39 32,3
20 4 6 36 42 28,0
21 15 16 24 40 37,5
17Performance measurement of agile teams

18. |
Advantages / disadvantages
Advantages
 Reduced influence of non-functional backlog items;
 The use of an ISO/IEC FSM standard – ability to benchmark.
Disadvantages
 Depending on accurate story point assignment (subjective);
 Possible for the team to tweak the performance figures;
 As the product owner is present, this risk is considered to be small;
 Impossible to measure ANS when the functional size delivered is 0.
Performance measurement of agile teams 18

19. |
Productivity measurement
Sprint Size (FP) ANS (nFP) Hours Hours/FP Hours/nFP
16 20 27,5 500 25,0 18,2
17 25 39,3 480 19,2 12,2
18 18 33,0 530 29,4 16,1
19 29 32,3 468 16,1 14,5
20 4 28,0 534 133,5 19,1
21 15 37,5 522 34,8 13,9
19Performance measurement of agile teams

20. |
The effect in multiple sprints
20Performance measurement of agile teams

21. |
The example
Sprint Size (FP) Functional SP Non funct. SP Story Points ANS (nFP)
16 20 32 12 44 27,5
17 25 28 16 44 39,3
18 18 24 20 44 33,0
19 29 35 4 39 32,3
20 4 6 36 42 28,0
21 15 16 24 40 37,5
22 0 0 41 41 n.t.b.
23 18 24 20 44 33,0
21Performance measurement of agile teams

22. |
Sprint 22: no productivity measurement
Sprint Size (FP) ANS (nFP) Hours Hours/FP Hours/nFP
16 20 27,5 500 25,0 18,2
17 25 39,3 480 19,2 12,2
18 18 33,0 530 29,4 16,1
19 29 32,3 468 16,1 14,5
20 4 28,0 534 133,5 19,1
21 15 37,5 522 34,8 13,9
22 0 N/A 512 N/A N/A
23 18 33,0 508 28,2 15,4
22Performance measurement of agile teams

23. |
Issue: completely non-functional sprints
 In sprint 22, zero function points were delivered;
 Size in FP is 0, ANS impossible to determine (dividing by zero);
 Impossible to determine productivity.
 Solution: progressive approach.
23Performance measurement of agile teams

24. |
Progressive approach
 Size measurement and productivity measurement not per sprint, but until
the last sprint;
 Does not focus on sprint, but on overall performance
(∑1-n functional size / ∑1-n functional story points)
* ∑1-n total story points
24Performance measurement of agile teams

25. |
Progressive approach
Sprint Size (FP) ANS (nFP) Hours Hours
(cumulative)
ANS
Progressive
Hours (cum) /
nFP (prog)
16 20 27,5 500 500 27,5 18,2
17 25 39,3 480 980 66,0 14,8
18 18 33,0 530 1.510 99,0 15,3
19 29 32,3 468 1.978 132,2 15,0
20 4 28,0 534 2.512 163,6 15,4
21 15 37,5 522 3.034 199,2 15,2
22 0 N/A 512 3.546 231,4 15,3
23 18 33,0 508 4.054 264,3 15,3
25Performance measurement of agile teams

26. |
Difference between the methods
Performance measurement of agile teams 26

27. |
Starting points
Documentation
After each sprint the functional documentation should be made up-to-
date and it must be clear:
 Which functionality was added in the sprint;
 Which functionality was changed in the sprint and in which way;
 Which functionality was deleted in the sprint;
 This should be part of the definition of done.
Effort administration
 The effort hours need to be booked in the effort administration in such a
way that it is possible to clearly identify the effort hours in scope and out
of scope of the performance measurement.
Performance measurement of agile teams 27

28. |
Conclusions and recommendations
 The productivity of an agile team in a contract can be measured and
benchmarked while taking into account the effect of non-functional
requirements;
 The customer now understands that non-functional backlog items have
impact on the PDR when using only Nesma/IFPUG function points in agile
projects when measuring on a sprint level. Customer is able to explain
that internally and politically. Pressure is less now, because targets are
met.
 The method can help other organizations as well!
Performance measurement of agile teams 28