Presentation at FSE15 on an empirical study looking at organizational understanding of technical debt.

1. © 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Measure it? Manage it?
Ignore it? Software
Practitioners and
Technical Debt
Neil A. Ernst
with Stephany Bellomo, Ipek Ozkaya, Robert Nord
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213
and Ian Gorton
College of Computer and Information Science
Northeastern University

2. 2
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Copyright 2015 Carnegie Mellon University and IEEE
This material is based upon work funded and supported by the Department of Defense under Contract
No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering
Institute, a federally funded research and development center.
References herein to any specific commercial product, process, or service by trade name, trade mark,
manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or
favoring by Carnegie Mellon University or its Software Engineering Institute.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING
INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY
MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER
INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR
MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL.
CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH
RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
This material has been approved for public release and unlimited distribution.
This material may be reproduced in its entirety, without modification, and freely distributed in written or
electronic form without requesting formal permission. Permission is required for any other use. Requests
for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.
DM-0002708

3. 3
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Background
• Cunningham, 1992: “Shipping first time code is like going
into debt. A little debt speeds development so long as it is
paid back promptly with a rewrite... The danger occurs
when the debt is not repaid”
• Our definition: “the obligation that a software organization
incurs when it chooses a design or construction approach
that's expedient in the short term but that increases
complexity and is more costly in the long term.”
(McConnell)

4. 4
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Research Questions
• RQ1. Is there a commonly shared definition of technical
debt among professional software engineers?
• RQ2. Are issues with architectural elements (such as
module dependencies, external dependencies, external
team dependencies, architecture decisions) among the
most significant sources of technical debt?
• RQ3. Are there practices and tools for managing
technical debt?

5. 5
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Methodology
• Pilot Survey
• Full survey
• Triangulate with:
• Multiple choice
background
• Open-ended
questions
• Likert agreement
• Follow-up interviews
(opportunistic)

6. 6
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Demographics
• 1831 surveys were started (across all three collaborators)
and 536 surveys fully completed (all questions answered),
an overall response rate of 29%
• Respondents over 6 years experience
• Roles selected included developers (42%), and project
leads/managers (32%)
• Most projects were web systems (24%) or embedded
(31%).
• Projects generally consisted of 10-20 people
• The systems averaged 3-5 years old, but a significant
number (29%) were over 10 years old.
• The systems between 100KLOC and 1MLOC in size.

7. 7
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Findings
1. Architecture is a key source of technical debt
2. Technical debt is broadly temporally distributed (source
and symptom not synched)
3. Issue trackers are currently the best tool approach to
managing technical debt.
Triangulate answers to these questions by looking at quantitative
(closed-ended) questions, and coding of open-ended and interview data.

8. 8
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
RQ1: Defining Technical Debt
• “(A2) I think the vocabulary of technical debt is useful for getting the
interests aligned.”
• ‘convincing product managers and stakeholders on the value proposition
of managing the debt.’

9. 9
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
RQ2: Architecture choices key
9

10. 10
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Coding of open-ended questions

11. 11
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
RQ2: Architecture as source

12. 12
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
RQ2 (2)
“(B2) the work that we’re doing now to introduce a service
layer and also building some clients using other technology is
an example of, you know, decisions that could have been
done at an earlier stage if we had had more time and had
the funding and the resources to do them at the time instead
of doing it now.”
“‘platform’ was not designed with scalability in mind”
“In retrospect we put messaging/communication ... in the
wrong place in the model view controller architecture”

13. 13
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
RQ2: Drift
Lehman’s concept of entropy present in our data:
• “over the years, other sites would begin using the system and
would require changes to how the workflow operated”
Weak association between system age and the perceived
importance of architectural issues, using Yule’s Q.
89% of those with longer-lived systems (>=6 years old)
agreed or strongly agreed with the notion that architectural
issues were a significant source of debt, compared to 80% of
those with newer systems (<3 years old).

14. 14
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
RQ3: Management of TD
How tracked …
Where tracked …

15. 15
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
RQ3: Tool Use
15
None/Unknown: 58%

16. 16
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
RQ3: Management of TD
“(B1) regarding static analysis we have the source code static
analysis tools, but this is to assure proper quality of source
code. But how architectural changes are impacting I don’t
know. And, in fact, this is something we don’t do.”
“(C1) it showed up on Jenkins - the CI server - there’s a
billion little warnings. And so it seems a little bit
overwhelming.”
“[we track] occasionally by explicit tech debt items, usually by
pain, or not at all...”

17. 17
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
• Timeline for TD management
• Track TD items like we do defects
• Connect Architecture Technical Debt to architectural analysis
(e.g., quality attribute prioritization)
Future Work and Open Questions

18. 18
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
18
• Software practitioners agree on the usefulness of the metaphor
• Different interpretations of what makes up technical debt in
particular contexts.
• Leading sources of technical debt are architectural choices
• take many years to evolve
• Managing this drift is vital in managing technical debt.
• Developers perceive management as unaware of technical debt
issues
• Desire for standard practices and tools to manage technical debt that
do not currently exist
http://bit.ly/sei-td – @neilernst – nernst@sei.cmu.edu

19. 19
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Call for Papers: Negative Results in Software Engineering
Special Issue of J. Empirical Software Engineering.
We welcome your well-conducted yet ‘negative’ empirical studies.
We also are looking for suitable reviewers and reviewer experience
reports.
Deadline for submission: October 7, 2015
http://bit.ly/emse-negative
If you have questions/comments or would like to volunteer to be a
reviewer of the papers, please contact the guest editors.
Richard Paige richard.paige@york.ac.uk
Jordi Cabot jcabot@uoc.edu
Neil Ernst nernst@sei.cmu.edu
Plug

20. 20
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Backup

21. 21
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Coding Process
Id Statement Coder 1-
A
Coder 1-B … Coder
2-C
1 Technical debt is built over years & multiple
versions by the combination of factors like
monolithic code design, mix of obsolete and
new technologies, cost over quality etc.
Code Code … Code
Cohen’s K: 0.45

22. 22
TD Survey
September 2, 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public
Release; Distribution is Unlimited
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Code definitions
Final Coding Term Definition Subsumed Codes
Interest The pain caused by technical debt, but not the debt itself.
Rework Additional work needed to remove technical debt ‘principal’
Architecture choice
While arch choice is pain caused perhaps by context shifts vs design shortcut that is deliberative we
combine the two as “architecture choice” because participants may not even know whether it was
deliberate or not. Must be a specific case/instance.
Design Shortcut, Bad Architecture Choice
Legacy modernization Changes and evolution in operating environment e.g. new tech or requirements changes.
Obsolete Technology, Legacy, evolution, changing requirements, External Dependencies, Prototype become
Product
Limited Knowledge Respondent had no good understanding of TD No clue
Awareness People in respondent’s org had no understanding of problems TD caused Management, avoidance strategy, culture, deferred integration design, metaphor
Defects
Responses referring to problems in code externally visible. Interest code is more suitable when text
refers to linking those bugs to original decision.
Bugs, Maintenance, Software Quality
Time Pressure Must release to make deadline Schedule
Cost Pressure Lack of financial resources or motivation to fix problem New code
Code Problems Issues relating to code-level problems such as detected by FindBugs Overly complex code, inter-module dependencies, Code with Debt, Code Duplication
Process Some problems arising from poor development processes. Inadequate testing, Inefficient CM, Lack of Documentation
None A comment that is not possible to code
Measurement
codes about need for measuring, things we might need to measure such as complexity and
accuracy
Tools
this category is about the things that are technical in nature including tool support for making
intelligent decisions. Platform dependence, ways to find TD
Requirements Shortfall
Requirements not changing significantly, but system does not meet them. Could be due to
ambiguity, lack of clarity, etc.
Lack of documentation
Issues due to incomplete understanding of the architecture which was often attributed to lack of
documentation or some way to better understand the impact of making changes on the quality or
maintainability of the system
Inadequate Testing
This category covers inadequate testing due to issues such as inadequate test coverage, limited test
resources, lack of test automation or not enough time to complete tests