Agile Software Architecture based overview of the technical debt metaphor … idea is that developers sometimes accept compromises in a system in one dimension (e.g., modularity) to meet an urgent demand in some other dimension (e.g., a deadline), and that such compromises incur a "debt": on which "interest" has to be paid and which the "principal" should be repaid at some point for the long-term health of the project. (ACM)
WordPress Websites for Engineers: Elevate Your Brand
Restructuring Technical Debt - A Software and System Quality Approach
1. A D N A N M A S O O D
S R . A R C H I T E C T / D O C T O R A L C A N D I D A T E
S C I S . N O V A . E D U / ~ A D N A N
A D N A N . M A S O O D @ O W A S P . O R G
Restructuring Technical Debt -
A Software and System Quality Approach
2. Technical Debt
“The technical debt metaphor … idea is that
developers sometimes accept compromises in a
system in one dimension (e.g., modularity) to meet
an urgent demand in some other dimension (e.g., a
deadline), and that such compromises incur a
"debt": on which "interest" has to be paid and which
the "principal" should be repaid at some point for
the long-term health of the project. ”
“Managing technical debt in software-reliant systems” - Proceeding FoSER '10 Proceedings
of the FSE/SDP workshop on Future of software engineering research Pages 47-52 – ACM -
http://dl.acm.org/citation.cfm?id=1882373
3. Reasons for Poor System Quality
There are many reasons that system quality decays as
the system ages. Some common reasons:
The business is blind to the concept of technical debt
and make decisions without considering the long
term implications
The business pressures the development
organization to release the system sooner without
taking the necessary time to complete all of the
necessary changes
Lack of quality processes or quality understanding
4. Reasons for Poor System Quality
Lack of documentation, where code is created
without necessary supporting documentation.
• Tightly coupled components.
• Inflexibly coded functionality.
• Lack of test suite
• Accumulation of quick band-aid “fixes” to
defects.
5. Reasons for Poor System Quality
Parallel development on two or more (merge debt).
Delayed refactoring.
As requirements evolve, unwieldy code must be refactored in
order to support future requirements.
The longer that refactoring is delayed, and the more code is
written to use the current form, the more debt that piles up
that must be paid at the time the refactoring is finally done.
Lack of commitment from the people involved.
6. Reasons for Poor System Quality
Hidden behind these issues:
dysfunction in organizational culture
poor technical practices
individual neglect
problematic hiring processes
insufficient oversight and governance
Incorrect alignment of project priorities with stakeholder
priorities taking into account all stakeholders
7. Correcting Poor System Quality
These issues are correctable, but will not be resolved
overnight.
To correct, the organization needs to
Make quality a first priority
Foster widespread understanding of system quality concepts
and best practices
Identify and communicate quality goals.
Enact positive change over time.
Apply principles for good design decision-making across the
teams.
Act as good stewards of system quality, balancing concerns of
all stakeholders (not just product development management)
8. System Quality Stakeholders
All stakeholders in a system have a keen interest in system quality:
Business
Information Security
Product Development Organization
Developers
Analysts
Data Teams
Architects
Quality Assurance
Configuration Management
Performance Engineering
Project and Program Managers
Dev Managers
Execs
Tech Ops
Help Desk
Customer Support
Internal and External Business Partners
Anyone else?
10. Technical Debt
Technical Debt: those things that
you choose not to do now which will
impede future development if left
undone
Technical debt = work that remains to be done before a job
can be considered complete.
Debt metaphor refers to eventual consequences of poor
architecture and development quality within a codebase.
Technical debt accumulates when focus remains on
immediate completion while neglecting quality and
changeability aspects of the system over time.
11. Sources of Technical Debt
Design Debt: cost of adding average sized features
is increasing to more than writing from scratch
Platform Experience Debt: availability and cost
of people to work on system features are becoming
limited
Quality Debt: diminishing ability to verify
functional and technical quality of entire system
Configuration Management Debt: integration
and release management become more
risky, complex, and error-prone
12. Accumulation of Technical Debt
Accumulation of technical debt:
does not impact delivery immediately.
creates a sense of increased volume of feature delivery.
causes business to respond positively to pace of delivered
functionality, creating an incentive/reward feedback loop
resulting in further accumulation of debt.
fools the business into thinking they will realize earlier returns
on their technology investments.
continually degrades quality of delivered functionality
Results in runaway accumulation of technical debt via
“compounding” (band-aids upon band-aids)
Eventually leads to either “technical debt restructuring” or flat-
out “technical bankruptcy”
13. Software Maintenance:
The Costs of Technical Debt
Software maintenance involves:
Correcting (removing functional errors),
Adapting (adding new functionality)
Perfecting (improving system quality) of software and systems
after initial release for production use
It can count for more than 50% of the costs in the
lifetime of a software system.
14. Software Maintenance:
The Costs of Technical Debt
Maintenance is difficult because a maintainer must
first understand the system before making changes.
This currently accounts for about half of a
maintenance programmer‟s time.
The system is hard to understand if it is poorly
architected, designed, coded, documented, or when
previous maintenance has degraded the system‟s
structure and integrity.
15. Software Maintenance:
The Costs of Technical Debt
Catch 22: It is always very difficult
to improve the maintainability of a
codebase which was not written to
be maintainable.
Code needs to be maintainable (with adequate test coverage) in order to
safely apply wholesale change
We cannot improve its maintainability without refactoring (changing) it.
This is why many projects designed to improve the quality of legacy
applications fail to do so.
When teams find themselves suffering from crushing technical debt, it is
time for tough decisions.
16. Technical Bankruptcy
Notions about application “rewrites” are often
unpopular.
However, the real costs of improving existing
systems may exceed the costs of rewriting them (or
at least, large portions of them).
Such rewrites might be likened to "declaring
bankruptcy"
This may be the most responsible course of action,
since the debt is really owed to us and to our
customers and stakeholders.
17. Technical Bankruptcy
Rewriting an application module is not necessarily
“starting from scratch”.
The team has significant experience with the current
implementation and the business.
The team has a greater ability to write quality,
maintainable applications
There are better frameworks and tools at their disposal
now than in the past.
Code is not the only artifact or output of combined
experience (that‟s the kind of thinking that led to the
problem in the first place)
18. Systems At the Verge of Bankruptcy
Characteristics of systems at the verge of technical bankruptcy:
Big Ball Of Mud: The overall structure of the system may never have been well
defined or may have eroded beyond recognition.
Information is shared promiscuously among distant elements of the system
Much of the important information is essentially global or duplicated
Small enhancements and superficial changes are doable, but larger changes are
very difficult
Larger changes often require expensive rediscovery of winding paths of
spaghetti code
The system shows obvious signs of unregulated growth
The system shows signs of repeated, expedient repair
Programmers with architectural backgrounds often shun working on such
systems
Programmers who are unconcerned with the overall architecture are more
content to work on such systems because of the inertia generated by constant
hole-filling and patching.
19. Avoiding Technical Bankruptcy
In order to avoid rewrites, design debt must be
regarded and respected early enough that
bankruptcy is not risked.
Ideally, design debt should be managed right from
the beginning of development projects.
20. Software Restructuring
Software restructuring is an alternative to
software rewriting.
Restructuring transforms a program‟s source code
into a more „„structured‟‟ form, thereby improving its
maintainability and prolonging its life.
21. Software Restructuring
Software restructuring creates an „„equivalent
structured replacement‟‟ from an unstructured
source program or system.
This can involve many types of transformations
whose common goal is to make programs and
systems easier to understand and maintain.
22. Software Structure
The structure of a program refers to
the structure of the code (called structured programming or
programming-in-the-small)
the structure of the system (called module structure or
programming-in-the-large).
23. Structured Programming
Structured programming is an „„art of reasoning‟‟, being able to
abstract different levels of understanding of a problem, and connect
them together in a hierarchy.
Good module structure can be achieved by applying
information hiding, in which every module hides a design
decision, allowing the modules to be understood
independently .
Good software structure can make the task of understanding the
software much easier
This can reduce the cost of maintenance to as little as one
third of the cost of maintaining unstructured programs.
However: Rewriting the unstructured programs from scratch using
structured programming and information hiding techniques is often
impractical because of high costs.
Bankruptcies are never very practical affairs.
24. Stages of Software Restructuring
Restructuring is a sequence of specialized
restructuring stages:
code restructuring
data restructuring
remodularization
25. Code Restructuring
Code restructuring transforms program code to adhere
to structured programming principles.
Poorly structured code is difficult, if not
impossible, to understand and maintain properly
Code level restructuring is a logical first step in a
complete restructuring because it simplifies the
subsequent stages by allowing them to make
additional assumptions about the source code.
26. Data Restructuring
Data restructuring makes the data structures and
variable usage of the program more sensible.
The data is much easier to understand if it is in a
representation that abstracts its relevant similarities.
Data structure analysis includes making sure that
all components of the data structures are related,
closely related data are not in separate structures,
the best type of data structure is used.
27. Data Restructuring Is Difficult
Data restructuring is difficult primarily because of tight coupling.
Coupling is a measure of the degree of dependence between
two items. The more highly coupled two things are, the
greater the chance that a change in one will require a change
in another.
Relational Databases are potentially coupled to a wide variety of
things:
Application source code
ETL source code
Persistence frameworks/layers
Other database schema
Data migration scripts
Test code
Documentation
28. Remodularization
Remodularization transforms bloated components
and subsystems into layers and hierarchies of
components.
Remodularization aims to enable implementation of
missing system quality attributes
Remodularization is achieved through isolation and
encapsulation of business functionality into many
small, tightly-limited components.
29. Remodularization is Difficult
Remodularization is also difficult because of tight
coupling.
Logic is frequently tightly coupled to data which is in
turn coupled to data stores.
Loosening these couplings may require elaborate
scaffolding (messaging adapters, service stubs. Etc.)
to be temporarily introduced
Temporary fixtures often become permanent
fixtures, so this process must be guided carefully
30. Phases In Each Stage
Each of these stages has three phases:
Information gathering
Decision making
Execution
31. Information Gathering Phase
The Information Gathering phase consists of the
following activities:
Information about the system and its current structure is
compiled
The existing source code and system documentation is
analyzed and reviewed
Information is presented to the restructuring team for further
analysis and decision making
32. Decision Making Phase
• The restructuring team analyzes the information and findings
from the information gathering and analysis phases.
• The restructuring team determines an appropriate structure
for the software.
• A restructuring effort is planned
Architectural master plans are created
Efforts are sized and prioritized
Project plans and timelines are created
Plans must be flexible to allow for reprioritization
33. Execution Phase
Includes all the actions that implement the
restructuring decisions
Includes actively monitoring progress toward
completion
Must include adequate quality monitoring and
interventions to prevent accumulation of technical
debt in the new parts
Plans and tasks are monitored and reprioritized
based on changing business needs as well as delivery
realities
34. Defining Future-State
High level modules and subsystems should be
defined and designed up front
Services
Components
Databases
Data Architecture (data types, taxonomies, models)
Equal attention must be given to Functional
Requirements and Non-Functional Requirements
(aka System Quality Attributes)
35. Functional Requirements
In general, functional requirements define what a
system is supposed to do
usually in the form of "system shall do <requirement>“
Functional requirements define specific behavior or
functions.
The plan for implementing functional requirements
is detailed in the system design.
36. Non-Functional Requirements
Non-functional requirements define how a system is
supposed to be.
Non-functional requirements specify criteria that
can be used to judge the operation of a system,
rather than specific behaviors.
The plan for implementing non-functional
requirements is detailed in the system
architecture.
37. System Quality Attributes
Non-functional requirements specify system
quality attributes.
Other terms for non-functional requirements are:
Constraints
Quality attributes
Quality goals
Quality-of-service (QoS) requirements
Non-behavioral requirements
Informally, these are known as the “-ilities"
38. System Quality Attributes
Business Perspective
Suitability: Functionality is suitable to all end users
Compliance: Functionality is compliant with applicable regulatory
guideline
Certifiability: System is able to be certified to some standard by an
independent third party
• Conformance: Conformance to industry and operational
standards
Accessibility: The user interface can be accessed and utilized by differently
abled users
Adaptability: Ability to change the system components to meet
new business needs
Extensibility & Modifiability: Ability to easily add new features
and customizations
Replaceability: Ability to replace system in the future
Reporting: The system supports generating various reports
39. System Quality Attributes
Security Perspective
Security: System is secure:
confidentiality, integrity, availability, accountability and
assurance
Privacy: System does not reveal (or allow revealing) of
information to improper parties
Nonexploitability: The system, its components, and its
configurations are hardened against known and
unknown security vulnerabilities
Accountability: The system records all user interactions
and data changes
Auditability: The system makes accountability
information available to managers and operators
40. System Quality Attributes
Performance Perspective
Efficiency: System does not waste valuable resources
Performance: Perceived response is immediate
Scalability: Able to handle increased usage on the
appropriate amount of resources, able to add resources
quickly and easily to handle increases in demand
Responsiveness: The system responds to interactions
with minimal delay and/or latency
Capacity: The system can handle sudden peaks and
surges in demand
Throughput: The system can handle a specified number
of interactions or transactions within a specified duration
without undesirable degradation in terms of other
qualities
41. System Quality Attributes
Configuration Perspective
Configurability: Ease of making configuration changes
Integrity: The system, its configuration, and its data
cannot be changed by unauthorized
parties(including accidental or deliberate
misconfiguration)
• Deployability: Ease of
installation, reinstallation, and updating
Portability: System supports changes in choice of
underlying technologies and architectures (such as
database platform, operating system platform, etc.)
Interoperability & Compatibility: Functionality
interoperates with other systems easily
42. System Quality Attributes
Operations Perspective
Availability & Fault Tolerance: System continues operating properly in the event of failure by
one or more of its components
Isolation: System survives and gracefully handles situations in which dependent “downstream”
systems become unavailable
Continuity: System recovers from failures in surrounding environment
Disaster Recoverability: The system, its data, and its configuration can be completely and
quickly recovered from backups and configuration management repositories, even in the event
of a catastrophic loss of a data center
Operability: System can readily be kept in a functioning and operating
condition by manual or automated means
• Maintainability: System upgrades can quickly and safely be performed with a
minimum of downtime
• Monitorability: Ability for operators to easily see how the system is operating
• Alerting: The system notifies operators of faults and exceptional conditions
• Maturity: System components are proven stable by others
• Stability and Reliability: The system exhibits infrequent failures
43. System Quality Attributes
Data Perspective
Durability: The system does not lose critical operational data, whether “in-
flight” or “at-rest”
Reversibility: Undesirable system and data changes can be reversed quickly
and easily with minimal disruption to the business
Cachability: The data is structured with appropriate separation of volatile
(changing) and non-volatile data to support caching
Archivability: Older data can be relocated to separate data stores
Partitionability: The system supports splitting like-kind data across
multiple databases (for scalability, risk mitigation, etc.)
Idempotency: Interactions performed twice result in same change as if
performed once
Concurrency: The system supports a high degree of simultaneous
interactions with the database without degrading in performance or
sacrificing data integrity (ex: optimistic locking, snapshot isolation)
Backup : The database and message queues support being backed up
without the system being taken offline
44. System Quality Attributes
Diagnostics Perspective
Supportability: System operators can quickly and easily
identify faults and perform root cause analysis within an
environment of controlled access to information and
restricted privileges
Instrumentability: System is capable of recording various
operational metrics that can be analyzed and reported
on
Tracability: System can be configured to record its own
interactions with itself in order to aid support personnel
in diagnosing problems
• Logging: The system logs external and
interactions with configurable degrees of detail
45. System Quality Attributes
Enhancement Perspective
Analyzability: Ability to figure out how the
system functions
Documentability: Ability to communicate
about the system in writing
Understandability: Able to use system with little
training
Learnability: Supports learning of system functionality
with little external interfacing
• Testability: Ability to create repeatable and
specific tests of the system and potential for
some to be automated
46. Architectural Tactics
An architectural tactic is a design decision that
affects how well a software architecture addresses a
particular quality attribute.
• Architectural tactics provide a means of
reducing maintenance costs and improving
quality.
48. References & Further Reading
Brown, Nanette, et al. "Managing technical debt in
software-reliant systems."Proceedings of the FSE/SDP
workshop on Future of software engineering research.
ACM, 2010.
Klinger, Tim, et al. "An enterprise perspective on
technical debt." Proceedings of the 2nd Workshop on
Managing Technical Debt. ACM, 2011.
Seaman, Carolyn, and Yuepu Guo. "Measuring and
monitoring technical debt."Advances in Computers 82
(2011): 25-46.
Black, Sue, et al. "Formal versus agile: Survival of the
fittest." Computer 42.9 (2009): 37-45.
Keywords: cost-benefit analysis, design , design decision, trade-off, economics , large-scale system development , life cycle
management, measurement, productivity software, metrics, technical debt