This document summarizes a presentation about contextually-driven system architecture reviews. The presentation discusses what architecture is, how it exists at the intersection of management, technology, and design. It also discusses problem statements and how they provide important context for reviews by outlining critical success criteria. The methodology of contextually-driven reviews is then outlined, including preparation, the review meeting, and follow-up steps. Benefits include identifying risks early and potential cost and schedule savings. Drawbacks include risks if the client is out of touch and the method's dependency on expertise and face-to-face meetings.
2. Michael Dedolph
Levi Deal Consulting
Michael Dedolph is self-employed as a consultant and innkeeper. For the
past seven years, Michael led quality process improvement efforts at CSC
and Lucent. Previously, he was a systems architecture review leader at Bell
Labs (Lucent) and facilitated numerous risk identification and project
retrospective sessions. Prior to 1997 Michael worked in the Risk and
Process programs at the SEI where he was the technical lead for teams
that developed the SCE and CBA-IPI appraisal methods. His IT career
began with ten years as an Air Force computer officer, working in satellite
systems, management information systems, and as an instructor of
software engineering at the Air Force Institute of Technology.
3. Contextually-Driven
Architecture Reviews
For Better Software
Conference East
F. Michael Dedolph
fmdedolph@netscape.net
14 November 2013
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
1
Author’s Note
Most of this presentation material was
used previously in these venues.
– SEI’s SATURN 2009 conference
– Washington DC Area SPIN meeting in Feb 2009
– PNSQC 2010
Architecture reviews were a good idea then, and
they still are
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
2
1
4. What is “Architecture”?
Ø Before we can review it,
we should know what “it” is
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
3
“A System Architecture . . .”
Provides a solution to a problem for a
client
– Early on, it is a conceptual or potential solution
Architecture has also been called
“design with constraints”
– Alternatively, you could say architecture includes
a design that works within the constraints,
including cost and schedule!
Ø Any given architecture is NOT the ONLY
solution, but, some solutions are better than
others
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
4
2
5. Where Architecture Exists
• Architecture exists at the intersection
of management, technology, and
design
Management
Technology
Design
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
5
Architecture Problem Statements
• The problem space has multiple
aspects: “FFETO”
– Function – what it does, how well it does it.
– Form – what it “looks like”; includes major
environmental interfaces
– Economy – cost, including development,
maintenance, material, system retirement, etc.
– Time – relationship to past, present, and future
– Operational/ Developmental – Things that
pertain to development constraints and system
operating environment, rather than the system
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
6
3
6. Problem Statements
Ø are the basis for the review!
Provide a succinct summary of critical
success criteria
– Short: 2 pages is good
– Includes major constraints
– Client-centric
– Cover Function, Form, Economy, Time, and
(optionally) Operational
– Be expressed in sufficient detail to make
judgments about the proposed solution, but are
not unnecessarily proscriptive of prescriptive
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
7
Problem Statements
Ø Are NOT
– A requirements document, but may summarize
the most critical high level requirements
Problem statements also have unstated
"always" criteria, e.g.:
– Possible to construct
– Can make money/ will provide value
– Solution won't result in harm
– Legal/ regulatory constraints
Ø The team must understand these as well.
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
8
4
7. Prob. Statements Provide Context
• Problem statements can be applied to
almost any product or service.
• Advantage: representational
independence
• Many architecture review techniques depend on the way
the design is represented.
• But, design can be represented in many different ways
—e.g., network diagrams, UML, physical prototypes,
circuit diagrams, flow charts, use cases . . .
Ø No representation captures all aspects of the
system, and focusing too much on representation
can actually obscure problem areas
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
9
Classical Architecture -
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
10
5
8. Taj Problem Statement Included:
Function: tomb, mosque, hotel, memorial
Form: symmetry, untarnished finish,
unprecedented weight, shearing force of
the river, visual perspective maintained
Economy: irrelevant, it will cost what it costs
Time: it will take as long as it takes, but must
last for all time
Operational: expertise is needed from other
kingdoms, supply chain issues
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
11
Architecture Review Method, in a
Nutshell
• Define the problem the client wants
solved
• Compare it to the architecture
(proposed solution)
• Identify the gaps (or risks)
Then:
Ø Let the project resolve the gaps
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
12
6
9. Architecture Review Questions
Questions to answer before a review:
• Who decides? (Know the client)
• What problem are we trying to solve?
• Are key stakeholder interests represented?
Questions to answer during a review:
• How good is the proposed solution?
• What are the issues/ risks/ gaps in the solution
Questions to answer after a review:
• Which things will we (the project team) address?
• How will we (the project team) address them?
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
13
Ground Rules for Reviews
Ground rules are covered in the pre-review, and
again in the review meeting
• No attribution—review products, processes, and
ideas, not people
• Team is there to identify, not solve problems
• Client requests (and pays) for the review, but,
the project owns the findings and responsibility
for correcting them
Ø One exception, “Management Alert” (covered later)
• Reviewers can ask questions at any time
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
14
7
10. Review Steps
• Preparation
• Respond to initial contact/request; develop problem
statement; select team and develop agenda; arrange
logistics – travel, space, connectivity; pre-review call
• Review Meeting
• Project presentations with Q&A; review team Caucus;
conduct readout; client meeting (optional)
• Follow-up
• Write and distribute the Review Report; present
findings to lessons learned/governing groups (Board
Report); review the Project’s action plans; Close the
review by meeting with project and/or sponsor
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
15
Preparation – Some Thoughts
Toughest part: Problem Statement
•
•
•
I have seen projects cancel a review because
they “didn’t have time to develop a problem
statement” How do you think the project fared?
In many cases, the initial review schedule was
delayed while the project and client worked out
the problem statement
The Problem Statement is reviewed in detail
during the pre-review meeting, used to finalize
agenda and assign pre-reading
Team selection
•
•
Based on the Problem Statement
Key roles – Leader, “Angel”, SMEs
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
16
8
11. Review Meeting - Overview
• Presentations – by the project team,
any format, reusing existing materials
• Questions by the review team, at any
time, moderated by the Leader or
Angel
• Strengths, Issues, Observations
recorded on index cards (Low Tech)
• Team Caucus determines findings
• Readout and (Optional) Sponsor/
Client meeting
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
17
Snow Card Example - 1
Save for
Sequence
Number
Write large,
One thought
per card,
with enough
detail to be
useful – but
not too
wordy.
Save for Severity
{
Paper-based
reviews can be
hard to transcribe!
FMD
Note here if card is a Strength or
Observation; default is an Issue
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
Don’t Forget
Your Initials!
11/14/2013
18
9
12. Review Meeting - Caucus
• Led by review Leader and Angel.
• Categorization is done to provide a
consistent “story” for the read-out
– A “typical” review produces 80–350 observations,
arranged into 10-25 topics
• Issues are Prioritized:“Management
Alert”, Critical, Major, Minor
• Summary of each topic area written by
team subject matter experts
• Strengths should be preserved if changes
are made to the architecture
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
19
Prioritization by the Review Team
Management Alert:
• Goes to client/upper management
• “In the unanimous opinion of the Review Team,
the project will fail unless these issues are
immediately addressed: <List of issues>”
Critical, Major, Minor, Strengths, &
Suggestions go to the project
• Readout ensures no surprises in the report
• The review team will NOT withdraw a finding or
lower the priority (project owns the action plan)
• Priority may be increased at project request
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
20
10
13. Follow-up
• Readout and Sponsor/ Client meeting
• Management Alert and Critical Issue
Letters within a week
Ø Most management alerts are related to cost,
schedule, and resources, NOT purely technical
issues. Hence, upper management is responsible
for the action plan for management alerts
• Critical Issue letters goes to the Project, Project
management is responsible for the action plan
• Two Reports, one to the project, one to
the Board.
• Review team Leader and Angel review
the action plans.
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
21
What Happens When?
• . . . a project ignores a Management
Alert?
• . . . a project ignores a Critical issue?
• . . . a project fixes everything?
• . . . the review team identifies an issue
that turns out to not be a problem
(“false positives”)?
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
22
11
14. What's Different (or the Same)
About this Kind of Review?
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
23
What's Different?
• Focus
– problem/solution congruence, Client ownership of
the decision making, ownership by project team
• Context/ Domain independent
– Notation and representation independent, not
model or standard based (although this can be
included in the problem statement, if needed)
• Follow up
– Clear statement of potential failure points, Project
sees and owns findings – can ignore or post on
their web site
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
24
12
15. What's the Same?
Outside eyes—external to team
Products, processes, ideas are
reviewed, not people
Not a problem solving session
Ø Balances a defined review
process with extreme flexibility.
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
25
Sponsor Survey - Benefits
Saved money/time
• Estimated at > $1M per large project, similar estimates
were derived from Monte Carlo simulations based on
analyzing the impact from correcting review findings
• Direct results from specific reviews, e.g., re-architecting
after the review trimmed 9 months off the schedule,
saved > $7 million.
Intangible benefits include:
• Preparation (especially preparing the problem
statement) forces people to think through the problem
• Cross-pollination of techniques across the organization
• Learning as a review team member
• Synching up project teams; “on the same page”
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
26
13
16. Drawbacks to the Method
• Risks of a Client-centered method
• The Client can grow out of touch with the market,
customer needs, and end user issues
• In contracting, many of the success criteria are
negotiated into the contract, and may represent political
compromises that can’t be changed easily
• Method dependencies
• Strongly oriented to face-to-face meetings, requires a
deep pool of reviewers with expertise
• Incomplete findings/level of detail
• Any review’s findings are inherently incomplete. A high
level review is not suited for all problem spaces; 2 to 3
days will not flush out all of the “devils in the details”
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
27
Institutionalizing the Method
• Requires building a “learning culture”
Oversight
Executive
Champion
Review
Board
“Angels”
Push
Review
Organization
Project
Project
Project
Project
Project
Project
Better Software East
Review
Leaders
Reviewers,
Reviews
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
28
14
17. Suggested Steps for Making
Reviews Part of the Culture
•
•
•
•
Start with a high-level champion/ executive sponsor
Obtain central funding for an extended period – say 5 years
Train a core team of leaders and angels to do reviews
Charter a board; recruit advocates. Board members are
expected to serve as review “Angels”
• Perform the reviews; capture data and lessons learned to
document the method’s value
• Move to a sponsor-based funding model after the initial
time period, when the method has proven itself
• Rigorously retain the focus on client-centered problem
statements, project ownership of findings, limited but
pertinent management alerts, and non-attribution
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
29
11/14/2013
30
Questions???
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
15
18. Presenter Info
F. Michael Dedolph
412-477-7163
fmdedolph@netscape.net
BIO:
F. Michael Dedolph is currently an innkeeper and owner of a bustling B&B, and part
time consultant to the IT industry for reviews, risk management, quality methods,
and process improvement.
From 1997 to 2004, Michael was a systems architecture review leader at Bell Labs
(Lucent); he also managed and taught Lucent’s Systems Architecture Introduction
class. In addition to leading architecture Review Teams, he facilitated numerous
risk identification, problem solving, and project retrospective sessions.
Prior to 1997, Michael worked in the Risk and Process programs at the SEI. While
at the SEI, he was the technical lead for the teams that developed the SCE and
CBA-IPI appraisal methods, and was the team leader for several Risk Reviews.
He started his IT career by spending 10 years as an Air Force computer officer, and
also spent 7 years at CSC leading process improvement teams.
Better Software East
Contextually Driven Architecture Reviews – Dedolph
11/14/2013
31
References:
1. CMU/SEI-2000-TR-004, "ATAM: Method for Architecture Evaluation";
Kazman, Klein, Clements, 2000
2. Problem Seeking: An Architectural Programming Primer, 4th Ed;
Pena and Parshall, 2001, ISBN 0-913962-87-2
3. Handbook of Walkthroughs, Inspections, and Technical Reviews;
Freedman and Weinberg, 1990, ISBN 0-932633-19-6
4. IEEE Software, March/April 2005, "Architecture Reviews: Practice and Experience";
Maranzano, Rozsypal, Zimmerman, Warnken, Wirth, and Weiss
5. STQE Jul/Aug 2002 (Vol. 4, Issue 4) Feature: Measurement & Analysis
"A Blueprint for Success: Implementing an architectural review system“;
Daniel Starr, Gus Zimmerman
6. CMU/SEI-2006-TR-012, "Risk Themes Discovered Through Architecture Evaluations";
Bass, Nord, Wood, Zubrow; 2006
7. CMU/SEI-2010-TN-018, “Relating Business Goals to Architecturally Significant
Requirements for Software Systems”; Clements, Bass; 2010
8. CMU/SEI Webinar, “SoS Architecture Evaluation and Quality Attribute Specification
(Webinar)”; Gagliardi; 2010
Better Software East
Contextually Driven Architecture Reviews – Dedolph
Contextually Driven Architecture Reviews
Dedolph – 10/18/2010
11/14/2013
32
16