A global development organization—in seven cities on three continents—has developers all using agile practices to develop complex applications. In addition to the common problems faced by distributed teams, they must deal with attrition rates in excess of 50 percent, possible loss of intellectual property, and the need to integrate the work of multiple Scrum teams into a single build. James Lynn describes an organizational structure that includes implementation teams, known as “the Factory.” Developers in the Factory often have little in-depth knowledge of the application or customer. Assisting the Factory are architecture teams that provide oversight, communication, coordination, and technical direction. The architecture teams include new roles such as Code Trolls who develop reference implementations for services and patterns to ensure the production of consistent code to common coding standards and quality measurements. These trolls review code against metrics standards, test case standards, and coverage. Learn how one organization successfully coordinated the efforts of massive, distributed development projects.

1.
AT12
Session
6/6/2013 3:45 PM
"Using Scrum for Global Scale
Development"
Presented by:
James Lynn
SUTO Consulting
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. James Lynn
SUTO Consulting
A partner with SUTO Consulting (sutoconsulting.com), James Lynn is passionate about
delivering value to his clients by improving software development productivity and quality
through enhancements to organization, process, and metrics; by transitioning work off-shore or
reallocating work on-shore; and by developing business intelligence and metrics in his
organization’s SaaS products to increase effectiveness for the end-user, resulting in higher
customer satisfaction and improved renewal rates. For the past ten years, Jim has been an
agent of change for several organizations in transition from waterfall to agile in the quest to
improve performance and as the foundation to migrate heterogeneous applications to a single
enterprise architecture. Reach Jim at jim@sutoconsulting.com.

3. Using Scrum for
Global Scale Development
Jim Lynn
SUTO Consulting
Presentation Progress
7
6
5
4
3
2
1
1

4. The Disclaimer
• The specific issues have
been generalized
• The specific solutions
have been rationalized
• What we are going to
talk about is the
intersection of these
experiences
Company A
Company
B
Company
C
No Individual Company Solution or Results - Contained in the Presentation
Background
7
• Based on several companies
– Shared common theme of goals and issues
– Had a large number of applications
• Different technologies, languages, security
• Developed in different geographic areas of the
company
• Documentation was limited, SDLC was tribal
and inconsistent within the company , quality
hero based
A Composite Project is Illustrated in This Presentation
2

5. Migrating - Standard Architecture
• 6 applications needing a refresh
– Each developed by a different center
– Each different technology and some obsolete
– Each center in a different geographical location
• Migrate applications to enterprise architecture
Large Application
•
•
•
•
•
6
Project estimated at 20,000 function points
Over 110 developer person years effort
Selected a Service-oriented Architecture (SOA)
Investors need a 24 month implementation
Nearly all employees will stay on deployed
solutions
– Only 10% of staff will move to the migration
3

6. Application Goals
• Create foundation for next 10 years
– Refreshed technology
– Standards for security, communications, technology
– Add mobile delivery
– Adequate documentation to support
• Improve the product quality
• Reduce the cost for maintenance
• Reduce the redundancy of services
Project Objectives
• Unify the geographic development centers
– SDLC, documentation, communication
•
•
•
•
Introduce agile development
Introduce UML for documentation
Common tools, documentation, architecture
Introduce Metrics for management and
communication
• Transform the organization while supporting
current clients and employees
4

7. Issues to Consider
• Attrition rates in India could exceed 30%
• Little documentation on existing applications
• Engage contractors for most of implementation
– They would have no background in applications
– Needed a short process to get them productive
– As the application evolves transition to employees
• Communication with over 100 people
• 20,000 FP large risk of project failure
– Needed to demonstrate the migration
– Investors and clients
Architecture
• Selected Service-oriented Architecture (SOA)
– Atomic stand-alone , loosely coupled
– No calls between services
– Services and objects are orchestrated
• Standardize on Unified Modeling Language (UML)
– Visual models of object oriented systems
– Pictures are better than words for communicating
• Standardize service communication using
Extensible Markup Language (XML)
5

8. Scrum of Scrums
A Common Agile Approach for Large Teams
11
Agile and Global Issues
•
•
•
•
•
•
•
Most large developments are NOT co-located
Time zones, communication
Customer interaction
Testing and defect resolution
Multiple teams on a single Build lines
High attrition issues and keeping teams coherent
Transition the work from contractors to employees
Only 20 Employees Would be Available at the Start
6

9. 3 Dimensions to the Team
13
Project Organization
• Implementation Teams (121 people)
–
–
–
–
Limited application exposure
Contractors
15 – 8 person teams – 1 manager
All offshore 3 different locations
• Architecture Team (50 people)
– Provides all technical leadership
– Customer for the implementation teams
– Mostly onshore (40/10)
• Project Management Office (PMO) (4 people)
Employees Were all in Architecture Team
7

10. Implementation Teams
5
• Contractors – no product experience
– Need to transition to employees
• Focused on producing high quality code
– Using Style Guides, Testing Standards
– Work assignment/coordination from Architecture Team
– Based on work packages
• Product owner for each team comes from
Architecture team – multiple teams
• Performance and quality held to common
standards
Also known as The Factory
Implementation Teams
• Highly skilled developers
• Develop Services
– UML Documents
– Using Reference
implementations
• Support specific
Architecture teams
• Architecture team
responsible for accepting
output of Factory
8

11. Factory Sprint Team
Scrum Master
5 Developers
2 Test Engineers
Factory Sprint Team
• 8 members per team
• 15 factory teams established
• 1 team was an integration/test
team only
• Teams in 2 Cities in India &
Philippines
• New employees joined the
factory
• Growth to architecture teams
• Factory metrics and
management by a Factory
Manager
Onboarding – A Factory Team
Process
Tools &
Technology
Training
Hands on
(Mock Sprint)
Evaluation
2 Weeks onboarding process
Factory
18
9

12. Team Bilbo Onboarding
Effort (Hrs.)/FP
20
15
10
17.3
Goal 10.24
11.63
9.77
5
• Standard ~ 10.24 hours/fp
• Based on 15fp/dev-mo
• Goals always improved
0
History
POC
Manage scale & grades
Demo Sprint
Sprint -xxx
Sprint – yyy
Mock sprint
evaluation
Production sprint evaluation
Defects /FP
0.2
0.16
0.15
0.1
• 1 defect 600loc
• Goal was 1/Kloc
• Moving to 1/10Kloc
0.05
0.1
0.09
Goal .054
0
History
POC
Manage scale & grades
Demo Sprint
Sprint -xxx
Sprint – yyy
Mock sprint
evaluation
Production sprint evaluation
19
Factory – Supported by Specialist
PMO
QMO
Factory Manager
Team Darth
Team Orrin
Team Dengar
Team Jarael
Team Paploo Team Skywalker
City 1 – Star Wars Characters
Architects & BAs
Team Nimrodel Team Legolas
Team Bilbo
Team Tango
Team Eowyn
Team Gimli
City 2 – Lord of Rings
Code Quality
Evangelist
20
10

13. Factory Metrics
• Evaluated teams like
“machines” against
standards and control
limits
• Presented by Factory
Manager
• Focusing on productivity
and quality
• Teams not compared to
each other
• Leveraged the best for
the rest
• Weekly metric
• Monthly metrics
• Goals always improve
Factory Support
• The Factory is a structure that specializes
– Producing production deployable software
– According to user requirements
– Through an assembly process
• Supported by the Specialist in the Architecture
Teams and the PMO
11

14. Architecture Team
4
• Employees/Contractors
• Specialist
–
–
–
–
Code Quality Evangelist
Test Architects
Business Analysis
Technical Architects & DBAs
•
•
•
•
BAs and Architects formed Sprint Teams
Sequenced work packages
Provided direction to the Factory
Accepted the results of sprints
• Central control of the Intellectual Property
Code Quality Evangelist
• Dedicated Team within the Architecture Group
– All off-shore to be closer to the Factory
• Highly skilled in the language and environment
• Established coding style guides
• Developed reference implementations
– Classes, services, UX and design patterns
• Develop training materials for factory
• Review the code and provide direction and mentoring
to implementation teams. They trawl through the code
• Responsible for the factory teams to produce high
quality consistent code -mentors
Trawling became Troll and thus the Code Troll
12

15. Code Trolls
• We needed our 100+ developers to produce
– Code that meet the standards
– Could be maintained and appeared as if only one person
wrote the code
• More money in maintenance than development
• Worked w/Architects to insure we could build
• Set the standards and review exceptions
– i.e. Cyclomatic Complexity – maintenance indicator
• Tasked to develop a tool to review all code
– Created a tool called – Sledge Hammer
• Identify Factory staff that could be Code Trolls
SLEDGE Hammer
• NCover & NDepend used to find potential trouble in the
code base and reported to the Code Trolls via import into
the Sledge Hammer tool
• Trolls review source code and make recommendations to
sprint teams – work with each team
• Second set of eyes reviews the code during sprints
• Recommendations get added to the product backlog
• Trolls coordinate with the Technical Architects on
recommendations to ensure they are sensible
• Recommendations are tracked and managed through the
Sledge Hammer tool.
13

16. Code Troll Role in Factory
• Fixes/refactors by the sprint team MUST be reviewed
by a Code Troll and be VERIFIED before it is accepted.
• New Code/Changed code gets reviewed after sprint is
delivered.
Code Troll Tasks & Activities During a Development Sprint Cycle
Code Quality Metrics
Code Generated on Day 7 &
Review
Day 10
Code Quality Data Moved to
Code Quality Improvement
Management System
Sledge
Hammer Tool
Development Sprint Cycle
Recommendations
Code Quality
Review
Sprint Teams
Test Architects
•
•
•
•
•
•
•
Create sprint test guidance
Own the test architecture
Review test cases
Insure all the sprint test cases support goals
Manage the Integration sprint team
Automated testing
Enter defects in the backlog for architect
scheduling
• Review and verify requirements delivered
• Coordinate Build lines with architects
14

17. Technical Architects/DBAs & BAs
•
•
•
•
•
Includes UX/UI team
Document Requirements
Design work packages
Ensure sprint teams deliver desired requirements
Architecture communication
Requirements &
Design Verification
Verify
Implementation
Design Work
Packages
Product
Backlog
Architects & BAs
Sprint
Teams
Architecture
Communication
Production
Shippable
Incremental
Build
Factory
Work Package
• UML package
– Reusable code identified
– Reference implementations are included
– Common solutions
• Video of the UX/UI elements
– UX/UI Style Guide
•
•
•
•
Contains all of the requirements
Sized and sequenced for sprint implementation
BA and Architect support the WP during the sprint
BA and Architect are customers for sprint
acceptance
15

18. BA & Architects w/WP
Scope
Definition
• Senior BA – Provides a scope definition of each package
• Lead Solutions Architect - Reviews/Approves the scope definition of each package
Package
Reviews
• Senior BA – Validate that the requirement and use case for each package is correct
and meets our define standards
• UI Lead – Validate that the mock UI for each package follows defined standards
• Architect – Validate requirements and use cases are clear and follows defined
standards
Package
Approvals
• Lead Solutions Architect –Perform final approval on requirements, use cases and UI
for each package
Package
Handoff
• BA – Handoff requirements and use cases for each package
• Architect – Handoff design artifacts for each package
• Scrum Master – Accepts approved package for development
Sprint
Reviews
• BA – Verify that all requirements are meet prior to sprint demo. Review the test
case coverage for each package.
• Architect – Verify that the design of each package is correct.
31
Architecture Sprint - Process Flow
Package Analysis
(Day 1-2)
Joint
Application
Design (JAD)
Development
(Day 3- 11)
Scope Sequence , UI/UX
movies,
Reviews
(Day 11 -14)
Review/Approval by
Leads
Approval
(Day 15)
Approval Lead
Solution
Architect
Architecture
Teams
JAD Backlog
Package
Acceptance
Factory
Backlog
32
16

19. Project Management
3
• Project scheduling and tracking
– Planning was done on a 90 day rolling wave
• 1 planner dedicated to the factory and metrics
• Scope watchdog – highlighted scope creep
• Coordinated communication
– Conducted audits for documents and results
– Acted as the honest brokers
– Management communication
– Customer communication
• Risks and Action Register
The PMO
• Managed the connections to organizations
outside of development
– Transformation of processes to include UML/XML
– Training for other organizations
• Owner of the build lines and release schedule
– Periodic releases for clients and internal
stakeholders
– Manages the Change Control Board
17

20. Factory Sprint - Process Flow
Requirement Analysis
(Day 1-2)
Work
Package
Analysis
Development
(Day 3- 11)
Work
Package
Handoff
Approve
Sprint
Scope
Demo
Code
Walk-through
Post Questions
on design &
requirements
BA/,
Architect
Test Case Execution
and Bug Fixes
Estimate Scope
& Seek
Approval
Sprint
Teams
Demo
(Day 15)
Testing
(Day 11 -14)
Development, Code
Review &
Test Case Creation
2
No
Scope
affected
Yes
Provide
Clarification
Log Change
Request
Sprint
Acceptance
CCB Review
Retrospective
35
Take Away
1
• Architects wanted to write code
• Code Trolls were very effective
• The Factory
– High degree of employee satisfaction
• Liked the routine of the sprints
• Liked that software developers were the clients
– Needed firm plans for transition of employees
– Needed to add training for “what” we were building
• Maintained library of reusable code and reference
implementations
– Needed standards and tools for easy reuse
– CMS to manage content
18

21. Learning
• Managing the build lines was a full time job
– Multi Factory building on a line
– 1 Final Integration line
• Clients struggled with build lines that had partial
functions – so many things going on
• Communication always needed to be monitored
– Establish overlap hours for meetings
• Difficult for companies to adapt agile without a plan and
executive buy in and commitment
• Selecting the right contractors to help train staff was key
Questions
19

22. Jim Lynn
jim@sutoconsulting.com
Notes
•
Certain images and/or photos in this presentation are the copyrighted property of 123RF Limited, their Contributors or Licensed Partners
and are being used with permission under license. These images and/or photos may not be copied or downloaded without permission
from 123RF Limited.
40
20