This document describes the experience of a large team developing and maintaining a large scale C# and SQL application over 15+ years using agile practices. Key aspects included:
- A team of 60 developers maintaining over 10 million lines of code
- Strict consistency enforced through architecture, naming conventions, patterns, and over 343,000 automated tests
- Continuous integration and distributed testing on developer machines to run the large test suite regularly
- A focus on reducing technical debt and making improvements that benefit all developers
2. What was the project?
C# Winforms
SQL Server database
Large CRUD system
App deployed to client sites
Started in 2001
Market leading product
3. What was the project?
Low volume
Performance important, but not critical
Goal was a stable, consistent, feature-rich app
Rapid release cycle (after any successful build)
4. In 2006…
60 Developers
5 million lines of code (including tests)
~100 solutions, 250 projects
300+ db tables
160,000 automated tests
5. Today…
60 Developers
~10 million lines of code
485 solutions
470 db tables
343,090 automated tests
6. The Pit of Success
We want our customers to simply fall into winning
practices by using our platform and frameworks
– Rico Mariani
Build platforms [where] developers just fall into
doing the “right thing”... Types should be defined
with a clear contact that communicates effectively
how they are to be used (and how not to)
– Brad Abrams
7. Pit of Success – Technical
Factors
Inheritance
Template Method pattern
Strict naming conventions
Common reusable controls
Own type system
Single generic repository – handled common
scenarios in CRUD, transactions, referential
integrity, caching, persistence
8. Pit of Success
Tried to avoid too much divergence in code
Made it difficult for people to stray from the “true
path”
12. Consistency
70 % of a dev’s time is understanding existing code
– Peter Hallam [2]
If you can reduce that even by a small amount, it’s
significant productivity gain
13. Consistency
Consistent approach to everything
Naming conventions – project, solutions, file locations,
namespaces, class names, methods, database tables
and columns
Formatting
UI look and feel (achieved through base controls)
Patterns of design
14. Consistency
Standardised on a core set of technologies that
best fit the application
Avoided “technology soup” support and
maintenance
Reduced learning curve for new developers
15. Insane things we did
60 devs, no feature branches!
SourceSafe
Highly sensitive to the build breaking
True continuous integration
Mitigated by clients opting in to “alpha”, “beta” or
“stable” versions
Two large re-writes of architecture
Result was more suited to our needs than if we had
kept the first and refactored it progressively
16. How did we build?
Full build from batch file
Update source files
Solutions built in dependency order defined in xml
All binaries output to one folder
Build on dev machine = 20-25 minutes
Build server with RAID = 4 minutes
Quick build tool (written in c#)
Only build solutions that have changes
17. Development - Architecture
Built our own architecture
Re-usable controls
Comprehensive base classes
Own type system
Primitive types as well as Custom types
Could define helper methods .IsEmpty,
.SubstringSafe() , IsInRange()
18. Development - Architecture
Architecture constantly extended
Approach was always unified though the
architecture
Enabled good ideas to have widespread effect
19. Development – Technical debt
Management support for improving bad code
Aggressive attitude to technical debt
Very little dead code
Example: methods with 7 overloads that can’t be
deleted because don’t know who’s consuming them
20. Development – Technical debt
Constant small amount of pain
Rare occurrences of large amounts of pain
Once a hard cost has been paid, make sure that
nobody ever has to pay the cost again
21. Development - Custom Tools
Automated repetitive and error prone tasks
Custom ORM generator
Database upgrading tool
GUI binding helpers to ensure targets exist
Build monitor system tray application
Code Generators for new modules
Automated deployment tool
22. Testing
Two types of tests
Business Tests: user functionality
Meta Tests: code design and contracts
23. Testing
Most of our tests were actually integration tests of
varying scope
Didn’t use a mocking framework (not yet well
known)
Created our own stubs by inheriting from the
concrete class and overriding any methods that
used resources, etc.
24. Business Tests
Generally state based logic
Easy to read and write
Fairly resilient throughout refactoring
Derived from a base TestCase with ~25 basic tests
Array properties don’t return null
Properties don’t throw NullReferenceExceptions
Don’t alter the state of the object before shown in GUI
26. Meta Tests
Don’t test direct functionality
Enforced consistency, patterns and code contracts
Implemented using reflection and attributes
Slow to run, but very valuable
27. Meta Tests – Examples
Check code conforms to conventions and contracts
Public constructor with a specific parameter exists
Assemblies are strong named
Check for memory leaks
Check that each model had an associated test
[TestsSubclassesOf(typeof(BusinessBase))]
public abstract class BusinessBaseTest
{
// contract tests here
}
28. Meta Tests
The Meta Tests help to enforce the rules, rather
than having a “bad cop” in your team
Not political or personal
Once established, it gave power to the rest of the
team to contribute new tests and rules
29. Meta Tests
Allowed the architecture to grow rapidly
Already know the code conforms to a particular
standard
30. How do you run 343,000 tests?
Initial approach similar to CruiseControl
One machine building and running all tests
Results emailed after completion
… stalled once we hit 25,000 tests
… Took 4 hrs to run
Feedback cycle wasn’t fast enough
32. Distributed Testing Framework
Highly customised
Tests distributed to up to 60 agents
Build time around 1 hour
if built on single agent = 30-40 hours
Package on successful build
Agents ran on idle developer machines
Ran 24/7
34. Distributed Testing Framework
Developer submitted builds could be processed
(i.e. pre-checkin)
Task tray to signal progress
Red light = broken tests, broken build or checkin
embargo
Green light = OK to check in
35. Team
Very Flat structure
6-8 devs in architecture team (“Blue Code”)
Business & GUI architecture
Development tools
Testing framework
Database development
Upgrading code
Most other developers building business rules into
software (“Red Code”)
37. Team - Architecture
Made up less than 20% of team
Improved productivity by much more than 20%
Allowed devs to focus on smaller slice of stack and
get quicker feedback cycles
38. Team - Business
Low barrier to entry for development
Junior or new developers could be productive very
quickly
Varied skill sets in the team were utilitsed
accordingly
Technically minded people drawn to architecture
Business minded people focused on implementing
customer requirements
39. Team – Sharing Information
Free flowing collaboration
Some development overhead in helping other
people
Some information silo-ing
40. Team – Pair Programming
Initially strict pairing
Became “organic”
Not enforced
Never discouraged
41. Team – Work Ethic
People took a lot of responsibility and
accountability of their own accord
Boy scout rule – leave things cleaner than when
you found it
Always fix broken windows
42. Work satisfaction
Low turnover rates for first 4 years
Lots of smart people to work with
More meritocracy than political
Developers took pride in their work
Ability to implement large scale change open to
everyone
43. Work Satisfaction – flip side
Hard to have interesting work for 60 developers
Could be pigeon-holed into the business layer
More stress than an average agile environment
Long work hours implied because there’s only one
build – responsibility to keep the dev machine
going. No leaving the build broken or failing tests
throughout lunch or overnight.
44. Conclusion
Team made a lot of investment in design and
conformity
Easy to do with a CRUD system, harder with other
types of business software
Project has expanded and moved on, but the
architecture has evolved with it and is still in use
46. References
[1] Brad Abrams, The Pit of Success
http://blogs.msdn.com/b/brada/archive/2003/10/02/50420.aspx
[2] Peter Hallam, What do Programmers Really do Anyway?
http://blogs.msdn.com/b/peterhal/archive/2006/01/04/509302.aspx