Denis Vanpoucke. Agile kelionės:nuo transformacijos iki planavimo dideliu mastu
Sigis jermolovicius chasing code quality in huge multi location team project
1. CHASING CODE QUALITY IN HUGE MULTI-
LOCATION TEAM PROJECT
October 15, 2012 www.ExigenServices.com
2. GLOBAL EXIGEN SERVICES
Riga St. Petersburg Nizhniy Novgorod Kazan
R&D Center R&D Center R&D Center R&D Center
San Francisco New York
Headquarters
Europe
Eastern Europe
Wuxi
US London Asia-Pacific R&D Center
Minsk Suzhou
R&D Center R&D Center
Frankfurt
Odessa
R&D Center Dnepropetrovsk
• Founded in 2000 R&D Center Adelaide
• 1’500 professionals
Vilnius Johannesburg
• Revenue approx. $70M San Paulo R&D Center
Christchurch
2 www.ExigenServices.com
3. EXIGEN SUITE COMPONENTS
Sales and Customer Service DISTRIBUTIONCORE Distribution Management
Business Invoice & Payment New
New Policy
Policy Product FNOL Adjudicaton
Business Servicing
Processes Collection Processes Business Servicing Management & Settlement
&
Core BILLINGCORE POLICYCORE CLAIMCORE
Functions
Bill Plan Payment PRODUCT Under- Claim
Claim Claims
Claim
FACTORY
Management Management Rating Writing Processing Financials
Processing Financials
Common Accounts Reporting & Tasks & 3rd Party Correspondence
Sub-Ledger
Services & CRM Compliance Decisions Mgmt Management
Data & Sys Billing Claims Commission Self-Service Reinsurance Analytics 3rd Party
Integration Interface Interface Interface Portal Interface Interface Interface Interface
Operations Configuration Document Content Business Process & Business
Security Activity
Management & Scheduling Generation Management Rules Task Mgmt Monitoring
3 www.ExigenServices.com
4. WHEN YOU START FAILING
Agile works says:
One location, team, scrum master and product owner
Everybody are universal and senior;
But in sometimes we have
Multi location, teams, scrum masters and PO’s
And mixed level;
And here problems arise.
4 www.ExigenServices.com
10. WHAT TO DO?
Options
Fight with continuous integration failures every day
Fail fast. Setup your continuous integration to no accept failing
code.
10 www.ExigenServices.com
11. FAILING FAST
“Failing fast” for development means getting response about your
code quickly, from:
compiler
deployment
unit and integration test
UI tests
static quality parameters
And all those answers should be given before code gets to source
control.
11 www.ExigenServices.com
12. MULTIPLE TEAMS – MULTIPLE
CONTINUOUS ENVIRONMENTS
12 www.ExigenServices.com
14. BENEFITS
Team continuous integration environment is effected only by team
changes.
Only finished and tested features will get to central. Gatekeeper
would be responsible to merge.
14 www.ExigenServices.com
15. STILL TO BE IMPROVED
There is still possibility that gatekeeper will push bad code
We don’t reach our ultimate goal - STABILITY of central.
15 www.ExigenServices.com
16. ONE STEP FURTHER - TEAM-MERGE
To achieve our goal we created CI job that doesn't allow bad code.
It does automatic code merge, compile, test and push.
– rights write to central are revoked for anyone and only "special"
"team-merge" user can do merge.
– only possible way to commit code from team-branch to central
is through "team-merge" script.
16 www.ExigenServices.com
19. FINAL TIPS
Your environment should be stable enough to do this approach.
Better have small number of working test rather than huge number
of failing test.
Each team has own CI plans – hardware is required
19 www.ExigenServices.com
21. CODE REVIEW OR NOT CODE REVIEW
Code review, if:
– You plan long life cycle for your product
– Your customer insist on high quality and you have maintenance
contract.
Don’t code review, if:
– Your product has short life cycle (demos, POC, etc.)
– You don’t care about maintenance after it’s done
21 www.ExigenServices.com
22. CODE REVIEW
There are various proposals on code review –
Pair Programming
Review about do less than 100%, with rules, like
Review only Selected commits.
Review only commits from juniors
Random reviews
Review 100% of code
We chose 100% review.
22 www.ExigenServices.com
23. CODE REVIEW AFTER COMMIT
Guarantee that reviewed code is committed
No special actions required if reviewed code passes
Issues:
In case code doesn’t pass you have bad code in repository until it’s
fixed.
Bad code effects others
23 www.ExigenServices.com
24. CODE REVIEW BEFORE COMMIT
Review patches or review branch.
Bad code doesn’t effect others.
Issues:
You have no guarantee that good code reviewed will get to
repository. As somebody needs to merge it.
What we needed is:
Automated merge on code review pass.
24 www.ExigenServices.com
25. 100% CODE REVIEW
What we want is
1. Work as a team.
2. Review all code changes
3. Do not effect others
Therefore we need code review systems what would assure 100%
review.
Other requirements:
DVCS-backed review system
Acts as a front end for to central DVCS-repo
When review is complete auto merge with default branch
25 www.ExigenServices.com
26. CODE REVIEW TOOLS
For git users there is good tool called gerrit
http://code.google.com/p/gerrit/
If you use mercurial (hg), it’s bit more complicated – as where is
no analogue for gerrit for hg.
Our approach was to use reviewboard.org and mercurial enhanced
with python scripts what are mimicking gerrit approach.
To initiate review you should call “hg postreview”
26 www.ExigenServices.com
30. CONTROL CODE QUALITY
We want to control basic parameters of code quality
Static code violation
Code unit and integration test coverage
Other code parameters
We can do it effectively with tool called Sonar
30 www.ExigenServices.com
31. METRICS
What you want to measure?
– Test Coverage?
– Static code violations?
– Code metrics?
Answer to this is SONAR
Automates metrics and creates reports.
Allows to analyze code to the code line details.
sonarsource.org
31 www.ExigenServices.com
37. WHAT’S NEXT WITH SONAR
Good is that we have tool to control
Bad is that what people are still people and tend to do crappy
code.
number of violations is increasing.
still repeating same errors.
code is unmaintainable
37 www.ExigenServices.com
38. OUR APPROACH
Decide that violations are critical for you and what you can live
with.
What we do:
Using multistage commit we don’t accept code with blocker
violations.
Commit is declined and developer should rework their code
before next commit.
Use personal violations view to work with particular developers
coding style.
38 www.ExigenServices.com
41. TIPS
Declare violations as critical first.
Allow teams to fix them.
Increase severity to blocker after all existing are fixed
Decline new violations of same type
41 www.ExigenServices.com
Project size over 100 people total Large development activities in Vilnius, Suzhou, MinskProject management located in San Francisco
In the Component View, Business Processes and Functions (Claims, Billing, Policy) are composed of Common services which themselves adhere to a set of defined interface contracts.Services are implemented internally as JavaBeans and orchestrated through IoC via Spring configuration.All interface implementations rely on a common set of operational facilities for configuration, security, document generation, content management, rules and BAM.
If you succeed and you product gets many customer, with money paying for news features. You will need to expand your teams and here problems start. If you succeed and start growing, sometimes you get into multi-location, multi-team, multi product owners situation…
This is view Jenkins for one team CI environment, in most cases all is good, we have green build. Even if build fails it’s easy to fix it as people who broke it are near and can do it quickly.
CI is always red, one issues is find, while fixing it, new issues is created and this is eternal cycle. Teams get annoyed by the fact that CI is never working, they are not confident to take new source code from source control, just because it’s never stable and nobody knows then it will be fixed.
Fight – go investigate each incident, find who can be blamed, request fix, wait for fix and later find out that previous problem get fixed, but meanwhile new is introduced by other developer. After some time you will understand that fighting is not the answer, you will waist all you time and effort for it. Fail fast -
Each team has it’s own CI environment + central CI environment exist as well.
Each team has it’s own branch or repository close there they do all the coding. Only after Feature is done and team decided that code is good enough for integration it gets pushed to central. Push is done by one of the team members who does the “gatekeeper” role, he has to ensure the pushed code will go through.
Team CIPeople are less annoyed by CI failures. It’s easier to fix as people are co-located. It can be tested much earlier in the team scope.GatekeeperOne person who does the push will do merge and adequate testing before push. Less pushes, less builds, less failures.
People, even seniors do mistakes or sometimes are lazy to do adequate testing.
Environment stability: this is most important! If you environment produces unpredictable errors, this approach will not work.Test: If you have large number of failing test, it better to @ignore all of them and keep only stable/working test… Later you can revisit failing test and fix them one by one. CI: you will need huge number of hardware to run such setup.
It is generally accepted that the cost of a software bug goes up dramatically, perhaps exponentially, as the time between introduction and correction goes up.Typically customers don’t pay for code review as such, but if you need to achieve high quality this unavoidable and you should plan in ahead.Decision on code review should be made by project, there are many decision points – you need to know you project quality requirements (it doesn’t make sense to invest into short time project), project size (it doesn’t make much sense to do code review for POC’s or demos). Code review give benefits for larger, longer time to be supported and high quality projects.
Pair programming practically speaking does code review due the fact that two people are working on same code. But on the other end this is not true code review as it involves two developers who develop same thing for same deadline. Less than 100% 30% this is bare minimum to make sure developers care about their code and know that it might happen that their code will be reviewed. It’s Best Cost-Effect ration.This is level then developers are aware about possible code review and write better code. You can chose to review only selected review items – only complex changes, only juniors code, only new features, etc. Or 100%just to make sure nothing is missed.sounds quite much as it accounts small fixes as well, but this is practically this is the only approach ensuring highest quality.
Work as a team – everybody should feel part of team and understand that happens in other parts of code. Review all code changes – even small changes can create big problems… Code review minimizes this.Do not effect others– make sure code is compatible with others people work.Assure – most important is that this system has to assure code review is done. Don’t leave open windows to small hacks.
Coverage is calculate by testing code lines touched (executed by Unittests). It allows you to understand where you have good coverage and where you need add some work.