Slide deck from a talk I gave to the BCS Configuration Management group in London. In this talk I discussed the issues of large scale software development and how you can control change in these environments. I highlight methods we have tried to improve our change control processes and where these have been a success or failure.

1. Coordinating Change and Release Management in Games Development Gordon Milligan QA and Release Manager Realtime Worlds

2. Intro Change control from games perspective Case study discussing our initial forays into change control for a large team Discuss what we learned A further case study on how we work now

3. Who are we? Software technology company in entertainment sector Based in Dundee, Scotland Largest independent games company in Scotland Approximately 300 staff

5. Change Control “process of managing, and controlling, how a product is changed” Michael E. Bays

6. Change Control Considerations Build changes Test changes Review changes Quality Risk

7. Maintaining Quality: Stability Continuous Integration Daily smoke test at a minimum Full regression passes Targeted feature testing Automated Testing

8. Maintaining Quality: Overall Continuous Review Designers Artists Producers Creative directors

9. Risk Analyse risk of changes and likely impact Highlight concerns to developers, producers, etc. Attempt to make developers aware of the risk of their changes

10. Evolution of Change Processes Initially we were naive Lack of understanding of the issue Size of team was bigger than any of us had worked on Hit many problems: no real control Educated ourselves and continued to review and evolve or change processes

11. Development Team Make Up ~55 % Software Engineers ~20 % Artists ~20 % Level/Script Designers ~5% Production

15. Daily Build is Life Blood ...for non-programmers Relied on to allow artists/designers to receive new code, design & art content Allows non-coders to develop their content against recent changes Provides buffer against bleeding edge

16. Crackdown Approach

18. Process Changes Introduced stringent check-in process Could take 2-3 hours all in (or much longer on occasion)

19. Example Check-in Process Build data & code Blow up car Drive car Kill NPC Complete mission X Etc.

20. Process Changes Introduced stringent check-in process Could take 2-3 hours all in (or much longer on occasion) Analysed and improved build verification speed on build servers

21. Reaction to Changes Less check-ins so code would change underneath feature development Also code/data sat on dev machines for longer so issues hidden Friday afternoon became favoured check-in time Coders loathed submitting changes

22. Further Process Changes Restricted times when check-ins could happen Created check-in queue (FIFO)

23. Problems Change submission bottleneck Whose next in queue? Coders not checking in regularly enough so always seeing BIG CUMULATIVE changes Major breakages of build and resultant loss of prodctivity

24. Crackdown: Lessons Learned Say NO to... Large # devs submitting change to same location Long build verification cycles (if possible) Allowing developers to check-in code without review

25. Main thing I learned... Trade Off Make check-ins easy and less restrictive V Maintain a stable product at all times

26. Project X Consists of multiple SCRUM teams working on separate areas Mixture of industry veterans through to new graduates Have a measured approach to quality Similar team make-up to previous project

27. Development Process Alterations Introduced buddy check-in process to share knowledge/responsibility & reduce risk Use unit, integration and higher level automated testing In addition to black box testing Code leads (& QA) meet most mornings

28. Development Process Alterations Mainline is no longer a development codelineand is owned by QA Used to consolidate features SCRUM teams each have own development codeline which they own Allows bugs to be found early and code to mature

30. Development Process Challenges Educate developers on: Benefits of branching In particular task & prototype branches to reduce risk How to use version control feature set Encourage regular short check-ins

31. Benefits of Current Setup Lightweight check-in process for team branches Allows quick change submission Changes submitted earlier and so have time to mature and for issues to be found SCRUM Masters own team branches

32. Benefits of Current Setup Strict policy for mainline integration helps to ensure its quality Rarely is it broken I have closer control of changes hitting mainline More time to review these and assess risk

33. Hurdles crossed during changes Resistance to methodology change Stakeholder buy in Educating developers on: Branching: when to, how to Using version control system features Communication

34. Summary Discussed types of changes & dependencies involved in game builds Talked in detail about the problems of trying to over control change Put forward a new strategy we have developed to control change

35. Tools Perforce CI System CruiseControl.net Moving to Team City Jira: Feature tracking (with help of p4 jobs)

37. Resources Software Configuration Management Patterns Berczuk and Appleton Software Release Methodology Michael E. Bays Practical Perforce Laura Wingerd