![Page 1: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/1.jpg)
Coordinating Change and Release Management in
Games DevelopmentGordon Milligan
QA and Release ManagerRealtime Worlds
![Page 2: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/2.jpg)
Intro
• Change control from games perspective• Case study discussing our initial forays
into change control for a large team• Discuss what we learned• A further case study on how we work now
![Page 3: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/3.jpg)
Who are we?
• Software technology company in entertainment sector
• Based in Dundee, Scotland• Largest independent games company in
Scotland–Approximately 300 staff
![Page 4: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/4.jpg)
![Page 5: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/5.jpg)
Change Control
“process of managing, and controlling, how a product is changed”
–Michael E. Bays
![Page 6: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/6.jpg)
Change Control Considerations
• Build changes • Test changes• Review changes
–Quality–Risk
![Page 7: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/7.jpg)
Maintaining Quality: Stability
• Continuous Integration• Daily smoke test at a minimum• Full regression passes• Targeted feature testing• Automated Testing
![Page 8: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/8.jpg)
Maintaining Quality: Overall
• Continuous Review–Designers–Artists–Producers–Creative directors
![Page 9: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/9.jpg)
Risk
• Analyse risk of changes and likely impact–Highlight concerns to developers,
producers, etc.• Attempt to make developers aware of the
risk of their changes
![Page 10: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/10.jpg)
Evolution of Change Processes
• Initially we were naive–Lack of understanding of the issue–Size of team was bigger than any of us
had worked on–Hit many problems: no real control
• Educated ourselves and continued to review and evolve or change processes
•
![Page 11: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/11.jpg)
Development Team Make Up
• ~55 % Software Engineers• ~20 % Artists• ~20 % Level/Script Designers• ~5% Production
![Page 12: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/12.jpg)
![Page 13: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/13.jpg)
![Page 14: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/14.jpg)
![Page 15: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/15.jpg)
Daily Build is Life Blood
• ...for non-programmers• Relied on to allow artists/designers to
receive new code, design & art content• Allows non-coders to develop their content
against recent changes• Provides buffer against bleeding edge
![Page 16: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/16.jpg)
Crackdown Approach
![Page 17: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/17.jpg)
Compunding Problems
• Lack of due diligence on check-ins• Long build verification turnaround• 70 people working on same codeline ==
continuous churn–Constant changes to common files–Productivity loss through broken builds
![Page 18: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/18.jpg)
Process Changes
• Introduced stringent check-in process–Could take 2-3 hours all in (or much
longer on occasion)
![Page 19: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/19.jpg)
Example Check-in Process
• Build data & code• Blow up car• Drive car • Kill NPC• Complete mission X• Etc.
![Page 20: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/20.jpg)
Process Changes
• Introduced stringent check-in process–Could take 2-3 hours all in (or much
longer on occasion)• Analysed and improved build verification
speed on build servers
![Page 21: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/21.jpg)
Reaction to Changes
• Less check-ins so code would change underneath feature development–Also code/data sat on dev machines for
longer so issues hidden• Friday afternoon became favoured check-
in time• Coders loathed submitting changes
![Page 22: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/22.jpg)
Further Process Changes
• Restricted times when check-ins could happen
• Created check-in queue (FIFO)
![Page 23: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/23.jpg)
Problems
• Change submission bottleneck–Whose next in queue?
• Coders not checking in regularly enough so always seeing BIG CUMULATIVE changes
• Major breakages of build and resultant loss of prodctivity
![Page 24: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/24.jpg)
Crackdown: Lessons Learned
• Say NO to...–Large # devs submitting change to same
location–Long build verification cycles (if
possible)–Allowing developers to check-in code
without review
![Page 25: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/25.jpg)
Main thing I learned...
Trade OffMake check-ins easy and less restrictive
VMaintain a stable product at all times
![Page 26: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/26.jpg)
Project X
• Consists of multiple SCRUM teams working on separate areas
• Mixture of industry veterans through to new graduates
• Have a measured approach to quality• Similar team make-up to previous project
![Page 27: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/27.jpg)
Development Process Alterations
• Introduced buddy check-in process to share knowledge/responsibility & reduce risk
• Use unit, integration and higher level automated testing– In addition to black box testing
• Code leads (& QA) meet most mornings
![Page 28: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/28.jpg)
Development Process Alterations
• Mainline is no longer a development codeline and is owned by QA–Used to consolidate features
• SCRUM teams each have own development codeline which they own–Allows bugs to be found early and code
to mature
![Page 29: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/29.jpg)
![Page 30: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/30.jpg)
Development Process Challenges
• Educate developers on:– Benefits of branching
• In particular task & prototype branches to reduce risk
–How to use version control feature set• Encourage regular short check-ins
![Page 31: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/31.jpg)
Benefits of Current Setup
• Lightweight check-in process for team branches–Allows quick change submission–Changes submitted earlier and so have
time to mature and for issues to be found
• SCRUM Masters own team branches
![Page 32: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/32.jpg)
Benefits of Current Setup
• Strict policy for mainline integration helps to ensure its quality–Rarely is it broken
• I have closer control of changes hitting mainline–More time to review these and assess
risk
![Page 33: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/33.jpg)
Hurdles crossed during changes
• Resistance to methodology change–Stakeholder buy in
• Educating developers on:–Branching: when to, how to–Using version control system features
• Communication
![Page 34: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/34.jpg)
Summary
• Discussed types of changes & dependencies involved in game builds
• Talked in detail about the problems of trying to over control change
• Put forward a new strategy we have developed to control change
![Page 35: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/35.jpg)
Tools
• Perforce • CI System
–CruiseControl.net–Moving to Team City
• Jira: Feature tracking (with help of p4 jobs)
![Page 36: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/36.jpg)
![Page 37: Coordinating Change And Release Management In Games Development](https://reader036.vdocument.in/reader036/viewer/2022062511/54c1e7f74a7959ea328b4585/html5/thumbnails/37.jpg)
Resources
• Software Configuration Management Patterns–Berczuk and Appleton
• Software Release Methodology –Michael E. Bays
• Practical Perforce–Laura Wingerd