build infrastructure: what it is, why you need it, and how to use python to fuel it

COPYRIGHT 2015 @ UNITY TECHNOLOGIES

BUILD INFRASTRUCTUREWhat It Is, Why You Need It, and How to Use Python to Fuel It



WHAT IS BUILD ENGINEERING? BUILD ENGINEERING IS THE GLUE THAT HOLDS THE SOFTWARE DEVELOPMENT PROCESS TOGETHER.


Text Styles

switch between levels

Level 1 = Bullet 24 pt. Level 2 = Bullet 24 pt. Level 3 = Bullet 20 pt. Level 4 = Bullet 18 pt. Level 5 = Bullet 16 pt.

+ TAB

Alternatively,

EXAMPLES OF BUILD ENGINEERING

• Choosing, configuring, customizing, and maintaining a CI server

• Applying/optimizing development models (branch-based, patch-based, etc)

• Build optimizations (distributed builds, bulk builds, etc)

• … and much more



WHY DO I CARE?


• Na’Tosha J. Bard • Copenhagen, Denmark • Engineering Tools Lead at Unity

Technologies (5+ years) • Open-Source & Linux Enthusiast • Married • 2 cats

ABOUT THE SPEAKER


Text Styles



+ TAB

Alternatively,

FOR YOUR REFERENCE

Slides will be available here:

http://natoshabard.com/speaking

Keep in touch:

Twitter: @natosha_bard E-mail: [email protected]



WHAT IS UNITY?


Text Styles



+ TAB

Alternatively,


Text Styles



+ TAB

Alternatively,

WHAT IS UNITY?

• Game Engine & Content Creation Toolkit • Editor for Mac OS X & Windows • Experimental build for Linux

• Runtime ported to 22 platforms • Mostly written in C++ • Gameplay scripts written in C# • Scripting engine for gameplay code built on

top of Mono



BUILD ENGINEERING AT UNITY TECHNOLOGIES


Text Styles



+ TAB

Alternatively,

2010

• Unstable SVN Server • All developers commit directly to trunk • Unstable CI Server • 6 VisualStudio and Xcode projects wrapped

in Perl • Automated Tests nightly on 2 platforms • 4 machines for building • Deployment scripts running from developer

machines • No automatic verification of deployed builds


Text Styles



+ TAB

Alternatively,

2015

• DVCS with Mercurial • Branch-based development • Project files generated with Jamplus • Stable CI Solution (Internally-Developed) • Automated tests running on developer branches

and on our mainline on various platforms • Build cloud of over 200 virtual and physical

machines • Web-based publishing portal for deploying

builds



OUR RELEASE CYCLES WENT FROM AS LONG AS 18 MONTHS TO EVERY 3 MONTHS WHILE OUR DEVELOPMENT TEAM GREW 400%.



FOUNDATION OF BUILD INFRASTRUCTURE



THE 5 PILLARS OF BUILD INFRASTRUCTURE • ONE-CLICK BUILDS • BUILD AUTOMATION • CONTINUOUS

INTEGRATION • AUTOMATED TESTING • AUTOMATED

DEPLOYMENT



WHAT ON EARTH DOES THIS ALL HAVE TO DO WITH PYTHON? PYTHON IS AN AMAZING LANGUAGE FOR WRITING DEVELOPMENT TOOLS.



ONE-CLICK BUILDS: BUILDING YOUR SOFTWARE WITH JUST ONE CLICK


Text Styles



+ TAB

Alternatively,

BENEFITS OF ONE-CLICK BUILDS

• KISS principle • Build early, Build often • Groundwork for other software

development principles


Text Styles



+ TAB

Alternatively,

GETTING STARTED WITH ONE-CLICK BUILDS

• Depends on your application • Possible Solutions: SCons, Waf • Cohesive buildsystem • Keep platform-dependent issues in mind!


Text Styles



+ TAB

Alternatively,

WHAT UNITY DOES

• Currently using Jamplus and Perl • Didn’t find a good Python-based tool that

met all of our requirements :-( • Migrating to Gradle • http://gradle.org/migrating-a-large-c-

codebase-to-gradle/



BUILD AUTOMATION:A BUILD SERVER TO RUN YOUR ONE-CLICK BUILDS


Text Styles



+ TAB

Alternatively,

BENEFITS OF BUILD AUTOMATION

• Verification • Parallelization • Groundwork for other software

development principles


Text Styles



+ TAB

Alternatively,

GETTING STARTED WITH BUILD AUTOMATION

• Depends on feature set you need • Possible Solution: Buildbot • Give careful thought to buildserver/

buildsystem configuration split


Text Styles



+ TAB

Alternatively,

HOW UNITY DOES IT

• Custom solution built on top of Buildbot - “Katana”

• https://github.com/Unity-Technologies/katana/ • http://mariangemarcano.blogspot.dk/

• Katana’s Build Cloud configured with Salt • http://saltstack.com/



CONTINUOUS INTEGRATION:CONTINUOUSLY RUN THOSE AUTOMATED BUILDS


Text Styles



+ TAB

Alternatively,

BENEFITS OF CONTINUOUS INTEGRATION

• Increased Developer Productivity • Helps avoid “Integration Hell” • Automatically identifies exact failure point


Text Styles



+ TAB

Alternatively,

GETTING STARTED WITH CONTINUOUS INTEGRATION

• Generally, the same solutions as for Build Automation apply here

• Note: Sometimes “Frequent Integration” is good enough


Text Styles



+ TAB

Alternatively,

HOW UNITY DOES IT

• We don’t…yet • Frequent integration • Nightly builds on mainlines • On-demand builds on other branches



AUTOMATED TESTING:RUN TESTS ON THOSE CONTINUOUSLY-RUNNING BUILDS


Text Styles



+ TAB

Alternatively,

BENEFITS OF AUTOMATED TESTING

• Provides protection against regressions • Saves human resources • Keeps test suites from bit-rotting


Text Styles



+ TAB

Alternatively,

GETTING STARTED WITH AUTOMATED TESTING

• Find a test framework for your technology • PyUnit (unittest), nose • Sometimes creating your own frameworks

is an answer (i.e, High-Level Integration Tests)


Text Styles



+ TAB

Alternatively,

HOW UNITY DOES IT

• No Python :-( • Custom test frameworks in C# • Heavy use of NUnit • Custom test runner in Perl



AUTOMATED DEPLOYMENT:EASILY DEPLOY THOSE TESTED BUILDS


Text Styles



+ TAB

Alternatively,

BENEFITS OF AUTOMATED DEPLOYMENT

• Less error-prone than manual publishing • Easy to verify • Easy to track release history


Text Styles



+ TAB

Alternatively,

GETTING STARTED WITH AUTOMATED DEPLOYMENT

• Often related to build automation/CI solution

• Highly dependent on your organization’s server infrastructure and type of software

• Start with a simple script and expand • Consider integrating directly with your

already-existing build server solution


Text Styles



+ TAB

Alternatively,

HOW UNITY DOES IT

• Web-based publishing portal - “Saihai” • Promoting builds to internal QA and

publishing to external testers. • Publishing final releases to CDN • Provides API regarding published releases

for other internal services



The 5 pillars of Build Infrastructure.

ONE-CLICK BUILDS

BUILD AUTOMATION

CONTINUOUS INTEGRATION

AUTOMATED TESTING

AUTOMATED DEPLOYMENT


Text Styles



+ TAB

Alternatively,

A CALL TO ARMS: TIPS FOR GETTING STARTED

• Just do something • When advocating for change, focus on the

benefits • Don’t underestimate the value of having

dedicated build engineers on your team


Text Styles



+ TAB

Alternatively,

COME JOIN US!

• Release Managers • Web Developers (Full-Stack & Front-End

Only) • Low-level Build Engineers • High-level Build Engineers • Version Control Experts • Packaging & Distribution Engineers

http://unity3d.com/jobs


Text Styles



+ TAB

Alternatively,

Thank You!@natosha_bard