02 - build and deployment management
TRANSCRIPT
BUILD AND DEPLOYMENT MANAGEMENT
2
PURPOSE OF CURRENT TRAINING
Theory and practice of build and deployment management
Why do we need to manage builds
Why do we need to manage deployments
3
CONTENTS
Build managementWhat is build?Why do we need to build?Build types.Tools and their specifics: Ant, Maven, make,
etcTips
Deployment managementBuild vs deploymentWhy do we need to deploy?How to deploy and when?Tools that might be useful
4
MAIN CONCEPTS
WHAT IS BUILD
From ToProcessof conversion
Source code Artifactsrun at the end-user
workstation
5
6
WHY DO WE NEED TO BUILD?
Configuration
BuildRESULT OF
defined set of components having specific versions
components = functional unitscomponents and their versions are chosen to
meet specific objectives or tasks
Main objectives
Functionality Performance
7
WHY DO WE NEED TO BUILD?
Conclusion. Why we need to build is: To incorporate additional functionality (feature) To reach defined level of performance (as an
example of non-functional requirement)
And what about data? Data has its own lifecycle Which is incorporated into the database
integration process
8
WHY DO WE NEED TO MANAGE BUILDS? When application grows in size, it becomes more
complex
As a result, it turns out that routine daily activities require additional management
Organizing build process is one of such activities
Therefore, build process also requires management
What means complex application?
From configuration management point of view, most complex application incorporate all possible variant properties
9
BUILD VS VARIANT
AppName-1.1.8.en.x32_beta.msi
install.msiBuild:
Variant:
HTC_Touch_Diamond2_WM6.5_RUU_Topaz_S_HTC_RUS_2.16.411.2_Radio_Sign_Topaz_61.44tc.25.32_4.49.25.17_Ship.exe
Real world example:
10
CONNECTION BETWEEN CONFIGURATION AND VARIANTS
build
source code changes (changeset)
variant
configuration +
#1 #2 #3 #N…
Functional requirements for the variant being built
static runtime
version Maturity
Platform
OS
Localization
Build type
11
WHAT IS VARIANT
Variant is the result of the build characterized by the set of specific
properties and their values
12
VARIANT PROPERTIES
Version Revision Date Build type: debug or release Operating system, hardware platform (x32,
x64, …) Localization (en, ru, ua, …) Platform (development, test, integration,
production) Maturity (alpha, beta, release candidate, stable)
Issue number (fix#387)
License (MIT, GPL, BSD, CC, …)
Project ID (EPM-SIGN) + Discipline ID (PD)
13
EXAMPLE. PLAIN DOCUMENT
Ctrl + N Ctrl + S Ctrl + S Ctrl + S
Build #1Properties: • Language (en)• Date (22-12-2010)• Revision (3)
Build #2Properties: • Language (en)• Date (24-12-2010)• Revision (4)
14
DEMO EXAMPLE
15
EMERGED QUESTIONS
Why open source repositories are not structured properly?
How to assign unique marker (version) for our changes?
What repository model should we use? How do we need to manage experimental
changes or development? What is initial codebase?
16
BUILD MANAGEMENT
17
WHY DO WE NEED TO MANAGE BUILDS?
Builds should be CRISPy!
CRISP means:
Complete
Repeatable
Informative
Schedulable
Portable
18
HOW TO MANAGE BUILD PROCESS?
Define what build tasks you need
Define what build types you need
Choose build phases you will need
Choose build tool
Structure your build file
19
HOW TO MANAGE BUILD PROCESS?
Define what build tasks you need
Define what build types you need
Choose build phases you will need
Choose build tool
Structure your build file
20
BUILD TASKS BY PRIORITY
CompilationDeploymentDB integrationUnit testing Code coverage Static analysis Source code metrics Dynamic analysis Documentation generation …
21
LET’S TALK ABOUT DIFFERENCES BETWEEN
BUILDS AND DEPLOYMENTS
22
BUILD VS DEPLOYMENT
Deployment is the process of software installation (software system available to use)
While build is the process of installation artifacts creation
Is this always true? Not really For interpreted languages (no compilation phase)
there might be no significant difference between build and deployment
But it is more convenient to think of all related activities (unit-testing, code coverage, inspections, etc) as a build, not deployment.
23
RESULTS OF THE BUILD
Executables .exe, .bin, .msi, .cab, …
Libraries .dll, .lib, .so, …
Archives .zip, .tar.gz, .bz2, .7z, .ear, .jar, .war, .sar, …
Packages .rpm, .deb, …
24
RESULTS OF DEPLOYMENT
Installed application Updated database Added incremental changes Unpackaged library Loading and installing
dependencies …
25
BUILD VS DEPLOYMENT
Runs on single machine
Does not require remote connection
Creates application installation
Makes sense always
Primary
Might be executed on several machines
Requires connection to the remote server
Installs application
Makes sense only in case of distributed or remote application
Secondary
Build Deployment
28
HOW TO MANAGE BUILD PROCESS?
Define what build tasks you need
Define what build types you need
Choose build phases you will need
Choose build tool
Structure your build file
29
BUILD TYPES
CLASSIFICATION BY:
Use
• Private (developers build)• Integration (centralized build)• Release (delivery to the end-user)
30
BUILD TYPES
CLASSIFICATION BY:
Structure, stage, result
• Pre-build (ensuring zero build errors: checking space, clean, …)• Post-build (build verification tests, security tests, …)• Clean (rebuilding whole project from scratch)• Incremental (only changed files and artifacts)• Broken
31
BUILD TYPES
CLASSIFICATION BY:
Staging
• Lightweight (recompilation, unit-tests, …)• Heavyweight (deployment, system tests, inspections, …)
32
Define what build tasks you need
Define what build types you need
Choose build phases you will need
Choose build tool
Structure your build file
HOW TO MANAGE BUILD PROCESS?
33
BUILD PHASES
BuildCompilationDeployment Inspections
Pre-buildClean InitCheck disk space
Post-buildDocumentation generationChecking build errorsSecurity testsTagging
34
HOW TO MANAGE BUILD PROCESS?
Define what build tasks you need
Define what build types you need
Choose build phases you will need
Choose build tool
Structure your build file
35
BUILD TOOLS
GNU Autotools
• make• automake• autoconf• autoheader• libtool• gettext• gcc • pkg-config
Autotools alternatives
• pymake• CMake• Cons• SCons• qmake• makepp• JAM• waf
36
BUILD TOOLS
Java based
• Ant• Maven• Gant
.NET based
• MSBuild• Nant• Byldan• NMaven
Other
• Phing (PHP)• Rake (Ruby)
38
GNU AUTOTOOLS
Did you build applications from sources on UNIX platform?
What steps did you need to accomplish this?1. ./configure2. make3. make install
When does actual compilation happens? Right. During make command execution make is the father of all build tools Heart of the build with make is the makefile
39
GNU AUTOTOOLS MAKEFILE EXAMPLE
target1 target2 target3 : prerequisite1 prerequisite2command1command2command3
myprogram: main.o part1.o part2.o gcc -o myprogram main.o part1.o part2.o
part1.o: part1.c part1.h header.h gcc -O -c part1.c
part2.o: part2.c header.h gcc -O -c part2.c
main.o: main.c header.h gcc -O -c main.c
clean: rm -f myprogram main.o part1.o part2.o
40
GNU AUTOTOOLS LIST
autoconf creates a configuration script for a package from a
template file From Makefile.in to Makefile
automake Creates template file from another template file From Makefile.am to Makefile.in
libtool helps manage the creation of static and dynamic
libraries gettext
internationalization and localization library gcc (GNU compiler collection)
default compiler used for building from sources
41
APACHE ANT
Implemented in Java For the purpose of building java applications And for the purpose of make replacement
Advantages over make: Cross-platform, does not require buildfile
generation Resolves circular dependencies Works with file hierarchies Easily determines outdated files Conforms to “java way of thinking”
42
APACHE ANT BUILDFILE EXAMPLE
43
APACHE MAVEN
Convention over configuration Declarative execution Reuse of build logic Coherent organization of
dependencies
Principles:
Features: Project Object Model Extensive plugin architecture Predefined build lifecycle Dependencies management
44
APACHE MAVEN BUILDFILEEXAMPLE
45
MAVEN VS ANT
Description of project (convention over configuration)
Invocation of defined goals (targets)
Project knowledge build lifecycle, standard
project layout reusable plugins,
repositories
Development of a build script per project
Invocation of project specific targets
"Just" the build process too complex scripts
scripts are not reusable
Maven Ant
46
HOW TO MANAGE BUILD PROCESS?
Define what build tasks you need
Define what build types you need
Choose build phases you will need
Choose build tool
Structure your build file
47
BUILD FILE STRUCTURE
CompilationDeploymentDB integrationUnit testing Code coverage Static analysis Source code metrics Dynamic analysis Documentation generation …
compiledeploydeploy-dbrun-testscheck-coveragerun-inspectionsgather-metricsperformance-testsgeneratedoc …
Tasks Targets
48
BUILD FILE TEMPLATE
49
BUILD TYPES AND BUILDFILESTRUCTURE
Use types: private, integration, release
50
LOCAL BUILD VS PRIVATE BUILD?
Local build:CompilationUnit-testing
Private build:CompilationDeployment Database integrationLoading initial dataUnit-testingSimple static analysis
What is the difference?
CompilationDeployment Database integrationLoading initial dataUnit-testingSimple static
analysis
Omitting unnecessary steps:
51
LOCAL BUILD
CompilationDeployment Database integration
Loading initial dataUnit-testingSimple static analysis
Local build is required after minor source code changes for the purpose of having executable artefact
ant compile run-tests
52
PRIVATE BUILD
CompilationDeployment Database integration
Loading initial dataUnit-testingSimple static analysis
Private build assumes that changes should be deployed to the application container or web-server
ant compile deploy deploy-db load-data run-tests inspections
53
PRIVATE BUILD
54
INTEGRATION BUILDIntegration build assumes presence of continuous integration practice as a key part of software lifecycle
CompilationDeployment Database integration
Loading initial dataUnit-testingStatic analysisDynamic analysisDocumentation generationGathering metrics
55
INTEGRATION BUILD
56
RELEASE BUILDRelease build is the result of development will be visible to the end user
CompilationDeployment Database integration
Loading initial dataSecurity testsintegrity checksperformance tests
57
RELEASE BUILD
58
DEPLOYMENT MANAGEMENT
59
DEPLOYMENT TYPES
Staticwhen the container starts
HotAbility to deploy/undeploy deployables into a
running container Incremental
Deploying only changed files/artifacts Remote
Requires connection via specified protocol Local
Does not require connection
60
DEPLOYMENT PROTOCOLS
FTP/SFTP
SSH/SCP
Rsync No protocol (copy/move)
Mail/e-mail (!)
61
DEPLOYMENT PHASES AND TASKS
DeploymentDeployRestart application server
Database integrationGenerate ORM models, mappingDump data Instantiate DB (DDL usage)Data load into DB (DML usage, CRUD
operations)Migrate data, structure or both
62
INCREMENTAL DEPLOYMENT
Rsync synchronizes files and directories from one location
to another SVN export only changed files
svn diff –summarize /path1 /path2 Deploy only files modified/added since last
revision svn status
In most cases it’s just a…
HEADACHE!
63
SIMPLE DEPLOYMENT WORKFLOW
WC (working copy, trunk)
dbinstance
local deployment
64
LOCAL DEPLOYMENT WORKFLOW
WC (working copy, trunk)
dev db test db prod db
dev instance test instance prod instance
local deployment
65
REMOTE DEPLOYMENT WORKFLOW
65
WC (working copy, trunk)
local deployment
dev
+
test
+
prod
+
dev
+
test
+
prod
+
remote deployment
66
MANAGE YOUR DEPLOYMENTS
1. Define what deployment type you will use (inherits from corresponding build type)
2. Define what deployment tasks you need3. For each deployment type prioritize
chosen tasks4. Define destinations (production, test,
development)5. Map destinations and deployment types6. Configure deployment destinations
environment
67
DEVELOPMENT WORKFLOW EXAMPLE
68
DEVELOPMENT WORKFLOWEXAMPLE
WC (working copy)Deployed application Repository
svn up
ant deploy-private
Second deployed application
changeset (#146)
manual merge
r1049
r1050
r1051
r1052
…
r1126svn up
ant deploy-private –Dinstance=2 resolve conflicts
development
OK?no
yessvn commit
manual mergemake correctionssvn commit
69
DEVELOPMENT WORKFLOW EXAMPLE NOTES
Does this example remind you something? Yes! Distributed version control What is the point? Svn working copy is the repository While deployed application is a working
copy Difference is that all operations between
‘repository’ and ‘working copy’ are manual And operations history is not tracked
70
DEVELOPMENT WORKFLOW EXAMPLE NOTES
Why do we need two builds? To ensure there are no integration errors Because of “Don’t break the build“ ruleAdvantages: You see your source code many times during merge You can find errors/inconsistencies easily Keeping unfinished development both safe and
actualFlaws: A lot of time spent before commit Works only for web-projects written in interpreted
languages
71
CONCLUSION
72
BEST PRACTICES
Separate build from IDE Run faster tests first
Introduce staging into build process Fail build fast
Organize build targets by priority Build components separately
Recursive build Centralization of project assets
Put sources to sources (VCS) Put binaries to binaries (dedicated repositories)
Consistent directory structure Develop project structure conventions
73
BUILDS AND DEPLOYMENT MANAGEMENT1. Define what build/deployment tasks you
need2. Define what build types you need3. Choose build/deployment phases you will
need4. Prioritize chosen tasks with regard to
corresponding build types5. Choose build tool6. Structure your build file7. Define deployment destinations8. Map build types and deployment
destinations9. Configure deployment destinations
environment
74
RECOMMENDED READING
1. The Build Master: Microsoft's Software Configuration Management Best Practices By Vincent Maraia
75
RECOMMENDED READING
2. Ant: The Definitive Guide, 2nd Edition by Steve Holzne
76
RECOMMENDED READING
3. Maven: The Definitive Guide by Timothy M. O'Brien
77
USEFUL LINKS
1. http://freshmeat.net/articles/build-and-release-management - Build and release management
2. http://onjava.com/pub/a/onjava/2003/12/17/ant_bestpractices.html - Ant best practices
3. http://www.sonatype.com/books/mvnref-book/reference/public-book.html - Maven complete reference
4. http://repo.exist.com/dist/maestro/1.7.0/BetterBuildsWithMaven.pdf - Great book about Maven
5. http://kent.spillner.org/blog/work/2009/11/14/java-build-tools.html - Ant vs Maven
6. http://martinfowler.com/articles/rake.html - article about rake by Martin Fowler
7. http://www.finalbuilder.com/finalbuilder.aspx - FinalBuilder home page