jérémy nguyen xuan 27.02.2013 on behalf of be/co be-abp/ph-sft/be-co workshop

Build & Continuous Integration for C/C++ in Accelerator Controls

Jérémy Nguyen Xuan 27.02.2013on behalf of BE/CO

BE-ABP/PH-SFT/BE-CO workshop

2

Agenda

Introduction – Motivation and Goals

Maven NAR – Build system for C/C++

Bamboo – CI server, automatic builds

Deployment into Operations

Integration and System Testing

3

Overall goal

Minimize Operations downtime due to software defects

Provide High quality software Well tested Maintainable

Unified modern development process for CO CO Java development environment already mature Common software development process Good set of tools

4

SW development processPlanning

• Change management in JIRA

Development

• Eclipse IDE• Maven NAR and Makefiles to

build• SVN source repository• Bamboo CI server• Maven artifact repository

Testing

• Unit tests and Code Coverage• Testbed for System and

Integration tests

Release

• Maven NAR & Makefile• Publish to repository

Deployment

• From repository to operations

5

C/C++ Teams and Projects in CO 5 Development Teams (FESA, CMW, TIMING, Drivers, Diamon)

~30 developers at CERN +20 projects on SVN

Many clients At CERN Outside CERN (GSI)

Lines of Code > 250’000

Supported platforms LynxOS ,SLC5, SLC6, Windows

Java ~200 developers ~1000 projects on SVN

6

DEP

beecrypt

IceUtil omniORB

serializer util

rbac rda

directory-client

proxy

cern-framework

core-framework

tgm

tgv

CMW

FESA

TIMING

6

tim

omniThread

curl (ppc4)

boost(headers

ppc4)

libxml2

example-get

3rd Party

DRIVER

drvrutil

UnitTestAllTypes_prj

stomp

log-stomp

log

7

Agenda

Introduction – Motivation and Goals

Maven NAR – Build system for C/C++

Bamboo – CI server, automatic builds

Deployment into Operations

Integration and System Testing

8

Generic Makefile

For Java, we already have Guidelines / best practices Recommended set of tools Common build tool

C/C++ teams asked for the same recommendations

9

SIP4C++

Working group with various teams from BE/CO

Agreed on common standards and tools Directory structure and naming Google test/mock, Valgrind, Doxygen Common implementation of Makefiles Common set of compilation flags Manifest implementation Tracing message (who, when, where)

On-going work Quality assurance (Coverity) Post-crash analysis (Google BreakPad)

10

Project

NFS

• Project information

PROJECT = fesaclassPRODUCT = FesaTestEnvironmentLIB_NAME = FesaTestEnvironmentVERSION = 0.0.1

• Dependencies information

FESAFWK_VERSION = 0.0.1-alpha1

FESAFWK_HOME = (INSTALL_LOCATION)/$(CPU)/fesa/fesa-fwk/$(FESAFWK_VERSION)

DEPENDENT_COMPILER_OPTIONS += -I$(FESAFWK_HOME)/include

DEPENDENT_LINKER_OPTIONS += -L$(FESAFWK_HOME)/lib -lfesa-core -lfesa-core-cern -lcmw-rda -lxml2

• Compiler/Linker rules• Google test/mock support• Doxygen generation• Valgrind integration• Tag SVN• Install on NFS

• Installation location• SVN url• Specific compilation flags

• cross-compilers definition

MakefileMakefile.dep

Make.fesa Make.generic

Make.common

11

Motivation for Maven

C/C++ build tool Started ~1 year ago Evaluating Maven as CmmnBuild

successor Opportunity to unify the build tools Common software development process Built a prototype with Maven NAR

Reuse same philosophy as Java

12

Build tool Build management, making it easy Provide uniform build process Encouraging best practices

Dependency manager Versioning Resolve and download artifacts Deal with conflicts

Plugin based Has numerous plugins You can develop your own

What is ?

13

Maven - How does it work?

Source Code

Build Process

Artifact

14

Maven - Build lifecycle

Source Code

Artifact

generate-sources

process-sources

process-resourcescompile

test-compile

test

package

integration-test

verify

install

validate

15

Maven - Convention over configuration

<project><groupId>cern.testbed</groupId><artifactId>testbed</artifactId><version>1.0.0</version>

<description>The TestBed focuses on testing Core Software Components.</description><url>http://wikis/display/ABCO/Control+System+Test+Bed+Facility</url><dependencies>

<dependency> <groupId>cern.japc</groupId> <artifactId>japc-ext-cmwrda</artifactId> <version>[2.0.0,3.0.0)</version></dependency><dependency> <groupId>cern.japc</groupId> <artifactId>japc</artifactId> <version>2.11.0</version></dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.8.2</version> <scope>test</scope></dependency>

</dependencies></project>

16

Maven - Convention over configuration

<project><groupId>cern.testbed</groupId><artifactId>testbed</artifactId><version>1.0.0</version>

<description>The TestBed focuses on testing Core Software Components.</description><url>http://wikis/display/ABCO/Control+System+Test+Bed+Facility</url><dependencies>

<dependency> <groupId>cern.japc</groupId> <artifactId>japc-ext-cmwrda</artifactId> <version>[2.0.0,3.0.0)</version></dependency><dependency> <groupId>cern.japc</groupId> <artifactId>japc</artifactId> <version>2.11.0</version></dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.8.2</version> <scope>test</scope></dependency>

</dependencies></project>

17

Maven - Convention over configuration

<project><groupId>cern.testbed</groupId><artifactId>testbed</artifactId><version>1.0.0</version>

<description>The TestBed focuses on testing Core Software Components.</description><url>http://wikis/display/ABCO/Control+System+Test+Bed+Facility</url><dependencies>

<dependency> <groupId>cern.japc</groupId> <artifactId>japc-ext-cmwrda</artifactId> <version>[2.0.0,3.0.0)</version></dependency><dependency> <groupId>cern.japc</groupId> <artifactId>japc</artifactId> <version>2.11.0</version></dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.8.2</version> <scope>test</scope></dependency>

</dependencies></project>

18

Maven - Repositories

Local repository Nexus

artifacts.cern.chMaven Centralrepo.maven.org

CERN Network

query

query

download

download

Internet

19

Maven NAR - Basic idea

Maven NAR Maven plugin Compiles native code for various architectures

and linkers Made some extensions

The idea is to separate the build tasks between Makefiles and Maven Compilation handled by Makefiles Dependencies & Versioning by Maven

20

Maven NAR - Example

<project><groupId>cern.cmw.cpp</groupId><artifactId>cmw-rbac</artifactId><packaging>nar</packaging><version>3.6.2</version><dependencies>

<dependency><groupId>cern.cmw.cpp</groupId><artifactId>cmw-serializer</

artifactId><version>1.3.0</version><type>nar</type>

</dependency><dependency>

<groupId>cern.cmw.cpp</groupId><artifactId>cmw-util</artifactId><version>1.2.1</version><type>nar</type>

</dependency></dependencies>

</project>

Pom.xml

mvn nar:makedep

21

Maven NAR - Example

R = /nfs/cs-ccr-nfsdev/vol1/u1/jnguyenx/demo_phsft/cmw-rbac-cpp/target/nar/

DEPENDENT_COMPILER_OPTIONS += -I$(R)cmw-serializer-1.3.0-noarch/include -I$(R)cmw-util-1.2.1-noarch/include

DEPENDENT_LINKER_OPTIONS += -L$(R)cmw-serializer-1.3.0-i386-SLC5-gpp-static/lib/i386-SLC5-gpp/static -L$(R)cmw-util-1.2.1-i386-SLC5-gpp-static/lib/i386-SLC5-gpp/static -lcmw-serializer -lcmw-util

Makefile.dep.L865

22

Workflow

Source code

Makefiles

Resolve dependencies

Maven NAR

Maven NAR

BinariesNAR

package

Maven NAR

Maven NAR

Dependencies

Compilation phase

Packaging phase

Installation phase

Dependency resolution phase

Binary repository

23

Makefiles & Maven - Benefits Dependencies and Versioning are automatically managed Developers keep their habits with Makefiles Enforced standards

directory structure directory names file names

Unification development/release/deployment process Google test/mock Compilation flags

Traceability Manifest versioning

Sharing projects for organizations outside CERN becomes easy

24

Agenda

Introduction – Motivation and Goals

Maven NAR – Build system for C/C++

Bamboo – CI server, automatic builds

Deployment into Operations

Integration and System Testing

25

Bamboo - CI Server

We use Atlassian tools: JIRA, Confluence, Crowd, Crucible, Clover and Bamboo

Good reports and notifications

Cascade builds

Several build agents are configured to build the code on different platforms

26

1.

27

1.

2.

28

29

Agenda

Introduction – Motivation and Goals

Maven NAR – Build system for C/C++

Bamboo – CI server, automatic builds

Deployment into Operations

Integration and System Testing

30

Deployment into Operations Deploy a binary on a target machine

Deployment tool - inhouse Common for all CO projects (Java & C/C++) Instance descriptor in XML Facilitates configuration & logging Reproducibility Backups and rollback Installation notifications

31

Agenda

Introduction – Motivation and Goals

Maven NAR – Build system for C/C++

Bamboo – CI server, automatic builds

Deployment into Operations

Integration and System Testing

32

Integration and System Testing Approach

To fulfill our goals, a mini-accelerator lab was built with the following aspects:▪ Completely separated from the operational

environment▪ Validate products before going in operations

Note: Staging development

Development

Testing

Production

Release Candidate Release

33

Hardware

First step to build our TestBed: Set up an environment where to install BE/CO software.

3 layers to simulate: Client side Middletier services Back-end

34

CO Testbed Hardware in placeTIMING

FEC03

FEC01 FEC02

FEC04FEC05

SERVER06

SERVER07

Niall TC may 2009

35

Software

Selected only essential components for Controls to:

Limit the complexity of the system

Give more flexibility to tweak components

Detect defects more efficiently

CO TestBed architecture

37

Deployment

The products were manually deployed Time consuming

Automatic deployment into the TestBed Speed up development cycle Faster feedback to the developers

38

Our idea

Use SVN Branches to identify release candidates

Automatically build a new release candidate and its dependees

Store the binaries in a separate repository from the release one

Somehow deploy these binaries into the TestBed

39

1. SVN Branch

2. Bamboo - CI server

3. Nexus – Binary repository

4. Deployment

40

Workflow

41

Type of tests

In our TestBed, we exercise

Functional testing System integration testing

Backward compatibility testing

42

Conclusions

Several years of experience in building Java SW

Developing common solution for Java and C/C++

Standardize as much as possible

Functional testing with the TestBed

43

Nexus - Release Management

Nexus stores several versions of a product development release candidates final release

Maven uses GAV (groupId, artifactId, version) to identify an artifact

Features Search Full integration with Eclipse IDE RSS feeds Etc…

44

45

46

47

48