CMTChristian Arnault arnault@lal.in2p3.fr

Structure the projects

Structure the packages

Define the development work models

Define recurrent patterns in configuration management

Identify the elements of the configuration

Query the knowledge base to parameterize the development tools

Identify and describe the activities of the software production

Structuring the projects

CMTPATH=/…/Project1 : /…/Project2 : /…/Project3

Project1 Project2 Project3

•Projects are located in dedicated disk areas

•Each project has an entry in the CMTPATH search list•Follows the standard platform dependent syntax

•‘:’ on Unix and ‘;’ on Windows

•The search list is prioritized like usual PATH search lists•Left (bottom) entries are considered first•Right (top) entries are considered last•Packages of a project may use packages of upstream projects

•Packages belong to projects•they are always installed in one of the CMTPATH entries•This is where packages are looked for

•Each project may define its own strategies•In /…/Project/cmt/project.cmt•Valid for all packages in this project•Valid for all bottom packages if not overridden

Structuring the packages•A package belongs to a project

•It is located in the corresponding CMTPATH entry•May be located at the top position or below any directory tree

•A package has a unique name•must be unique in the complete software base•Unique amongst the different projects

•A package has versions•Implicitly specified in the directory structure•Or explicitly in the version.cmt text file

•A package is recognized as a CMT package by•a cmt directory below its directory structure•a requirements file in the cmt directory

/.../Project/cmt/project.cmt

/A/cmt/requirements /cmt/version.cmt

/B/v1/cmt/requirements

/C/D/E/cmt/requirements /version.cmt

The project file

Package A with explicit

version definition

Package B with implicit

version definition

Package E In a sub-directory

Define the development work models

•Project management•Split the complete software base into natural sub-projects

•Based on the release cycle constraints•Typical:

•Core software•Offline/online•Simulation/reconstruction/analysis/…

•Reflect the multi-project organization•Eg. Atlas uses Gaudi which uses LCGCMT•Decomposition based on the responsibilities

•Each individual project have•A specific release cycle

•Every n-week•Random

•A dedicated CMTPATH entry•An interface package to its used sub-projects

•Defines the upstream CMTPATH items•Selects the release id of all used sub-projects

•Development/Integration work models•The work area becomes the bottom project

•i.e. it is the top left entry in $CMTPATH•Used packages override those from the selected production release•Structure the software base with levels of integration areas

Define recurrent patterns in configuration management

•Build strategy•E.g. Define whether the result of the build will go to a global installation area

•Setup strategy•E.g. Define what kind of automatic environment variables will be generated by CMT

•Compiling, linking•E.g. Construct DLL on different platforms

•Generating code, documentation•E.g. Building rootcint generator•Complec Doxygen generation with tag-file management

•Include file search style•E.g. Specifying the project-wide conventions for include search strategy

•Test patterns•Driving CppUnit•Enforcing the Unit tests •Managing the integration tests

•QA activities

Identify the elements of the configuration•Projects

•Project names are defined in the project files<project>/cmt/project.cmt

•Packages•The name is the primary identifier of packages•Must be unique

•Constituents•Libraries•Applications•Generated documents

•Activities•Code generation•Build

•One make target per constituent•Actions

•Generic activity•Activities can be grouped by type (testing, documentation, installation, deployment…)

•Contextual parameters values according to•Sites•Platforms (hardware, OS, …)•Step in the Life cycle•User defined conditions

•Specified in symbols•Macros, sets, paths, actions

•Variants with tags or tag mixtures

Identify and describe the activities of the software production

•Versioning activities•Generally handled through tools like CVS or SVN•CMT provides a plugin to CVS to perform queries about CMT structures

•helper command to perform conformant checkout operations: cmt co•SVN is more flexible and can be taught to recognize CMT packages

•Through SVN properties•Build activities

•Standard compiling and linking is automatically handled from the definitions of constituents•Specialized sequences of operations defined in CMT patterns

•Testing, QA assessment•Driven by external tools like JUnit, CppUnit, Qmtest, OVAL, … CodeWizard, RuleChecker•Enforced by automatic applications through CMT patterns

•Deployment•CMT query mechanisms to the knowledge base used to generate the manifest files of different packaging systems: Pacman, RPM, Fink•Specify adaptation scheme to configuration parameter values when installed on remote sites

Query the knowledge base to parameterize the development tools

•The cmt command•A C++ application•Available on any platform (Unixes, MacOSX, native Windows)

•All configuration parameters can be reached by query commands•cmt show macros•cmt show uses•cmt show macro xxx•…

•The cmt broadcast command•Uniquely sort the dependency graph (leaves first, root last)

•Make it an ordered list of packages reached from the current package•Traverse the ordered list•Apply an action to every node

•A normal shell command•May stop at first error or continue•May select a subset of the graph•May exclude a subset of the graph

•The cmt filter command transforms any text files•expand all $(mmm) and ${mmm} patterns in the file using symbol values

•The common options•-quiet•-tag_add=t1,t2•-public•-private

> cmt broadcast gmake> cmt broadcast –select=/Graphics/ gmake test

project P1

package Buse A

package Cuse B

package Duse B

package Euse B

package Guse Euse F

project P2

package A

project P3

package F

Integration area

package Cuse B

package Duse B

development area

package Guse Euse F

http://www.cmtsite.org/

CMTChristian Arnault arnault@lal.in2p3.fr

The dependency graph

•Describe the primary relationship between packages•Three identification elements

•The name•The version request•The directory offset in the project

•The dependences form a Direct Acyclic GraphClients packages use used packages

•Packages may be reached by several paths•CMT supports cycles in the graph

•Cycles introduce non deterministic traversal paths

•A package exports its public features to its clients•All requirements files of all (visibly) used packages are concatenated to form the complete configuration•Constituents are always private•Automatic (global) patterns are applied to all visible packages

•Use statements installed in private sections define a hidden region of the dependency graph when observed from the client perspective

Dependency graph

Configuration tags

Constituents

Symbolic parameters

Configuration patterns

Package semantics

Actions

Scoping sections

Policy packages:•Provide policy statements for a software domain

•include dir strategy•CMT strategies•CMT patterns•Generic symbols (macros, sets, …)

•Examples:•AtlasPolicy is the main policy package, mandatory for all Atlas packages.•ExternalPolicy, AtlasCxxPolicy, LCG_Policy

macro m1 “a public value”private macro m2 “a private value” use my_packageend_private# back to public section

Scoping sections

•A private section is not exported to the clients•A public specification found in the graph overrides a private region of the dependency graph•The cmt broadcast and cmt show uses reach the private sections by default

•The –public common option always hides the private regions•The –private common option always reaches the private regions

•Scoping sections end:•At the end of the requirements file•With the end_private or end_public statements

•Ending statements return to the previous scope

Container and release packages:•Purely managerial packages

•Not involved when using the software•Only provide a CMT requirements file with a set of explicit use statements •No wild cards

•Purpose:•Specifies the set of exact version for a release of a software domain•Drive the cmt broadcast commands over the complete software domain

•One top container can provide a release package <project>Release.•It describes the complete software release of the project•Every new release of the project implies a new release of the <project>Release container.

Interface packages:•Provide an interface between a software project managed by CMT and a so-called external product (i.e. not managed by CMT)•Specify the native version value defined by the product (as opposed to the CMT version tag)•Adapt the policies and conventions to the project’s ones

•Eg Include directory•Construct the compiler and linker options for this product•Set the runtime options (PATH, LD_LIBRARY_PATH, etc…)•Specify the material to export when constructing a distribution kit•Examples

•MySQL•ROOT•SEAL•Geant4•Java kits

•The LCGCMT project is entirely made of such interface packages•Connect to LCG internal (SEAL, POOL, PI, …) or external products

package XercesCuse LCG_Settings v*macro XercesC_native_version "2.3.0"macro XercesC_home "$(LCG_external)/XercesC/$(XercesC_native_version)/..."include_dirs $(XercesC_home)/includemacro XercesC_linkopts "-L$(XercesC_home)/lib -lxerces-c -lpthread" \ WIN32 "/LIBPATH:$(XercesC_home)/lib xerces-c_2.lib"path_prepend LD_LIBRARY_PATH "$(XercesC_home)/lib" WIN32 ""path_prepend PATH "" WIN32 "$(XercesC_home)/bin"macro XercesC_export_paths " $(XercesC_home)/include $(XercesC_home)/lib"

The symbolic parameters•Generically define a configuration parameter

•May receive different values under different conditions•According to tag expressions

•Is exported to client packages (if installed in a public section)•Can be modified/overridden by client packages

•Modifications installed in a private section are not propagated to clients.

•Several possible semantics share the same syntax•macro generic internal CMT variable

and make’s macro definition•set environment variable definition•alias alias definition of the shell•path PATH-like environment variable•action shell command

•And associated modifiers•<kind>_append•<kind>_prepend•<kind>_remove•<kind>_remove_all•<kind>_remove_regexp•<kind>_remove_all_regexp

macro m1 “default value” \ Unix “on any Unix” \ rh73 “only on this machine” \ Win32&debug “for Windows and debug”

macro m2 “je réutilise $(m1)” \ Linux “but on any Linux”

path_remove PATH “/A/bin”path_prepend PATH “” \ CERN “/afs/cern.ch/sw/lcg/A/bin” \ LAL “/lal/A/bin”

action directory “ls $(dir_options)” \ Win32 “dir $(dir_options)”

cmt do directory

The configuration tags•Describe the contextual and operational conditions of the software configuration

•The current sub-project•The platform and operating system•The site•The compiler•The step in the life cycle•The user defined conditions

•Examples:•Debug or optimized mode•Selection of a graphical environment•Running some special firmware

•The version tags of CMT itself•Various production modes:

•Some are automatically constructed by CMT•By discovery of the context (OS, compiler version)•To reflect the project strategies•During the activities

•While building a constituent•While running a user defined activity (through actions)

•Some are externally specified by the user through environment variables

•CMTSITE: the current site name•CMTCONFIG: the binary tag name

•Tags can be specified on the cmt command line•Using the –tag_add=<tag-list> common option

•Tags can be explicitly activated in requirements file•Using the apply_tag <tag> statement

•Some are deduced from other tags•Using the tag association statement

•Active tags are visualized using cmt show tagstag i686-rh73-gcc32 Linux redhat73 gcc-3.2 opttag i686-rh73-gcc32-opt Linux redhat73 gcc-3.2 opttag i686-rh73-gcc32-dbg Linux redhat73 gcc-3.2 debugtag i686-rh73-gcc32-pro Linux redhat73 gcc-3.2 opt profiled

tag redhat73&gcc-3.2 redhat73-gcc32

tag OSF1 Unixtag Linux Unixtag HP-UX Unixtag Darwin Unix

pattern ld_library_path \ path_remove LD_LIBRARY_PATH "" Unix "/<package>/" ; \ path_append LD_LIBRARY_PATH "" Unix "${<PACKAGE>ROOT}/${CMTCONFIG}" ; \ path_remove PATH "" Win32 "/<package>/" ; \ path_append PATH "" Win32 "${<PACKAGE>ROOT}/${CMTCONFIG}"

Configuration patterns

•Describe recurrent configuration operations or policies in a project or a software domain

•Enforce project policies•Automate and parameterize complex operations•Extend the CMT concepts

•Pattern application can be:•Explicit (manual) using the apply_pattern keyword

•Application is on a package basis•Automatic when patterns have the –global keyword

•Applied to all packages that see the pattern in the dependency graph

•Definition of a pattern•A sequence of templated cmt statements

•Templates:•Automatic: <package> <PACKAGE> <version>

<path> <project>•User defined: <xxx>

•Examples:•Defining the project-wide policy for include search path

•Defining automatic shared library access path

•Enforcing a systematic QA test application

pattern -global include_path include_path ${<package>_root}/<package>/

pattern quality_test \ application <package>test –s=../test <package>test.cxx <other_sources>

package Pause ProjectPolicyquality_test other_sources=*.cxx

In ProjectPolicy

In package Pa

The constituents•Describe constituents to be constructed from sources

•Applications executable•Libraries static, shared, dynamically loadable•Documents anything else

•Applications and libraries obey a predefined strategy•Sources are always explicitly specified•Default source location is ../src

•But can be explicitly specified•Absolute location•Or relatively to ../src/

•Sources are transformed by a language compiler•Support for new languages is described by users

•The language statement•Automatic detection of languages from file suffixes

•Documents construction method can be freely designed•Make fragments

package doc_provider

make_fragment doc_headermake_fragment doc_trailermake_fragment doc \ –header=doc_header \ –trailer=doc_trailer

../cmt/fragments/ /doc_header /doc /doc_trailer

Provider package use

doc_header

doc (a.txt)

doc (b.txt)

doc (c.txt)

doc_trailer

package doc_client

document doc mydoc a.txt b.txt c.txt

Client package

mydoc.make