2009 - the waf book (v1.5.4)

8/8/2019 2009 - The Waf Book (v1.5.4)

1/43

The Waf Book (v1.5.4)Thomas Nagy

Copyright 2008-2009 Thomas Nagy

Copies of this book m ay be redistributed, v erbatim, and for non-com mercial purposes. The

license for this boo k is by-nc-n d license

Table of Contents

Preface

About build systems in generalThe Waf approach

Objectives of this book

1. Getting Started

Obtaining W af

InstallingRunning Waf

2. Introduction to t he Waf scripting system

Empty project exampleWorkflow illustration on a more complicated exampleBuilding a project - Introduction to TasksConfiguring a pro jectAdding c ustom command-line optionsInstalling targets

3. The c onfiguration system

Overview of the configuration filesThe co nfiguration environmentsConfiguration helper ex ecutionAdding new configuration helpers

4. Packaging and redistributing the source code

Quick tarb all creationCustom archivesCleaning the project t ree

5. The Task system

Task creation and executionTask execution

Task execution in parallelTask execution orderExec uting tasks only when som ething changes

6. Task encapsulation using task generators

Ex plicit rules for simple transfo rmations (Make-like)Implicit processing based on file name or extension

Custom p rocessing by task generator methodsTask generator method execution

7. The scheduler for ex ecuting the tasks

The task execution modelJob controlWeak task order con straints

8. C and C++ projects

Com mon sc ript for C /C++ applicationsLibrary interaction (uselib)

Customizing object filesUsing configuration headersThe include system

Configuration helpersPkg-config

9. Advanced scenarios

Simple file tr ansformat ionsSame targets, different configurations (variants)Building th e co mpiler firstWriting the output of a program into a header

A compiler producing source files with names unknown in advanceA task without any file dependency

10. C ustomizing th e scripting sy stem

Providing custom commandsChaining com mands

Providing a custom command contextCreating aliases / Injecting new commands

10/10/2010 he Waf Book (v1.5.4)

freehackers.org/~tnagy//single.html 1/43

8/8/2019 2009 - The Waf Book (v1.5.4)

2/43

11. Using the dev elopment version

Tracing the executionBenchmarking

ProfilingTracing parallel task executionObtaining the latest source c ode

Programming co nstraints

12. Ov erview of the Waf architecture

The core library

Build contex t instances

Overv iew of the Waf execu tionPerformance and b uild accuracy

13. C onclusionGlossary

List of T ables

11.1. Debugging zones12.1. The core library

12.2. Build contex t p ersistence12.3. Build context access13.1. Recomm ended links

Preface

Table of Contents

About build systems in generalThe Waf approachObjectives of this book

About build systems in general

As software is becoming increasingly complex, the process of creating software is becomingmore c omplex t oo. To day's so ftware uses v arious languages, requires various com pilers, andthe input data is spread into many files.

Software is now used to express the process of building software, it can take the form ofsimple scripts (shell scripts, makefiles), of co mpilers (CMake, Qmake), or c omp lete

applications (SCons, Waf). The term "build system" is used to design the tools used to buildapplications.

The Waf approach

Build systems make assumptions on software it is trying to build, and are typically limited

where it comes to processing other languages or different projects. For example, Ant isbetter suited than Make for managing Java projects, but is more limited than Make formanaging simple c projects.

There are so many different languages that creating a build system to support all thepossible kind of projects out-of-the-box is likely to be difficult. It looks much more

interesting to create one which can be extended easily.

The Waf approach is to provide a minimalistic and very flexible framework, which is packagedit with a significant amount of default build rules to be usable for most projects, and manyrecipes for the most complicated scenarios.

Objectives of this book

The objectives of this book is to exp ose the use of the W af build system though the use ofWaf in practice, the description o f the W af extension system, and detailed review of the W afinternals. Those who are impatient to see Waf in action should open the folder demos/ inthe Waf distribution, and start looking at the examples.

The first chapters contain a detailed description of the basic use of Waf. The chapters on

Configuration, Tasks, and Task generators describe the internals. These chapters should beread carefully before trying to ex tend W af. The next chapters co ntain examples for thelanguages such as C++, and for special scenarios which may occur in the most complicatedbuilds. The last c hapters are provided as a reference for those who may need to modify W afitself.

Chapter 1. Getting Started

Table of Contents

Obtaining W afInstalling

Running Waf

Obtaining Waf

The Waf project is located on Google Code The current W af version requires Py thon 2.3compiled with support for the bzip2 compression algorithm. Installing Waf is optional, if youwant to install Waf on the system, jump t o the nex t section.

Using the Waf binary

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

3/43

The Waf binary is ready to use and can be ex ecuted directly. Just rename it into "waf":

Generating Waf from source

The Waf binary is usually generated from the source distribution:

Installing

Waf installation is optional and there are many advantages of not installing Waf, amongthem:

1. Installation: installation is cumberso me, and req uires administrative privileges

2. W af versions: users rarely install the approp riate software version (too old, too new)3. Bugs: newer Waf versions may result in broken builds4. Size: W af is ext remely small (about 70 kB)

5. Operating systems: W af cannot be installed on W indows (yet)

If, despite the advantages enumerated previously, you really want to install the Wafapplication on the system, y ou m ay use th e following command on Unix-like systems such asLinux:

Root privileges may be required depending on the final location.

Waf can be uninstalled using:

Running Waf

The W af script can be used directly:

Or, if the executable permissions are not set properly:

By default, the Waf script uncompresses its library in the current directory, in a hiddenfolder. This means the W af script must reside in a writable folder. The n aming is allow

different Waf v ersions to run from the same folder:

Chapter 2. Introduction to the Waf scripting system

Table of Contents

Empty project exampleWorkflow illustration on a more complicated example

Building a project - Introduction t o TasksConfiguring a p rojectAdding custom comm and-line options

Installing targets

Empty project example

The Waf scripts are based on the following concepts:

System files: files and folders which do not belong to the projectSource directory: directory containing the source files for an application, it is meant to

be ultimately packaged, and redistributed to other dev elopers or to end users.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

4/43

Build directory: all files produced by Waf will be output in that folder

The Waf philosophy is to avoid the pollution of the source directory by letting all files intothe build directory. The build directory can be located on t he system, out of the source

directory (like in /tmp for example).

W hen W af is launc hed, it looks for the u ser-defined Waf scr ipts which are files written in thePython language. The most important one is the top-level Waf script file in which severalfunctions and attributes must be prov ided for defining a valid W af project:

srcdir: string representing the source d irectoryblddir: string representing the build directoryset_options: function used for adding custo m c ommand-line optionsconfigure: function called for configuring the projectbuild: function used for building the project

The top-level Waf script file name is "wscript"

A simple empty Waf project can be declared using a wscript file containing the followingPython code:

The minimum workflow for any project consists of the following steps:

Configuring th e project: searching for system parameters and com pilersBuilding the project: building the software

To do this, the following sh ell com mand s will be:

Workflow illustration on a more complicated example

Additional funct ions are usually prov ided in th e top -level wscript file:

APPNAME: project name

VERSION: project versioninit: function called before anything is executedshutdown: function called after a successful build

dist: function called for packaging the application

We will now provide a more complicated wscript file, execute the typical commands, and

look at the output:

The output will be:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

5/43

3 4

A few files will be pro duc ed by W af, let us look at t hem no w:

A build directory was created with the name given in the wscript file. It contains aconfiguration log, and various cache files. The last command produced a tarball of theproject (an archive) containing the source files. For now it only contains the wscript file.

The execution can be summed up in the following diagram:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

6/43

Building a project - Introduction to Tasks

The function build indicates the start of the actual construction of the software, it is usedfor two m ain purpo ses:

Declaring the location of scripts that provide other build functionsDeclaring the actual targets to build

Subdirectory recursion

The function build may reference build functions in wscript present in sub folders, for

example:

The top-level wscript may indicate there is a build function to execute in src/wscript using:

The wscript files that are not on the top-level often contain a large build function, with noadditional configuration or command-line options. In this case the file only contain one

method and the indentation is an annoy ance. For this reason, the wscript_build files

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

7/43

3 4

3 4

contain the body of a build function defined in a wscript file. A folder may only contain eithera wscript or a wscript_build but not bo th.

Waf task declaration

The process of building a piece of software require the transformation of source code (input)

into the result (output). Since the source code is usually split in various files, theconstruction of the result requires the processing of each of the files. Several new conceptsderive from this:

Intermediate results (object files) may be produced to speed up the build (no need to

rebuild everything when small changes occur)Source files may be processed in parallel to speed up the build

Source files may be transformed into other source files (sequences of transformations)

In a m ore general scope, the input necessary for a build may come from arbitrary datainstead of fi les (database, comm ands to execute). The c oncepts o finput, output andtransformationand the transformation constraints(in parallel, in sequence) remainidentical.

In Waf, the operation of transforming data is performed by tasks, which are instances ofthe c lass TaskBase (in the W af library ). The tasks in th e following ex ample illustrate thedeclaration of Tasks from the base class. The task instances must be located in a buildcontex t (the function build) to b e taken into ac count in the bu ild phase, and to o btain thereference on the source and build folders:

The ex ecut ion trace will be the fo llowing:

Task generator encapsulation

In practice, when lots of tasks are declared one by one, scripts tend to become c omplicated,and con tain a lot o f redundant co de. Special objects named task generatorsare prov idedto facilitate the creation of several tasks at once. In the following example, a task generator

is used to create the compilation and link tasks needed by a c++ project.

This example provides a modified configurefunction, which is the to pic of the nex t section.The task generator system and the default generators will be reviewed in detail in Chapter 6,Task encapsulation using task generators.

Configuring a project

The function configureis used for four main purposes:

Indicating new scripts that provide configuration functions to callProv iding co nfiguration contex ts to hold persistent data

Providing configuration helpers

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

8/43

3 4

Loading various Waf to ols

Indicating configuration sub folders

The function configure may reference configure functions in wscript present in sub folders,for ex ample:

The top-level wscript may indicate there is a configure function to execute in src/wscriptusing:

As a matter of fact, any wscript file may indicate a sub-configuration function to execute.

Storing and lo ading configuration parameters

The ex ecut ion will result in:

This configuration modifications (on conf.env) are persistent, they are reused during the

build (using bld.env). Let us look at the files produced:

The contents of the file default.cache.py is reproduced here:

Using configuration helpers

Setting up many parameters manually is a tedious proc ess, and W af provide configurationhelpers to automate the process of finding the settings. For example:

W ill produc e the following results:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

9/43

3 4

3 4

Loading Waf tools

The W af tools are extensions that prov ide task types (for languages such as c, java, ...),more co nfiguration t ests, and comm and-line options. To make certain that the W af tools

used during the configuration are also available during the build, they are stored during theconfiguration and loaded automatically afterwards. Here is an example:

By default, too ls are loaded from the W af library, but it is possible to load too ls from th esource directory using the following:

Adding custom command-line options

The function set_options is used for three main purposes:

Declaring c ustom comm and-line optionsIndicating that scripts provide command-line optionsAdding c ommand-line opt ions from W af tools

Adding options directly

Waf uses the optp arse module from P ytho n for adding comm and-line options, for example:

The command-line options can be displayed afterwards:

and us ed like this:

Adding options from wscript files located in sub folders

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

10/43

The function set_options may reference build functions in wscript present in sub folders, for

example:

The top-level wscript may indicate there is a build function to execute in src/wscript using:

Adding options from Waf tools

Waf too l options are not added by d efault because the exc ess of command-line options isannoying. Here is an example for adding the command-line options defined from a Waf tools

named compiler_cx x

Installing targets

Targets are installed when W af is run with the following c omm and-line waf install. If thetargets are not built, they will be installed as soo n as they are built. We will now d escribe thethree main ways of installing the targets.

Installing files directly

Source files may be installed directly by using the install methods of the build context bld:

The variables in curly brackets are substituted by values found in bld.env, for example

${PREFIX}is substituted by the v alue ofbld.env['PREFIX']. An environment parametermay b e provided if necessary:

Though the substitution is only performed on the first parameter, it is possible to re-use themethod bld.get_install_path or the function Utils.subst_vars

Installing targets (most task generators)

By pro viding the attribute install_path, tasks g enerator s will install the target (in thisexample, a program) in the given path.

The variables in curly brackets are substituted as described in the previous paragraph, usingmain.envinstead of bld.env. The chmodattribute is optional, and is used to change thepermissions of the installed file (by default it is set to 064 4).

To prevent the installation of a target (by default, programs and shared libraries are

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

11/43

installed), set install_path to None

Custom tasks

Task installation is disabled by default for most task types. If a method named installisprovided, it will be executed immediately after the task is successfully executed. Unlike thecompilation, the file installation is not run in parallel (there is little gain in copying files in

parallel).

Chapter 3. The configuration system

Table of Contents

Overview of the configuration filesThe co nfiguration environments

Configuration helper ex ecutionAdding new configuration helpers

Overview of the configuration files

Configuring a project consists in finding the parameters that will be used during the build

step

Several configuration files are produced and the configuration parameters are stored in thefolder c4chelocated under the build directory

Contains a reference to the build directory and contains the command-lineconfiguration fl ags. It is located at the root of the source directory.

Contains serialized data corresponding to the build information (cache,filesystem representation, etc)

Contains the Waf version and the list of Waf modu les to open automatically

Represents a configur ation environment (parameters such as configurationflags used during the build)

Contains the configuratio n test execution details

The file default.cache.py is called a configuration environment, and it may be edited byhand. W e will give the det ails on th e following section. The o ther files are not meant to bemodified by hand, even if they are editable.

The configuration environments

The build phase uses configuration environments created during the configuration phase.

The con figuration environments are Python Dictionarieswhich map keys to values. Theconfiguration environment files created take the following form:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

12/43

3 4

By default, the conf object comes with a default built-in configuration environment:

To replace the default configuration environment, a new one must be created and assigned:

Duplicates the default configuration environment (shallow copy)

Binds the configuration environment to the name 'debug'

Replaces the default configuration environment by the 'debug' one

Changes the CXXFLAGS entry in the 'debug' configuration environment. Sincewe are using a shallow cop y, we have to use the methods append_value,

prepend_valueor append_uniqueinstead of '+=' or '[].append'

In the prev ious exam ple, the file con tents will be:

In the build section, the values are retrieved using bld.env_of_name, for example

Configuration helper execution

Configuration helpers are methods prov ided by the conf object to help finding parameters,for example the method conf.find_program

Wh en a test fails, the ex ception Configure.ConfigurationExceptionis raised.

When calling the configuration from a tool, for example:

several different configuration helpers may be called: the first one for finding the compilerand the others for finding compiler flags. In some cases, it is interesting to disable some tests

(unnecessary, time consuming, etc), to insert new tests, or to modify the ex isting tests. Thecheck_toolmethod has a parameter for passing the list of c hecks to perform:

If no check is performed, the configuration helpers are still attached to the conf object, andmay be used later:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

13/43

3 4

To ease the creation of projects split into modules, conf.check_tool will not load the toolstwice for the same environment and t he same parameters.

An error h andler attached to the con f object is used for catc hing the Con figurationexceptions and processing the errors. Here is how to replace the default configuration error

handler by a custom method which may modify the list of tests, stop the ev aluation, or re-raise the exception:

The following diagram illustrates the test execution loop performed from conf.check_tool

Adding new configuration helpers

The con figurat ion tests specific for the different languages belong to th e different W af toolsrepresenting the languages supported. For example, The configuration tests for the Clanguage are located in the tool config_c.py. To enable this, decorators are used to attach

new configuration tests dynamically, for example:

Then the new configuration test may be used easily in wscript:

Chapter 4. Packaging and redistributing the sourcecode

Table of Contents

Quick tarb all creation

Custom archivesCleaning the project t ree

Quick tarball creation

Creating a tarball of the project is as simple as running

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

14/43

3 4

3 4

By default, the archive name uses the v ersion and t he project n ame found in the top-levelwscript file

The default compression format is bzip2. It may be changed to gzip by using the followingcode:

Custom archives

Custom archives can be created by providing a function named " dist" in the top -level scriptfile. The following example prints a checksum immediately after the archive is created:

A function may be defined for excluding or adding more files to the archive. The hook isexecuted from the temporary directory after the dist function has finished to run:

When copying the files, the dist function automatically excludes temporary files. Thefunction dont_dist in the module Scripting does the filtering, the code is reproduced below:

To exclude new files, it is possible to modify the extensions, or to provide a new methoddont_dist, for example, to disable most of the filtering:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

15/43

Cleaning the project tree

By default, all the files produced by Waf are produced into the build directory. An additionalfile named .lock-wscript is created to store information o n the build directory:

The command waf distcleanremoves these two elements from the filesystem. Afterwards, itis necessary to recompile the project again.

To remov e the build files without remov ing the configuration, use th e comm and waf cleaninstead. The files created in the build directory may be removed manually, but the

modifications will not be detected by Waf.

Chapter 5. The Task system

Table of Contents

Task creation and executionTask executionTask execution in parallel

Task execution orderExecut ing tasks only when something changes

Task creation and execution

Wh en W af tasks are created, they are not executed immediately, for ex ample it is not

necessary to rebuild an application if the source files have not changed. Some tasks mayalso depend on other tasks which may not be c reated yet. For this reasons Waf delays theexecution of t asks to the point when all build functions have been execut ed.

Creating all tasks by hand is a tedious process that the task generators ( Chapter 6, Taskencapsulation using task generators) may automate. Before starting the build, Waf asks eachtask generator to produce the corresponding tasks. If Waf is launched from a sub folderinside the source directory, it will try to avoid the creation of the tasks that are not relevantfor that particular sub folder (optimization).

Once the tasks are created, Waf will review each of them one by one to decide whether toexecute them o r not. A summary can be found on the following diagram:

Task execution

Executing a task consists in calling the method runon that task, and setting the taskexecution state. The following diagram is a summary of the process:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

16/43

8/8/2019 2009 - The Waf Book (v1.5.4)

17/43

Task execution order

Runn ing tasks in parallel is a simple problem, but in pr actice it is more c omp licated:

Dependencies can be discov ered during the build (dy namic task creation)New ordering constraints can be discovered after files are compiledThe amount of tasks and ordering con straints (graph size) can b e huge and

performance may be a problem

To make the problem more simple, it is divided by the different concerns, and the ordering

constraints can be given on three different levels:

1. groups of tasks may run only after another group of tasks has finished to run, this

represents a strict sequential order between groups of tasks, for example a compiler isproduced and used to compile the tasks in the nex t group2. task ty pes to indicate the instanc e will run after other task ty pe instances, for examp le

linking object files may only occur after compiling the source files3. specific con straints for task instanc es that can only run after a few other task

instances

Task groups

The following is used to declare groups of tasks to run one after the other:

The effect o f task gro ups when runn ing tasks in parallel is illustrated by the followingdiagram. Three groups of tasks have been added, and th e execution of th e next group o nlystarts when the ex ecution of the t asks in the previous group is complete.

Precedence constraints

The attributes beforeand afterare used to declare ordering constraints between tasks:

Another way to declare precedence constraints is to declare a file extension production, forexample:

The extensions have to match to add a v alid precedence constraint, but th ey are onlyannotations, they do not m ean the tasks actually have to produce fi les of that ty pe.

Precedence constraints o n task instances

The method set_run_after is used to declare ordering constraints between tasks:

unlike the previous constraints, it is used on the instances of class Taskwhich is a subc lassof class TaskBase

Executing tasks only when something changes

The direct instances of TaskBase are quite limited and do not track the changes to thesourc e files. The c lass Taskprovides the necessary features for the m ost co mmon builds inwhich source files are used to produce target files. The idea is to create a unique signaturefor tasks, and to represent the dependencies on files or other tasks by including them in the

signature. A hashing function is used for computing the signature, by default it is md5.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

18/43

3 4

The following diagram illustrates the task proc essing including th e signature, it is only v alidfor Task instance (not TaskBase instances):

The signature computation uses the following data:

1. ex plicit dependenc ies: input files and depend encies set ex plicitly using task.deps_man

or bld.depends_on2. implicit dependen cies: dependenc ies searched b y th e task itself (like source files

included from other source files).

3. parameter s: compilation flags and comm and-line parameters.

Here is an examp le illustrating t he different kinds of depend encies:

Environment variable dependencies (compilation flags)

Implicit dependencies: a method returns a list containing the list ofadditional no des to take into account, and the list of the files that could notbe found (cache)

Explicit dependencies as input files (nodes)

Explicit dependencies as manual dependencies

Manual dependencies on source files, the second parameter can be a string, anode object or a function returning a string

Manual dependencies with nodes, the first node represents a target (whichmay or may not exist in the buil d), and the second parameter represents afile in the source directory.

Chapter 6. Task encapsulation using task generators

Table of Contents

Ex plicit rules for simple transfor mations (Make-like)Implicit processing based on file name or extensionCustom pro cessing by t ask generator methodsTask generator method execution

Although task instances may be created directly, they act like global variables in the scripts,and they may lead to the c reation o f spaghetti code. The containers named task generatorsare used for creating the tasks and keeping track of them.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

19/43

3 4

Explicit rules for simple transformations (Make-like)

For simple transformations, a make-like scheme enables the creation of tasks by means oftask generat ors. Th e task is created lazily, fo r ex ample only if waf is called from asubdirectory of the pro ject co ntaining the declaration.

Simple rules

The following examples illustrate how to perform simple rule-based transformations, theattribute ruleis mandatory, but the input or output files are optional:

A basic example with an i nput and an output. The task is run whenever theinput or the rule change

The task will run whenever the input or the rule change, but there is nooutput

Creates a task with no input, the output is regenerated whenever the PREFIXvariable changes

Creates a task without inputs our outputs, it is r un whenever the rule textchanges

Advanced features

The next examples illustrate various attributes which may be used to alter task execution.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

20/43

Make a task which i s always run (attribute always)

Demonstrates how to add an orderi ng constraint (attribute before) toexecute the rule before the one that produces test.k1. It also demonstrateshow to set the directory from which the command is run (attribute cwd)

Rules may be given as functions. In this case the task has no input or o utputfiles

the dependant task copy_svnversion is executed when the actual ou tput ofsvnversion changes (attribute on_results)

By binding the rule to the actual declaration, this system limits code reuse. For thisreason, it is recommended for solving very specific problems in which a library would not

make sense. The creation of lots of small rules may also impact performance negatively.

Implici t processing based on file name or extension

The exclicit rules described in the previous section becomes limited for processing severalfiles of the same kind. Instead of adding loo ps, we will now describe ho w to p roc ess filesusing the name or the extension, for files declared in the sourceattribute. This scheme isthe most c ommo n technique used in the W af library.

Task chain declaration

The following example illustrates how to add a rule for transforming .luafiles int o .luac files:

In practice, the chaining declaration is located in a separate python file which may be re-used for other projects.

Using the extension decorator

A special method named apply_coreprovides file extension processing using the attributesource. On the user side, this takes the following form:

The method to process .c files (main.c) is called aut omat ically by apply_core, and isdeclared in the following m anner:

When the method apply_coreis called, the following op erations are perfor med:

1. If a method is mapped (@extension) to a n ame listed in sour ces, it is called with thatname as a parameter

2. The names that do not have a mapping are supposed to be files and are transformedinto Node instances

3. For each node (temporary variable self.allnodes), a method corresponding to the fi leex tension is applied

In this system, the list of sources for an application does not have to exactly represent files.Also, method s may re-inject new no des to process by extension (in the temporary variableself.allnodes), without keeping an exact reference on the method that will be called.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

21/43

Custom processing by task generator methods

In some cases the information to process is very specific and does not really fit in the list ofsources. The data may be attached to v arious attribute of the task generator, and a distincttask generator method may process the attributes dynamically.

In the following example, a new attribute 'tst' is added to task generators instances, and is

used to call a method to print the c ontents of that attribute at the start o f the build:

To attach a new method to the task_gen class (the type offoo), the decorator @taskgenis used in the following mann er:

At this point, the function print_tst is attached to the class as a new method, but to pro cessthe attribute foo.tst, the method has to be called explicitly:

To schedule that method for automatic execution, it is necessary to bind it somehow. Thealiases are given in the task generator attribute features:

For mapping the method to all features, the wildcard *may be used, a complete example isreproduced below:

Upon ex ecution, this produces the following output:

Task generator method execution

Methods may be added to task generators, and those methods are bound to features. At thebeginning o f the build, the task generator instances are executed one by one, and in turn

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

22/43

3 4

their methods are executed to generate the tasks or to modify additional data. We will nowdescribe what m ake task generator metho ds with an ex ample:

Task generator methods take a single parameter selfThey are bound t o particular names representing features(or to '*')They hav e order restrictions beforeand after

Here is an example of a task generator method from the W af source code:

In general, the methods cannot be called several times or in random order. For example, inc++ programm ing co ntexts, t he com pilation t asks must be created before link tasks because

link task will reference t he co mpilation t asks. This leads to a particular workflow, repro duc edin the following illustration:

The list of metho ds to execute is kept in the attribute methsof the task generator instance.

The list of methods t o ex ecute may be given directly.

It is however more co nvenient to use the attribute featureto add the methodsautomatically. For ex ample, for creating a c application, the methods t o ex ecute depend o n

the operating system, so the feature system actually simplifies the code reuse.

The order c onstraints on the metho ds (after/before), are used to sort the list o f methods inthe attribute meths. The sorting is performed once, and the list is consumed as methods areexecuted. Thoug h no new feature may be added once the first method is execut ed, newmethods may be added dynamically in self.meths. Here is how to make an infinite loop:

Chapter 7. The scheduler for executing the tasks

Table of Contents

The task execution modelJob control

Weak task order c onstraints

The task execution model

Task dependencies and task ordering specify the ex act order in which t asks must beexecuted. When tasks are executed in parallel, different algorithms may be used to improvethe compilation times. For example, tasks that are known to last longer may be launched

first. Linking tasks that use a lot of ram (in the context of c++ applications) may be

launched alone to avoid disk thrashing by saving RAM.

To make this possible, the task execution is organized in the following manner:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

23/43

Job control

Job control is related to the parallelization algorithms used for launching the tasks. While theaim of parallelization is to maximize the amount of tasks in execution, different algorithmsmay b e used in practice

In the NORMAL ordering, task groups are created, and a topological sort is performed in

adv ance. The ov erall performance pen alty for c omplete bu ilds is usually sm all, like a fewseconds on builds during minutes.

In the JOBCONTROL ordering, groups are created in advance, and a flag indicates themaximum amoun t of jobs to b e used when the consumer threads execute the tasks. Thisprevents parallelization of tasks which use a lot of resources. For example, linking c++ object

files uses a lot of RAM.

In the MAXPARALLEL ordering, Each task holds a list of tasks it must run after (there is only

one list of tasks waiting to be ex ecuted ). Thoug h this v ersion parallelizes tasks ver y well, itconsumes mo re memory and processing. In practice, Waf may last 20% more on buildswhen all tasks are up-to-date.

Warning

Because most task classes use ordering constraints, the maximumparallelization can only be achieved if the constraints betweentask classes are relaxed, and if all task instances know theirpredecessors. In the example graph, this was achieved by

removing the ordering constraints between the compilation tasksclasses and the lin k tasks classes

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

24/43

From this, we can immediately notice the following:

An assumption is made that all tasks have input and output nodes, and that ordering

constraints can be deduced from themDeducing the co nstraints from the input and output nodes exhibits a n^2 behaviour

Note

In practice, the NORMAL algorithm should be used wheneverpossible, and the MAXPARALLEL should be used if substantialgains are expected and if the ordering is specified between alltasks. The JOBCONT ROL system may be useful for tasks thatconsume a vast amount of resources.

Weak task order constraintsTasks that are known to take a lot of time may be launched first to improve the build times.The general pro blem of finding an opt imal order for launching t asks in parallel and withcon straints is called Job Shop. In practice this problem can often be reduced to a critical

path pro blem (approx imation).

The following p ictures illustrate t he difference in sch eduling a build with d ifferent independ enttasks, in which a slow task is clearly identified, and launch ed first:

Waf pro vides a function for reordering the tasks before they are launched in the m oduleRunner, the default reordering may be changed by dynamic method replacement in Python:

If the reordering is not to be performed each time a task is retrieved, the list of task may bereordered when the next group is retrieved:

It is possible to measure the task execution times by intercepting the function calls. The task

execution times may be re-used for optimizing the schedule for subsequent builds:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

25/43

Chapter 8. C and C++ projects

Table of Contents

Comm on script for C/C ++ applicationsLibrary interaction (uselib)Customizing object filesUsing configuration headersThe include system

Configuration helpersPkg-config

Common script for C/C++ applications

The c/c++ builds consist in transforming source files into object files, and to assemble theobject files at the end. In theory a single programming language should be sufficient forwriting any application, but in practice slight variations exist:

Applications may be divided in dynamic or static librariesAdditional files may enter in the link step (libraries, objec t files)Source files may be generated by other c ompilers

The con struction of c /c++ applications can be quite complicated, and several measures must

be taken to ensure coherent interaction with new compilation rules. The canonical code for atask generat or b uilding a c/c ++ application is the following:

Task generator declaration; each element in the list represent a feature; it ispossible to add several languages at once (ocaml and c++for example),but the one ofobjects, staticlib, shlib or programmust be chosen.

List of source, it may be either a python list, o r a string containing the filenames separated with spaces. This list may contain fil e names of differentextensions to make hybrid applications.

Target name, it is concerted to the name of the binary name.soorname.exedepending on the pl atform and the features.

List of include paths, it may be either a python li st, or a string containing thepaths separated by spaces. The paths are used for bo th the command-li ne

and for finding the implicit dependencies (headers). In general, include pathsmust be relative to the wscript file and given expli citly. See the section called

The in clude system.

Installation directory, this is where to install the library or programproduced. The ${} expression is a reference to a variable to be extractedfrom tgen.env. By default it is set to ${PREFIX}/bin for pro grams and${PREFIX}/lib for libraries. To disable the installation, set it to None.

Command-line defines: list of defines to add to the command-line with the -Dprefix. To reduce the size of the command-line, it is possible to use aconfiguration header, see the following section for more details.

Command-line compilation flags, for the c++ language the attribute is calledcxxflags

Shared libraries may be given directly (use staticliband staticlibpathfo rstatic libraries)

Additio nal pararameters may be added from a task generator reference. Thenext section describes a technique to gather the conditions i nto theconfiguration section.

Library interaction (uselib)

To link a library against another one created in the same Waf project, the attribute

uselib_localmay be used. The include paths, the link path and the library name are

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

26/43

automatically ex ported, and the dependent binary is recompiled when th e library c hanges:

A static library

Include paths to export for use with uselib_local (include paths are notadded automatically). T hese folders are taken relatively to the current target.

A program u sing the static library declared previously

A list o f references to existing l ibraries declared in the proj ect (either apython list or a string containing the names space-separated)

To link an application ag ainst v arious system libraries, sev eral comp ilation flags and linkflags must be g iven at once. To reduce the m aintenance, a system called uselibcan be usedto g ive all the flags at the sam e time:

Declare a few variables during the configuration, the variables follow theconvention VAR_NA ME

Add all the VAR_NA ME corresponding to the uselib NA ME, which is 'TEST' inthis example

The v ariables used for c /c++ are t he following: S TATICLIB, LIB, LIBPATH, LINKFLAGS,RPATH, CXXFLAGS, CCFLAGS, CPPP ATH, CPPFLAGS, CXXDEFINES, FRAMEWORK,

FRAMEWORKPATH, CXXDEPS. The uselib is similar to the cascading style sheet (CSS)principle.

Customizing object files

In some cases, it is necessary to re-use object files generated by another task generator toavoid recompilations. This is similar to copy-pasting code, so it is discouraged in general.Another u se for th is is to en able some co mpilation flags for spec ific files. The attr ibute"add_objects" can be used, like in the following example:

Files will be compiled in c mode, but no program or library will be produced

Different compilation flags may be used

The objects will be added automatically in the link stage

Using configuration headers

Adding lots of command-line define values increases the size of the command-line andconceals the useful information (differences). Some projects use headers which aregenerated during the configuration, they are not m odified during the b uild and they are notinstalled or redistributed. This system is useful for huge projects, and has been made popular

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

27/43

by autoconf-based projects.

Writing configuration headers can be performed using the following methods:

The code snipped will produce the following config.h in the build directory:

The contents of the c onfig.h for this example are

The include system

The Waf preprocessor

Include paths are used by the c/c++ compilers for finding the headers. When one header

changes, the files are recompiled automatically. Includes must be given in the following form:

Local folders are given relative to the folder of the current script. The equivalent builddirectory folder is then added automatically for headers or source files produced in the builddirectory.

Waf uses a preproc essor written in Py thon for adding the dependencies on the headers. Asimple parser looking at #include statements would miss constructs such as:

Using the compiler for finding the dependencies would not work for applications requiring filepreprocessing such as Qt. For Qt, special include files having the .mocextension must bedetected by the build system and produced ahead of time. The c c ompiler could not parse

such files.

Using th e W af preprocessor also makes the scripts strongly compiler-independent.

Portability recommandations

By default, the preprocessing does not climb to system headers. This may lead to misseddepdend encies in the cases similar to the following:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

28/43

3 4

To write portable code and to ease debugging, it is strongly recommended to p ut all theconditions used in the p roject into a config.h file.

Although it is discouraged, it is possible to enable the preprocessor to look into system

headers by using the following code:

Debugging dependencies

The W af preprocessor co ntains a specific debugging zone:

To display the dependencies obtained or missed, use the following:

The dependency c omputation is performed on ly when the files are not up-to -date, so thesecom mands will display someth ing only wh en there is a file to c omp ile.

Configuration helpers

The methods check_ccand check_cxxare used to detect parameters using a small buildproject. The main parameters are the following

msg: title of the test to executeokmsg: m essage to display when the test succ eeds

errmsg: message to display when the test failsmandatory: when true, raise a configuration exception if the test failsenv: env ironment t o use for the bu ild (conf.env is used by d efault)define_name: add a define for the configuration header when the test succeeds (in

most cases it is calculated automatically)

Here is a concrete example:

Try to compile a program using the configuration header time.h, if presenton the system, if the test is successful, the define HAVE_TIME_H will beadded

Try to compile a program with the function printf, adding the header stdio.h(the header_name may be a list of additional headers). The parametermandatory will make the test raise an exception if it fails.

Try to compile a piece of code, and i f the test is successful, define the nameboobah

Modifications made to the task generator environment are not stored. Whenthe test is successful and when the attribute uselib_store is pr ovided, thenames lib, cflags and defines will be converted into uselib variabl es LIB_M,

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

29/43

3 4

3 4

CCFLA GS_M and DEFINE_M and the flag values are added to theconfiguration environment.

Try to compi le a simple c program against a library called 'linu x', and reusethe previous parameters for libm (uselib)

Execute a simple program, collect the output, and put it in a define whensuccessful

After all the tests are executed, wri te a configuration header in the buil ddirectory (optional). The configuration header is used to limit the size of thecommand-line.

Here is an example of a config.hproduced with the previous test code:

The file default.cache.py will con tain th e following variables:

Pkg-config

Instead of duplicating the configuration detection in all dependent projects, configuration

files may be written wh en libraries are installed. To ease the interac tion with bu ild sy stemsbased on Make (cannot query databases or apis), small applications have been created forreading the cache files and to interpret the parameters (with names traditionally ending in -config): pkg-config, wx-config, sdl-config, etc.

Waf prov ides the method check_cfg for querying c onfig parameters:

Check for the pkg-config version

Check for a module version

Obtain the flags for a package and assign them to the uselib PANGO(calculated automatically from the package name, can be overridd en with theattribute "uselib_store='MYPANGO'")

Retrieve the module version for a package. The returned obj ect is a stringcontaining the version number or an empty string in case of any errors. Ifthere were no errors, 'PANGO_VERSION' is defined, can be overridden withthe attribute "uselib_store='MYPANGO'".

Obtain the flags for a different configuration program (sdl-config). Theexample is applicable for other configuration programs such as wx-config,pcre-config, etc

Due to the amount of flags, the lack of standards between config applications, and to t heoutput changing for different operating systems (-I for gcc , /I for msv c), the outp ut of pkg-config is parsed, and the variables for the corresponding uselib are set in a go. The function

parse_flags(line, uselib, env)in the Waf module config_c.py performs the outputanalysis.

Chapter 9. Advanced scenarios

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

30/43

Table of Contents

Simple file transformat ionsSame targets, different configurations (variants)

Building th e co mpiler firstWriting the output o f a program into a headerA compiler producing source files with names unknown in advanceA task without any file dependency

This chapter provides examples for advanced scenarios to d emonstrate the practical use of

the Waf library.

Simple file transformations

The Waf tool misccontains various routines to ease file manipulations such as substitutingparameters or executing custom compilers. The objective of this section is to illustrate the

principles of the current apis.

File substitutions

The following example illustrates how to produce a pkg-config file from a template:

The variables must be declared in the template file by enclosing the names between @characters(m4 syntax).

The default substitution funct ion may be replaced by changing the attribute funcof the taskgenerator.

Compiler output

The command-output system enables execu tion of system co mmands without requiring ashell. The following example illustrates how to execute a script present in the sourcedirectory each time it is executed.

misc is the Waf module providi ng the command-output features

The standard output will be redirected to the file test1

The file 'wscript_build' will be passed to the standard input

The progr am to execute is assumed to be present in the directory of thewscript file, unless command_is_external is set to True

Additio nal parameters are provided in the argv attribute, as a list.

Parameters representing fi les must be added through the special classesinput_file, output_file, input_dir and output_dir to resolve the paths properly

An additio nal dependency is added on the time, the task will be executedwhenever the time changes

Same targets, different configurations (variants)

In many builds, it is necessary to build the same targets for different purposes: debugging,

profiling, optimizations. A system is provided for duplicating the targets easily.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

31/43

Variants and environments

The following code demon strates how to create a configuration for a new variant named

'debug'

After running waf configure, the bu ild folders corresponding to the defaultand to thedebugvariants are produced:

To use the debugvariant, the environment must be g iven d irectly, by default, a cop y o fthe environment named default is set:

Cloning targets

Task generators may be copied for different variants easily using the clonemethod. Theparameter is the name of the env ironment to use (not the variant). The namedenvironments are created du ring t he co nfiguration, as shown in the previous subsection.

It is also possible to clone all task generators from a project using a code similar to the

following:

Cross-variant dependencies are discouraged

The variant system is meant for automating the duplication of targets, which are supposedto run in in isolation. As a consequence, it is not specified how linking a program against a

library created in a different variant should be processed. The general recommendation is toavoid optimizations and to duplicate the targets as necessary. The following bug trackerentry prov ides mo re details http://code.google.com/p/waf/issues/detail?id=171

Building the compiler first

The example below demonstrates how to build a compiler which is used for building theremaining targets. The requirements are the following:

Co mpile the co mpiler with all its intermed iate tasksRe-use the compiler in a second build step

This scenario d emonstrates the bld.add_group to segment the build into strictly sequentialparts:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

32/43

3 4

Create the compi ler.

Create a new group

The rest of the build follows

Writing the output of a program into a header

The example below demonstrates how to write the outpu t of svn version into a header. The

command is always run, and the c/c++ compilation will be executed if the header changes(implicit dependencies obtained by the Waf preprocessor):

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

33/43

3 4

Example in a user scrip t, the attribute svn_headeris used to ind icate aheader to create as a build fil e

Process the attribute svn_header when present, create a task instance

A function for obtaining the svn version of the project

Obtain the svn version and wri te it to the header

Declaration of the task which will execute the svn processing

The task must be run each time - there is no si gnature involved

Mark the outputs as changed if necessary by changing the signature

Task creation is performed through a new task generator metho d (proc ess_svn) which isexecuted only when the task generator creates all its tasks. This organization makes itpossible to compile only the code necessary for a specific target (tasks are created lazily,

and are grouped by task generators). To address the scenario described, a new task typ e iscreated, and metho ds are replaced dynamically: runnable_status (execute each time), andpost_run (ch ange the signature of the outputs acco rding to the co ntents of the fi le

created).

A compiler producing source files with namesunknown in advance

The example below demonstrates how to tackle the following requirements:

A com piler produces source files(.c files) for which tasks must be created

The source file names are not kno wn in advanceThe task must be run o nly if necessaryOther tasks may depend on the tasksprocessing the source pro duced (co mpile andlink the .c files)

The main difficulty in this scenario is to store the information on the source file produced

and to c reate the corresponding tasks each time.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

34/43

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

35/43

3 4

An example. The source line contains a directive foo.shpipwhich triggersthe creation o f a shpip task (it does not represent a real file)

This m ethod is used to create the shpip task when a file ending in .shpip isfound

Create the new task type

Disable the warnings raised because the task has no i nput and outputsExecute the task

Retrieve the information on the source files created

Create the c++ tasks used for processing the source fil es found

If the tasks are created during a task execution (in an execution thread), thetasks must be re-injected by adding them to the attribute more_tasks

If the tasks are created during the task examination (runnable_status), thetasks can be injected directly in the bui ld by usi ng the attribute outstandingof the scheduler

A task without any file dependency

Given the following requirements

Atask x produces a headerused in c/c++ compilationsThe c/c++ task depends on the production of that headerThat c task must be executed whenever the header actually changesThe header may or may not be upd ated, but the header production task must runeach time

The following example demonstrates how to create a task type fulfilling the requirements,and how to use it.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

36/43

3 4

Create a new task type, in this example, we pretend it runs a command toretrieve the svn version and stores i t in a header.

Replace runnable_statusby a new method which i ndicates the task must

be run each timeIndicate that the task must always be run before c++ ones

By default, the task signature is assigned to the node info rmation. In ourcase the task signature is always the same, and we need a way to ind icate todependent tasks that something has changed. The soluti on is to com pute ahash of the file produced, and to assign it to the node information.

Demonstrate the manual creation of the task

The task outputs are Node instances, but there are no inputs.

Chapter 10. Customizing the scripting system

Table of Contents

Providing custom commands

Chaining co mmandsProviding a custom command context

Creating aliases / Injecting new commands

Providing custom commands

Waf commands are functions defined on the top-level wscript file, for example, with thefollowing wscript file:

executing the script with " foo" as argument will produce th e following output

The parameter ctxin the function foo above is provided to enable recursion and sharingdata with the scripts. In the following example, the script subdir/wscript_foo will be looked

up, and executed if found. Else, the script subdir/wscript will be look ed up an d loaded as a

module. If successfully loaded, the function foowill be execu ted, else an ex ception is raised.

For backward co mpatibil ity reasons, the comm ands init, shut down, d istand distcheckdonot accept a con text as parameter yet.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

37/43

8/8/2019 2009 - The Waf Book (v1.5.4)

38/43

3 4

3 4

Chapter 11. Using the development version

Table of Contents

Tracing the executionBenchmarkingProfiling

Tracing parallel task executionObtaining the latest source c odeProgramming con straints

Tracing the execution

The generic flags to add m ore information t o the stack traces or to th e messages is -v(verbosity), it is used to display the command-lines executed during a build:

To display all the traces (useful for bu g repo rts), use th e following flag:

Debug ging information c an be filtered easily with the flag zones:

Tracing zones m ust be c omma-separated, for example:

The Waf module Logsreplaces the Py thon module logging. In the sourc e code, traces areprovided by using the debugfunction, they must obey the format "zone: m essage" like in

the following:

The following zones are used in Waf:

Table 11.1. Debugging zones

Zone Meaning

runnercommand-lines executed (enabled when -v is provided withoutdebugging zones)

task_gen task creation (from task generators)

action functions to execute for building the targets

env environment contentsenvhash hashes of the environm ent objects - helps seeing what changes

build bui ld context operations such as f i lesystem access

deps implic it dependencies found (task scanners)

Benchmarking

The script utils/genbench.py generates a simple benchmark for Waf. The habitual use is the

following:

The project created contains 50 libraries with 100 classes for each, each source file having15 include headers pointing to the same library and 5 headers pointing to the headers ofother libraries in the project.

The time taken to c reate the tasks and to resolve the dependencies can b e obtained by

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

39/43

3 4

3 4

injecting code to disable the actual compilation, for example:

Profiling

Profiling information is obtained by calling the module cProfile and by injecting specific code.The two most interesting m ethods to profile are flushand compile. The most importantnumber from the profiling information is the amount of function calls, and reducing it results

in noticeable speedups. Here is an example on the method compile:

Here the output obtained on a bench mark build created as ex plained in the prev ious section:

From the profile information, it appears that the hot spots are, in order:

The persistence implemented by the cPickle module (the cache file to serialize takesabout 3Mb)Accessing configuration data from the Env ironment instances

Computing implicit dependencies (the test project contains lots of interleaveddependencies)

In practice, the time taken by these methods is not significant enough to justify codechanges. Also, profiling must be carried out on builds which last at least several seconds.

Tracing parallel task execution

A special W af tool named ParallelDebugis used to inject code in Waf mo dules and o btaininformation on the execution. This module is provided in the folder playground and must be

imported in one's project b efore use:

After executing a full build, a trace of the execution is output in /tmp/test.dat; it may beprocessed by other applications such as Gnuplot:

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

40/43

Obtaining the latest source code

Waf is hosted on Google code, and uses Subversion for source co ntrol. To obtain thedevelopment copy, use:

To avo id regenerating Waf each t ime, the environment v ariable W AFDIR c an be used topoint to the directory containing wafadmin:

Although the Waf binary depends on Python 2 .3, the Waf source code depends on Py thon2.4. W hen generating Waf, a parser modifies the source code and performs the followingoperations:

Move the decorators into simple function callsAdd the imports for the module setsEliminate redundant spacesStrip comments to reduce the size

Programming constraints

Though Waf is written in Py thon, additional restrictions apply to the source c ode:

Identation is tab-only, and t he maximum line length should be abou t 20 0 c haracters

The development code is kept compatible with Py thon 2 .3, to the ex ception ofdecorators in the To ols directory. In particular, the W af binary c an be generated usingPython 2.3

The wafadmin modules must be insulated from t he Tools modules to keep the Waf core

small and language independentApi compatibility is maintained in the cycle of a minor version (from 1.5.0 to 1.5.9)

More code always means more bu gs. Whenev er possible, unnecessary code must beremoved, and the existing code base should be simplified.

Chapter 12. Overview of the Waf architecture

Table of Contents

The core library

Build contex t instancesOverview of the Waf executionPerformance and build accuracy

The core library

Waf is based on 1 2 mo dules which constitute the co re library. They are located in thedirectory wafadmin/. The modules located under wafadmin/Tools add support for

programming languages and more too ls, but are not essential for the W af core.

Table 12.1. The core library

Module Role

BuildDefines the build context class, which holds the data for one bui ld(paths, configuration data)

ConfigureContains the configuration context class, which is used forlaunching confi guration tests, and the extension system

Constants Provides the constants used in the project

EnvironmentContains a dictionary class which supports a lightweight copyscheme and pro vides persistence services

Logs Provide a logging system

Nod e C ontai ns th e fi le system repr esen tati on cl ass

OptionsProvides a custom command-line option processing system basedon optparse

Runner Contains the task execut ion system (threaded producer-consumer)

ScriptingConstitutes the entry point of the Waf application, use thecommand-line for launching the configuration, the build, etc

TaskGenProvide the task generator system, and its extension system basedon method addition

Task Con tains the task cl as ses, and the task container s.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

41/43

UtilsContains the support function s and classes re-used in other Wafmodules

The essential classes and methods from the core library are represented on the followingdiagram:

Build context instances

Executing tasks, accessing the file system and consulting the results of a previous build arevery different c oncerns which have to b e encapsulated properly. The co re class representinga build is a build context.

Build context and persistence

The build context holds all the information necessary for a build. To accelerate the start-up,a part of the information is stored and loaded between the runs. The persistent attributesare the fo llowing:

Table 12.2. B uild cont ext persistence

Attribute Information

r oo t Node r ep resen ti ng the r oo t o f the fi le s ys tem

srcnode Node representing the source directory

bldnode Node representing the bui ld directory

node_sigs File hashes (dict mapping Node ids to hashes)

node_deps Impli cit dependencies (dict mapping Node ids)

raw_deps

Impli cit file dependencies which could not be resolved (dict mapping

Node ids to lists of strings)

task_sigsSignature of the tasks previously run (dict mapping a Task id to ahash)

id_nodesSequence for generating unique node in stance ids (id o f the last Nodecreated)

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

42/43

Build context access

In previous Waf releases, the build context was supposed to be a unique object (one buildactive at a time). To enable the use of Waf as a library, the dependency on the singletonBuild.bldwas removed. This implies that each object should be able to obtain its buildcontext from its attributes. Here are a few examples:

Table 12.3. B uild cont ext access

Object type Build context access

Node self.__class__.bld

task_gen self.bld

Task self.generator.bld

Parallelization concerns

Build contexts perform an os.chdircall before starting to execute th e tasks. When runningbuild con texts within build contex ts (tasks), the current working directory m ay c ause

various problems. To work aroun d them, it may be necessary to c hange the com pilationrules (compile from the file system root) and to inject code (replace bld.compile).

Direct Nodeinstances are not used anywhere in the W af code. Instead, each b uild co ntextcreates a new Node subclass (bld.node_class), on which the build context instance isattached as a class attribute.

Threading concerns

Nearly all the code is executed in the main thread. The other threads are merely waiting fornew tasks, and ex ecuting the methods run from the task instances. Such methods shou ldcontain as little code as possible, and access the BuildContext in a read-only manner.

If the run methods hav e to modify the build cont ext, it is recommended to overload themethod get_outof the scheduler and to execute methods in an ev ent-like manner (data isattached to the task, and the method g et_out ex ecutes the code). Adding more tasks froma running task is demonstrated the section called A compiler producing source files withnames unknown in advance.

Overview of the Waf execution

File system access

File system access is performed thro ugh an abstraction layer formed b y the build c ontextand Nodeinstances. The data structure was carefully designed to maximize performance, soit is unlikely t hat it will change again in the fut ure. The idea is to represen t on e file or one

directory by a single Node instance. Dependent data such as file hashes are stored on thebuild context object and allowed to be persisted. Three kinds of nodes are declared: files,build files and folders. The nod es in a particular directory are unique, but build files used inseveral variant add dup licate entry on t he build co ntext cache.

To access a file, the methods Node::find_resource, Node::find_build(find an existingresource or declare a build node) and Node::find_dirmust be used. While searching for aparticular node, the folders are automatically searched once for the files. Old nodes (whichdo no t have a corresponding fi le) are automatically removed, ex cept for the build no des. In

some cases (lots of files added and remo ved), it may be necessary to perform a Waf cleanto eliminate the information on build files which do not exist anymore.

Task classes

The whole process of generating tasks through W af is performed by m ethods added on the

class task_gen by code injection. This often puzzles the p rogrammers used to staticlanguages where new functions or classes cannot be defined at runtime.

The task generators automatically inherit the build context attribute bld when created frombld.new_task_gen. Likewise, tasks created from a task generator (create_task) automaticallyinherit their generator, and their build context. Direct instantiation may result in problems

when runn ing Waf as a library.

The tasks created by task generator metho ds are automatically stored on t he build con texttask manager, which stores the t ask into a t ask group. The task groups are later used by thescheduler to obtain the task which may run (state machine). Target (un)installation isperformed right after a task has run, using the method install.

Performance and build accuracy

From the exp erience with too ls such as SCons, users may be c oncerned about performanceand think that all build systems based on interpreted languages such as Python would notscale. We will now describe why this is not the case for Waf and why Waf should be chosen

for building very large projects.

Comparing Waf against o ther build systems

Since Waf considers the file contents in the build process, it is often thought that Waf wouldbe much slower than make. For a test project having 500 0 fi les (generated from the script

located in tools/genbench.py), on a 1.5Ghz computer, the Waf runtime is actually slightly

faster than the Gnu/Make one (less than one second). The reason is the time to launch a

new process - make is usually called recursively, once by directory.

10/10/2010 he Waf Book (v1.5.4)


8/8/2019 2009 - The Waf Book (v1.5.4)

43/43

For huge projects, calling make recursively is necessary for flexibility, but it hurtsperformance (launch many processes), and CPU utilization (running tasks in parallel). Make-

based build systems such as CMake or Autotools inherit the limitations of Make.

Though Waf uses a similar design as SCons, Waf is about 15 times faster for similar featuresand without sacrificing build accuracy. The main reasons for this are the following:

The Waf data structures (file system representation, tasks) have been carefully chosento minimize memory usage and data duplicationFor a project of the same size, SCons requires at least 10 times as many function calls

A few benchmarks are maintained at this location

Waf hashing schemes and build accuracy

To rebuild targets when source file change, the file contents are hashed and compared. The

hashes are used to identify the tasks, and to retrieve the files from a cache (folder definedby the environment v ariable WAFCACHE). Besides command-lines, this scheme also takesfile dependencies into account: it is more acc urate than c aching systems such as ccache.

The W af hashing scheme u ses the md5 algorithm prov ided by the Py thon distribution. It isfast enough for up to about 1 00Mb of data and abou t 100 00 fi les and very safe (virtually norisk of collision).

If more than 1 00Mb of data is present in the project, it may be necessary to use a fasterhashing algorithm. An implementation of the fnv algorithm is present in the W af distribution,and can replace md5 without really degrading accu racy.

If more t han 100 00 fi les are present, it may be necessary to replace the hashing system bya file name+size+timestamp hash scheme. An example is provided in the com mentsection o f the mod ule Utils.py. That scheme is more efficient but less accurate: the Waf

cache should not be used with this scheme.

Chapter 13. Conclusion

The features provided by Waf aim at building software in a maintainable manner, in whichthe dependencies for other piece of software is limited to the compilers and a Pythoninstallation. Due to the amoun t of features prov ided by Waf, the book is forcibly incomplete.For greater understanding and practice the following links are recommended to the reader:

Table 13.1. Reco mmended links

Link Description

http://code.google.com/p/waf The Waf project page

http://code.google.com/p/waf/w/listThe Waf wiki, including the frequentlyasked questions (FAQ)

http://groups.google.com/group/waf-

users The Waf mailing-list

Glossary

Waf essential terms

Task Generator

A task generator is an object instance of the class Task.task_gen. The task generatorsencapsulate the creation of various task instances at a time, and simplify the creationof ordering co nstraints between them (for example, compilation tasks are execut edbefore link tasks).

Task

A Waf task is an object instance of the class Task.TaskBase. Waf tasks may be simple(Task.TaskBase) or related to the filesystem (Task.Task). Tasks represent the

production of something during the build (files in general), and may be executed insequence (with ordering constraints) or in parallel.

Tool

A Waf tool is a pyt hon m odule containing W af-specific ex tensions. The W af tools arelocated in the folder wafadmin/Tools/ and usu ally c ont ain a global variable detect

which may reference functions to execute in the configuration.

10/10/2010 he Waf Book (v1.5.4)

2009 - the waf book (v1.5.4)

Documents