1

Integration of cfengine and scliVladislav Marinov

Jacobs University Bremen*Supervisor: Prof. Juergen Schoenwaelder

Networks and Distributed Systems Seminar, Spring Semester 2007

Abstract—The cfengine program is used to configure Unixmachines. It is a policy engine which reads a file describing apolicy how a system should be configured and tries to enforce thispolicy. The scli program is an SNMP management programwhich has among other features the ability to configure VLANs.The goal of this project is to integrate cfengine and scli sothat the former can monitor and configure SNMP enabled devicesvia the latter. For this purpose a communication protocol has tobe developed and the implementations of the two tools have tobe adapted so that they can support the protocol.

Index Terms—Network management, communication protocol,thread-safe support, open source integration

I. INTRODUCTION

The objective of this project is to marry two open sourcesoftware packages. The first one is cfengine, a policy hostconfiguration system [1]. The second one is scli [2], acommand line interface to network devices running on top ofSNMP [2]. In this section we provide a short backgroundabout these two tools. In section II we present the motivationbehind this topic and the research question which describesthe work that had to be carried out. In section III we describethe communication protocol that was developed and in sectionIV we discuss the implementation work that was carried outin order to enable thread support for scli and to make thecode thread safe. Finally in section V we present some relatedwork in terms of algorithms and tools that can be used totest whether some particular implementation is thread safe. Insection VI we present some final thoughts.

A. cfengine

Cfengine [1] is a configuration management environmentthat can be used for configuring and testing Unix-like systemsattached to a TCP/IP network. It consists of a high levelpolicy language and a low level logic engine. Cfengine tiesconfiguration policy to system monitoring, thereby allowingfeedback and reactive configuration. Cfengine does notcurrently interface with SNMP.Cfengine focuses on a few key functions which are handledrather poorly from scripts. It eliminates the need for lotsof tests by allowing administrators to organize the networkaccording to classes. A single configuration file (or set of files)is specified, using classes, describing how the system shouldbe configured. cfengine will then parse the file and carry

*Jacobs University Bremen is formerly known as International UniversityBremen

Fig. 1. scli architecture

out the instructions, warning or fixing errors as it goes. Thebasic functions [3] include:

• Checking and configuring network interfaces• Editing textfiles• Maintaining symbolic links• Checking and setting the permissions and ownership of

files• Tidying junk files which clutter the system• Systematic, automated mounting of filesystems (Unix)• Checking for the presence of important files and filesys-

tems• Controlled execution of user scripts and shell commands

Another component of cfengine performs machine-learningfunctions. This is the daemon cfenvd which is collectinglonger term statistics in order to detect anomalies based on theTwo-Dimensional Time Series (2DTS) and the Leap DetectionTest algorithm (LDT). [4].

B. scli

scli [2] is a tool which provides an efficient to usecommand line interface to display, modify and monitor data re-trieved from SNMP agents. It runs on simple ASCII terminalsand does not require any graphical user interface capabilities.The tool provides command line editing, completion andhistory capabilities to make it easy for network operators andsystem administrators to use this tool, even if they cannotremember the precise command syntax.

2

The overall software architecture is shown in Figure 1. Thepackage uses the glib library to achieve portability and toreuse generic data structures such as lists and dynamic strings.The SNMP engine gsnmp has been derived from the gxsnmppackage and was subsequently modified to fix bugs and toimprove stability. The SNMP engine itself uses glib.The gsnmp library provides roughly the same low-levelfunctionality as many other SNMP APIs. A MIB compiler isused to generate C stubs from MIB modules. The stubs consistprimarily of C structures which represent MIB table rows orgroups of scalars plus a set of stub functions which can beused to read/write these structures. The implementations ofthe stub functions serialize/deserialize the C structures intoSNMP varbind lists. They also validate the data to ensure thatthe elements of the varbind lists have appropriate types andsizes.The scli command implementations either use the stubsdirectly or they use so called MIB procedures. MIB proceduresextend the stub interface with specific functions for commonoperations like creating rows in certain MIB tables or iteratingover certain MIB tables. MIB procedures are by definitionMIB specific and implemented entirely by using the stubinterface.The scli interpreter core provides all the infrastructureneeded to register commands, to tokenize the input stream,to locate and execute the function implementing a recognizedcommand and to finally display the results on stdout or viaa pager. The interpreter uses the GNU readline and historylibraries for command line editing and the curses library forscreen management. It also uses glib data types internally.All state information is bound to the scli interpreter.The interpreter core and some command implementations alsouse the libxml2 library to create and manipulate XMLdocuments. By using a dedicated XML library, it is possibleto ensure that any generated XML output is well-formed.

II. MOTIVATION AND RESEARCH QUESTION

This project is carried within the framework of workpackage 6 (WP6) of the European Network of Excellencefor the Management of Internet Technologies and ComplexServices (EMANICS) [5]. The primary focus of WP6 is thedevelopment of open source initiatives and joint software.Open Source support and coherent joint software developmentis a very important initiative for ensuring strong integration,network survivability and transfer of technology developedwithin the network. In the management community, manyOpen Source components have been produced over the years,some of them being widely used today for several purposes.Recent evolutions in management and networking technolo-gies however did slow the use of these very valuable softwarecomponents and the rhythm of new productions has decreaseddrastically. Today, there is a need to support the Open Sourceinitiatives in the area of management to foster again their useand acceptance on large scale.Cfengine as described is used by millions of users world-wide and has an established user base. Scli has been de-scribed in [2] but so far is only used by a rather small number

of sites. The integration of these two packages will (a) addimportant features to cfengine in order to configure networkinterfaces via cfengine and (b) enhance the technologyon which scli is based and at the same time increase itsvisibility.By integrating these two packages, software coming from andused by slightly different communities is integrated, namelysystem administrators and network managers. This integrationwill hopefully open the floor for some synergies and ideallynetwork managers will start to adopt more automated ap-proaches to configuration management while system admin-istrators will take advantage of very lightweight monitoringprotocols.The main goal is to integrate SNMP access to networkdevices into cfengine. cfenvd is a cfengine daemonwhich is collecting long term statistics in order to detectanomalies in system parameters. Currently the data that isanalyzed by cfenvd is obtained from localhost. The goal isto extend this daemon so that it can also collect key parametersfrom remote systems using SNMP. The cfagent componentof cfengine enforces policies written in the cfenginelanguage on the local host system. The goal is to enhance thecfengine language in such a way that it is possible to definepolicies that should be enforced on remote network elementsthat can talk SNMP. In the end, cfengine should be able tomonitor, control and configure remote devices such as layertwo switches via SNMP.The integration of cfengine and scli will be achievedby a loose coupling where a cfengine process may startscli processes in order to collect data or to enforce certainconfiguration policies. A communication channel will be usedbetween cfengine and scli in order to send and receivemessages. The gsnmp engine and the compiler generatedstubs of scli will be used to talk SNMP to remote devices.Thus, scli will serve as a proxy between cfengine andthe remotely managed device. In order to achieve the goalsseveral tasks have to be carried out:

• A communication protocol between cfengine andscli has to be defined. This will define the syntax ofthe messages that are exchanged between cfengine andscli.

• The scli implementation has to be adapted so thatit can handle multiple requests concurrently. The mostconvenient way of doing that will be to enable threadsupport for scli. Hence the scli code has to beadapted so that it is thread-safe and the functions are re-entrant. This will allow that various requests are handledby a separate synchronized threads.

• The cfengine/cfenvd programs will be extendedsuch that they can initiate a pipe to scli and sendcommand and parse the returned output. This task wasdelegated to Mark Burgess (University of Oslo) who isthe creator of cfengine.

III. SCLI PROTOCOL

The scli protocol is a request/response protocol whichprovides a way for an external program like cfengine to

3

read data from managed devices and to configure them byutilizing simple command messages.

A. Protocol Specification

An scli message is either a request or a response message.A request message is sent to an scli interpreter. scli hasone master interpreter and can have many slave interpreters.A request message sent to the master interpreter consists of anumber of tokens. Each token can contain letters, digits andspecial characters excluding single and double quotes and the# character. Tokens themselves can be encapsulated into singleor double quotes. A request sent to a slave interpreter is alwaysprefixed with the name of the slave interpreter. Interpreternames can contain letters, digits as well as dots and colonsand must always start with a letter. All requests sent to anscli interpreter must end with a carriage return and linefeed (CRLF).A response message consists of a message from the masterinterpreter and can be optionally followed by a message froma slave interpreter. A message from the master interpretercontains a response code followed by a short description ofthe code and it can optionally carry data. Whenever a responsecontains data each line of data is prefixed with a responsecode followed by a ”-”. The last line of the master interpreterresponse contains the final response code together with a shortdescription. A message from a slave interpreter might followthe message from the master interpreter (in the case whenthe request message was sent to the slave interpreter). Theresponse from the slave interpreter has the same structureas the response from the master interpreter, however eachresponse line is prefixed with the name of the slave interpreter.All lines in the response end with a carriage return and linefeed (CRLF).The ABNF specification of the scli protocol is shown in theAppendix.

B. Response Codes

Response codes can be classified into one of the followingcategories:

• 1xy - transient messages• 2xy - positive completion• 3xy - generic error codes• 4xy - errors detected before command processing (syntax

errors)• 5xy - communication failure

C. Communication Architecture

An external program establishes a communication channelwith an scli interpreter and sends commands, subsequentlycalled request messages. The scli interpreter generatesSNMP messages based on the data contained in the sclirequest messages and sends them to the respective manageddevice. When SNMP responses are received, the data isconverted in an scli format and sent back to the externalprogram as an scli response message. The communicationbetween cfengine and scli is shown in Figure 2

Fig. 2. Communication between cfengine and scli

An entity which runs scli contains a master interpreterand can optionally contain several slave interpreters. When arequest message is sent via an I/O stream to the entity runningscli, the master interpreter handles the message. If thecommand inside the request message is destined for the masterinterpreter, then it executes the command. If the commandinside the message is destined for a slave interpreter, the masterinterpreter communicates the command to the respective slaveinterpreter via an scli internal communication mechanismafter checking if the respective slave interpreter exists. Ascenario of scli with a master interpreter and three slaveinterpreters each one serving a different SNMP agent is shownin Figure 3.

2

Fig. 3. The scli master interpreter and several slave interpreters

D. Processing of scli request/response messages by aninterpreter

The master scli interpreter waits for request messages(commands). The set of specific commands as well as thesyntax of the commands is specified in the scli documenta-tion [6]. Whenever a slave interpreter is created it registers itsidentifier as a command with the master interpreter. It is theresponsibility of the external program that invokes the scli

4

interpreter to ensure that the syntax of the command invokedis correct.Whenever a command is received by an interpreter it isinitially parsed and the tokens are extracted. If tokenizingfails, then the error code 405 (syntax error in tokenizer)is returned by the respective interpreter. If tokenizing issuccessful then the command is matched against the set ofregistered commands. If there is no such command registered,then the error code 406 (syntax error unknown command) isreturned by the interpreter. At this point if the command isdestined for a slave interpreter of the current interpreter thecommand as well as the arguments are communicated to therespective slave interpreter and the current interpreter returnsa status code of 200 (ok) thus declaring that it is done withprocessing the command. Any error or status codes as well asdata that results from the execution of the command by theslave interpreter will be returned by the slave interpreter. If thecommand requires communication with an SNMP agent thescli interpreter checks whether the interpreter is associatedwith an SNMP agent and whether the output format requestedis supported by the command. If the command requires andSNMP association and there is no SNMP agent associatedwith the scli interpreter executing the command, then theerror code 301 (no association to SNMP peer) is returnedby the respective interpreter. Furthermore, if the output isrequested in XML format and the command does not supportXML output format, then error code 302 (command doesnot support XML) is returned by the interpreter. Otherwise,the interpreter checks the number of arguments provided. Ifa wrong number of arguments is provided, then error code401 (syntax error in number of arguments) is returned. Afterverifying that the correct number of arguments is provided,each argument is checked whether it belongs to the correcttype and range. If the check fails, then one of the syntax errorcodes is returned by the interpreter: error 402 (syntax errorin regexp), error 403 (syntax error in number) or error 404(syntax error in value). If any other syntax error is detected,then error 400 (generic syntax error) is returned. If all checksare successful the respective interpreter starts executing thecommand. If there is a runtime error during the execution ofthe command, then a respective error code is returned by theinterpreter executing the command. The runtime error codesare: error 500 (snmp error return code) or error 501 (snmpname lookup error). If the command has been successfullyexecuted, then the respective scli interpreter returns the data(if any) together with a status code of 200 (ok).The processing of commands by scli interpreters is de-scribed in Figure 4. Some example sessions of using the scliprotocol mode of operation are shown in the Appendix.

2

IV. THREAD SUPPORT FOR SCLI

In this section we describe what the requirements for writinga multithreaded application are and how to write thread-safeand re-entrant code. Several approaches for making C codethread safe and re-entrant after [7] are listed. For section IV-Awe closely follow [7]. Then in section IV-B we discuss how

Fig. 4. Processing of commands by scli interpreter

thread support was enabled for scli. Section IV-C discusseswhat possibilities exist for adapting the scli code so that itis thread-safe and reentrant and section IV-D discusses whichsolution was implemented in practice.

A. Requirements for a multi-threaded application

In single-threaded processes there is only one flow ofcontrol. The code executed by these processes thus need notto be reentrant or thread-safe. In multi-threaded programs,the same functions and the same resources may be accessedconcurrently by several flows of control. To protect resourceintegrity, code written for multi-threaded programs must bereentrant and thread-safe.Reentrance: A reentrant function does not hold static dataover successive calls, nor does it return a pointer to staticdata. All data is provided by the caller of the function. Areentrant function must not call non-reentrant functions.A non-reentrant function can often, but not always, beidentified by its external interface and its usage. For example,the strtok() subroutine is not reentrant, because it holdsthe string to be broken into tokens. It has the followingsignature: char *strtok(char *str, const char

*delim); and it parses the string held by *str into tokensby using *delim as a separator. During the first invocationthe string to be parsed is provided as the *str argument. Insuccessive calls a NULL is provided as a *str argument.The implementation of the function keeps the string to beparsed as static data over multiple calls and thus if severalthreads try to access the function concurrently each threadwill get inconsistent result. The ctime() subroutine isalso not reentrant; it returns a pointer to static data that isoverwritten by each call.Thread-Safety: A thread-safe function protects sharedresources from concurrent access by locks. Thread-safetyconcerns only the implementation of a function and does not

5

affect its external interface.In C, local variables are dynamically allocated on the stack.Therefore, any function that does not use static data orother shared resources is trivially thread-safe. The use ofglobal data is however thread-unsafe. It should be maintainedper thread or encapsulated, so that its access can be serialized.

1) Making a Function Reentrant: In most cases, non-reentrant functions must be replaced by functions with amodified interface to be reentrant. Non-reentrant functionscannot be used by multiple threads. Furthermore, it maybe impossible to make a non-reentrant function thread-safe.Many non-reentrant functions return a pointer to static data.This is the case with many of the functions that implementthe scli interface. An example of such function is thestrtoupper() function, converting a string to uppercase.It could be implemented as shown in Fig 5

/∗ non−r e e n t r a n t f u n c t i o n ∗ /char ∗ s t r t o u p p e r ( char ∗ s t r i n g ){

s t a t i c char b u f f e r [ MAX STRING SIZE ] ;i n t i n d e x ;

f o r ( i n d e x = 0 ; s t r i n g [ i n d e x ] ; i n d e x ++)b u f f e r [ i n d e x ]= t o u p p e r ( s t r i n g [ i n d e x ] ) ;

b u f f e r [ i n d e x ] = 0 ;

re turn b u f f e r ;}

Fig. 5. Non-reentrant version of strtoupper()

There are two approaches that could make this functionthread safe:

• Returning dynamically allocated data. In this case, it willbe the caller’s responsibility to free the storage. Thebenefit is that the interface does not need to be mod-ified. However, backward compatibility is not ensured;existing single-threaded programs using the modifiedfunctions without changes would not free the storage,leading to memory leaks. A reentrant implementation ofstrtoupper() using this approach is shown in Figure6:

• Using caller-provided storage. This method is recom-mended, although the interface needs to be modified.Using this approach the strtoupper() function wouldbecome as shown in Figure 7. Here the caller mustprovide the storage for both input and output strings. Thisapproach was used to make the scli functions reentrant.

Another issue that has to be solved when making a functionreentrant is that no static data should be kept over successivecalls because different threads may successively call the func-tion. If a function needs to maintain some data over successivecalls, such as a working buffer or a pointer, this data should beprovided by the caller. An example of non-reentrant function

/∗ r e e n t r a n t f u n c t i o n ( a poor s o l u t i o n ) ∗ /char ∗ s t r t o u p p e r ( char ∗ s t r i n g ){

char ∗ b u f f e r ;i n t i n d e x ;

/∗ e r r o r−c h e c k i n g s h o u l d be per fo rmed ! ∗ /b u f f e r = m a l loc ( MAX STRING SIZE ) ;

f o r ( i n d e x = 0 ; s t r i n g [ i n d e x ] ; i n d e x ++)b u f f e r [ i n d e x ] = t o u p p e r ( s t r i n g [ i n d e x ] ) ;

b u f f e r [ i n d e x ] = 0 ;

re turn b u f f e r ;}

Fig. 6. Reentrant version of strtoupper() by using dynamically allocatedbuffer

/∗ r e e n t r a n t f u n c t i o n ( a b e t t e r s o l u t i o n ) ∗ /char∗ s t r t o u p p e r r ( char ∗ i n s t r , char ∗ o u t s t r ){

i n t i n d e x ;

f o r ( i n d e x = 0 ; i n s t r [ i n d e x ] ; i n d e x ++)o u t s t r [ i n d e x ] = t o u p p e r ( i n s t r [ i n d e x ] ) ;o u t s t r [ i n d e x ] = 0

re turn o u t s t r ;}

Fig. 7. Reentrant version of strtoupper() using buffer provided fromthe caller

in this aspect is a function that returns the successive lowercasecharacters of a string. The string is provided only on the firstcall, as with the strtok subroutine. The function returns 0 whenit reaches the end of the string. A non-reentrant version of thisfunction is shown in Figure 8

This function is not reentrant because the buffer and theindex variable keep the state over successive calls of thefunction. Thus, if several threads access this function theywill get inconsistent result. To make the function reentrant,the static data, the index variable, needs to be maintainedby the caller. The reentrant version of the function could beimplemented as in the fragment in Figure 9.

The interface of the function changed and so did its usage.The caller must provide the string on each call and mustinitialize the index to 0 before the first call.

2) Making a Function Thread-Safe: In multi-threaded pro-grams, all functions called by multiple threads must be thread-safe. However, there is a workaround for using thread unsafesubroutines in multi-threaded programs. Non-reentrant func-tions usually are thread-unsafe, but making them reentrantoften makes them thread-safe.In order to make a function thread-safe shared resources have

6

/∗ non−r e e n t r a n t f u n c t i o n ∗ /char l o w e r c a s e c ( char ∗ s t r i n g ){

s t a t i c char ∗ b u f f e r ;s t a t i c i n t i n d e x ;char c = 0 ;

/∗ s t o r e s t h e s t r i n g on f i r s t c a l l ∗ /i f ( s t r i n g != NULL) {

b u f f e r = s t r i n g ;i n d e x = 0 ;}

/∗ s e a r c h e s a l o w e r c a s e c h a r a c t e r ∗ /f o r ( ; c = b u f f e r [ i n d e x ] ; i n d e x ++) {

i f ( i s l o w e r ( c ) ) {i n d e x ++;break ;

}}re turn c ;

}

Fig. 8. Non-Reentrant version of lowercase_c() using static buffer andstatic counter

/∗ r e e n t r a n t f u n c t i o n ∗ /charl o w e r c a s e c r ( char ∗ s t r i n g , i n t ∗ p index ){

char c = 0 ;

/∗ no i n i t i a l i z a t i o n −∗ t h e c a l l e r s h o u l d have done i t∗ /

/∗ s e a r c h e s a l o w e r c a s e c h a r a c t e r ∗ /f o r ( ; c = s t r i n g [∗ p index ] ; ( ∗ p index )++){

i f ( i s l o w e r ( c ) ) {(∗ p index ) + + ;break ;

}}re turn c ;

}

Fig. 9. Reentrant version of lowercase_c() - no static variables

to be locked. Functions that use static data or any other sharedresources, such as files or terminals, must serialize the accessto these resources by locks in order to be thread-safe. Anexample of a thread-unsafe function is shown in Figure 10

To be thread-safe, the static variable counter needs to beprotected by a static lock, as in the following example onFigure 11.

In a multi-threaded application program using the threads li-brary, mutexes should be used for serializing shared resources.

/∗ t h read−u n s a f e f u n c t i o n ∗ /i n t i n c r e m e n t c o u n t e r ( ){

s t a t i c i n t c o u n t e r = 0 ;

c o u n t e r ++;re turn c o u n t e r ;

}

Fig. 10. A thread unsafe function

/∗ pseudo−code th read−s a f e f u n c t i o n ∗ /i n t i n c r e m e n t c o u n t e r ( ) ;{

s t a t i c i n t c o u n t e r = 0 ;s t a t i c l o c k t y p e c o u n t e r l o c k =LOCK INIT ;

l o c k ( c o u n t e r l o c k ) ;c o u n t e r ++;un lo ck ( c o u n t e r l o c k ) ;re turn c o u n t e r ;

}

Fig. 11. A thread safe function

B. Thread Support for scli

Before the integration work started all scli interpretersused to run in a single thread. Communication between themaster interpreter and the slave interpreters was accomplishedvia function calls.Thread support was added to the scli code which allowseach interpreter to run in a separate thread. For this purpose thethread API of the glib library was used [8]. During sclistartup g_thread_init() is called in order to initializethe thread support of glib. Then the master interpreter iscreated which runs in the main thread. The thread in whichan interpreter is running is stored in the scli_interpstruct which keeps all data associated with the interpreter.Whenever a user invokes the create scli interp com-mand the function create_scli_interp() is called.This function invokes g_thread_create() which createsa new thread and the pointer for this thread is stored inthe newly created slave interpreter. The new thread sits ina loop in the scli_thread() function which was addedas part of the thread support for scli. The main task ofscli_thread() is to keep polling the command queue forthe slave interpreter via calls to g_async_gueue_pop().Whenever a command for a slave interpeter is sent to scli itis first handled by the master interpeter. The master interpetercalls scli_interp_eval() on this command in order toparse it. Thus, it extracts the name of the slave interpreterthat has to execute the command as well as the argumentsfor the command. The master interpreter creates a messageof type scli_message_t which consists of the name ofthe interpreter, the number of arguments (argc) and a vectorof the arguments (argv). The created message is pushed

7

Fig. 12. Processing of commands by a slave interpreter running in a separatethread

onto the queue of the respective slave interpreter via call tog_async_queue_push(). As mentioned above the threadexecuting the slave interpreter keeps polling this queue andextracts the message via a call to g_async_gueue_pop().The command and the arguments are extracted and the com-mand is executed via a call to scli_eval_argc_argv()which is the core function that handles the execution of com-mands in scli. Before the integration work this function usedto be called inside the thread in which the master interpreteris running and thus blocking scli until the execution of thecommand is complete. In the multi-threaded version of sclithe communication between the master interpreter thread andthe slave interpreter threads is performed via passing messagesof type scli_message_t. The communication involvedin processing commands by a slave interpreter is shown inFigure 12. The result is returned via the result field of thescli_interp struct.

When the scli program has to be terminated the masterinterpreter iterates over all slaves and pushes the commandexit onto their queues. This terminates the slave interpretersafter which the slave interpreter threads join the master inter-preter thread.In a multithreaded model each interpreter runs in its ownthread and the threads are synchronized. All messages willagain be initially handled by the master interpreter. An externalprogram using scli in order to communicate with a manageddevice will be able to communicate with several interpreterssimultaneously. The multithreaded model will allow severalrequest/response messages to be handled by scli simultane-ously. Response messages that come from different interpreterscan be mixed when sent over the communication channelbetween the external program and the scli master interpreter.Since responses coming from slave interpreters are alwaysprefixed with the interpreter identifier they can be easilymatched with request messages when they are received bythe external program. Responses coming from the masterinterpreter will always follow the order of the correspondingrequest messages because all request messages are processedby the master interpreter serially and the response is returnedto the external program before a command is forwarded tothe respective slave interpreter (if command is addressed to aslave interpreter).

C. An Approach to Make the scli Code Thread-Safe andReentrant

Based on the various techniques studied in [7] a top-downapproach for making the scli code thread safe and reentrantwas taken. It is based on the scli architecture shown inFigure 1 and consists of the following steps:

1. Introduce a global lock in the scli code. This willenable multithread support for scli but will slow downthe performance significantly since only one thread willbe able to execute at a time. This solution is similar tothe previous status of operation of scli and is a fastand simple hack.

2. Modify the signatures of reentrant and thread-unsafefunctions from the scli command implementations byusing the recommendations given in IV-A. Any staticbuffers should be eliminated and replaced with buffersprovided by the caller. No function should return a pointerto a static buffer. The scli command implementationsare based on the compiler generated stubs and procedureswhich are not thread safe. Therefore, after making thescli command implementations thread-safe a lock willbe introduced at the entry point of the stubs and the MIBprocedures. The global lock introduced in step 1 can beremoved.

3. Modify the smidump compiler so that it generates reen-trant and thread-safe stubs and procedures. Introduce alock at the entry point of the gsnmp library which isnot thread-safe. At this point the lock in the stubs andprocedures can be removed.

4. Modify the gsnmp library so that it is thread-safe andreentrant. Remove all locks. The gsnmp library based onthe glib library which is thread-safe. Therefore, at thispoint all scli code will be thread-safe.

D. Implemented Solutions

Initially we implemented the solution from point 1of section IV-C by introducing a global lock. Thefunction scli_eval_cmd() was chosen as anentry point to the scli commands implementation.A G_STATIC_MUTEX was used as a lock andg_static_mutex_lock(&scli_global_lock)was called at the beginning of this function andg_static_mutex_unlock(&scli_global_lock)was called just before the function returned. This constitutesa quick and fast hack which however slows down theperformance since only one thread can execute at a time.During the second phase of the implementation the sclicommands implementation functions were modified inorder to make them reentrant. This step involved mainlyadapting the formatting interface functions of scli manyof which either used or returned static char buffers in theirimplementation. The signatures of those functions weremodified after the approach shown in section IV-A - namelythe implementation was rewritten so that a reference toa result GString buffer was provided by the caller andthe return type was changed from char* to gboolean.

8

The whole formatting API was also changed consistentlyso that each formatting function contains a reference to aresulting GString buffer in its signature. The modificationof the formatting API required modifying the code fragmentswhere the formatting functions are called. This code had tobe rewritten in order to reflect the new API and deal withallocation and deallocation of return GString buffers. Asan example of a function which was modified we show thefmt_seconds() which takes the number of seconds thathave passed and returns a formatted string in the formathour:minute:seconds. The initial implementation of thefunction is shown in Figure 13.

char c o n s t ∗f m t s e c o n d s ( g u i n t 3 2 s e c o n d s ){

s t a t i c char b u f f e r [ 8 0 ] ;u i n t 3 2 sec , min , hour ;

s e c = ( s e c o n d s ) % 6 0 ;min = ( s e c o n d s / 60) % 6 0 ;hour = ( s e c o n d s / 60 / 6 0 ) ;

g s n p r i n t f ( b u f f e r , s i z e o f ( b u f f e r ) ,”%4d:%02d:%02d ” , hour , min , s e c ) ;

re turn b u f f e r ;}

Fig. 13. The non-reentrant version of fmt_seconds()

In this version the fmt_seconds() function uses a staticbuffer to write the formatted string which is later returned. Ifseveral threads enter this function then an inconsistent stateof the buffer will result. Therefore, the function was rewrittenafter the approaches discussed in section IV-A. The reentrantversion is shown in Figure 14.

g b o o l e a nf m t s e c o n d s ( G S t r i n g ∗ s , g u i n t 3 2 s e c o n d s ){

g u i n t 3 2 sec , min , hour ;

s e c = ( s e c o n d s ) % 6 0 ;min = ( s e c o n d s / 60) % 6 0 ;hour = ( s e c o n d s / 60 / 6 0 ) ;

g s t r i n g p r i n t f ( s , ”%4d:%02d:%02d ” ,hour , min , s e c ) ;

re turn TRUE;}

Fig. 14. The reentrant version of fmt_seconds()

In this version the function writes the string to a GStringbuffer which is allocated by the caller. In Figure 15 we showthe old and the new way of calling the function.

/ / c a l l t o t h e o l d f u n c t i o nchar ∗e ;e = f m t s e c o n d s ( u s e c s ) ;

/ / c a l l t o t h e new f u n c t i o nchar ∗eG S t r i n g g= g s t r i n g n e w (NULL ) ;( void ) f m t s e c o n d s ( g , u s e c s ) ;e=g−>s t r ;

/∗ use e , make a copy i f n e c e s s a r y ∗ /

g s t r i n g f r e e ( g , 1 ) ;

Fig. 15. Calling the reentrant and the non-reentrant version offmt_seconds()

In the modified version the caller function has to take careof allocating and deallocating the return GString buffer.During the next step of the implementation work the stubs andthe MIB procedures were made reentrant. For this purpose thesmidump compiler was modified so that it does not generatestatic buffers in the implementation of the stubs. This involvedmodifying several printing functions inside the compiler afterthe approach mentioned in section IV-A.The final step of the implementation the global lock wasremoved from the scli command implementations and a lockwas introduced at the entry of the gsnmp library. Thus, wereached at step 3 from the top-down approach specified insection IV-C.In addition to making the functions in the formatting API reen-trant and thread-safe a XML formatting API was implemented.These functions mainly deal with formatting data and insertingnodes in the XML tree when XML output is requested. Thenewly developed API utilizes many of the functions of theformatting API and thus makes handling of XML data easier.Since the formatting API was used during the implementationthe XML API is reentrant and thread safe.

V. RELATED WORK

In this section we present a short literature survey ofalgorithms, techniques and tools that can be used fordetection of data races in programming code. All of thepresented algorithms are designed to detect races in thepresence of dynamically allocated shared data. Therefore,the tools that are presented are runtime analyzers and thechecks for data races are performed at runtime. Attemptshave been made to create race detection systems that performthe check at compile time and work in the presence ofdynamically allocated static data. Examples of such arethe Sun’s lock lint [9] and the Extended Static Checkerfor Modula-3 [10]. However, these approaches have beenreferred to as problematic in [11] since they require staticalreasoning about the program semantics.In section V-A we describe the Lamport’s happens beforealgorithm which is briefly outlined in [11]. Then in sectionV-B we present the Lockset algorithm for race detection and

9

in section [11] we discuss how it has been improved duringthe implementation of Eraser tool [11]. In section 19 wepresent another tool for data race detection, VisualThreads[12], which further improves the algorithm used by Eraser.

A. Happens Before Algorithm

The happens before algorithm introduces a partial order onall events of all threads in a concurrent execution. Within anysingle thread, events are ordered in the order in which theyoccurred. Between threads, events are ordered according to theproperties of the synchronization objects they access. If onethread accesses a synchronization object, and the next accessto the object is by a different thread, then the first access isdefined to happen before the second. For example, Figure16 shows one possible ordering of two threads executing thesame code segment. The three program statements executedby Thread 1 are ordered by happens-before because they areexecuted sequentially in the same thread. The lock of mu byThread 2 is ordered by happens-before with the unlock ofmu by Thread 1 because a lock cannot be acquired before itsprevious owner has released it. Finally, the three statementsexecuted by Thread 2 are ordered by happens-before becausethey are executed sequentially within that thread.

Fig. 16. Ordering of events based on the happens before algorithm

Happens before signals a warning about a possible data racewhen accesses to shared variables by different threads are notordered by the partial order defined. If those accessed are notordered by this partial order then in another execution of theprogram in which the slower thread ran faster and the fasterthread ran slower the accesses to the shared data could havehappened simultaneously and a race would occur.One disadvantage of happens before is that it might miss somedata races. One such example is shown in Figure 17.

While there is a potential data race on the unprotectedaccesses to y, it will not be detected in the execution shownin the figure, because Thread 1 holds the lock before Thread

Fig. 17. An example where happens before algorithm will miss a data race

2, and so the accesses to y are ordered in this interleaving byhappens-before. A tool based on happens-before would detectthe error only if the scheduler produced an interleaving inwhich the fragment of code for Thread 2 occurred before thefragment of code for Thread 1.

B. Lockset Algorithm

The Lockset algorithm monitors whether there is a consis-tent locking discipline for each shared variable. It ensures thateach shared variable is protected by some lock when it isaccessed for reading and writing and that the lock is held byany thread when it accesses the variable. Since there is noway of knowing which locks are intended to protect whichvariables, this must be inferred from the execution history.For each shared variable v, a set C(v) of candidate locks for vis maintained. This set contains those locks that have protectedv for the computation so far. That is, a lock l is in C(v) if, inthe computation up to that point, every thread that has accessedv was holding l at the moment of the access. When a newvariable v is initialized, its candidate set C(v) is considered tohold all possible locks. When the variable is accessed, C(v) isupdated with the intersection of C(v) and the set of locks heldby the current thread. This process, called lockset refinement,ensures that any lock that consistently protects v is containedin C(v). If some lock l consistently protects v, it will remain inC(v) as C(v) is refined. If C(v) becomes empty this indicatesthat there is no lock that consistently protects v. The followingis a pseudocode for the Lockset algorithm.

Let locks held(t)← set of locks held by thread tfor all v do

C(v)← set of all locksend forfor each access to v by thread t do

10

C(v)← C(v) ∩ locks held(t)end forif C(v) = ∅ then

issue a warningend if

C. Eraser Tool

The Eraser tool [11] uses the Lockset algorithm and intro-duces two additional optimizations that prevent false alarmsin the following cases:

1. Initialization - Shared variables are frequently initializedwithout holding a lock. Thus, an alarm for data raceshould not be produced during the initialization of avariable although the thread that does that does not hold alock. For this reason the refinement of a locations candi-date lock set is delayed until after it has been initialized.Eraser considers a shared variable to be initialized when itis first accessed by a second thread. As long as a variablehas been accessed by a single thread only, reads andwrites have no effect on the candidate set.

2. Read-Shared Data - Some shared variables are writtenduring initialization only and are read-only thereafter.These can be safely accessed without locks. Thus, inorder to avoid reporting alarms for data races in suchcases races are reported only after an initialized variablehas become write-shared by more than one thread. Basedon 1. and 2. the state transition diagram shown in Figure18 has been developed for the Erase algorithm. When

Fig. 18. State transition diagram for Eraser

a variable is first allocated, it is set to the Virgin state,indicating that the data are new and have not yet beenreferenced by any thread. Once the data are accessed,it enters the Exclusive state, signifying that it has beenaccessed, but by one thread only. In this state, subsequentreads and writes by the same thread do not change thevariable’s state and do not update C(v). This addressesthe initialization issue, since the first thread can initializethe variable without causing C(v) to be refined. When

and if another thread accesses the variable, then the statechanges. A read access changes the state to Shared. Inthe Shared state, C(v) is updated, but data races are notreported, even if C(v) becomes empty. This takes careof the read-shared data issue, since multiple threads canread a variable without causing a race to be reported.A write access from a new thread changes the statefrom Exclusive or Shared to the Shared-Modified state,in which C(v) is updated and races are reported, just asdescribed in the original, simple version of the algorithm.

3. Read-Write Locks - Read-write locks allow multiplereaders to access a shared variable, but allow onlya single writer to do so. Thus, an alarm should bereported if a reader and a writer try to access shared datasimultaneously. However, no alarm should be reported ifmultiple readers access the shared data at the same time.In order to avoid reporting such false alarms Eraserrequires that for each variable v, some lock m protectsv, meaning m is held in write mode for every write ofv, and m is held in some mode (read or write) for everyread of v. The state transition diagram in Figure 18 isstill used but the check in the Shared-Modified state ifperformed in the following way:

Let locks held(t)← the set of locks held in any modeby thread tLet write locks held(t) ← the set of locks held inwrite mode by thread tfor all v do

C(v)← set of all locksend forfor each read of v by thread t do

C(v)← C(v) ∩ locks held(t)if C(v) = ∅ then

issue a warningend if

end forfor each write of v by thread t do

C(v)← C(v) ∩ write locks held(t)if C(v) = ∅ then

issue a warningend if

end for

D. Visual Threads Tool

The Visual Threads tool [12] is another tool that utilizes therefined Lockset algorithm from Eraser. However, the Eraseralgorithm is further extended in order to reduce the number offalse alarms. The motivation is that not all data shared betweenthreads without the use of a lock might lead to data race.There are types of indirect synchronization that may eliminatethe need for explicit locks. In particular, a very commonparadigm is for the initial thread to allocate and initialize somedata, create worker threads to perform some transformationson this data, and then after the threads have all completed,display the result. The original Eraser algorithm was extendedto reduce the number of false reports due to this type of

11

implicit synchronization, by introducing the concept of threadsegment. A thread segment delineates time in addition to justthread identity. By utilizing the thread segment identifiers inthe algorithm instead of simply using the thread identity, onecan distinguish accesses that cannot happen concurrently. Nothread segment spans beyond the creation of a new thread, ora join. When a parent thread creates a new child thread, theparent’s thread segment id is updated, and the child’s threadsegment id is assigned. Similarly, after a join operation, theparent’s thread segment id is updated (the child no longerexists). Each running thread is represented as a leaf in thegraph. Figure 19 shows how thread segments are assigned overtime. These thread segments are kept in a directed graph that

Fig. 19. Thread Segment Assignment in Visual Threads

indicates which thread segments cannot occur concurrent witheach other. From Figure 19 we can observe that TS1 happensbefore TS2 − TS7, TS3 happens before TS5 − TS7, TS2happens before TS6 and TS7, and so forth. The modificationof the Eraser algorithm to incorporate thread segments is asfollows:

• When a variable v is in Exclusive state, associate it withthe thread segment id of the current thread instead of thethread id

• If variable v is in Exclusive state and associated withthread segment TSi, and is being touched by TSj (wherei 6= j), and TSi happens before TSj in the graph, theninstead of moving the variable to one of the shared states,associate v with TSj . The state for v remains Exclusive.

This extension allows exclusive-use data to be ”passed” fromparent thread to child thread (and back) as long as the threadsegments involved in the access cannot be concurrent dueto the known points of thread creation and termination (asdetermined by the thread segment graph). If the two threadsegments are potentially concurrent, then the data is markedas either in Shared or in Shared-Modified state as appropriateand the standard checking is applied.

E. The Helgrind Tool

Helgrind [13] is a valgrind tool for detecting data races inC and C++ programs that use the Pthreads library. It utilizesthe Lockset algorithm together with the refinements introducedin Eraser and Visual Threads

VI. CONCLUSION

During this project significant progress was achieved inthe integration of cfengine and scli. A communicationprotocol was specified in ABNF and multithread supportwas implemented for scli by using the glib library.Furthermore, several solutions were discussed for makingthe scli code thread-safe and reentrant. A top-downapproach was followed during the implementation work.We started from a quick and easy solution with very poorperformance and gradually drilled down by making thecommand implementations and the compiler generated stubsthread-safe. At the end the lock was placed at the entry pointof the gsnmp library.Further work in the integration should consider making thegsnmp library reentrant and thread safe. At the end of theproject work we discovered that in addition to the majorcomponents listed in the scli software architecture therealso exist some caching mechanisms which are not threadsafe. Thus, making the caches reentrant and thread safeis another issue that should be taken into consideration.It is recommended to make a further check that passingresults from the slave interpreters to the master interpreters isthread safe. In order to complete the integration process thecfengine code has to be extended so that cfengine canopen a communication channel to scli, send a request andparse a response.As part of the project a number of tools and algorithms thatdetect data races were surveyed. We concluded that most ofthese tools nowadays are based on the Lockset algorithm formonitoring consistent locking policy. Therefore, the sclicode could be tested using some of the appropriate tools.The integration of cfengine and scli will foster theusage and development of open source software in networkmanagement. Furthermore, it will give new insights in thedirection of automated configuration management to bothnetwork managers and system administrators. IntegratingSNMP access for cfengine enables the latter one to enforcepolicies on remote agents and to collect data for anomalydetection from remote devices.

REFERENCES

[1] M. Burgess, “A site configuration engine,” j-COMP-SYS, vol. 8, no. 3,pp. 309–337, Summer 1995.

[2] J. Schoenwaelder, “Specific simple network management tools,” in Proc.LISA 2001, December 2001, pp. 109–119.

[3] O. F. R. S. E. L. G. P. J. M. Burgess, V. Cridlig, “Emanics specificationof the static and dynamic content for dissemination environment,”eMANICS WP6 Interim Report.

[4] M. Burgess, “Two dimensional time-series for anomaly detection andregulation in adaptive systems,” in DSOM ’02: Proceedings of the 13thIFIP/IEEE International Workshop on Distributed Systems: Operationsand Management. London, UK: Springer-Verlag, 2002, pp. 169–180.

[5] EMANICS project web page, http://www.emanics.org.[6] scli project page, https://trac.eecs.iu-bremen.de/projects/scli/.[7] General Programming Concepts: Writing and Debugging

Programs. IBM Corporation, September 1999, available athttp://www.unet.univie.ac.at/aix/aixprggd/genprogc/toc.htm.

[8] GLib reference manual, http://developer.gnome.org/doc/API/2.2/glib/.[9] LockLint Users Guide, http://developers.sun.com.

12

[10] D. L. Detlefs, K. R. M. Leino, G. Nelson, and J. B. Saxe, “Extendedstatic checking,” Palo Alto, USA, Tech. Rep. #159, 1998. [Online].Available: citeseer.ist.psu.edu/detlefs98extended.html

[11] S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson,“Eraser: A dynamic data race detector for multithreaded programs,”ACM Transactions on Computer Systems, vol. 15, no. 4, pp. 391–411,1997.

[12] J. J. Harrow, “Runtime checking of multithreaded applications withvisual threads,” in SPIN, 2000, pp. 331–342. [Online]. Available:citeseer.ist.psu.edu/harrow00runtime.html

[13] Helgrind:a valgrind data race detector,http://valgrind.org/docs/manual/hg-manual.html.

13

MESSAGE = REQUEST / RESPONSE

REQUEST = MREQUEST / SREQUESTMREQUEST = TOKEN *(WSP TOKEN) CR LFSREQUEST = INTERP WSP MREQUESTTOKEN = DQTOKEN / SQTOKEN / NQTOKENDQTOKEN = DQUOTE NQTOKEN DQUOTESQTOKEN = SQUOTE NQTOKEN SQUOTENQTOKEN = 1*REQCHAR

RESPONSE = MRESPONSE WSP [SRESPONSE]MRESPONSE = *(CODE "-" DATA CR LF) CODE WSP DATA CR LFSRESPONSE = *(INTERP WSP CODE "-" DATA CR LF) INTERP WSP CODE WSP DATA CR LFDATA = *RESCHAR

CODE = "100" ;arbitrary messageCODE =/ "200" ;normal return codeCODE =/ "201" ;return and exit the command loopCODE =/ "300" ;generic error return codeCODE =/ "301" ;no association to SNMP peerCODE =/ "302" ;command does not support XMLCODE =/ "400" ;generic syntax errorCODE =/ "401" ;syntax error in number of argsCODE =/ "402" ;syntax error in regexpCODE =/ "403" ;syntax error in numberCODE =/ "404" ;syntax error in valueCODE =/ "405" ;syntax error in tokenizerCODE =/ "406" ;syntax error unknown commandCODE =/ "500" ;snmp error return codeCODE =/ "501" ;snmp name lookup error

INTERP = ALPHA *(ALPHA / DIGIT / %x2E / %x3A);Combination of letters, digits, dots and colon;starting with a letter

REQCHAR = WSP / %x21 / %x24-26 / %x28-7E

Fig. 20. scli protocol ABNF definition

14

S: 100 scli version 0.3.1 (c) 2001-2007 Juergen SchoenwaelderC: set scli mode protocolS: 200 ok; scli 0.3.1 readyC: show system infoS: 301 no association to a remote SNMP agentC: open curveS: 200 ok; scli 0.3.1 readyC: show system infoS: 200-Name: curveS: 200-Agent: snmp://public@192.168.1.33:161//S: 200-Description: ProCurve J4903A Switch 2824, revision I.08.105, ROMS: 200- I.08.07 (/sw/code/build/mako(ts_08_5))S: 200-Contact: <j.schoenwaelder@iu-bremen.de>S: 200-Location: Jacobs University Bremen, GermanyS: 200-Vendor: Hewlett Packard <http://www.hp.com/>S: 200-Services: datalink transport applicationS: 200-Agent-Boot-Time: 2007-01-29 00:49:09 +01:00S: 200-Interfaces: 39S: 200-Interface Swap: 2007-03-28 11:39:15 +02:00S: 200-Entity Swap: 2007-01-29 00:49:09 +01:00S: 200-Bridge Ports: 24S: 200-Bridge Type: transparent-onlyS: 200 ok; scli 0.3.1 readyC: show system info fooS: 401 wrong number of arguments: should be ‘show system info’C: set scli timeout 100S: 200 ok; scli 0.3.1 readyC: set scli timeout fooS: 403 invalid number "foo"C: set scli "timeout" 100S: 200 ok; scli 0.3.1 readyC: set scli "timeout 50S: 405 failed to tokenize input: Text ended before matching quote was found for ". (The text was ’set scli "timeout 50’)C: set system name fooS: 500 noAccess @ varbind 0C: fooS: 406 invalid command name "foo"C: exitS: 201 ok; scli 0.3.1 exiting

Fig. 21. Example Communication Using scli protocol mode

15

S: 100 scli version 0.3.1 (c) 2001-2007 Juergen SchoenwaelderC: create scli interp fooS: 200 ok; scli 0.3.1 readyC: foo show scli infoS: foo 200-Version: 0.3.1S: foo 200-Epoch: 1175195954S: foo 200-Format: scliS: foo 200-Interactive: falseS: foo 200-Delay: 0 secondsS: foo 200-Regex: extendedS: foo 200-Debugging:S: foo 200-Rows: 32S: foo 200-Columns: 80S: foo 200-Pager: scliS: foo 200 ok; scli 0.3.1 readyS: 200 ok; scli 0.3.1 readyC: foo gagaS: foo 406 invalid command name "gaga"S: 200 ok; scli 0.3.1 readyC: foo set scli timeout 400S: foo 200 ok; scli 0.3.1 readyS: 200 ok; scli 0.3.1 readyC: foo set scli timeout oopsS: foo 403 invalid number "oops"S: 200 ok; scli 0.3.1 readyC: exitS: 201 ok; scli 0.3.1 exiting

Fig. 22. Example Communication Using scli protocol mode and a slave interpreter