Transaction-based Anomaly Detection

Roland Buschkes, Mark BorningAachen University of Technology - Department of Computer Science

Informatik 4 (Communication Systems)D-52056 Aachen, Germany

froland, borningg@i4.informatik.rwth-aachen.de

Dogan Kesdogano.tel.o communications GmbH & Co

Dept. Enterprise SecurityD-51063 Koln

Dogan.Kesdogan@o-tel-o.de

January 22, 1999

Abstract

The increasing complexity of tele and data communi-cation networks yields new demands concerning net-work security. Especially the task of detecting, repuls-ing and preventing abuse both by in- and outsiders be-comes more and more difficult. This paper deals with arelatively new technique that appears to be suitable forsolving these issues, i.e. anomaly detection based onthe specification of transactions. The traditional trans-action and serialization concepts are discussed, before anew model of anomaly detection based on the conceptof transactions is introduced. Applying this model toknown attacks gives a first insight concerning the feasi-bility of our approach.

1 Introduction

Modern tele and data communication networks providetoday’s users with all kinds of services. In the future, thevariety of available services will increase, offering theusers any service anytime and anywhere. But with thegrowing range of available services also the complex-ity of the underlying networks increases. Therefore, itbecomes more and more difficult to detect, repulse andprevent abuse by both in- and outsiders. Classical secu-rity mechanisms, i.e. authentication and encryption, andinfrastructure components like firewalls cannot provideperfect security. Thereforeintrusion detection systems

(IDS) have been introduced as a third line of defense.

The techniques classically applied within IDS can besubdivided into two main categories [22]:

� Misuse Detection, and

� Anomaly Detection.

Misuse detection (see e.g. [8, 12, 14]) tries to detectpatterns of known attacks within the audit stream of asystem, i.e. it identifies attacks directly. The main disad-vantage of this approach is that the underlying databaseof attack patterns must be kept up-to-date and consis-tent. Because misuse detection techniques depend onthe knowledge of attack patterns, they cannot detect newattacks.

Misuse detection, by explicitly describing the sequenceof actions an attacker takes, is based on the specificationof theundesirableor negative behaviorof users and pro-cesses. The dual approach is the specification of the de-sired orpositive behaviorof users and processes. Basedon this normative specification of positive behavior at-tacks are identified by observing derivations from thenorm. Therefore this technique is calledAnomaly De-tection.

The main difficulty for anomaly detection techniquesis to determine the positive behavior. Two general ap-proaches exist:

Figure 1: Anomaly Detection Approaches.

1. Learning of user and process behavior, and

2. Specification of user and process behavior.

The first approach is often based on statistical methodslike the calculation of means, variations and multivariatestatistics [10]. Other methods use learning algorithmslike e.g. neural networks or Bayesian classifiers [3].This approach is particular popular for the profiling ofusers.

The second approach, specification-based anomaly de-tection, was first proposed by [11]. In this paper wedescribe a new model called transaction-based anomalydetection and its application in communication net-works. The transaction-based approach is in generalsimilar to the specification-based approach as it formallydescribes positive behavior. In contrast to the speci-fication based-approach it specifies the desired actionsand sequence of actions by the definition of transactions.This explicit definition of allowed transactions becomesan integral part of the local security policy.

Fig. 1 summarizes the different approaches concerninganomaly detection and their possible application. On theone hand, the expected behavior of the network protocolstack is well defined. This allows the application of non-intelligent techniques to monitor it. On the other hand,users are less predictable in their behavior and thereforeintelligent techniques must be applied. For processesit would be theoretically possible to define their behav-ior formally, but often enough their complexity makesthis approach impossible. Nonetheless, from our pointof view non-intelligent monitoring techniques should be

preferred over intelligent ones whenever possible. Al-though intelligent techniques can improve the securityof a system, they rarely give the security administrator aclear picture about the level of security they can guaran-tee. By way of contrast non-intelligent techniques likee.g. specification-based approaches naturally extend thegeneral security policy and thereby clearly define theirguaranteed level of security.

The paper is subdivided into six main parts. After thisintroduction we first define the requirements for a for-mal specification of positive behavior. Thereafter weintroduce our transaction-based approach. We do thisby shortly reviewing the general transaction model asknown from database systems and adapting it to the IDSscenario. In section 4 we give examples for the appli-cation of our model. After comparing the transaction-based concept with other approaches in section 5, we fi-nally draw some conclusions and close with an outlookon future work.

2 Specification Requirements

Any misuse or anomaly detection component should ad-here to some basic design principles. These design prin-ciples also define the common criteria for the compari-son of individual techniques. We will therefore shortlyreview them here.

Obviously the specification of positive behavior de-mands some formalism. Although formalisms are of-ten very expressive and powerful, they also involve thedanger of being too complicated. Complicated mecha-nisms tend to introduce errors and errors open new secu-rity holes.

An appropriate formalism should therefore at least pro-vide the following properties:

1. ease of specification,

2. ease of monitoring,

3. completeness (no false negatives),

4. soundness (no false positives),

5. efficiency, and

6. universal validity.

As the first five desired properties in this list are well-known, we will here only shortly elaborate the require-ment of universal validity to motivate our approach. Cur-rent IDS still mainly focus on the monitoring of oper-ation system environments like UNIX or NT. Only re-cently the first papers dealing with the monitoring of net-work infrastructures [2, 4, 15, 17, 20, 21] have been pub-lished. But future communication platforms like mid-dleware (CORBA, DCOM), electronic commerce, intel-ligent network or mobile phone (UMTS) architectureswill also be potentially vulnerable to attacks.

Misuse detection techniques can obviously not be stud-ied yet in this context, because the systems are not inplace and therefore no attacks are known. Anomaly de-tection techniques have the advantage that they can beapplied right from the start of such systems. And, asan additional advantage, they can provide the adminis-trators not only with information concerning potentialattacks, but with general information concerning sys-tem activity, faults and performance. Following thisapproach anomaly detection components can build anessential part of the general network management plat-forms.

Two general observations hold - from our point of view- for present and future communication platforms:

� Communication protocols can generally be speci-fied as state transition systems.

� The protocols can be considered to define validtransactions.

Therefore our approach tries to maximize its universalvalidity by falling back on the transaction model, whichis typically for many processes and especially commu-nication processes.

3 Transaction-based approach

Transactions are a well known concept, originating fromthe field of database managing systems(DBMS) andnowadays widely applied in other environments like dis-tributed systems [5]. Before we will discuss their ap-plication in the context of intrusion detection, we willshortly review the general concept.

Process1 Time Process2

read(x) 12 read(x)

update(x) 34 update(x)

write(x) 56 write(x)

Table 1: Lost update problem.

3.1 The Classical Transaction Concept

Transactions generally consist out of a sequence of readand write operations. If several transactions read andwrite to a database concurrently, the following problemscan arise:

1. Lost Update Problem

2. Dirty Read

3. Phantom Updates

The Lost Update Problem e.g. is depicted in Tab.1. Process1 reads the variable x. Immediately afterprocess1 process2 also reads x. Thereafter both pro-cesses update x and write it back to the database. Ob-viously the update of process1 is lost.

To avoid these problems transactions are considered todescribe atomic operations. The provision of an atomicoperation [5] means that the effect of performing anyoperation on behalf of one client is free from interfer-ence from operations being performed on behalf of otherconcurrent clients; and either an operation must be com-pleted successfully or it must have no effect at all inthe presence of server crashes. The properties of suchatomic transactions are often characterized by the ACIDprinciple [7]:

1. Atomicity: All operations of a transaction must becompleted, i.e. a transaction is treated as a single,indivisible unit.

2. Consistency: A transaction takes the system fromone consistent state to another consistent state.

3. Isolation: Each transaction must be performedwithout interference from other transactions.

4. Durability: After a transaction has completed suc-cessfully all its effects are saved in permanent stor-age.

Operation Return value Comment

BeginTransaction TransId Starts a new transac-tion

EndTransac-tion(TransId)

Commit_ Abort Ends a transaction

Commit(TransId) Commits a transac-tion

Abort(TransId) Aborts a transaction

Table 2: Transaction commands.

Obviously the simple serial execution of transactionspreserves the ACID properties, although it is not ef-ficient. Therefore the task of ascheduleris to max-imize concurrency and to execute transactions inter-locked. The scheduler ensures that the transactions areexecuted in aserially equivalentway. Schedulers usu-ally belong to one of the following categories:

1. Blocking scheduler

2. Non-blocking scheduler

A blocking scheduler applies read/write-locks in orderto serialize transactions. A non-blocking scheduler doesnot use such locks. Instead it coordinates the transac-tions with the help of timestamps or other techniques. Anon-blocking technique of particular interest in the con-text of intrusion detection is the optimistic concurrencycontrol. This technique is based on the assumption thatthe likelihood of conflicts (attacks) is low. Therefore thetransactions simply proceed and at their end the sched-uler checks them for any conflicts.

What actually has to be considered by the schedulerto be a transactions must be specified by the clients.The sequence of operations which should be exe-cuted according to the ACID principle is grouped by aBeginTransaction /EndTransaction pair. Thisbracketing groups several valid commands and defines asingle meta command.

Depending on whether the scheduler is able to executethe transaction without conflicts or not the transaction iscommitted or aborted (see Tab. 2).

3.2 Transaction-based Anomaly Detection

One common argument concerning the difficulty of de-tecting intrusive behavior is that attacks normally con-sist out of single steps, which perform legal operations.

Figure 2: Layered Transaction Model.

These legal steps are generally used to interfere with an-other process also performing legal operations. Consid-ering the victim’s and the attacker’s sequence of opera-tions as transactions an attack will obviously not be suc-cessful, if the ACID properties are guaranteed for thevictim’s transaction.

Although it will probably not be efficient to designtransaction-based operating and network systems, thetransaction concept itself can be used to detect intru-sions. Like an optimistic scheduler checks each trans-action at its end concerning its serially equivalent exe-cution and its adherence to the ACID properties, an IDScan check these properties, too. If the check is not suc-cessful, the transaction is trapped. This is the generalidea of transaction-based intrusion detection.

As we focus on communication systems in this paper,our model is based on the ISO/OSI reference model [6],which is made up of seven protocol layers. The proto-cols of each of these layers can be described bydeter-ministic finite state machines(DFSM) (Fig. 2).

A DFSM A is a 5-tupleA = (Q;�; q0; �; F ) with Qbeing the set of states which can occur during a transac-tion,� being the union of all inputs,q0 being the initialstate,� : Q� � ! Q being the transition function, andF the set of final states. The syntax and semantic of aDFSM are unambiguously determined by the transitionfunction.

The initial state of the DFSM is considered to be a con-sistent state. The DFSM passes through the operations(input events) that form the transaction. The transactionis successful, if the final state is also consistent. To en-sure this the atomicity, consistency and isolation proper-

Figure 3: General architecture.

ties1 have to be checked. The atomicity is checked on thebase of the DFSM. The DFSM defines a languageL(A).By checking whetherw 2 L(A) holds for a trace w ex-tracted from the audit stream the transaction is tested forthe atomicity property.

Thereafter the assertions (consistency property), whichare expressed with the help of first order logic (FOL),are checked. Each DFSM can include assertions. Asser-tions can be valid only for a specific transaction (localassertions) or on a more global level, e.g. for a certainlayer (layer assertions, see Fig. 2).

Finally, and only if necessary, the serially equivalent ex-ecution (isolation) of the transaction is checked.

The overall architecture is shown in Fig. 3 and describedin more detail in the following subsections.

1We do not consider the durability property in detail in this paper.

3.2.1 The Splitter

The audit stream contains all events received from thedifferent sources. As we focus on network protocolsin this paper the audit stream consists of packetsreceived from the network. The task of the splitter isto distribute these single events to the correspondingstate machines. The assignment of events to a statemachine is straightforward as it follows the rules ofthe corresponding communication protocols. A TCPsession e.g. is uniquely identified by the 4-tuple(source address, source port, desti-nation address, address port) , whereasa UDP packet is sufficiently identified by its combi-nation of an IP address with a specific port number,i.e. the socket address. For a fragmented IP packetthe 16-bit identification field in combination with themore fragments bit of the flags field, the sourceaddress, the destination address and the protocol typeallows the splitter to feed the corresponding audit datainto the appropriate state machine.

The splitter is also responsible for the scheduling of theisolation tester. The scheduler monitors the event streamfor any potential conflicts and feeds the isolation testerwith the necessary information.

3.2.2 Template State Machine

As mentioned above the theory of DFSMs is in gen-eral sufficient to represent the protocol state machines.Nonetheless, a major disadvantage of the DFSM theoryis the nonexistence of variables. Therefore, a separatetransition has to exist for each possible value of a vari-able during a protocol run. This blows up the state ma-chines substantially.

To keep the state machines small we introduce the con-cept of aTemplate State Machine(TSM). A TSM rep-resents a protocol in a value independent manner. Theactual instance of the protocol template, i.e. the con-crete DFSM, is derived during run time from the auditdata.

A TSM A is a 6-tuple(Q;�; V; q0; �; F ) with

� Q - set of states which can occur during a transac-tion,

� � - the union of all inputs,

� V - set of variables,

� q0 - initial state,

� � - transition function, and

� F - set of final states.

The transition function� is defined as follows:

� : Q� (AExp(�; V ))+�! Q

AExp(�; V ) is defined by

e ::= z j x j e1 + e2j e1 � e2 2 AExp(�; V )

with z 2 �; and

x 2 V:

The semantic of the TSM, i.e. the derivation relation),is defined by:

) ��Q��+ � (V ! �?)

�� (Q� (V ! �?))

with

(q; (z1; : : :); �)) (q0; �0), ifex. (e1; : : :) 2 (AExp(�; V ))+ with�(q; (e1; : : :)) = q0 and8i : [(�(ei) = zi ^ �0 = �)_(�(ei = ? ^ ei 2 V ^ �0 = �[ei=zi])].

In contrast to the DFSM, the semantic of a TSM is notdetermined by the transition function solely, but also bythe instantiation of the variables.

The TSM mechanism serves several purposes:

� It enables the formal and compact specification of aprotocol, which can dynamically be fed into a mon-itor.

� It prevents state explosions.

� It can be used for the graphical specification of pro-tocols.

� It can be used for a consistency and correctnesscheck of the protocol specification.

In this paper we concentrate on the first two points.

3.2.3 Consistency Tester

After the atomicity of a transaction has been checkedwith the help of a TSM, the next step is to ensure thatthe transaction leaves the system behind in a consistentstate. The consistent state itself is defined by so-calledassertions. In general we distinguish two kinds of asser-tions, local assertions and layer assertions.

To express the assertions we use first order logic (FOL)with the restriction that the negation is only allowed foratomic formulas (FOL+)2.

Local Assertions The local assertions are related to aspecific TSM and are checked during the execution oftransitions. Therefore we have to add Boolean expres-sions to our TSM concept.

The transition function is adapted in the following way:

� : Q� (AExp(�; V ))+ � BExp�! Q

BExp is defined by

b ::= true j false j e

j e1 = e2 j e1 < e2 j e1 6= e2

j b1 ^ b2 j b1 _ b2

The semantic of a TSM changes in the following way:

) ��Q��+ � (V ! �?)

�� (Q� (V ! �?))

with

(q; (z1; : : :); �)) (q0; �0), ifex. (e1; : : :) 2 (AExp(�; V ))+, b 2 Bexp(�; V ) with�(q; (e1; : : :)) = q0 and�(b) = true and8i : [(�(ei) = zi ^ �0 = �)_(�(ei) = ? ^ ei 2 V ^ �0 = �[ei=zi])].

Obviously we do not need the whole expressiveness ofFOL+ for local assertions, because quantifiers do notnecessarily make sense within the context of a TSM.

2Any formula can be transformed in an equivalent negation-freeformula.

Each transition is triggered by a single packet, which ischecked for obeying the local assertions. No quantifiersare necessary in this context.

Layer Assertions Layer assertions are valid for allTSMs belonging to a single layer. In opposite to localassertions layer assertions can make use of the completeFOL+, including universal and existential quantifica-tion. They can be checked either at the initial state or ata final state, i.e. before or after checking for atomicity.In general, a violation of an assertion should be detectedas early as possible. Therefore layer assertions will bechecked at the initial state of a TSM whenever possible.

As layer assertions can be expected to be more complexthan local assertions, they will not be checked duringthe execution of the corresponding TSM. In general it ispossible that the check of a layer assertion during run-time would fail, because intermediary states of the TSMmust not necessarily fulfil the layer assertions. Inter-mediary states only describe the transformation processfrom one consistent state to another. This does not implythat all intermediary states have to fulfil the layer asser-tions. They only have to fulfil the local assertions. Thelayer assertions must hold for the initial state and for thefinal state.

3.2.4 Isolation Tester

The isolation tester is responsible for checking the iso-lation property, i.e. for checking whether a transactionperformed without interference from other transactions.For general processes running on a single node the ap-plication of the isolation tester is obvious: It can detectattacks based on race conditions. To do so, the splitterhas to monitor the audit stream for accesses to criticalobjects (e.g. directories and files) and start the isolationtester, which checks for a violation of the isolation prop-erty.

With regard to communication processes the use of anisolation tester is not immediately obvious. Nonetheless,certain attacks on the protocol level can be interpreted asa violation of the isolation property.

This is especially true if we take the distributed natureof a communication process into account, i.e. if we donot only take the effects of a protocol run on a singlesystem into account, but instead broaden our transactionconcept to include all nodes involved in the communi-cation process. This results in the so-called compound

transaction concept.

3.3 Compound Transactions

The core of modern tele and data communication net-works are the signaling and communication protocols.They define the communication between two or morenodes. Therefore it is not sufficient to consider onlylocal transactions. Instead, it must also be possible tomonitordistributed transactions.

Distributed transactions describe the parallel executionof single transactions on different nodes. This is not al-ways sufficient in our context. Therefore we enhancethe concept of distributed transactions and take alsothe communication between the single transactions intoaccount, i.e. the transactions communicate with eachother and the progress of one transaction depends on theprogress of the other. The communication itself can alsobe considered as a transaction. Together with the localprotocol actions of the sender and receiver the commu-nication forms acompound transaction.

Compound transactions can not only be used to describethe horizontal communication within the protocol stack,but also the vertical communication between differentlayers.

Whether a communication process is monitored as a dis-tributed or a compound transaction demands on the levelof granularity demanded by the IDS. We use the fol-lowing syntax to distinguish between these two kinds oftransactions:

1. Distributed Transactions:A k B

2. Compound Transactions:A$ B

In order to describe the communication process betweenthe involved components, we useMessage SequenceCharts(MSCs).

MSCs [9] are a well-known technique for the visualiza-tion of communicating processes. A single MSC con-sists of different instances, which can communicate witheach other. Each instance itself represents a DFSM,which is additionally allowed to communicate with an-other DFSM. Fig. 4 shows the MSC for TCP’s three-wayhandshake protocol in graphical notation.

A MSC is not a description of the complete behaviorof a system. Instead it expresses one execution trace.

Figure 4: MSC for TCP’s three-way handshake.

msc ::= mscmscid; mscbodyendmsc;mscbody ::= instdef mscbodyj �instdef ::= instanceinstid; instbodyendinstanceinstbody ::= event instbodyj �event ::= in msgidfrom instid;

j in msgidfrom env;j outmsgidfrom instid;j outmsgidfrom env;j actionactid;

Figure 5: Basic Message Sequence Charts.

To specify a system more detailed a collection of MSCsmay be used [13].

The concrete textual syntax of the core language ofMSCs [13] is given in Fig. 5. The eventsin andoutare used for the communication with other instances andthe environment.

The correspondence between message outputs and mes-sage inputs has to be defined uniquely by message nameidentification. A message input may not be executedbefore the corresponding message output has been ex-ecuted [13]. In addition, the activities along one singleinstance axis are completely ordered, but no notion ofglobal time is assumed. Events of different instances areordered only via messages, which imposes a partial or-dering on the set of events being contained.

This partial ordering is checked by the IDS. The com-munication processes on both sides are identified by aunique transaction identifier (TransId ). The transac-tion identifier is composed of the session characteristicsalready mentioned in section 3.2.1. For a TCP sessione.g. the socket address pair and the corresponding se-quence numbers are used. To check the compound trans-action for its correct execution two checks have to be

performed:

� Check the local transactions for their successful ex-ecution (aborted or committed?).

� Check the communication between the local trans-actions according to the MSC.

Step 1 is similar to the general approach taken for dis-tributed database transactions. Each node involved in thetransactions is polled by a coordinator about the successof its local transaction. Based on the information gath-ered from the nodes the coordinator takes a decision onwhether to commit or abort the transaction and informsthe nodes (two-phase commit protocol).

In our case the coordinator will either be the receiveror a trusted third party (TTP). Our monitoring of dis-tributed transactions follows the approach taken by thetwo-phase commit protocol. Nonetheless, for compoundtransactions the communication between the nodes hasalso to be taken into account. The communication be-tween the nodes is checked to conform with the corre-sponding MSC. As we do not assume a global synchro-nized time, this check focuses on the partial order of theevents defined by the MSC, particular the message in-and outputs. In the sense of the state operator�M de-fined in [13] we check that a message input is not exe-cuted before the corresponding message output has beenexecuted. The coordinator therefore requests the inputand output events from the participating nodes and per-forms this check.

Actually compound transactions can be mapped ontodistributed transaction. The checking of the MSC sim-ply defines an additional local transaction that leads toa local decision about committing or aborting the wholetransaction based on its course of communication.

This method will only work if the source of the auditdata is reliable. Otherwise an attacker could send therequesting node a faked set of in- and output events. TheIDS components therefore need to communicate in anauthentic way.

4 Examples

In order to give an insight concerning the feasibility ofour approach, we will examine two aspects related toour model with the help of some practical examples. On

the one hand we will elaborate the aspect of how to sub-divide a communication process horizontally and verti-cally into transactions for monitoring. On the other handwe will have a closer look at certain attack scenarios anddescribe how they will lead to trap events based on vi-olations of the ACID principle. For the latter we willdistinguish between attacks which are to be consideredin the context of local transactions (e.g. Ping of Death)and attacks to be considered in the context of compoundtransaction (e.g. IP spoofing).

The typical scenario considered for these examples is aVirtual Private Network(VPN).

4.1 Scenario

Future applications will increasingly involve severalspatial distributed sites, which are e.g. interconnectedby a VPN. As a matter of fact, VPNs are expected toplay a vital role in future communication scenarios forenterprises and organizations. As stated in [1] such net-works will need to combine strong security propertieswith new functionality, such as conferencing and mul-ticast support. Any risk of misuse or intrusion shouldtherefore be minimized.

VPNs are a well suited scenario for studying IDS anddistributed IDS functionality as they involve two or moreparties belonging to different security domains with dif-ferent trust relationships. We consider three general sce-narios for our approach:

1. Local security domain

2. Co-operation of trusting security domains (e.g.sites interconnected via a VPN)

3. Co-operation of non-trusting security domains (e.g.VPN sites and network provider)

The last two scenarios are of special interest concerningdistributed and compound transactions.

4.2 Atomicity

The check for atomicity of a transaction helps to detectvarious attacks as abnormal behavior. This includes theSYN flooding, adenial of serviceattack.

Figure 6: TSM for the 3-Way-Handshake.

With SYN flooding a server is hit by a great numberof connection requests. The sender address is normallyspoofed.

By defining the TCP connection establishment process,namely the 3-way-handshake, as a transaction, it be-comes obvious that the corresponding DFSM on the vic-tims side, is not completely passed. No final state isreached and therefore the atomicity property is violated.

Fig. 6 shows the corresponding TSM of the 3-way-handshake.

The 5-tuple obeys the following scheme:

(b0; b1; b2; c0; c1)

with

� b0 - Boolean value, indicating whether a packet wassent (b0 = 0) or received (b0 = 1);

� b1 - Boolean value, indicating whether the SYNflag was set or not;

� b2 - Boolean value, indicating whether the ACKflag was set or not;

� c0 - sequence number of the local node;

� c1 - sequence number of the remote node, incre-mented by 1 (acknowledgement number).

Obviously a SYN flooding attack can be detected by theviolation of the atomicity, i.e. the state machine doesnot reach the final stateq4. The protocol itself recog-nizes the incomplete 3-way-handshake on the base of atimeout. Due to this timeout the receiver sends a reset(RST bit set). As a result the next message received byour monitor is an invalid 5-tuple, which aborts the trans-action.

An example for an attack involving two sides in the de-tection process (second scenario) isIP-spoofingbasedon sequence numbers. Actually the already describedSYN flooding attack is part of the attack. If hostC hi-jacks a connection between senderA and receiverB,the local transaction monitored at siteB will end with aCommit(TransId) . By considering the communica-tion as a distributed or compound transaction the wholecommunication is decomposed into two or three localtransactions. Only if all of these local transactions com-mit, the distributed or compound transaction also com-mits. For IP-spoofing the local transaction at the senderA will obviously abort and therefore the whole commu-nication transaction will be aborted.

Other attacks that can be detected by monitoring theatomicity of a transaction include e.g.port scanning at-tacks.

4.3 Consistency

The consistency of a transaction guarantees that thecommunication process is error free and adheres to theprotocol specification. This also helps to detect variousattacks.

One example is thePing of death. This attack is basedon the fragmentation of IP packets. According to [18]IP packets, including the header, can have a size of up to65535 octets. Without any option fields the header hasa size of 20 octets. Packets, which exceed the size ac-cepted by layer 2 of the protocol stack3, are fragmentedand reassembled on the receiver side.

An ICMP ECHO request consists of an eight octet ICMPheader [19], which is followed by the request data. Themaximum size of the data field is therefore65507 octets.

Nonetheless, it is possible to construct and send an in-valid packet, in which the data part contains more than65507 octets. This is based on the fragmentation pro-cess. The fragmentation is done by specifying an offset

3Defined by the maximum transmission unit (MTU).

Figure 7: TSM for the reassembling of IP fragments.

and the size of the fragment. For the last fragment it istherefore possible to exceed the 65535 octet limit for thewhole packet.

A layer assertions therefore specifies that the total sizeof a packet cannot exceed 65535 octets. This layer as-sertion has to be put into action by the TSM, i.e. thelayer assertion is transformed into a local assertion anda corresponding check. In our example the check can bedone when receiving the final fragment, because for thisfragment the sum of the fragment offset and fragmentlength has to be less than 65536.

Fig. 7 shows the corresponding TSM using the follow-ing abbreviations:

� FO (Fragment Offset) - offset (in 8-byte units) ofthe fragment from the beginning of the originaldatagram;

� MF (More Fragments) - one bit in theflagsfieldturned on for each fragment comprising a datagramexcept the final fragment;

� AF - Boolean expression, indicating whether allfragments have been received or not;

� TL (Total Length) - total length of the IP datagram;

� SourceIP - IP source address of the fragmentsender;

The TSM given in Fig. 7 uses both, local and layer as-sertions. The local assertions are part of the inscriptionof the transitions, the layer assertions are explicitly spec-ified below the TSM.

The local assertions ensure, that the fragment withMF = 0 is checked for the condition that the sum ofthe fragment offset and the fragment length is actuallyless than 65536 octets. This puts the second layer asser-tion into action.

The first layer assertion defines the admissible generalconditions of the communication. In this simple exam-ple the set of valid sources is limited to IP addresses ofthe form137:226:� :� or written as aFOL+ expression:

9a3; a4 2 f0; : : : ; 255g : SrcAddr = 137:226:a3:a4

This assertion is checked before the run through theTSM.

4.4 Isolation

Isolation means that each transaction must be performedwithout interference from other transactions, i.e. the in-termediate effects of a transaction must not be visibleto other transactions [5]. As already stated the viola-tion of the isolation property is not immediately obviousin the network context discussed so far, although e.g.IP-spoofing based on sequence numbers could also beinterpreted as a violation of the isolation property. Wewill therefore not elaborate on this topic in this paper.Nonetheless, as one major design goal of our approachis universal validity, we will shortly stress its validity inthe context of processes. For processes the monitoringof the isolation property helps to prevent security fail-ures from improper synchronization of processes or raceconditions.

An abstract example for improper synchronization isgiven in Tab. 1. A concrete example for such a lost up-date problem is given in [11]. [11] describes a scenarioin which a user invokes thepasswdprogram to changehis password while an administrator is editing the pass-word file. The two programs modify the password filesimultaneously, leaving it with an incorrect content.

Following our approach we can consider either one orboth of the accesses to the password file as a transaction.By doing this, the transaction is monitored for its iso-lated execution and the password file cannot be left with

an incorrect content. In contrast to the network scenario,the allowed transactions are not already defined by anykind of protocol specification. When entering the pro-cess and user level depicted in Fig. 1, the allowed trans-actions must be explicitly specified in the security pol-icy. In order to avoid restrictions for the normal user itis sufficient to define transactions on the base of validaction sequences for administrators and security criticalprocesses.

4.5 Durability

We have not considered the property of durability in de-tail so far. As atomicity, consistency and isolation areproperties which focus on the execution of the transac-tion, the durability property mainly influences the pe-riod after a successful execution of a transaction, i.e. theeffects of a successful transaction have to survive evenhard- and software failures or, in our case, attacks.

What does that mean in the context of an IDS? Fordatabases the durability property (in combination withthe atomicity) ensures that after a failure the databasestate to be reconstructed is well defined. This propertyis also useful for general systems. After an attack hasbeen detected it is often difficult to determine the lat-est safe state. If the administrator can not determine theexact time of a successful intrusion, he cannot be surethat his backups are free from security leaks. There-fore an IDS should not only support the detection ofattacks, it should also support the administrator in thereconstruction of the system and the reestablishment ofa safe state. Nonetheless, the details of this approachwill be discussed elsewhere.

5 Related Works

The approach most similar to ours is [11]. It followsa specification based approach and also describes themeaning of atomic actions in relation to intrusion de-tection. Nonetheless [11] does not follow the parallelsto the classical transactional model and therefore, fromour point of view, looses some of its generality. E.g.[11] cites the examples of a superuser editing the pass-word file and an user changing his password in the samefile. The given solution is a specification in the form of aso-calledParallel Environment Grammar(PEG). In the

Environment Variables1. Int E=0;

Start Expression2. <progA> || <progB>

Hyperrules3. <progA> -> <writeA, E>.4. <writeA, 0> -> <openA> <closeA>{E=E-1;}.5. <open> -> open_A {E=E+1;}.6. <close> -> close_A.

7. <progB> -> <writeB, E>.8. <writeB, 0> -> <openB><closeB>{E=E-1;}.9. <openB> -> open_B {E=E+1;}.10. <closeB> -> close_B.

Figure 8: Parallel Environment Grammar (PEG).

PEG the parallel execution of two programs4 is specifiedas shown in Fig. 8.

Although we focus on communication processes in thispaper, it is obvious that the transaction model also ap-plies to this process centric scenario. Following thetransaction model it is sufficient to specify the mosthigh-level (su level) action as a transaction. It is notnecessary to explicitly specify the parallel execution, be-cause of the check for the serially equivalent executionperformed by the scheduler.

Another system related to ours is the Bro system [16].The specifications made by Bro can be integrated intoour approach as a part of theAssertion -section of atransaction or a layer. As we do not have yet specifiedthe exact data types to be used in our approach, we arecurrently considering the reuse of some of the data typesalready specified by Bro.

The main limitation of our model is related to the speci-fication process. In general the specification for commu-nication processes can be extracted immediately fromthe protocol specification. Therefore the specificationcan either be provided by the vendor or any other trustedthird party (ease of specification). Any attack basedon an implementation error can be indirectly detectedby our approach, because it will be recognized as ananomaly. The closer examination and classification (er-ror or attack) of the anomaly can be done either by thesecurity officer himself or by a misuse intrusion detec-tion component.

4The concrete and therefore longer PEG for the password file ex-ample can be found in section 5.4 of [11]. The general structure is thesame as for this simple example.

Nonetheless the transaction model cannot deal withspecification or management errors, which form theother two sources of vulnerabilities. The transactionmodel only deals with implementation errors. If thespecification of a communication protocol or its trans-formation into a specification for the anomaly detectionsystem itself is faulty, attacks based on these errors willremain undetected, because by definition they follow thespecification and can therefore not be considered to beanomalies. The same holds for any management errorsof the environment.

6 Conclusions and Outlook

In this paper we have proposed a new technique foranomaly based intrusion detection. The detection ofanomalies is based on the definition of correct transac-tional behavior. This definition of correct, desired be-havior defines the system’s multi-level security policy,which is monitored during run-time by the IDS. In com-parison with classical database and other transactionalsystems we do not enforce the distinct transactions tobe executed according to the ACID properties. Instead,in the sense of an optimistic scheduler, we monitor thesystem only for any potential conflicts.

Obviously it is not desirable and feasible to monitor allhost activities and connections. The monitoring willtherefore be performed dynamically, i.e. the differentsensors will be activated and configured on demand by acontrol instance being part of the general network man-agement environment. This is one of the main designcriteria for ourAachener Network Intrusion DetectionArchitecture(ANIDA), which forms the broader contextto which the described anomaly detection approach be-longs.

Our approach is currently under development and firstresults of the prototype will soon be available.

References

[1] K. Birman. Building Secure and Reliable Network Ap-plications. Prentice Hall, 1996.

[2] K. Bradley, S. Cheung, N. Puketza, B. Mukherjee, andRonald A. Olson. Detecting disruptive routers: A dis-tributed network monitoring approach. InProceedingsof the IEEE Symposium on Security and Privacy. IEEE,May 1998.

[3] R. Buschkes, D. Kesdogan, and P. Reichl. How to in-crease security in mobile networks by anomaly detec-tion. In Proceedings of the 14th Annual Computer Se-curity Applications Conference (ACSAC’98). ACM, De-cember 1998.

[4] S. Cheung and K. Levitt. Protecting routing infras-tructures from denial of service using cooperative intru-sion detection. InProceedings New Security ParadigmsWorkshop 1997, Cumbria, UK, September 1997.

[5] G. Coulouris, J. Dollimore, and T. Kindberg.DistributedSystems - Concepts and Design. Addison-Wesley, 2ndedition, 1994.

[6] Fred Halsall.Data communications, comuter networks,and open systems. Addison-Wesley, 3rd edition, 1992.

[7] T. Harder and A. Reuter. Principles of transaction-oriented database recovery.Computing Surveys, 15(4),1983.

[8] K. Ilgun, Richard A. Kemmerer, and Phillip A. Porras.State transition analysis: A rule-based intrusion detec-tion approach.IEEE Transactions on Software Engineer-ing, 21(3):181–199, March 1995.

[9] ITU-T. ITU-T Recommendation Z.120: Message Se-quence Chart (MSC). ITU-T, Geneva, 1993.

[10] H. Javitz and A. Valdes. The SRI IDES statisticalanomaly detector. InProceedings of the IEEE Sympo-sium on Research in Security and Privacy, pages 316–326, May 1991.

[11] C. Ko. Execution Monitoring of Security-Critical Pro-grams in a Distributed System: A Specification-BasedApproach. PhD thesis, University of California, Davis,1996.

[12] S. Kumar.Classification and Detection of Computer In-trusions. PhD thesis, Department of Computer Science,Purdue University, August 1995.

[13] S. Mauw and M. A. Reniers. An algebraic semantics ofbasic message sequence charts.The Computer Journal,37(4), 1994.

[14] A. Mounji. Languages and Tools for Rule-Based Dis-tributed Intrusion Detection. PhD thesis, Facult´es Uni-versitaires Notre-Dame de la Paix Namur, Belgium,September 1997.

[15] B. Mukherjee, L. Heberlein, and K. Levitt. Net-work intrusion detection.IEEE Network, pages 26–41,May/June 1994.

[16] V. Paxson. Bro: A system for detecting network intrud-ers in real-time. InProceedings of the 7th USENIX Se-curity Symposium, January 1998.

[17] Phillip A. Porras and Peter G. Neumann. Emerald: Eventmonitoring enabling responses to anomalous live distur-bances. InProceedings of the 20th National InformationSystems Security Conference, October 1997.

[18] J. Postel.RFC 791: Internet Protocol, September 1981.[19] J. Postel.RFC 792: Internet Control Message Protocol,

September 1981.[20] S. Staniford-Chen, S. Cheung, R. Crawford, M. Dilger,

J. Frank, J. Hoagland, K. Levitt, C. Wee, R. Yip, andD. Zerkle. Grids: A graph-based intrusion detection sys-tem for large networks. InProceedings of the 19th Na-tional Information Systems Security Conference, Balti-more, 1996.

[21] S. Staniford-Chen and L. Todd Heberlein. Holding in-truders accountable on the internet. InProceedings of the1995 IEEE Symposium on Security and Privacy, Oak-land, CA, 1995.

[22] A. Tucker Jr., editor.CRC Computer Science and Engi-neering Handbook. CRC Press, December 1996.