VAST: Visibility Across Space and Time

Matthias Vallentin1,2 and Himanshu Sharma1

{vallentin,himanshu}@cs.berkeley.edu

1 UC Berkeley2 International Computer Science Institute

AbstractKey operational networking tasks, such as troubleshoot-ing and defending against attacks, greatly benefit fromattaining views of network activity that are unified acrossspace and time. This means that data from heteroge-neous devices and systems is treated in a uniform fash-ion, and that analyzing past activity and detecting futureinstances follow the same procedures.

Based on previous work that formulated principlesfor comprehensive network visibility [1], we presentthe practical design and architecture of Visibility AcrossSpace and Time (VAST), an intelligent database thatserves as a single vantage point into the network. Weimplemented a proof-of-principle prototype that canarchive and index events from a wide range of sources.Our preliminary performance results indicate that the un-derlying event engine needs further tuning in order to de-ploy the system in large-scale operational networks.

1 Introduction

Many administrative tasks in operational networks todayrely on descriptions of network activity that are frag-mented in both space and time. By fragmented in space,we allude to the heterogeneity of disparate data sourcesand programmatic interfaces that the network operatorhas to incorporate to obtain a unified view of the networkdata. By fragmented in time, we refer to the discrepancyin analysis procedures between past and future activity(e.g., processing plain-text IDS logs versus configuringfirewall rules; using scripts to extract information fromthe logs versus pro-actively configuring alert generation).

In this paper, we provide the practical component toprevious research which set out principles of comprehen-sive network visibility [1]. We devised the architectureand implemented the first prototype of Visibility AcrossSpace and Time (VAST), a platform that serves as a sin-gle vantage point on network activity. VAST is designed

to archive descriptions of activity from numerous sourcesin the network and to query past activity for retrospectiveanalysis. Based on a policy-neutral core, the architectureallows a site to customize operations to coalesce, age,sanitize, and delete data.

The realization of such a system requires addressing anumber of challenges. First, we need a flexible and ex-pressive data model to unify heterogeneous descriptionsof activity. This corresponds to principal D.1 from [1].A particularly apt model for this purpose is the pub-lish/subscribe scheme, which is based on asynchronouscommunication using events. In this model, interactingpeers publish the availability of an event and subscribe toevents published by peers according to local interest.

Second, we need to separate mechanism from policy(principle D.6 in [1]). Many security devices limit theirfocus only on real-time detection but ignore the addi-tional step of providing policy-neutral output that can behighly useful in the future.

Third, we have to realize that the temporal dimensionsof analysis extend beyond the duality of past and fu-ture. By considering an additional what-if perspective,we seek to automatically explore the suitability of anal-ysis patterns codifying future behavior if applied to pastactivity (principle C.2 in [1]). For instance, false posi-tives can be greatly reduced by testing detection proce-dures against previous network behavior.

Fourth, a large measurement window with respect totime is needed (principle D.2 in [1]). Most intrusion de-tection systems operate in real-time today and discard theincoming data after processing it. However, an extensivearchive allows re-inspection of activity that became in-teresting only in retrospect.

Finally, we argue to avoid “clean-slate” designs (prin-ciple B.2 in [1]). In operational practice, the very chal-lenge in creating a coherent picture of network activity isoften exacerbated by the chaotic details such as hetero-geneity and policy frictions. Therefore, we emphasizeincremental deployability to achieve practical usability.

The remainder of the paper is organized as follows.In § 2, we sketch possible scenarios that would signifi-cantly benefit from the existence of VAST system. Then,we refer to related work in this field in § 3 and compareour approach to existing ones. After introducing the tech-nological building blocks of VAST in § 4, we explain itsarchitecture in § 5 and prototype implementation § 6. Wepresent a performance study in § 7 and give recommen-dations for future work in § 8.

2 Application Scenarios

Different scenarios in operational networks face substan-tial problems due to the fragmented nature of descrip-tions of network activity. In the following applicationfields presented by [1], we point out practical hurdles andproblems and argue why an integrated view of networkactivity would significantly improve the situation.

Troubleshooting. Troubleshooting network problemsrequires an operator to gather detailed informationfrom disparate sources. Pin-pointing the exact er-ror source is made difficult by the layered, modu-lar structure of network architecture and protocols.The more context they can draw upon, the easier itbecomes. Being able to characterize the problemsources from as many perspectives as possible is ofhigh value.

Also, allowing the operator to integrate analogousanalysis procedures for future activity would be agreat utility. For example, consider the example ofa failing automated remote backup caused by wrongpermissions of the SSH configuration directory. Anoperator could codify the automated monitoring ofSSH server logs to trigger an automated inspectionof the directory permissions on the failing client.

Detecting Intrusions. Most Intrusion Detection Sys-tems (IDS) focus on a particular type of monitoringdata (either host logs or network packets). A hy-brid system that synthesizes data from a variety ofsources (virus scanners, user keystrokes and inter-actions with peripheral devices, firewall logs, DNSlookups, internal NetFlow records, and honeypotdata) enhances the analysis context of the IDS.

Post facto forensics constitute an important aspectof comprehensive incident response. A high qualityarchive allows the security analyst to assess the realdepth of an intrusion and quantify the scope of thecaused damage. A unified log archive makes foren-sic analyses less error-prone and more precise, andsaves a significant amount of time, which is key todefend against an ongoing attack.

It is also highly beneficial for a security analyst be-ing able to codify security breaches in a consistentfashion in order to automatically detect future in-stances of a certain attack pattern. Moreover, the an-alyst could extend the temporal dimension to what-if, i.e., automated exploration of how analyses in-tended for future activity would have behaved if ap-plied in the past.

Combating Insider Abuse Dealing with securitybreaches from the inside is similar to intrusiondetection, but it is difficult to comprehend thefull scope of the attack at the time of detectionbecause the complete attack comprises of a specificsequence of actions, each of which the attackerwas authorized to do. For example, an employeeaccessing a sensitive machine, copying documentsto an intermediate machine to conceal the maliciousintent, and finally sending the sensitive data to anon-authorized party.

Insider attacks can manifest over long time inter-vals. When trying to understand the full extent ofa breach, a comprehensive archive that reaches farback in time is particularly helpful to reveal initialattempts to hide the abuse or to uncover similar non-authorized activity.

3 Related Work

The term visibility has been used in different contexts inthe past. Cooke et al. [6] use the term network-wide vis-ibility to refer to the ability of accessing clear-text dataat the network-level, which is hardly possible in the pres-ence of encryption and tunneling. To regain visibility, theauthors present Anemone, a monitoring infrastructure de-ployed on endpoints where data flows can be accessed inclear-text.

The Time Machine (TM) is the closest to our work. Itefficiently records network traffic streams to support in-spection of activity that turns out to be interesting onlyin retrospect [14]. Similar to VAST, the TM aims to im-prove key operational networking tasks — such as trou-bleshooting and forensic analysis. While the TM pro-vides high-performance network traffic capture, index-ing and retrieval, VAST extends these concepts to arbi-trary events that can stem from a wide range of sources.Similar to the TM’s architecture, VAST is parallelizedfrom scratch to exploit the performance benefits of mod-ern multi-core CPUs.

One major aspect in the design of VAST is the storagecomponent which continuously archives and indexes in-coming events for later retrieval. Approaches geared tooff-line network traffic analysis [5, 20] are not designed

to deal with live streaming data [8, 21]. In the follow-ing, we briefly sketch some of the existing approachesby streaming databases.

The CoMo project [10] provides a network monitor-ing infrastructure featuring both live and retrospectivequeries on streaming data. In a distributed setup, mul-tiple CoMo systems can propagate queries to pinpointrelevant data locations in the network. Despite its high-speed layout, network traffic is stored in a circular on-disk buffer which imposes functional limitations.

TelegraphCQ [3] builds an adaptive stream processingengine for high-volume data streams based on the open-source database PostgreSQL. Recently, the system hasbeen extended to allow querying both historical and real-time data simultaneously [19, 4]. To work around theexpensive I/O operations for fetching historical data, theapproach retrieves only a sampled — and thus incom-plete — version of the incoming data stream.

An interesting hybrid approach follows Hyperion [7],a system for archiving, indexing, and on-line retrievalof high-volume data streams. Their write-optimizedstreaming file system, StreamFS, poses an alternative tothe traditional relational database storage mechanisms.The low-level programming and file system implemen-tation is only available for Linux.

4 Technologies

VAST continuously receives and relays streams ofevents, and stores them permanently for later querying.Traditional data base management systems (DBMS) sup-port historical queries over data that already exists beforethe query is issued. In contrast, data stream manage-ment systems (DSMS) answer live queries that incorpo-rate new arriving data after the query is issued. VASTis a hybrid system that answers both historical and livequeries.

Queries sent to VAST usually process large quantitiesof historical data. At the same time, the system has toarchive a continuous data stream. Since the high I/Ocosts of delivering historical query results can prevent thesystem from processing the incoming stream of data, it isimportant to accelerate historical query through indices.

On the one hand, B-Tree [2] indices exhibit similarcomputational complexities for searching and updatingand are hence apt for on-line transaction processing ap-plications (OLTP) where write and update operations oc-cur more often than searching.

On the other hand, Bitmap indices [17, 24] performbetter than B-tree indices for on-line analytical pro-cessing (OLAP) applications where search frequency ishigher than update frequency [23, 24, 16]. Although up-dating bitmap indices is known to be expensive [24], ithas been shown recently that efficient updates are indeed

1

0

0

1

0

0

1

01

B0

3

6

0

7

0

0

0

0

0

1

12

0

B2

0

2

ID

0

1

B1

5

foo

1

0

3

2

0

0

1

0

1

0

0

1

B3

0

0

0

0

0

4 3 0

0

3210

Figure 1: A bitmap index generated from 4 distinct val-ues

possible [19]. As historical data is never modified, newdata can be appended without disturbing existing data.Additionally, some indices can be built on unsorted datawhich allows adding new data in time that is a linearfunction of the number of new records.

In general, bitmap indices are based on bit arrays(called bitmaps) and answer queries by performing bit-wise logical operations which are well supported bymodern processors. To index a single column in a ta-ble, the bitmap index generates c bitmaps, where c is thenumber of distinct values in the column (also referred toas column cardinality). Each bitmap has N bits, with Nbeing the number of records in the table. If the columnvalue in the kth row is equal to the value associated withthe bitmap, the kth bit of the bitmap is set to 1, but re-mains 0 for any other column value.

Figure 4 depicts a simple bitmap index for the columnfoo that consists of integers ranging from 0 to 3. Sincethe column cardinality is 4, the index includes 4 bitmaps,named B0, B1, B2, and B4. For example, the third and5th bit of B3 are set to 1 because the values in the corre-sponding rows are equal to 3. Answering a query such as“foo < 2” translates into bitwise OR (|) operations be-tween successive long-words of B0 and B1. The result isa new bitmap that can be used in further operations.

FastBit To archive, index, and query historical data,VAST uses FastBit [22] for its storage engine.FastBit implements many efficient bitmap indexingmethods with various encoding, compression, andbinning strategies. A distinctive feature of Fast-Bit is its efficient compression method, the WordAligned Hybrid (WAH) method [23, 24]. WAH em-ploys run-length encoding (RLE) to reduce the size

of each bitmap in a bitmap index so that the totalsize of the index remains modest regardless of thetype of data.To minimize I/O operations on indexlookups, FastBit stores bitmaps in one index in lin-ear order.

Bro To describe network activity, VAST leveragesthe event model of the flexible open-source BroNIDS [18] developed and maintained by Vern Pax-son. Raw packets from the network interface arereceived through libpcap [13]. Employing libpcapenables flexible deployment in various UNIX envi-ronments and greatly reduces the traffic in the ker-nel by applying a BPF filter expression [15]. Thepre-filtered packet stream is then handed up to thenext layer, the event engine, which re-assembles thepacket streams and elicits events.Since the resultingevents represent a policy-neutral snapshot of visi-ble network activity, they perfectly fit into our eventmodel.

Broccoli To enable third-party applications partaking inthe event communication, the light-weight client li-brary Broccoli1[12] has been developed, allowingother applications to request, send, and receive Broevents.

5 VAST Architecture

This section describes the architecture of VAST. Afterframing the technical challenges we follow throughoutthe system design, we give a high-level overview aboutVAST’s components and then delve into each componentin detail.

5.1 ChallengesVAST can be seen as an intelligent fusion of DBMS andDSMS. Consequently we face challenges and limitationsof both technologies. It is an ambitious undertaking tomerge the functionality of both approaches as they aregeared towards different application domains.

Real-Time Nature. As a single sink for incoming datain form of a continuous stream of events from allcomponents interfacing with the VAST, the systemmust provide enough resources to archive the arriv-ing events quickly and entirely, without foregoingnew data.

High Concurrency. System must be able to adequatelyprocess the incoming event stream, while at thesame time offering enough resources to perform

1Broccoli is the healthy acronym for “Bro Client CommunicationsLibrary”.

IDS

Web Server

Custom Application

IDS

Security Analyst

Network Operator

V A S T

IndexArchive

Incoming Events Bidirectional query and result channel

Figure 2: High-level overview of VAST.

parallel tasks, such as answering retrospectivequeries, dispatching event feeds, reorganizing indexstructures, and constantly aggregating old events tomaximize storage capacities; emphasizing the needfor a concurrent and asynchronous system architec-ture.

Compatibility. The system should provide the abilityto transform internal state of arbitrary devices intoevents that can be sent to the VAST system yieldinga powerful mechanism to unify the communicationof heterogeneous components.

Commodity Hardware. Finally, the system should notrequire expensive custom hardware to achieve thedesired performance. Instead we prefer to em-ploy commodity hardware to enable cost-effectivedeployment. Ideally, performance bottlenecks canbe countered by adding a new machine that over-takes a dedicated task to unburden a component thatreached its resource limits.

5.2 Overview

VAST is a scalable, distributed system that archivesevents from a wide range of sources and offers a queryinterface to extract a stream of events. As illustratedin Figure 2, an event source could be a NIDS, web server,logging daemon, or an arbitrary Broccoli-enabled appli-cation. VAST archives the incoming event streams bystoring a raw copy in the archive and indexing the eventdetails in a separate index.

An interactive query interface allows users to extracthistoric activity in form of events. For example, the in-terface supports retrospective queries like “which eventsinclude activity of IP address X today” or “which eventsin the last month included access of URL Y”. The queryresult consists of a list of events that match the given con-dition. In a further step, the user can inspect and parsethe event to extract the desired details.

Further, users should have the ability to install livequeries that remain active. By subscribing to a specificevent pattern, such as “all URLs of this form”, VASTinstantiates a filter for all future events matching thegiven condition. The corresponding query is registeredas an event feed which continuously relays the qualify-ing events to the user or a remote application.

5.3 Data RepresentationVAST leverages the event model of Bro and uses Broc-coli to outfit third-party applications with the capabilityto communicate in the same event model. VAST actsboth as sink and source of events, that is, it represents atransparent actor in the network since events can be sentto and received from the system in a uniform fashion.This flexibility is particularly beneficial when integrat-ing the VAST system into larger setups that span acrossthe boundaries of the local administrative domain.

5.4 Event SpecificationEvents in the Bro event model consist of a name andzero or more typed arguments. To minimize the space-overhead when sending events across the network, onlyevent name, argument data, and the type information aretransferred, but not the argument names. We refer tothese compact representations that do not contain the ar-gument names as raw events.

The missing semantic context is codified in the eventspecification, a document that VAST maintains to man-age event meta data. Separating the event structure fromtheir semantics gives us a flexible mechanism to changethe meaning and behavior of events when they are pro-cessed. For example, it is possible to rename arguments,change the way they are indexed, or link together similararguments of different events. VAST’s specification lan-guage is similar to the Bro scripting language and offersboth primitive types such as addr, count, string,etc., and compound types such as record, vector,set, and table.

To group related or similar semantic activity, an eventis assigned to zero or more classes. A class consist ofa list of arguments and the contributing events. By de-fault, every event generates a separate class because acomputer cannot deduce semantic relationships betweenevents reliably. The following examples illustrates thebenefit of classes:

event http_request (src: addr,dst: addr,request: string

)

event ftp_request (

client: addr,server: addr,command: string

)

From a user perspective, the query “which events fromthe last hour involve communication between host A andhost B?” should ideally consider both events. The cor-responding query that connects (or joins) the two eventswould be similar to:(http_request.src = A OR ftp_request.client = A)

AND(http_request.dst = B OR ftp_request.server = B)

With hundreds of events, this explicit connection atquery time not only introduces a significant overhead forthe query issuer, but is also quite error-prone. There-fore, the event specification allows factoring out relatedinformation. The following change uses the &map toattribute in the specification to create a separate classconn which holds IP endpoint information:event http_request (

src: addr &map_to="conn:orig"dst: addr &map_to="conn:resp"request: string

)

event ftp_request (client: addr &map_to="conn:orig"server: addr &map_to="conn:resp"command: string

)

Although not explicitly listed as such in the specifica-tion, the class conn has the following internal structurebased on the above mappings:

conn ( orig: addr, resp: addr )

Our initial query looks now much simpler and ex-presses the intent of the user in a more natural fashion:

conn.orig = A AND conn.resp = B

5.5 Archiving EventsA high-level overview of the event archival process isvisualized in Figure 3. All three components involvedin the archival process are designed to operate in a dis-tributed fashion. That is, each component can be de-ployed on a separate machine and they communicateover a network connection. Because each component isself-contained and has an associated event queue, scal-ing components across multiple machines is naturally en-abled by the system design.

In order to interact with VAST, event producers con-nect to the event dispatcher and send their events througha Broccoli communication channel. The dispatcher as-signs each incoming event a unique monotone 64-bit

IndexArchive

ArchiveManager

Event Dispatcher

Index Manager

Bro Apache Broccoli-enabled Application

Feed Manager

Figure 3: Archiving events from multiple sources.

identifier and creates two copies of the event: The firstcopy is the raw event, an unmodified version of the se-rialized Broccoli event augmented with the obtained 64-bit ID that is compressed and forwarded to the archivemanager. The second copy is the VAST event which isconverted from a raw event into an internal data struc-ture. It is forwarded to the index manager which accessthe event arguments through a well-defined interface. Wewill discuss the feed manager later in the context of livequeries (see §5.6.2).

The archive represents a comprehensive record of allevents that have ever been sent to the VAST system. Itstores events unmodified, i.e., in their serialized formas obtained from Broccoli. Archiving raw events ratherthan the VAST events has the advantage that query re-sults can directly be sent over the network using Broc-coli. The archive consists of a collection of containersoffering a table-like interface to append data. Figure 4visualizes an archive and index container. The formerhas two columns to store evens associated with their ID,the latter has an ID column and one column for each ar-gument of the class that this container represents.

Archive Containereventid

Index Containerarg2arg1arg0id

Figure 4: Archive and index containers.

Common among archive and index is a specific slicingscheme. A slice is an abstraction that maps an object with

an identifier to specific bucket. The archive uses event idsas identifier and the objects are containers. The indexuses timestamps as identifier that map a set of containers(also referred to as index slice). Slicing is similar to thebinning strategy employed by FastBit, only at a higherlevel of abstraction.

Archive

0 1 n...

{ 42 → 1, 43 → 1, 84 → 2, 85 → 2 }

21

Index

1 n...1 20

Event ID → Archive Slice

Figure 5: Slicing of archive and index.

This technique partitions data horizontally. As illus-trated in Figure 5, an incoming event is both sent to thearchive and index, where the corresponding slice is de-termined. Because object identifier are monotone, theevent is usually appended to the last slice.2 When look-ing at the index to find specific events, the list of returnedevent ids allows us to efficiently locate the correspondingevents in the particular archive slice (see §5.6.1).

Slicing helps only to bypass FastBit’s maximum con-tainer size, but we also expect to use it in conjunctionwith VAST’s aging and aggregation framework to expireslices with no more valid entries.3 The slice interval iscustomizable by the user and denotes the maximum num-ber of events to be stored per container for the archive,and a time interval in seconds for the index. For exam-ple, if the slice intervals were 20,000 and 60, an archiveslice would then hold 20,000 events and all events arriv-ing in the same 60 seconds interval would belong to thesame index slice.

When issuing queries like “which events include hoststhat accessed IP address X more than n times in the lasthour”, we usually formulate a query condition that refersto a particular event argument. In the above example,we are interested in events that have an argument that

2Due to multi-threading or components distributed across multiplemachines, it is possible that out-of-order events arrive that have to besent to a previous slice.

3At the time of this writing, aggregation and aging only exists con-ceptually. In this context, slicing helps us to overcome the append-onlynature of FastBit partitions by deleting a partition entirely after all itsentries are marked as invalid through an aggregating process.

represents a destination IP address. One approach to ex-tract the relevant events would to iterate over the entirearchive and inspect each event. This is clearly an ineffi-cient strategy for large event archive. To improve queryperformance, we therefore index each argument value byassociating it with the corresponding event ID.

Continuing the above example, consider the fictitiousevent transfer with the following structure:event transfer (

from: addr,to: addr,data: string

)

An event with such a simple structure translates to aclass that consists of the same arguments. The corre-sponding container would contain columns for the threearguments, plus one column for the event ID. That is, thecolumn names would be id, from, to, data. Whenindexing an instance of this event, the ID and each ar-gument are written in one container row. Unfortunately,we encounter also more complex scenarios in practice:events can include arguments which belong to severalclasses (e.g manually remapped arguments).

Index Manager

Queue

Index

Index Sice

Index Slice

Containerarg2arg1arg0id

VAST event Plucker

Figure 6: Pluckers that index events.

To counter these intricacies, we introduce the plucker,a dedicated entity to distribute all arguments of one eventto the corresponding containers. Upon slice creation, theplucker consults the event specification to generate a listof containers that have to be accessed for a particular

event. When an event arrives, the plucker performs thefollowing steps. First, it sends the event ID to all con-tainers that have to be accessed for the given event. Sec-ond, the plucker iterates over the event arguments andwrites the data of each argument to the correspondingcontainer. Similar to a database transaction, the pluckerfinally commits the involved containers when all argu-ments have been processed successfully.

5.6 Event RetrievalVAST offers operators and security analysts a central-ized vantage point where they can retrieve informationabout past and future activity. Moreover, remote appli-cations such as a NIDS can interact with VAST to obtaina policy-neutral event stream to analyze or subscribe tospecific set of events that are forwarded when they en-ter the system. To issue a query, a client connects to thequery manager. This component provides a well-definedinterface to access to archive and index over the network.We will cover the query manager in-depth later in thissection.

Queries sent to VAST are represented as events aswell. The advantage of this approach is that we can em-ploy Broccoli to create an asynchronous, bi-directionalcommunication channel. While the client uses this chan-nel to issue queries, VAST uses the same channel to sendthe qualifying results to the client.4

Furthermore, a client can specify a query target differ-ent from itself to redirect the query result to a differentmachine. For example, this proves beneficial when ananalyst issues a manual query to feed a stream of eventsinto an intrusion detection system. In the following, wefirst discuss the design of historical queries and thereafterturn to live queries.

5.6.1 Historical Queries

The schematic process of a historical query is sketchedin Figure 7 and involves the six following steps.

1. Both users and remote application can issue queriesby creating a query event locally and sending it tothe query manager. Each incoming query is associ-ated with a dedicated query queue that is processedasynchronously to deliver matching events back tothe client.

2. Executing the query instructs the query manager toparse the event, extract the query condition and ac-cess the index to obtain a list of event ids that matchthe condition.

4Note that query events take a different data path into the systemthan events that are archived and indexed. Rather than connecting to theevent dispatcher, clients that issue queries connect to the query managerwhich listens on different port.

Security Analyst

Network Operator Index Manager

Queue

Index

Archive Manager

Queue

LibraryIDS

1

3

5

2

4

1

6

6

n Historical Query ProcessingHistorical Query EventList

Query Manager

QueueQueue Event Dispatcher

Event Client

ID

ID

ID EventIDs

Figure 7: Historical query processing.

3. The index evaluates the condition and sends a list ofmatching ids back to the index manager.

4. If the list was empty the query manager signals theclient that the query did not yield a result and ter-minates the connection. Otherwise, it consults thearchive to extract the events with the given list ofids.

5. The archive manager then sends the extracted eventsback to the query manager which inserts them in tothe corresponding query queue.

6. As soon as events are inserted into the client queue,the query manager forwards the events over the ex-isting Broccoli connection to the client.

5.6.2 Live Queries

In contrast to historical queries, live queries do not con-sult the event archive. Instead, they install an event feedwhich continuously relays matching events to the queryissuer.

On the client side, the only difference to historicalqueries is a flag indicating the query type. On VASTside, live queries differ significantly from their historicalcounterpart. While historical queries can simply be con-verted into one or more index lookups, live queries ex-hibit a different nature. Instead of “pulling” data from

disk, a constant stream of events is “pushed” into thesystem. This subtlety has significant implications forthe design of the live query architecture. In particular,a very high or bursty event arrival rate can impose anunpredictable workload and thus imposes limitations onthe system resources required for processing the eventstream.

In historical query architectures, queries are brought tothe event data. Live queries have diametrically opposedcharacteristics: data is brought to the queries. This twistnecessitates a different architecture for live queries. Ourdesign is depicted by Figure 8. Issuing a live query con-sists of the following steps:

1. Both users and remote applications can issuequeries by creating a query event and sending it tothe query manager.

2. The query manager parses the event and forwardsit to the feed manager which creates a live querybased on the given condition. Each VAST eventis “pushed” into the feed manager and comparedagainst all live queries.

3. If a match was successful, the feed manager re-quests the corresponding raw event from the dis-patcher and sends it to the associated query queueof the query manager.

1

1

4

4

Feed Manager

Live Query ProcessingnLive Query Raw Event VAST Event

Query Manager

QueueQueue

2

3

Event Dispatcher

EventClient

Security Analyst

Network Operator

IDS

Figure 8: Live query processing.

4. As soon as events are inserted into the client queue,the query manager forwards the events over the ex-isting Broccoli connection to the client.

A key challenge in the design of the live query archi-tecture is avoiding big performance penalties. Becauseeach incoming event has to be matched against all livequeries, the processing overhead can be non-negligiblewhen facing large event volumes. To cope with a veryhigh event load, concurrent query execution could sub-stantially ameliorate the scalability of the system.

Another issue represents the retention strategy for rawevents. Because the feed manager asks for any raw eventafter encountering a positive match, the dispatcher wouldhave to keep the raw events in memory until the corre-sponding VAST event is processed. One possible solu-tion would be to forward the raw event to the feed man-ager as well. While it would remove the dependency tothe dispatcher, it greatly increases the data sent across thenetwork in a distributed setup.

Note that the live query architecture is still in a veryfledgling stage. Hence we have not yet found an idealsolution but would highly appreciate feedback to comeup with a tractable solution.

6 Implementation

We now delve into technical facets of VAST and de-scribe interesting aspects of the implementation. VAST

is implemented entirely in C++ and currently consists of7,102 LOC including comments. Since our long-termgoal is not only to create a prototype but an applicationto be employed by large institutions, we emphasized acareful development process and distilled separate mod-ules that have small interfaces. Our implementation has alayered architecture with lower layers not depending onhigher layers.

utilcomm

metaevent

store

dispatch query

age

Figure 9: VAST implementation: higher layers only de-pend on lower layers.

As shown in Figure 9, the building blocks of the sys-tem are the meta and event layer. While the formerimplements the event specification with event and typemeta information, the latter contains data structures usedduring event processing, such as raw events and VAST

events. On top resides the store layer which providesstorage abstractions for archive and index, the slicinglogic, and an enhanced FastBit interface for data stream-ing. Both the dispatch and query layer make useof the storage engine. Finally, aging and aggregationis the highest layer as it act like a client that performsqueries and inserts new events. Note that the shaded lay-ers (query and age) are not yet implemented.

6.1 Event Compression

When archiving a high event volume over a very longtime window, already small efforts to keep data growthunder control have a substantial effect in the long run.

Recall that the event dispatcher creates a raw byte copyof the incoming Broccoli event. Storing raw data (alsoreferred to as blob data) in a FastBit is only supportedby means of NULL-terminated C-strings. However, thecontents of the raw Broccoli event can include NULLbytes and thus cannot be stored in its pristine form. Inorder to avoid the NULL bytes, we encode the raw bytesconstituting the Broccoli event. To this end, we created acustom run-length encoding that achieves a compressionrate of approximately 30%.5

Our byte-stuffing algorithm is simple and straight-forward: each non-NULL character is copied withoutmodification. NULL bytes are replaced with a specialencoding symbol followed by a null symbol. The byteafter the null symbol holds the number of consecutivezeros in the raw byte stream. If this number becomes256, we write the value 255 (or 0xFF), reset the NULLbyte counter to 1, and advance to the next byte. We con-tinue this procedure until the number of zeros remainsless than 256. If we encounter the encoding symbol inthe raw byte stream, we simply duplicate it in the en-coded byte stream.

Although we could use more sophisticated algorithmswith higher compression rates, we explicitly chose a sim-ple encoding scheme given that disk space is cheap asopposed to CPU cycles.

7 Evaluation

In the following we present our preliminary performanceresults of VAST’s building blocks, the event and storagelayer. At first we explain our methodology in § 7.1, thenevaluate the performance of the event layer and storagelayer in § 7.2, and finally discuss the storage overheadin § 7.3.

5Raw Broccoli events contain apparently enough consecutiveNULL bytes to yield such a compression rate

7.1 Methodology

In order to generate reproducible measurements we re-sort to trace-based offline analysis. As event source, werun Bro on a pcap trace, enabling a large fraction of itsanalyzers to induce a high event load. VAST listens ona TCP socket at port 42,000, waiting for event produc-ers to connect. Although VAST supports multiple eventsources in parallel, we only use one connecting Bro in-stance in our measurements.

Generally, trace-based analysis entails a “compressed”notion of time because the trace is processed as fast pos-sible, without respecting the inter-packet arrival gaps. Togenerate a more realistic workload, we use Bro’s pseudo-realtime mode which insert processing delays while re-ceiving packet input from a trace. These delays equalto the natural inter-packet arrival gaps. Internally, Brodefers packet processing until the time difference to thenext packet timestamp has elapsed. All our measure-ments are conducted with a pseudo-realtime factor ofone, thus reflecting real-time trace replay.

We conduct our measurements with a packet trace oftraffic from the International Computer Science Insti-tute (ICSI) in Berkeley. The trace has been recorded onNovember 6 and spans 24 hours, ranging from 01:45:02AM to 01:45:04 the next day. It contains 32,923,230packets from 447,893 IP flows (avg. 73.51 pkts/flow)that incorporate 155,394 distinct source addresses. Theobserved average rate is 1.81 Mbps (standard deviation3.97 M) and the peak rate amounts up to 92.45 Mbps.95.15% bytes are TCP traffic, whereas 4.56% constituteUDP traffic. The most prevalent protocols in terms ofbyte are HTTP (66%) and HTTPS (2.81%).

7.2 Performance Analysis

To get a detailed picture of the resource usage, we em-ploy the Goggle perftools [9] to both understand memoryfootprint and CPU usage. The heap profiler indicates thatVAST has very low memory footprint of 29.7 MB. Ourexpectation was that most of CPU time would be spentin VAST’s storage layer. However, it turned out that 52%of time was spent in the event layer. We then conducteda new measurement to analyze this layer in isolation. Tothis end, we turned off the storage engine by uncomment-ing the relevant portions of code in the Broccoli eventhandler that is executed when event arrives.

According to the perftools profiler, the functionbro ht free() and its callees spent 51.7% of CPU

time. We assumed that the small object allocation of nu-merous empty tables caused this high figure and henceworked with the Bro developers to investigate the issue.After patching bro ht free() to prevent the allo-cation of slots when the hash table is empty, the CPU

consumption of this particular function could be reducedto to 6.3%. As a result, the maximum processing rate ofevents doubled from roughly 1,500 to 3,000 events persecond.

We now turn to the actual event processing overheadof VAST’s storage layer. As baseline, we run VASTwith empty event handlers as above, thereby disablingthe storage component. The event processing rate is dis-played in Figure 10(a). The corresponding event queuesize on the Bro side is shown on the right side in Fig-ure 10(b). The average event arrival rate per second is420.3 with a standard deviation of 460.9, and a peak rateof 2,959. Then, we enabled the storage component ofVAST and observed event rates and queue sizes as shownas in Figure 10(c) and Figure 10(d). In this case, the av-erage event arrival rate was 307.8, the standard deviation243.3, and the maximum number of events per second1707.

Unfortunately, we could not run our measurementsto completion. The sheer volume of the event streamcaused Bro to terminate the network connection. Toomany events on the VAST side arrived that could notbe consumed by Broccoli and handed up to VAST. AfterBro reaches a threshold of 250,000 events, it closes thenetwork connection. Therefore, our measurement withenabled storage engine reflects the first 10% of the mea-surement with empty event handlers. That is, Bro termi-nates the connection after reaching the first peak in Fig-ure 10(a).

Our results confirm an earlier result: 52% of CPU timeis spent in Broccoli, thus reducing the event arrival rateby a factor of two. Generally speaking, the analysis sug-gests to concentrate on further improvements in the eventlayer before optimizing higher layers.

7.3 Storage Overhead

We now investigate the storage overhead of raw eventson disk. VAST could archive and index 1,180,000 eventsuntil Bro closed the network connection. The size of thearchive amounts to 4.2 GB and the index occupies 377MB (11.48%) of disk space. Archive meta data (64bitidentifier and container meta data) constitutes 0.098% ofthe entire archive volume.

Figure 11 characterizes the types of events that havebeen archived. The event with the highest frequency wasconn weird (428,778 times). Bro generates this eventwhenever it encounters a “weird” connection, such as anunexpected TCP state change. The second highest fre-quency has packet contents (249,013 times) whichcontains the raw packet contents of a particular connec-tion.

conn

ectio

n_at

tem

ptco

nnec

tion_

esta

blis

hed

conn

ectio

n_pa

rtia

l_cl

ose

conn

ectio

n_pe

ndin

gco

nnec

tion_

rese

tco

nnec

tion_

reus

edco

nnec

tion_

stat

e_re

mov

eco

nnec

tion_

SY

N_p

acke

tco

nnec

tion_

timeo

utco

nn_s

tats

conn

_wei

rdflo

w_w

eird

gnut

ella

_not

_est

ablis

hht

tp_s

tats

icm

p_ec

ho_r

eply

icm

p_ec

ho_r

eque

stic

mp_

sent

icm

p_tim

e_ex

ceed

edic

mp_

unre

acha

ble

load

_sam

ple

net_

stat

s_up

date

new

_con

nect

ion

new

_con

nect

ion_

cont

ents

ntp_

mes

sage

pack

et_c

onte

nts

rem

ote_

log

softw

are_

vers

ion_

foun

dtc

p_pa

cket

term

inat

edud

p_co

nten

tsud

p_re

ply

udp_

requ

est

udp_

sess

ion_

done

020

0000

0

Figure 11: Distribution of different event types.

8 Future Work

As illustrated by our performance study, the bottleneckof the system represents the event layer. Given that ourlong-term goal is operational deployment of VAST, wewill focus on increasing the peak event processing rateof Broccoli to keep up with a high-volume event stream.We currently experiment with different tuning parame-ters of the Broccoli that reads from the network socket inrounds,in which each reads with fixed number of bytes.

Our next near-term goal is to implement the query en-gine. As described, it will support SQL-like queries andallow users to install event feeds. In the long run, we en-vision to provide graphical user interface to further im-prove the usability of the system.

Future work further includes the implementation ofan aging and correlation layer which performs continu-ous housekeeping tasks to optimize space utilization andquery response times. The idea is to elevate events tohigher semantic abstractions by condensing them into amore succinct form. For example, events reporting nu-merous unsuccessful connection attempts, could be com-bined to a summary like “N connections in last T sec-onds”.

To cope with a high-volume event stream over a longperiod of time, we could use degradation mechanismsthat gracefully transform data into more compact but stilluseful forms. We can have different levels of abstraction.Despite the loss of data over time, abstracted informationcan constitute the missing link in a chain of previous ac-tivities.

The global nature of network attacks makes them dif-ficult to counter with only a strictly local vantage point.We envision VAST to act as a platform to facilitate op-

0 5000 10000 15000 20000 25000

050

010

0015

0020

0025

0030

00

time in seconds

# ev

ents

(a) Event processing rate (empty handler).

0 5000 10000 15000 20000 25000 30000

050

000

1500

0025

0000

time in seconds

# ev

ents

(ch

unks

/2)

(b) Bro queue size (empty handler).

0 1000 2000 3000 4000

050

010

0015

00

time in seconds

# ev

ents

(c) Event processing rate.

0 500 1000 1500 2000 2500 3000

050

000

1500

0025

0000

time in seconds

# ev

ents

(ch

unks

/2)

(d) Bro queue size.

Figure 10: Event processing rate and Bro queue size.

erationally viable, cross-institutional information shar-ing of attack details. Sharing details in today’s opera-tional practices is highly inefficient, cumbersome and re-quires often human intervention.We can automate signif-icant elements of cross-organizational security analysesvia event-oriented analysis scripts that reduce human in-volvement.

9 Conclusion

Analysis of network activity is fragmented across spaceand time. There are several fields of application thatwould significantly benefit from a unified view. We pre-sented the architecture of VAST, an intelligent distributeddatabase that processes network activity logs in a com-prehensive and coherent fashion. It accumulates datafrom disparate sources and provides a central vantagepoint to facilitate global analysis. We implemented a firstprototype that supports archival of events from multiplesources.

The design of VAST is based on Bro’s rich-typed event

model which provides a generic abstraction for activity.Due to its component-based design, the system can bescaled across multiple machines to distribute the workload. Further, we carefully implemented a concurrentstorage engine on top of FastBit to make full use ofmany-core machines.

Our initial performance analysis showed that the per-formance of the event layer needs further improvementto handle event rates of operational networks.

Acknowledgements

We would like to thank Robin Sommer for the numer-ous discussions about the system architecture. ChristianKreibich has also been a great help with his immediateresponse to the Broccoli issues we raised. Finally, wehighly appreciate the invaluable feedback of Vern Pax-son on this project.

References

[1] M. Allman, C. Kreibich, V. Paxson, R. Sommer,and N. Weaver. Principles for developing compre-hensive network visibility. In Proceedings of theWorkshop on Hot Topics in Security (HotSec), July2008.

[2] R. Bayer and E. M. McCreight. Organization andmaintenance of large ordered indexes. In Record ofthe 1970 ACM SIGFIDET Workshop on Data De-scription and Access, pages 107–141, Rice Univer-sity, Houston, Texas, USA, Nov. 1970. ACM.

[3] S. Chandrasekaran, O. Cooper, A. Deshpande,M. J. Franklin, J. M. Hellerstein, W. Hong, S. Kr-ishnamurthy, S. Madden, V. Raman, F. Reiss, andM. Shah. TelegraphCQ: Continuous Dataflow Pro-cessing for an Uncertain World. In Proceedingsof the 1st Biennial Conference on Innovative DataSystems Research (CIDR’03), Asilomar, CA, Jan-uary 2003.

[4] S. Chandrasekaran and M. J. Franklin. Remem-brance of Streams Past: Overload-Sensitive Man-agement of Archived Streams. In VLDB, pages348–359, 2004.

[5] B.-C. Chen, V. Yegneswaran, P. Barford, and R. Ra-makrishnan. Toward a query language for networkattack data. In ICDEW ’06: Proceedings of the22nd International Conference on Data Engineer-ing Workshops (ICDEW’06), page 28, Washington,DC, USA, 2006. IEEE Computer Society.

[6] E. Cooke, R. Mortier, A. Donnelly, P. Barham, andR. Isaacs. Reclaiming network-wide visibility us-ing ubiquitous endsystem monitors. In ATEC ’06:Proceedings of the annual conference on USENIX’06 Annual Technical Conference, pages 32–32,Berkeley, CA, USA, 2006. USENIX Association.

[7] P. J. Desnoyers and P. Shenoy. Hyperion: High Vol-ume Stream Archival for Retrospective Querying.In Proceedings of the 2007 USENIX Annual Tech-nical Conference, Santa Clara, CA, June 2007.

[8] L. Golab and M. T. Ozsu. Issues in data streammanagement. SIGMOD Rec., 32(2):5–14, 2003.

[9] Google perftools. http://code.google.com/p/google-perftools/.

[10] G. Iannaccone, C. Diot, and D. McAuley. TheCoMo white paper. Technical Report IRC-TR-04-017, Intel Research, Sept. 2004.

[11] S. Kornexl, V. Paxson, H. Dreger, A. Feldmann,and R. Sommer. Building a time machine for ef-ficient recording and retrieval of high-volume net-work traffic. In IMC’05: Proceedings of the In-ternet Measurement Conference 2005 on InternetMeasurement Conference, pages 23–23, Berkeley,CA, USA, 2005. USENIX Association.

[12] C. Kreibich and R. Sommer. Policy-controlledevent management for distributed intrusion detec-tion. In ICDCSW ’05: Proceedings of the FourthInternational Workshop on Distributed Event-Based Systems (DEBS) (ICDCSW’05), pages 385–391, Washington, DC, USA, 2005. IEEE ComputerSociety.

[13] libpcap. http://www.tcpdump.org.

[14] G. Maier, R. Sommer, H. Dreger, A. Feldmann,V. Paxson, and F. Schneider. Enriching networksecurity analysis with time travel. In SIGCOMM’08: Proceedings of the 2008 conference on Ap-plications, technologies, architectures, and proto-cols for computer communications, New York, NY,USA, August 2008. ACM Press.

[15] S. McCanne and V. Jacobson. The BSD Packet Fil-ter: A New Architecture for User-level Packet Cap-ture. In Proc. Winter USENIX Conference, 1993.

[16] E. O’Neil, P. O’Neil, and K. Wu. Bitmap in-dex design choices and their performance impli-cations. In IDEAS ’07: Proceedings of the 11thInternational Database Engineering and Applica-tions Symposium, pages 72–84, Washington, DC,USA, 2007. IEEE Computer Society.

[17] P. E. O’Neil. Model 204 architecture and perfor-mance. In Proceedings of the 2nd InternationalWorkshop on High Performance Transaction Sys-tems, pages 40–59, London, UK, 1989. Springer-Verlag.

[18] V. Paxson. Bro: A System for Detecting Net-work Intruders in Real-Time. Computer Networks,31(23–24):2435–2463, 1999.

[19] F. Reiss, K. Stockinger, K. Wu, A. Shoshani, andJ. M. Hellerstein. Enabling real-time querying oflive and historical stream data. In SSDBM ’07:Proceedings of the 19th International Conferenceon Scientific and Statistical Database Management,page 28, Washington, DC, USA, 2007. IEEE Com-puter Society.

[20] M. Siekkinen, E. W. Biersack, G. Urvoy-Keller,V. Gol, and T. Plagemann. Intrabase: Integrated

traffic analysis based on a database managementsystem. In E2EMON ’05: Proceedings of the End-to-End Monitoring Tecques and Services on 2005.Workshop, pages 32–46, Washington, DC, USA,2005. IEEE Computer Society.

[21] M. Stonebraker, U. Cetintemel, and S. Zdonik.The 8 requirements of real-time stream processing.SIGMOD Rec., 34(4):42–47, 2005.

[22] K. Wu. Fastbit: an efficient indexing technologyfor accelerating data-intensive science. Journal ofPhysics: Conference Series, 16:556–560, 2005.

[23] K. Wu, E. Otoo, and A. Shoshani. On the per-formance of bitmap indices for high cardinality at-tributes. In VLDB ’04: Proceedings of the Thirtiethinternational conference on Very large data bases,pages 24–35. VLDB Endowment, 2004.

[24] K. Wu, E. J. Otoo, and A. Shoshani. Optimizingbitmap indices with efficient compression. ACMTrans. Database Syst., 31(1):1–38, 2006.