Embedded Virtual Machines: Architecture Description,

Technical Report ESE-2009-01

Miroslav Pajic and Rahul Mangharam

September, 2009

Abstract

Embedded wireless networks have largely focused on open-loop sensing and monitoring.To address actuation in closed-loop wireless control systems there is a strong need to re-thinkthe communication architectures and protocols for reliability, coordination and control. Asthe links, nodes and topology of wireless systems are inherently unreliable, such time-criticaland safety-critical applications require programming abstractions where the tasks are assignedto the sensors, actuators and controllers as a single component rather than statically mappinga set of tasks to a speci�c physical node at design time. To this end, we introduce the Embed-ded Virtual Machine (EVM), a powerful and �exible programming abstraction where virtualcomponents and their properties are maintained across node boundaries. In the context of pro-cess and discrete control, an EVM is the distributed runtime system that dynamically selectsprimary-backup sets of controllers to guarantee QoS given spatial and temporal constraintsof the underlying wireless network. The EVM architecture de�nes explicit mechanisms forcontrol, data and fault communication within the virtual component. EVM-based algorithmsintroduce new capabilities such as predictable outcomes and provably minimal graceful degra-dation during sensor/actuator failure, adaptation to mode changes and runtime optimizationof resource consumption. Through the design of a natural gas process plant hardware-in-loop simulation we aim to demonstrate the preliminary capabilities of EVM-based wirelessnetworks.

1 Introduction

Automation control systems form the basis for signi�cant pieces of our nation's critical infrastruc-ture. Time-critical and safety-critical automation systems are at the heart of essential infrastruc-tures such as oil re�neries, automated factories, logistics and power generation systems. Discreteand process control represent an important domain for real-time embedded systems with over atrillion dollars in installed systems and $90 billion in projected revenues for 2008 [1].

In order to meet the reliability requirements, automation systems are traditionally severelyconstrained along three dimensions, namely, operating resources, scalability of interconnectedsystems and �exibility to mode changes. Oil re�neries, for example, are built to operate withoutinterruption for over 25 years and can never be shutdown for preventive maintenance or upgrades.They are built with rigid ranges of operating throughput and require a signi�cant re-haul to adaptto changing market conditions. This rigidity has resulted in proprietary systems with limited scopefor re-appropriation of resources during faults and retooling to match design changes on-demand.For example, automotive assembly lines lose an average of $22,000 per minute of downtime [2]during system faults. This has created a culture where the operating engineer is forced to patch afaulty unit in an ad hoc manner which often necessitates masking certain sensor inputs to let theoperation proceed. This process of unsystematic alteration to the system exacerbates the problemand makes the assembly line di�cult and expensive to operate, maintain and modify.

Embedded Wireless Sensor-Actuator-Controller (WSAC) networks are emerging as a practicalmeans to monitor and operate automation systems with lower setup/maintenance costs. While

1

Figure 1: (a) A wireless sensor, actuator and controller network. (b) Algorithm assignment to a setof controllers, each mapped to the respective nodes. (c) Three Virtual Components, each composed ofseveral network elements

the physical bene�ts of wireless, in terms of cable replacement, are apparent, automation manu-facturers and plant owners have increasing interest in the logical bene�ts.

With multi-hop WSAC networks, it is possible to build modular systems which can be swappedout for o�-line maintenance during faults. Modular systems can be dynamically assigned tobe primary or backup on the basis of available resources or availability of the desired calibra-tion. Modularity allows for incremental expansion of the plant and is a major consideration inemerging economies. WSAC networks allow for runtime con�guration where resources can bere-appropriated on-demand, for example when throughput targets change due to lower price elec-tricity during o�-peak hours or due to seasonal changes in end-to-end demand.

While WSAC networks facilitate both planned and unplanned mode changes, runtime pro-grammable WSAC networks allow for �exible item-by-item process customization. For example,a high demand for fuel-e�cient Toyota Prius' will require major retooling of a traditional wiredfactory that is designed for the Toyota Camry chassis. With re-programmable WSAC, the assem-bly line stations can adapt to a schedule where every 3 Camrys are interleaved with 2 Prius' withsynchronized changes in operation modes and assembly line operations.

1.1 Embedded Virtual Machines

The current generation of embedded wireless systems has largely focused on open-loop sensing andmonitoring applications. To address actuation in closed-loop wireless control systems there is astrong need to re-think the communication architectures and protocols for reliability, coordinationand control. As the links, nodes and topology of wireless systems are inherently unreliable, suchtime-critical and safety-critical applications require programming abstractions where the tasks areassigned to the sensors, actuators and controllers as a single component rather than staticallymapping a set of tasks to a speci�c physical node at design time (see Fig. 1). Such wirelesscontroller grids are composed of many wireless nodes, each of which share a common sense ofthe control application but without regard to physical node boundaries. Our approach, as shownin Fig. 2(a), is to decouple the functionality (i.e. tasks) from the inherently unreliable physicalsubstrate (i.e. nodes) and allow tasks to migrate/adapt to changes in the underlying topology.

To this end, we introduce the Embedded Virtual Machine (EVM), a powerful and �exibleprogramming abstraction where virtual components and their properties are maintained acrossnode boundaries. EVMs di�er from classical virtual machines (VM). In the enterprise or on PCs,one (powerful) physical machine may be partitioned to host multiple virtual machines for higherresource utilization. On the other hand, in the embedded domain, an EVM is composed acrossmultiple physical nodes with a goal to maintain correct and high-�delity operation even underchanges in the physical composition of the network. The goal of the EVM is to maintain a set of

2

functional invariants, such as a control law and para-functional invariants such as timeliness con-straints, fault tolerance and safety standards across a set of controllers given the spatio-temporalchanges in the physical network. By incorporating EVMs in existing and future wireless automa-tion systems, our aim is to realize:

1. Predictable outcomes in the presence of controller failure. During node or link faults, EVMalgorithms determine if and when tasks should be reassigned and provide the mechanisms fortimely state migration.

2. Provably minimal QoS degradation without violating safety. In the case of (unplanned)topology changes of the wireless control network, potential safety violations are routine occurrencesand hence the EVM must reorganize resources and task assignments to suit the current resourceavailability (i.e. link bandwidth, available processing capacity, memory usage, sensor inputs, etc.).

3. Composable and recon�gurable runtime system through synthesis In the EVM approach,a collection of sensors, actuators and controllers make a Virtual Component as shown in Fig. 1.A Virtual Component is a composition of interconnected communicating physical componentsde�ned by object transfer relationships. At runtime, nodes determine the task-set and operatingpoints of di�erent controllers in the Virtual Component. This machine-to-machine coordinationrequires task-set generation, task migration and remote algorithm activation which are executedvia synthesis at runtime.

4. Runtime Resource Re-appropriation and Optimization for dynamic changes in service.For planned system changes such as a factory shift, increase in output or retooling for a di�erentchassis, nodes are required to be re-scheduled in a timely and work conserving manner.

1.2 Challenges with Wireless Control

While there has been considerable research in the general area of wireless sensor networks, a ma-jority of the work has been on open-loop and non-real time monitoring application. As we extendthe existing programming paradigm to closed-loop control applications with tight timeliness andsafety requirements, we identify four primary challenges with the design, analysis and deploymentof extending such networks:

1. Programming motes in the event-triggered paradigm is tedious for control

networks. It is hard to provide any analytical bounds on the response time and stability oftasks in an event-driven regime [3, 4]. Real-time tasks are time-triggered while sensor inputs areevent-triggered. It is easier to incorporate sporadic tasks in a time-triggered regime than viceversa.

2. Programming of sensor networks is at the physical node-level where the tasks arebound to the node at compile-time. It is non-trivial to decompose a large control problem intode�ning components and applications for each mote.

3. Design of systems with �exible topologies is hard with physical node-level program-ming as the set of tasks (or responsibility) is associated with the physical node. Thus, a changein the link capacity, node energy level or connectivity in the current topology will render the ap-plication useless. It is necessary to associate a logical mapping of tasks to nodes and incorporatemechanisms to transfer responsibilities during physical and environmental changes in the network(as in Fig. 2(a)).

VT1 ...VT2 VTn

N1 N2 N3 ... Nm

(a) Decoupled virtual tasks and physicalnodes with runtime task mapping

29

Runtime Task Management

• Task_Migrate– Task instructions

– Stack

– Data

– Associate libraries

– Control/Schedule/Resource meta data

Node

1

Node

2

Current Task-set: T1, T2, T3 Current Task-set: T’1, T’2, …, T’7

Task_Migrate(T2)

(b) Task migration for real-time operation (instructions, stack,data & timing/fault tolerance meta-data) on one physical nodeto another.

Figure 2: Programmatic control with decoupled virtual tasks and physical controller nodes

3

4. Fault diagnostics, repair and recovery are manual and template-driven for amajority of networked control systems. In the case of WSAC networks, it is not plausible toexhaustively capture all possible faults at design time and thus provisions must be made forruntime diagnostics and recovery.

1.3 Overview of the EVM Approach

While wireless system engineers optimize the physical, link and network layers to provide an ex-pected packet error rate, this does not translate accurately to the stability of the control problem atthe application layer. For example, planned and unplanned changes in the network topology withnode/link failures are currently not easily captured or speci�able in the metrics and requirementsfor the control engineer. For a given plant connected to its set of controllers via wireless links it isnecessary that the controller process the sensor inputs and perform actuation within a boundedsampling interval. While one approach is to design specialized wireless control algorithms thatare robust to speci�ed range packet errors [5, 6], it is non-trivial to design the same for frequenttopological changes. Furthermore, it is di�cult to extend the current network infrastructure toadd/remove nodes and to redistribute control algorithms to suit environmental changes such asbattery drain for battery-operated nodes, increased production during o�-peak electricity pricing,seasonal production throughput targets and operation mode changes.

The EVM approach is to allow the control engineer to utilize the same wired-network controlalgorithms on the wireless network without knowledge of the underlying network protocols, node-speci�c operating systems or hardware platforms. The virtual machine executing on each nodewithin the virtual component instruments the VC to adapt and recon�gure to changes whileensuring the control algorithm is within its stability constraints. This approach is complementaryto the body of network control algorithms [5, 6] as it provides a logical abstraction of the underlyingphysical node topology.

2 EVM Design Flow

Our focus is on the design and implementation of wireless controllers and in providing suchcontrollers with runtime mechanisms for robust operation in the face of spatio-temporal topo-logical changes. Our focus is exclusively on robust controller design and not on sensors or ac-tuators, as the latter are largely physical devices with node-bound functionality. In order toenable control engineers to design wireless control systems in a manner that is both largely plat-form/protocol/hardware architecture independent and extensible to di�erent domains of controlsystems in process, discrete, aviation, medical, etc. we present a three-layered design process. Wenow describe the design �ow from control problem formulation, automatic translation of controlmodels to platform-independent EVM interpreter-based code and �nally to platform-dependentbinaries (see Fig. 2). These binaries are assigned to nodes within a VC using assignment andscheduling algorithms presented in [7]. The binaries are executed as virtual tasks within thenode-speci�c architecture described in Section 3.

At design time, control systems are usually designed using software tools, such as Mat-lab/Simulink, that incorporate both modeling and simulating capabilities. Therefore, in orderto automatize whole design �ow, the EVM is able to automatically generate functional modelsfrom the Simulink control system description to de�ne the processes by which input sampleddata is manipulated into output data for feedback and actuation. These functional models arerepresented by generated code and meta data that are platform and node independent system

description, thus providing the system designer to exclusively focus on the control problem design.From the functional description in the platform-independent and domain speci�c language,

the EVM design �ow automatically extracts additional para-functional properties, like timing,

inter task dependencies, etc from the Simulink model. All these properties, along with functionaldescription are used to de�ne a platform optimized binary for each virtual task (VT). VTs have

4

the option of static or dynamic binding depending on the capabilities and timing requirements forthe given control problem speci�cation.

2.1 Control Problem Synthesis

Control System Description

(Matlab/Simulink, C…)

Platform

IndependentDomain-specific EVM Description

(Common-EVM and Control-EVM)

Design

Time

Platform

Dependent Bin.

Format

Virtual Component Manager/RTOS

(nano-RK)

Figure 3: EVM Design �ow

A functional description of a VT is automaticallyextracted from the Simulink design, using thefact that each block can be represented as a com-position of other Simulink blocks. Thus, eachmodel can be presented as a hierarchical com-position of basic Simulink blocks.This organiza-tion of Simulink models allows for natural extrac-tion of a functional description using prede�nedwords from the EVM programming language dic-tionary (described in the next sub-section). Simi-larly, when a new block in Simulink is de�ned asa composition of previously de�ned blocks, forthe EVM functional description, a new word isdescribed using previously de�ned words, until alevel is reached where all words belong to theEVM dictionary. Therefore, a VT description is obtained using a parser that processes theSimulink .mdl �le as a .txt �le by searching for a new block de�nitions along with the inter-connections between blocks. Since each block in Simulink design has to have an unique name, astraightforward functional extraction would result in number of EVM words equal to the numberof used (sub)blocks. In order to avoid this, before each new word is de�ned the parser checkswhether a word with same structure already exist. This check can contain several hierarchicallevels and thus it can increase parser's execution time. An example of creation of a VT's func-tional description from a Simulink design model in Fig. 4(a) is presented in Fig. 4(b). As theplatform-independent language is stack-based, the notation is in reverse.

2.2 Platform Independent Domain Speci�c Language

In order to generate functional description of the designed system, the EVM programming languageis based on FORTH, a structured, stack-based, extensible programming language[8]. On the otherhand, since the goal of EVM design is to allow �exibility and designing utilities independentof chosen programming language, the intermediate programming language is not constrained toFORTH. This allows users to have a degree of freedom to replace Simulink with C, .NET, VB orother descriptions for control system implementation.

The use of the EVM intermediate programming language enables domain-speci�c con-

structs, where basic programming libraries are related to the type of application that is beingdeveloped. For example, for use in embedded wireless networks for industrial control, we devel-oped two prede�ned libraries, Common EVM ( Table 2), based on the standard FORTH library[8], and Control EVM that contains functionality widely used for developing control applications

out4

4

out3

3

out2

2

out1

1

Sum 3

Sum 2

Sum 1

Sum

Saturation 1

Saturation

R2

R1PID_controller 1

in out

PID_controller

in out

Fcn1

f(u)

Fcn

f(u)

in3

3

in2

2

in1

1

(a) Example of a control algorithm (VT) modeled inSimulink

: VTout1 SWAP f SUM ( x x – y ) ;: VTout2 NEGATE R1 SUM kp1 ki1 kd1 0 0 PID SATURATE ( x – x ) ;: VTout4 NEGATE R2 SUM kp ki kd 0 0 PID SATURATE ( x – x ) ;: VT_ID ROT DUP ROT VTout1 DUP VTout4 2SWAP SWAP VT out1 DUP

VTout4 2SWAP ( in1 in2 in3 – out1 out2 out3 out4 ) ;

(b) Generated code

Figure 4: Generation of EVM functional description from Simulink model

5

Table 1: Control EVMPID ( p KP KI KD addr_in addr_out - ) Proportional-Integral-Derivative controller

calculating p outputs in one invocation,result is updated directly in memory,

since previous values are neededPI ( p KP KI addr_in addr_out - ) Proportional-Integral controllerTF ( p m n addr_alpha addr_beta addr_in add_out - ) Transfer function described with mth (nth)

order polynomial in nominator (denominator)with coe�cients stored in α and β

LTI ( p m n addr_A addr_B addr_C a_in a_x a_out - ) LTI system, x[k + 1] = Ax[k] +Bu[k],y[k] = Cx[k], where p,m, n

are sizes of vectors n, y and x

(a list of all API's can be seen in Table 2 and Table 1). In addition the extensibility of EVMallows to de�ne the Automotive EVM, Aviation EVM or Medical EVM libraries that will containfunctionalities speci�c to each of these application �elds. Using these libraries code generatorcreates a system description from a prede�ned component, thus creating a task description �le foreach of the Virtual Tasks (VTs).

Timing parameters (period and worst-case execution time) are also extracted from the model.Since we consider only discrete controllers as potential VTs, Simulink design rules force the designerto de�ne a sampling rate for each discrete block, allowing us to extract period as least commondivider for all task timing. These values are also extracted from the model (.mdl) �le and addedto the VT's description �le, along with the functional description.

List of the Control EVM words is presented in Table 1. First three words speci�ed in the tableare used for the Singe-Input-Single-Output (SISO) systems. Although all these systems can bedescribed using the last word (LTI - describing Linear Time Invariant system), the fact that theyare widely exploited recommended its use.

6

Table 2: Common EVM: De�ne a wordArithmetic operations (on 16bit):

+ ( n1 n2 - n3 ) Adds n1 and n2, result returned on the stack- ( n1 n2 - n3 ) Subtract n2 from n1, result returned on the stack1+ ( n1 - n2) Add 1 to n1, return the result on the stack1- ( n1 - n2) Subtract 1 from n1, return the result on the stack* ( n1 n2 - n3 ) Multiply (signed) n1 and n2, result returned on the stack/ ( n1 n2 - n3 ) Divide n1 with n2, quotient returned on the stack/MOD ( n1 n2 - n3 n4 ) Divide n1 with n2, quotient n3 and reminder n4 returned on the stackMOD ( n1 n2 - n3 ) Divide n1 with n2, reminder returned on the stackL� ( x1 u - x2 ) Logical left shift of u places on x1, result returned on the stackL� ( x1 u - x2 ) Logical right shift of u places on x1, result returned on the stackA� ( x1 u - x2 ) Arithmetical left shift of u places on x1, result returned on the stackA� ( x1 u - x2 ) Arithmetical right shift of u places on x1, result returned on the stackABS ( n - |n|) Replace the top stack item with absolute valueMIN ( x1 x2 - x3 ) Lower of the two values is returned on the stackMAX ( x1 x2 - x3 ) Higher of the two values is returned on the stackLogical operations (on 16bit):

INV ( x1 - x3 ) Invert x1, result returned on the stackAND ( x1 x2 - x3 ) ands x1 and x2, result returned on the stackXOR ( x1 x2 - x3 ) xors x1 and x2, result returned on the stackOR ( x1 x2 - x3 ) ors x1 and x2, result returned on the stackNOR ( x1 x2 - x3 ) nors x1 and x2, result returned on the stackNAND ( x1 x2 - x3 ) nands x1 and x2, result returned on the stackComparison and testing:

0< ( n - �ag )0> ( n - �ag )0<> ( n - �ag )0= ( n - �ag )= ( n1 n2 - �ag )<> ( n1 n2 - �ag )> ( n1 n2 - �ag )< ( n1 n2 - �ag )Memory manipulation:

@ ( addr - x) fetch! ( x addr - ) writemalloc ( size - addr ) Allocation of 128B memory blockmfree ( addr - ) Frees 128B memory blockSensor/Actuator handling:

RDSensG ( sensID - n1 ) Read sensor value from the sensor described by the global sensor IDRDSensL ( sensID - n1 ) Read sensor value from the sensor described by the local sensor IDWRActG ( value actID - ) Write value to the actuator described by the global actuator IDWRActL ( value actID - ) Write value to the actuator described by the local actuator IDNetworking:

PktSendG ( addr n nodeID - ) Send n-byte message storedat address addr to node described by global ID

7

3 EVM Architecture

We now describe the node-speci�c architecture which implements the mechanisms for the virtualmachine on each node. The Common-EVM and Control-EVM description for each set of controllersare scoped within Virtual Tasks that are mapped at runtime by the the task Assignment Algorithmdescribed in [7]. This description is interpreted by the interpreter running on each node. The EVMruntime system is built on top of the nano-RK real-time operating system[9] as a supertask, allowingnode-speci�c tasks to execute natively and virtual tasks (VTs), i.e. those that are dynamicallycoupled with a node, to run within the EVM. The EVM services are shown in Fig. 6 and the EVMblock-level reference architecture is presented in Fig. 5. This allows EVM to maintain node speci�cfunctionalities and be extensible to runtime task evocation of existing or new virtual components.

The interface between nano-RK and all VTs is realized using the Virtual Component Manager(VCM) supertask. The VCM maintains local resource reservations (CPU, network slots, memory,etc.) within nano-RK, the local state of the VTs and global mapping of VTs within the virtualcomponent. The VCM is responsible for memory and network management for all VTs-to-physicalnodes and presents a mapping between local and remote ports which is transparent to all local VTs.The VCM includes a FORTH-like interpreter for generic and domain-speci�c runtime operationsand a Fault/Failure Manager (FFM) for runtime fault-tolerant operation.

3.1 nano-RK Real-Time OS

To address the need for timing precision, priority scheduling and �ne-grained resource managementthe nano-RK resource kernel [9] has been previously developed with timeliness as a �rst-classconcern. nano-RK is a fully preemptive RTOS with multi-hop networking support that runs ona variety of sensor network platforms (8-bit Atmel-AVR, 16-bit TI-MSP430, Crossbow motes,FireFly). All networking is conducted over the RT-Link [10] real-time link protocol that is nativeto nano-RK. It supports �xed-priority preemptive scheduling for ensuring that task deadlines aremet, along with support for and enforcement of CPU and network bandwidth reservations. Taskscan specify their resource demands and the operating system provides timely, guaranteed andcontrolled access to CPU cycles and network packets in resource constrained embedded sensorenvironments. It also supports the concept of virtual energy reservations that allows the OS toenforce energy budgets associated with a sensing task by controlling resource accesses. nano-RKprovides various medium access control and networking protocols including a low-power-listenCSMA protocol called B-MAC, an implicit tree routing protocol and RT-Link.

For networked control systems, it is essential that the underlying sensor operating system ex-pose precision timing, scheduled tasks and synchronized networking so that the trade-o�s betweenenergy-consumption (node lifetime), reliability and responsiveness are speci�able and enforceableboth at design-time and runtime. Support for the above services is required for low-duty cycle andenergy-constrained sensor networks too because the computation and communication are packedinto a short duration so all nodes maximize their common sleep time.

nano-RK has been design as fully static OS, con�gured at the design time. Therefore to allow

...

native tasks

nanoRK

Task nTask 1VCM

Interp.

Task

FFM

Task

VT 1

VT m

...

virtual tasks

Figure 5: EVM architecture with Virtual Component Manager running as a supertask along side nativenano-RK tasks

8

State Migration

Task

Motor Control

Task

Overload Detection

Task

Focus of EVM work

App

s K

erne

l H

ardw

are

Task Management

Peripheral Drivers

Microcontroller

Real-Time Scheduler

Reservations

Reserves Reserves Reserves

Adaptive Virtual Machine Runtime S

802.15.4 Radio

Time Sync RX Power Control

RX Buffer

TX Buffer

RT-Link

Network Management ystem

Parametric Control

Task Partitioning

Scheduleability Analysis

Software Attestation

Algorithm Activation

Protocol Adaptation

Policy Negotiation

Data Migration

Online Fault Diagnosis

Figure 6: nano-RK RTOS architecture with EVM extensions

parametric and programmatic run-time changes in the code nano-RK was redesigned and extendedwith several new features (see Fig. 6):• Runtime Parametric Control: Support for dynamically changing the sampling rates, runtimetask and peripheral activation/deactivation and runtime modi�cation to the task utilization hasbeen added. These facilities are exposed and executed via the Common-EVM programmer inter-face.• Runtime Programmatic Control: As a part of EVM design dynamic task migration has beenimplemented. This requires runtime schedulability analysis, capability checks to migrate a subsetof the task data, instructions, required libraries and task control block. Based on the Assignment[], tasks may be activated or migrated between primary and backup nodes. Such facilities aretriggered by the primary-backup policy implemented on top of the assignment algorithm.• Dynamic Memory Management: Both Best-�t and First-�t memory allocation methods aresupported. In addition a Garbage Collector (GC) has been designed to reclaim all memory seg-ments owned by tasks that had been terminated. The GC is scheduled only when its executiondoes not in�uence execution of other tasks. To allow for dynamic memory (re)allocation dur-ing code execution, a memory management mechanism was implemented with functions void ∗nrk_malloc(uint16_t) and void nrk_free(void ∗).

3.2 Virtual Component Interpreter

The Virtual Component Interpreter provides the EVM programmer an interface to de�ne and exe-cute all VTs. Every VT is de�ned as a word within the VCM library. When a new VT descriptionis received over the network, the VCM calls the interpreter which de�nes a new word using thedescription �le of a task and existing virtual component libraries. When a VT is activated, eachexecution of VT is implemented as a scheduled activation of the interpreter with the VT's word asan input. To allow preemptivity of the tasks, each call of the interpreter uses a VT-speci�c stackand dedicated memory segments. In addition, since each VT is capable of dynamically allocatingmemory during its execution in cases when there is no space in already dedicated memory, theEVM's memory manager allocates a new block of �xed size (currently 128B). Therefore, the inter-preter is designed to use logical addresses in form (block_index, local_address_inside_block).In order to be able to use the node's physical memory, the VCM supports address translation andgarbage collection for the interpreter.

Each node has a local copy of standard Common-EVM dictionary and Control-EVM dictio-naries. If a new word needs to be included in the existing library, upon reception of a messagethat contains the word, VCM initiates execution of the interpreter. The interpreter �rst checksthe global word identi�er and revision number in order to discard obsolete versions. If a newerversion is at hand, the interpreter (re)de�nes the word in the dictionary. Since all VTs executions

9

(a) Virtual Task's Description Table

Global IDProcessing Units neededDeployment constraints Type (Fixed/Flexible)

Number of nodes (n)Node 1

...Node n

Memory requirements Stack sizeNumber of Blocks

Network Dependencies Number of Tx slotsNumber of Rx slots

EVM description �le

(b) nanoRK TCB extension

Local IDCPU ReserveMemory requirements Stack size

1st Block pointer...

nth Block pointerNumber of Tx slots Pointer to slot

index arrayNumber of Rx slots Pointer to slot

index array

Table 3: Local and Global task description

are performed using the interpreter, there is no need for any additional kind of provider-subscriberrelation as described in [11]. In addition, potential failure modes that may result from inconsistentversions on each node are avoided using this simple approach. In the case when a when a nodereceives an update for a word used to de�ne a task, it checks the revision number in the de�nition�le of the task in the case of a newer revision, after updating de�nition of the word, the interpreterinforms the VCM to update the VT's description �le. Each local update, also updates the revisionnumber of the VT word, therefore keeping VT fully updated without requiring consensus acrossthe VC.

3.3 Virtual Tasks

User Defined

- Common EVM

- Control EVM

INTERPRETER

Standardized

- Message format

...

Figure 7: VCM/Interpreter relation

Each VT is described using Virtual Task's De-scription Table, comprised of a global and localdescription of a VT (see Table 2(a)). Copies ofthe Descriptor Table are stored on all membersof the VC assigned within the VT. While this re-quirement for consistency currently results in anissue of scalability, the current project with Hon-eywell Process Control (OneWireless) requireshigh-speed control networks with less than 20nodes and is hence within the practical limitsof the current approach. Each task speci�es itsDeployment constraints which describe speci�cdistribution requirements of a task as to whethera task is Fixed, meaning that it has to run ona speci�c subset of physical nodes, or Flexible,i.e. a task can be deployed on any physical nodein the VC. If a task is declared as Fixed, the number of the suitable nodes is speci�ed along withthe node IDs. Each VT's global description has information about memory requirements, stacksize and number of �xed size memory blocks (e.g.128B) that are used. In addition to the abovemeta data, Network requirements in terms of number of active RT-Link transmit and receive slotsare speci�ed at design time. The above descriptors are speci�ed as within the VCM's Task ControlBlock (TCB) for each task. The EVM TCB is an extension to the native nano-RK TCB[9].

Upon VT de�nition on a node, VCM creates Task Control Block (TCB) for the task. Besideparts of native TCB (more details can be found in [6]) VCM initializes the TCB extension presentedin Table 2(b) which describes local resources used by the VT.

10

3.4 Virtual Component Manager

The fundamental di�erence between native nano-RK and the VCM is that the scope of nano-RK'sactivities is local, node-speci�c and de�ned completely at design time, while the scope of theVCM is the Virtual Component that may possibly span multiple physical nodes. The current setof functionalities supported is:

1. Virtual Task handling:1.1 VC state is maintained, keeping every node with a consistent mapping of VTs. The VCMin each controller node within the VC periodically broadcasts its information about VT mappingin order to keep consistency between all members of the VC. Currently a centralized consensusprotocol is used and a distributed consensus protocol is needed to scale operations.1.2 Task migration that can be triggered as a result of a fault/failure procedure or by request ofeither the VT or VCM. As a part of a task migration, task's VTDT is sent along with all memoryblocks used by the task. If the VT is already de�ned on a node (checked by exchange of hashvalues), only task parameters are exchanged. In addition, due to physical limitations (network,di�erences between controller/actuator nodes, etc) before migrating VT to a particular node, net-work and CPU schedulability analysis are performed for nodes that are potential candidates. Ifanalysis show that no node can execute the task correctly, an error message is returned.1.4 VT activation: After a VT is de�ned, the VCM performs local CPU and network schedulabil-ity analysis prior to task activation. This is done in order to ensure that task will not adverselya�ect correct execution of previously de�ned VTs.1.5 Control of tasks executed on other nodes: For all VTs in Backup mode, the VCM shadows theexecution of the VT in Primary mode. If a departure from the desired operation is observed (i.e.low battery level, decreased received packet signal strength), Backup nodes may be assigned toPrimary mode based on policy.

2. Network Management:2.1 Transparent radio interface: Using the message header which contains information about mes-sage type, the VCM determines which task should be informed about message arrival. Messagescontaining tasks and their parameter de�nitions are �rst processed by a VCM, before the VCMactivates the Interpreter (for remote ports). All local task-speci�c messages are transferred totheir respective local ports.2.2 Logical-to-physical address mapping: All communication between VTs is via the VCM. Sincea VT does not have information on which nodes other VTs are deployed, the VCM performslogical-to-physical address mapping. In cases when both tasks are on the same node, the VCMdirectly passes a message to a receiving task bu�er.

4 Implementation

To evaluate the EVMs performance in a real setting with multiple coordinated controller opera-tions, we used a factory simulation module (FischerTechnik model factory). The factory consistsof 22 sensors and actuators that are to be controlled in a coordinated and timely manner. A blockof wood is passed through a conveyor, pushed by a rammer on to a turn table and operated uponby up to three milling/cutting/pneumatic machines. The factory module was initially controlledby wired programmable logic controllers (PLCs). We converted it to use wireless control withFireFly nodes controlling all sensors and actuators via a set of electical relays. FireFly [12] is alow-cost, low-power, platform based on the Atmel ATmega1281 8-bit micro-controller with 8KBof RAM and 128KB of ROM along with a Chipcon CC2420 IEEE 802.15.4 standard-compliantradio transceiver. FireFly nodes support tight global hardware-based time synchronization forreal-time TDMA-based communication with the RT-Link protocol [10]. The EVM also workson TI MSP-EXP430F5xxx and MSP430FG4xxx architectures for more e�cient execution of theFORTH-like stack-based language on a Von Neumann architecture. We have demonstrated:1. On-line capacity expansion when a node joins the VC

11

2. Redistribution of VTs when adding/removing nodes3. Planned VT migration triggered by the user4. Unplanned VT migration as a result of node failure or communication link failure5. Multiple coordinated work-�ows

In a second experiment setup we considered multiple concurrent work�ows, where each work-�ow uses a subset of the actuators. Each new block type triggers the assignment of the associatedVT and migrates from one controller to the next as the block physically moves through the factory.Therefore, with every new type of block a recon�guration of the VC is performed. We have testedthe setup with a batch of 9 input blocks consisting of 3 di�erent types. This is an example of thelogical bene�ts of the EVM as it enables a more agile form of manufacturing. Details on bothexperiments with videos can be seen in [13].

5 Related work

There have been several variants of virtual machines, such as Maté [4], Syclla [14] and SwissQM[15],and �exible operating systems, such as SOS[11], Contiki[16], Mantis[17], Pixie[18] and LiteOS[19],for wireless sensor networks. The primary di�erences that set EVM apart from prior work is thatit is centered on real-time operation of controllers and actuators. Within the design of EVM'soperating system, link protocol, programming abstractions and operation, timeliness is a �rst-classcitizen and all operations are synchronized. The EVM does not have a single node-perspectiveof mapping operations to one virtualized processor on a particular node but rather maintainscoordinated operation across a set of controllers within a virtual component. Furthermore, EVMhas been designed to operate in an autonomous manner and to be extensible at runtime.

One of the �rst virtual machines (VM) for sensor networks was Maté. Maté implementsa simple, communication-centric VM built on top of the TinyOS [3]. It is designed as a highlevel interface where code is written using limited instruction set, de�ned at design-time, andexecuted with a FORTH-like interpreter. EVM utilizes a similar FORTH-like interpreter but isextensible at runtime and allows for fully preemptive tasks. Syclla is a more conventional systemVM that allows code mobility by providing a virtualized processor abstraction on each node.It uses a standard approach where one physical machine exposes interfaces to a single logicalmachine. On the other hand, EVM uses several physical nodes and allows user to consider thevirtual component as a single logical entity. SOS operating systems for sensor nodes [11] wasdesigned to allow �exible operation of nodes in a sensor network. As TinyOS, it has an event-drive execution of all components, with a di�erence that components can be installed/modi�edduring runtime. To allow this, SOS uses dynamically-loaded modules while kernel implementsmessaging, dynamic memory, and module loading and unloading. EVM allows for dynamic taskevocation and runtime function loading but in addition has mechanism for interaction with othernodes, which SOS lacks. In addition, SOS utilizes a signi�cant portion of the TinyOS codeand all tasks are run-to-completion. EVM is built on the nano-RK sensor RTOS and hence alltasks are scheduled by �xed priority scheduling and are fully preemptable. Finally, the VirtualNode Layer [20] provides a programing abstraction for a virtual infrastructure where each virtualnode is identi�ed with a particular region and it is emulated by one of the physical nodes in itsregion. In last few years several di�erent systems for macro-programming in WSN have beendeveloped[21, 22, 23]. Welsh et al. [21] have de�ned a set of abstractions representing localcommunication between nodes in order to expose control over resource consumption along withproviding feedback on its performance. EVM along with data sharing incorporates performancetracking of algorithm execution and automatically readjust available resource in order to maximizesystem robustness. An extension of these ideas is used to develop the Regiment [23], a high-levellanguage based on the functional reactive programming. The Regiment uses two types of objects,Signals used to represent sensor states and Regions, collection of signals, where membership inRegion can vary in time. In addition the authors de�ne operators for creating new regions alongwith three basic operations (applying function to a signal and aggregating values in the region).EVM is not a generic macroprogramming system as it focuses on closed-loop control and actuation

12

problems with native support for task migration and on-line task assignment. Kairos [22] allowsa programmer to describe code execution for each of the nodes in a network using a centralizedapproach where details about code generation, remote data access and management along withnode interactions are hidden from the programmer. Kairos uses three constructs: remote dataaccess, iterating through 1-hop neighbors and addressing arbitrary nodes. Basic of Kairos isa data sharing algorithm which assumes eventual consistency between shared memories on thenodes. Although EVM provides consensus between shared memories store on di�erent nodes witha shared memory access for all nodes in the networks its list of features is not only limited to that.

References

[1] Frost and Sullivan, North American Sensor Markets, Technical Report A-761-32, 2004.

[2] Nielsen Research, Downtime Costs Auto Industry, March 2006.

[3] J. Hill and et al. System architecture directions for network sensors. ASPLOS, 2000.

[4] P. Levis and D. Culler. Mate: A tiny virtual machine for sensor networks. ACM ASPLOS-X, 2002.

[5] J. P. Hespanha, P. Naghshtabrizi, and Y. Xu. A survey of recent results in networked control systems.Proc. of the IEEE, 2007.

[6] W. Zhang, M.S. Branicky, and S.M. Phillips. Stability of networked control systems. IEEE ControlSystems Magazine, 2001.

[7] M. Pajic and R. Mangharam. The Assignemnt Problem. http://mlab.seas.upenn.edu/evm, 2009.

[8] L. Brodie. Starting FORTH. 2nd Ed., Prentice-Hall. ISBN 0-13-843079-9, 1987.

[9] nano-rk sensor rtos. http://nanork.org.

[10] A. Rowe, R. Mangharam, and R. Rajkumar. RT-Link: A Time-Synchronized Link Protocol forEnergy-Constrained Multi-hop Wireless Networks. IEEE SECON, 2006.

[11] C.C. Han and et.al. SOS : A Dynamic Operating System for Sensor Nodes. ACM Mobisys, 2005.

[12] R. Mangharam and et.al. FireFly: A Cross-layer Platform for Real-time Embedded Wireless Net-works. Real-Time System Journal, 2007.

[13] EVM website - http://mlab.seas.upenn.edu/evm, 2009.

[14] P. Marbell and L. Iftode. Scylla: A smart virtual machine for mobile embedded systems. InWMCSA,2000.

[15] R Muller, G Alonso, and D Kossmann. A virtual machine for sensor networks. ACM EuroSys, 2007.

[16] A. Dunkels, B. Gronvall, and T. Voigt. Contiki - A Lightweight Operating System for Tiny NetworkedSensors. IEEE LCN, 2004.

[17] Shah Bhatti and et. al. MANTIS OS: an embedded multithreaded operating system for wirelessplatforms. Mob. Netw. Appl., 2005.

[18] K. Lorincz and et. al. Resource aware programming in the Pixie OS. ACM SenSys, 2008.

[19] Q. Caoand and et. al. The LiteOS: Towards Unix-Like Abstractions for Wireless Sensor Networks.IEEE IPSN, 2008.

[20] M. Brown, S. Gilbert, N. Lynch, C. Newport, T. Nolte, and M. Spindel. The Virtual Node Layer: AProgramming Abstraction for Wireless Sensor Networks. WWSNA, 2007.

[21] M. Welsh and G. Mainland. Programming Sensor Networks using Abstract Regions. NSDI, 2004.

[22] R. Gummadi et al. Macro-programming Wireless Sensor Networks Using Kairos. In Dist. Comp. inSensor Sys., 2005.

[23] R. Newton et al. The Regiment Macroprogramming System. ACM IPSN, 2007.

13