the new msr plugin download environment

WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

[email protected] http://www.arl.wustl.edu/~jdd

The New MSRPlugin Download

Environment

2WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

John DeHart - 04/21/23

First: The Old Way• Compile plugin code on NetBSD• Copy plugin directory onto File System

destined for SPC• Boot SPC• Log in to SPC

– requires user space and serial line or telnet access

• Use modload(1) on SPC to load plugin– links plugin against kernel in /netbsd on SPC

• Use modstat(1) on SPC to see list of loaded plugins



The New Way• Compile plugin code on NetBSD• Verify plugin has no undefined symbols

– There is no link against running kernel anymore– msr_lkm.h provides macros to needed kernel fucntions

and structures• We can add more as we need them.

• Use pluginDownload utility from CP (Linux) to download plugin to SPC– Not sure I’ve even tried it on NetBSD…

• Use sendcmd utility from CP to configure plugin, filters, etc

• Use pluginStat utility from CP to view list of loaded plugins



pluginDownload• Uses gnu binary utilities cross-linker

– Runs on Linux, builds a NetBSD binary– Used to:

• prelink to get size information• final link to relocate

• Uses new AAL5 control data channels to download plugin directly into kernel memory.– uses sendcmd API to:

• allocate kernel memory for plugin• set up AAL5 data channel• call plugin entry function to complete load operation

• Side Note: sendcmd API– should be expanded to include all of sendcmd, but right

now we have about 15 “commands”.



• The rest of the slides are old notes



Plugin Loading Steps• Compile Plugin code on NetBSD• Do one of these two, currently Linux seems to work better for later steps:

– Pre-link Plugin on NetBSD to get size info ready ld -e example -T 0 -o pl_combined combined.o

– Pre-link Plugin on Linux to get size info ready ld-cross –o pl_combined.o –e _example –Ttext 0 combined.o



Plugin Loading Steps• On Linux CP: open file (pl_combined.o), get size information and allocate MSR/SPC

memory for plugin sendcmd –p # -c rp_pcu –s allocate –d <size> -d <sym_size> -d <sym_symsize>

• sendcmd code in kernel calls msr_lkmiotcl(LMRESERV, <ptr> , FWRITE)• <ptr> points to a (struct lmc_reserv) which passes in:

– size = a_text + a_data + a_bss (all from a.out header)» but we don’t need to load the bss since it is all 0

– sym_size = a_syms + stb.st_size – N_STROFF(info_buf)» a_syms is symbol table size» string table is at the end so, the string table size is full file size (stb.st_size minus string tab offset)

– sym_symsize = a_syms• and returns:

– address of block of memory allocated in kernel (resrvp->addr)– slot in lkm table (resrvp->slot)– address of symbols (resrvp->sym_addr)

• msr_lkmioctl() returns 0 if successful, non-zero if there was an error.– sendcmd will return

• Kernel address (PlAddr) for plugin to be loaded at• Slot number• Address of location of symbols (what do we use this for?)

– This is used for adjusting the string table pointers before they are loaded (see modload.c for details)

– useful command: objdump



Plugin Loading Steps (continued)• On Linux CP: re-link plugin to relocate to PlAddr

ld-cross –o plugin –e _example –Ttext <PlAddr+0x20> combined.o• relocates to plAddr+0x20 and sets ‘_example’ as the entry point

– the +0x20 is because we are currently downloading the a.out header– This sets the a_entry field in the a.out header of the file

• file plugin is now ready for downloading• On Linux CP: request two data channels to MSR Kernel

– One for plugin text and data: sendcmd –p # -c dchan -s get -d <PlAddr> -d <size>

– Another for the plugin symbols (sym_addr was returned by allocate command earlier)

sendcmd –p # -c dchan -s get -d <sym_addr> -d <sym_size>– sendcmd returns the dchan# in its responses

• On Linux CP: download file plugin into MSR/SPC memory: pluginDownload -f plugin –S <sym_addr> -d <dchan#1> -d <dchan#2> -V <vci> -L

<lkm slot>– This needs to adjust the string table before downloading. Thus it needs the

sym_addr which was returned by the allocate sendcmd above– text and data go to dchan#1 and symbol and string table to go dchan#2



Plugin Loading Steps (continued)• On Linux CP: Finish the loading of the plugin

sendcmd –p # -c rp_pcu –s load -d <entry_addr> -d <size> -d <sym_size>– I don’t think we actually need the PlAddr here.– I think in this command (or an earlier one) we do need to send the entry

symbol address– We are also sending the amount of data sent for the plugin (size) and symbols

(sym_size)• the msr_lkm code will use these two values to update

– curp->offset– curp->sym_offset

– This sendcmd causes a call to msr_lkmioctl(LMREADY, <data>, FWRITE)– We may also need to do some other stuff. See page of ioctl’s used by modload.

• On Linux CP: release the data channel sendcmd –p # -c dchan -s release -d <dchan#>

• When all done with plugin:– On Linux CP: unload and deallocate the plugin

sendcmd –p # -c rp_pcu –s unload –d <id>

• We should never have to use the LMLOADBUF and LMLOADSYMS cmds



Data Transfer• Options for the download phase, CP side:

– hack up modload from NetBSD and build it on Linux• modload uses ioctl’s to load the module into the local kernel• we want to send AAL5 frames

– modify Stage3 of AAL5_download• Stage3 already processes NetBSD a.out files • Need to do one of these:

– change Stage3’s interaction with Stage2, i.e. have it not expect any acks– add a Stage2 like thing to the MSR kernel for accepting plugin frames

– Merge modload and Stage3• modload does some manipulations to get the string table correct• Stage3 has all the code to send AAL5 frames and wait for ACK/NACK

• Options for the downloading phase, MSR side:– Build a new AAL5 control data channel structure

• Uses AAL5 frame buffer pool that the IP fwding channels use• Re-uses some of the frame processing code in kernel• Arriving frames are copied into previously allocated memory associated with the data

channel.– It would be nice if APIC descriptors pointed directly at the allocated memory, but that

breaks the current model and would make it hard to build in safeguards against corruption.



Data Transfer (continued)

VaddrStart of Virtual address block where data frames should go (32b)

LengthLength of block in bytes (32b)

Next AddrWhere next frame should go. Initially set to <Vaddr> (32b)

Seq #Next expected frame sequence number (first frame should be 1) (16b)Index(dchan#)

0

1

. . .

DCHAN_MAX

MSR Kernel Dchannel Table



Data Transfer (continued) MSR Control Data Channel RATM AAL5 Frame Format

(as it appears in the MSR Kernel)

dchan#(8bits) seq# (16bits)

framelength (16bits) pad (16bits)

AAL5 Trailer (8 Bytes)

AAL5 Frame Data (<= 1992 Bytes)

Shim Reserved space (8 Bytes)

1992

B 2000

B

2016

B

APIC Rx descpoints here flags (8bits)

flags:1: Data2: Ack4: Nack



External Symbols in Plugin• We will implement a structure to contain pointers to Kernel

symbols that Plugins are allowed to use.– This structure will surely grow over time as we discover more kernel

functions that we want to expose to Plugins• When a plugin is loaded, it will be given the pointer to this

structure in the kernel via the call to its entry function• The plugin will therefore not need to link against the running

kernel to resolve symbols.• This strategy will achieve two goals:

– Put bounds on what plugins can do in the kernel– Make the linking and downloading of plugins easier



External Symbols in Plugin (continued)• Current kernel functions used in example plugin:

– malloc related:• free()

– via FREE() macro from sys/malloc.h• malloc()

– via MALLOC() macro from sys/malloc.h– MSR Related:

• msr_printf()• PCU Related:

– pcu_deregister_class()– pcu_free_all_instances()– pcu_register_class()

– LKM Related:• lkm_nofunc()• lkm_dispatch()

– via DISPATCH() macro from sys/lkm.h• lkm_exists()



The rest of the slides aresupporting notes

some from other places



Data Transfer• Options for the download phase, CP side:

– hack up modload from NetBSD and build it on Linux• modload uses ioctl’s to load the module into the local kernel• we want to send AAL5 frames

– modify Stage3 of AAL5_download• Stage3 already processes NetBSD a.out files • Need to do one of these:

– change Stage3’s interaction with Stage2, i.e. have it not expect any acks– add a Stage2 like thing to the MSR kernel for accepting plugin frames

• Options for the downloading phase, MSR side:– When we get a request to load a plugin of size <size>

sendcmd –p # -c rp_pcu –s allocate –d <size>• Allocate an APIC buffer of that size• associate the plugin load descriptor with that buffer• Tie the descriptor to the plugin load VCI• Go

– Once we have the plugin in the allocated buffer:• in response to a

sendcmd –p # -c rp_pcu –s load –d <PlAddr> -S “example • use calls to lkmioctl() to do the rest



Implementing modload in MSR• Issues:

– Where will we download plugin from?• presumably just from the CP

– Will the plugin on the CP already be pre-linked with the current MSR kernel?

• if not, we may have Linux vs. NetBSD problems.• if so

– how do we plan to keep plugins and MSR kernels in sync– how do we know the address where the plugin will go in the kernel

– Dynamic scheme:• allocate memory on demand in the kernel for a new plugin

– Static scheme:• predefined slots at predefined addresses in kernel• Each plugin is built to fit in a specific slot at a predefined address• Still need to resolve symbols with actual kernel

– if we allow this at all…



Implementing modload in MSR• Option 1: Use sendcmd utility to execute modload on

SPC– Download plugin binary to a file on MSR

• e.g. /var/run/rp/plugin/fileABC.o– and run modload

• e.g. modload –o /var/run/rp/plugin/fileABC.o –e example fileABC.o– Something like:

sendcmd –p # –c rp_pcu –s load –S “fileABC.o” –S “example”• uses proposed new option –S for including strings• causes file “fileABC.o” to be downloaded to MSR and loaded as

kernel module named “example”– Requires presence on MSR file system of:

• /sbin/modload• /usr/bin/ld• up to date symbol version of kernel being executed



Implementing modload in MSR• Option 2: Prelink on CP, use sendcmd to download directly into

memory– Link plugin against copy of kernel on CP ld -A netbsd.MSR -e _<entry> -o <outfile> -T <addr=0> <module>– Reserve Kernel memory:

sendcmd –p # –c rp_pcu –s allocate –d <size>• returns kernel load address

– Relink at kernel load address (LA) ld -A netbsd.MSR -e _<entry> -o <outfile> -T <addr=LA> <module>

– Open relinked module and load it into kernel sendcmd –p # –c rp_pcu –s load –S <module>• uses proposed new option –S for including a strings

– Adjusting symbol table entry pointers and load it.• What does this actually do in modload(1)

– sync();• What does this actually do in modload(1)

– Call the module’s entry function to test it.– Post-install if called for...

• ioctl(lkm_fd, LMSTAT, ...)



Implementing modload in MSR• Option 3: Prelink on CP, Plugin must use kernel function

pointer array to access predefined, limited number of kernel functions, then use sendcmd to download directly into memory– plugin code uses local pointer for function pointer array which will be

reset later when kernel calls entry function– Reserve Kernel memory:

sendcmd –p # –c rp_pcu –s allocate –d <size>• returns kernel load address

– Relink at kernel load address (LA) ld -A netbsd.MSR -e _<entry> -o <outfile> -T <addr=LA> <module>

– Open relinked module and load it into kernel sendcmd –p # –c rp_pcu –s load –S <module>• uses proposed new option –S for including strings

– Call the module’s entry function to test it.• kernel will provide the actual location of the function pointer array

– Post-install if called for...• ioctl(lkm_fd, LMSTAT, ...)



Implementing modload in MSR (continued)

• Current ld commands issued by modload on NetBSD:– modload links it the first time so it can get size information: ld -A /netbsd -e _example -o /var/run/msr/example -T 0 combined.o– modload links it the second time to resolve symbols and relocate: ld -A /netbsd -e _example -o /var/run/msr/example -T 100000

combined.o• Equivalent command on Linux:

– First do this on NetBSD (‘pl_’ implies pre-linked) ld -e _example -o pl_combined -T 0 combined.o– This is just for relocating, everything else should be done on NetBSD: ld-cross –o plugin –e _example –Ttext 0x100020 pl_combined.o– Linux and NetBSD seem to be off by 0x20 on their location of the text

segment, still investigating that. – After this command, the file plugin should be ready for downloading



modload ioctl’s• We need to know exactly what each of these does:

– ioctl(LMRESERV)• reserves kernel memory• reserves lkm slot for module• reserves kernel memory for the symbols

– ioctl(LMLOADBUF)• loads module into memory, uses and affects lkm struct curp

– ioctl(LMLOADSYMS)• loads symbol table into memory , uses and affects lkm struct curp

– ioctl(LMLOADSYMS)• loads string table into memory , uses and affects lkm struct curp

– ioctl(LMREADY)• if DDB is turned on, adds modules symbols to the DB symbol table• clears modules BSS space• calls entry procedure of module , uses and affects lkm struct curp

– ioctl(LMSTAT)• post install retrieval of stats from module

– ioctl(LMUNRESRV)• release slot for module if something goes wrong after LMRESERV.



(From Crossbow Tutorial)NetBSD Kernel Topics

IP Processingmbuf structure

Loadable Kernel ModulesInterrupts

Miscellaneous



Loadable Kernel Modules

• Mechanism in NetBSD to dynamically load code into running kernel• NetBSD System Utilities:

– modload(8)– modunload(8)– modstat(8)

• Module Types supported:– System Call modules

• When unloaded, returns to original system call• Any system call can be replace• Take care when replacing ioctl(2) since LKM uses it to load/unload modules!!

– Virtual File System modules– Device Driver modules: Block and character device drivers– Execution Interpreters: For binaries not normally usable by OS– Miscellaneous modules

• No defined interfaces• Up to user/developer to provide hooks to get to the code• This is what Router Plugins uses



Loadable Kernel Modules: modload• Open /dev/lkm

– ioctl’s will be performed on the open file descriptor• Prelink module, open it and calculate size info

– system(“ld -A /netbsd -e _<entry> -o <outfile> -T <addr=0> <module>”);

• Reserve Kernel memory:– ioctl(lkm_fd, LMRESERV, size_info)– returns kernel load address

• Relink at kernel load address• Open relinked module and load it into kernel

– ioctl(lkm_fd, LMLOADBUF, ...)• Adjusting symbol table entry pointers and load it.• sync();• Call the module’s entry function to test it.• Post-install if called for...

– ioctl(lkm_fd, LMSTAT, ...)



Overview (From MSR Tutorial)

• Introduction to hardware environment

• APIC core processing and buffer management

• Overview of SPC kernel software architecture and processing steps

• Plugin environment and filters

• Command Facility



Packet Classification & Plugins• Classification provides and opportunity to bind

flows to registered plugin instances.

• General classifier - Network Management– classification using 5-tuple

• <saddr, sport, daddr, dport, proto> ,

• Prefix match on address, exact match port and proto

• 0 is a wildcard for all fields

– input and output ports– filters added/removed via the command facility



Flow Bound to a Plugin

...DQ/ In Queuing

pluginpluginPlugin

Plugin Environment

...DRR/Out Queuing

Flow Classifier/(channel map)

Route Lookup(Shim, FIPL, Simple, cache)

...

Fra

me/

Bu

ffer

an

d I

P P

roce

ssin

g

Ingress/Egress ?

NMFilter

instance->handle_packet(instance, packet, flags)

Call packet handler for bound instance with pointer to IP packet (struct ip *).

AAL5 Frame

Fragment offsetVersion H-len TOS Total length

Identification flagsTTL protocol Header checksum

Source AddressDestination Address

Options ??

IP data (transport header and transport data)

AAL5 padding (0 - 40 bytes)

CPCS-UU (0) CPCS-UU (0) Length (IP packet + LLC/SNAP)CRC

Shimpkt (struct ip *)

handle_packet(inst, pkt, flags) { /* Plugin may read and/or * modify content but not * delete it unless COPY. * On return the framework * forwards packet */ ... return;}

Rule 1Rule 2Rule 3Rule 4Rule 5Rule 6Rule 7Rule 8Rule 9

Rule 10

i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5

SearchInvoke instance handler

General Match Classifier: Linear search of {src_addr, dst_addr, src_port, dst_port, proto}. General Classifier options: {First, Last, All}

Rule Actions: {Deny, Permit, Active}.Rule flags {All, Copy, Stop}

Send packet to exactmatch classifier

Flow Table

flow flow

hash Instance 1{Active}

Flow entry to plugin has a one-to-one relationship.

Exact Match Classifier: Hash{src_addr, dst_addr, src_port, dst_port},

then linear search for flow spec. Exact Match Classifier options: None.

Rule Actions: {Deny, Permit, Active, Reserve}.Rule flags {Pinned, Idle, Remove}

Exact Match: active processing same as general match. The AAL5 length is and IP header checksum are calculated so plugin does not have to perform these operations.



Search

Rule 1Rule 2Rule 3Rule 4Rule 5Rule 6Rule 7Rule 8Rule 9

Rule 10

i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5i1 i2 i3 i4 i5

Invoke instance handler

General Match Classifier: Linear search of {src_addr, dst_addr, src_port, dst_port, proto}

• General Classifier options: {First, Last, All} • Rule Actions: {Deny, Permit, Active}. • Rule flags {All, Copy, Stop}

General Match Classifier Notes



Flow Table

flow flow

hash Instance 1{Active}

Flow entry to plugin has a one-to-one relationship

General Match Classifier: Linear search of - {src_addr, dst_addr, src_port, dst_port, proto}.

• Exact Match Classifier options: None.• Rule Actions: {Deny, Permit, Active, Reserve}.• Rule flags {Pinned, Idle, Remove}

Exact Match Classifier Notes



Instance 1{Active}

Instance 2{Active, All}

Instance 1{Deny}

Rule N

General/Exact Match Classifier

Class A“plugin x”

Class B“plugin y”

Class C“plugin z”

Rule P

Instance 1{Active}

•Plugin instance maps to at most one rule/filter.•General classifier: rule maps to at most 5 instances.•Exact match classifier: rule maps to at most 1 instance.

Active Processing Environment



Creating an InstanceClass A

classid = 100inst_t *create_instance(class_t *, inst_id)

Instance of Class A - (Base Class extended by Developer)

<Fields defined by the Base Class>class_t*classinst_t *nextinst_id idfid_t bound_fidvoid (*handle_packet) (inst_t *, ip_t *, flag32_t);void (*bind_instance) (inst_t *);void (*unbind_instance) (inst_t *);void (*free_instance) (inst_t *);int (*handle_msg) (inst_t *, buf_t *, flag8_t, seq_t,

len_t *)<Class Specific Data>

...

create class instance

Return referenceto instance

create_instance() Called by PCU framework in response to

receiving command.

struct my_inst { inst_t base;subclass defs};



Plugin Class Specific Interface• All plugins belong to a class. At run time a class (i.e.

plugin) must be instantiated before it vcan be referenced.• Plugin is passed its instance pointer (like c++) as the first

argument.• Developer may extend the base class (struct rp_instance) to

include additional fields which are local to each instance. • Plugin developer must implement the following methods:

– void(*handle_packet)(struct rp_instance *, struct ip *, u_int32_t);

– void(*bind_instance)(struct rp_instance *);

– void(*unbind_instance)(struct rp_instance *);

– void(*free_instance)(struct rp_instance *);

– int (*handle_msg)(struct rp_instance *, void *, u_int8_t, u_int8_t, u_int8_t);



Plugin Framework Enhancements• Integrated with Command framework

– send command cells to PCU:• create instance, free instance, bind instance to filter,

unbind instance

– Send command cells to particular plugin instances– Send command cells to plugin base class

• Enhanced interface to address limitation noticed in crossbow:– instance access to: plugin class, instance id, filter id– pcu reports describing any loaded classes, instances

and filters

the new msr plugin download environment

Documents