Networking Virtualization Using FPGAs

Russell Tessier, Deepak Unnikrishnan, Dong Yin, and Lixin Gao

Reconfigurable Computing GroupDepartment of Electrical and Computer Engineering

University of Massachusetts, Amherst, USA

2

Outline

Network virtualization

Advantage of FPGA for virtualization

Architecture of FPGA-based virtualization system

Partially-reconfigurable implementation

Results

3

Virtual Networking Internet growing to include new applications and services

Cloud computing, Data center networking

Challenges Lack of innovation at network core – Fixed internet routers Coupled infrastructure-service provider model

Solution Network virtualization Router hardware shared

across multiple networks

4

Network Virtualization

Many logical networks over a physical infrastructure

Virtual nodes

Shared network resources among multiple virtual networks

Reduces costs

Independent routing policies

FA

C E

DB

C E

F

DB

A

B D

CE

A F

5

Virtual Router Independent routing policies

for each virtual router

Key challenges• Isolation• Performance• Flexibility• Scalability

Forwarding Table

Routing Control

Virtual router B

Forwarding Table

Routing Control

Physical routerVirtual router A

DEMUX MUX

6

Traditional Network Virtualization Techniques Software

Full virtualization Container virtualization Limitations

Limited performance (~100Mbps) Limited isolation

ASIC Supercharging PlanetLab Platform1

Juniper E series Possible Limitations

Flexibility Scalability

HW

VM VM

Full virtualization

HW

OSinstance

OSinstance

Container virtualization

1 “Supercharging PlanetLab – A High Performance, Multi-Application, Overlay Network Platform”, J. Turner et al., SIGCOMM 2007

7

Virtualization using FPGAs

A novel network virtualization substrate which

• Uses FPGA to implement high performance virtual routers

• Introduces scalability through virtual routers in host software

• Exploits reconfiguration to customize hardware virtual routers

FPGA

VirtualRouter 1

VirtualRouter 2

VirtualRouter 3

VirtualRouter 4

8

System Overview

SRAM

SRAM

PCI

1GEthernet

I/F

SoftwareVirtual Router

SoftwareVirtualRouter

SoftwareVirtualRouter

Kernel driverSoftware bridge

Linux

NetFPGA

PHY

SDRAM

SDRAMHW VRouter

HW VRouter

NIC NIC

9

Architecture

HardwareVirtual

Router 1

VIP TYPE ID0 HW 01 HW 12 SW 03 SW 1

User space

Kernel space

Host Workstation

NetFPGA

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

InputArbiter

DynamicDesignSelect

HardwareVirtual

Router 2

CPUTransceiver

OutputQueue

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

NetFPGAPCI I/F

NetFPGAdriver

Bridge

PC NIC

Software Virtual

Router 1

SoftwareVirtual

Router 2

Bridge

NetFPGAdriver

NetFPGAPCI I/F

Kernel space

PC NIC

CONTROL

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PCI BUS

To Switch

FromSwitch

10

Scalable Network Virtualization Approaches for implementing scalable virtual routers

• Single receiver approach• All packets routed through NetFPGA hardware

• Multi-receiver approach• Use a switch to separate packets to NetFPGA and software

11

Virtual Router Customization Multiple virtual routers share the substrate

Individual virtual routers may need customization

Challenge

• Minimize the impact on traffic in shared hardware virtual routers during modification

12

Multi Receiver Approach

HardwareVirtual

Router 1

VIP TYPE ID0 HW 01 HW 12 SW 03 SW 1

User spaceKernel spaceHost OS

NetFPGA

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

InputArbiter

DynamicDesignSelect

Hardware Virtual

Router 2

CPUTransceiver

OutputQueue

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

NetFPGAPCI I/F

NetFPGAdriver

Bridge

PC NIC

SoftwareVirtual

Router 1

SoftwareVirtual

Router 2

Bridge

NetFPGAdriver

NetFPGAPCI I/F

Kernel space

PC NIC

CONTROL

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

Switch Switch

PCI BUS

So

urc

e

De

stin

atio

n

13

HardwareVirtual

Router 1

VIP TYPE ID0 HW 01 HW 12 SW 03 SW 1

User spaceKernel spaceHost OS

NetFPGA

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

InputArbiter

DynamicDesignSelect

Hardware Virtual

Router 2

CPUTransceiver

OutputQueue

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

NetFPGAPCI I/F

NetFPGAdriver

Bridge

PC NIC

SoftwareVirtual

Router 1

SoftwareVirtual

Router 2

Bridge

NetFPGAdriver

NetFPGAPCI I/F

Kernel space

PC NIC

CONTROL

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

Switch Switch

PCI BUS

So

urc

e

De

stin

atio

n

Dynamic Reconfiguration

AddressRemap

Sw VirtualRouter

New HW VirtualRouter

Reconfigure FPGA

14

HardwareVirtual

Router 1

VIP TYPE ID0 HW 01 HW 12 SW 03 SW 1

User space

Kernel space

Host Workstation

NetFPGA

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

MAC RX Q

CPU RX Q

InputArbiter

DynamicDesignSelect

HardwareVirtual

Router 2

CPUTransceiver

OutputQueue

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

MAC TX Q

CPU TX Q

NetFPGAPCI I/F

NetFPGAdriver

Bridge

PC NIC

Software Virtual

Router 1

SoftwareVirtual

Router 2

Bridge

NetFPGAdriver

NetFPGAPCI I/F

Kernel space

PC NIC

CONTROL

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PHY MAC

PCI BUS

To Switch

FromSwitch

Single receiver approach

15

Partial Reconfiguration Use partial reconfiguration to independently configure

virtual routers

16

Partial Reconfiguration Up to 2 partially reconfigurable virtual routers in Virtex 2

Up to 20 partially reconfigurable virtual routers in Virtex 5

17

Experimental Approach Metrics

• Throughput• Latency

Packet generation• NetFPGA packet generator• iPerf

Ping utility used for latency measurements

Software virtual routers run on 3Ghz AMD X2 6000+ processor, 2GB RAM, Intel E1000 GbE in PCIe slot

SourceNetFPGA Pktgen/

iPerf

VirtualRouter

SinkNetFPGA Pktcap/

iPerf

1Gbps 1Gbps

18

Dynamic Reconfiguration Overhead

Full FPGA reconfiguration• Source pings destination through a single h/w virtual router• Migrate traffic to software• Reconfigure FPGA• Migrate traffic to hardware• 12 seconds reconfiguration overhead

Partial FPGA reconfiguration• 0.6 seconds reconfiguration• Remaining virtual routers in FPGA remain active

19

Throughput of Single Hardware Virtual Router Hardware virtual router 1 to 2 order better than the software

virtual router Consistent forwarding rates vs packet size

1

10

100

1000

64 128 512 1024 1470Packet size (bytes)

Thro

ughp

ut (M

bps)

Hardwarevirtual router

Softwarevirtual router

20

Full FPGA Reconfiguration Two virtual routers (A, B) initially in FPGA During reconfiguration router A migrated to software, the

other eliminated After reconfiguration two virtual routers (A, B’) again in FPGA

ReducedThroughput

21

Partial FPGA Reconfiguration A remains in hardware and operates at full speed 20X speedup in reconfiguration down time due to partial

reconfiguration SustainedThroughput

22

Scalability - Average Throughput of Virtual Routers Aggregate throughput of 1Gbps up to 4 h/w virtual routers

Adding software virtual routers causes drop in aggregate throughput

1

10

100

1000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Virtual routers

Thro

ughp

ut (M

bps)

Avg hardware andsoftware

Software only

23

Scalability – Average Latency of Virtual Routers Latency of h/w virtual router substantially better than

software virtual router

0

0.2

0.4

0.6

0.8

1

1.2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Virtual routers

Late

ncy

(ms)

Software only

Avg hardware andsoftware

24

Throughput Effect of Reconfiguration Frequency Partial reconfiguration beneficial during frequent updates

25

Average throughput for combined hardware/software Rigid placement constraints limit #hardware virtual routers Larger FPGAs sustain high throughputs for more virtual routers

26

Altera DE4

27

Throughput Measurements on DE-4 Reference Router Consistent packet forwarding rates of 1Gbps for all packet sizes

(64-1024) bytes. Packet throughput matches NetFPGA reference router in all cases. Packet buffering in on-chip SRAM (300Kbit).

28

Resource Utilization

Altera DE4 NetFPGA

Used Available % Used Available %

Combinational LUTs

24,727 182,400 14% 19,783 47,232 41%

Logic Registers

34,218 182,400 19% 14,682 47232 31%

Memory bits

1,448,612

14,625,792

9% 24 232 10.3%

IOs 118 888 13% 356 692 51%

Logic Utilization 14% LUTs 19% Logic registers 9% Memory bits 13% IOs

29

Reconfiguration Time

Partial Reconfiguration

Full Reconfiguration

20X Faster virtual router updates with partial reconfiguration

30

Resource Usage and Power Consumption Virtex 2 can support

• 5 h/w virtual routers (without support for software routers) • 4 h/w virtual routers (with support for software routers)• 2 partially reconfigurable hardware routers

Virtex 5 can support up to 32 virtual routers

A single h/w virtual router consumes • 156mW static power

• 688mW dynamic power

Clock gating saves 10% power

31

Dynamic Virtual Network Allocation Assign virtual networks to software/hardware virtual

routers based on bandwidth-resource requirements 15-20% more successful network upgrades with dynamic

allocation

32

Conclusion FPGAs are ideal for a variety of networking applications

• Partial reconfiguration growing in importance

Virtual networking is an emerging networking area• Combined FPGA and software system gives high performance and

scalability• A novel heterogeneous network virtualization substrate using

FPGAs

NetFPGA platform used to implement up to five virtual routers• Two order of magnitude throughput increase versus software.

Partial FPGA reconfiguration reduces the number of implemented routers but reduces reconfiguration time to 0.6s• Allocation algorithm migrates highest throughput routers to

hardware