Download - NIC Teaming - 3
1
NIC Teaming/Bonding
2
ESM Port Assignments
All ports tasked for a specific roll and can not be reallocated
ESMModule
14 ports1000 MbpsInternal linksto BladeServers
2 ports -100 MbpsInternal links to the Management Modules
4 or 6 ports 10/100/1000 Mbps RJ45
or4 ports GigE SFP Fiber
for external network connections
Service port
ESM Connections
INT1INT2INT3INT4INT5INT6INT7INT8INT9INT10INT11INT12INT13INT14
Mgt15Mgt16
EXT1EXT2EXT3EXT4
BladeServerSlot1BladeServerSlot2BladeServerSlot3BladeServerSlot4BladeServerSlot5BladeServerSlot6BladeServerSlot7BladeServerSlot8BladeServerSlot9
BladeServerSlot10BladeServerSlot11BladeServerSlot12BladeServerSlot13BladeServerSlot14
MM1MM2
Serial
BladeCenter
3
BladeCenter Ethernet Port Assignments
1
BSS1
2
1
BSS14
2
1
BSS2
2
1
BSS3
2
1
BSS4
2
1
BSS5
2
1
BSS6
2
1
BSS7
2
1
BSS8
2
1
BSS9
2
1
BSS10
2
1
BSS11
2
1
BSS12
2
1
BSS13
2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Bottom ESM (Bay 2) 20
MM1
MM2
Top ESM (Bay 1) 17 18 19
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 20
ESMExternal uplinks
ESMExternal uplinks
MM1Uplink
MM2Uplink
BladeCenter Chassis
Eth1
Eth1
Eth0
Eth0
(BSS = Blade Server Slot)
4
Broadcom BASP
Broadcom Advanced Control Suite (BACS) is the GUI tool for BASP. Windows Only
Broadcom offers no support for Linux and is now ending development.
Broadcom now supports the TG3 drivers for Linux http://www.broadcom.com/support/ethernet_nic/faq_drivers.php#tg3
IBM is distributing and supporting BASP???
5
What can Broadcom do?
Create a team using: Smart Load Balancing, IEEE 802.3ad (LACP) or Generic Trunking 802.3ad-draft (Cisco EtherChannel).
Create VLAN using 802.1q (max. 64). Both of the above can be done concurrently (team w/ VLANs) Enable 802.1p (Quality of Service) New adapters are TCP Offload Engine (TOE) Enabled.
iSCSI and RDMA Capable Included on planar board on HS21, LS21, LS41 (5708s, 5706s) Requires Microsoft Scalable Networking Pack
http://www.microsoft.com/technet/itsolutions/network/snp/default.mspx
BASP Teaming can be used with third-party NICs except for VLAN support. http://www.broadcom.com/collateral/wp/570X-WP100-R.pdf
6
Flow Control IEEE 802.3x
Flow Control is a parameter that allows the user to enable or disable the receipt or transmission of PAUSE frames.
PAUSE frames enable the adapter and the switch to control the transmit rate. The side that is receiving the PAUSE frame temporarily stops transmitting.
The Server link is 1000 Mbps, Full-Duplex, Flow Control Rx/Tx. -> You can’t change it!!!
Flow Control is especially important when the uplinks are using 100Mbps, since all servers are at 1000 Mbps.
When the switch buffer memory is getting full, the GbESM issue a PAUSE frame to the server (hold-on for a moment).
To see the link status: /info/link
7
Smart Load Balancing (SLB)
Proprietary solution from Broadcom. Only works in a flat Layer 2 environment (no routers). Team MAC address is one of the physical MACs of the NICs in the team.
It’s used to initially reply to ARP requests. Transmit Load Balancing.
When the server receives an ARP request, it replies using the Team MAC. Based on the TCP/UDP information; the teaming driver will change the Source
MAC address with one of the MACs of the physical NICs.
Receive Load Balancing In order to distribute all incoming traffic among all NICs, BASP periodically issues
a Gratuitous ARP (GARP) directly to the destination MAC address with the Source MAC address of one of the physical NICs.
Router will receive this GARP and keep updating the ARP cache. ARP doesn't have an IP header, so it can’t be routed.
8
Smart Load Balancing (SLB)
Always configure SLB in Active-Standby mode. Make a note of the physical MAC address of each NIC before
creating the Team.
Once a NIC is part of a Team, it can’t be used in another Team.
SLB with auto Fall-Back means that once the primary NIC is back on-line, the Team will start using that NIC again. Keep in mind that when you replace a faulty switch, it doesn't have any
configuration, so it might not be good to automatically use it. The MM also perform a series of Link UP/Down tests during the boot process.
SLB with no failback is preferable for these reasons.
9
Generic Trunking
Equivalent of Cisco EtherChannel or Nortel Static MLT.
Team uses one MAC and IP address. Requires both links to end on the same
device also configured with the same Trunking protocol. Only possible in a BladeCenter using the HS40 or LS41 or the BCE card.
If the NICs are connected to different devices, Active-Standby is the only possible option…
If both NIC’s in the team are in Active-Active, the external GbESM ports need to end on a single switch where all ports use same trunking protocol.
Active Active
Core Network
Generic Trunk
10
Linux Bonding (802.3ad)
Functionally very similar to what BASP provides. The “bond” uses the MAC address of the first physical NIC. Output traffic is allocated to physical NICs based on (src hw addr
XOR dst hw addr). This selects the same NIC for traffic to a specific destination hw address.
SLES Bonding https://secure-support.novell.com/KanisaPlatform/Publishing/133/3929220_f.SAL_Public.html
Red Hat Bonding http://kbase.redhat.com/faq/FAQ_85_6313.shtm
VMWare ESX Bonding http://www.vmware.com/vmtn/resources/432
Link to Linux Networking Stuff.
11
Linux Bonding config in Active-Backup
sys1# insmod bonding.o mode=1 miimon=100 sys1# /sbin/ifconfig bond0 10.0.0.1 up sys1# /sbin/ifenslave bond0 eth0 sys1# /sbin/ifenslave bond0 eth1 sys1# /sbin/ifconfig bond0 Link encap:Ethernet HWaddr 00:A0:CC:D0:DA:10 inet addr:10.0.0.1 Bcast:10.0.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:7224794 errors:0 dropped:0 overruns:0 frame:0 TX packets:3286647 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 eth0 Link encap:Ethernet HWaddr 00:A0:CC:D0:DA:10 inet addr:10.00.0.1 Bcast:10.0.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:3573025 errors:0 dropped:0 overruns:0 frame:0 TX packets:1643167 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:10 Base address:0x1080 eth1 Link encap:Ethernet HWaddr 00:A0:CC:D0:DA:10 inet addr:10.0.0.1 Bcast:10.0.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING SLAVE NOARP MULTICAST MTU:1500 Metric:1 RX packets:3651769 errors:0 dropped:0 overruns:0 frame:0 TX packets:1643480 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 Interrupt:9 Base address:0x1400
Bond need to be mode=1 (mode=0 is Active-Active).
The NIC in backup is the one with the “NOARP” statement.
By default, preempt is enable. -> Fall-back to the original NIC once it’s OK.
12
Locating BASP software
Software and documentation is included in driver package (.zip or .iso packages) NetXtreme (xseries servers)
http://www-3.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-43815
NetXtreme II (newer, system x servers) http://www-3.ibm.com/pc/support/site.wss/document.do?lndocid=MIGR-64505
Default extraction location is C:\Drivers\v94B2 Click on Launch.exe file
13
Installing BASP
This application allows you to update the Windows Drivers (recommended)
Management Applications is the BASP Software Default installation is acceptable
14
BACS Control Panel
Tray icon Unconfigured Control Panel
15
Creating a Team
Expert Mode Wizard Mode Be careful of Team Name, as
this becomes the Interface Name Can cause problems if name
has a space
16
Configuring BASP
Choose the Team type you wish to configure SLB is most common
17
Add adapters to Team
18
Configuring BASP ALWAYS configure the SLB Team with one NIC Active and One NIC Standby. Make sure to properly identify the Active and Standby NIC, it‘s not always the one
we think... Smart Load Balancing (SLB) is supported with or without FallBack. SLB Auto-Fallback means the Team will go automaticaly to the primary NIC when
is comes (back) online.
Note: Before a team is
configured on the server
ipconfig /all will list the NIC
MAC address. On the switch
/i/l2/fdb/dump will show you
the MAC database.
19
LiveLink
Do not configure Livelink Trunk Monitoring performs this function
20
Add VLAN
Choose whether or not to create VLANs for the TEAM VLAN tagging not supported with non-broadcom adapters
21
Give Name to VLAN
Friendly name for VLAN Not the actual VLAN ID
22
To Tag, or Not To Tag
Only one untagged VLAN per team. GbESM will handle these frames in the PVID
23
Assign the VLAN tag value
This is the actual VLAN id. VLAN must be configured on GbESM Must assign IP address for the adapter (static or DHCP)
24
A chance to review or commit configuration
Almost complete
25
Preview
Shows how configuration will look prior to saving.
Make any changes in this view. (don’t click cancel thinking you will return to the wizard)
26
Changes to interfaces in the OS
27
High-Availability at the external Network level
Active Backup
Core Network
NICs on server blades operate in “Teaming” mode.
The “NIC Teaming” option is unaware of any outages on the external network
When a failure occurs on the uplinks, the “NIC Teaming” function is not aware of the failure and the servers continue to send data to a “dead end” path!
All data is dropped by the primary switch. No error message is sent to the administrator
Although two switches are used in the BladeCenter the application sees no resilience to failures
Layer 2 switches
28
Active Backup
Core Network With the GbESM “Trunk Monitoring” option,
the switch constantly monitors the uplinks. When a failure is detected the GbESM
disables the internal ports. The “NIC Teaming” function then becomes
aware of the failure and falls back to the secondary NIC.
A Syslog message and SNMP trap are generated and sent to the Network administrator.
High-Availability at the external Network level
29
Enhanced L2 Trunk Fail-Over
Core Network
GbESM is VLAN aware and will only disable internal ports associated with the same VLAN. No un-necessary Fail-Over.
Triggers can be configured to force a Fail-Over when only some of the links fail, for bandwidth-sensitive applications.
Link Aggregation with Fail-Over can be configured with IEEE 802.3ad (LACP).
Internal ports are only re-activated once Spanning-tree is in “Fowarding“ mode.
Wait until
Spanning-Tree is
Forwarding.
30
Configuring Trunk Fail-Over
/cfg/l2/trunk 1
ena
add ext1
add ext2 /cfg/l2/failovr ON /cfg/l2/failovr/trigger 1
limit 0
ena /cfg/l2/failovr/trigger 1/amon
addtrnk 1
Create a normal Trunk. (i.e.: LACP or
EtherChannel)
Enable the Fail-Over feature.
Create a Trigger Profile.
Configure the number of required active link in the Trunk.
0 means that all links needs to be Down for the trigger to
activate.
Apply the trigger conditions to your Trunk.
31
Team 1BC-H
Team 2BC - H
Team 3BC
Team 4BC-H
MM: 192.168.70.10ESM1: 192.168.70.11ESM2: 192.168.70.12
MM: 192.168.70.20ESM1: 192.168.70.21ESM2: 192.168.70.22
MM: 192.168.70.30MMint: 192.168.70.35ESM1: 192.168.70.31ESM2: 192.168.70.32
MM: 192.168.70.40ESM1: 192.168.70.41ESM2: 192.168.70.42
Team 5BC
MM: 192.168.70.50MMint 192.168.70.55ESM1: 192.168.70.51ESM2: 192.168.70.52
Team 6BC-H
MM: 192.168.70.60ESM1: 192.168.70.61ESM2: 192.168.70.62
Team 7BC
MM: 192.168.70.70MMint 192.168.70.75ESM1: 192.168.70.71ESM2: 192.168.70.72
Team 8BC-H
MM: 192.168.70.80ESM1: 192.168.70.81ESM2: 192.168.70.82
OK
TX/RX
Link
IP
AC
DC
AC
DC
PS
1
4
PS
2
3
7
8
9
10
MM 13
MM 24
1
2
5
6
1
2
OK
TX/RX
Link
IP
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
PS
1
4
PS
2
3
7
8
9
10
MM 13
MM 24
1
2
5
6
1
2
OK
TX/RX
Link
IP
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
PS
1
4
PS
2
3
7
8
9
10
MM 13
MM 24
1
2
5
6
1
2
OK
TX/RX
Link
IP
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
PS
1
4
PS
2
3
7
8
9
10
MM 13
MM 24
1
2
5
6
1
2
OK
TX/RX
Link
IP
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
PS
1
4
PS
2
3
7
8
9
10
MM 13
MM 24
1
2
5
6
1
2
OK
TX/RX
Link
IP
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
OK
TX/RX
Link
IP
AC
DC
AC
DC
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
OK
TX/RX
Link
IP
AC
DC
AC
DC
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
TX/RX
Link
TX/RX
Link
TX/RX
Link
TX/RX
Link
1
2
3
4
OK
Unavailable
Unavailable
USERID/PASSW0RD Administrator/eserver1