vmware emc service talk
TRANSCRIPT
Copyright © 2013 EMC Corporation. All Rights Reserved.
VMware EMC Service Talk
May 2014Kevin Wang
Copyright © 2013 EMC Corporation. All Rights Reserved.
Question 1: The detailed procedure for HBA initiates login to SAN to complete the connection to the SP (Answers cover both FC and FCoE)
Copyright © 2013 EMC Corporation. All Rights Reserved.
This lesson covers the following topics:• FC protocol make-up• FC components
Lesson 1: Fibre Channel Protocol Overview
Fibre Channel and FCoE Technology 3
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fibre Channel Layered Architecture: FC-0
ULP Upper Layer ProtocolSCSI-3, IP, ESCON/FIPS, etc.
FC-4 FCP (SCSI-3 Fibre Channel Link Encapsulation),Single-Byte Command Code Sets (FICON), etc.
FC-3 Common Services (Not Used)
FC-2 Exchange and Sequence Management, Frame Structure ,Flow Control
FC-1 Encode/Decode, Link Control Protocols,Ordered Sets
FC-0 Physical Interface, Optical and Electrical Interfaces, Cables, Connectors, etc.
Fibre Channel and FCoE Technology 4
Copyright © 2013 EMC Corporation. All Rights Reserved.
Transceivers Cable Connectors Comments
SFP, SFP+ LC
• Most popular for Fibre Channel today
• SFPs up to 8 Gbps• SFP+ > 8Gbps• versions for different distances
mSFP mLC
• mini-SFP and mini-LC • Designed by Brocade for 64-
port blade
GBIC SC
• Technology found in older 1 Gbps equipment
• Replaced by LC technology
Optical Transceivers and Cable Connectors
Fibre Channel and FCoE Technology 5
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fibre Channel Cables and Distances
Transceiver Type Speed
Multimode media maximum distance
62.5 um(OM1)
50 um(OM2)
50 um(OM3)
50 um(OM4)
SW
2 Gbps 150m 300m 500m 500m
4 Gbps 70m 150m 380m 400m
8 Gbps 21m 50m 150m 190m
10 Gbps 33m 82m 300m 550m
16 Gbps 15m 35m 100m 125m
Single-mode media maximum distance (9 um core optic fiber)
LWL
4 Gbps
10 km to 40 km depending on vendor
8 Gbps
10 Gbps
16 Gbps
Note: For greater distances, you may connect using DWDM / CWDM between switches .
Fibre Channel and FCoE Technology 6
Copyright © 2013 EMC Corporation. All Rights Reserved.
Basic Port Types
Node Port: Transmits and receives Fibre Channel data frames in a Switched Fabric environment (N_Port)
Fabric Port: Port on a switch that N_Port connects to. (F_Port)
Expansion Port: Port on a switch that has another switch plugged into it (two switches that are merged) (E_Port)
Node Loop Port: Transmits and receives Fibre Channel frames in an Arbitrated Loop (private) environment (NL_Port)
Fabric Loop Port: Allows a Fabric to communicate with an Arbitrated Loop (public) (FL_Port)
Generic Port: Port on a switch that detects the attached port type and auto-configures to match that port type (G_Port)
Fibre Channel and FCoE Technology 7
Copyright © 2013 EMC Corporation. All Rights Reserved.
The Host Bus Adapter
Fabric
Target Storage Array
Server
HBA
NIC
LAN
Ethernet / IP
See EMC Support Matrix at https://elabnavigator.emc.com
Fibre Channel and FCoE Technology 8
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fibre Channel Layered Architecture: FC-1
ULP
FC-4
FC-3
FC-2
FC-1 Encode/Decode, Link Control Protocols,Ordered Sets
FC-0
Data encoding/decoding
• Less than 10 Gbps (20% overhead for encoding) 8-bits 10-bits
• 10 Gbps and 16 Gbps (3% overhead for encoding) 64-bits 66-bits
• Special Characters Used with data characters to create ordered sets Ordered sets include: Idle, SOF, EOF, R_RDY
Fibre Channel and FCoE Technology 9
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fibre Channel Layered Architecture: FC-2
ULP
FC-4
FC-3
FC-2 Exchange and Sequence Management, Frame Structure ,Flow Control
FC-1
FC-0
• Defines structure of Fibre Channel frame Frames, Sequences, Exchanges
• Flow control mechanisms
• Error handling and recovery
• Provides class of services Class 1, Class 2, Class 3, Class 4, Class 6, and Class F
Fibre Channel and FCoE Technology 10
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC-2
• Defines the structure of the Fibre channel frame How information to large to fit into a single frame is transferred by
using a sequence of frames How operations that consist of exchanges of sequences of frames
are managed
• Describes how the rate of frame transmission is controlled by the flow control mechanisms
• Error handling and recovery• Class of Service
Class 1, Class 2, Class 3, Class 4, Class 6, and Class F
Fibre Channel and FCoE Technology 11
Copyright © 2013 EMC Corporation. All Rights Reserved.
Exchange, Sequence, Frame
• Exchange is an operation manager SCSI task, an IP connection Has an Originator Has a Responder Has one or more Sequences
• Sequence is a ULP object to be transmitted in one direction that may be one frame or longer
• Frame is the unit of information transfer• Manages flow control (buffer-to-buffer and end-to-end)
Fibre Channel and FCoE Technology 12
Copyright © 2013 EMC Corporation. All Rights Reserved.
Conversations
Session
Exchange Exchange Exchange
Seq Sequence Sequence SequenceSeqSeq
Frame #1 Frame #1
Frame #2
Frame #3
Frame #1
Frame #2
Frame #3
Frame #4
Frame #5
Frame #6
Frame #1
Frame #2
Frame #3
Frame #4
Frame #1
Frame #2
Frame #1
Fibre Channel and FCoE Technology 13
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fibre Channel Frame
Data Field End-of-Frame
CRC
Start-of-Frame
Header
Optional Header
Fill Bytes
Payload(information being transported)
Optional Header
R_CTL
CS_CTL
TYPE
SEQ_ID DF_CTL SEQ_CNT
Frame Control (F_CNT)
Source Address (S_ID)
Destination Address (D_ID)
OX_ID RX_ID
Parameter Field (PARM)
Fibre Channel and FCoE Technology 14
Copyright © 2013 EMC Corporation. All Rights Reserved.
Credit-Based Flow Control
Fibre ChannelSwitch
Physical ISL
FC Frame R_RDY
Credit = 16 1516
Fibre ChannelSwitch
Long Distance links may require additional BB_Credit
Fibre Channel and FCoE Technology 15
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fibre Channel Layered Architecture: FC-4
ULP
FC-4 FCP (SCSI-3 Fibre Channel Protocol) mapping,Single-Byte Command Code Sets (used for FICON) mapping
FC-3 Common Services (Not Used)
FC-2
FC-1
FC-0
• Mapping Layer Maps Upper Level Protocol to lower levels SAN typically uses SCSI FCP mapping
Fibre Channel and FCoE Technology 16
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC-4 and Upper Level ProtocolExchange, Sequence, and Frame
...Information Unit # 1 Information Unit # 3Information Unit # 2...FC-4
FC-2
Sequence # 1 Sequence # 3Sequence # 2... ...
Frame 1
...
Frame 1 Frame 2 Frame N Frame 1
Payload: SCSI CDB DATA Payload: Status
Exchange (Operation)
DATA DATA
SCSI Command SCSI ResponseSCSI DATAULPSCSI-3:
One SCSI Read I/O Operation
Fibre Channel and FCoE Technology 17
Copyright © 2013 EMC Corporation. All Rights Reserved.
EMC Fibre Channel Implementations
• Point-to-Point – Link between two ports using Arbitrated Loop
• Arbitrated Loop (FC-AL) – Shared interconnect between 2 to 126 nodes
• Switched Fabric (FC-SW) – Dynamic connectivity support for more than 15 million nodes
Fibre Channel and FCoE Technology 18
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fibre Channel Fabric
Target Storage Array
Fabric
Server
HBA
Server
HBA
ISL
Fibre Channel and FCoE Technology 19
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fibre Channel Address (FCID)
• Required to route frames from source to target• 24-bits (three-bytes) - specified in Hex
Assigned at Fabric Login (FLOGI)
• Divided into three parts:
Domain_ID Area_ID Port_ID
Domain (01-EF)
(1-239 in decimal)
00-FF 00-FF
Example: 6b 04 00
Fibre Channel and FCoE Technology 20
Copyright © 2013 EMC Corporation. All Rights Reserved.
Common Fabric Services: Well Known Addresses
Service NameWell
Known Address
Description
Login Service(F_Port Server)
FFFFFE All nodes register in login service when performing FLOGI
Name Service FFFFFCStores information about nodes attached to fabric during the Fabric Login process
Fabric Controller FFFFFD
State change notification to nodes registered in the fabric and inter-switch communications
Fibre Channel and FCoE Technology 21
Copyright © 2013 EMC Corporation. All Rights Reserved.
World Wide Name
• A World Wide Name (WWN) is a 64-bit value used in Fibre Channel networks to uniquely identify each element in the network
• Usually assigned to a Host Bus Adapter (HBA), switch port, or storage port by the vendor at the time of manufacture
• Often linked within a component, such as an HBANode WWN – A name for all ports within the single HBAPort WWN – A name for each individual port
Fibre Channel and FCoE Technology 22
Copyright © 2013 EMC Corporation. All Rights Reserved.
WWPN Examples
EMC VNX World Wide Port Name Example
5 0 0 6 0 1 6 0 4 7 2 0 5 5 9 0
EMC Company ID (24-bits) Port VNX Seed (from SPE/DPE midplane)
EMC VMAX World Wide Port Name Example
5 0 0 0 0 9 7 2 0 8 1 3 4 9 a d
EMC Company ID for VMAX (24-bits) VMAX Serial Number - Decodes to country where manufactured/Model /
etc. (HK192601234)
CPU/Dir/P(12g port B)
Emulex World Wide Port Name Example
1 0 0 0 0 0 0 0 c 9 2 0 d c 4 0
Reserved (12-bits)
Company OUI (24-bits) Company Specific (24-bits)
Fibre Channel and FCoE Technology 23
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC Port Initialization Process - Step 11 - Link online2 - Fabric Login - get FCID3 - Port Login to fabric name server4 - Registrations with name server5 - Query Name server for FC-4 devices6 - State Change RegistrationRepeat for each Target:
7 Port Login to Target port 8 Process Login to Target port 9 SCSI INQ
Fabric 1
5
1 2
3 4HBA
WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7FCID = ??????
6
Fibre Channel and FCoE Technology 24
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC Port Initialization Process - Step 21 - Link online2 - Fabric Login - get FCID3 - Port Login to fabric name server4 - Registrations with name server5 - Query Name server for FC-4 devices6 - State Change RegistrationRepeat for each Target:
7 - Port Login to Target port 8 - Process Login to Target port 9 - SCSI INQ
Fabric 1
5
1 2
3 4HBA
FLOGI
ACC
Accept frame includes FCID for HBA 030500
FLOGI FrameDestination Address = FFFFFESource Address = 000000
WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7FCID = ??????
6
Fibre Channel and FCoE Technology 25
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC Port Initialization Process - Step 31 - Link online2 - Fabric Login - get FCID3 - Port Login to fabric name server4 - Registrations with name server5 - Query Name server for FC-4 devices6 - State Change RegistrationRepeat for each Target:
7 - Port Login to Target port 8 - Process Login to Target port 9 - SCSI INQ
Fabric 1
WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500
5
1 2
3 4HBA
PLOGI
ACC
PLOGI FrameDestination Address = FFFFFCSource Address = 030500
6
Fibre Channel and FCoE Technology 26
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC Port Initialization Process - Step 41 - Link online2 - Fabric Login - get FCID3 - Port Login to fabric name server4 - Registrations with name server5 - Query Name server for FC-4 devices6 - State Change RegistrationRepeat for each Target:
7 - Port Login to Target port 8 - Process Login to Target port 9 - SCSI INQ
Fabric 1
5
1 2
3 4HBA
WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500
6
Fibre Channel Services Frames• Register Class of Service • Register FC-4 Type• Register Port type• Register other (vendor
dependent)
FC Serv
ACC
FC Serv
ACC
Fibre Channel and FCoE Technology 27
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC Port Initialization Process - Step 51 - Link online2 - Fabric Login - get FCID3 - Port Login to fabric name server4 - Registrations with name server5 - Query Name server for FC-4 devices6 - State Change RegistrationRepeat for each Target:
7 - Port Login to Target port 8 - Process Login to Target port 9 - SCSI INQ
Fabric 1
5
1 2
3 4HBA
WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500
6
Fibre Channel Services Frames• Query: Get PIDs • Query: Get Port WWN for PID1• Query: Get Port WWN for PIDn
FC Serv
ACC
FC Serv
ACC
Name Server Database:• PID = 040600
• FC-4 = SCSI FCP• WWPN =
50:06:04:8a:d5:2e:76:c4• . . .
• PID = 030100 . . .• PID = 040400 . . .
Fibre Channel and FCoE Technology 28
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC Port Initialization Process - Step 61 - Link online2 - Fabric Login - get FCID3 - Port Login to fabric name server4 - Registrations with name server5 - Query Name server for FC-4 devices6 - State Change RegistrationRepeat for each Target:
7 - Port Login to Target port 8 - Process Login to Target port 9 - SCSI INQ
Fabric 1
5
1 2
3 4HBA
WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500
6
SCR
ACC
SCR FrameDestination Address = FFFFFDSource Address = 030500
Fibre Channel and FCoE Technology 29
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC Port Initialization Process - Step 71 - Link online2 - Fabric Login - get FCID3 - Port Login to fabric name server4 - Registrations with name server5 - Query Name server for FC-4 devices6 - State Change RegistrationRepeat for each Target:
7 - Port Login to Target port 8 - Process Login to Target port 9 - SCSI INQ
Fabric 1
5
1 2
3 4HBA
WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500
6PLOGI
Fibre Channel and FCoE Technology 30
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC Port Initialization Process - Step 81 - Link online2 - Fabric Login - get FCID3 - Port Login to fabric name server4 - Registrations with name server5 - Query Name server for FC-4 devices6 - State Change RegistrationRepeat for each Target:
7 - Port Login to Target port 8 - Process Login to Target port 9 - SCSI INQ
Fabric 1
5
1 2
3 4HBA
WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500
6PRLI
SCSIProces
s
SCSIProces
s
Fibre Channel and FCoE Technology 31
Copyright © 2013 EMC Corporation. All Rights Reserved.
FC Port Initialization Process - Step 91 - Link online2 - Fabric Login - get FCID3 - Port Login to fabric name server4 - Registrations with name server5 - Query Name server for FC-4 devices6 - State Change RegistrationRepeat for each Target:
7 - Port Login to Target port 8 - Process Login to Target port 9 - SCSI INQ
Fabric 1
5
1 2
3 4HBA
WWN = 50:06:04:8a:d5:2e:76:c4FCID = 040600
WWN = 10:00:00:00:c9:48:f1:c7FCID = 030500
6
SCSIProces
s
SCSIProces
sSCSI INQ
Fibre Channel and FCoE Technology 32
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fibre Channel Login at a Glance
Process FLOGI PLOGI PRLI
Requirements - Link initialization- Cable- HBA and driver- Switch port
- FLOGI- Zoning- Persistent binding- Driver setting
- PLOGI- Device masking (Target)- Device mapping (Initiator)- Driver setting (Initiator)
Information passed - WWN- S_ID- Protocol- Class- Zoning
- WWN- S_ID- ULP- Class- BB Credit
- LUN
Session members - N_Port to F_Port - N_Port to N_Port - ULP (SCSI-3 to SCSI-3)
Verification UNIX
WINDOWS
- Syslog- Switch Utilities
- Syslog- Driver Utilities
- Syslog- Host based volume management
- Event Viewer- Switch Viewer
- Driver utilities - Driver utilities- Host based volume management- Device manager
Fibre Channel and FCoE Technology - 33
Copyright © 2013 EMC Corporation. All Rights Reserved.
Zone Set: Fabric 1Zone A: Initiator A; Target wZone B: Initiator B; Target xZone C: Initiator C; Target y; Target z. . .
Zoning Overview
Storage(Targets)
Hosts(Initiators)
I
T
Initiator (HBA) Target
Zone A
Fabric 1
Fibre Channel and FCoE Technology 34
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fabric 1
How to Define Zone Members
I
T
Target Port WWPN:50:06:04:8a:d5:2e:76:c4
Zone A
Initiator Port WWPN:10:00:00:00:c9:48:f1:c7
5 3
1 2
3 4
Zone A10:00:00:00:c9:48:f1:c7;50:06:04:8a:d5:2e:76:c4
Zone Members Defined by WWPN
Zone A0305ef; (Domain_ID 03, port_05)0403ef; (Domain_ID 04, port_03)
Zone Members Defined by Domain/Port
Fibre Channel and FCoE Technology 35
Copyright © 2013 EMC Corporation. All Rights Reserved.
This lesson covers the following topics:• Architecture of an Ethernet solution
Lesson 2: Ethernet Overview
Fibre Channel and FCoE Technology 36
Copyright © 2013 EMC Corporation. All Rights Reserved.
Ethernet and the OSI Model
7 - Application
6 - Presentation
5 - Session
4 - Transport
3 - Network
2 - Data Link
1 - Physical
LLC - Logical Link ControlMAC - Media Access Control
Ethernet is defined at Layer 2 of the OSI model
Fibre Channel and FCoE Technology 37
Copyright © 2013 EMC Corporation. All Rights Reserved.
Ethernet PDU
Destination MAC6 bytes
Source MAC
6 bytes
802.1Q tag
(optional)
4 bytes
Type2
bytes
Data Payloadvariable # bytes
FCS4
bytes
Ethernet_II Frame Format
Header
Fibre Channel and FCoE Technology 38
Copyright © 2013 EMC Corporation. All Rights Reserved.
Ethernet MAC Address
IG
GL
Organizationally Unique Identifier (OUI)
(Assigned by IEEE)Vendor assigned
24 bits 24 bits
Examples:
Broadcast address: ff:ff:ff:ff:ff:ff
Physical address of port on PC: 00-26-5E-82-3E-6E
47 46 45 24 23 0
Fibre Channel and FCoE Technology 39
Copyright © 2013 EMC Corporation. All Rights Reserved.
MAC Address Used to Direct Frame Through Network
MAC: 00-26-5E-82-3E-6E MAC: 00-26-5E-77-2F-58
…2F:58 …3E:6E 08-00
Payload FCS
Ethernet Frame
This is the source address This is the destination address
Destination MAC
Fibre Channel and FCoE Technology 40
Copyright © 2013 EMC Corporation. All Rights Reserved.
Switching Frames Using MAC Addresses - Step 1
MAC TablePort 1 - EmptyPort 2 - EmptyPort 3 - EmptyPort 4 - Empty
PC A PC C
PC B
PC D
Port 1
Port 2
Port 3
Port 4
Frame
Dest Src
PC A
Fibre Channel and FCoE Technology 41
Copyright © 2013 EMC Corporation. All Rights Reserved.
Switching Frames Using MAC Addresses - Step 2
MAC TablePort 1 - PC APort 2 - EmptyPort 3 - EmptyPort 4 - Empty
PC C
PC B
PC D
Port 1
Port 2
Port 3
Port 4
No entry for PC D, so flood the frame out all
ports
PC A
Fibre Channel and FCoE Technology 42
Copyright © 2013 EMC Corporation. All Rights Reserved.
Switching Frames Using MAC Addresses - Step 3
MAC TablePort 1 - PC APort 2 - EmptyPort 3 - EmptyPort 4 - Empty
PC A PC C
PC B
PC D
Port 1
Port 2
Port 3
Port 4
Frame
Dest Src
PC A PC D
PC D
Fibre Channel and FCoE Technology 43
Copyright © 2013 EMC Corporation. All Rights Reserved.
The Problem of Loops and Broadcast Storms
Switch A
Switch C
Switch B
Broadcast
Fibre Channel and FCoE Technology 44
Copyright © 2013 EMC Corporation. All Rights Reserved.
How STP Solves the Problem
• STP is a loop-prevention protocol
• Uses spanning tree algorithm to elect a Root Bridge and block certain ports.
Switch A
Switch C
Switch B
Block
Fibre Channel and FCoE Technology 45
Copyright © 2013 EMC Corporation. All Rights Reserved.
How Does STP Protocol Elect a Root Bridge?
Switch A
Switch C
Switch B
Priority = 32768MAC = 0000.1111.2222
Priority = 32768MAC = 0000.2222.3333
Priority = 32768MAC = 0000.2222.2222
• Switch with lowest Bridge ID (BID) wins election
• If all switches have equal priority, then switch with lowest MAC address wins
Since all switches are set to default priority, this switch will be elected Root Bridge because it has the lowest MAC address
Bridge ID
Fibre Channel and FCoE Technology 46
Copyright © 2013 EMC Corporation. All Rights Reserved.
Result of running STP Protocol
Switch A
Switch C
Switch B
Root Bridge
Designated Port (F)
Root Port (F)
AlternatePort (D)
Nonroot Bridge
Nonroot Bridge
Root Port (F)
Designated Port (F)
Designated Port (F)
• Forwarding ports Root ports Designated ports
• Discarding port Alternate
Fibre Channel and FCoE Technology 47
Copyright © 2013 EMC Corporation. All Rights Reserved.
Ethernet Hardware Overview
• Ethernet hardware supports the following speeds 10 Mbps 100 Mbps 1 Gbps 10 Gbps 40 Gbps 100 Gbps
Hardware Popular Implementations
NIC(Network Interface Controller)
• Stand-alone card• Built into host motherboard• Wireless
Cables
• Copper UTP (Unshielded Twisted Pair)
• Copper Twinax• Optic
Connectors• RJ45 (for copper UTP cables)• SFP+ (for copper twinax cables)• LC (for fiber optic cables)
Most common
Data Center Core (Backbone)
Fibre Channel and FCoE Technology 48
Copyright © 2013 EMC Corporation. All Rights Reserved.
Network Interface Controller - NIC
• Provides access to the network• Different implementations include:
Stand-alone card (could have multiple ports) Built into motherboard (most common now) Wireless
• Ports can be “teamed” to act as a single logical interface for failover
NIC LAN
Fibre Channel and FCoE Technology 49
Copyright © 2013 EMC Corporation. All Rights Reserved.
Example Ethernet Cabling
UTP (Unshielded Twisted Pair)
Twinax with SFP+Fiber Optic Cable with LC Connector
Media Copper Copper Optic
IEEE Standard 1000BaseT (802.3ab)
10GSFP+CU (Direct Attach)
1000BaseLX (802.3z)
Distance/speed
100 meters @1 Gbps 5 meters @ 10 Gbps 10 km @ 1 Gbps
Cat 5
RJ45
Twinax / SFP+ 9/125 SM FiberLC Connector
Fibre Channel and FCoE Technology 50
Copyright © 2013 EMC Corporation. All Rights Reserved.
Issues With a Flat Switched Network
• Single broadcast domain
• No Security between users
Broadcast
Fibre Channel and FCoE Technology 51
Copyright © 2013 EMC Corporation. All Rights Reserved.
VLAN200
VLAN200
VLAN100
VLAN100
VLAN Overview
• Each VLAN is a separate broadcast domain
• Each switch port is assigned to a single VLAN
• Devices in different VLANs must use a router to communicate
• VLAN Trunks allow the data from multiple VLANs to pass on a single link between switches
Trunk
Trun
k
Fibre Channel and FCoE Technology 52
Copyright © 2013 EMC Corporation. All Rights Reserved.
How a VLAN Solves the Problems of the Flat Network
• Smaller broadcast domains
• Security between VLANs
VLAN100
VLAN200
Trunk
VLAN100
VLAN200
VLAN100
VLAN200
TrunkTrunk
Broadcast Broadcast
Fibre Channel and FCoE Technology 53
Copyright © 2013 EMC Corporation. All Rights Reserved.
How it Works: VLAN Tagging
• 802.1Q - IEEE Standard Each port on switch assigned to specific VLAN ID Unassigned ports automatically belong to Default VLAN (VLAN ID = 0001)
• When port receives frame: Compare VLAN ID to filter table.
If valid, and MAC on another switch - forward to trunk port If destined to exit an access port on switch, remove VLAN ID
• Trunk carries both tagged and untagged traffic
Destination MAC6 bytes
Source MAC
6 bytes
802.1Q tag
(optional)
4 bytes
Type2
bytes
Data Payloadvariable # bytes
FCS4
bytes
Ethernet_II Frame Format
VLAN Tag
Fibre Channel and FCoE Technology 54
Copyright © 2013 EMC Corporation. All Rights Reserved.
This lesson covers the following topics:• FCoE Protocol and FCoE Fabric design basics
Lesson 3: FCoE Protocol Overview
Fibre Channel and FCoE Technology 55
Copyright © 2013 EMC Corporation. All Rights Reserved.
FCoE Protocol Stack
Volume managers, File systems, etc.
SCSI Command Descriptor Blocks, data, and responses
FCoE frame encapsulation / de-capsulation, FCoE entities
Frame switching, MAC address, transport Connection to the Physical Layer
Fibre Channel Protocol frame construction FC-4, FC-3, FC-2
Application
SCSI
FCP
FCoE
Enhanced Ethernet
FCoE Protocol Stack
Fibre Channel and FCoE Technology 56
Copyright © 2013 EMC Corporation. All Rights Reserved.
FCoE Encapsulation
SO
F
EO
F
EthernetHeader
FCoEHeader
Encapsulated FC Frame(including FC-CRC)
FC
S
FC Header
Payload
Fill
Dest. MAC
Address
SourceMAC
AddressVersion Reserved
VLANTAG C
RCEther
Type
Fibre Channel Frame
Ethernet Frame
FCoE PDU
Fibre Channel and FCoE Technology 57
Copyright © 2013 EMC Corporation. All Rights Reserved.
FCoE Frames
• Frame Efficiency iSCSI, FCIP, iFCP rely on IP over Ethernet which result in additional
overhead in frames FCoE frames only use Ethernet protocol Efficiency can be calculated by the formula
• Jumbo Frames Ethernet frames with more than 1500 bytes of payload Considered unreliable on regular Ethernet CEE allows reliable use of Jumbo Frames FCoE must use jumbo frames (2240 bytes)
Number of Data Bytes
Total Bytes Sent
Fibre Channel and FCoE Technology 58
Copyright © 2013 EMC Corporation. All Rights Reserved.
Converged Network Adapter (CNA) Overview
CNACNA = HBA + NIC + FCoE
Host
Fibre Channel and FCoE Technology 59
Copyright © 2013 EMC Corporation. All Rights Reserved.
• CNA is both a Network Adapter Fibre Channel adapter
• Host sends I/O to appropriate adapter: NIC – NFS, Web, Email, etc. HBA – SCSI block data
• Host receives data through lossless Ethernet MAC Directs I/O to appropriate adapter Based on Ethernet “Type”
Converged Network Adapter Architecture
2 x 10 Gbps Ethernet Ports
Ethernet NIC
Host Networking Drivers
FCoE Encapsulation
Existing FC Host Drivers
CNA Adapter
Lossless Ethernet MAC
Is it FCoE?NO YES
HBA Architecture
Fibre Channel and FCoE Technology 60
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fibre Channel Forwarder – FCF
• FCF function provided by FCoE switch or FCoE blade Services FCoE Login requests Provides services associated with a Fibre Channel switch
Stand-aloneFCoE Switch Fibre Channel Switch
with FCoE blade
Fibre Channel and FCoE Technology 61
Copyright © 2013 EMC Corporation. All Rights Reserved.
• Lossless Ethernet: FCoE requires reliable frame delivery Can’t drop frames from buffer overflow
• Improves regular Ethernet: Eliminates Ethernet loss, therefore more
reliable Speed increased to 10 Gbps
• Additions to Ethernet: Priority-based Flow Control (PFC) Enhanced Transmission Selection (ETS) DCB Capability Exchange (DCBX)
Notify neighbor node of DCB support
Data Center Bridging (DCB)
FCoE Requires LosslessEthernet
Fibre Channel and FCoE Technology 62
Copyright © 2013 EMC Corporation. All Rights Reserved.
Enhanced Ethernet Features
Feature Enhancement
Priority Flow Control IEEE 802.1Qbb – class of service flow control by enabling PAUSE functionality on IEEE 802.1p lanes
Data Center Bridging Exchange Auto-negotiation of Enhanced Ethernet capabilities DCBX (switch to NIC)
Bandwidth ManagementIEEE 802.1Qaz Enhanced Transmission Selection – manage bandwidth and assign priorities to groups of IEEE 802.1p lanes based on class of traffic.
Fibre Channel and FCoE Technology 63
Copyright © 2013 EMC Corporation. All Rights Reserved.
Priority-based Flow Control
STOP
Transmit Queues Receive Buffers
0
1
2
3
4
5
6
7
Buffers are almost full - send “PAUSE”
Priority
Fibre Channel and FCoE Technology 64
Copyright © 2013 EMC Corporation. All Rights Reserved.
Enhanced Ethernet - DCBX
• DCBX runs between the switch and CNA
• DCBX negotiates capabilities• Switch distributes configuration
to all attached adapters• Devices need to discover the
capabilities of its peers.• DCBCX utilizes the link-layer
discovery protocol and handles local operational configuration for each feature
Priority Flow Control
Data Center Bridging Exchange
Bandwidth Management
Rules for Capabilities Result
Capability and configuration match Enabled
Adapter configured to accept switch config. Enabled
Adapter not configured to accept switch config. Disabled
Adapter does not support/implement DCBX Disabled
Fibre Channel and FCoE Technology 65
Copyright © 2013 EMC Corporation. All Rights Reserved.
Enhanced Ethernet – Bandwidth Management
• Enables consistent management of QoS
• Provides consistent scheduling of different traffic types
Priority Flow Control
Data Center Bridging Exchange
Bandwidth Management
Fibre Channel and FCoE Technology 66
Copyright © 2013 EMC Corporation. All Rights Reserved.
Addressing
MAC = AFCID = 050100
MAC = BDomain ID = 05 Domain ID = 01
FCID = 011C00
FC SANConvergedNetwork
FC Frame
D_ID = 011C00S_ID = 050100
Dest = MAC BSource = MAC A
D_ID = 011C00S_ID = 050100
Ethernet Frame
Fibre Channel and FCoE Technology 67
Copyright © 2013 EMC Corporation. All Rights Reserved.
A Note About CNA MAC Addresses
FCoE Controller
Lossless Ethernet MAC
FCoE_LEP
VN_Port
FC-3 / FC-4 Layers
Ethernet_Port
Ethernet NIC
Host Networking Drivers
Existing FC Host Drivers
Physical MAC
ENode MAC
FPMA MAC
CNA
Fibre Channel and FCoE Technology 68
Copyright © 2013 EMC Corporation. All Rights Reserved.
How are MAC Addresses Assigned to the VN_Port?• Server-provided MAC Address (SPMA):
Uses single MAC Address for all FCoE traffic FCoE has MAC address independent of other protocols Support for different MAC addresses for each Virtual Port This method isn’t used
• Fabric-provided MAC Address (FPMA): Uses virtual port MAC addresses Each virtual port MAC address is supplied by the fabric Most common method Have the following format:
Fibre Channel and FCoE Technology 69
Copyright © 2013 EMC Corporation. All Rights Reserved.
Server Provided MAC Addresses (SPMA)
• Adapter uses burned-in or configured MAC address Consistent with the Ethernet model
• FCF needs a table to map between MAC addresses and FC_IDs
FCoE Node Port
MA
C
00.05.69.00.00.03
MAC Address
Fibre Channel and FCoE Technology 70
Copyright © 2013 EMC Corporation. All Rights Reserved.
Fabric Provided MAC Addresses (FPMA)
• MAC address assigned for each FC_ID: Consistent with the Fibre Channel model
• Uses OUIs with U/L = 1 FC-MAP (Local addressing) • No table needed for encapsulation
FC-MAP(0E-FC-00)
FCID07.08.09
24 bits 24 bits
48 bit MAC Address
FC-MAP(0E-FC-00)
FCID07.08.09
Fibre Channel and FCoE Technology 71
Copyright © 2013 EMC Corporation. All Rights Reserved.
FCoE Initialization Protocol (FIP) and FC Logins
CNA
CNA (ENode)Lossless Ethernet
Switch (FIP Snooping Bridge)
FCoE Switch(FCF)
Storage
FCoE Switch(FCF)
FC SAN
FC SAN
Storage
ENode (FCoE Controller)
Send FIP VLAN Request
Send FIP Solicitation
Send FIP FLOGI
FCF
Reply FIP VLAN Notification
Reply FIP Advertisement
Reply FIP FLOGI ACC
FIP Multicast
Fibre Channel and FCoE Technology 72
Copyright © 2013 EMC Corporation. All Rights Reserved.
FIP, the FCoE Initialization Protocol – How does it work?
Fibre Channel and FCoE Technology 73
Copyright © 2013 EMC Corporation. All Rights Reserved.
FIP VLAN Request
Fibre Channel and FCoE Technology 74
Copyright © 2013 EMC Corporation. All Rights Reserved.
FIP Discovery Solicitation
Fibre Channel and FCoE Technology 75
Copyright © 2013 EMC Corporation. All Rights Reserved.
FIP Discovery Advertisement
Fibre Channel and FCoE Technology 76
Copyright © 2013 EMC Corporation. All Rights Reserved.
FIP Virtual Link Instantiation Request (FLOGI)
Fibre Channel and FCoE Technology 77
Copyright © 2013 EMC Corporation. All Rights Reserved.
FIP Virtual Link Instantiation Request (FLOGI ACC)
Fibre Channel and FCoE Technology 78
Copyright © 2013 EMC Corporation. All Rights Reserved.
FIP Snooping Bridge (FSB)
Fibre Channel and FCoE Technology 79
Copyright © 2013 EMC Corporation. All Rights Reserved.
Question 2: How SP coordinates the host to handle hardware assist locks (Reservation, ATS)
Copyright © 2013 EMC Corporation. All Rights Reserved.
Verify VAAI Support Details
• The compare write(F1) operation is exclusively used by VAAI enabled VMware host
Copyright © 2013 EMC Corporation. All Rights Reserved.
VAAI – Hardware Assisted Locking• Scalable Lock Management – Atomic Test and Set (ATS)
• A number of VMFS operations cause the LUN to temporarily become locked for exclusive write use by one of the ESXi nodes, including:– Moving a VM with vMotion– Creating a new VM or deploying a VM from a template– Powering a VM on or off– Creating a template– Creating or deleting a file, including snapshots
• VAAI feature, atomic_test_and_set (ATS) allows the ESXi host to offload the management of the required locks to the storage and avoids locking the entire VMFS file system
Copyright © 2013 EMC Corporation. All Rights Reserved.
Atomic Test & Set• Original file locking technique
1. Acquire SCSI reservation2. Acquire file lock3. Release SCSI reservation4. Do work on VMFS file/metadata5. Release file lock
• New file locking technique
1. Acquire ATS lock2. Acquire file lock3. Release ATS lock4. Do work on VMFS file/metadata5. Release file lock
Note - The main difference with using the ATS lock is that it does not affect the other ESXi hosts sharing the datastore
Copyright © 2013 EMC Corporation. All Rights Reserved.
VMFS Scalability with Atomic Test and Set
Makes VMFS more scalable overall, by offloading block locking mechanism
Using Atomic Test and Set (ATS) capability provides an alternate option to use of SCSI reservations to protect the VMFS metadata from being written to by two separate ESXi hosts at one time.
Normal VMware Locking (No ATS)
Enhanced VMware Locking (With ATS)
Copyright © 2013 EMC Corporation. All Rights Reserved.
How to resolve SCSI Reserves on VMware ESXi servers• Refer to the following VMware KB articles:
– SCSI Reservation Issue with Fibre Channel HBAs (4365932) – Troubleshooting SCSI Reservation failures on Virtual Infrastructure 3.x and
vSphere 4.x (1005009) – VMware KB Article 1002293
• Warning! DO NOT release the reservation with INLINES when customer is using vSphere.
• It is also NOT recommended to try to release via Solutions Enabler
• The CORRECT way to release a SCSI Reserve in ESXi is to ask the customer to use OS tools as described in the VMware documentation and KB articles that are listed.
Copyright © 2013 EMC Corporation. All Rights Reserved.
Question 3: The terms or tricks to match WWPN with SP port, which would help us to understand mpath failover sequence
Copyright © 2013 EMC Corporation. All Rights Reserved.
During this lesson the following topics are covered:• PowerPath configuration and PowerPath’s function in the I/O
path• How PowerPath supports EMC and non-EMC arrays• Key functionality included in current releases of PowerPath• How PowerPath is licensed• Hardware, software, clusters, and storage arrays that
PowerPath supports
Lesson 1: PowerPath Architecture
87
Copyright © 2013 EMC Corporation. All Rights Reserved.
EMC PowerPath - A Family of Products
• PowerPath traditionally viewed only as “Path Management Software” Performed Multipathing, load balancing, failover, etc.
• Functionality Expanded PowerPath Family now has 4 distinct products
PowerPath can now be used for solutions previously handled by separate software
Common Interface allows for simplified management
• Optimized for heterogeneous host / storage environments
88
PowerPath Multipathing Path
Management /
OptimizationPowerPath/VE
PowerPath Migration Enabler Data
ProtectionPowerPath Encryption
Copyright © 2013 EMC Corporation. All Rights Reserved.
• PowerPath sits between the applications and the HBA driver
• Applications direct I/O to PowerPath
• PowerPath directs I/O to an optimal path based on current workload and path availability
PowerPath Configuration
89
O p
e n
S y
s t
e m
s H
o s
t
Logical volume manager
File system
PowerPath Driver
Applications
ManagementutilitiesDBMS
Storage Array
Host busadapter
Host busadapter
Host busadapter
Host busadapter
Storage Area Network
Copyright © 2013 EMC Corporation. All Rights Reserved.
• PowerPath keeps a table of paths
• Presents a single device to the application
• Combines equivalent paths into a single “Path Set”
Volume Path Set
90
PowerPath
HBAHBAS
ER
VE
RS
TO
RA
GE
Front-end Ports
Storage Area Network
Volume Path Set
Application
HBAHBA
POWERPATH
“Power”Device
Copyright © 2013 EMC Corporation. All Rights Reserved.
PowerPath Array Support Concepts
91
SE
RV
ER
ST
OR
AG
E
SE
RV
ER
ST
OR
AG
E
Active-Active Arrays Active-Passive Arrays
Owned by SP-A
HBA HBA HBAHBA
SAN
SP-A
Active
Active
SP-B
Passive
Passive
PowerPath
0 1 0 1
Host Application(s)
Mapped to allFront-end ports
Active
Active
Active
Active
SAN
Front-end Ports
HBA HBA HBAHBA
PowerPath
Host Application(s)
Copyright © 2013 EMC Corporation. All Rights Reserved.
• Dead indicates the path is
not usable. PowerPath driver
will not direct user I/O to this logical I/O path.
• Alive indicates the path is
usable The PowerPath
driver can direct I/O to this logical I/O path.
Device (LUN) Path States
92
SE
RV
ER
ST
OR
AG
E
Front-end Ports
Storage Area Network
Host Application(s)
HBA HBA HBAHBA Host BusAdapter
PowerPath
A
B
Copyright © 2013 EMC Corporation. All Rights Reserved.
PowerPath Licensing
• PowerPath Unlocks the full load balancing and path failover capabilities of PowerPath and
supports all EMC and certain non-EMC arrays Configures optimized default load balancing policies for EMC VMAX, VNX and
supported third-party arrays
• PowerPath/VE Full feature set specifically for virtual server environments based on VMware
vSphere and Windows Hyper-V
• PowerPath SE Backend (between switch and array) failover only Single HBA supported only No load balancing available Path Failover across two backend paths
93
PowerPath SE
Copyright © 2013 EMC Corporation. All Rights Reserved.
During this lesson the following topics are covered:• Benefits of PowerPath/VE• Features and functionality of PowerPath/VE• PowerPath/VE configuration in an ESXi environment
Lesson 2: Multipathing in ESXi
94
Copyright © 2013 EMC Corporation. All Rights Reserved.
What is PowerPath/VE?
• PowerPath advanced multipathing capabilities for virtual environments hosted on VMware vSphere
• Features include: Dynamic load balancing Device prioritization Automated optimization of server, storage, and path utilization Dynamic path failover and path recovery Monitoring performance and path status, I/O statistics and reports Automatic path testing Automatic path restore
95
x86 Architecture
PowerPath
Copyright © 2013 EMC Corporation. All Rights Reserved.
PowerPath/VE Key Benefits
• Automates the virtual server-storage connection: Assures tunable, predictable performance Optimizes server, storage, and data-path utilization Maximizes data availability
• Uses standard management features across physical and virtual environments: PowerPath/VE experience similar to rest of environment (physical servers) Common user experience across all classes of supported platforms
• Saves Money: Simple to implement application service levels for performance and
availability Eliminates need to monitor and rebalance the dynamic environment Reduces number of multi-pathing software variants to manage
96
Copyright © 2013 EMC Corporation. All Rights Reserved.
• PowerPath manages complexity: Constantly adjusts I/O path usage
and changes in I/O loads from VMs Simplifies provisioning by pooling
all connections Optimizes overall I/O performance
in VMware ESX environments
PowerPath/VE Load Balancing
97
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
OS
APP
ST
OR
AG
E
SAN
PowerPath PowerPath PowerPath PowerPath
Copyright © 2013 EMC Corporation. All Rights Reserved.
PowerPath/VE supports:• Changes to Permanent
Device Loss (PDL) handling • VMFS 5 Filesystem
vSphere 5.0 Kernel Support
98
EMC and Non-EMC Storage
Host Bus
Adapter
Host Bus
Adapter
Host Bus
Adapter
Host Bus
Adapter
vSphere KernelPowerPath/VE
Guest OS
Applications
DBMS
Manageme
ntUtilitiesFile System
Logical Volume Manager
PowerPath/VE runs in the kernel
Copyright © 2013 EMC Corporation. All Rights Reserved.
PowerPath/VE – Two Major Components
99
PowerPath Remote Management Server
ESX Hosts
PowerPath remote tools (rpowermt) installed
Used to remotely manage PowerPath/VE on any number of ESX hosts
PowerPath software is installed
Copyright © 2013 EMC Corporation. All Rights Reserved.
PowerPath Remote Management Server
• Supported platforms: Can be a physical or virtual host Check ESM for version compatibility
• Software: rpowermt
Sends PowerPath management commands to a remote ESX host Is the only way to manage PowerPath/VE (no local commands)
VMware vSphere CLI Installs PowerPath remotely onto an ESX host
License file Uses rpowermt to register license with an ESX host
• Can manage any number of ESX hosts.
100
PowerPath Remote Management Server
Copyright © 2013 EMC Corporation. All Rights Reserved.
PowerPath Remote Tools
• Are installed on the PowerPath Remote Management Server• Provide remote versions of existing PowerPath commands:
Example: Local command used by previous versions of PowerPath: powermt display dev=all Remote command, supported by PowerPath/VE: rpowermt host=<hostname> display dev=all
• Note: Remote Tools version and PowerPath version must be compatible: Possible to update either Remote Tools or PowerPath, without updating the
other
101
Copyright © 2013 EMC Corporation. All Rights Reserved.
Installation and Configuration Steps
1. Upload Files to ESXi Server2. Install PowerPath software on the ESXi server3. Install PowerPath Remote Tools on PowerPath Remote
Management Server (Optional)4. Configure and register license file
102
Copyright © 2013 EMC Corporation. All Rights Reserved.
Upload Files to ESXi Server
103
Create New Folder
Upload File or Folder
Copyright © 2013 EMC Corporation. All Rights Reserved.
Install PowerPath on ESXi
104
# cd /vmfs/volumes/Storage1/PPVE/# lsPowerPath_VE_5_7_for_VMWARE_vSphere-Install_SW_Bundle.zip# unzip PowerPath_VE_5_7_for_VMWARE_vSphere-Install_SW_Bundle.zip. . .# esxcli software vib install --depot=/vmfs/volumes/Storage1/PPVE/ PowerPath_VE_5_7_for_VMWARE_vSphere-Install_SW_Bundle/EMCPower. VMWARE.5.7.b173.zip
Installation Result Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true VIBs Installed: EMC_bootbank_powerpath.cim.esx_5.7.0.00.00-b173, EMC_bootbank_powerpath.lib.esx_5.7.0.00.00-b173, EMC_bootbank_powerpath.plugin.esx_5.7.0.00.00-b173
VIBs Removed: VIBs Skipped:
Copyright © 2013 EMC Corporation. All Rights Reserved.
Reboot ESXi Server
105
Copyright © 2013 EMC Corporation. All Rights Reserved.
Verify PowerPath /VE Installation
106
# cd /opt/emc/powerpath/bin/opt/emc/powerpath/bin # lspowermt/opt/emc/powerpath/bin # ./powermt versionEMC powermt for PowerPath (c) client Version 5.7 (build 173)EMC PowerPath (c) Version 5.7 (build 173)
Warning: License not installed.
/opt/emc/powerpath/bin #
Copyright © 2013 EMC Corporation. All Rights Reserved.
THANK YOU