SHARKFEST '08 | Foothill College | March 31 - April 2, 2008

File and Disk Sharing ProtocolsApril 2, 2008

Richard SharpeSenior Software Engineer | Data RoboticsWireshark Contributor

SHARKFEST '08Foothill CollegeMarch 31 - April 2, 2008

SHARKFEST '08 | Foothill College | March 31 - April 2, 2008

Agenda

Disk serving/sharing protocols

File serving/sharing protocols

Ethereal issues

File Sharing or Disk Sharing

Network Oriented Disk or File blocks?

Disk/Storageblocks or

file blocks?

Disk Sharing Basics

LUN0

LUN1

Read 64 LBAs from LBA128 on LUN 0

Write 128 LBAs from LBA256 on LUN 1

LUN0

LUN1

Read 64 LBAs from LBA128 on LUN 0

Write 128 LBAs from LBA256 on LUN 1

Disk Sharing Dimensions

How many Logical Units (LUNs) Size of each LUN

Does the protocol restrict LUN size Management Storage protection

Mirroring or striping or P+Q Provisioning

Do you back all advertised blocks with real blocks? When do you alert user more storage is needed? Mixed LUN sizes?

Shared access?

LUN0

LUN1

Disk Sharing/Serving Protocols

Protocol SCSI ATA Proprietary

Transport USB FireWire (IEEE1394) Ethernet FibreChannel and FibreChannel over Ethernet (FCoE) Point-to-point cable (SATA, SAS, PATA, Parallel SCSI)

SCSI Introduction

Parallel 8, 16, and 32-bit cables Terminators, LVDS, etc

Serial Attached SCSI USB and FireWire TCP/IP – ie, iSCSI SCSI over FibreChannel SCSI over Ethernet - HyperSCSI

SCSI Standards Architecture

www.t10.org/scsi-3.htm

Original SCSI Model

Eight Phases Bus Free Arbitration Selection Reselection Command (CDB) Data Message Status

Parallel Cable8/16/32 bits

InitiatorTarget

Command (CDB)

Data In or Out

Status

SCSI Data Phases

COMMAND

• Target requests Command Descriptor Block (CDB)

• 6, 10, 12, 16 or variable byte CDBsBits

Bytes 7 6 5 4 3 2 1 0

0 Operation Code

1 Reserved (MSB)

2 Logical Block Address

3 (LSB)

4 Transfer Length or Parameter List Length etc

5 Control

SCSI Data Phases, Command

10-Byte CDBsBits

Bytes 7 6 5 4 3 2 1 0

0 Operation Code

1 Reserved Service Action (If required)

2,3 (MSB) Logical Block

4,5 Address (If required) (LSB)

6 Reserved

7 (MSB) Transfer Length or

8 Parameter List Length (LSB)

9 Control

SCSI Data Phases, Command

12-Byte CDBs Provides for 6-byte LBAs 32-bit transfer sizes

16-Byte CDBs Provides for 8-byte LBAs 32-bit transfer sizes

SCSI Data Phases

Data Transfer Target requests transfer to or from initiator

Status Target requests transfer of status to initiator

Message Target can request transfer of messages

See www.t10.org, eg http://www.t10.org/ftp/t10/drafts/spc2/spc2r20.pdf

USB Storage Details

SCSI over USB Bulk Only Transport Command Block Wrapper

Wraps CDB (max 16-bytes)

Data in (to host) or data out (from host) Command Status Wrapper

Wraps the status and residue No sense data

See: http://www.usb.org/developers/devclass_docs/usbmassbulk_10.pdf

Command Block Wrapper

• Sent to Bulk endpoint by the host

Bits

Bytes 7 6 5 4 3 2 1 0

0-3 dCBWSignature (0x43435355 – USBC)

4-7 dCBWTag

8-11 bmCBWFlags

13 Reserved (0) bCBWLUN

14 Reserved(0) bCBWLength

15-31 CBWCB(Up to 16 bytes)

Command Status Wrapper

• Sent on a Bulk In endpoint by the device

Bits

Bytes 7 6 5 4 3 2 1 0

0-3 dCSWSignature(0x53425355 - USBS)

4-7 dCSWTag

8-11 dCSWDataResidue

12 bCSWStatus

Protocol Dissection Issues

Capture files? Wireshark understands Linux usbmon captures

Request and response matching Match data in or data out to requests Conversations Using the existing SCSI dissector

Use the source

epan/dissectors/packet-usb-masstorage.c

USB Storage Details

ATA

Serial ATA ATA over Ethernet (ATAoE) SATA over FibreChannel ATA over FireWire?

Why not, given Firewire extension to 3+Gbps?

File Sharing Basics

File 1

File 2

File n

...

Open File 1, Read bytes 10-1000

Open File 2, Write bytes 100-2000

File Operations

• Create a file

• Delete a file

• Rename a file

• Open a file

• Read/write a file

• Lock portions of a file

• Change permissions

• Read/write file attributes

• Extend/truncate a file

Further File Sharing Dimensions

One tree of files per server or a forest Namespaces

Single network-wide namespace or one (or more) per server

Stateless vs Stateful Access permission model Sharing model Locking Caching mechanisms Performance

File Serving/Sharing Protocols

NFS NFS v3 NFS v4

CIFS/SMB NetWare Parallel NFS Andrew File System PanFS

NFS v3

RFC1813 (http://www.faqs.org/rfcs/rfc1813.html) Improvement of NFS v2 Stateless

Operates over UDP and TCP Server does not need to keep client-related state File handles encapsulate state

Based on XDR and SunRPC File attributes returned with most calls

Reduces number of calls and round trips

NFS v3 continued

Locking protocol external to NFS v3 NFS lock manager

Mount protocol separate

NFS v4

RFC3010 (http://www.faqs.org/rfcs/rfc3010.html) Servers keep state

Operates over TCP and UDP Open request establishes state Lease based, with release on timeout

Compound operations Locking Caching and delegation

CIFS/SMB

Originally NetBEUI Over Ethernet Frame Protocol

Server Message Block protocol Renamed CIFS in 1996 State based

TCP connection oriented NetBIOS header

Forest of trees Shares are the point of access

CIFS/SMB continued

Sharing defined in the protocol Read/Write/Delete specified at file open time

Locking part of the protocol Ranges of bytes can be locked for reading/writing

Client caching protocol part of the protocol OpLock based (callbacks)

CIFS/SMB Packets

Request/Response Protocol

• Basic Header fields repeated in request and response

• Makes response matching relatively simple

Header (Variable Length – Starts with 0xFFSMB)

Data, length defined by header

CIFS/SMB Header

Type Offsets Field

uchar 0-3 Protocol (0xFFSMB)

uchar 4 Command

ulong 5-8 Status (Several variants)

uchar 9 Flags

ushort 10-11 Flags2

uchar 12-23 Security/Extra

ushort 24-33 TID, PID, UID, MID

uchar 34 WordCount

ushort 35-xx ParameterWords[WordCount]

ushort xx+1 ByteCount

uchar xx+2 - yy Buffer[ByteCount]

packet-smb.c

Now look at the source code epan/dissectors/packet-smb.c

NetWare

Based on XNS Internet Packet Exchange Protocol – IPX

Network layer - connectionless

Sequenced Packet Exchange – SPX Transport protocol – connection oriented

Service Advertisement Protocol Very popular at one time

Parallel NFS

Extension to NFSv4 (http://www.pnfs.com/) Allows direct, parallel, access to storage devices Provides clients with layout information

Clients

Storage ServersBlock (FC), Object (OSD), File (NFS)

MetadataNFS v4.1

Metadata

ParallelDataAccess

Andrew File System

PanFS

These Protocols and Ethereal

Protocol dissectors reusable Capture formats can inject any packet types

Eg, SCSI over FC over Ethernet

State keeping

Problems

More capture file format need to be included Especially for proprietary tools

USB sniffers FireWire sniffers Other sniffer products