1

Solaris CIFS ClientInternals

Gordon RossProject Leadwww.opensolaris.org/os/project/smbfs

1

May 2008

2

Solaris CIFS Client Components

• nsmb – netsmb driver

• smbfs – Loadable file system module

• libsmbfs – Used by CIFS client commands

• smbutil(1) – List shares, lookup names, etc.

• mount_smbfs(1M) – Mount a CIFS share

• pam_smbfs_login(5) – Save p/w hash at login

• mdb(1)/kmdb(1) modules for nsmb and smbfs

3

SMBFS NSMB

VFS layer

mount_smbfs smbutilUser Apps

Data

CIFS Server

libsmbfs

Solaris CIFS Client Architecture

4

netsmb

smbfs

VFSvfs_t

smbmntinfo_t smbnode_t (FID)

vnode_t

smb_vc_t (UID)smb_share_t (TID)

network

In-Kernel Data Structures

5

Internals of nsmb ― Data Structures• nsmb data structures are a hierarchy,

the top of which is smb_vclist

• A virtual circuit (VC) in nsmb is an object that holds all state associated with a connection to one server with one set of credentials (smb_vc_t)

• Below VCs are share connections (smb_share_t)

• Below shares are file handles, but those are not managed by the nsmb driver (see smbfs)

• Message data carried in STREAMS mblk_t chains.Need ability to split and concatenate messages

6

Internals of nsmb ― MT Locking• Lock order is top-to-bottom in the hierarchy ―

see comments on locking in smb_conn.h

• With each VC is a list of current requests,and this uses a “readers/writer” lock

• A blocking semaphore is used to serialize sending of network data

• In most cases we don't need to hold multiple locks simultaneously

• The code has many locking assertions; use a debug build to test changes

7

Internals of nsmb ― Entry Points• ioctl calls from libsmbfs ― smb_dev.c: nsmb_ioctl(), smb_usr_X()

• Connection management ― smb_conn.c: smb_sm_X(), smb_vc_X(), smb_share_X()

• Request handling ― smb_rq.c:smb_rq_init(), smb_rq_simple()smb_rq_enqueue(), smb_rq_reply(), smb_rq_done()

• Packet building/parsing ― subr_mchain.c: mb_init(), mb_put_X(), mb_get_X()where X is uint8, uint16le, etc.

8

Internals of nsmb ― Build/Parse Request

• Create requestssmb_rq_init() or smb_rq_alloc()

• Mark start/end of word or byte sectionssmb_rq_wstart(), smb_rq_wend(),smb_rq_bstart(), smb_rq_bend()

• Build body of each part by callingmb_put_X(), where X is uint8, uint16le, etc.

• Dispatch the request smb_rq_simple()― see next slide

• Parse results with by callingmb_get_X() functions

9

Internals of nsmb ― Dispatch a Request

1.Put the request on the list ― typical call stack:smb_rq_simple / smb_rq_simple_timed /smb_iod_addrq, or smb_iod_multirq

2.Send pending requests (if necessary)smb_iod_sendall ― usually via _addrq

3.Wait for reply ― typical call stack:smb_rq_simple / smb_rq_simple_timed /smb_rq_reply / smb_iod_waitrq

● In step 3, the I/O Daemon (IOD) does its work...● “trans” requests may send/wait multiple times,

and those stay in the list until fully complete.

10

Internals of nsmb ― I/O Daemon

● The I/O Daemon (IOD) is a “reader” threadstarted for each active VC, running this loop:1.Wait for a packet and read it.2.Parse the SMB header for the “multiplex ID” (MID),

which identifies the request this response belongs to.3.Search the list of current requests for the one with the

matching XID. If found, link the response onto the list of responses and signal the condition variable, which continues the thread waiting for that response.

● Responses with no matching request happen after requesting thread wait timeout.

11

Request List Management

• All requests go on the request list• Only the first (M) of them are sent, where

M is the negotiated “max. multiplex” value• Every thread adding or removing requests

checks whether “send work” is needed.• Normally, the thread adding a request

will do its own send.• Avoid sends from the IOD thread due to

deadlock concerns (send could block).

12

Internals of smbfs ― Data Structures

• smbfs.h: smbmntinfo_tFS-specific, per-mount

• smbfs_node.h: smbnode_tFS-specific, per-vnode

• Node identity is: Path Name!No NFS-style file handle

• SMB file ID (FID) is held in smbnode_t,currently shared (carefully) by user open

• A lot like NFS, i.e. locking, zone support

13

Internals of smbfs ― Code Organization• Lots of files due to differing licenses,

tracking copies of originals> smbfs_vfsops.c: VFS operations vector> smbfs_vnops.c: Vnode operations vector> smbfs_node.c: Node identity and attributes> smbfs_subr2.c: Node hash (from NFS)> smbfs_subr.c: Other miscellaneous helpers> smbfs_rwlock.c: Reentrant read-write lock> smbfs_io.c: Input/output shim layer> smbfs_client.c: Zone code (from NFS)> smbfs_smb.c: message building & parsing

14

Internals of smbfs ― Surprises

• Single credentials per mount is unusual• All access uses the credentials of

the user who made the mount (!)• Owner, group, and modes are “fake”,

specified by client mount options• Server does most access checks,

client does “unix mode” checks• Some UNIX file names are illegal in CIFS• Other CIFS differences...

> No link, symlink, delete open file, ...

15

Internals of libsmbfs ― Data Structures

• smb_ctx_t ― “Handle” per server/share> Open FD into the netsmb driver> Copy of ioctl data going to or from driver> CIFS server address, etc.> Authentication information

• BSD-style “mbuf” simulation> Used for building/parsing RAP messages, etc.> Intentionally similar to in-kernel,

but not identical

16

Internals of libsmbfs ― Entry Points• Server or share context ― ctx.c:

> smb_ctx_init() ― Create a "handle"> smb_ctx_readrc() ― defaults, SMF, .nsmbrc> smb_ctx_resolve() ― Connect, authenticate> smb_ctx_lookup() ― Does tree connnect (!)

• Remote administration protocol (RAP) > rap.c: smb_rap_X()

• List shares using RAP: netshareenum.c

• Build & parse network messages:> mbuf.c: mb_get_X(), mb_put_X()

where X is uint8, uint16le, ...

17

Internals of libsmbfs ― Authentication

• smb_ctx_resolve() drives authentication, but actions differ significantly by authentication type:> NTLM uses simple password hash, computed

in the driver during ioctl SMBIOC_NEGOTIATE> With NTLM, the driver may be asked to get the

password from the “keychain” instead of using one passed in See: smb_dev.h: SMBVOPT_*

> Kerberos, used with Active Directory (AD), requires SPNEGO protocol (see: libsmbfs)In this case, the driver treats the auth. data as opaque and copies to or from the libsmbfsSee struct members: ioc_intok, ioc_outtok

18

Internals of smbfs Commands• cmd/fs.d/smbclnt/smbutil/view.c

> smbutil view ― Nice, simple example> Uses the RAP functions in libsmbfs

• cmd/fs.d/smbclnt/mount/mount.c> Mount is almost as simple

• Both basically do> smb_ctx_init()> smb_ctx_readrc()> smb_ctx_resolve()> smb_ctx_lookup()

19

mdb modules ― usage• Examine nsmb data structures:

> ::nsmb_vc -rv (List all VCs, shares)> ADDR ::nsmb_rqlist (All requests on a VC)> ::nsmb_pwtree (List all p/w hashes)

• Examine smbfs data structures:> ::smbfs_vfs (List all smbfs mounts)> ::smbnode (List all smbfs nodes)

• Example outputs (next slide)

20

mdb modules ― sample output$ mdb -k 0

> ::nsmb_vc -rv

// smb_vc_t uid server user

d71331c0 0 CHELUB cifs_user

d824f558 PUBLIC

> ::smbnode

// smbnode vnode rpath

d84d6b48 d85e5540 \file

e5056e20 dd88ac00 \

21

Handy dtrace Probes

• Debug messages use dtrace probes• To see debug messages, use:

#!/usr/sbin/dtrace -ssdt:nsmb::debugmsg2{ printf("\n\t %s: %s", stringof(arg0), stringof(arg1));}sdt:nsmb::debugmsg3{ printf("\n\t %s: %s", stringof(arg0), stringof(arg1)); tracemem(arg2, 16);}

• More dtrace scripts on our wiki

22

More Information• CIFS Client Project (smbfs) on OpenSolaris:

http://opensolaris/org/os/project/smbfs

• External Wiki: (search for CIFS)http://www.genunix.org/wiki

• Mailing lists:cifs-discuss@opensolaris.org