1

USB Debugging and Profiling TechniquesKishon Vijay Abraham I and Basak Partha

2

AgendaAgenda

Introduction

USB Generic Linux System Architecture

USB Mass Storage Architecture

Challenges in debugging

USB Debugging Techniques (sysfs, usbmon, dynamic debug interface, tracepoint, protocol analyzer)

Gadget Zero

Other profiling tools

3

IntroductionIntroduction

Widespread use of USB in embedded space

HOST MODE:

To connect Ethernet/Hub

To connect Modem

To connect mass storage devices

DEVICE MODE

Acts as mass storage device

USB Speakers

USB serial device

USB webcam

4

IntroductionIntroduction

Ease of use

Hot pluggable

Speed

Reliability

Power devices from the bus

Bus expansion using hub

USB On-The-Go

5

USB Generic Linux System ArchitectureUSB Generic Linux System Architecture

PHY

Device Controller

Firmware Driver

Gadget Driver

Input, Sound, FS..Subsystem

USB Device Stack

UDC

PHY

Host Controller

Firmware Driver

VFS

HCD/usbcore

Class Driver

libusb

Application

ulpi/utmi ulpi/utmi

HSIC/TLL

Device Host

HOSTDEVICE

HWcomp

SWcomp

6

USB Mass Storage ArchitectureUSB Mass Storage Architecture

PHY

Device Controller

Firmware Driver

Mass StorageFunction Driver

USB Device Stack

PHY

Host Controller

Firmware Driver

SCSI

HCD/usbcore

Mass storage Class driver

ulpi/utmi ulpi/utmi

HSIC/TLL

Host

UDCUDC

VFS

Block Layer

Backing Store

Device

Block Layer

VFS

File System

HWcomp

SWcomp

7

Challenges in debuggingChallenges in debugging

Data on the bus is encoded

Timing issues

Out-of spec signaling errors

Protocol errors

Multiple layers makes it difficult to identify where exactly the problem originates

Too many formatted prints lead to skewed results

8

USB debuggingUSB debugging

Using linux kernel facilities

sysfs/debugfs

usbmon

Dynamic debug interface

Tracepoints

Using debug tools and Analyzers

Elisys/Lecroy/total Phase analyzer tools

ETM

9

Sysfs Entry in hostSysfs Entry in host

ls/sys/bus/usb/devices/

10:1.01111.111.1:1.011:1.0usb1

The names that begin with "usb" refer to USB controllers

The devices are named by a scheme

bus-port.port.port

The interfaces are indicated by suffixes having this form

:config.interface

All information about the device will be under this entry

TheDocumentationinformationforthisentrycanbeobtainedfromhttp://www.linuxusb.org/FAQ.html#i6

10

Debugfs Entry in hostDebugfs Entry in host

cat/sys/kernel/debug/usb/devicesT:Bus=01Lev=00Prnt=00Port=00Cnt=00Dev#=1Spd=480MxCh=3B:Alloc=0/800us(0%),#Int=1,#Iso=0D:Ver=2.00Cls=09(hub)Sub=00Prot=00MxPS=64#Cfgs=1P:Vendor=1d6bProdID=0002Rev=3.06S:Manufacturer=Linux3.7.0rc2next2012102600003g72580a5ehci_hcdS:Product=OMAPEHCIHostControllerS:SerialNumber=ehciomap.0C:*#Ifs=1Cfg#=1Atr=e0MxPwr=0mAI:*If#=0Alt=0#EPs=1Cls=09(hub)Sub=00Prot=00Driver=hubE:Ad=81(I)Atr=03(Int.)MxPS=4Ivl=256ms

T:Bus=01Lev=01Prnt=01Port=00Cnt=01Dev#=2Spd=480MxCh=5D:Ver=2.00Cls=09(hub)Sub=00Prot=02MxPS=64#Cfgs=1P:Vendor=0424ProdID=9514Rev=2.00C:*#Ifs=1Cfg#=1Atr=e0MxPwr=2mAI:If#=0Alt=0#EPs=1Cls=09(hub)Sub=00Prot=01Driver=hubE:Ad=81(I)Atr=03(Int.)MxPS=1Ivl=256msI:*If#=0Alt=1#EPs=1Cls=09(hub)Sub=00Prot=02Driver=hubE:Ad=81(I)Atr=03(Int.)MxPS=1Ivl=256ms

TheDocumentationinformationforthisentrycanbeobtainedfromDocumentation/usb/proc_usb_info.txt

11

Debugfs Entry in host contd..Debugfs Entry in host contd..

T:Bus=01Lev=02Prnt=02Port=00Cnt=01Dev#=3Spd=480MxCh=0D:Ver=2.00Cls=ff(vend.)Sub=00Prot=01MxPS=64#Cfgs=1P:Vendor=0424ProdID=ec00Rev=2.00C:*#Ifs=1Cfg#=1Atr=e0MxPwr=2mAI:*If#=0Alt=0#EPs=3Cls=ff(vend.)Sub=00Prot=ffDriver=smsc95xxE:Ad=81(I)Atr=02(Bulk)MxPS=512Ivl=0msE:Ad=02(O)Atr=02(Bulk)MxPS=512Ivl=0msE:Ad=83(I)Atr=03(Int.)MxPS=16Ivl=1ms

TheDocumentationinformationforthisentrycanbeobtainedfromDocumentation/usb/proc_usb_info.txt

12

Debugfs (in device controller)Debugfs (in device controller)

drivers/usb/dwc3/debugfs.c

Show the entire register dump

Change the port mode (device, host, OTG)

Force the controller to work at a particular speed (super, high..)

13

Debug example using debugfsDebug example using debugfs

PROBLEM STATEMENT

Added PM support for dwc3 and realized once the system goes to off mode and comes back the device is not enumerating. Adding some debug prints showed the we are not getting any interrupts in dwc3.

Configuration Diagram

MPUDWC3CORE

EVENTBUFFERHWWrite SWRead

interuptDWC3WRAPPER

14

Debug example using debugfsDebug example using debugfs Took the register dump before the system goes to suspend and after

resuming using cat /debug/dwc3.0/regdump

After going to OFF mode and coming back, DEVTEN register lost the contents.

DEVTEN controls the generation of device specific events.

The fix is to restore the contents of DEVTEN after coming back from OFF mode.

WorkingRegdump(BeforeSuspend)

NonWorkingRegdump(AfterResume)

15

usbmonusbmon

Facility in kernel to be used to collect URB traces

USB monitoring facilities for Linux consists of a kernel part and user part

Reports requests made by peripheral-specific drivers to Host Controller Drivers (HCD)

Requires a reliable working HCD

Has a "text" and "binary" API

16

usbmon Architectureusbmon Architecture

usbmon mainbinary

Character device(for each host

controller)

text

HCD

urb_submit/urb_complete/

add/remove bus Debugfs entry(for each host

controller)

/dev/usbmonX/sys/kernel/debug//usb/usbmon/Xu

tcpdump

wireshark vusb-analyzer

Userspacetools

Linuxkernelcomp

17

usbmon ASCII captureusbmon ASCII capture

mounttdebugfsnone_debugs/sys/kernel/debug

modprobeusbmon

ls/sys/kernel/debug/usb/usbmon/

0s0u1s1t1u

cat/sys/kernel/debug/usbmon/1u>usbmon.mon

./vusbanalyzerusbmon.mon

18

usbmon binary captureusbmon binary capture

modprobeusbmon

ls/dev/usbmon

usbmon0usbmon1

sudotcpdumpiusbmon1wusbmon.pcap&

./wiresharkusbmon.pcap

19

Decoding usbmon capturesDecoding usbmon captures

Usbmontrace

d2263780469874560SCi:3:003:0s80060100000000088usb_nw_dev.mon

catusb_w.mon|grep1:019>usb_w_dev.mon

./vusbanalyzerusb_nw_dev.monusb_w_dev.mon

The non working log has some additional commands compared to working log

21

Debug examples using usbmon contd.Debug examples using usbmon contd.

From the diff, there are additional commands in the non-working case

like the UDISK shown in the command

Then the URB return status shows broken PIPE error.

The host sends CLEAR ENDPOINT HALT and this happens again and again

So it's concluded some user space program send bogus commands causing the issue.

NONWORKINGLOG

22

Debug examples using usbmonDebug examples using usbmon

PROBLEM STATEMENT2

Webcam does not work when connected to USB 2 port but works on USB 3

Obtain usbmon traces for working and non working setup

Filter the usbmon traces for only the required devices

catusb2.mon|grep1:006>usb2_nw.mon

catusb3.mon|grep3:003>usb3_w.mon

./vusbanalyzerusb2_nw.monusb3_w.mon

23

Debug examples using usbmon contd.Debug examples using usbmon contd.

Both traces show that the webcam stopped being used for a short time and was suspended

when it was resumed again, it worked okay in USB3 but did not work in USB-2

The workaround was to disable auto-suspend (and debug the device)

NONWorkingusbmonlog Workingusbmonlog

24

Dynamic Debug InterfaceDynamic Debug Interface

Extended version of printk

Use dev_dbg and dev_vdbg to control debug messages

Enabled using DDEBUG and DVERBOSE_DEBUG compiler options

If CONFIG_DYNAMIC_DEBUG is NOT set, everything under dev_dbg turns to normal printk

If CONFIG_DYNAMIC_DEBUG is set

a new debugfs entry /sys/kernel/debug/dynamic_debug/control gets created

Writing to this file will enable or disable specific debugging functions

e.g., echo file gadget.c line 269 +p > .../dynamic_debug/control

25

Debug Example using Dynamic Debug Debug Example using Dynamic Debug InterfaceInterface

GOOD: out transfer on dwc3 device mode:[274.694458]=>>>queingrequested7cdea0toep2outbulklength512[274.694458]dwc3dwc3.0:ep2outbulk:reqed7cdea0dmaac9f0000length512last[274.713378]dwc3dwc3.0:ep2outbulk:cmd'StartTransfer'params00000000ad5ab11000000000[274.713378]dwc3dwc3.0:CommandComplete>0[274.726989]dwc3dwc3.0:ep2outbulk:TransferComplete[274.732482]=>>requested7cdea0fromep2outbulkcompleted16/512===>0

BAD: out transfer on dwc3 device mode:[217.927062]=>>>queingrequestecb1ecc0toep1outbulklength24[219.333374]dwc3dwc3.0:ep1outbulk:reqecb1ecc0dmaaca1b000length24last[219.333374]dwc3dwc3.0:ep1outbulk:cmd'StartTransfer'params00000000ad55e00000000000[219.349822]dwc3dwc3.0:CommandComplete>0[219.360168]dw