epics 2011 spring collaboration meeting, hsinchu, june 13-17, 2011 pcigeneral pci device support of...
TRANSCRIPT
EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
pciGeneralPCI Device Support of EPICS
which is used in TPS Control System
Presented by: Jenny ChenTPS Control Group
June 14, 2011
2EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Outline
Goals of the Device Support –
pciGeneral
Rules of the Kernel Driver for PCI Cards
Details of the Kernel Driver Programs
Details of the Device Support Programs
Performance Tests
Access TLS control system via EPICS
3EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Goals of the pciGeneral Programs
Manipulate as many PCI cards as possible
Manipulate the relevant resources
Handle interrupts
Convert byte order (big to little endian) if
needed
Keep maintenance programs to a minimum
4EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Manipulate as Many PCI Cards as Possible Generalized kernel driver + pciGeneral + asynDriver(EPICS),
all PCI cards we have were tested successfully The tested PCI cards: cPCI-7452, 24DSI32, cPCI-EVG-300,
cPCI-EVR-300, TCP201 + DAC8402, TCP201 + DAC8415, TCP201 + ADC8417
Properties of PCI cards:– address space: IO-PORT or MEMORY
– DMA
– byte order: little-endian or big-endian
– daughter boards One kernel driver may handle several cards of the same model Special kernel driver (without accessing hardware): creates
shared memory – may exchange data between processes
5EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Manipulate the Relevant ResourcesExtract 3 regions of the address spaces of a PCI card
Fully access the address spaces
plx -- the chip sets of PCI interface, ex: plx9000 series,
the mostly accessed registers are interrupt control and
status
card – includes the main control/status registers of the
PCI cards which are mostly for configuration
data – separated from the main registers (card), ex:
DMA, transient data or processed data (may be done by
the kernel driver or a running process)
6EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Handle Interrupts
APs call read() to wait for the interrupts instead of
using ioctl() or signal(), which is easy to program
APs may directly access interrupt registers (plx), ex:
reset the registers when interrupts are not handled
properly
Kernel drivers serve enable or disable interrupts that
are requested by APs (ioctl)
7EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Keep Maintenance Programs to a Minimum
Keep the software simple and small
Minimize the maintenance of IOC software
One kernel driver program per card + 2 device
programs– Ex: kernel driver: pci_adl7452.c for cPCI-7452– EPICS device support programs: pciGeneralDrv.c,
pciGeneralLinux.c
8EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Rules of Kernel Driver Implementation
Create char device file in /dev, file name ex: pciCardName-x,
x: pci-slot, ex: pciadl7452-1, pciadl7452-2
ioctl() -- get the card information, enable/disable the interrupt,
setup/start/abort DMA, special service for the card
mmap() -- map the PCI address spaces into AP's
read() -- block AP to wait for an interrupt
9EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Details of a Kernel Driver Program
Interface to a PCI card
Create a char device interface file
Serve char device access requests --
open/close/ioctl/mmap/read
10EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Interface to PCI cardCreate char device interface file
#define MY_VENDOR_ID 0x144A
#define MY_DEVICE_ID 0x7452
#define SUB_VENDOR_ID PCI_ANY_ID
#define SUB_DEVICE_ID PCI_ANY_ID
#define MYCARD_NAME "pci7452"
static struct pci_device_id MY_DRIVER_PCI_TABLE[2] __devinitdata = {
{MY_VENDOR_ID, MY_DEVICE_ID, SUB_VENDOR_ID, SUB_DEVICE_ID, 0, 0, 0},
{0, 0, 0, 0, 0, 0, 0}
};
Declaration
static struct file_operations mypci_fops = {
.owner = THIS_MODULE,
.open = mypci_open,
.read = mypci_read,
.ioctl = mypci_ioctl,
.mmap = mypci_mmap,
.release = mypci_release,
};
11EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Interface to PCI cardCreate char device interface file
Probe callback
- alloc_chrdev_region allocate char device resource, assign the char device name (should be unique)
- cdev_jnit
- cdev_add
- class_create create a class in directory /sys/class, ex: /sys/class/pci7452
- device_create create a device in /sys/class/device, ex: /sys/class/pci7452/pci7452-1
to insert/remove kernel module dynamic – according to a rule listed in /etc/udev/rules.d, the udev daemon sends
uevent to OS to create interface file: /dev/pciadl7452-1 (kernel version: linux2.6.13 or higher)
Create device interface file:
Request to access PCI address spaces- pci_request_regions request pci resources and associate with the device file: pci7452-1
- pci_resource_start get pci address spaces information
- pci_resource_end
- pci_resource_len
- pci_set_master to be the dma master
Initialize a waiting queue for interrupt request and variables
- init_waitqueue_head, atomic_set- kzalloc
12EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Interface to PCI cardCreate char device interface file
Remove callback
- device_destroy destroy created device structure, ex: delete /sys/class/pci7452/pci7452-1
- class_destroy destroy created class structure, ex: delete /sys/class/pci7452
- cdev_del delete cdev structure
the char device interface file will be removed then
Delete device interface file:
Release PCI resources
- pci_release_regions release pci resources
- pci_disable_device release pci access
Free allocated memory
- kfree
13EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Provide char device interface: open/close/ioctl/mmap/read
- open allow AP to access the char-device, call request_irq
- close release to access, call free_irq
- ioctl serve command: IOCTL_CARD_INFO, IOCTL_PLX_INFO, IOCTL_DATA_INFO,
IOCTL_INT_ENABLE, IOCTL_INT_DISABLE, IOCTL_DMA_SETUP, IOCTL_DMA_START,
IOCTL_DMA_ABORT, IOCTL_INITIAL, IOCTL_COMMAND
transfer PCI address information to AP, process interrupt requests, process DMA requests,
special service request to the kernel driver
- mmap map PCI address (memory type only) to user process address space: card, plx and data
- read serve interrupt request: block AP, wake up AP and copy interrupt count to AP
Serve the char device access requests
14EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Details of the Device Support Programs for EPICS
They are based on asynDriver Programs:
– Header: pciGeneral.h
– pciGeneralDrv.c – the interface to EPICS, create records, register interrupt, parse arguments, ...
– pciGeneralLinux.c -- the interface to PCI cards, open/close device file, mmap pci address spaces, access the io-port/memory and create interrupt threads (call read() to wait)
15EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
EPICS Record types of pciGeneral
bi/bo/mbbi/mbbo ai/ao/longin/longout Waveform with DTYP: asynInt32ArrayIn/Out,
asynFloat32ArrayIn/asynFloat32ArrayOut asynFloat64ArrayIn/asynFloat64ArrayOut
16EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
DTYP: asynUInt32Digital for bi/bo/mbbi/mbbo
. bi/mbbi INP: @asynMask(portName,byteOffset,bitMask,timeout) command location
command: input/input16 location: plx/card/data
. bo/mbbo OUT: @asynMask(portName,byteOffset,bitMask,timeout) command location
command: output/output16/write/write16 location:
plx/card/data
(output/output16 – get readback first)
. bo (special) OUT: @asynMask(portName,0,0,timeout) command location actionState
command:
intEnable/intDisable/initial/dmaSetup/dmaStart/dmaAbort location: card
actionState: 0/1/-1 (0/1:process when state is 0/1, -1:process every
time)
INP/OUT Field of EPICS Record
record(bi,"diBank0Ch0") {
field(SCAN, “I/O Intr”)
field(DTYP,"asynUInt32Digital")
field(INP,"@asynMask(pci7452-1,0xc,0x1,1.0) input
card")
field(ZNAM,"low")
field(ONAM,"high")
}
record(bo,"doBank0Ch1") {
field(DTYP,"asynUInt32Digital")
field(OUT,"@asynMask(pci7452-1,0x0,0x2,1.0) output
card")
field(ZNAM,"low")
field(ONAM,"high")
}
appendix
17EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Db example
record(mbbi,"erClockControlSts") { field(DTYP,"asynUInt32Digital") field(INP,"@asynMask(er230-2,0x050,0x0000ffff,1.0) input card") field(DESC, "Clock Control Status")}
record(mbbo,"setIntMaskBank0") { field(DTYP,"asynUInt32Digital") field(OUT,"@asynMask(pci7452-1,0x8,0xffffffff,1.0) write card")}
record(bo,"enableInt7452") { field(DTYP,"asynUInt32Digital") field(OUT,"@asynMask(pci7452-1,0,0,1.0) intEnable card 1") field(ZNAM,"null") field(ONAM,"enable") field(DOL, "clearIntBank0") field(OMSL, "1") field(HIGH, “0.1”) field(FLNK, "enableIntBank0")
}
st.cmd: pciGeneralDrvInit("tcp201-0","/dev/pcitcp201-0-IP"))
pciGeneralDrvInit("adc8417-0","/dev/pcitcp201-0-IPB"))
pciGeneralDrvInit("pci7452-1","/dev/pci7452-1")) pciGeneralDrvInit("er230-2","/dev/er230-2"))
appendix
18EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
DTYP: asynInt32 for ai/ao/longin/longout
. ai/longin INP: @asyn(portName,byteOffset,timeout) command location extractMask
command: input/input16, location: plx/card/data, extractMask: extract
several bits of 32/16 bits
. longin (special) INP: @asyn(portName,0,timeout) intCount card
. ao/longout OUT: @asyn(portName,byteOffset,timeout) command location extractMask
command: output/output16/write/write16/ioctl, location: plx/card/data
extractMask: extract/update several bits of 32/16 bits
INP/OUT Field of EPICS Record
record(longin,"diBank0") {
field(SCAN,"I/O Intr")
field(DTYP,"asynInt32")
field(INP,"@asyn(pci7452-1,0xc) input card 0xffffffff")
}
record(longout,"erPrescaler0Set") {
field(DTYP,"asynInt32")
field(OUT,"@asyn(er230-2,0x100) output card 0x0000ffff")
}
record(longin,"intCountEg230") {
field(SCAN,"I/O Intr")
field(DTYP,"asynInt32")
field(INP,"@asyn(eg230-3,0x0) intCount
card")
}
Ex: request kernel driver service: Ioctl card codecode parameterCode may be the pv value
record(longout,"$(IOC)-adc8417-$(S):ioctlCmd") { field(DTYP,"asynInt32") field(OUT,"@asyn(adc8417-$(S),0x0) ioctl card 0 $(HNUM)") field(DOL, "$(IOC)-adc8417-$(S):dataProc") field(OMSL, "1") field(FLNK, "$(IOC)-adc8417-$(S):ch0WfProc")}
appendix
19EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
DTYP: asynInt32ArrayIn/asynInt32ArrayOut for waveform
. waveform input
INP: @asyn(portName,byteOffset,timeout) command location extractMask increaseAddrOffset
command: inArray/inArray16, location: card/data
. waveform output
INP: @asyn(portName,byteOffset,timeout) command location extractMask increaseAddrOffset
command: outArray/outArray16, location: card/data
IncreaseAddrOffset depend on the data layout:
typeA ch0ch0ch0... one data block for one channel
(addrOffset:2/4)
ch1ch1ch1...
typeB ch0ch1ch2... one data block for N channels
(addrOffset:2N/4N)
INP/OUT Field Format of EPICS Record
record(waveform, "dac8402ch0Wf") {
field(DTYP,"asynInt32ArrayIn")
field(INP,"@asyn(dac8402-1,0x0) inArray16 data 0xffff
2")
field(NELM, "2048")
field(FTVL, "LONG")
field(PINI, “YES”)
}
record(waveform, "dac8402ch0WfOut") {
field(DTYP, "asynInt32ArrayOut")
field(INP,"@asyn(dac8402-1,0x0,1) outArray16 data 0xffff
2")
field(NELM, "2048")
field(FTVL, "LONG")
field(FLNK, "dac8402ch0Wf")
}
appendix
20EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
DTYP: asynFloat32ArrayIn/asynFloat32ArrayOut for waveform
INP: @asyn(portName,byteOffset,timeout) command location extractMask increaseAddrOffset factor
offset
input waveform command: inArray/inArray16, location: card/data
output waveform command: outArray/outArray16, location: card/data
IncreaseAddrOffset: depend on the data layout
INP/OUT Field Format of EPICS Record
record(waveform, "dac8402ch0WfE") {
field(DTYP,"asynFloat32ArrayIn")
field(INP,"@asyn(dac8402-1,0x0) inArray16F data 0xffff 2 3.051e-4 -
10")
field(NELM, "2048")
field(FTVL, "FLOAT")
}
Refer to: http://www.icg.nsrrc.org.tw/wiki/EPICS_Drivers
appendix
21EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Performance Tests
int_value = *((int *)mapped_ptr ;
vs.
ioctl(fd, IOCTL_READTEST, &int_value) ;
- Memory access reduces about 25% CPU time compared to ioctl
- Interrupt service test – EPICS client:
get the 5ms interrupt counter by LabCA (matlab)
ADLINK cPCI-6910:
CPU:dual-core Intel Xeon 2GHz,
cPCI:64bit/66MHz
Jitters:. OS scheduler. Dmc controller + network
22EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Interrupt Service Test
- Interrupt test: the response time of a 5ms interrupt triggered by a
network process (measured by an AP)
Shared memory
kernel driver
char dev file: /dev/dmc4000-0
Data receiver
(from DMC4000)
Ethernet
as field bus
DMC4000 controller
send data
UDP
at every 5ms
Prepare for insertion devices control:
the field compensation by setting
correction coils (setting follows gap/phase)
needs to be done as fast as possible.
Wake up(read)
Ethernet kernel driver
(socket interface UDP)
open/mmap
ioctl
Control NetworkEPICS
ioc
EPICS ClientLabCA
open/mmap
23EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Interrupt Test with cPCI-7452 DIO Card- Hardware interrupt test: connect a TTL signal to a DI channel, the state change would trigger the
PCI interrupt, the figures show the response time of 100Hz and 200Hz signal; the reason of some missing changes was unknown but it was probably not caused by the OS scheduler (maybe the interrupts were not delivered)
cPCI-7452 kernel driver
char dev file: /dev/pci7452-0
Test Program
open/mmap/read
Test condition:1. hi/low signal connect to ch02. testing loop: 500003. test process priority adjusted
cPCI-7452 DI Card
Wake up
Interrupt
200Hz/100Hz
ADLINK cPCI-6910:
CPU:dual-core Intel Xeon 2GHz,
cPCI:64bit/66MHz
24EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
EPICS ioc and LabCA Monitor Test- update test: the ioc updates data at every 5ms, the interrupts trigger
the ioc to update the dmc4000 and cPCI-7452 DIO channels
Shared memorykernel driverchar dev file:
/dev/dmc4000-0
Receive data
sent by DMC4000
Private Ethernet
DMC4000 controller send data
UDPat every 5ms
Wake up
Ethernet kernel driver
open
mmap
Ethernet
EPICS ioc
EPICS ClientLabCA
cPCI-7452 DI Card
Interrupt
Figure 1: ioc CPU load: 8%(top/Irix), db record: about 40
Figure 2: ioc CPU load: 20%(top/Irix), db record: about 170
Note: in figure 2, 128 DI
records scaned but not
in figure 1
kernel driver
char dev file:
/dev/pci7452-0
Wake
up
100Hz
ADLINK cPCI-6910:
CPU:dual-core Intel Xeon 2GHz,
cPCI:64bit/66MHz
25EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Access Existing TLS Control System via EPICS Modify pciGeneral to tlsControl, because the data byte order
of TLS control system is not standard Start up a softIOC(+tlsControl) on a TLS console as a
gateway A simple char device driver (kernel) allocates memory to store
DDB which is updated by TLS ILCs and serves file operations: open/read/ioctl/mmap
TLS DDB-receiver flushes the DDBs into the memory which is also mapped by the softIOC, then two control systems access the DDB simultaneously
With the kernel driver running, the softIOC may be waked up by the interrupt which is triggered by the DDB receiver in order to update data synchronously
To control TLS devices from EPICS, streamDevice is used, a server waits for the string commands (name+value) then converts it (access SDB of TLS) to the format that ILC accepts
26EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
TLS Network
TLS ILCs
TLS control
software on consoleEPICS
IOC
Shared memorykernel driverchar dev file:
/dev/ilc-x
Setting server for
EPICSStreamDeviceTCP socket interface
TLS setting functions
TPS Network
File interface mmap()ioctl()File interface
mmap()read()
TLS SDB/DDB
The Conjunction for TPS and TLS Control System
10Hz DDB
On-demand Setting
Wake up
DDB Upload
Setting server
TLS ILC control
system
Run EPICS ioc on an ILC of TLS control system
TLS ILC
kernel driverchar dev file
TLS Network
EPICS IOC
TCP socket
Access any device of TLS control system via EPICS
TLS Console
Setting server (ca-lib) for ILC
IO hardware
27EPICS 2011 Spring Collaboration Meeting, Hsinchu, June 13-17, 2011
Summary
• Efficiency
• Flexibility
• Maintenance