fast 2d detector control at esrf beamlines

A. Homs-Puron, NOBUGS 2008, Sydney, 3-5 Nov 2008

Fast 2D Detector Control at ESRF beamlines

David Fernández Alejandro Homs

Emmanuel PapillonManuel Pérez

Vicente ReyCyril Guilloud V. Armando SoleAssen Kirov


Talk outline

•Introduction

ESRF FReLoN CCDs

2D control arquitecture

•Espia card based systems

Espia card

FReLoN 2k, Maxipix

Dalsa, Sarnoff, Aviex

•Latest developments

Driver improvements

Data saving

Common CCD acq. platform


FReLoN 1000 (1996):

• 1024 x 1024 - 14 bit• 4 parallel readout channels• Readout time: 100 ms

FReLoN 2000 (1998):

• 2048 x 2048 - 14 bit• Readout time: 220 ms

FReLoN – Fast Readout & Low Noise


FReLoN 1000/2000 acquisition setup

Gigabit Ethernet

TACO Device serverTACO Device server

SPECSPEC


Princeton Sensicam, PCO Photometrics Matrox Meteor II [H. Witsch, J. Meyer] Trixell Pixium [L. Claustre] Andor ImagePro [A. Gotz] Direct-Show

Other CCD-like devices supported

Mar CCD [L. Claustre] IDS Falcon [D. Miraut, J. Meyer] Firewire (IEEE 1394) [A. Gotz] Dalsa Dalstar


ADSC Q315r for MX beamlines

Gigabit Ethernet

Local Gigabit


Designed to interface the FReLoN 2k Collaboration ESRF + SECAD, S.A. 2 Gbps fiber optic link (> 100 m) PCI 64 bits / 66 MHz ~180 MB/s maximum data rate Bus master + Scatter Gather + IRQ Driver source code available rewritten!

Can write on 32-bit addresses only (4 GB)

Espia card


Control workstation (2005)

Transtec (2005): Dual Intel Xeon 2.6 GHz, HT 2 GB ECC DDR-RAM 3 x 73.4 GB Ultra320 SCSI 15Krpm SuSE 8.2 + Linux 2.4.20-64GB-SMP

• Remote NFS through 1 Gbps link 90 MB/s• 2 disks in software RAID 0 100 MB/s (8 MB files)• 4 disks in software RAID 0 150 MB/s (1 GB files)

for ultra-fast tomography


New FReLoN Generation

FReLoN 2k (ATMEL): No Mux – only one fiber optic! Frame Transfer Mode 2048 x 1024 @ 16 fps 66 MB/s

FReLoN 4M (KODAK): Only 4 channels – Full Frame Mode

FReLoN HD (PSB-2): 2048 x 1024 @ 32 fps ~125 MB/s


• Connects to the Espia• 2 x Camera Link connectors:

• Base configurationMultiplexed• Multiple pixel packing formats

• Test image generator @ ~180 MB/s

Camera Link: FOCLA


Frame transfer technology 1024 x 1024 @ 60 fps 120 MB/s 1 x Camera Link [Base]

Connector Compact! Image not reconstructed FOCLA does reconstruction

Dalsa Pantera 1M60


Dalsa setup

Fast online displayImage reconstruction


512 x 512 @ 300 fps 150 MB/s 512 x 128 @ 500 fps 125 MB/s 2 x 8 ADCs – 12 bit 2 x Camera Link [Base]

Connectors Separate RS-232 serial line

Sarnoff CAM512


Sarnoff setup

Camera Link

Fiber optic

RS-232


Sarnoff results (2007)• 300 fps• 75 frames• 250 ms• 40x

slowdown

No sound! B&W Flickering

(room light)

ID15: Tomography

in 0.5 sec @ 500 fps


The real image

Espia #1

Espia #2


Image reconstruction in kernel

Espia #1

Espia #2


“Very High memory” support

Physical Memory

Low RAM0 MB

3.2 GBPCI

Very high RAM

800 MB

4 GB

5.8 GB

High RAM

32 bit

boundary

• Driver always has 32-bit aux. buffers• If next target frame has 64-bit pages:

Program each Espia to write on aux buffer

Frame end (all adapters finished) copy from aux. into target buffer

Inform waiting threads• Adapters are not synchronous• Memory copy with IRQ disabled

Kernel thread

Implementation in driver

High frame rate many image buffers


Aviex PCCD-8448

2048 x 3584 @ 1.2 fps 18 MB/s

Binning 16 x 16 8 fps


Software image reconstruction

ADC

ADC

ADC

ADC

• Per frame operation• Warranted only once in a multi-frame (big) buffer• Callback thread safe


Medipix2 / Maxipix

Pixel detector photon counting 256 x 256 – 13 bit 0.3 ms readout time 1000 fps 130 MB/sMax: 1400 fps 180 MB/s

Chipboard: University of Geneve PRIAM: ISG/ESRF Interfaced to the Espia

Espia


Maxipix results (2007)

• 1000 fps !!


Multichip MaxipixMedipix2 array 5 x 1 1280 x 256

1000 fps 650 MB/s PRIAM multiple FO links Future configs: 2 x 2, 5 x 2


New control workstation

OPALE server from Ecrin Systems (FR):

Trenton Single Host Board and backplane Dual Quad Core Intel Xeon @ 2.33 GHz 8 GB of FB DDR2 3 x 140 GB SAS disks @ 15 Krpm

Chelsio T320 10 Gbps Ethernet controller


Driver evolution

32-bit application 3GB address space 8GB RAM Dynamic mapping of buffersMove to 64-bit architecture No so trivial ESRF/Linux 2.1 [Red Hat Enterprise Linux 5]Kernel version 2.6.18Copy to very-high-memory: 1.3 GB/s


Sarnoff @ ID15

512 x 512 12 bits @ 300 fps 150 MB/sWrite to local disks: 300 MB/sWrite to remote disks: 300 MB/s

NetApp FAS6070NetApp

FAS6070

10 Gbps Ethernet10 Gbps Ethernet


“La Bestia”Dell PowerEdge 2900 Similar to Ecrin/Trenton + 16 GB RAM 4 + 6 = 10 disks 530 MB/s (800 MB/s raw) Super Sarnoff: 2x FOCLA test imageAcquisition rate: ~360 MB/s

Running


Turbo Bestia 4 + 6 + 8 = 18 disks Single RAID 0 array 840 MB/s raw 3 RAID 0 arrays + parallel write 1.5 GB/s raw

4x

6x

8x


Espia Next Generation

PCI-Express 8x lanes 4 fiber optic links @ 250 MB/s 1

GB/s Full access to 64-bit memory Maximum register compatibility


Common CCD acq. Library

Reuse of common code generic procedures + interfaces Software “features” fallback if hardware has limited

capabilitiesHigh level Python implementationCommon TACO CCD interface Interfaces: implicit explicit Image processing and data reduction as plugins (C++) Extensible interface through user-defined “features”

(Maxipix) Generic TANGO interface Collaboration framework (ALBA)


Current status

x 10 + 1[+1] (C++)

x 1 (Python)

x 20

x 5 + 1


Conclusions

Significant number of supported 2D detectors This number keeps raising … New detectors demand high performance control Experience in high speed acquisitions

Stable low level driver (Sarnoff failed only once…)Functional high level interface

Never-ending project … (sub-projects)Common image acquisition library will help!


Acknowledgements

SECAD, S.A. F. Hereson

Instrument Support Group:

J.C. Labiche E. Collet L. Siron J.J. Thevenin D. Pothin C. Ponchut J.M. Rigal J. Clement P. Fajardo

Computing Services:

P. Makijarvi B. Regad P. Pinel L. Garçon F. Andrei D. Gervaise S. Ohlsson B. Lebayle


Thank you!

fast 2d detector control at esrf beamlines

Documents