25th & 26th august 2009icat developer workshop 1

125th & 26th August 2009

ICAT developer workshop

NXextract

Extracting too l for NeXus filesS téphane P o ir ie r – S ynch ro tron S o le il



Overview

Introduction

Writing templates

Real example

Main usages

Future



Introduction

Why does we need a extracting tool ? Because the analysis software the users use to process data are not yet

able to read NeXus files

What were the Requirements A tool able to extract any kind of data from NeXus files and write it a

arbitrary file format (in ASCII or binary mode) Output description as simpliest as possible Efficiency

2 possibilities Extending a scripting language like Python Developing a specific tool that process template files describing the

output

Why choosing the second way ? The templates have to be as short as possible while keeping a good

readability: they must be understood even by a scientist ! Fast to execute Easy to deploy (very few dependencies, apart from NeXus and HDF5)



Introduction

CapabilitiesExtracting any kind of data

Writing one or more files (ASCII or binary, it's not a matter) from a single NeXus file

Directly writing 2D data as jpeg or bmp files (png format will be added soon)



Writing a template

In a template you describe the extraction in a “WYSIWYM”-like way (What You See Is What You Mean)

Just write that you want to get

The result is written on the console

Defining a output file

No keywords, only symbols !

Actually, it's not a language just a way to describe a output format

"Some text\n""Value = %?\n" nxs:/path/to/a/dataset

Some textValue = 1

> "out.txt"



Writing a template

Looping over a collectionThe NXentry groups

Testing values, attributes or dataset presence

Defining data blockNot directly written on

output

Defining and using variablesRecursive evaluation of

content

@( entry = [nxs:/<NXentry>/]...@)

?( nxs:/path/to/dataset = 2...?)

( binarydata # output into this bloc as 2 bytes unsigned integers 'ui2' nxs:/path/to/a/group<NXdata>/data)

% image_grp = /path/to/a/group"value: %g\n" nxs:$(image_grp)/a_dataset



Writing a template

Looping over values index is in range [0,

array_size[

Inserting sub-templateThe sub-template inherit

variables and data blocks of its parent

@( i = 0, nxs:/path/to/a/group/dataset._size_(1) "%g\n" nxs:/path/to/a/group/dataset[i]@)

@( entry = [nxs:/<NXentry>/] + 'operation.nxe'@)



Real example

A script used on the SWING beamline used to extract images data from Nexus file and output them as EDF (ESRF Data format) files.

Characteristics of the output formatThe file is made of 2 parts: a ASCII header and a binary image

data

ASCII Header length must be a power of 2

a lot of cabbalistics fields must be filled

bina ry data must be 2 bytes integer

Input NeXus files contains several image, each image in a dedicated NXdata group and named image#i with i in range [0, image_count[ (I known, a image stack would be a better way for storing data in that case)



Main usages at Soleil

During the data acquisition process a extraction is (most of the time) applied automatically for each NXentry group or for the whole NeXus file (on 7 out of 11 beamlines)

Our data portal application allows users to download NeXus files as well as on the fly extractions using nxextract on server sideTemplates depends of the beamline



Future

Adding PNG format support

Defining standard template(s) for exporting data in CIF or imgCif files ? (but no interest for Soleil up to now)

Embeding the Python engine ?Allowing to define data block filled by Python script, like this:

( integer_array 'i2' nxs:/path/to/a/dataset)py( toto import os toto = integer_array[0]py)'b' toto

25th & 26th august 2009icat developer workshop 1

Documents