standards: saving the world? · hello, my name is amy low energy community meeting, daq working...
TRANSCRIPT
STANDARDS:
SAVING THE WORLD?
AMY ROBERTS,
BRETT GEDVILAS
HELLO, MY NAME IS AMY
Low Energy Community Meeting, DAQ Working Group,
Michigan State University, 9/10/2018AMY ROBERTS 2
Faculty @ CU Denver
Member of Cryogenic Dark Matter Search Collaboration
Our detectors measure phonons!
Focus on low-energy detector response, DAQ
Dark Matter @ University of South Dakota
Ultra cold neutrons @ LANL
Started out working on cryo for nEDM experiment
Ended up focusing on ultra-cold neutron simulation
Neutrons @ Notre Dame Nuclear Lab
(3He,n)74,76Ge
Neutron time-of-flight
Got to make a nice muon veto
Low Energy Community Meeting, DAQ Working Group, Michigan State University, 9/10/2018
AMY ROBERTS 3
DATA DESCRIPTION STANDARDS
• We all need to read and
write data
• Writing this code is
tiresome and error-prone
• Such code (often) is not
citable or open-source
Can we build something
that handles custom-
binary-format data
reading and writing for
(nuclear) scientists?
DATA DESCRIPTION STANDARDS
Low Energy Community Meeting, DAQ Working Group,
Michigan State University, 9/10/2018AMY ROBERTS 4
One read-write library to rule them all?
Option 1
Common format (HDF5, ROOT, JSON)
My collaboration uses the MIDAS DAQ, which
outputs a custom format. Moving to a different
format is non-negotiable.
Option 1I
Data description language (Kaitai, DFDL)
24 bytes of ‘FFFF’ starts the record
The next two bytes indicate the size of the record
The next byte indicates the type of record
If the type of record is “scal” then the next four bytes …
DATA FORMAT DESCRIPTION LANGUAGE & TOOLS
Low Energy Community Meeting, DAQ Working Group,
Michigan State University, 9/10/2018AMY ROBERTS 5
• Description languages and tools already exist!
• None of them quite do what I need …
Kaitai
• Produces read code
in multiple languages
DFDL
• Reads and writes
data, Java
Construct
• Reads and writes
data, python
DATA FORMAT DESCRIPTION LANGUAGE & TOOLS
Low Energy Community Meeting, DAQ Working Group,
Michigan State University, 9/10/2018AMY ROBERTS 6
Kaitai
• kaitai.io
• CDMS example:
https://github.com/det-
lab/dataReaderWriter
• User specifies data format,
kaitai produces data-reading
c++/python/ etc. library
• No writing
• Human readable!
• Flexible enough for CDMS
• Nice data-viewing tools
• Ruby viewer too slow
for ~MB data sets
DATA FORMAT DESCRIPTION LANGUAGE & TOOLS
Low Energy Community Meeting, DAQ Working Group,
Michigan State University, 9/10/2018AMY ROBERTS 7
DFDL
• Open-source
implementation an Apache
Incubator project,
https://daffodil.apache.org/
• IBM implementation also
• OpenGrid specification
(really, really long)
• Java, Scala
• Custom-format data => XML
• XML => custom-format data
• Kinda human readable
DATA FORMAT DESCRIPTION LANGUAGE & TOOLS
Low Energy Community Meeting, DAQ Working Group,
Michigan State University, 9/10/2018AMY ROBERTS 8
Construct
• https://construct.readthedo
cs.io/en/latest/
• Will output a kaitai
definition file!
• Have hit snags writing code
for CDMS data
• User has to directly write
parsing code
Not what I want, but you might
find it useful.
DATA DESCRIPTION STANDARDS: NEXT STEPS
We’re actively looking for collaborators
Want some help with your data? We’d love to collaborate!
Want to join us? Get in touch! [email protected]
Low Energy Community Meeting, DAQ Working Group,
Michigan State University, 9/10/2018AMY ROBERTS 9
Strongest as a collaboration
that affects multiple disciplines
Approaching this with public
data
DATA DESCRIPTION STANDARDS: NEXT STEPS
We’re actively looking for collaborators
Want some help with your data? We’d love to collaborate!
Want to join us? Get in touch! [email protected]
Projects we’re considering
Note: we really like the Katai description language
Add write capability to Kaitai
Improve the Ruby Katai-based data viewer so it works with GB files
Try describing CDMS data with DFDL
Low Energy Community Meeting, DAQ Working Group,
Michigan State University, 9/10/2018AMY ROBERTS 10