xbox software development – an alternative approach...alternative approach kai papke current...
TRANSCRIPT
![Page 1: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/1.jpg)
Xbox Software development – an alternative approachKai Papke
![Page 2: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/2.jpg)
Current Workflows
● LabView
● LabView’s DIAdem
– Limited number of loaded data sets
– Very very slow● Matlab
– requires very memory intensive and time consuming conversion of files
● Python
– No conversion required, direct use of tdms files
TDMS file
![Page 3: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/3.jpg)
Motivation● Open source
– No license issues– Avoid huge software packages e.g. Matlab, LabView
● Memory efficient data evaluation– Conversion between .tdms and .mat files very intense– Prefer formats designed for large data amounts (partial load)
● Fast data evaluation– High-level languages e.g. Matlab, LabView, Python may be very slow in
certain situations → Avoid bottle-necks e.g. loops● Maintainability and adaptability
– Modular but not too modular framework (e.g. avoid all-in-one-scripts)– Generic (e.g. avoid fixed size arrays)– Clear variable scope (e.g. no global variables)
![Page 4: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/4.jpg)
So, lets try C++ and ROOT ...
![Page 5: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/5.jpg)
Why C++ and ROOT?● ROOT files
– 20 years of experience with big data
– very user definable
– Various options to read and evaluate data in single and multithreading
ROOT HDF5 PB SQL Parquet Avro
Self-describing
x x - x x x
Nested types x ? x ? x x
Columnar layout
x ? - - x -
Compres-sion
x ? x - x x
x supported, - unsupported, ? Difficult/ unclear
https://indico.cern.ch/event/567550/contributions/2628878/attachments/1511966/2358123/hep-file-formats.pdfhttps://indico.cern.ch/event/613842/contributions/2585787/attachments/1463230/2260889/pivarski-data-formats.pdfhttps://indico.cern.ch/event/613842/contributions/2590628/attachments/1463381/2261193/hep-file-formats.pdf
![Page 6: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/6.jpg)
Why C++ and ROOT?● Flexible framework
– Interactive environment as in Matlab or IPython available
– Bindings to Python opens the door to higher level scripting language
– Kernel for Jupyter
– Creation of user defined libraries and stand alone apps
Jupyter notebook
![Page 7: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/7.jpg)
Why C++ and ROOT?● Scientific libraries
– Fitting, statistics, and minimiser tool boxes widely used features of ROOT
– Various other scientific packages available (linear algebra, fft, machine learning, image processing, ...)
Math libraries and packages of root
https://root.cern.ch/root/html534/guides/users-guide/ROOTUsersGuideA4.pdf
![Page 8: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/8.jpg)
Drawbacks of using C++● Requires knowledge of data types, functions,
pointers, classes, ... ● Compiling and linking
– done by ROOT as long as within the ROOT framework
2nd point must be
as simple as possible
… but what if we want to go beyond ROOT?
![Page 9: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/9.jpg)
Framework
InteractionInterpreter like Environment,
Gui apps
BuildingAutomatized process,
facilitates to extend the project, add packages,resolve dependencies,
Support of Linux, Windows, and MacOS
I/OReading TDMS files,Writing ROOT and
HDF5 files
AnalysisFiltering,
Breakdown localization,Pulse width, gradient, ...
CoreInternal data representation
![Page 10: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/10.jpg)
Core ● Representation of
DAQ channels● Simple operations
– Interval selection– Max, Min, mean, ...– Calibration
(scaling)
Example of a file converter panel
![Page 11: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/11.jpg)
I/O● Reading of TDMS files and mapping to common
format between Xboxes● Reading and writing of ROOT files● Writing of HDF5 files (for Matlab users)
command size Location of original
Time
Copy to local folder 500 Workspace 7 sec
python load tdms file 150 Local folder 2-4 sec
root load tdms file 150 Local folder 0.6 sec
python load tdms file 150 Workspace 33 sec
root load tdms file 500 Workspace 7.5 sec
Performace for loading tdms files
Conversion of TDMS to ROOT files ~60 times faster
than conversion of
TDMS to MAT files
![Page 12: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/12.jpg)
Analyses● Signal interpolation and re-
sampling to improve accuracy
● Polynomial Noise filter to increase SNR
● Edge detection based on derivative, differences, and thresholds
● Signal alignment and localization of the divergence point
● Phase wrapping and unwrapping
![Page 13: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/13.jpg)
Analyses● Breakdown localization still in progress ...
MATLAB ROOT
downmixer not locked to masterclock (1.4.2018)
![Page 14: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/14.jpg)
Analyses● ROOT’s Draw option allows to plot filtered data directly
– E.g. power and pulse length of non-breakdown events:
● Famous new tool: RDataFrameshttps://root-forum.cern.ch/t/rdataframe-a-modern-tool-to-manipulate-and-analyze-root-datasets/29384
![Page 15: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/15.jpg)
Interaction● Web based applications
– Future of ROOT graphics– Uses XML and Java– A bit too early stage
Example of a file converter panel
https://indico.cern.ch/event/697389/contributions/3062039/attachments/1712196/2760672/webwindow.pdf
![Page 16: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/16.jpg)
Interaction● QT gui applications
– relatively easily implemented
– Potential for real time applications
– Very similar to PyQt
Example of a converter app
![Page 17: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/17.jpg)
What next?● QT Data Viewer● Extending and simplifying analyser package● Adding TrendData● Getting users● Machine learning for BD detection
![Page 18: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/18.jpg)
What next?● What about going beyond data analysis?
http://www.ni.com/white-paper/14623/it/
![Page 19: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/19.jpg)
Thank you
![Page 20: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/20.jpg)
● Misleading Dark current
reflectedamplitude
transmittedamplitude
transmittedphase
dark current
![Page 21: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/21.jpg)
reflectedamplitude
transmittedamplitude
transmittedphase
dark current
● Breakdown before pulse not detected
![Page 22: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/22.jpg)
reflectedamplitude
transmittedamplitude
transmittedphase
dark current
● Oscillations pulse disturb trigger
![Page 23: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/23.jpg)
● Loading root files without root installation
`
![Page 24: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/24.jpg)
● HDF5 Files
![Page 25: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/25.jpg)
● Platform independent building tool: cmake ● facilitates to extend the project, add packages,
resolve dependencies
├── analyser├── cmake├── CMakeLists.txt├── converter├── core│ ├── CMakeLists.txt│ ├── xboxbase│ └── xboxchannel├── io│ ├── CMakeLists.txt│ ├── hdf5│ ├── root│ └── tdms├── macros├── README.md
Macros to find installed packages (Boost, GSL, HDF5, …)
Adaptation to Windows, Linux, MacOSAutomatic ROOT Dictionary generation
Automatic library and executable generation
Analysis of Xbox data (BD localisation, Bulse length, Gradient,
...)
Conversion from TDMS to root or HDF5 files
Basic xbox data representation during reading, analysing and writing
Library for providing fundamental io features:
Reading TDMS filesReading and writing ROOT filesWriting HDF5 files (for Matlab)Example macros to be used directly
within the ROOT frame work
![Page 26: Xbox Software development – an alternative approach...alternative approach Kai Papke Current Workflows LabView LabView’s DIAdem – Limited number of loaded data sets – Very](https://reader033.vdocument.in/reader033/viewer/2022050610/5fb0fb664fe89c44210ce870/html5/thumbnails/26.jpg)
● ROOT files
– Conversion from .tdms to .root files about 60 times faster than conversion from .tdms to .mat files
– Compression rate 30 - 50 % with respect to tdms files
– Comparing apples with beans: ten to twenty times faster evaluations with respect to pure python solution
0 10 20 30 40 50 60 70 80 90 100
ROOT using ROOT files
Python using ROOT files
Python using TDMS files
Run time [%]