remi database: a data management system for nuclear ...remi facts about remi developed for the...

REMI Database: A Data Management System forNuclear Medicine Studies

Antall Johann Fernandes

Florida Institute of Technology

[email protected]

30th November, 2011

Antall Johann Fernandes (Florida Tech.) REMI Database 30th November, 2011 1 / 43

REMI

Facts about REMI

Developed for the Nuclear Imaging Group of the Life SciencesDivision at Lawrence Berkeley National Laboratory, California.

Supported by a subcontract on NIH grant 5-R01-EB007219-04,”Molecular Imaging of Cardiac Hypertrophy Using microPET andPinhole SPECT”.

Currently is in production and houses 180 experiments, and 8105 datafiles.


Motivation

Typical Work Process by the Nuclear Imaging Group

Perform a SPECT, DT-MRI or Micro-PET experiment.

Copy experimental data to individual researcher’s workstation.

Perform data analysis, image reconstruction and various otherprocessing tasks on experimental data.

Extract results from processed data.

Publish results and findings based on results.

What happens to the experiment raw data or processed data?

Data resides on individual researcher’s workstation under a researchergenerated directory structure.

Data is moved or copied into a central file server into a researchergenerated location and directory structure.

Data about the experimental data may or may not be captured.


Motivation

Knowledge about Data

What experimental data has been collected?

Where does all of the experimental data reside?

Data Sharing

Can we share the experimental data with others?


Motivation

The Need for a System

Aware of all the experimental data captured over the years.

Knowledge of the location of the experimental data.

Provide the capabilities to share the experimental data.

Sharing Experimental Data

Data should be stored in a standardized manner.

Data should be query-able.

Meta-data needs to be captured in a timely manner. (Meta-data isexplained later).

Users should not need to deal with file systems.


Design Considerations

Typical Scientific Data Management Systems should have...

Creation of logical collections - physical data to logical collections.

Physical data handling - storage of the physical data files.

Security support - data access authorization and change verification.

Data ownership - who is responsible for data quality and meaning.

Persistence - data lifetime.

Knowledge and information discovery - identify useful informationinside the data collection.

Meta-data collection, management and access.


Design Considerations

REMI’s Design Considerations

Open access to downloading of experimental data.

Users should be able to access the database from anywhere.

Authorized researchers should be able to upload experimental data.

REMI should handle the physical storage of data files (explainedlater).


Understanding Experiments and its Data

SPECT Experiment

Data and Meta-Data

Patient Coordinate System

Detector Coordinate System


Meta-data

What is Meta-data?

Information about data.

Describes how data is measured, acquired, and computed.

Enables data browsing, data transfer, and data documentation

Makes it possible to build data independent automated tools.


Entities and Relationships

Entity

A ”thing” or ”object” in the real world that is distinguishable from allother objects. Eg. Person or Vehicle.

An entity has a set of properties and values that identify an entity.

Relationship

An association among several entities. Eg. A Person owns a Vehicle.


Entities and Relationships

Entity-Relationship Data Model

Based on the perception of the real world that consists of a set of basicobjects called entities, and of relationships among these objects.

Mapping Cardinalities

Represents constraints on the relationship.

Expresses the number of entities to which another entity can beassociated via a relationship.


REMI Meta-data Database Development

Logical Entity Relationship Diagram



Physical Table Design Diagram



Physical Table Design Diagram: Machine



Physical Table Design Diagram: Tracer



Physical Table Design Diagram: File Tag



Physical Table Design Diagram: File Association

Owner ID maps to the entity which owns the File.



Physical Table Design Diagram: User


REMI File Storage Development

REMI’s Design Consideration (mentioned earlier)

REMI should handle the physical storage of data files.

File Storage Schemes

(c) Single Directory (d) Directory Hierarchy

All Files under a single directory.

Individual directory for each experiment.



REMI File Storage

Concatenate experiment ID and filename and generate a 64 lengthSHA256 string.

Create directory structure based on SHA256 string, directory levels,and directory name size

Parameter directory name size determines how many characters touse of the hash value

Parameter directory levels determines the depth of the directorystructure

Example 1

SHA256 string: fb80f1735b3153e7a41d38390de5d9773c35259965..directory levels : 2directory name size : 2=>ROOT DIR/fb/80/<experiment id> <filename>.<extension>



REMI File Storage: Example 2

SHA256 string: fb80f1735b3153e7a41d38390de5d9773c35259965..directory levels : 2directory name size : 3=>ROOT DIR/fb8/0f1/<experiment id> <filename>.<extension>

REMI File Storage: Example 3

SHA256 string: fb80f1735b3153e7a41d38390de5d9773c35259965..directory levels : 3directory name size : 2=>ROOT DIR/fb/80/f1/<experiment id> <filename>.<extension>



File Storage

directory levels : 2directory name size : 2


REMI Application Development

Rails Model-View-Controller (MVC) Architecture



System Design



System Component Design



Reasons to upload Data Files in Parts

Web Browsers can upload a maximum of 4 GB of data per request.

Certain data files are in excess of 50 GB in size.



File Upload Process



File Upload Process: Save the chunk files



File Upload Process: Concatenate the chunk files


REMI User Interface Development

JSON (JavaScript Object Notation)

A lightweight data-interchange format.

Easy for humans to read and write.

Built on two structures.

A collection of name/value pairs.Realized as an object, record, struct, dictionary, hash table, keyed list,or associative array.An ordered list of values.Realized as an array, list, or sequence.



JSON Syntax

(e) Object Syntax

(f) Array Syntax

(g) Value Syntax



REMI Menu

id: Translates to the element ID on the HTML page.

value: Display name shown on the user interface.

link: URL to be called.

submenu: Array to contain sub-menus.Antall Johann Fernandes (Florida Tech.) REMI Database 30th November, 2011 33 / 43


Download Menu Options



Searching for Experiment based on Modality



Creating a New SPECT Experiment



Various Sections within the SPECT Experiment


Weaknesses

Weaknesses within REMI

Lack of full support for Semi-Structured Data.

Semi-Structured Data does not conform with the formal structure oftables and data models associated with relational databases.Meta-data is all semi-structured data.

Resume functionality for uploading and downloading data files.

Server side file compression on large files.

Meta-data Templates and User Personalization.


Future Enhancements

Provide support for Semi-Structured Data.

Relational databases with XML support.

Move from a relational model to Entity-Attribute-Value model.

Document based databases.

User Personalization.

Provide user specific views of owned experiments.

Support saving of personalized search queries.

Meta-data

Best approach to modeling meta-data.

Make the meta-data capture process less cumbersome.

Reading meta-data from header files.


Future Enhancements

Resume functionality for Data File Uploads.

Files uploaded in chunks.

Query the server for existing file size on the server.

Transfer file chunks greater than the size on the server.

Resume functionality for Data File Downloads.

Look at HTTP Byte Range Retrieval Extensions.

See how the resume functionality affects the compression process onthe server.


Questions


Demonstration

REMI Website


http://host75-177.harris-cc.fit.edu

Thank You


remi database: a data management system for nuclear ...remi facts about remi developed for the...

Documents