Simulation and Software Technology

Deutsches Zentrumfür Luft- und Raumfahrt e.V.in der Helmholtz-Gemeinschaft

Simulations- und SoftwaretechnikKöln/ Braunschweig/ Berlin

Miriam NeyMiriam.Ney@dlr.de

Andreas SchreiberAndreas.Schreiber@dlr.de

www.dlr.de/sc/datafinder

WebDAV Server

FTP Server

Tivoli StorageManager

File System

GridFTP Server

Meta Data ServerExternal Medias

(CD, DVD,…)

Department

Employee

Simulation

Geometry

Grid Generation

Flow Solution

Visualisation

Application: Datafinder (User

Interface)For executing scripts in grids and for accessing

and managing data stored on different

backends

Use Cases:Using a grid to

calculate and execute Geometry, Flow

Simulation,… and also storing Data in Grids or

Clouds

Storage System:All possible

Backends, that can be used, including Grids and Clouds

S3 Cloud- Storage System

Storage Resource Broker

Wizards:Help to configure Use

Cases and manage the Data Access

Why develop the DataFinder?

When conducting experiments large datasets are produced. The DataFinder supports therefore:

organizing large data sets•handling describing meta •dataexecution of scripts for the •automation of common procedures, like simulations and visualizations of experiment conditions

With the DataFinder the produced datasets are easily accessible and processable.

Who could use the DataFinder?

A typical use case is archiving of data on Cloud storage. Even if the computation producing the data is performed locally or in the Grid, storing on a lightweighted storage backend is possible.The DataFinder could be especially interesting for smaller companies or institutions which do not want to maintain their own archiving infrastructure.

How are Cloud and Grid connected with the DataFinder?

The same user interface is provided for all data stores, so users have uniform access to backends for Grid and Cloud storage. This allows easy data management throughout the whole virtual storage space, for example copying and moving data between Grid and Cloud resources.

What kind of backends does the Datafinder support?

A major concept of DataFinder is to store data on different backends:

WebDAV server•FTP und GridFTP server•localfilesystem•Tivoli Storage Manager •(TSM)forhigherflexibility:•Amazon Simple Storage Service (S3)

The user has the option to either store his data on a classical server, on Grid resources, or in the Amazon Cloud.

DataFinder - A Data Management Application for Grids and Clouds

��������

Example Use CaseD-Grid Project: „AeroGrid“

Technicalandscientificdata•management on distributed resources (Grid and Cloud)Documentation of CFD simulations•Assure quality of results and trace •errorsRepeatability of simulations•UserFrontEnd„Datafinder“(Python/•PyQT)SimulationCode: Cluster optimized •CFDcode„Trace“(C++)Jobs with Globus and UNICORE•