Download - Poster for ISGC
Simulation and Software Technology
Deutsches Zentrumfür Luft- und Raumfahrt e.V.in der Helmholtz-Gemeinschaft
Simulations- und SoftwaretechnikKöln/ Braunschweig/ Berlin
Miriam [email protected]
Andreas [email protected]
www.dlr.de/sc/datafinder
WebDAV Server
FTP Server
Tivoli StorageManager
File System
GridFTP Server
Meta Data ServerExternal Medias
(CD, DVD,…)
Department
Employee
Simulation
Geometry
Grid Generation
Flow Solution
Visualisation
Application: Datafinder (User
Interface)For executing scripts in grids and for accessing
and managing data stored on different
backends
Use Cases:Using a grid to
calculate and execute Geometry, Flow
Simulation,… and also storing Data in Grids or
Clouds
Storage System:All possible
Backends, that can be used, including Grids and Clouds
S3 Cloud- Storage System
Storage Resource Broker
Wizards:Help to configure Use
Cases and manage the Data Access
Why develop the DataFinder?
When conducting experiments large datasets are produced. The DataFinder supports therefore:
organizing large data sets•handling describing meta •dataexecution of scripts for the •automation of common procedures, like simulations and visualizations of experiment conditions
With the DataFinder the produced datasets are easily accessible and processable.
Who could use the DataFinder?
A typical use case is archiving of data on Cloud storage. Even if the computation producing the data is performed locally or in the Grid, storing on a lightweighted storage backend is possible.The DataFinder could be especially interesting for smaller companies or institutions which do not want to maintain their own archiving infrastructure.
How are Cloud and Grid connected with the DataFinder?
The same user interface is provided for all data stores, so users have uniform access to backends for Grid and Cloud storage. This allows easy data management throughout the whole virtual storage space, for example copying and moving data between Grid and Cloud resources.
What kind of backends does the Datafinder support?
A major concept of DataFinder is to store data on different backends:
WebDAV server•FTP und GridFTP server•localfilesystem•Tivoli Storage Manager •(TSM)forhigherflexibility:•Amazon Simple Storage Service (S3)
The user has the option to either store his data on a classical server, on Grid resources, or in the Amazon Cloud.
DataFinder - A Data Management Application for Grids and Clouds
��������
Example Use CaseD-Grid Project: „AeroGrid“
Technicalandscientificdata•management on distributed resources (Grid and Cloud)Documentation of CFD simulations•Assure quality of results and trace •errorsRepeatability of simulations•UserFrontEnd„Datafinder“(Python/•PyQT)SimulationCode: Cluster optimized •CFDcode„Trace“(C++)Jobs with Globus and UNICORE•
