wp-8: data handling and visualization
DESCRIPTION
WP-8: Data Handling And Visualization. Review Meeting Report Felix Hupfeld, Andrei Hutanu, Andre Merzky, Thorsten Schütt, Brygg Ullmer Zuse-Institute-Berlin. Overview. Formal Matters Visualization Tangible Devices Remote File Access Replica Management / File Movement Metadata Service. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/1.jpg)
WP-8, ZIB
WP-8: Data Handling And Visualization
Review Meeting Report
Felix Hupfeld, Andrei Hutanu, Andre Merzky, Thorsten Schütt, Brygg Ullmer
Zuse-Institute-Berlin
![Page 2: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/2.jpg)
WP-8, ZIB
Overview
Formal Matters
Visualization
Tangible Devices
Remote File Access
Replica Management / File Movement
Metadata Service
![Page 3: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/3.jpg)
WP-8, ZIB
Formal Matters
Deliverables: most right on time, some delivered earlier
Milestone 8.2 (M24): Dynamic data access techniques are designed and implemented
Replica catalog (Location independant file access)Remote File Access (Pitfalls and GridFTP)Visualization service
Plan for the rest of this year:Further integration with other WPsImplement additional features (user management)
![Page 4: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/4.jpg)
WP-8, ZIB
Formal Matters (II)
B. Ullmer, A. Hutanu, W. Benger, H.-C. Hege. (2003). Emerging Tangible Interfaces for Facilitating Collaborative Immersive Visualization. Workshop on Collaborative Virtual Reality and Visualization, October 2003.
H.-C. Hege, A. Hutanu, R. Kähler, A. Merzky, T. Radke, E. Seidel, B. Ullmer. (2003). Progressive Retrieval and Hierarchical Visualization of Large Remote Data. Workshop on Adaptive Grid Middleware, September 2003.T. Schütt, A. Merzky, A. Hutanu, F. Schintke, Remote Partial File Access Using Compact Pattern Descriptions, IEEE CCGrid 2004.
![Page 5: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/5.jpg)
WP-8, ZIB
Visualization context
Grid = “virtualization” of resources
Broad progress toward virtualization of hardware resources (compute power, disk space, network)
Less common: virtualization of graphics hardware, software
“Virtualization” of visualization resources
Scientific visualization has demanding hardware and software requirements, unavailable to many users
Software is frequently complex and “heavy-weight”
Our approach:Automation for core visualization operations
Provision of grid-based “visualization services”
Develop simplifying mobile services & devices
![Page 6: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/6.jpg)
WP-8, ZIB
Visualization and mobile devices
Mobile devices• Access to viz services
from mobile phones, PDAs
• Development of new grid-enabled “mobile”viz devices
Visualization• Access to grid resources
from existing viz applications
• Access to existing viz applications from the grid
![Page 7: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/7.jpg)
WP-8, ZIB
Visualization service
1D
2D
3D
![Page 8: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/8.jpg)
WP-8, ZIB
Mobile + viz = “viz artifacts”
MotivationAdvantages of stereo, collaborative interactionImportance of the meeting room, teleconferencingSimplifying interaction + gaining real estate by breaking out into the physical world
![Page 9: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/9.jpg)
WP-8, ZIB
Data cards
Represent online data, user credentials, remote users, parameters, etc.
![Page 10: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/10.jpg)
WP-8, ZIB
Interaction pads
targ
auth
sens?
net
source target
source
targ
auth
sens?
net
upper left upper right
center
lower left lower right
targ
auth
sens?
net
targ
auth
sens?
net
param 1 param 2
“binding pad” “placement pad”
“parameter pad” “control pad”
![Page 11: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/11.jpg)
WP-8, ZIB
Remote File Access (I)
Needed forVisualization of remote files (intermediate results of large jobs)Job input files (extract and transmit only data of interest)…
Existing mechanismsGridFTP remote partial file access (read, seek, read, seek, .. )
I/O wrapper for local and remote access methods (to be used in GAT)
But does not fit Grid requirements, because
too many small remote operations!
![Page 12: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/12.jpg)
WP-8, ZIB
Remote File Access (II)
Need to execute high-level operations on the server side
File format specific approach(HDF5)HDF5 – structured binary file format.
Datasets, Groups, Attributes
Forwards HDF5 calls to the remote side
Plugin for the Globus GridFTP server
Generic approach (much better ) next slide
![Page 13: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/13.jpg)
WP-8, ZIB
Remote File Access (III)
Generic approach
Compact access pattern description: Pitfalls
Speedup of several orders of magnitude achieved!
Example: Access part of a Cube: (339, 65874, 262144, (0, 511, 2048, 32,
(0, 511, 2048, 32, (0, 3, 16, 32)))
just 1 (in words: one!) remote call
![Page 14: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/14.jpg)
WP-8, ZIB
Replica Management Components
Replica catalogSupports hierarchical data organization
POSIX filesystem like interface
Uses C++ with SOAP/GSI interface
Adaptor inside of GAT
File movement serviceUsed for replication
Based on GridFTP
Adaptive service (WP6)Ranks replica locations
Optimizes GridFTP parameters
![Page 15: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/15.jpg)
WP-8, ZIB
StorageBox Metadata Service
Unified storage for metadata of persistent Grid objects (archived files, job input/output, messages/notifications)
Any number of namespaces, configurable via pluggable access policies as public or private archives, group collaboration spaces or private annotations for public data
Public Sensor Data Archive
User ‘A’: Sensor Data Annotations and Private Messages
Project ‘X’: Collaboration
Space
User ‘B’: Sensor Data Annotations and Private Messages
AB
![Page 16: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/16.jpg)
WP-8, ZIB
StorageBox Metadata Service
Portlet as user interface
Integrated with WP12’s MessageBox service
SOAP web service interface
Pluggable database backendswith PostgreSQL implementation
Portable C++ implementation Win32, Linux, Mac OS X
Planned: Improved query language / processing
Remote administration interface
More access policies
![Page 17: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/17.jpg)
WP-8, ZIB
Continuing plans
Visualization, mobile devicesFurther integration with portal
Implement additional viz operations
Deploy, test both services & devices w/ end users
Data managementProvide a flexible namespace for organizing files
Remote File Access: Generic vs. File Format Dependant
![Page 18: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/18.jpg)
WP-8, ZIB
Questions?
![Page 19: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/19.jpg)
WP-8, ZIB
Visualization and mobile devices
Mobile devices• Access to viz services from mobile phones, PDAs
Completed + demoable from mobile phones, iPAQCompleted + demoable from mobile phones, iPAQ• Development of new grid-enabled “mobile” viz devices
Demoable: use of new devices to load remote content, Demoable: use of new devices to load remote content, activate and control video conference sessions activate and control video conference sessions
Visualization• Access to grid resources from existing viz applications
Completed: remote data access from AmiraCompleted: remote data access from Amira• Access to existing viz applications from the grid
Demoable: grid automation, presentation of reports Demoable: grid automation, presentation of reports integrating content from gnuplot, OpenDX, Amira;integrating content from gnuplot, OpenDX, Amira;grid access to streaming Amira contentgrid access to streaming Amira content
![Page 20: WP-8: Data Handling And Visualization](https://reader035.vdocument.in/reader035/viewer/2022062519/5681555e550346895dc327d0/html5/thumbnails/20.jpg)
WP-8, ZIB
Videoconferencing example
24.02.04, Berlin-Munich, surgical planning