TeraGrid Quarterly MeetingDec 5 - 7, 2006

Data, Visualization and Scheduling (DVS) Update

Kelly Gaither, DVS Area Director

TeraGrid Quarterly MeetingDec 5 - 7, 2006

Data Movement

The alpha draft of the data toolkit has been completed and has been initially reviewed by Lee: Needs some tweaking to make it complete and consistent with

other toolkit definitions Will be distributed to the data working group after that

Key items to focus on over the next several weeks: Formal testing plan to ensure that deployed tools are tested and

stay working (configuration and change management plan) Complete analysis of data transfer speeds

• PSC Speedpage (http://gridinfo.psc.edu/gridftp/speedpage.php) has the raw numbers for point to point transfer times. Group at PSC will be investigating and documenting bottlenecks, and what can be reasonably expected given the current infrastructure.

TeraGrid Quarterly MeetingDec 5 - 7, 2006

Data Management

Phil Andrews is leading discussions about globally available file systems going forward. Current examples of this are GPFS-WAN and Lustre-WAN

Will be looking at viability of Amazon S3 storage for TG user community

TeraGrid Quarterly MeetingDec 5 - 7, 2006

Data Workshop

January 9-11, San DiegoThe outcome will be a draft of a TG wide data

movement/data management plan going forward

If you are attending: Please email Mark Sheddon to register Please make your hotel reservations by the end

of this week

TeraGrid Quarterly MeetingDec 5 - 7, 2006

Data Collections

Data Collections RAT Report is complete Defined what constitutes a formally designated TG data

collection: Classes:

• Research data collection• Resource or community data collection• Reference data collection

Types:• Type 1: Generally Accessible Data Collections

– Satisfied basic requirements• Type 2: Compute Associated Data Collections

– Requires routine usage of TG compute or visualization resources to process data (e.g., Purdue Environmental Data Portal consolidates several earth observation data collections)

• Type 3: Globally Available Data Collections– Large data collections available on global file systems (e.g., NVO)

• Type 4: TG Affiliated Data Collections– Data collections demonstrating a link to TG, for example a

demonstrated TG user community desiring access

TeraGrid Quarterly MeetingDec 5 - 7, 2006

Data Collections

Data Collections Working Group: Charter is complete Working group is expected to begin

meeting beginning of 2007Going forward in 2007 we expect

folks to approach TG for inclusion as a TG data collection: Will be completing a process for

becoming a TG data collection before the review

TeraGrid Quarterly MeetingDec 5 - 7, 2006

Visualization

A half-day visualization tutorial “Remote/Collaborative TeraScale Visualization on the TeraGrid,” was taught at IEEE Visualization on October 29, 2006: Taught by Kelly Gaither (TACC), Mike Papka (ANL),

Joe Insley (ANL) and David Ebert (Purdue) Visualization Workshop for Users:

Coordinating a full day workshop on the Monday of TG ’07 (June 4, 2007)

Limiting enrollment to 25-30 maximum Picking ~8 power users to begin working with now

and visualize their data as examples for the workshop Will be distributing a call for participation by Jan, 2007

TeraGrid Quarterly MeetingDec 5 - 7, 2006

TeraGrid Visualization Gateway

Q4FY06 Accomplishments Presentations & demonstrations of TeraGrid Visualization beta portal at SC06 Collaborative and Remote Visualization Functionality

• Remote Paraview on UC/ANL systems• Remote & Collaborative Visualization on Maverick

Offers support for TeraGrid User Portal accounts and community accounts• Current TeraGrid User Portal users can login with their TGUP account and have

full access to Viz Gateway.• Community users can create their own accounts with restricted access.

Q1FY07 & Future Plans Continue development to production quality portal Integrate Additional Visualization Tools

• Expand current set of visualization tools and work with other RP sites to see what they can offer (e.g., Purdue has expressed interest in including a Maya portlet).

• Look into registering ‘visualization services’ with the portal, either through web services or importing additional functionality.

Milestones:• Have portal v1.0 complete and in production by TG07 Conference (and TG Viz

User’s Workshop)• Paper submission to TG 07 Conference

TeraGrid Quarterly MeetingDec 5 - 7, 2006

Remote Visualization Portlet

TeraGrid Quarterly MeetingDec 5 - 7, 2006

ParaView Portlet

TeraGrid Quarterly MeetingDec 5 - 7, 2006

Scheduling

Co-scheduling/metascheduling RAT: Leadership and Members

• Warren Smith (TACC) and Patricia Kovatch (SDSC)• Members from each RP site

Accomplishments to Date: Performed a user survey, summarized and discussed the

results Performed an RP survey, summarized and discussed the

results Performed a review of metascheduling tools, summarized and

discussed the results Drafted a set of recommendations Finishing final report (1/5/2007) Developing a scheduling working group charter

TeraGrid Quarterly MeetingDec 5 - 7, 2006

Scheduling

RAT Recommendation Topics: Advance Reservations Co-scheduling On-demand Scheduling

• Highest priority

• Preemption Automatic Resource Selection Ensemble Workflow

Four Primary Tasks: Update user metascheduling requirements Compile information about scheduling environments on RP systems as

well as RP requirements and preferences in the area of metascheduling Identify and perform a paper evaluation of metascheduling tools Develop recommendations for metascheduling in TG