tpac digital library talk overview presenter:glenn hyland tasmanian partnership for advanced...

TPAC Digital LibraryTalk Overview

Presenter: Glenn HylandTasmanian Partnership for Advanced Computing &Australian Antarctic Division

Outline:

• TPAC Overview

• Digital Library Holdings

• Recent DevelopmentsOceans & Climate Digital Library Portal

• Australian National Grid ProgramEnhancing access to and use of digital library data

TPAC Digital LibraryWho is TPAC?

• Tasmanian Partnership for Advanced ComputingState based APAC (Australian Partnership for Advanced Computing)

• Located at University of Tasmania, Hobart

• Partnership between:University of Tasmania CSIRO Atmospheric Research

CSIRO Marine Research Australian Maritime College

Australian Antarctic Division Antarctic Climate & Ecosystems CRC

Bureau of Met. Research Centre

TPAC Digital LibraryWho is TPAC?

• Expertise centre for modelling oceans & atmospheres

• 1) Computational Tools and Techniques ProgramAims to develop numerical models for high performance computers withreference to the oceans, atmosphere, Antarctic ice sheet & the environment

• 2) Education, Outreach and Training ProgramAims to establish courses in the application of high performance computermodelling and visualisation of computer models

• 3) Grid ProgramAims to create the tools and facilities that enable digital library data to bemade transparently available to users of the APAC National Grid, and tocreate compute grid applications relevant to earth system science

TPAC Digital LibraryArchitecture

• Distributed library of oceanographic and climate data

• Built on OPeNDAP framework- Open-source Project for a Network Data Access Protocol- Makes local data accessible to remote locations over the web

(uses web-based transport protocols – http)- Supports a variety of self-describing formats

(netCDF, HDF, JGOFS, Matlab, etc)- users can access data of interest (sub-sampling)

• Accessible using web browsers & common analysis applications(Matlab, Ferret, GrADS, ODC, Excel, etc). Eg.

mydata = loaddap(“http://tpac.org.au/test.nc?sfcpr[1:2:2000]”)

TPAC Digital LibraryOPeNDAP Network

APAC NF (Canberra)International IPCC model results (10-50Tb)TPAC 1/8 degree ocean simulations (7Tb)

Met Bureau Research Centre (Melbourne)Near real-time LAPS analyses products (<1Gb)Sea- and sub-surface temperature products

TPAC & ACE CRC (Hobart)NCEP2 (150Gb), WOCE3 Global (90Gb)Antarctic AWS (150Gb), Climate modelling (4Gb)Sea-ice simulations, 1980-2000

CSIRO Marine Research (Hobart)Ocean colour products & climatologies (1Tb)Satellite altimetry data (<1Gb)Sea-surface temperature product

CSIRO HPSC (Melbourne)IPCC CSIRO Mk3 model results (6Tb)

AC3 Facility (Sydney)Land surface datasets

TPAC Digital LibraryArchitecture

TPAC Digital LibraryOceans & Climate Portal

Front-end

• set of GridSphere portlets – navigational, administrative, and help

• common and easy to navigate format

• functionality for searching file meta-data

Data API

• relational database + Java class interfaces

Search Crawler

• console based Java application

• crawls over OPeNDAP web sites

• traverses hyperlinks to find OPeNDAP files

• automatically populates the database

TPAC Digital LibraryOceans & Climate Portal

TPAC Digital LibraryOceans & Climate Portal

TPAC Digital LibraryOceans & Climate Portal

TPAC Digital LibraryOceans & Climate Portal

TPAC Digital LibraryOceans & Climate Portal

Stage 1 completehttp://digitallibrary.tpac.org.au:8080/gridsphere

Stage 2 in progress …- improvements to user interface & search capabilities- provide accounting & authorisation functionality- optimise the back-end database performance- provide support for APAC grid-enabled certificates

TPAC Digital LibraryOceans & Climate Portal

• The APAC Grid program is building a national grid infrastructure integrating the APAC and partner facilities to give researchers seamless access to computational and data resources

• Also provide services to support research collaboration

• Activity aligned along two streams – infrastructure and applications

• 3 infrastructure projects will provide servicescomputing, information, user interface & visualisation

• 6 scientific areas have been chosen for applicationsastronomy, bioinformatics, chemistry, earth system science,geoscience and high-energy physics.

• TPAC activities fall within the earth systems science area

TPAC Digital LibraryAPAC National Grid

TPAC Digital LibraryTPAC Grid Program

• Data GridCreating the tools and facilities that enable digital library data to be made transparently available to users of the APAC National Grid

• Compute GridCreating compute grid applications relevant to earth system science and environment

TPAC Digital LibraryData Grid Activities

• Expanding our digital library data holdings• Globus based OPeNDAP for Grid compatibility

Development currently stalled

• Server-side analysis & visualisation toolseg. GrADS Data Servers & Live Access Servers

• Catalogues of OPeNDAP repositoriesPortal provides cataloguing and searching capabilitiesFuture versions of OPeNDAP to have inbuilt cataloguing?

• OPeNDAP-XML interoperability toolXML version of OPeNDAP in development, eventually leadingtowards access via a web service

TPAC Digital LibraryCompute Grid Activities

The Grid will support three classes of applications:

User written codesAnalysis performed on the Grid within problem solving

environmentssuch as Matlab/IDL/GrADS/Ferret

Pre-installed monolithic codesSelected Earth System models will be ported to all HP computingplatforms that will be participating in the National Grid.

Earth Systems Science analysis toolkitAn Earth Systems Science toolkit will be developed which

performsa set of standard discrete operations, such as averaging a set ofmodel output fields, flux calculations, etc.

What we have …

• Data Discovery ServiceOceans & Climate Digital Library Portal

Working on …

• Data Analysis ServicesEarth Systems Science Analysis ToolkitEarth Systems Science Analysis Workflow Portal

• Compute ServicesSelected ESS models running on APAC National Grid

TPAC Digital LibraryGrid Services Summary