www.gridlab.org

Grid Portals: Bridging the gap between scientists and the Grid

Michael Russell, Jason Novotny, Gabrielle AllenMax-Planck-Institute fuer GravitationphysikGolm, Deutschland

www.gridlab.orgThe promises of Grid computing are grand

Uniform access to heterogenous resources The ability to pool distributed resources together

on demand Resources are either transparently available to

users or they simply don’t have to worry about them

Support virtual organizations of distributed researchers collaborationg across institutoional, geographic, and political boundaries…

Support applications with enormous computing and/or data management requirements.

www.gridlab.org

Grid computing is quickly evolving

In the 2 years I’ve been here Globus 1.3 went 2.0

Grid Portals became “the next big thing” Luckily, portlets have come to the rescue Meanwhile, Globus went pro to bring in IBM

and other heavyweights Now OGSA is stepping up to bat Global Grid Forum is already a hit Can’t wait for the video games!

www.gridlab.org

So where is the Grid?

How can we use it? Or at least that’s so-called users are probably asking.

The Grid is a work in progress, one of the biggest undertakings in the history of humankind. Most of you are in Scotland to do your part in building the Grid.

And most of you have reasons why you need the Grid, you want to use it right? Or perhaps you want to help others to use it.

The point is, a large gap exists between the Grid and its would-be users.

www.gridlab.org

Grid Portals

So you want to build a Grid portal to bridge that gap. But what does it take to build a Grid portal? Well, it’s going to take a lot…

It takes a solid understanding of the state-of-the-art in Web portal development.

It takes a solid understanding of the state-of-the-art of in Grid computing.

And much more…

www.gridlab.orgThe Astrophysics Simulation Collaboratory

The ASC seeks to: Enable astrophysicists to develop better

and more powerful simulations with the Cactus Computational Toolkit.

Enable astrophysicists to run and analyze simulations on Grids.

Build a Grid portal to support these activities and all the rest that comes with Web portals.

www.gridlab.org

The ASC and Cactus

So we worked with the Cactus Project to develop support for working with Cactus from the ASC Portal.

We developed application Web components to: Install Cactus software from multiple cvs repositories onto

target resources. Build executables with those installations using the autoconf

and make capabilities in Cactus. Upload and edit parameter files. Run simulations on target resources, as well as for connecting

to simulations and monitoring their progress. Launching the appropriate visualization applications on

Cactus data. In most cases, the Cactus Project developed extensions to

Cactus software to support these components.

www.gridlab.org

The ASC and Globus

We worked with the Globus Project to develop support for working with Globus from the ASC Portal.

We developed Grid Web components to: Enable logging on with one or more Grid proxy certificates

stored in MyProxy. Submit and monitor jobs with Globus Gatekeeper and for

maintaing job history. Manipulate files (listing, copying, moving, deleting files) with

GSI-FTP. We tried to use MDS, but at that time MDS did not meet our

needs, so we developed our own components for storing static information about resources and polling for whatever dynamic information could be reliably retrieved from MDS.

We asked Globus to build GSI-CVS and now we’re building Web components and services to use and extend GSI-CVS.

We’ve added support for GSI-authentication in the MindTerm Java SSH implementation.

www.gridlab.org

The ASC and ASC

We realized we needed to support the people within ASC directly from our portal, so we built administrative Web components to: Manage user accounts. Assign security roles to users and to which

pages users have access. Manage the proxy certificates users require to

authenticate to other services and determine which certificates authenticate to which services.

Manage the resources and services to which we provide access from the ASC Portal.

www.gridlab.org

Problems we faced

The ASC is a Virtual Organization in every sense of the word with users, developers, administrators, software, and resources distributed across the U.S. and Europe.

With our developer(s) in the University of Chicago and our users in Washington University, St. Louis and Max Planck Institute in Golm, Germany, there wasn’t nearly enough direct contact between our users and developers. This made it difficult to meet their needs.

It was easy enough to prototype Web components, but we needed to build a Grid portal framework that would support future development and sustain a production-quality Grid portal, and that took several months to develop.

We were closely associated with Globus, but this made us too reliant upon Globus as our interface to resources…

For while we had identified the resources our users required, the Grid as we knew it then and today just wasn’t enough.

www.gridlab.org

You need to build a virtual organization

or otherwisejoin a virtual organization and plug into their work!

It takes more than just a portal

www.gridlab.org

Other lessons we learned

Put the needs of scientists at the very top of your list. Be familiar with their research and the day-to-day problems they face in using computing technology to conduct that research.

Next, consider what it is you really need in the form of: Enhancements to the applications scientists are

developing. For new applications, consider how you can build-in support for Grid operations. For legacy applications, consider how you can provide better support for their applications with external services.

Enhancements to the Grid infrastructure with respect to your applications. These enhancements should build off other…

High-level services that coordinate the use of resources. We’re beginning to see Globus as the “system” layer with respect to Grids.

www.gridlab.org

Don’t forget your infrastructure

Create a vialable testbed that includes both the resources your users need and use in their everyday reserch as well as resources with which you can experiment.

In a VO like the ASC, you will not have administrative control over these resources. In fact, this is a very difficult problem to overcome, it takes a lot of effort to see changes applied where and when you need them.

You need tools for testing things out and you’re going to need to keep track of all your resources and providers, the change requests you make, the problems your users experience, and so forth…

www.gridlab.org

Get into production mode

In order to build a viable Grid portal, realize you need to build a production system, something your users can rely upon to work every time. For instance, what happens to your production database when your software and data model change?

What makes a production system? Solid engineering practices and attention to user requirements, security issues, persistence management, quality assurance, release management, performance issues…

Project management at the VO level is complex, make sure you understand Grid-level management issues before you start writing those cool Grid portal proposals!

www.gridlab.org

Build a team

It’s important to communicate with partners in the Global Grid Forum, but try to cover all the bases within your project. If you don’t have the funds to build a large team, then allocate funds towards developing explicit links between yours and other projects.

Because you need application experts, Grid portal experts, Grid service experts, Grid testbed administrators, and so on… division of labor is a cornerstone of project management.

www.gridlab.org

GridLab

Well, so we’re taking the lessons we learned in the ASC and elsewhere (GPDK, for example!) through the collective experience of everyone involved in GridLab and we’re applying them towards building a…

Production Grid across Europe (and the U.S.)

Grid Application Toolkit for developing applications with built-in support for Grid operations, like migrateApplication()

Grid Portal to support GridLab

www.gridlab.org

Key points

We’re going to work as closely as possible with scientists and their applications throughout the project.

We’ve created a testbed of resources that our scientists really want to use.

We’re working to build higher-level services to better coordinate the use of those resources, and the requirements for these services will come either directly from our application groups or indirectly through using a Grid portal.

We’re developing application frameworks to enable scientists to make use of these higher-level services as basic function/method calls within the applications.

And we’re coordinating these activities through the development and use of the GridLab Portal.

www.gridlab.org

The GridLab Portal

We’re using the ASC Portal software to allow us to focus on the needs of scientists from the very beginning of our project development.

We’re simultaneously building a new framework that takes the best of current practices in Web and Grid computing, and we’re documenting just about every bit of its requirements, design, and development.

As we develop this framework, we’ll be preparing the Web interfaces we develop with the ASC Portal for migration to the new framework.

www.gridlab.org

Bridging the gap

Before you begin your Grid portal efforts, look at what’s already out there. For example: GPDK - JSP/Servlets, Great way for getting an

introduction to Grid Portal development GridPort - Perl-CGI, well-managed project and they have

real application groups using NPACI resources. Bear in mind that Portlets are where it’s all

heading, but there is no Portlet API quite yet. A lot of people looking at building Grid portals with the JetSpeed codebase (but not everyone!).

Or, you should consider working with us…

www.gridlab.org

A possible collaboration with GridLab

GridLabPortal

GridLabResource Manager

GridLabInformation

Service

GridLabApplication

Manager

Grid CVS

Grid Make

ComputeResources

Run my application!

GridLabApplication

Toolkits

•Cactus•Triana

Help us design better Grid technologies. Plug in your own application and supporting services.Then develop your own Web pages with our Grid portal…

www.gridlab.org

Some online references

Astrophysics Simulation Collaboratory: http://www.ascportal.org

Cactus Project: http://www.cactuscode.org Globus Project: http://ww.globus.org GridLab: http://www.gridlab.org GridPort: http://www.gridport.org Grid Portal Development Toolkit:

http://www.doesciencegrid.org/Projects/GPDK Jakarta JetSpeed: jakarta.apache.org/jetspeed OGSA: http://www.globus.org/ogsa Portlet Specification: http://www.jcp.org/jsr/detail/168.jsp