a pparc funded project deploying and configuring a registry

A PPARC funded project

Deploying and Configuring a Registry

http://wiki.astrogrid.org/bin/view/Astrogrid/AgTechWorkshopJan06_Registry

Registry – Kevin Benson, Jeff Lusted 2

OverviewAbout the Registry

Overview Various Resource types

Exercises: Installation Setting properties through tomcat Self Registration Querying Entering Resource(s) Removing Resource(s) *Interacting directly at the database Harvesting Using the workbench *Extending the Registry

XFormsOther

(* = A brief pause right “before” that task)


About the Registry{Brief talk on the registry}. The yellow pages of the VO.Identifiers: The primary key of a xml resource.

Ex: ivo:// + {authorityid} + (/{resource key); ivo://org.workshop/registry

Definitions: Publishing Registry – A Registry that holds xml resources

managed by your authority id’s. And is harvestable by Full Registries. Does not require a Query interface, but typically has one.

Full Registry - A Registry that holds all xml Resources. Has a Query and a Harvest Interface.

Ref: http://www.ivoa.net/twiki/bin/view/IVOA/IvoaResReg


Managed Authoritieshttp://localhost:8080/astrogrid-registry/browse.jspNow click on “XML” link for your vg:Registry type entry.Look at the bottom.<vg:managedAuthority>org.session.workshop</vg:managedAuthority>


Resource TypesRef: http://www.ivoa.net/twiki/bin/view/IVOA/VOResourceV010

Go to: http://localhost:8080/astrogrid-registry/resourceHelp.jsp Various Types:

Resource Service

SkyService->TabularSkyService ConeSearch SIA (Simple Image Access) OpenSkyNode

Registry CeaService

DataCollection TabularDB

Authority CeaApplication


Installing RegistryDownload the latest beta war file for this session at:http://hydra.star.le.ac.uk:8080/misc_astrogrid/downloads/registry_session.war And place it into your tomcat/webapps directory. Tomcat will automatically deploy the war file. And you can now go to http://localhost:8080/registry_session , even though it is not setup yet.In your Tomcat Administration GUI change properties by hitting save after each property change and then hit “commit” at the end.

See the next slide on information about properties. reg.amend.authorityid - Your main Authority ID for this

registry, normally an institution domain name e.g.: mssl.ucl.ac.uk. For this exercise change it to “org.session.workshop”.


Properties

All the registry properties are defined in JNDI or more rarely changed properties are in the webapps/{context}/WEB-INF/astrogrid.properties in this sample {context} is registry_session.Also Tomcat has an administration GUI for editing these properties.For the purpose of this workshop everything can be changed in the Administration GUI.Go to your http://localhost:8080Click on “Tomcat Administration”And login. (Go to the next slide)


Properties cont.Now go to the “Environment Entries” of your registry. You should see what is shown in the screen shot below, then go to the next slide.


Installing Registry cont.

Note: Feel free to look at other properties and change them. The property names with “amend” are ones you may typically change and “custom” are rarely changed. See the “Registry Reference” docs for ideas of how the “custom” ones can be used. Some of the custom properties are so rare to be used that they have been moved out of JNDI (environment entries) to WEB-INF/classes/astrogrid.propertiesNow go to your http://localhost:8080/registry_session click on the “Admin” link in the menu login and on the first page click on “Self Registration”.


Installing Registry cont.Fill out the form for Self Registration, be sure to use the same authority id that you used in setting the properties which is “org.session.workshop”, for the other boxes just fill in some “test” text strings. Then hit submit.url: http://localhost:8080/registry/admin/selfRegisterForm.jsp

You are now at a text box that should have your completed registry type. If you want you may look at the last XML element “managedAuthority” and add more to manage other authority ID’s, or you can come back and edit this Registry type later.You are now all setup and ready to add XML Registry Resources to your new registry.


ExercisesYour first objective is to familiarize yourselves with various pages of a registry. It might be helpful to interact and walk through each page.The next series of slides demonstrate the navigation menus. We will walk through each slide, but feel free to go ahead.You should be able to go through the pages at http://localhost:8080/registry_sessionThe credentials for the Admin pages login are as follows:

Username: workshop Password: qwerty


Access Menu - Takes the user to the index page - Setup and Administration

-Registry Documentation - Resource Lookup - Browse Resources - ADQL & SQL queries - Search by keywords - Typical Help doc for Resource elements - - Register or Update new resource - Back Door to Register or Update - {not used} XForms, being reimplemented - Harvesting registries manually - Removing a Resource - Registry and System config and properties - Setup and Administration - External Registry tests.


Exercises I – JSP’s

Having familiarised yourselves with the navigation options of the registry, let us explore the Registry functionality.The following set of slides demonstrate

Querying Entering Resources Removing Resources of a Registry.

We will walk through each exercise, but feel free to skip ahead.


Loading some SamplesGo to your registry webapp:http://localhost:8080/registry_session Were going to upload several xml Resources

in case you wish to view them. Go to Enter Resource in your menu. Enter in the URL box:

http://wiki.astrogrid.org/pub/Astrogrid/AgTechWorkshopJan06_Registry/workshop_samples.xml

Click “Browse” on your menu to verify more resources are in your registry.


BrowseClick on Browse this is the most commonly used page for querying. (Below is an example of the galahad registry.)


Browse cont.

Type swire into the “IVORNs Including” text box and hit “List”. Now you have a reduced list.


Browse cont.Staying on the browse page hit “XML” link on the line for org.session.workshop/org.astrogrid.registry.RegistryService. Now you should see the full xml for the small registry service.


Entering ResourceClick the “Back” button on your browser and then click “Edit” next to the org.session.workshop/org.astrogrid.community. You will see the text in the last text area, change the contact name to yours and your contact e-mail. Feel free to make other textual changes as well. XML gets validated - if you tick the “validate” checkbox - after the submission. When ready hit “Submit”.Note: The blue text at the top is simply stating that Validation is on, but will still try to insert the Resource in an invalid document, you will see later how to set this up in the properties. Ticking the “Validate” checkbox will cause the document to be sent to the server only after it ensuring its validity. We will see what happens to an invalid document later in this tutorial.The following slide gives a walkthrough of entering a resource.


Entering Resource Screenshot


Entering Resource cont.

Click the “Back” button on your browser button.In the XML text remove “<title>…</title>” line. Check mark the validate checkbox and hit “Submit”. Notice that it is not sent because title is a required XML element.

document failed to schema validate

cvc-complex-type.2.4.a: Invalid content was found

starting with element 'identifier'. One of

'{"http://www.ivoa.net/xml/VOResource/v0.10":title}‘

is expected.


Entering a Non-Managed Resource

To verify you cannot edit non-managed resources. Go to your browse screen and try to change the identifier to have a different authority id. Example ivo://will.fail/SWIRE/object-catalogue/TDB Try to “Submit”.Notice it does not allow to submit (in fact you will get a warning message plus an exception) because you do not manage this Authority ID: it is managed and owned at a different registry in this case the unknown ghost of a registry.


Removing a ResourceClick on “Remove Resource”Try to remove:Example entry to be placed. ivo://org.session.workshop/SWIRE/object-catalogue/TDB


A Pause

At your own pace, look at the other Querying type pages such as: IVORN Lookup Query (which allows ADQL or SQL queries to the registry) Keyword Query

Try the different query mechanism, aiming to understand how they works. Interesting enough there is a XQuery interface which will see later with the workbench, but never made the jsp page.Look at the samples just loaded. Will take a brief pause. Let people catch up or other questions that might come from querying or looking at the xml resources.Also try editing your Registry Type resource to have more managedAuthorities then you can try changing the identifier of one of your other XML resources to your new managed authority to see if it is “added”.


Exercises II – The Database -eXisteXist XML database can be found with all the documentation at http://exist.sourceforge.net. Lets peak inside it a bit.Go to a command prompt, then go to where your Tomcat is installed, and then in the webapps/registry_session/WEB-INF directory. Now run this the command below ( you can add a –s for shell mode if you have no GUI, but prefer GUI if you can):

Run: client.{bat/sh} –ouri=xmldb:exist://localhost:8080/registry_session/xmlrpc

Linux/Mac users need to set a variable: EXIST_HOME=. Linux/Mac Seems that you can run this:

EXIST_HOME=. client.{bat/sh} -ouri=xmldb:exist://localhost:8080/registry_session/xmlrpc

Username=admin password= (leave password blank) hence just hit “OK” for the login box.Now you are inside the database. If you’re in shell mode type “help” and you will see the list of commands, and you can “cd” and “ls” just like a normal shell. If your in the GUI then experiment: you have a wide range of options, try various things just don’t “delete”/”remove” collections. Note: eXist uses the term “Resources” for its documents and collections/tables so don’t confuse it with the actual XML registry resources.


eXist cont.Take a few minutes to get acquainted with eXist.http://exist.sourceforge.net/client.html has various other client information.


WebDav (new)When you try the below abilities in an application. It will ask for a user and password from the eXist database.username: admin password:See: http://wiki.exist-db.org/space/WebDAV and see if you can put your database into your file manager or one of your applications. If not you may access the db via webdav through http-get http://localhost:8080/registry_session/webdav/db (but this is not as nice, but a good test).It may ask for usernames and passwords. This is your eXist xmldb user and password. Use username=admin password= (blank)


HarvestingGo to Enter Resources and in the url box put this url to a xml file which contains registries for us to harvest. http://wiki.astrogrid.org/pub/Astrogrid/AgTechWorkshopJan06_Registry/registries_workshop.xml

Go to your Harvest page in the menu. The following slide is a screen shot of the Harvest page, click on “harvest” from the menu. Click on “jvo” (japan)Click Harvest.Now in your browser you should see your jvo entries.You may wish to come back and harvest other registries if you desire. And you can harvest by date. WARNING: CDS is huge 11,000 entries don’t harvest till after workshop .


AstroGrid HarvestingHarvest by date

Click the radio button of the Registry you

want to harvest.

Calendar


A PauseStart your workbench. Keyword search at RegistryBrowser->RunClick on Astrogrid on the left side of the workbench. There you will see a Registry link to do various queries. Even has the XQuery setup whereby the jsp one has not been created.Sample XQuery:

for $x in //vor:Resource where vr:identifier &= ’*works*’ return $x/vr:title


Properties and Moving eXist

Open up a file manager or through command prompt create a directory where eXist data will go on your machine. Hence create a directory on your system.

Copy the conf.xml, and the whole “lib” and “data” directory from your Tomcat installation {context name}/WEB-INF directory. So inside your new directory you should now have conf.xml, data {directory} and lib {directory}.

Using the Tomcat Administration GUI, change the "reg.custom.exist.configuration" property and point to this directory plus the conf.xml hence /workshop/exist/conf.xml Or optionally you can try to change the xml file via tomcat/conf/Catalina/localhost/registry_session.xml

Be sure to hit "save" then hit "commit changes". Now when the data is being stored it is using this new data

directory: be sure during upgrades to set that property to your directory. Note: "Windows" OS users may need to escape their slashes, e.g. c:\\existstore\\conf.xml.


A PauseFor final exercise go back to your tomcat/webapps/registry_session/WEB-INF directory see if you can run backupandrestore.{bat/sh} look at the top of these files for instructions you need to set a EXIST_HOME variable to the directory of where your new conf.xml lives hence /workshop/exist or c:\workshop\exist (for windows no double slashes here or conf.xml in the path). See if you can backup and restore.


Extending the RegistryResource schemas doesn’t carry everything, sometimes you need to extend. Lets say you need to extend it for a particular purpose. For example the new vospace standard.You can already enter extended resources into the registry and query (with known namespaces). But you can’t query on your particular extension namespace.You should make a new schema with your namespace and extension. But were not going to bother on this exercise, but would be normally needed for validation of the xml.Add this XML tag <workshop:vospace xmlns:workshop=“http://www.regsession”>VOSpace</workshop:vospace>to one of your resources such as inside the <content> tag.Go to WEB-INF/classes/astrogrid.properties change and add to the end of “reg.custom.declareNS.0.10”:declare namespace workshop = http://www.workshop.sample/vospaceFinal note, no need to do in this exercise (and sometimes never) but in WEB-INF/classes/xsl there is a file called ExistRegistryResult0.1.xsl in this file schemalocations are placed into the result of the responses of the registry. You may add yours to the top layer VOResources to ensure schemaLocations are placed on your responses or if you know xsl see the bottom where it does other schemaLocation additions.


XFormsIt is assumed that you are familiar with XML or have an external tool such as XMLSpy or Oxygen to help make the XML conform to the Registry schemas. This can be a daunting task at times because of the complexity of schemas.XForms makes things easier, since you can fill out a form (much like an html form) and it will validate and conform to the schemas and send the data as XML to your server.At the time of this writing XForms in the Registry webapp is more experimental and the implementation of it is getting changed. The current implementation is probably usable, but slower and considerably more buggy. See your “astrogrid-registry” XForms if you wish to have a look, the registry_session at this time of writing is being changed. Things will change to “FormsFaces” implemenation soon. By following the installation exercise of accessing the latest war file, you will find now an “XForms” link in the menu (for creating brand new xml resources) and also an “Xedit” link on your main query pages such as Browse.


Xforms cont.

More information on Xforms Read the first couple of chapters of:

http://xformsinstitute.com/essentials/ Read the first couple of chapters of

http://www.w3.org/TR/2003/REC-xforms-20031014/ or finally

http://www.w3.org/MarkUp/Forms/There are no exercises to this since it is basically filling out an HTML type form similar you have done in the past.


Other Discussions LaterRegistry not only works with eXist embedded but you may download eXist and run it externally as a database. Only property to be changed is xmldb.uri. See docs and http://exist.sourceforge.orgSince the registry uses the xmldb api it has the ability to connect to other xml databases. But this has not been tested or attempted. Chat with the instructor if you have the desire to use a different xmldb such as X-Hive, tamino, xindice.Since eXist has a rpc ability you could potentially access the registry db via rpc.


Suggestions?Welcome any suggestions?From the Past Big Astrogrid logos and “astrogrid” text in

jsp’s change one of the logos to be custom. Add the XQuerySearch jsp. Finish off XForms to be used on other

components. Better error messages.

Report invalid xml.


Whats Next?Security on the Update web service.Support Interfaces Heartbeat Logging

New 1.0 Resource schemaInterface Spec with paging.

a pparc funded project deploying and configuring a registry

Documents

registry properties

registry cont

publishing registry

custom properties

registry type entry

registry reference docs

properties cont

installationsetting