dap servers and services section 2 apac ‘07 opendap workshop 12 oct 2007 james gallagher thanks to...
TRANSCRIPT
![Page 1: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/1.jpg)
DAP Servers and Services
Section 2
APAC ‘07 OPeNDAP Workshop
12 Oct 2007
James GallagherThanks to Jennifer Adams, John Caron,
Roberto De Almeida, Nathan Potter and Patrick West
![Page 2: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/2.jpg)
Outline
• Different DAP Servers
• Hyrax Architecture and customization
• Example Configuration of Hyrax
![Page 3: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/3.jpg)
There are several different DAP servers
• Hyrax, developed by OPeNDAP• TDS, developed by Unidata• GDS, developed by COLA• PyDAP, developed by Roberto De Almeida• Others we won’t cover: Dapper; FDS;
CODAR; OPeNDAP Server3; et cetera• Terminology: We often say ‘server’ when
describing what the technical press calls a ‘web application.’
![Page 4: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/4.jpg)
Hyrax
• Supports multiple protocols– Data: DAP using HTTP/GET and HTTP/SOAP;
Direct access (via HTTP); WCS/WFS funded, in development
– Catalog: THREDDS; HTML directories
• Data formats: In binary distribution: NetCDF; HDF4; HDF5; FreeForm; many more available as source code.
• Includes ASCII data dump, HTML data access form, Info metadata page
![Page 5: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/5.jpg)
Hyrax Architecture
• Two (or more) cooperating processes:– Front-end provides DAP interface– Back-end(s) read(s) data
• Both parts can be customized– Front-end: different network protocols– Back-end: different data formats/systems
• N-Tier design is flexible, secure
OLFS BES Data
![Page 6: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/6.jpg)
TDS
• Java Servlet network interface• Supports multiple protocols
– Data: DAP; WCS; NetCDF Subset; Direct access (via HTTP)
– Catalog: THREDDS
• Data formats: NetCDF; HDF5; GRIB-1,2; NEXRAD; DORADE; BUFR; DMSP; GINI; more in development
• Can also read from any other DAP server• Can serve aggregations
![Page 7: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/7.jpg)
THREDDS Data Server HTTP Tomcat Server
Datasets
catalog.xml
motherlode.ucar.edu
THREDDS Server
NetCDF-Javalibrary
Application
IDD Data
•HTTPServer
•NetcdfSubset
•WCS
•OPeNDAP
configCatalog.xml
![Page 8: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/8.jpg)
PyDAP
• Python native implementation from the DAP2 specification; WSGI application
• Server uses run-time loaded modules• Protocols:
– Data: DAP (HTTP/GET); WMS/KML; EditGrid/Google Spreadsheets; JSON
– Catalog: THREDDS using a script/batch system; full support planned
• Formats: CSV; netCDF; SQL Matlab; GRIB; HDF5
![Page 9: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/9.jpg)
PyDAP Server Architecture:Plugins and Responses
![Page 10: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/10.jpg)
GDS
• Java Servlet network interface and C/C++ back-end• Supports multiple protocols:
– Data: DAP (HTTP/GET)– Catalogs: THREDDS
• Formats: GRIB; HDF4; netCDF; BUFR; Binary; GrADS Station
• Can also read from many DAP servers• GDS can perform sophisticated server-side
operations and build interim data sets as a result.• Server-side operations use GrADS command syntax.
![Page 11: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/11.jpg)
GRIB, HDFNetCDF
Binary
datasets in any format supported by GrADS
Result cache
holds temporary data (uploaded, generated by a previous operation, or transferred directly from another server) for use in remote analysis
GrADSbatch mode
interface code
DODS server libraries
Serverperforms analysis
operations
manages sessions, translates dataset
names
Java servlet
supports extended request types for analysis, upload
GrADS Data Server
![Page 12: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/12.jpg)
DAP Servers Summary• Four servers, all developed by different
groups, were described (Hyrax, TDS, PyDAP and GDS)
• Each supports DAP and THREDDS• Each has unique features:
– PyDAP: WMS, KML and python scripting– GDS: Extensive server-side operations– TDS: Aggregation, WCS– Hyrax: Customizable, SOAP interface, WCS/WFS
support funded
• These are more properly called ‘web applications.’
![Page 13: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/13.jpg)
Hyrax Architecture
• Front-end (OLFS) customization
• Authentication & Authorization
• Back-end (BES) customization
• Single/Multiple Machine Installations
• Security
![Page 14: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/14.jpg)
OLFS
Java Servlet Engine
BES
Unix Daemon
BES Commands
XML- encapsulated object
File system with data files,SQL Database, …
DAP2
THREDDS
HTML
Optional THREDDScatalogs
Hyrax Architecture
![Page 15: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/15.jpg)
• The OLFS uses an ordered list of Dispatch Handlers.
• Each handler on the list is asked if it can handle the incoming request.
• The first handler on the list that claims the request gets to handle it.
• List order is important, as some handlers (for example THREDDS) may claim to handle requests that should be handled by a different handler.
• Handlers are free to do anything they need to handle a request: Contact a remote system/process, read files, spawn threads, et cetera.
OLFS Dispatch Handlers
![Page 16: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/16.jpg)
Extending the OLFS
• Extension ‘modules’ written in Java• Added to a directory within Tomcat• The new modules have complete access to
the request information• Both HTTP GET and POST requests• There is some significant processing done
before the handler is called:– Conditional GET Requests– Authorization & Authentication
![Page 17: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/17.jpg)
OLFS and Authorization/Authentication
• Authorization & Authentication (Az/Ac) actually provided by Apache or Tomcat
• Apache/Tomcat provides a range of Az/Ac features:– Realms: Role-based Az/Ac
• Usernames & passwords -> Authorization• Roles -> Authentication• Realm info in RDBs, XML docs, LDAP, …
– TLS/SSL:• Server and client Az/Ac• Secure data transmission
• Tomcat provides single sign-on capabilities – Clients must support cookies– Suitable for portals
![Page 18: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/18.jpg)
BES
Network Protocol andProcess start/stopactivities
Data Store Interfaces
BES Framework
PPT*Initialization/Termination
DAP2Access
NetCDF3 HDF4 FreeForm…
DataCatalogs
Commands**BES Commands/ XML Documents
*PPT: point to point protocol**Some commands are built in, most load at run-time
![Page 19: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/19.jpg)
BES Extensibility
In what ways can you extend the BES?
• New request handlers (data handlers like netcdf, hdf4)
• New response handlers (Cedar's Flat, Tab, Info responses)
• New commands (like our hello world example)
• Initialization/Termination
• Exception Handlers
• Reporters
• Container and Definition storage
• Aggregation engines
• Methods of returning your data (return as netcdf)
![Page 20: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/20.jpg)
Possible Installations
• Both OLFS and BES run on one machine
• OLFS on one machine and BES on another
• One OLFS and several instances of BES on different machines
• OLFS communicating with one or more BESs and other backend processors
![Page 21: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/21.jpg)
Installation Security
• A separate issue from Ac/Az• The BES must be protected:
– With a firewall or– TLS & Client certificates
• Running the OLFS and BES on separate machines limits the scope of a compromise of the OLFS
• Ensure that the BES, Tomcat and Apache all run with limited access to the server host
![Page 22: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/22.jpg)
Hyrax Architecture Summary
• Hyrax can be installed on one machine or several
• Installation security merits serious consideration
• Authentication & Authorization are handled by the web servers (Tomcat and Apache)
![Page 23: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/23.jpg)
Hands on: Hyrax Configuration
• Choices:– Single or Multiple machine– Single or Multiple back-end servers– Tomcat or Apache web server– Data formats– Catalog customization– Security
• Testing– Command line tools for system administrators– Web browser
![Page 24: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/24.jpg)
APAC Workshop Configuration
• Run a single BES and the OLFS on one host (the virtual machine running SLAX Linux)
• Use Tomcat running on port 8080
• Data formats: NetCDF and FreeForm
• No custom THREDDS catalog
• No firewall
![Page 25: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/25.jpg)
APAC Hardware
• SLAX Linux VMware Virtual Appliance
• All the software needed has been built, installed and configured
• Start the virtual machine now…
![Page 26: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/26.jpg)
Starting the Virtual Machine
• Insert the CD-ROM• Drag the opendap_vm and Wintools folders to the Windows
desktop• Copy the ISO image from the CD ROM to the harddisk (in a
command window, mkdir C:\SLAX and then use copy or dd to copy the iso to the new directory).– The dd command is in the Wintools folder– The C:\SLAX directory is coded into the virtual machine; use Vmware
Workstation/Fusion to edit
• Open the opendap_vm folder and double click on the opendap_vm.vmx virtual machine configuration file or start VMware Player/Workstation/Fusion and open opendap_vm.vmx from there.
• The virtual host will boot and the SLAX (slackware Linux with KDE) desktop will appear.
![Page 27: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/27.jpg)
Installing the Software
• Already present on the virtual machine
• But, if it wasn’t, you’d go to the OPeNDAP web site* and download: – Binaries or source for the BES and the
data handlers needed. – Get the OLFS web archive file (which is a
compiled java servlet).– You also need the Tomcat servlet engine
* http://opendap.org/download/hyrax.html
![Page 28: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/28.jpg)
Location of Server Files on the Virtual Machine
• On your virtual machine:– The Tomcat servlet engine is in /usr/local/javadev/apache/tomcat-5.5.12 (this is the value of $CATALINA_HOME)
– The BES, data handlers and related source files are in /usr/local/src. The BES has been built and installed in /usr/local/ ($prefix)
– The OLFS web archive file is $CATALINA_HOME/webapps/opendap.war
– BES: bes.conf, found at $prefix/etc/bes/bes.conf– OLFS: olfs.xml and catalog.xml, found at $CATALINA_HOME/content/opendap
![Page 29: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/29.jpg)
Background: Starting the Server
• Start the BES (back-end data processing component)
• Use bescmdln to verify it’s working• Start Tomcat: This automatically starts
all installed servlets– Servlets are installed by copying the .war
file to the servlet’s webapps directory
• Verify it’s working using a web browser
![Page 30: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/30.jpg)
Start the BES
![Page 31: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/31.jpg)
Verify the BES is running
![Page 32: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/32.jpg)
Start Tomcat & the OLFS
• Typical steps:– Unpack the olfs jar-file– Copy the opendap.war file to Tomcat’s
webapps directory– Start Tomcat
• Since all but the last step has been done already, start Tomcat:
– /usr/local/javadev/apache-tomcat-5.5.12/bin/startup.sh
![Page 33: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/33.jpg)
…terminal view
![Page 34: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/34.jpg)
Verify Tomcat is running
http://localhost:8080
![Page 35: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/35.jpg)
…and Hyrax
http://localhost:8080/opendap
![Page 36: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/36.jpg)
Complete the Configuration
• Steps you would typically perform:– Security: Set up a firewall! Limit access to
port 10002 to this host only – Custom catalogs: Edit the catalog.xml
configuration file. By default automatic catalogs are generated
– Logging: Edit the log4j.xml file. By default all accesses are logged
![Page 37: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/37.jpg)
Stopping Hyrax
First, stop Tomcat using ‘shutdown.sh’
…then stop the BES using ‘besctl’
![Page 38: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/38.jpg)
Hyrax Configuration Summary
• For our chosen configuration, we used only Tomcat, plus the Hyrax web application which consists of the BES and the OLFS
• The tools bescmdln, getdap and a web browser were used to test the installation
• There are a lot of options, but the default settings produce a working server
• Security is a must for a web application; use a firewall to isolate the BES so only the local host can connect to it.
![Page 39: DAP Servers and Services Section 2 APAC ‘07 OPeNDAP Workshop 12 Oct 2007 James Gallagher Thanks to Jennifer Adams, John Caron, Roberto De Almeida, Nathan](https://reader036.vdocument.in/reader036/viewer/2022062308/56649f4d5503460f94c6dbfa/html5/thumbnails/39.jpg)
Summary
• There are a number of DAP-compliant servers (Hyrax, TDS, PyDAP and GDS) were described
• All of the servers (read ‘web applications’) actually support several other protocols
• While DAP does nothing to support cataloging data, THREDDS supports just that
• Hyrax is customizable and can be installed in several different configurations