Gfdnavi:its design and implementation

with Ajax and Ruby-on-Rails

Seiya NISHIZAWA,Takeshi HORINOUCHI,Chiemi WATANABE,

T. KOSHIRO, A. TOMOBAYASHI, S. OTSUKA,Y. MORIKAWA, Y.-Y. HAYASHI, M. SHIOTANI, and

GFD-Dennou Davis project

Introduction

• What’s “Gfdnavi”– A tool to archive, share, distribute, analyze, and

visualize geophysical fluid data and knowledge– desktop use to data provide server– fundamental technologies

• Ruby on Rails• GPhys

An introduction was done by T. Horinouchi yesterday.

Ruby on Rails

• an open source web application framework• written in Ruby• Model-View-Controller (MVC) architecture• Convention over Configuration (CoC)• Don't repeat yourself (DRY)

• swift development– ActiveRecord– helper methods (HTML, JavaScript, ajax)

ActiveRecord (AR)• a part of Rails products• a ruby implementation of the object-relational

mapping pattern

• Do not need to use SQL– For better performance, SQL can be used on AR.

node = Node.find(:first, :conditions=>[”id=?”,2]) # SELECT * FROM nodes WHERE id=2 LIMIT 1;path = node.path #=> “/samples”kas = node.keyword_attributes # SELECT * FROM keyword_attributes WHERE node_id=2;

id path parent

1 / null

2 /samples 1

id node_id keyword value

1 2 description sample directory

2 2 notice just sumple

nodes table keyword_attribute table

Metadata DB

used for search

1. name-value attributes2. geospatial- and time-coordinate information3. owner, groups and access mode4. link among data5. time-stamp, size, etc

1. Name-Value attributes• attributes in data file (self-describing files)

– unified access to attributes in differently formatted files with GPhys

• attributes in text file– YAML format– any name-value attributes

gphys_nc = GPhys::IO.open(“fname.nc”,”T”) # NetCDFgphys_nc.att_names #=> [“long_name”, …]gphys_nc.get_att(“standard_name”) #=> “air_temperature”

gphys_grib = GPhys::IO.open(“fname.grib”, “TMP”) # GRIBgphys_grib .att_names #=> [“long_name”, …]gphys_grib.get_att(“standard_name”) #=> ”air_temperatrue”

description: NCEP/NCAR reanalysisgfdnavi: owner: user1 other_mode: 0 rgroups: - groupA - groupB

YAML

• a human-readable data serialization format– easier to read/write than XML

puts “Array (list)”ary = [0,1,2]puts ary.to_yaml

puts “\nHash (associative array)”hash = {“key0”=>”value0”, ”key1”=>”value1”}puts hash.to_yaml

Array (list)---- 0- 1- 2

Hash (associative array)---key1: value0key0: value1

Back

2. Geospatial- and time-coordinate information– spatial region

• rectangle in longitude-latitude section

– temporal region• start time and end time

global, regional, or point swath

3. Owner, Groups, and Access mode– permission system like i-node

• readable and writable for groups and others

– Multiple groups are allowed.

4. Link among data– e.g. This data was calculated from these variables

Directory tree structure• nodes in the tree structure

– node types: directories, variables, images, knowledges, etc

• Each node can have some metadata.– inherited to children nodes

local or opendap directoriesdata files

variables

image files

attributesdescription = “……..”owner = userAgroups = [groupA,groupB]

description = “……..”param1 = value1param2 = [value21,value22]

attributesgroups

virtual aggregated files

fork a child process

Analysis model

open data file(s)

calculate draw

NetCDF PNG

analysis

draw

storage

GPhysDB

cached get cache

output output

cache

yes

no

programmable

Analysis and Visualization

• Analysis model (Analysis class)– all the parameters for analysis or visualization

• the form in the analysis page• instance variables of the Analysis object

It is able to construct one from the other• enable to reconstruct the analysis page from

• drawn image• history list

• Draw method and analysis function are not hard-coded.– Their definitions are in YAML files (editable)

• one method in one file

:name: spectrum:description: |FFT|^2 along a specific dimension:nvars: 1:script: | [gphys0.fft(*arg0).abs ** 2]

:arguments:- :description: the dimensions for spectrum :value_type: array_string :default: []

spectrum.yml

simple coding due to GPhys

can create and modifyvia web-browser

• examples of original draw methods in a Gfdnavi server providing an ensemble forecast data

User Interface

• bottleneck of network application– network bandwidth– machine power and system load of the server

• better usability– ajax

• Rails has many helper methods to write HTML and JavaScript to use ajax.

– cache

• Animation

Web service• local programming• cross-site use

– other Gfdnavi servers– non-Gfdnavi servers

SOAPAPIs for all the analysis functions and draw methods

use the Analysis classWSDL

(REST)

Summary• Metadata and Directory tree structure

– attributes in self-describing data files and YAML files– inheritanceunified access to attributes with GPhyseasy and swift development with AcriveRecord (Rails)

• Analysis/Visualization– programmable (with text editor or web-browser)easy and extensible coding with GPhys

• User Interface– good usability with ajax and cacheeasy development with helper methods (Rails)

Thank you

MVC architecture• Model

– Relational Database• ActiveRecord: a ruby object that wraps a row in a

database table

• View– HTML

• ERuby: a templating system that embeds Ruby into a text document

– JavaScript• ajax

• Controller

Web server

Web server

G

Controller

Model

ActiveRecord, VizShot

View

ERubyhelper methods

User

Web Browser

RDB

create, update, deleteanalysis, draw

request

HTMLJavaScript

Gfdnavi server

GPhysDCL

ActiveRecord (AR)• a part of Rails products• a ruby implementation of the object-relational

mapping pattern– a row in a table ↔ an instance object– a column ↔ an instance method– a reference between tables ↔ an instance method

part = Part.newpart.name = “Sample part”part.price = 123.45part.save

INSERT INTO parts (name, price) \ values (‘Sample part’, 123.45);

name = “gearbox”part = Part.find(:first, :conditions=>[“name=?”,name])

SELECT * FROM parts \ WHERE name=‘gearbox’ LIMIT 1;

RDB tables

• Directory tree structure– nodes (basic unit of a tree)

• directories, variables, images, knowledges, etc.

• Meta data– keyword_attributes– spatial_and_temporal_attributes– node_relations

• User’s information– users– groups

• instance method is used for reference between two tables with foreign key

class Cart < AcriveRecord::Base has_many :itemsend

class Item < AcriveRecord::Base belongs_to :cartend

id cart_id name amount

1 1 orange 3

2 1 apple 2

id user

1 userA

cart = Cart.find_by_user(“userA”) #=> cart whose # user = “userA”items = cart.items #=> array of # items whose cart_id = 1items[0].name #=> “orenge”items[1].amount #=> 2

carts items

Directory tree structure• Tree structure is good way to organize data

/…/data_top_dir/ncep.yml /ncep/UV.jan.nc (contains U and V) /UV_500hPa_vect.png /UV.jan.nc.SIGEN /UV_500hPa_vect.png.yml

id path name parent node_type

1 / / null directory

2 /ncep ncep 1 directory

3 /ncep/UV.jan.nc UV.jan.nc 2 directory

4 /ncep/UV.jan.nc/U U 3 variable

5 /ncep/UV.jan.nc/V V 3 variable

6 /ncep/UV_500hPa_vect.png UV_500hPa_vect.png 3 image

• acts_as_tree method of AR– “parent” column is assumed to have parent’s “id”– “parent” method return the parent node– “children” method return the array of children nodes

class Node < AcriveRecord::Base acts_as_treeend

node = Node.find(3) #=> node whose id=3node.children #=> array of nodes # whose parent column is 3node.parent #=> node whose id is value of parent column

• two types of meta data1. meta data in data file (self-describing file)

• NetCDF: variable attributes

variables: float U(level,lat,lon) ; U:long_name = “zonal velocity“ ; U:units = “m/s” ; U:missing_value = -999.f ; U:standard_name = “eastward_wind” ; U:statistic = “Long Term Mean\n” ; float V(level,lat,lon) ; V:long_name = “meridional velocity” ; V:units = “m/s” ; V:missing_value = -999.f ; V:standard_name = “northward wind” ; V:statistic = “Long Term Mean\n” ; float lon(lon) ; : snip

2. meta data in YAML and SIGEN files• ncep.yml

• UV_500hPa_vect.png.yml

description: vector plot of horizontal velocity at 500hPagfdnavi: owner: user2 references: - /ncep/UV.jan.nc/U - /ncep/UV.jan.nc/V other_mode: 0 rgroups: - groupA

description: NCEP/NCAR reanalysisgfdnavi: owner: user1 other_mode: 0 rgroups: - groupA - groupB

• keyword_attributes tableid node_id keyword num_value value

1 2 description NCEP/NCAR reanalysis

2 3 description horizontal velocity in January

3 4 long_name zonal velocity

4 4 units m/s

5 4 missing_value -999.0

6 4 standard_name

eastward wind

7 4 statistic Long Term Mean\n

8 5 long_name meridional velocity

9 5 units m/s

10 5 missing_value -999.0

11 5 standard_name

northward wind

12 5 statistic Long Term Mean\n

13 6 description vector plot of horizontal velocity at 500hPa

• spatial_and_temporal_attributes table

• node_relations table– UV_500hPa_vect.png (node id = 6) was drawn

from U (node id=4) and V (node id = 5)

id reference referenced_by title

1 4 6 drawn

2 5 6 drawn

id node_id lon_lb lon_rt lat_lb lat_rt starttime endtime

1 4 0 360 -90 90 2008-01-01-00:00 2008-01-31-18:00

2 5 0 360 -90 90 2008-01-01-00:00 2008-01-31-18:00

• user and groups– root, user1(groupA,groupB), user2(groupA), user3

• access permission– owner

• root: “/”, “/ncep”• user1: “/ncep/UV.jan.nc”, “/ncep/UV.jan.nc/(U|V)”• user2: “/ncep/UV_500hPa_vect.png”

– permission for reading• everyone: “/”• groupA and groupB: “/ncep“, “/ncep/UV.jan.nc/(U|V)”• groupA: “/ncep/UV_500hPa_vect.png”

• users and groups tables

• nodes table

id name owner

1 groupA user1

2 groupB user2

id name groups super_user

1 root 00000000 true

2 user1 00000011 false

3 user2 00000001 false

4 user3 00000000 false

id path owner other_mode rgroups wgroups

1 / 1 4 null null

2 /ncep 1 0 00000011 null

3 /ncep/UV.jan.nc 2 4 null null

4 /ncep/UV.jan.nc/U 2 4 null null

5 /ncep/UV.jan.nc/V 2 4 null null

6 /ncep/UV_500hPa_vect.png 3 0 00000001 null

The first bit is the flag for the group of id=1 (groupA),the second one is for that of id=2 (groupB),and so on.

The first bit is the flag for the group of id=1 (groupA),the second one is for that of id=2 (groupB),and so on.

They seem to be permitted to be accessed by anyone,but it is not so, because the permission for “/ncep” is inherited.They seem to be permitted to be accessed by anyone,but it is not so, because the permission for “/ncep” is inherited.

2: writable, 4: readable, 6:both2: writable, 4: readable, 6:both

• Permission of a node depends on all its ancestors.– calculated in advance– saved at additional Boolean-type columns– In order to keep consistency, callbacks can be

added to events (e.g. after saving) with AR.

• When permission is changed, the columns of its children are recalculated.– Using ActiveRecord, adding such callbacks to events

(e.g. after saving) is quite easy.

class Node < ActiveRecord::Base

after_save calculate_children_permissions # add a callback method

def calculate_permissions # calculation of permission other_readable = (other_mode & 4) & parent.other_readable groups_readable = rgroups & parent.groups_readable save! # save to database table end

def calculate_children_permissions # callback method children.each{|child| child.calculate_permissions } endend

ER diagram of RDB tables

(only important tables and columns are shown)

Registering script

• Directory tree structure on a storage is registered to the database table as it is.– The top directory to be registered can be

specified.

• Metadata contained in data files are registered to the database table.

• Any other metadata can be registered by writing in Yaml and SIGEN files.

Analysis and Visualization• GPhys

– IO handling– data slice– mathematical calculation

• DCL (Dennou-Club Library)– plot

• (Cache)– drawn plots are cached to DB

• Directory tree view

click

ajax request to get children directories

The ajax request will not be called again.Opening and closing the directory is doneby changing “display” attribute of CSS.

RESTful

• Representational state transfer• Uniform interface

– All resource share a uniform interface like HTTP methods such as GET, POST, PUT, and DELETE.

• Resource Oriented Architecture– Every resources are uniquely addressable using URI.

• Statelessness– Each request is treated independently.

improvable– still some separate codes

• web application, web service

– SOAP ignore HTTP methods.• Some cache systems could not work well.

– Programming with SOAP tend to be complicated.

“RESTful web service + client(s)”– one unified code

• Multiple output formats are still supported (HTML, XML).

– acts as definition of HTTP methods• cache system friendly

– simpler local programming

RESTful webservice• URI syntax

– variable• http://host/…/ncep/UV.jan.nc/U.(html|xml)• http://host…/ncep/UV.jan.nc/children.(html|xml) # U and

V

– analysis• http://host/ .../{variables}/analysis/{function}.(html|xml)?{options}

– drawing• http://host/ .../{variables}/draw/{draw method}.(html|xml)?{options}

clients• flex

– ActionScript– (AIR)

• (HTML + JavaScript)• (desktop ruby client)• another gfdnavi server

UI with ajax

• asynchronous JavaScript and XML (originally)– not only JavaScript and XML, but also other pairs

• update a part of the existing page– better usability– reduce bandwidth usage

• Rails has many helper methods to write HTML and JavaScript to use ajax.

click

get node list of the directory vie ajax call

• Analysis page

check

get dimension’s information with ajax

click

send valuesin the formwith ajax

• Animation– the 1st loop

• ajax requests (one request for one image)• may take a little long time

– the later loops• cached image (cached as JavaScript object)