© geodise project, university of southampton, 2001-2005. geodise & geodiselab simon cox...
TRANSCRIPT
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Geodise & GeodiseLAB
Simon Cox
University of Southampton
21st April 2005
Grid Enabled Optimisation and Design Search for Engineering
(GEODISE)Southampton, Oxford and ManchesterSimon Cox- Grid/ W3C Technologies
and High Performance ComputingGlobal Grid Forum Apps Working Group
Andy Keane- Director of Rolls Royce/ BAE Systems University Technology Partnership in Design Search and Optimisation
Mike Giles- Director of Rolls Royce University Technology Centre for Computational Fluid Dynamics
Carole Goble- Ontologies and DARPA Agent Markup Language (DAML) / Ontology Inference Language (OIL)
Nigel Shadbolt- Director of Advanced Knowledge Technologies (AKT) IRC
BAE SYSTEMS- Engineering
Rolls-Royce- Engineering
Fluent- Computational Fluid Dynamics
Microsoft- Software/ Web Services
Intel- Hardware
Compusys- Systems Integration
Epistemics- Knowledge Technologies
Condor- Grid Middleware
© Geodise Project, University of Southampton, 2001-2005http://www.geodise.org/
The GEODISE Team ...
• Richard Boardman• Sergio Campobasso• Liming Chen• Mike Chrystall• Trevor Cooper-Chadwick• Simon Cox• Mihai Duta• Clive Emberey• Hakki Eres• Matt Fairman• Mike Giles• Carole Goble• Ian Hartney• Tracey Hunt• Zhuoan Jiao
• Andy Keane• Marc Molinari• Graeme Pound• Colin Puleston• Nicola Reader• Angus Roberts• Mark Scott• Nigel Shadbolt• Wenbin Song• Paul Smart• Barry Tao• Lisa Telfer-Brunton• Jasmin Wason• Fenglian Xu• Gang “Luke” Xue
© Geodise Project, University of Southampton, 2001-2005http://www.geodise.org/
Distributed Systems 2005
Network
IP
HTTP
(HTML)
Compute/ Data
Moore’s Law
Future ServicesFlexible, Seamless, and Secure Sharing
Software
(HTML) XML
Web Services
ASCII
© Geodise Project, University of Southampton, 2001-2005http://www.geodise.org/
Negative Scarf InletConventional Inlet
0 1 2
Applications - GeodiseCFD 3D engine nacelle optimisation
Goal - reduce ground noise generated by fan when plane takes off.
Optimise aerodynamic performance when scarf angle is varied.
Problem definition
Design of Experiment
Response surface modelling
Optimisation on Response surface
Validation
ProEngineer CAD(Condor Pool)
Gambit Meshing(Globus Compute)
Fluent CFD(Globus Compute)
Workflow for aerodynamic shape optimisation using CAD, Gambit, and Fluent
© Geodise Project, University of Southampton, 2001- 2004.http://www.geodise.org/
Applications – GEM
Electromagnetic optimisation
• Large number of designs, parameters and solutions
• Query for a particular data range to post-process
Transmission of light properties through a photonic crystal
Parameters are radius of holes and light wavelength
APPLICATION SERVICE
PROVIDERCOMPUTATION
GEODISE PORTAL
OPTIMISATION
Engineer
Parallel machinesClusters
Internet Resource ProvidersPay-per-use
Optimisation archive
Intelligent Application Manager
Intelligent Resource Provider
Licenses and code
Session database
Design archive
OPTIONSSystem
Knowledge repository
Traceability
Visualization
Globus, Condor, OGSA
Ontology for Engineering,
Computation, &Optimisation and Design Search
CAD SystemCADDSIDEASProE
CATIA, ICAD
AnalysisCFDFEMCEM
ReliabilitySecurity
QoS
Geodise will provide grid-based seamless access to an intelligent knowledge repository, a state-of-the-art collection of optimisation and search tools,
industrial strength analysis codes, and distributed computing & data resources
GEODISE
© Geodise Project, University of Southampton, 2001-2005http://www.geodise.org/
Grid-Enabled Scripting Environment
• Why use scripting languages?– Flexibility – High-level functionality– Quick application development– Extend the user’s existing PSE– … Acts as execution/ enactment engine too
• Approach– Matlab initially chosen as the hosting environment– Now working with Python/ Jython
MatlabPython/ & /Jython
Hardware
Management
Grid
Enterprise
Applications
Collaboration
Coordination
Hardware
Management
Grid
Enterprise
Applications
Collaboration
Coordination
Engineering Applications of the Grid via Geodise
Problem Solving Environment
Bringing underlying infrastructure into the Problem
Solving Environment
“Useful into useable”
Grid
Hardware
Matlab, Python, etc
“Heartbeat”Enterprise
“Dial-tone”
Geodise Toolkits
Geodise deliverables
• Grid Compute toolbox Cycle stealing High-performance computing Integration with heterogeneous environments
(Windows, Linux, Solaris, …)• Grid Data toolbox
Raw files into journaling filesystem Metadata in relational database (Oracle, DB2, SQL
Server, mySQL…) XML processing capability
• Intelligent Workflow composition, execution and monitoring tool Rule and case-based composition of ‘best’
workflow (semantics and knowledge driven) Deployment to compute/ data resources System-level “heartbeat” User/ Application level “dial-tone” (security,
resource availability, network weather, …)
Problem Solving Environments e.g. Matlab,
iSight Python, ModelCenter,…
Grid-enabled Toolboxes for Engineers
Middleware (Grid)
Hardware
Opti
mis
ati
on
`Tools
Designed around web-services to inherit and leverage
Technologies (e.g. security, reliability) and Development tools
© Geodise Project, University of Southampton, 2001-2005http://www.geodise.org/
Geodise Tools
• A number of Grid-enabled design tools have been developed– Compute Toolbox: certificate based authentication, job
submission, and file transfer, etc.– Database Toolbox: archive files and data, custom metadata,
locate and retrieve data, etc.– XML Toolbox: convert Matlab structures to and from XML– Workflow Editor: graphical problem solving environment which
allows users to construct and configure workflows for the Grid– OptionsMatlab: Matlab interface to the Options design
exploration system– Knowledge Toolbox: Intelligent Design advice
Hardware
Management
Grid
Enterprise
Applications
Collaboration
Coordination
Hardware
Management
Grid
Enterprise
Applications
Collaboration
Coordination
Geodise Database
Globus Server
GraphicalWorkflow Tool
Matlab Scripting Environment
Compute ToolboxDatabase Toolbox
OptionsMatlab
Compute ToolboxAPI
Database ToolboxAPI
Knowledge ToolboxAPI
KnowledgeRepository Metadata
andStructure
Repository
Computation
Globus Server #1
Globus Server #N
Condor Cluster
Globus Server /Condor Node
Condor Master
Condor Node #1
Engineer
Knowledge Toolbox
Condor Node #N
Applications
Pro/Engineer
ICAD
Catia
IDEAS
CAD
Gambit/Fluent
Abaqus
Catia
HYDRA
Analysis
Globus Servers
Glob
us and
Con
dor
Job M
ana
gers
Java Matlab
MetadataWebservice
Physical files,scripts, and
outputs
Results by GridFTP
ResultsResults
LocalFilesystem
Matlab Script
Grid JobSubmission
Results
Metadata by webservice
Geodise Architecture
XML Toolbox
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Sponsors
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
GeodiseLab Activities for the OMII
• Harden & deliver Geodise toolboxes– Geodise Compute Toolbox
– Geodise Database Toolbox
– XML Toolbox for Matlab
• Documentation• Testing • Extend existing functionality• Train and support users• ... GGF: Simple API for Grid Applications (SAGA)
Working Group
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Compute Toolbox: Aims
• Support access to whichever compute resources users wish to exploit– Globus GT2– OMII services– Condor
• Provide access from alternative environments– Matlab– Jython– cPython
• Leverage existing APIs wherever possible
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Compute Toolbox: Hit-list
Matlab
Geodise Java API
Jython cPython
CondorNative
?
.py files .py files
WindowsHPC/Grid
CondorWeb Service
GlobusGT2
OMIIServices
Java Client Java CoG OMII API.NET Client
.m files
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
GeodiseLab Compute Toolbox
Available Now
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Compute Toolbox Release
Matlab
GlobusGT2
OMIIServices
Geodise Java API
Jython cPython
CondorNative
CondorWeb Service
Java CoG OMII APIJava Client
?
.py files .py files.m files
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Toolbox Architecture
Matlab Environment
Globus GT2 Resource GRAM
ServiceGridFTPService
gd_jobsubmit.m
Java CoG
gd_getfile.m
script.m
GridFTP file transfer
GRAMjob submission
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Certificate Management Functions
gd_certinfo Returns information about the user's certificate
gd_createproxy Creates a Globus proxy certificate
gd_proxyinfo Returns information about the user's proxy certificate
gd_proxyquery Queries whether a valid proxy certificate exists
gd_destroyproxy Destroys the local copy of the user's Globus proxy certificate
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Certificate Management
C:\GEODISE
>> gd_createproxy
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Certificate Management
C:\GEODISE
>> subject = gd_certinfosubject : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme poundissuer : C=UK,O=eScience,OU=Authority,CN=CA,[email protected] date : Wed Sep 22 16:31:24 BST 2004end date : Thu Sep 22 16:31:24 BST 2005 subject = /C=UK/O=eScience/OU=Southampton/L=SeSC/CN=graeme pound >> isvalid = gd_proxyinfosubject : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme pound,CN=proxyissuer : C=UK,O=eScience,OU=Southampton,L=SeSC,CN=graeme poundtype : full legacy globus proxystrength : 512 bitstimeleft : 11 h, 59 min, 30 sec
isvalid =
1
>>
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Job Submission Functions
gd_jobstatus Gets the status of a Globus GRAM job
gd_jobsubmit Submits a compute job to a Globus GRAM job manager
gd_jobpoll Queries the status of a Globus GRAM job until complete
gd_jobkill Kills a Globus GRAM specified by a job handle
gd_chmod Changes file permissions of a file on a Globus resource
gd_condorsubmit Submits a job to a Condor pool via a Globus resource
gd_submitunique Submits a GRAM job to a unique working directory
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Job Submission
C:\GEODISE
>> RSL = '&(executable="/bin/date")(stdout="date.out")';>> host = 'grid-compute.oesc.ox.ac.uk';>> jobhandle = gd_jobsubmit(RSL, host)
jobhandle =
https://grid-compute.oesc.ox.ac.uk:30001/30705/1098694366/
>> isdone = gd_jobpoll(jobhandle)
isdone =
1
>>
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
GridFTP File Transfer
gd_getfile Retrieves a remote file using GridFTP
gd_putfile Puts a remote file using GridFTP
gd_transferfile Performs a third-party file transfer using GridFTP
gd_makedir Creates a remote directory using GridFTP
gd_listdir Lists the contents of a directory on a GridFTP resource
gd_fileexists Tests the existence of files on a Globus resource
gd_rmdir Deletes a remote directory using GridFTP
gd_rmfile Deletes a remote file using GridFTP
gd_rmuniquedir Deletes a remote directory and its contents
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
File Transfer
C:\GEODISE
>> exists = gd_fileexists(host, 'date.out')
exists =
1
>> gd_getfile(host,'date.out','localfile.txt');>> type('localfile.txt')
Mon Oct 25 09:52:46 BST 2004
>> gd_rmfile(host, 'date.out');>> exists = gd_fileexists(host, 'date.out')
exists =
0
>>
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Resource Testing
gd_servermetrics Performs tests upon a list of Globus resources
gd_testauthentication Tests authentication with a Globus resource
gd_testfiletransfer Tests file transfer to a Globus resource
gd_testjobsubmission Tests the job submission to a Globus resource
gd_timeauthentication Times authentication to a Globus resource
gd_timefiletransfer Times file transfer to a Globus resource
gd_timejobsubmission Times a job submission to a Globus resource
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Resource Testing
C:\GEODISE
>> servers(1).name = 'grid-compute.leeds.ac.uk';>> servers(2).name = 'escience-dept2.sesnet.soton.ac.uk';>> results = gd_servermetrics(servers);>> disp(results(1))
name: 'grid-compute.leeds.ac.uk' authentication: 188 jobsubmission: 672 filetransfer: 984
>> disp(results(2)) name: 'escience-dept2.sesnet.soton.ac.uk' authentication: 125 jobsubmission: 485 filetransfer: 4938
>>
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
GeodiseLab XML Toolbox
Available Now
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Engineer’s familiar scripting environment
3C0003
2B0002
1A0001
ValueNameID
XML Toolbox
Web Technology & Services
Databases
Access
Communication
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Features
• Powerful Matlab to XML interface• Allows transparent conversion between proprietary Matlab
data types and standard XML• Brings flexibility of open XML Standard to Matlab• Easy-to-use interface (4 simple core functions:
xml_save, xml_load, xml_format, xml_parse)• Extensive documentation & direct help functionality• Large range of potential applications:
– Science, Engineering, IT, Web, …
Toolbox
Toolbox
<struct xmlns="http://www.geodise.org/matlab.xsd" idx="0" fields="name id param"> <char idx="1" name="name" size="1 14">eng_opt_design</char> <char idx="1" name="id" size="1 11">1022223-779</char> <struct idx="1" name="param" size="1 1" fields="width height material"> <double idx="1" name="width" size="1 1"> 100 </double> <double idx="1" name="height" size="1 1"> 87.9 </double> <double idx="1" name="material" size="1 1"> 0.07244 </double> </struct></struct>
<root> <name>eng_opt_design</name> <id>1022223-779</id> <param> <width> 100 </width> <height> 87.9 </height> <material> 0.07244 </material> </param></root>
DBFilestoreStandard File Formats
(eg. STEP-XML, …)
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
XML Toolbox in action
• Jan 2004 v2.0 released• Nov 2004
– Downloads from soton.ac.uk/~gridem: 227– Downloads from mathworks.com/matlabcentral: 882
• Apr 2005 v3.0– > 1500 downloads total
• Large user base– Academic & Commercial– Extensive testing & feedback community– Market pull
• So far very positive feedback from users• Matlab Central user rating: 5 / 5• Linked by Mathworks help for XML in Matlab
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Aalesund College: Read data from databaseAberdeen University: Export analysis results from Matlab to XMLADVA: to read existing XML-files in a Matlab structure and do some calculationsAdvanced Tech Center: configuring xml input into matlabAlfred-Wegener-Institut: Handle data from echo sounderAMAT: Evaluation of XML input in matlabAnna University: We are programmed to develop a new tool for analysing discrete event system using matlab.Applied Math Labolatory, Ecole Centrale ParisAston University: To read date from xml files in matlabAvantium Technologies BV: Transferring HTS data stored as XML into MatlabBAE Systems: Data import/export between MATLAB and Delphi; Generation of XML files to control a BAE SYSTEMS Software Defined RadioBeb CorpBosch GmbHCalypso Medical: looking at xml files in matlabCarnegie Mellon university: Use XML in our facial expression analysis projectCenter for Advanced Spatial Technologies: Writing least squares adjustments for geodetic observations as part of research in new techniques for parameter estimation combining photogrammetric and terrestrial observations. Data is varied but structured and I am interested in using XMLChouaib Doukkali Universit: Numericals simulations of the physics phenomenonsChristian-Albrechts-Universitat zu Kiel (CAU): http://busselab.uni-kiel.deCity College of CUNY: Projects using Matlab and XML technology.CMS: For trialCold Spring Harbor Laboratory: Storing and parsing neuronal dataConestoga college: intro to telecommeCrystal EngineeringCU: Sending sending commands to matlab via xml, outputting results of calculations from matlab in xml to be used by other products
Daimler Chrysler: test
DeAM: xml reading
DECO
DEIMOS Engenharia: Reading/writing configuration files
DGA Defence, France: For managing data files used in simulation's input
DSTO: input parameters to radar model
EADS CRC: InformationEmyo: For teachingENI: Reading & Modifying XML documentsEPFL: Evaluation for data exchange between matlab and other softwareFairchild Dornier: Currently I make a research, how I can efficiently store my matlab data structure in a text editable data format, that's why I'am looking for a xmlread and a xmlwrite function.Financial EnginesFIOTFraunhofer Germany: The last aim of our research is to provide user a platform for maintenance oriented decision via web service.
38 sample institutions
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
GeodiseLab Database Toolbox
Available Now
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Database Toolbox – Aims
• Make data easier to manage, find and share• Store data with descriptive information
– Standard and user defined metadata– Query over metadata to easily locate data
• Familiar interface for engineers– Wrap toolbox as Matlab/Jython functions– Easy integration into user scripts
© Geodise Project, University of Southampton, 2001-2005.http://www.geodise.org/
Database Toolbox – Features
• Archive files and data structures with metadata
• Group related data• Simple query interface• Retrieve data based on ID rather than
location• Security
– User grants permission for others to access their data– Certificate-based authentication and authorisation– Secure Web service access to central database
© Geodise Project, University of Southampton, 2005.http://www.geodise.org/
Functions Overview
gd_archive Stores a file or structure with metadata into an archive
gd_query Queries over metadata or structures in the archive
gd_retrieve Retrieves a file or structure from the archive to the local machine
gd_addusers Grants users permission to access some data
gd_datagroup Creates a new datagroup to aggregate files and structures
gd_datagroupadd Adds a file or structure to a datagroup
gd_display Displays the results of a query
© Geodise Project, University of Southampton, 2005.http://www.geodise.org/
GeodiseLab Database Toolbox Worked Example
GridClient
QueryService
Globus ServerXML Toolbox Java CoG
LocationService
AuthorisationService
MetadataService Database
Generate Data
GridClient
QueryService
Globus ServerXML Toolbox Java CoG
LocationService
AuthorisationService
MetadataService Database
Metadata
GridClient
QueryService
Globus ServerXML Toolbox Java CoG
LocationService
AuthorisationService
MetadataService Database
Archive
XML Toolbox Java CoGGlobus Server
LocationService
AuthorisationService
MetadataService Database
GridClient
QueryService
Globus ServerXML Toolbox Java CoG
LocationService
AuthorisationService
MetadataService Database
Generate More Data
Java CoGGlobus Server
XML Toolbox
LocationService
AuthorisationService
MetadataService Database
GridClient
QueryService
Globus ServerXML Toolbox Java CoG
LocationService
AuthorisationService
MetadataService Database
Group Files
XML Toolbox
AuthorisationService
MetadataService Database
GridClient
QueryService
Globus ServerXML Toolbox Java CoG
LocationService
AuthorisationService
MetadataService Database
Query
XML Toolbox
QueryService
AuthorisationService
Database
GridClient
QueryService
Globus ServerXML Toolbox Java CoG
LocationService
AuthorisationService
MetadataService Database
Retrieve
Java CoGGlobus Server
LocationService
AuthorisationService
Database
© Geodise Project, University of Southampton, 2001-2005http://www.geodise.org/
Applications - GENIE
GENIE supports environmental scientists modelling long term climate change.
Optimising a parameterised model over a multi-dimensional state space.
Database used for monitoring, sharing, post-processing and fault tolerance.
Earth system model tuning studies
© Geodise Project, University of Southampton, 2005.http://www.geodise.org/
Distributed Systems: 2005 and beyond?
Network
IP
HTTP
(HTML)
Compute/ Data
Moore’s Law
Software
(HTML) XML
Web Services
ASCII
Experiences
Services
RobustnessSecurityReliabilityVisualizationQuality of ServiceKnowledge TechnologiesCollaboration tools & frameworks
Platform StrategiesDevelopment Tools
Systems ArchitectureDistributed Data ManagementProblem Solving Environments
Algorithms, Methods and LibrariesHeterogeneous Distributed
Computing