![Page 1: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/1.jpg)
Developing an Agricultural Monitoring Developing an Agricultural Monitoring System from Remote Sensing Data System from Remote Sensing Data
Using GridRPC on Ninf-GUsing GridRPC on Ninf-G
Shamim Akther, Yann Chemin, Honda KiyoshiAsian Institute of Technology, Thailand
Itthichok Jangjaimon, Putchong UthayopasKasetsart University, ThailandAnd Thai National Grid Project
![Page 2: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/2.jpg)
2
MotivationMotivation
• Recent advances in data assimilation of satellite imagery for the purpose of agricultural monitoring led to the increasing demand for computing power
• There is a need for technology and platform that can meet the requirement
![Page 3: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/3.jpg)
3
SWAP Model and StructureSWAP Model and Structure
Adopted from Van Dam et al. (1997)Drawn by Teerayut Horanont (AIT)
![Page 4: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/4.jpg)
4
SWAP-GASWAP-GA
• Know SWAP Input parameter – sowing date, soil property, Water management, etc
• Finding SWAP Crop model
Comparison of Satellite LAI and Simulated LAI
0
1
2
3
4
5
0 30 60 90 120 150 180 210 240 270 300 330 360
DOY
LA
I
LAI_sat
LAI_sim
![Page 5: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/5.jpg)
5
SWAP Model Parameter identificationSWAP Model Parameter identification - Data Assimilation using RS and GA- Data Assimilation using RS and GA
0.00
1.00
2.00
3.00
4.00
0 45 90 135 180 225 270 315 360
Day Of Year
Eva
po
tra
nsp
iratio
n L
AI
RS ObservationRS Observation
SWAP Crop Growth ModelSWAP Crop Growth Model
SWAP Input ParametersSWAP Input Parameters
sowing date, soil property, sowing date, soil property, Water management, and etc.Water management, and etc.
LAI, LAI, EvapotranspirationEvapotranspiration
0.00
1.00
2.00
3.00
4.00
0 45 90 135 180 225 270 315 360
Day Of Year
E
avpo
tran
spira
tion
LA
I
FittingFitting
LAI, LAI, EvapotranspirationEvapotranspiration
Assimilation by Assimilation by finding Optimized finding Optimized
parametersparameters
By GABy GA
RSRS ModelModel
![Page 6: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/6.jpg)
6
Pixel pool
Result pool
Best fit population
Sequential Version
Sequential VersionSequential Version
• Take satellite image and loop through each pixel
• For each pixel perform GA processing
![Page 7: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/7.jpg)
7
ChallengesChallenges
• The calculation time for identify SWAP parameters only for 1 pixel (1 sq.km) takes several minutes to 30 minutes.
• Thus, a RS image of 1000 x 1000 sq.km of 1000x1000 pixels will take more than 50 years (30min x 1000 x 1000) is not acceptable.
• Solutions– Parallel Computing on
Cluster and Grid
• Longitude: 100.008133
• Latitude: 14.388195
![Page 8: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/8.jpg)
8
Thai Grid InfrastructureThai Grid Infrastructure
Network
SPIRIT
PALM
OPTIMA
ENQUEUE
GASS
AMATA
MAEKA
WARINE
KU
ThaiGrid User
ThaiGrid PortalTgcc.cpe.ku.ac.th
Submit
Grid Job Scheduling
CU
KMUTNB
KMUTT
AIT
SUT
CAMETAINCA
![Page 9: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/9.jpg)
9
Programming ModelProgramming Model
• Message Passing Using MPI– Widely adopted standard– Work efficiently on Cluster– Can work at some level on Grid
• Grid RPC– Remote procedure call programming model
on Grid system– GGF adopted as a standard programming on
Grid– Several Implementation : GridSolve, Ninf-G
![Page 10: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/10.jpg)
10
GridRPC Programming ModelGridRPC Programming Model
• Application partition into client/server– Multiple client can be used to achieve
parallelism using asynchronous call
• Role– Server side
• Implement the algorithm as a remote procedure
• Using Grid for security mechanism and execution management
– Client side • Implement main workflow of the application
• Handling input/output for user
• Call Remote Functions at Server using GridRPC API
![Page 11: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/11.jpg)
11
Ninf-G programming modelNinf-G programming model
• A software package which supports programming and execution of Grid applications using GridRPC
• Advantages– Simple programming model that work
efficiently over lossely copuple heterogenous system
– Hiding most of the grid complexity from programmer and developer
Low level Grid Middleware
Grid RPC
Grid application
Grid Fabric Layer
![Page 12: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/12.jpg)
12
Partitioning strategy on clusters and GridPartitioning strategy on clusters and Grid
• There are 3 possible partitioning strategies– Population Distribution – Pixel Distribution– Hybrid Distribution
• Group compute nodes as virtual group• each virtual group take care one pixel• inside the group, distributing population to members
• Pixel and population distributed has been implemented using MPI and reported in our previous work
![Page 13: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/13.jpg)
13
Population Distribution StrategyPopulation Distribution Strategy
Pixel pool Result pool
Eval Eval Eval
GA
Set of Population
fitness
![Page 14: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/14.jpg)
14
Pixel Distribution StrategyPixel Distribution Strategy
Pixel pool Result pool
GA GA GA
Image
Pixel
Results
![Page 15: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/15.jpg)
15
Hybrid Distribution StrategyHybrid Distribution Strategy
GA
Pixel pool Result poolImage
Pixel
Results
EvalEval Eval
population
Cluster
GA
Pixel
Results
EvalEval Eval
population
Cluster
![Page 16: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/16.jpg)
16
Extension to the GridExtension to the Grid
• Modify and adjust code to change distributing middleware from MPI to Ninf-G
• Implement secure data-transferring method for Grid instead of using NFS for cluster environment
• Gain benefit of Grid computing– More effective– More scalable– More security
![Page 17: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/17.jpg)
17
ArchitectureArchitecture
![Page 18: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/18.jpg)
18
Table 4: Server Program Model with Ninf_G //GIS program extract pixels from the Modis RS image and //pixel information will be written in separate file for each pixel. for pix_no = first to last { //Grid Remote Procedure Call environment initialization. grpc_initialize(config_file); for(pixel is 1 to 16) //let one time we manipulate 16 pixels { //Each pixel create one handle grpc_function_handle_t handle[]; grpc_function_handle_init(handle[pixel],host_name,port_name,”lib_name”); } for (pixel 1 to 16) { //for the 16 pixels a asyncnous call occure grpc_call_async(handle[pixel], args….); } // Wait until every node finished grpc_wait_all(); for( pixel 1 to 16) { // all handles need to detruct grpc_function_handle_destruct(handle[pixel]); } grpc_finalize(); }
GA_SWAP(pixel_buffer, result){
Load(pixel_buffer);Run(swap);Send(result);
}
![Page 19: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/19.jpg)
19
Results & DiscussionsResults & Discussions
• Finish only pixel distribution over cluster
• Experiment run on 3 clusters in the Grid – MAEKA 31 nodes Opteron Dual 1.4&1.8 GHz– AMATA 14 nodes Athlon XP 1 GHz– MAGI 4 Nodes Athlon XP 2500+
![Page 20: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/20.jpg)
20
ResultResult
Pixelversion speedup
0.000
1.000
2.000
3.000
4.000
5.000
1 2 3 4 5nodes
speedup
pop10pix5
pop20pix5
![Page 21: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/21.jpg)
21
ResultResult
Popversion speedup
0.000
2.000
4.000
6.000
8.000
10.000
12.000
14.000
16.000
18.000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
nodes
speedup
pop10pix1
pop10pix5
pop20pix1
pop20pix5
![Page 22: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/22.jpg)
22
ResultResult
speedup 20 pop 5 pixels
0.000
2.000
4.000
6.000
8.000
10.000
12.000
14.000
16.000
18.000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
nodes
speedup
pixel-version
pop-version
![Page 23: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/23.jpg)
23
Lesson LearnedLesson Learned
• Grid programming is still difficult – Learning curve– Debugging? – Grid reliability. May be useful to have a small
development grid separate from deployment grid
• Ninf-g application on grid is working fine. A good approach for development of grid applications.
![Page 24: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/24.jpg)
24
Future WorkFuture Work
• Improve performance
• scheduler and node selecting (now is only round-robin)
• Test running across international Grid– collaboration with Prof. Kento Aida,Titech– PRAGMA testbed
• Deployment of ninf-g for Thai National Grid application development
![Page 25: Developing an Agricultural Monitoring System from Remote Sensing Data Using GridRPC on Ninf-G Shamim Akther, Yann Chemin, Honda Kiyoshi Asian Institute](https://reader030.vdocument.in/reader030/viewer/2022032721/56649cd85503460f949a104e/html5/thumbnails/25.jpg)
Thank youThank you
Q&A