co-resyf hands-on sessionsco-resyf.eu/wordpress/wp-content/uploads/2017/07/2-2-2-3...2017/07/02 ·...
Post on 02-Nov-2020
1 Views
Preview:
TRANSCRIPT
Coastal Waters Research Synergy Framework
Coastal Waters Research Synergy Framework
This project has received funding from the European
Union’s Horizon 2020 Research and Innovation Programme under
grant agreement no 687289
Summer School Meeting, Thursday 13th July 2017LNEC, Lisbon, Portugal
Co-ReSyF Hands-on sessions▶ Hervé Caumont (Terradue)
Coastal Waters ResearchSynergy Framework
http://co-resyf.eu/
2
http://geoportal.co-resyf.eu/
Cloud services
User profiles
Novice:- EO data access helpers- Processing ‘as-a-Service’
Intermediate:- Workflow composer service- Reuse of existing modules
Expert:- Cloud Sandbox service- Software integration
❏ Data Processing
❏ Algorithm integration
❏ Collaborative research
3
Research Apps
Aiming at ‘Platform’ services
Front-end Back-end
4
Data Search
Workflow Composer
Workflow Manager
Visualisation & Analysis
Platform Services v1.0
Hosted processor
Third-party tools repos
Co-ReSyF toolkit
repository
Workflow Engine
Data Agency
Cloud Controller
Processor as a Service
User Support
AAA
OGC WPS, Hadoop
OGC WPS
▶ OGC OSxGT
▶ yum, conda
5Innovation & automationBenefits for Platform users
Develop Jobs
Setup Workflow
Sandbox Environment
Validate App
Operational Cloud
Environment
Design Service
Deploy Service
Monitor Service
▶Hands-on focus
6
An application lifecycle
7
▪ Processing libs developed in any programming language supported by CentOS 6.5
▪ C/C++, Java, Python (most common)▪ Matlab and IDL▪ Other languages shall be
previously analyzed
▪ Then synchronized onthe Platform via softwarepackages management tools(RPM repos, GitHub, …)
Design & Libraries Declaration (1/4)
8
Python, CDAT, R, BEAM, BEAT
Service integration (2/4)
▪ Service developer uses the sandbox tools to implement, test and validate a scalable workflow
▪ Sandbox includes tools to proof test data discovery, parallel processing and access to results
9Cloud Compute deployment (3/4)
▪ After validation, the developer publishes the Service for integration on a selected Cloud Infrastructure
▪ A ‘Processor as a Service’ accesses complete pools of datasets,allows application intensive computing, and stores processor outputs, e.g. back to the lab
10Service Operations (4/4)
▪ Processing Services are operated on a selected Cloud provider
▪ Operators manage the service via their user account on Terradue Cloud Platform
▪ Resources can scale up/down▪ The development environment
remains active for maintenance activities on the Service
▪ Service End Users can access the service’s products from ad-hoc community portals
▪ Includes support for standard protocols like OGC Web Services
11
Hands-on sessions
12Hands-on sessions overview
Session 1 - Data access and adding your own data to the platform Cloud storage
What are the Platform APIs, along with integrated tools for data staging within Cloud Apps, that deliver custom data management services to registered developer users on the Platform ?
Session 2 - Develop your own processing chain, using existing modules
How registered users are able to create a work environment with state of the art tools like Git and GitHub, Maven, Oozie workflows, … and work collaboratively for the development of powerful Apps ?
13
Get the instructions
Visit
https://gist.github.com/HerveCaumont/
Select
simple-da-training-part-1
Session 1Data access and adding your own data to the platform Cloud storage
14
Get the instructions
Visit
https://gist.github.com/HerveCaumont/
Select
simple-dcs-training-part-1
Session 2Integrate and test your own processing chain, using existing modules
15
Co-ReSyF Project EC fundinghttp://www.copernicus.eu/projects/co-resyf
Co-ReSyF Websitehttp://co-resyf.eu/
Co-ReSyF GeoPortalhttp://geoportal.co-resyf.eu/
Co-ReSyF Twitter handlehttps://twitter.com/Co_ReSyF
Co-ReSyF repositories on GitHubhttps://github.com/ec-coresyf
User forumhttps://discuss.terradue.com https://discuss.terradue.com/c/co-resyf
Session 3Resources at your disposal
Terradue Cloud Platform, sign-uphttps://www.terradue.com
Private Cloud VPN setuphttps://access.terradue.com
Cloud Dashboardhttps://cloud.terradue.com
Dev Cloud Sandbox (DCS) documentationhttps://docs.terradue.com/developer-sandbox
DCS Community shared applicationshttps://github.com/Terradue?q=dcs
T2 API documentationhttp://docs.terradue.com/t2-api/
Operations Supporthttps://support.terradue.com
Looking forward hearing from you!CO-RESYF SUMMER SCHOOL 2017
16
Hervé CaumontCloud Platform Operations, Terradueherve.caumont@terradue.com
Back-up slides(from RPM-1)
17
Choose a language 18
Sen2Cor is a processor for Sentinel-2 Level 2A product generation and formatting; it performs the atmospheric, terrain and cirrus correction of Top-Of- Atmosphere Level 1C input data.
Sen2Cor creates Bottom-Of-Atmosphere, optionally terrain and cirrus corrected reflectance images; additional, Aerosol Optical Thickness, Water Vapor, Scene Classification Maps and Quality Indicators for cloud and snow probabilities.
http://step.esa.int/main/third-party-plugins-2/sen2cor/
https://anaconda.org/Terradue/sen2cor
19The sen2cor processing tool
20Understand the application structure
function main()
{
local reference=${1}
ciop-log "INFO" "**** Sentinel-2 Atmospheric Correction ****"
ciop-log "INFO" "------------------------------------------------------------"
ciop-log "INFO" "Input S-2 L1C product reference: ${reference}"
ciop-log "INFO" "------------------------------------------------------------"
ciop-log "INFO" "STEP 1: Getting input product"
local_product=$( get_data "${reference}" "${TMPDIR}" ) || return ${ERR_GET_DATA}
ciop-log "INFO" "------------------------------------------------------------"
ciop-log "INFO" "STEP 2: SEN2COR tool" output=$( sen2cor "${reference}" "${local_product}" ) || return ${ERR_GET_DATA}
ciop-log "INFO" "------------------------------------------------------------"
ciop-log "INFO" "STEP 3: Publishing results" publish_data "${output}" || return ${ERR_PUBLISH}
ciop-log "INFO" "------------------------------------------------------------"
}
21Understand the application structure
for reference in sys.stdin:
reference = reference.strip('\n')
ciop.log('INFO', '**** Sentinel-2 Atmospheric Correction ****')
ciop.log('INFO', '------------------------------------------------------------')
ciop.log('INFO', 'Input S-2 L1C product reference: ' + reference)
ciop.log('INFO', '------------------------------------------------------------')
ciop.log('INFO', 'STEP 1: Getting input product')
local_product = get_data(reference, os.environ['TMPDIR'])
ciop.log('INFO', '------------------------------------------------------------')
ciop.log('INFO', 'STEP 2: SEN2COR tool')
output = sen2cor(reference, local_product)
ciop.log('INFO', '------------------------------------------------------------')
ciop.log('INFO', 'STEP 3: Publishing results')
publish_data(output)
ciop.log('INFO', '------------------------------------------------------------')
22Inspect the logs - Job
23Inspect the logs - Task
24Inspect the logs - Attempt
top related