development of data integration & analysis system in japan
TRANSCRIPT
Development of Data Integration & Analysis System in Japan
Activities toward interoperability
Seishi NinomiyaInstitute of Sustainable Agro-ecosystem Services,The University of Tokyo
Productivity
Quality/Safety
Farmers’ benefits
Food loss/waste
Minimum emission
Limited resources/water
/land/energy
Ecosystem/Biodiversity
Food distributionDiet
transition
Climatic change/disaster
Toward Sustainable and sufficient food productionToward Sustainable and sufficient food production
SecurityFood
Security
What we need for data-centric science in agriculture
• Utilization of legacy datao Yield data, variety data, quality data, soil data, market data, …………o Need to rescue such data
• Sensor innovation IOTo To efficiently monitor the facts in fields, market, demands, logistics, processing,….o To collect knowledge of farmers, tacit knowledge
• Data integration and efficient usage/ Interoperabilityo Common platform for seamless data exchange with standardo Agricultural cloud and database
• Tools for analysis/analytics and for supporting decisionso Statistics, data-mining , knowledge extraction, risk managementso Big data-based optimizationo Enrichment of commonly usable APIs
• Communication innovationo Efficient Knowledge transfer to farmers
• Service science
Big data
+ Advancement of Agricultural Science
DIASData Integration & Analysis System
© GEO Secretariat
• The Group on Earth Observations is coordinating efforts to build a Global Earth Observation System of Systems (GEOSS).
• GEO was launched in response to calls for action by the 2002 World Summit on Sustainable Development and by the G8 leading industrialized countries.
GEO/GEOSS
S&D strategy in Japan
RECCA2
S&T Basic plan -5 will start since 2016
Data Integration and Analysis System (DIAS)
• DIAS was launched in 2006 as a Japanese contribution to GEOSS
• one of five National Key Technologies defined by the 3rd Basic Program for Science and Technology of Japan. o Total of USD 60 million for 10 years by MEXT from 2006
• The missions are:o to coordinate the cutting-edge information science and
technology and the various research fields addressing the earth environment;
o to construct data infrastructure that can integrate earth observation data, numerical model outputs, and socio-economic data effectively;
o to create knowledge enabling us to solve the sustainable worldo to generate socio-economic benefits
20PB by 2014
Data Storage Core System
User Communication & Management
Search & Discovery System
ScienceSocietal Benefit
Data
Mata Data
Document
DataCleansing
System(QC)
Processed Data
Data LoadingSystem
Data Integration & Analysis System
DataArchive
Data Download
System
AnalysisSystemsBy User
Communication Tool Management Tool
Data Provider
Meta dataRegistration
Document Registration
Meta dataStandard
Inter-operability
Portal
User Authentication
User
Authorized User
10
DB
DB
0
50
100
150
200
250
300
350
1992
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
2014
2016
2018
2020
A1:高度成長社会
A2：多元化型
B1:持続発展型
B2：地域共存型
DB
DB
DB
DB
DB0
50
100
150
200
250
300
350
1992
1994
1996
1998
2000
2002
2004
2 006
2 008
2 010
2 012
2 014
201 6
201 8
202 0
A1:高度成長社会
A2：多元化型
B1:持続発展型
B2：地域共存型
DB
DB0
50
100
150
200
250
300
350
1992
199 4
199 6
1 998
2 000
2002
2004
200 6
200 8
201 0
2012
2014
2016
201 8
202 0
A1:高度成長社会
A2：多元化型
B1:持続発展型
B2：地域共存型
DB
DB
DB
Meteorology
Ecosystem
AgricultureHydrology
Land Use
Climatology
Health
农业
เกษตรกรรม
농업
Where is data?How to access?
Technical Terms among
Different Disciplines
Quality?Reliability?
Upload
Meta Data
Meta Data Meta Data
Data Integration and Analysis System (DA-09-02a)
Quality Control
Data Provider (Observer)
User
Meta Data Registration
•Search with Metadata•Data Download
•Document Generation from Meta Data•Data Visualization
・・
ObservationData Meta Data
Data Upload＋(part of )Meta Data
ObservationData
Meta Data
Data Quality Control Process
Meta Data
Post-QCObservation Data
InputMeta Data
Data Download Search IF Document Generator Visualization System
Data A
rchivingD
ata IntegrationWeb-based Data Archiving & Integration System
Basic Information
Observation Point Inf.,Contact Person Inf.,……
Ontology development in DIAS
Challenge for Data Management & Fusion
Syntax InteroperabilityProposal of Standard Schema and Interface.It is not enough for diversified Geo-spatial Information.e.g. legacy data.
New Challenge of Data Management and Fusion
RegistryVisualizing diversified data, Helping data convergency .One stop service for data utilization.
Very Large& Heterogeneous Data Management and FusionData quality checking, Emergency response for disaster, Automatic processing and fusion, Change detection, etc.
Integrating Observation Data and
Model Simulation
Semantic Interoperability (Ontology )Semantic Interoperability for geo-spatial data by using data definitions, terminologies, relations, landnames, etc.
By Msahiko Nagai, U..Tokyo & AIT, 2015
Semantics Activity
Existing Glossaries
1 WMO Glossary
2 CEOS Missions, Instruments and Measurements(MIM) Database
3 CEOS Systems Engineering Office(SEO)
4 GEMET
5 INSPIRE Feature Concept Dictionary
6 SWEET
7 CUAHSI
8 CF Standard Names
9 GCMD
10 Eurovoc Thesaurus
11 International Glossary of Hydrology/UNESCO
12 Marine Metadata Interoperability
Close Match
We are forcing on the one hand the implementation of semantic interoperability arrangement with ontological information.
DIAS Vocabulary Registry
SKOS
146 observation parameters with SBA
Define and associate with EO Vocabulary and Existing Glossaries
By Msahiko Nagai, U..Tokyo & AIT, 2015
Vocabulary Registry to Find Similar Technical TermInput Keywords, “precipitation”
Similarity scorewith the input keywords
By Msahiko Nagai, U..Tokyo & AIT, 2015
INDEX DB RESULTS- Scholarly Journals- Social Data
- Lat / Lng- Scientific Values- Keywords- Date
Scientific Data[ Remotely Sensed Data, Meta - Data ]
- Description- Images- Date- Geo - Tags- Videos
]
Social Data[Ushahidi, Google News, Social
Networks]
INDEXED
DATABASE
Use
rIn
terf
ace
QUERY PARAMETERS- Location- COP / SBA- Date
Scholarly Journal Data[ Sci-Verse HUB, Mendeley api ]
• Keywords• Abstract / Full Text• Author(s)• Published Date• Scientific Models Used• Input Data / Output Data• Geo - Tags
ONTOLOGY DEVELOPMENT / CONCEPT TAGGINGEXTRACTION / INDEXINGONTOLOGY UPDATESEARCH REQUESTSEARCH RESULTS
ONTOLOGY
Application Service
EXTRACTOR
SEARCH ENGINE
JournalsKnowledge based Ontology Development / Update
JSP Service
24By Msahiko Nagai, U..Tokyo & AIT, 2015
MetBrokerWeather data integration
• Heterogeneity among data sources is a big issue in the Internet. (data structure, access methods, etc.)
• Data brokers provide consistent access to those heterogeneous data sources
MetBroker for various weather databases
MetaData
Heterogeneous and Autonomous DBs
Rice Growth Prediction
Farm Management
MetBroker
Pesticide Prediction
Heterogeneity is absorbed by brokers (mediators)
B-DB
C-DB
A-DB
MetBroker Since 2000 -Spatial integration of weather data
• MetBroker provides applications consistent access to heterogeneous weather databases and covers 30,000 weather stations of 25 DBs
• API MetXML
Crop model + MetBroker = Potential Rice Yield
28
IOT/Sensor data
Giving Interoperability to heterogeneous sensing datavia OGC Standard Web Service, SOS
OGC API
GetCapabilities
List of authorized SOS stations with its sensors
GetObservation
Sensor Data withTimestamp
1
2
Simulation System
User Interface for Famers
Kiyoshi HONDA, R. Chinnachodteeranun, A. Witayangkurn, APAN Meeting 4 Mar 2015
Sensor Infra. And Multi-Layered Web Service
Sensor Infra
Water Level/Temp
via NICT
NARO 1km Mesh Agri Weather (Past and 2w Forecast)
NIAES PointAgri. Weather
Other Sensor
Interpolation, Statistics, Visualization
Weather Generator
Rice Crop Simulation
Standard APIStandard APISOS
Open API
Open API
Application
Open APIOpen API
Application
Visualization
Analysis Appli.
Crop Simulation
Developed by single developer
Obtain necessary functionalities via Web Service
Anyone can access to high-level functionalities
Sensor Virtualization, New sensor, sensor transfer will be reflected application automatically
Kiyoshi HONDA, R. Chinnachodteeranun, A. Witayangkurn, APAN Meeting 4 Mar 2015
Farm data
33
Conceptual sketch of “CLOP”
APAN 39th in FukuokaAPAN 39th in Fukuoka T. Yoshida, NARO
USD 10 million for 5 years by MAFF from 2014
• Current range FIX-pms covers is limited to farm work and production process management.
34
Cover range of ‘FIX-pms’
APAN 39th in Fukuoka T. Yoshida, NARO
• Defined based on agroXML.
35
Outline structure of ‘FIX-pms’
APAN 39th in Fukuoka T. Yoshida, NARO
Structure of API mashup : 4 Layers
36
Term, Code Layer
Data Content Layer
Data Format (Container)
Layer
API Layer
agroXML, Sensor ML, GML/KML, GPX, …FarmXML(FIX-pms), BIX-pp, GPXX, …
Data structure?Data meaning?Data relation?
VVV
RDF, UML, …
SOS, WMTS, WMS, WFS, …MetXML, PDS, …
Content list in certain region of interest among certain stakeholders, …
Terminology, ontology, …Code system definition, …
(Language / Localization)APAN 39th in Fukuoka T. Yoshida, NARO
International collaboration
W3C Agriculture CGW3C Team | Posted on: October 15, 2014
The Agriculture Community Group has been launched:The initial mission of the Agriculture Community Group is to gather and categories
existing user scenarios, which use Web APIs and services, in the agriculture industry from around the world, and to serve as a portal which helps both web developers and agricultural stakeholders create smarter devices, Web applications & services, and to provide bird’s eye view map of this domain which enables W3C and other SDOs tofind overlaps and gaps of user scenarios and the Open Web Platform.
We’ll try to collect facts and knowledge from around the world through crowd-sourcing, while, at the same time, build a scaffold for it by quickly gathering key topics from Japanese agricultural stakeholders. Smart Platform Forum supports this early stages by connecting relevant stakeholders in Japan and organizing face-to-face meetings if needed to proceed faster.
https://www.w3.org/community/agri/
Future Internet PPP – 300 M€ EUfunding
39
