godan action wp1: data standards: survey, gap analysis and...

GODANActionWP1:datastandards:survey,gap

analysisandrecommendationsFocusonweatherdatastandards

ValeriaPesceGlobalForumonAgriculturalResearchandInnovation(GFAR)

LeighDodds,JeniTennison,PaulineL'HénaffOpenDataInstitute(ODI)

PanagiotisZervas(AgroKnow)

GlobalOpenDataforAgricultureandNutritioninitiative(GODAN)

TheGODANinitiativewasoneoftheoutputsoftheG8InternationalConferenceonOpenDataforAgricultureandwasannouncedattheOpenGovernmentPartnershipConferenceinOctober 2013.Theinitiativefocusesonbuildinghigh-levelsupportamonggovernments,policymakers,internationalorganizationsandbusiness.

Currentlyaround500partnersworldwidefromnationalgovernments,non-governmental,internationalandprivatesectororganizations.

www.godan.info

GODANAction

Three-and-a-half-yearprogramme launchedbytheUK’sDepartmentforInternationalDevelopment.

GODANActionbringstogetheragricultureandnutritionspecialistsandopendataexpertsandwillsupportGODANinitsmissionbybuildingpeople’scapacitytoengagewithopendata.

http://www.godan.info/godan-action

UndertheDFIDGODANfundingstream

GODANAction:Focalareas1) Standards - Enhancingdatastandardsandpromotingbestpracticein

agricultureandnutritiontoimproveinteroperability.1) Mapofagri-fooddatastandards2) Gapanalysisonuseandusabilityofdatastandards3) Recommendationstoaddressgaps4) à Specificationsà services,pilotimplementations

2) Research - Identifyingandimprovingtoolsandmethodsforevaluationoftheimpactofopendatausageininitiativesandinvestmentsinagricultureandnutrition.

3) Capacities - Buildingthecapacityanddiversityofopendatausers,leadingtomoreeffectiveuseofdataintacklingkeyagricultureandnutritionchallenges.

Purposeoftheglobalmapofdatastandards

• Themainpurposeofaglobalmapofdatastandardsinaspecificfieldistopromotethediscoveryandreuseofvocabularies andtheirproperties,classesandcontrolledvalues.Thereuseofexistingvocabulariespromotesgreaterinteroperabilitybetweenvocabulariesanddatasets.paraphrasingwhattheDublinCoreMetadataInitiativesaysabouttheirDCMIRegistry(http://dcmi.kc.tsukuba.ac.jp/dcregistry/ )

• Helpidentifyoverlaps,duplication,gaps andlimitstoadoption,>>encouragesnottoduplicateeffortsandtocollaboratetobothdevelopandusecommonstandards

Approachforbuildingmapofdatastandards

•Notduplicating,buildingonwhatexists(e.g.syncwithAgroPortal)

•Collaborativeeffort

•Broadcoverage

•Opendataangle

•Designedforgapanalysis (morelater)

Anybodycanaddorclaimastandard

Thesaurus CodelistOntology ISOspecification MessagingstandardTaxonomy

Agronomy Naturalresources Fisheries Valuechains

Allcontributorsacknowledged

Calltoactiontopartnersandexperts

Format APIs Mappings License

CategorizationofdatastandardsbycontentDomain-specific• Bysub-domain:Whichdomainclassification?Attemptedone

basedonFAO+USDAclassificationsHowfartogowithneighboringdisciplines?

• Bydatatype:alignmentwithGODAN“AgriculturalSectorPackage”fortheOpenDataCharter

Sub-domains• AgriculturalResearch,Technologyand

Engineering• Agro- Economics,Businessand

Industry• AnimalScienceandAnimalProducts• EducationandAgriculturalExtension• FarmsandFarmingSystems• FisheriesandAquaculture• FoodandHumanNutrition• ForestScienceandForestProducts• Government,AgriculturalLawand

Regulations• HealthandPathology• NaturalResources,Earthand

Environment• PlantScienceandPlantProducts• RuralandAgriculturalSociology

MapofstandardssofarVEST / AgroPortal MAP OF STANDARDSvest.agrisemantics.org

Numberofdatastandardsbydomain

Criteriaforgapanalysis

Toidentifygaps,needtoidentifyassessmentcriteria.Criteria developedbasedon:

• TheassessmentprocessusedbytheUKGovernment’sOpenStandardsBoard

• TheODIOpenDataCertificates criteria

Categoriesofassessment:Fitnesstopurpose Adoption Usability Openness

ExampleofadoptionassessmentcriteriaUsedinsoftware Isthisstandardusedinsoftwaretools(e.g.as

controlledvaluesforsomefields,orasexportformat,orasdatamodel)?

Yes,inmanytoolsthatareverypopular 5Yes,inafewtoolsthatareverymuchused 4Yes,inafewtools 3Yes,in1-2tools 2No 0Notclear- N/A 1

Usedindatasets Isthisstandardusedindatasets(e.g.asserializationformat,orasdatamodel/elementset,orascontrolledvaluesforsomecolumns/dimensions)?

Yes,inmanybymanyproviders 5Yes,inmany,byafewproviders 4Yes,inafew 3No 0Notclear- N/A 1

Endorsed Doesthestandardhaveastrongsupportfromdifferentinterestgroups?

Yes,verystrongly 3Yes,moderately 2No 0Notclear- N/A 1

Regulatory Isthestandardpublishedbyarecognizedstandardizationbodyorasagovernmentdirective?

Yes 3No 0Notclear- N/A

1

Long-term,sustainable

Isthemaintainingorganizationalong-standingandauthoritativebody?Isthemaintainercommittedtosustainandpreservethestandard?

Yes,highly 3Yes,reasonably 2No 0Notclear- N/A

1

Participatory,collaborative

Isparticipationinthecreationprocessofthestandardopentoallrelevantstakeholders?

Yes 3No 0Notclear- N/A 1

Exampleofusabilityassessmentcriteria

Versatile Isthestandardavailableindifferentformatsfordifferenttechnologies?(e.g.XML,JSON,RDF)?

Yes,manyformats 3Yes,2-3similarformats 2No 0Notclear- N/A 1

ServedbyAPIs ArethereAPIsandwebservicesthatallowapplicationstoworkwiththevocabulary?Chooseasmanyanswersasyouneed.

Yes,togetweb- oruser- friendlyresults3

Yes,tolookupterms/conceptsusingseveralparameters4

Yes,toautomaticallyannotatetextordata5

Yes,toperformcross-walksbetweenvocabularies6

Yes,toextract/lookupsubsetsofvocabularies7

No 0Notclear- N/A 1

Manageable Isthestandardmanagedinacollaborativeenvironment?Isitmanagedonaspecializedvocabularymanagementplatform?

Yes,onspecializedvocabularymanagementplatform3

Yes,onacollaborativeenvironment(e.g.Github)2

No 0Notclear- N/A 1

Exampleofopennessassessmentcriteria

Machine-readable

Isthestandardavailableinmachine-readableformats?(XML,CSV,RDF...)

Yes 3No 0Notclear- N/A 1

Meaningful Ifmachine-readable,isthestandardserializedusingtheappropriatevocabularylanguage(RDFs,OWL,SKOS,OBO)andsemanticallyappropriatevalues?

Yes 3No 0Notclear- N/A 1

Referenceable DoesituseURIsdereferenceableasURLsasidentifiersofclasses,propertiesandinstances?

Yes 3No 0Notclear- N/A 1

Linked IsthestandardavailableasLinkedData?I.e.serializedasRDFandabovealllinkingtoURIsinothervocabularies?

Yes 3No 0Notclear- N/A 1

Mapofstandards– topic1:weatherdata• Specificusecaseofweatherdatausedinfarmmanagementinformationsystems

• Coverage– Total of 65 weather and use-case related data standards

http://vest.agrisemantics.org/by-type-of-data/7623+7550+7626/7623/7550/7626

• Refined classification of weather and use-case data types

Weather / meteorological dataGeospatial data / objectsFarm management data

Weather / meteorological dataWeather observations (live or historical)Weather monitoring infrastructureClimate dataWeather forecasts

Farm management dataeBusiness data (inventory, sales, suppliers...)Crop management data from the farmCrop growth modelsFarm sensors infrastructureObserved field data from the farm

Examplesofdatastandardsrelevantfortopic1

Assessmentandgapanalysis• Weatherdatastandards• Specificusecaseofweatherdatausedinfarmmanagementinformationsystems

Assessmentmetadata+consultationswithexperts

Forgeospatialandweatherdata• BenSchaap(GODANSecretariat)• GiovanniL'Abate(CRAItaly)• SimonCox(CSIROAustralia,RDA)

Forweatherdataforfarmmanagement• SoonhoKim(IFPRI,ICASAstandards)• AndresFerreyra(AgGateway)• HugoBesemer(Wageningen UR)• FrancescoBenincasa(RDAWeatherIG,

BarcelonaSupercomputingCenter)• AllarddeWit(Alterra)• ChristopherBrewster(TNONetherlands)

Weatherdatastandards– Summary• Variety ofdatamodels,dataformatsandvocabulariesthatareusedto

exchangethesedata• Someolderstandardsarestillverymuchused,eitherforlegacy andcompliance

reasons(likeBUFRorGRIB)orbecauseoflong-termpracticeinresearch(NetCDF)

• Standardizationbodieshaveworkedongeospatialandobservationsmodelsandrelatedschemas(ISO/OGC,especiallytheISO19100series),startingtobeusedalsobythemeteorologicalcommunity(CSML, METCE,IWXXMschemas)

• RecentlyAPI-basedweatherdataservicesstartedservingdataondemand andinapplication-friendlyformatslikeJson anduser-friendlyformatslikeCSV

• Workonvariablenamingconventions• FMIS:subsetofweatherdatavariables;agreementwithweatherdata

providers;workonvariablenaming

Gapanalysisonweatherdatastandardsa) Thebiggestchallengeswithweatherdataarerelatedtoissuesofdataavailability,

discoverability,quality,coverageanddocumentation.

b) Datastandardsarenotwelldocumented,differencesbetweenoverlappingstandardsarenotclarified.

c) Bothforweatherdataandforfarmmanagementdata,standardizationofvariablenames acrossthedifferentcommunitiesandevenwithinthesamecommunityisanissue;moreingeneral,therearemanydifferentcodelists usedbydifferentauthorities,withlimitedalignment.

d) Fewdatastandardspublishedasmachine-readableandlinkedvocabularies.

à Intermediariesstillhavetodomostoftheworkconverting,processing,re-purposingthedatabetweenthedifferentstepsinthedatavaluechain.

Keyrecommendationsforbetteruseandusabilityofweatherdatastandards

• Addressdiscoveryissuesrelatingtoweatherdata(improvinguseofdiscoverymetadata tohelpcatalogueanddescribedata)• Improvethedocumentationandself-description ofexistingdatastandards(creatingdeveloperdocumentation;publishingexistingvocabulariesinnewways);offercommunitysupport,Q&Aservices• Identify2-3keycodelists thatshouldbepublishedinmorelinkable,versatileformats;linkkeycodelistsandpublishexistingalignments;providewebservicesforcross-walks.

Nextstepsondatastandards

• Datapublicationonlinehelpdesk• Q&Aservice;Facilitateauthors/expertsonstandardstoregisterandprovidetheirhelp• Facilitatedatapublisherstoregisterandaskforhelpfromtheregistered

authors/expertsonstandards• Sameiterationfornutritiondataandlanddataasforweatherdata

• Usecases>surveyofstandards>gapanalysis>recommendations• Specificationsforstandardinteroperabilityservices

• Basedontherecommendations,forall3typesofdata• Interoperabilityspecificationsthatmakestandards(forall3typesofdata)interoperable

andreusable• Standardinteroperabilityservicesforpilotinterventions

• Basedonrecommendationsandspecificationsofservices• Pilotsforprovidinginteroperabilityservicesforstandardsidentifiedtousecasesfromthe

thematictopics

Usefullinks• GODAN:http://godan.info• GODANActionmapofstandards:http://vest.agrisemantics.org• AgroPortal:http://agroportal.lirmm.fr• TheassessmentprocessusedbytheUKGovernment’sOpenStandardsBoard:Corequestions:https://standards.data.gov.uk/core-assessment-questions

• TheODIOpenDataCertificatescriteria:https://certificates.theodi.org/en/• AgSectorPackage:http://agpack.info/• DCKOSTypesvocabulary:http://wiki.dublincore.org/index.php/NKOS_Vocabularies

• Blogpostongapanalysisandrecommendations:http://www.gfar.net/news/gfar-and-odi-lead-work-gap-analysis-and-recommendations-weather-data-standards

Gapanalysisandrecommendationstobepublishedsoon.

Weatherdatastandards:survey,gapanalysisandrecommendations

Thankyou

valeria.pesce@fao.org