cobweb project deliverable report system... · extended by third-parties and can be installed and...
TRANSCRIPT
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
COBWEB PROJECT Deliverable Report
Grant Agreement number: 308513
Project acronym: COBWEB
Project title: Citizen Observatory Web
Funding Scheme: FP7 ENV.2012.6.5-1
Project website address: www.cobwebproject.eu
Deliverable: D4.7 System documentation
Version: 1.0
Delivery date: August 2016 (M46)
Lead partner: University of Nottingham (UNOTT)
Coordinating Editor: Julian Rosser
Editing contributors: Michael Koutroumpas (UEDIN), Conor
Muldoon (UCD), Stefan Wiemann (TUD)
and Didier G. Leibovici (UNOTT)
Reviewer: Conor Muldoon
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
1
Table of Contents
1. Document history .......................................................................................... 2
1.1 Versions........................................................................................................... 2
1.2 Document Release Schedule ........................................................................... 2
2. Executive Summary ...................................................................................... 3
3. COBWEB Mobile Client software ................................................................. 4
3.1 COBWEB Fieldtrip-Survey-Designer ................................................................ 4 3.1.1 Technical Overview ......................................................................................................... 4 3.1.2 Usage overview............................................................................................................... 4 3.1.3 Installation and source code: .......................................................................................... 5
3.2 COBWEB FieldTripOpen framework ................................................................ 6 3.2.1 Technical Overview ......................................................................................................... 6 3.2.2 Usage overview............................................................................................................... 7 3.2.3 Installation and source code: .......................................................................................... 8
3.3 Flooding App ................................................................................................... 9 3.3.1 Technical Overview ......................................................................................................... 9 3.3.2 Usage Overview .............................................................................................................. 9 3.3.3 Installation and source code: ........................................................................................ 10
4. Quality Assurance and Conflation ............................................................. 10
4.1 Quality Assurance .......................................................................................... 10 4.1.1 Technical Overview ....................................................................................................... 10 4.1.2 Usage overview............................................................................................................. 11 4.1.3 Installation and source code: ........................................................................................ 13
4.2 Conflation software ........................................................................................ 14 4.2.1 Technical Overview ....................................................................................................... 14 4.2.1.1 Usage overview............................................................................................................. 16 4.2.2 Installation and source code: ........................................................................................ 16
5. Sensor Observations Middleware .............................................................. 17
5.1 OpenSensorHub Integration .......................................................................... 17 5.1.1 Technical Overview ....................................................................................................... 17 5.1.2 Usage overview............................................................................................................. 18 5.1.3 Installation and source code: ........................................................................................ 19
Appendix A: Technology Readiness Levels .............................................................. 20
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
2
1. Document history
1.1 Versions
Version Date Who What
0.1 8 July
2016
UNOTT Initial Document structure
0.2 22 July
2016
UNOTT/UCD/ED/TUD Addition of installation and
source details
0.3 28 July
2016
UNOTT/UCD/ED/TUD Addition of usage details
0.4 2 August
2016
UCD Internal QA and STC review
by Conor Muldoon
0.5 3 August
2016
UNOTT Corrections following QA and
STC review
1.0 17 August
2016
Chris Higgins/UEDIN Final review prior to
submission
1.2 Document Release Schedule
This deliverable (D4.7) relates to the software documentation of the mobile client,
middleware, validation and conflation services.
Date (Project Month) Activity
M46 Software Documentation (D4.7)
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
3
2. Executive Summary
This deliverable (D4.7) relates to the documentation of the mobile client, quality
assurance, conflation and sensor data observation middleware. Though the
deliverable for this work is primarily software, this document is provided to detail the
different components and their technical basis. The aim is that the software is
documented to a sufficient degree at the systems-level such that it might be
extended by third-parties and can be installed and operated by users and
developers.
As an overview, the major software components are:
• The COBWEB Fieldtrip-Survey-Designer: a web-based form designer for use with
mobile/smartphones. It allows a project coordinator to create bespoke data capture
interfaces for citizen science/crowd source applications.
• The mobile application Fieldtrip-COBWEB which leverages the COBWEB survey
designer to provide the latest data-capture functionality on Android and iOS
platforms.
• PCAPI Storage-Middleware: a server based data synchronisation tool.
Documentation on PCAPI can be found in D3.6 (COBWEB User Guide &
Documentation).
• COBWEB-Flooding-App: a native Android client developed specifically for the
flooding thematic pilot case study area.
• A quality workflow authoring tool COBWEB QA-workflow-AT or QA-wAT, enabling
bespoke construction of quality assurance workflows.
• A Web Processing Service (OGC WPS) COBWEB QA-WPS enabling data
captured to pass through a quality assurance workflow composed of quality controls;
themselves part of the WPS repository
• A Web Processing Service COBWEB wps-conflation enabling data conflation
services.
• Quality Controls (QCs) for Quality Assurance (QA) on the mobile without internet
connectivity have been developed to work as java plug-ins that access the QA-
Mobile-library.
• Several plug-ins have been developed within the context of the flooding co-
design. The plugins are not specific to flooding surveys and can be used for QA,
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
4
capturing specific data either from sensors on the phone or via environmental
embedded sensors.
• A middleware Sensor Observation Service (SOS) allowing access to registered
environmental sensors supporting conflation and QA for additional dataset
processing.
Note:
It is important to mention that the system software we are describing is, by virtue of
COBWEB being a research project, overall still a prototype with various TRLs
(Technical Readiness Levels – see Appendix A) across the different components
(between TRL4 and TRL8).
3. COBWEB Mobile Client software
3.1 COBWEB Fieldtrip-Survey-Designer
3.1.1 Technical Overview
The COBWEB Fieldtrip-Survey-Designer is a Web browser-based thin client
application written in HTML5 for creating bespoke mobile surveys which will be used
by the mobile app during a survey. This authoring step is carried out first by the
project coordinator to create a custom interface. This interface can then be packaged
and downloaded onto a phone running a FieldtripOpen based mobile application,
discussed in further detail below in section 3.2.
3.1.2 Usage overview
The COBWEB Survey-Designer is used as a browser application and provides a
drag and drop interface (see Figure 1) for creating data-capture screens for the
mobile application. The interface provides a canvas area for inserting various form
elements which include:
- text boxes,
- range selection widgets,
- point alerts,
- optional field capture (see below),
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
5
- options on location, image and sound capture
- client side field prompting (including picture prompts) and QA validation.
A video demonstrating usage of both the COBWEB mobile app (described in further
detail below) and the Survey Designer is available providing an overview of the
operation of these tools:
http://cobwebsvc.edina.ac.uk/apps/cftopen.mp4
Further instructions on usage can be found at:
https://github.com/edina/survey-designer/wiki
Figure 1 Screenshot of the COBWEB Authoring Tool
3.1.3 Installation and source code:
Generic Application Designer (aka Survey Designer)
Source code:
https://github.com/edina/survey-designer
Installation and technical documentation:
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
6
https://github.com/edina/survey-designer/wiki
3.2 COBWEB FieldTripOpen framework
3.2.1 Technical Overview
The COBWEB FieldtripOpen framework is an open source mobile app toolkit which
is heavily based on plugins to maximise flexibility and to allow the construction of all
kinds of diverse data capture projects. The plugins are loosely coupled to be easily
used as independent building blocks by third-party developers. Currently the
following plug-ins have been developed for COBWEB:
custom map overlays
GPS tracking
mobile QA
conditional field capture
synchronisation with the COBWEB storage middleware
offline mapping
Under the hood, the mobile application is built on the open source Cordova
technology which provides a portability layer on top of major mobile platforms
including Android and iOS. This enables portability as it is based on HTML and
JavaScript, which is usable across operating systems. However, the Android plugins
developed for this work that deliver specific native functionality (e.g. Bluetooth
plugin) would need to be re-written for complete deployment on iOS.
A unique feature of the mobile app is its ability to leverage the COBWEB Mobile QA
library to perform quality checks instantly on the mobile without relying on internet
connectivity or the availability of server-side functionality. This is particularly useful
when pre-emptive data checks are needed to provide instant feedback to the user
and allow for corrections. As an example, this method is currently used for surveys
that require submission of photographs above a certain sharpness level. Each time
the user takes a photograph, a “sharpness test” is executed and if the image is too
blurry to satisfy the survey requirements the user is warned and given the
opportunity to try again before submitting the image. Another example is the “line of
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
7
sight” functionality which allows calculating the location of the observation rather
than the observer’s when a photograph is taken.
3.2.2 Usage overview
The COBWEB mobile application is built upon the COBWEB Fieldtrip Open
framework which takes advantage of Cordova technology with versions made
available for download from Google Play and with the potential to release in the
Apple App Store. Once compiled and installed on a device, the application can be
used to cache offline maps and capture observations. Once observations have been
captured and the device is online, the data may be synchronised with the server.
On the main screen a user can either choose to login and then join a private survey
or to immediately browse existing public surveys and pick the ones in the user's
interest. Each survey is essentially a data capture form designed with the COBWEB
Survey Designer and can have an entirely unique content and set of data capture
widgets. The COBWEB app implements all security protocols for user and access
management referenced in deliverable D5.4 (User Management and Policies to run
COBWEB for evaluation in the initial Biosphere Reserves). This is not mandatory
though as one can anonymously participate on public surveys without logging in.
Once the survey is loaded the user can start capturing data and preview existing
observations.
A common requirement for all observations is the association of at least one point
coordinate. Optionally, one can also add complex polygon captures and/or position
tracking depending on the survey. Once all observations are captured they reside on
the phone's storage until the user chooses to upload them. This step is the only step
after joining a survey that requires an internet connection.
A non-technical user can install the app by downloading a pre-compiled version
available at:
http://cobwebsvc.edina.ac.uk/apps/COBWEB.apk
Further step-by-step usage instructions on usage can be found at COBWEB
deliverable D3.6 page 21.
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
8
Figure 2 COBWEB app form data input (left) and map screen (right)
3.2.3 Installation and source code:
Generic COBWEB Application Framework (aka Fieldtrip Open)
Source code:
https://github.com/edina/fieldtrip-open
Installation and technical documentation:
https://github.com/edina/fieldtrip-open/wiki
Pre-compiled COBWEB application:
http://cobwebsvc.edina.ac.uk/apps/COBWEB.apk
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
9
3.3 Flooding App
3.3.1 Technical Overview
In parallel to the development of the COBWEB app described above, a native mobile
application for Android was implemented for the flooding pilot case study area. This
application enables capture of data with particular relevance to flood events. For
example, features such as annotation of photographs and grouping of observations
to denote flood areas as well as points, and live sharing of observations between
devices using wireless hotspot functionality.
Figure 3 Screenshot of the COBWEB Android Flooding app
3.3.2 Usage Overview
Full instructions on usage can be found at:
http://ucd.cobwebproject.eu/flooding_application/Flooding%20Application%20User%
20Guide.pdf
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
10
3.3.3 Installation and source code:
Source code:
https://github.com/cobweb-eu/flood-app
Installation and technical documentation:
http://ucd.cobwebproject.eu/floodapp/COBWEBFloodingApp.apk
https://github.com/cobweb-eu/flood-app
4. Quality Assurance and Conflation
4.1 Quality Assurance
4.1.1 Technical Overview
The COBWEB Quality Assurance components include a processing service of
individual quality control tests (QA-WPS) and an authoring tool for creating workflows
using the quality control tests (QA-workflow-AT). The QA-WPS is implemented as an
OGC Web Processing Service thereby providing an open and standardised interface
for executing quality control tests (grouped within 7 pillars, see D4.6). Adoption of the
WPS standard also allows simple integration of the Conflation tools for data fusion
within the system. The QA-workflow-AT uses Business Process Modelling Notation
(BPMN) thereby providing an open and standardised definition for the workflow
document.
The system uses:
1. 52North Web Processing Service (WPS) with the GeoTools extensions
enabled. The WPS application supports creation of Web Processing Services
in Java and R; quality control processes in this work are implemented in both
languages.
2. JBPM Workflow engine by JBOSS. This component constitutes both the user
interface and workflow engine and has been customised to enable execution
of WPS. Workflows (defined as BPMN documents) may be created using
either a web-editor or Eclipse development environment plugin. In order to
utilise the QA-WPS, the processes must be registered within the workflow
JBPM engine. Once this is done, the workflows may be constructed with
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
11
relevant process parameters and input and output data defined as Web
Feature Service or Web Coverage Service endpoints.
The components chosen enable the system to be implemented as loosely-coupled
distributed system. Figure 4 illustrates the relation between each of the components.
QA/QC WPS
(52North)
WFS / WCS
(GeoServer)
Web editor
user
Eclipse plugin
user
JPBM Workflow
Editor / Engine
Conflation WPS
(52North)
Figure 4 Architectural overview of the COBWEB QA system components
4.1.2 Usage overview
The creation of Quality Assurance workflows is achieved through use of the JBPM
Eclipse plugin or web-editor (found within Kie Workbench). Once the Quality Control
processes are registered within either of these environments using .WID definitions,
as detailed in the installation details, the workflow editor may drag and drop
processes into the canvas area. Double-clicking a process reveals a properties
window where data and parameters may be mapped into the inputs of the Quality
Control process. Output and temporary variables for mapping into further processes
may also be undertaken.
To run a process, the BPMN editor document must be compiled and is then
executed by the BPMN engine. In the Eclipse environment this can be handled by
defining and compiling a main Java method which sets up a service for running the
workflow, registers each of the processes used in the workflow as work items, and
starts the workflow. In the web-editor environment, the workflow is compiled from the
drag and drop environment and executed from the “Process Definitions” window.
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
12
Figure 5 Web-based workflow editor environment and process properties window
Figure 6 List of deployed process definitions
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
13
Further instructions on usage can be found at:
https://github.com/cobweb-eu/workflow-at/wiki/
4.1.3 Installation and source code:
Quality Assurance Web Processing Service (QA-WPS)
Source code:
https://github.com/cobweb-eu/QA_wps_processes
Installation and technical documentation:
https://github.com/cobweb-eu/QA_wps_processes/blob/master/readme.md
Quality Assurance Authoring tool (QA-workflow-AT):
Source code:
https://github.com/cobweb-eu/workflow-at
Installation and technical documentation:
https://github.com/cobweb-eu/workflow-at/wiki
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
14
4.2 Conflation software
4.2.1 Technical Overview
The conflation software implementation can be used to relate and combine spatial
data based on certain spatial, temporal and/or thematic characteristics. It can be
used as a stand-alone component or as a part of QA. Conflation functionality is
offered via the standardized OGC WPS service interface and can thus be accessed
by clients or orchestration engines capable of invoking WPS instances.
The implementation consists of two components: 1) a Java library that provides the
conflation functionality and 2) a WPS wrapper for the Java library, which is based on
the 52°North WPS framework. In addition to the processes already offered by the
WPS framework, the following processes are implemented:
Data parser
o GML – reads GML from file or URL; based on GeoTools library
o GBIF – reads GBIF observation data
o GridCoverage – reads raster image; based on GeoTools library
o OpenStreetMap (OSM) XML – reads OSM XML data from file or URL
o RDF Turtle – reads RDF relation data with Turtle encoding
Data enhancement & harmonization
o Line intersection – intersects linear network to avoid topological
inconsistency
o Multi-to-Single-part – Converts Multi- to Single-part features
o CRS reproject – reproject coordinate reference system for input
feature
Similarity measurement
o Angle Difference – angle difference between linear geometries
o Bounding Box Distance – intersection between feature bounding
boxes
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
15
o Damerau Levenshtein Distance – Damerau Levenshtein string
distance between two attribute values
o Geometry Distance – distance between input geometries
o Geometry Overlap – overlap between feature geometries
o Hausdorff Distance – Hausdorff distance between feature geometries
o Length Difference – length difference between linear feature
geometries
o Length in Polygon – relative length of geometry in polygon
o Sinuosity Difference – sinuosity difference between linear feature
geometries
o Topology Relation – DE9-IM code for intersection between feature
geometries
o Zonal Statistics – zonal statistics between source features and target
coverage
Feature mapping and resolving
o Best Correspondence Mapping – Filters a set of feature relations in
order to create best correspondences
o Attribute Transfer – Transfers selected attributes between related
features
o Missing Feature Transfer – Transfers missing features from the target
to the reference dataset
Data generator
o RDF Turtle – Feature relations encoded in RDF using the RDF Turtle
encoding
o RDF Triplestore – Feature relations in a triple store; uses SPARQL
Update queries
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
16
4.2.1.1 Usage overview
The developed conflation functionality can be accessed in three ways:
Java API: The developed Java framework specifies interfaces for other Java
programs to access and extend the implemented functionality. Each
operation implements the de.tudresden.gis.fusion.operation.IOperation
interface, which defines an execute operation to invoke the process, and a
profile operation that returns the process description profile.
JBPM Workflow engine: If the conflation functionality is wrapped behind a
WPS interface, it can be accessed and invoked by the COBWEB QA
workflow engine described in Section 4.1.
Fusion Web Client: for specialized conflation workflows, the custom Web
Client can be used to chain WPS processes directly on the Web. The
compiled workflow is encoded as BPMN and sent to an orchestration WPS
process for invocation. Once the workflow is finished, the result is directly
returned to the Web Client for further use.
Figure 7 Conflation workflow example using the Fusion Web Client
4.2.2 Installation and source code:
Java library offering functionality for spatial data fusion
Source code & installation guide:
https://github.com/GeoinformationSystems/SpatialDataFusion
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
17
Service implementation of the fusion process. Wraps the Java library behind a OGC
WPS interface
Source code & installation guide: https://github.com/cobweb-eu/wps-fusion
5. Sensor Observations Middleware
5.1 OpenSensorHub Integration
5.1.1 Technical Overview
OpenSensorHub represents a middleware platform for the development of OGC
compliant services that deliver data from sensors. Specifically, OpenSensorHub
supports the development of services that are consistent with the Sensor Web
Enablement (SWE) standards. The goal of the SWE standards is to enable sensors
to be discoverable, accessible, and usable through web based interfaces. The SWE
standards include the Sensor Observation Service (SOS), the Sensor Planning
Service (SPS), Observations and Measurements (O&M), the Sensor Model
Language (SensorML), and the SensorThings API. In the context of COBWEB, of
particular interest is SensorML and SOS. SensorML can be used to describe a wide
variety of sensor types including both stationary and mobile platforms. Models are
provided to describe sensors and measurement processes along with an XML
encoding. The SOS standard defines how data should be delivered from a web
service to enable the real time querying of sensor data in addition to sensor time
series data. OpenSensorHub enables SOS compliant data to be accessed either via
HTTP or WebSockets, enabling full duplex communication over a single TCP
connection. With WebSockets, sensor data can be streamed to web clients as it
arrives in real time.
In order to make data available through the use of OpenSensorHub, drivers must be
developed and launched by the platform. Documentation, from a coding perspective,
in relation to the development of OpenSensorHub drivers is available at:
http://docs.opensensorhub.org/dev/dev-guide/
In order to illustrate how OpenSensorHub drivers can be used in the context of
COBWEB, we shall discuss a driver that has been developed to enable the delivery
of data from sensors deployed in a peat bog in the Dyfi biosphere. This driver was
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
18
developed in the context of the co-design projects. The Royal Society for the
Protection of Birds (RSPB) is monitoring data coming from the bog, which is related
to a number of different phenomena, such as the water depth level, barometric
pressure, and temperature. This data is useful in measuring the impact of efforts at
reclaiming the natural habitat for birdlife. Specifically, a number of In-Situ sensors
have been deployed within a peat bog and these periodically record the various
sensor reading. Once per day, the sensors activate a 3G connection and transmit
the collected data to a FTP server located in Environment Systems. This data is
periodically harvested by services operating behind firewall, which do not enable
incoming FTP connections.
Once data is harvested from the FTP server, it is stored in a database locally.
Subsequently, the database will be accessed by the peat bog OpenSensorHub
driver. Rather than incorporating all of the data at once, which could be a substantial
amount and overload the system, the peatbog driver buffers sensor readings and
subsequently adds data in a periodic manner. The data can subsequently be
accessed through HTTP or WebSocket requests. When using WebSockets, typically,
an Apache redirect will be required to access the web container that hosts the
OpenSensorHub instance. In order for a developer to create a driver, for different
types of sensors, it is recommended that a pre-existing example be modified.
5.1.2 Usage overview
To use the OpenSensorHub peat bog functionality, a cron job must be created in
Linux to periodically execute the Peat Bog harvester, which downloads files from the
FTP server that the sensors send their data to, parses the files, and exports the
content to a database. An instance of OpenSensorHub must also be started. The
OpenSensorHub instance needs to be configured to execute the Peat Bog drivers by
modifying the OpenSensorHub config.json file and the driver jar files need to be
placed in the lib folder. OpenSensorHub contains an embedded Jetty web
application server and once operational, the local OpensSensorHub administration
console web application can be accessed. When viewing the administration console,
the peat bog sensor drivers will be enabled. Subsequently, data can be accessed
from the sensors by making standard SOS requests to the server.
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
19
Figure 8 OpenSensorHub GUI
Further instructions on usage can be found at:
http://ucd.cobwebproject.eu/osh-peatbog-driver/INSTALL.txt
http://docs.opensensorhub.org/
5.1.3 Installation and source code:
Source code:
http://ucd.cobwebproject.eu/osh-peatbog-driver/osh_pb0.tar.gz
http://ucd.cobwebproject.eu/osh-peatbog-driver/osh_pb1.tar.gz
http://ucd.cobwebproject.eu/osh-peatbog-driver/pb_harvest.tar.gz
http://ucd.cobwebproject.eu/osh-peatbog-driver/harvester.tar.gz
This project has received funding from the European Union’s
Seventh Programme for research, technological development
and demonstration under grant agreement No 308513
20
Installation and technical documentation:
http://ucd.cobwebproject.eu/osh-peatbog-driver/INSTALL.txt
Appendix A: Technology Readiness Levels
TRL 1 – basic principles observed
TRL 2 – technology concept formulated
TRL 3 – experimental proof of concept
TRL 4 – technology validated in lab
TRL 5 – technology validated in relevant environment (industrially relevant environment
in the case of key enabling technologies)
TRL 6 – technology demonstrated in relevant environment (industrially relevant
environment in the case of key enabling technologies)
TRL 7 – system prototype demonstration in operational environment
TRL 8 – system complete and qualified
TRL 9 – actual system proven in operational environment (competitive manufacturing in
the case of key enabling technologies; or in space)