d.6.2.3 fi-ware sw release
TRANSCRIPT
Private Public Partnership Project (PPP)
Large-scale Integrated Project (IP)
D.11.1.1: FIWARE Technical Whitepaper
Project acronym: FI-Core
Project full title: Future Internet - Core
Contract No.: 632893
Strategic Objective: FI.ICT-2011.1.7 Technology foundation: Future Internet Core Platform
Project Document Number: ICT-2013-FI-632893-WP11-D.11.1.1
Project Document Date: 2015-02-03
Deliverable Type and Security: PU
Author: FIWARE Chapter Leaders & Chapter Architects
Contributors: GEi owners
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 2
1 Foreword ............................................................................................................................................... 4
2 What is FIWARE? ................................................................................................................................... 5
3 Wanna start quickly using FIWARE? ..................................................................................................... 6
4 Quick FIWARE Tour Guide for developers ............................................................................................ 7
4.1 Development of context-aware applications ............................................................................... 7
4.1.1 How to update and query context information .................................................................... 8
4.1.2 How to connect to external context providers ................................................................... 11
4.1.3 How to subscribe to changes on context information ........................................................ 12
4.1.4 How to perform geo-location queries ................................................................................ 13
4.2 Connection to the Internet of Things .......................................................................................... 14
4.2.1 How to read measures captured from IoT devices ............................................................. 15
4.2.2 How to act upon IoT devices ............................................................................................... 19
4.3 Real-time processing of context events ...................................................................................... 19
4.3.1 How to define a CEP application ......................................................................................... 21
4.3.2 How to administrate the CEP .............................................................................................. 22
4.3.3 How to send input events to the CEP ................................................................................. 22
4.4 Handling authorization and access control to APIs .................................................................... 23
4.4.1 Keyrock GE .......................................................................................................................... 24
4.4.2 Access Control GE ............................................................................................................... 24
4.4.3 How to create your identity in FIWARE .............................................................................. 25
4.4.4 How to implement OAuth2 in your applications ................................................................ 26
4.4.5 How to send requests to a FIWARE GE ............................................................................... 28
4.4.6 How to secure your backend service .................................................................................. 29
4.4.7 How to manage Access Control in FIWARE ......................................................................... 31
4.4.8 References .......................................................................................................................... 33
4.5 Publishing (Open) Data in FIWARE .............................................................................................. 33
4.5.1 How to Publish (Open) Datasets in CKAN ........................................................................... 34
4.5.2 How to Publish Context Information as (Open) Data in CKAN ........................................... 37
4.5.3 How to Offer Datasets (including Context Information) through the WStore ................... 39
4.6 Big Data analysis of historic context information ....................................................................... 51
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 3
4.7 Creating application dashboards ................................................................................................ 55
4.7.1 How to find widgets and other components for your dashboard ...................................... 57
4.7.2 How to create your application dashboard from these building blocks ............................. 59
4.7.3 How to interconnect widgets and connect them to backend resources ............................ 59
4.7.4 How to share, sell or make available your new dashboard ................................................ 61
4.7.5 How to develop new widgets and operators ...................................................................... 62
4.7.6 How to make AJAX requests to external services ............................................................... 63
4.7.7 How to receive events ........................................................................................................ 64
4.7.8 How to send events ............................................................................................................ 65
4.7.9 How to use other FIWARE GEs from your widgets and operators ..................................... 67
4.8 Real-time processing of media streams ............................................................................... 70
4.8.1 What’s WebRTC and what’s a media server ....................................................................... 72
4.8.2 What’s the architecture of a Kurento enabled application ................................................ 73
4.8.3 How to develop a Kurento enabled application ................................................................. 75
4.9 Providing an advanced User Experience (UX) ............................................................................. 76
4.9.1 3D-UI: Interactive 3D graphics and Augmented Reality in any Web browser .................... 77
4.9.2 XML3D: Interactive 3D graphics and Augmented Reality via DOM extensions .................. 79
4.9.3 3D graphics on the Web with Three.js ................................................................................ 80
4.9.4 Multiuser 3D with WebTundra (Games, Virtual Worlds, Collaborative applications) ........ 80
4.10 Hosting your application on a FIWARE Cloud ............................................................................. 81
4.10.1 How to provision and manage your virtual infrastructure on FIWARE Cloud .................... 82
4.10.2 How to use Object Storage capabilities of FIWARE Cloud .................................................. 83
4.10.3 How to use Blueprints to manage complex applications hosted on FIWARE Cloud .......... 83
4.10.4 How to invoke FIWARE Cloud capabilities via an API ......................................................... 84
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 4
1 Foreword
This document provides a comprehensive overview of what FIWARE offers to developers
and software architects who design and implement Future Internet applications.
Contents of this document translate into contents of the web pages on the FIWARE website
(http://fiware.org) that guide developers in their journey learning how to use FIWARE.
The first two sections (“What is FIWARE?” and “Wanna start quickly using FIWARE?”) are intended
to capture the attention of developers and software architects by providing a very short and
concise summary of what FIWARE brings.
The Quick FIWARE Tour Guide for developers section brings useful information that will help
developers to learn how they can use FIWARE in order to incorporate certain functions in
their applications. It is aimed to work as the "suggested routes" in a given City Tour Guide,
where a tourist learns what he can do if he wants to do <X>. Therefore, sections of this
tour guide will refer to functionality developers may wish to incorporate in their
applications. Titles of these sections should trigger the reaction "hi, yes I want to
incorporate this functionality in my application, let me check how it could be done using
FIWARE". In the description of the main body of each of these sections, there will be links
to the entries in the FIWARE Catalogue linked to FIWARE GEris. This way, developers will
end landing on the FIWARE Catalogue but knowing what is the main purpose of the entries
in the catalogue they are visiting. Each of the sections in which the Tour Guide will be
structured will end with a list of links to how-to sections where more detailed info will be
provided. Each of the how-to sections will map into separate nested web pages in the
fiware.org website. However, the reader should not need to read the “how-to”s to
understand what is described in the main body of a given subsection of the Quick FIWARE
Tour Guide for developers section.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 5
2 What is FIWARE?
The FIWARE platform provides enhanced OpenStack-based cloud hosting capabilities
plus a rich library of added-value functions offered “as a Service”. Components of the
FIWARE platform, referred as Generic Enablers (GEs), provide open standard APIs that
make it easier to connect to the Internet of Things, process data and media in real-time at
large scale, perform Big Data analysis or incorporate advanced features for the interaction
with users.
Each FIWARE GE provides APIs that are public and royalty free and it comes with an open
source reference implementation. This allows that multiple commercial FIWARE providers
emerge faster in the market, all supporting the same set of APIs. Thanks to that, you will be
able to decide who will operate the environment where your application, and more
importantly your data, will be hosted. As a result, FIWARE becomes an open alternative
to existing proprietary Internet platforms.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 6
3 Wanna start quickly using FIWARE?
If you are an application developer, you can gradually incorporate FIWARE GEs in the
architecture of your application to transform it into a truly smart application.
As an example, you may first incorporate the Context Broker GE in order to gather and
manage context information so that your application exhibits a context-aware
behaviour. Your application can incorporate FIWARE GEs supporting connection to the
Internet of Things in order to gather context information from sensors or actuate
devices. You can add new FIWARE GEs helping you to process context information in real
time or perform Big Data analysis of historic data gathered about context. Real-time
processing of media contents or the incorporation of advanced user interface functions are
also examples of the kind of advanced features your application can support by means of
using FIWARE GEs. All the above GEs, as well as your own software components, can be
hosted on the FIWARE Cloud, automating their provisioning and lifecycle.
Discover all what FIWARE can do for you by following the Quick FIWARE Tour Guide for
developers.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 7
4 Quick FIWARE Tour Guide for developers
4.1 Development of context-aware applications
If you want your application to be “smart”, you have to first make it “aware”. FIWARE
provides you with means to produce, gather, publish and consume context information at
large scale and exploit it to transform your application into a truly smart application.
Context information is represented through values assigned to attributes that characterize
those entities relevant to your application. The Context Broker GE (open source reference
implementation: Orion) is able to handle context information at large scale and implements
the standard OMA NGSI-9/10 APIs (OMA NGSI APIs for short) which enable your
application to query on context information or subscribe to changes in context information
that will be received through notifications. It also enables your application or other
applications to modify the context information. If you are familiar with SNMP, the popular
standard for management of equipment, OMA NGSI APIs is said to work as “the SNMP of
Context”.
The Context Broker GE
Context information may come from many different sources:
Already existing systems Users, through mobile apps Sensor networks
One of the most important features of the Context Broker GE is that it allows to model and
gain access to context information in a way that is independent from the source of that
information. As an example, your application may need to be aware about “Places”
(identified by their specific geographical coordinates) and the “temperature” in them. The
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 8
way in which the temperature of a given place is obtained may vary from place to
place. Thus, the temperature of a given street may be measured through temperature
sensors deployed in that street, while in another street it may be obtained through
temperature sensors deployed on buses that circulate through the street and even in some
other streets may be obtained through users who report the temperature using their
smartphones. Using the OMA NGSI API exported by the Context Broker GE, the way in
which your application will be able to query the temperature of places, or subscribe to
changes on temperatures of places will be the same no matter which is the source of a
temperature and it will not vary if the source of the temperature for a given place changes
over time (e.g., the temperature of a given street turns to be measured through temperature
sensors deployed on the streets rather than through buses equipped with sensors).
If you are interested in more details about how to make your application a context-aware
application, check out:
How to update and query context information
How to connect to external context providers
How to subscribe to changes on context information
How to perform geo-location queries
4.1.1 How to update and query context information
Processes running as part of your application architecture that update context information
using operations that the Context Broker GE exports, are said to play a Context Producer
role. As an example, let’s consider an application for rating restaurants (let’s call it
NiceEating). The client part of that application running on the smartphone of users would
play the role of Context Producer, enabling them to rate restaurants.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 9
On the other hand, processes running as part of your application architecture that query
context information using operations that the Context Broker GE exports are said to play a
Context Consumer role. Continuing with our NiceEating app, the mobile application
running on the smartphone of users and enabling them to query for the rating of restaurants
would play the role of Context Consumer. Note that a given part of your application may
play both the Context Producer and Context Consumer roles. This would be the case of the
mobile client of the NiceEating app enabling end users to rate, and query about rates of,
restaurants.
Entities that would be relevant to the NiceEating application are of type Restaurant, Client
and Rating. For example, when a given user scores a restaurant (e.g. in a scale from 0 to 5,
“Client1234” scores “4” for the “LeBistro” restaurant) the smartphone application plays the
Context Producer role creating a Rating entity in the following way:
POST <cb_host>:<cb_port>/v1/contextEntities/type/Rating/id/LeBistro::Client1234
{
"attributes" : [
{
"name" : "score",
"type" : "integer",
"value" : "4"
}
]
}
Each time a new Rating entity is created, the average rating for the corresponding
restaurant is recalculated by the application backend which (playing also the role of Context
Producer) updates the Restaurant entity accordingly:
PUT <cb_host>:<cb_port>/v1/contextEntities/type/Restaurant/id/LeBistro
{
"attributes" : [
{
"name" : "average_scoring",
"type" : "float",
"value" : "4.2"
} ]
}
Finally, the user can get the information of a given Restaurant using the smartphone
application. In that case the application works as Context Consumer, querying the
Restaurant entity in the following way:
GET <cb_host>:<cb_port>/v1/contextEntities/type/Restaurant/id/LeBistro
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 10
getting a response such as the following one:
{
"contextElement": {
"attributes": [
{
"name": "name",
"type": "string",
"value": "Le Bistro"
},
{
"name": "average_scoring",
"type": "float",
"value": "4.2"
},
{
"name": "location",
"type": "location",
"value": "40.419697, -3.691281",
"metadatas": [
{
"name": "location",
"type": "string",
"value": "WSG84"
}
},
{
"name": "postal_address",
"type": "string",
"value": "Calle Alcala 57, 28028 Madrid, Spain"
},
{
"name": "cousine_type",
"type": "string",
"value": "french"
}
],
"id": "LeBistro",
"isPattern": "false",
"type": "Restaurant"
},
"statusCode": {
"code": "200",
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 11
"reasonPhrase": "OK"
}
}
4.1.2 How to connect to external context providers
Context Broker GE also implements a mechanism so part of the context information is not
managed directly by the Context Broker, but by external Context Providers. Thus, in this
way, Context Broker GE becomes the single point of access for all the context information
management, regardless the actual location of the source of context. This approach has
several advantages:
It’s easy for client applications to interact with just one API endpoint, instead of interacting with a different API endpoint for each source of context. Moreover, if the endpoint for a given source of context changes, the change is done at Context Broker GE in a way that is transparent for clients.
Scalability. When a single Context Broker GE is not enough to manage all the context information, several can be used. Each Context Broker GE (managing a subset of the context) is set as Context Provider in the Context Broker GE acting as single access point. This schema can be applied in a hierarchical way, which GE at different “level”, each level managing finer-grained context subsets.
Security. Clients may be not allowed to access directly to Context Providers for security reasons. In that case, the Context Broker GEs (which has the ability to authenticate and authorize request) acts as security enforcement point.
Let’s illustrate with an example based on the NiceEating case. Let’s consider an external
restaurant booking system that is able to provide in real time the number of free places of a
given restaurant and occupancy level (as a percentage). That system runs at
http://booking.restaurants.foo.com and plays the role of a Context Provider.
First, the Context Provider is registered as provider of free_places and occupancy_level for
“LeBistro” restaurant:
POST
<cb_host>:<cb_port>/v1/registry/contextEntities/type/Restaurant/id/LeBistro/attributes/free_
places
{
"duration" : "P1M",
"providingApplication" : "http://booking.restaurants.foo.com"
}
POST
<cb_host>:<cb_port>/v1/registry/contextEntities/type/Restaurant/id/LeBistro/attributes/occo
upancy_level
{
"duration" : "P1M",
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 12
"providingApplication" : "http://booking.restaurants.foo.com"
}
Thus, each time free_places or occupancy_level is queried at Context Broker GE (e.g. from
the user smartphone application), that query is forwarded to
http://booking.restaurants.foo.com. From a Context Consumer perspective, all the process
is transparent, as if the information were actually managed by Context Broker GE locally.
4.1.3 How to subscribe to changes on context information
Apart from getting information using queries in a synchronous way (as illustrated in the
“How to update and query context information” section above), Context Consumer can also
get context information in an asynchronous way using notifications. In this scenario, the
Context Broker GE is “programed” to send notifications upon given conditions (specified in
the subscription request).
In the case of NiceEating, the application backend could use a subscription so each time a
new rating is cast by any user, the backend gets notified (in order to recalculate restaurant
average score and publish it back in the Context Broker GE).
POST <cb_host>:<cb_port>/v1/subscribeContext
{
"entities": [
{
"type": ".*",
"isPattern": "true",
"id": "Rating"
}
],
"attributes": [ "score" ],
"reference": "http://backend.niceeating.foo.com:1028/ratings",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"score"
]
}
]
}
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 13
Another case would be an application that subscribes to changes in average ratings of a
given restaurant. This may be useful for restaurant owners in other to know how their
restaurants score is evolving.
POST <cb_host>:<cb_port>/v1/subscribeContext
{
"entities": [
{
"type": ".*",
"isPattern": "true",
"id": "Restaurant"
}
],
"attributes": [ "average_scoring" ],
"reference": "http://myapp.foo.com:1028/restaurant_average_scorings",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"average_scoring"
]
}
]
}
4.1.4 How to perform geo-location queries
One very powerful feature in Context Broker GE is the ability to perform geo-located
queries. You can query entities located inside (or outside) a region defined by a circle or a
polygon.
For example, to query for all the restaurants within 2 km of the Madrid city center (identified
by GPS coordinates 40.418889, -3.691944) a Context Consumer application will use the
following query:
POST <cb_host>:<cb_port>/v1/queryContext
{
"entities": [
{
"type": "Restaurant",
"isPattern": "true",
"id": ".*"
}
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 14
],
"restriction": {
"scopes": [
{
"type" : "FIWARE::Location",
"value" : {
"circle": {
"centerLatitude": "40.418889",
"centerLongitude": "-3.691944",
"radius": "2000"
}
}
}
]
}
4.2 Connection to the Internet of Things
You may start managing context information that arrives from external systems connected
to your application or from end users interacting with your application using some web
portal. However, you may require incorporating management of “things” using
sensors/actuators. This will require incorporating some GEs of the FIWARE IoT chapter as
part of your architecture.
The FIWARE IoT Backend Device Management GE (open source reference implementation
IDAS) will connect to the Context Broker GE (see Development of context-aware applications)
and will hide all the complexity linked to management of different IoT protocols.
The FIWARE IoT Backend Device Management GE is defined to provide both built-in
support of some of the most relevant IoT standards (SensorML/UL2.0, MQTT, ETSI M2M,
OMA-LWM2M/CoAP…) as well as the ability to integrate additional IoT protocols (e.g.,
protocols supported by the device manufacturer). Therefore, it covers the “southbound”
connection to devices. It accomplishes that goal by means of implementing an architecture
where interaction with IoT devices is handled through processes referred as “IoT-Agents”.
Each of these IoT-Agents handles interaction with a given set of devices, using the specific
protocols they support and working as Context Producers and/or Context Providers (see
Development of context-aware applications). You can use any of the built-in IoT-Agents already
developed and available or create your own IoT-Agent.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 15
FIWARE IoT Device Management GE architecture
Using the FIWARE OMA NGSI API, reading from a sensor or actuating a device will be as
easy as to read or write an attribute associated with some Context entity. Again, remember
the analogy to SNMP, which would be extended to the management of the so-called
“Internet of Things”.
If you are interested to connect your application to the Internet of Things, check out:
How to read measures captured from IoT devices
How to act upon IoT devices
4.2.1 How to read measures captured from IoT devices
The first thing you need to do is to connect your IoT devices or Gateways to the FIWARE
Cloud. This basically means that your IoT devices’ observations reach a ContextBroker.
Although you may always implement the NGSI protocol in your IoT devices or gateways
and then talk directly to the ContextBroker, here we present a more common (and IoT
focussed) option based on using standard IoT protocols and use FIWARE BE Device
Management GE (IDAS implementation) that will translate it for you into NGSI and send it to
a ContextBroker.
In the following paragraphs, we provide an example on how you would connect your
devices using Ultralight2.0 (UL2.0) - which is a proposed simplification of the SensorML
(SML) standard - and get those devices sending their measurements (observations) to the
FIWARE Lab public ContextBroker. Ultralight2.0 is selected in this example because of its
simplicity.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 16
Basically, there are five simple steps to follow:
1. Know the details of the IDAS (IoT Agent) you will be sending the measurements to
These details are available at the FIWARE catalogue, BE Device Man. instances section.
For UL2.0 we use IDAS2.6 implementation running at the FIWARE Cloud with the following
details:
IP Address (<idas_host>): In this case, it is 130.206.80.47. REST Admin Port (<idas_admin_port>): which is the port used to access the
Administrative API of IDAS: 5073. REST Devices Port (<idas_ul20_port>): which is the port used by your IoT devices
to send observations or request commands, in this case: 8002. Service (<service>): IDAS is able to manage different tenants, for instance for
different cities or smartspaces. You can always use our “OpenIoT” service for testing, as we actually do in this example.
APIKEY (<apikey>): This is a shared secret your IoT devices need to know prior to communicate to IDAS regarding a specific Service. Each service/tenant has its own APIKEY, in the case of the testing service “OpenIoT”, it is the string “4jggokgpepnvsb2uv4s40d59ov”.
2. Create a Model for your IoT device
Before a device is registered in IDAS, a model of that device needs to be created at this
time. In future releases we are removing this requirement.
However, you can always re-use the models that we have already created in the “OpenIoT”
service. Currently, they are:
SENSOR_TEMP: for a generic Temperature sensor. SENSOR_HUM: for a generic Humidity sensor. SENSOR_LUM: for a generic Illuminance sensor. SENSOR_MOV: for a generic Presence sensor. SENSOR_ZWAVE_4IN1: for the specific Everspring Zwave 4IN1 (Temperature,
Humidity, Light, Presence) sensor.
However, creating your own model is pretty simple, by performing for instance the following
query we used to create “”SENSOR_TEMP”:
POST <idas_host>:<idas_admin_port>/v1/m2m/v2/services/<service>/models/
(Example: POST: http://130.206.80.47:5371/m2m/v2/services/OpenIoT/models/)
Headers: {'content-type': 'application/json’; 'X-Auth-Token' : <Oauth2.0 TOKEN>}
{ "name": "SENSOR_TEMP",
"capabilities" : [
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 17
{ "name" : "Temperature",
"property" : "temperature",
"format" : {
"name" : "Temperature",
"alias" : "t" ,
"phenomenon": "urn:x-ogc:def:phenomenon:IDAS:1.0:temperature",
"type" : "Quantity",
"uom": "celsius”
} } ],
"protocol": "ul-2_0” }
The exact syntax for the JSON payload is documented in the IDAS manual linked in the
catalogue. The alias of the measured phenomenon (“t”) will be used later on to keep
observation packets at their minimum expression (“t” instead of “temperature”).
The NGSI entity forwarded to the ContexBroker will have the expanded attribute name
(“Temperature”, in this example).
3. Create (register) an asset/device for your IoT device
A device is understood as a physical device and an asset, which is its logical
representation. This is made to avoid changing the upper layer asset resource in the event
you replace the specific physical device (failure, technology update, etc.).
Therefore there is a device name, normally tied to a unique identifier of the device and/or
gateway (e.g. a zwave temperature sensor connected to a RaspberryPI that we identify as
”RPI:79:ed:af:zwave:temp:1”), and the logical asset name that you may choose for your
convenience (e.g. “TEMP-MainRoom”).
In order to register an IoT device (and create the associated asset), the following query has
to be performed:
POST <idas_host>:<idas_admin_port>/v1/m2m/v2/services/<service>/assets/
(Example: POST: http://130.206.80.47:5371/m2m/v2/services/OpenIoT/assets/)
Headers: {'content-type': 'application/json’; 'X-Auth-Token' : <Oauth2.0 TOKEN>}
{ "name": "RPI:79:ed:af:zwave:temp:1",
"model": "SENSOR_TEMP",
"asset": {
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 18
"name": "TEMP-LivingRoom",
"description": "asset model protocol”
}
}
4. Send Observations related to your IoT device
Sending an observation from a IoT devices is extremely efficient and simple with the
following query:
POST <idas_host>:<idas_ul20_port>/d?k=<apikey>&i=<device_name>
(Example: POST:
http://130.206.80.47:8002/d?k=4jggokgpepnvsb2uv4s40d59ov&i=RPI:79:ed:af:zwave:temp:
1)
Headers: {'content-type': 'application/json’; 'X-Auth-Token' : <Oauth2.0 TOKEN>}
“t|25”
In the event you want to provide several measurements at once, the query looks like this
(for a “4IN1” device based in other model with four pre-defined observations: temperature,
humidity, illuminance and presence/movement):
POST <idas_host>:<idas_ul20_port>/d?k=<apikey>&i=<device_name>
(Example: POST:
http://130.206.80.47:8002/d?k=4jggokgpepnvsb2uv4s40d59ov&i=RPI:79:ed:af:zwave:4IN1:
1)
Headers: {'content-type': 'application/json’; 'X-Auth-Token' : <Oauth2.0 TOKEN>}
“t|23#h|80#l|95#m|Quiet”
So it follows the convention “alias1|value1#alias2|value2#alias3|value3...”.
5. Reading measurements sent by your IoT device
Finally, after connecting your IoT devices this way you (or any other developer) should be
able to use the ContextBroker NGSI API to read the NGSI entity assigned to your device.
The Entity ID will be the asset name (<asset_name>), the entity type will be “Sensor” and
the attributes will be the phenomena defined in the model your device is based on
(“Temperature”, “Humidity”, etc.).
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 19
In our particular example described above, you may check in the Global ContextBroker,
where you will find an entity with Entity_ID=”TEMP-LivingRoom”, Entity_type=”sensor” and
the attribute “Temperature” with the correct updated value.
4.2.2 How to act upon IoT devices
In order to send commands to devices, developers will find with NGSI a really convenient
approach: they just need to update the attribute related to a command in the correspondent
NGSI entity.
In the particular example described above, the developer would send an “UpdateContext”
operation to change the attribute “Set_Temperature” for the entity “TEMP-LivingRoom”. For
this action to work properly, that NGSI entity attribute will be actually provided by an IoT-
Agent and therefore the update will get to that IoT-Agent that knows the device in charge –
a thermostat in this case- and will launch a command to perform the desired operation. All
this process in done in a transparent way from the point of view of the APP developer.
At this point, the only way for a developer to know that a particular attribute is related to
commands generation is via documentation of the NGSI entities or signalling those
attributes with a pre-defined metadata that the client may check, for instance
“Commands=True”.
4.3 Real-time processing of context events
You may want to perform some processing on available context information. As an
example, you may want to automatically detect patterns that require triggering some action
or raising some alarm. You can use the FIWARE Complex Event Processing (CEP) as part
of the architecture of your applications for this purpose. The Complex Event Processing GE
allows you to detect patterns above contexts. This way, instead of reacting to a single
context information you can identify and react to patterns over the contexts of several
entities or over a context that was changed over time. The Complex Event Processing
receives contexts information as input events and generates observations that sometimes
called situations, as output events. The CEP GE analyses input event data in real-time,
generates immediate insight and enables instant response to changing conditions. The
technology and implementations of the CEP provide means to expressively and flexibly
define and maintain the event processing logic of the application.
Applications connected to the CEP GE (external applications or some other GEs like the
Context Broker GE) can play two different roles: the role of Event Producer or the role of
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 20
Event Consumer. Note that nothing precludes that a given application plays both roles.
Event Producers are the source of events for event processing. Following are some
examples of event producers:
External applications reporting events on user activities such as "user placed a new order", and on operation activities such as "delivery has been shipped".
Sensors reporting on a new measurement. Events generated by such sensors can be consumed directly by the CEP GE. Another alternative is that the sensor events are gathered and processed through the IoT GEs, which publish context events to the Context Broker GE, having the CEP acting as a context consumer of the Context Broker GE.
Event Producers can provide events in two modes:
"Push" mode: the Event Producers push events into the CEP by means of invoking a REST API.
”Pull” mode: the Event Producer exports a REST API that the CEP can invoke to retrieve events.
Event Consumers are the destination point of events. Following are some examples of
event consumers:
Dashboard: a type of event consumer that displays alarms defined when certain conditions hold on events related to some entities user community or produced by a number of devices.
Handling process: a type of event consumer that consumes meaningful events (such as opportunities or threats) and performs a concrete action.
The Context Broker GE which can connect as an event consumer to the CEP and forward the events it consumes to all interested applications based on a subscription model.
The CEP sends output events to the event consumers in a “push” mode by activating their
REST API.
The CEP allows you to define patterns over selected events occurring in event processing
contexts (such as a time window or segmentation) with optional additional conditions. Those
patterns are defined using a Web based authoring tool without the need to write any code.
This makes it easier to write the event processing logic and to maintain and change it over
time. Examples for supported patterns are:
Sequence, meaning events need to occur in a specified order for the pattern to be detected. E.g., follow a sensor context, and detect if the sensor status was “fixed” and later was “failed” within a time window.
Aggregate, compute some aggregation functions on a set of incoming events. E.g., compute the percentage of the sensors events that arrived with a fail status out of all the sensors events arrived in the time window. Alert if the percentage of the failed sensors is higher than 10 percent.
Absent, meaning no event holding some condition arrived within the time window for the pattern to match. E.g., alert if within the time window no sensor events arriving from specific source have arrived. This may indicate that the source is down.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 21
All, meaning that all the events specified should arrive for the pattern to match. E.g., wait to get status events from all the 4 locations, where each status event arrives with the quantity of reservations. Alert if the total reservations are higher than some threshold.
Every pattern is associated with an Event processing context. Event processing context
groups event instances so that they can be processed in a related way. It assigns each
event instance to one or more processing context partitions. Event processing context can
be a temporal processing context, a segmentation processing context, or a composite
context that is to say one made up of other processing context specifications.
Temporal processing context defines a time window. If a pattern is associated with a temporal processing context, it will process only the events arriving within this time window. E.g., “within 5 seconds after sensor events with high value”, “A time window bounded by Order-Placed and Order-Delivered events”
Segmentation processing context defined a matching criteria based on the attribute values of the events. E.g., “Sensor’s ID”, “Shipment’s ID”, “Building’s ID”. Events that belong to the same matching criteria (e.g., have the same sensor ID) will be processed together within the same processing context partition.
4.3.1 How to define a CEP application
A CEP application is created using the CEP web based authoring tool. Using this authoring
tool, the application developer creates the building blocks of the application definitions. This
is done by filling up forms without the need to write any code.
The URL of the CEP authoring tool is:
<cep_host>:<port>/AuthoringTool/Main.html
(Example: http://130.206.81.23:8080/AuthoringTool/Main.html)
The building blocks of a CEP application are:
Event types – the events that are expected to be received as input or to be sent as output. An event type definition includes the event name and a list of its attributes.
Producers – the event sources and the way the CEP should pull events from those sources. If you are going to push events through REST calls, there is no need to define a producer.
Consumers – the event consumers and the way to send them the CEP output events.
Temporal processing contexts – time window contexts in which event processing agents are active.
Segmentation processing contexts – semantic contexts that are used to group several events to be processed by event processing agents.
Composite processing contexts – group together several different processing contexts.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 22
Event processing agents (EPAs) – responsible to detect patterns over incoming events in a specific processing context and to generate derived events.
The Authoring tool allows you to define a CEP application, validate it, and export the
application definition. The export action creates a JSON format representation of the CEP
application. This JSON can be exported either to the engine repository or to a local file (to
be later fed to the engine repository).
This JSON file that describes a CEP application is the input for the CEP engine. It can be
generated by the authoring tool, and it can be generated programmatically by any other
application and fed to the CEP engine.
4.3.2 How to administrate the CEP
There are REST services that allow you to manage the CEP definition repository that holds
the CEP application definitions available to the CEP engine at run time. These services
allow putting a new application definition to the repository, getting a specific application
definition from the repository, updating a repository application definition or deleting a
definition from the repository. In addition, there are REST services that allow controlling the
CEP engine at run time. These services allow starting and stopping a CEP engine, updating
CEP engine definitions and reading the state of the CEP engine (started/stopped and its
definition url).
4.3.3 How to send input events to the CEP
The CEP provides a REST service for receiving input events. Event has a list of attributes,
each attribute as a name and a value, and the name of the event is given in the "Name"
attribute. The CEP supports three formats for input events: tag-delimited, JSON and NGSI-
XML.
To send event in the JSON format use the following query:
POST <cep_host>:<port>/{instance_name}/rest/events
(Example: POST: http://130.206.81.23:8080/ProtonOnWebServer/rest/events)
Headers: {'Content-Type’: 'application/json’; 'X-Auth-Token' : <Oauth2.0 TOKEN>}
{ "Name":"TrafficReport",
"volume":"1000"
}
To send the same event in the tag-delimited format use the same query but with a different
content type and a different data format:
POST <cep_host>:<port>/{instance_name}/rest/events
(Example: POST: http://130.206.81.23:8080/ProtonOnWebServer/rest/events)
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 23
Headers: {'Content-Type’: ‘text/plain’; 'X-Auth-Token' : <Oauth2.0 TOKEN>}
Name=TrafficReport;volume=1000;
The NGSI-XML format allows the CEP to get input events from the Context Broker GE. If
you want the CEP to get input events from the Context Broker GE, you should add a
subscription to the Context Broker with the CEP URL given above.
4.4 Handling authorization and access control to APIs
In FIWARE we offer some services and tools to allow you to manage authentication and
authorization in your applications and backend services. If you want to manage identity in
your application without developing your own mechanisms, you can offer your users the
possibility to log in to your app using their FIWARE Accounts.
This is possible thanks to the OAuth2 protocol and Keyrock, the Identity Manager
component of FIWARE. In the same way that you usually log in to some services using your
Twitter or Facebook account, your users will use their FIWARE accounts to access your
service. But this is only the first step, because you can also secure your backends using
FIWARE Account. If your service or GE has a REST API that can be accessed from
Internet, probably you want to manage the access to the resources. For instance, you can
allow the access only to the users that have a FIWARE account.
Furthermore, if you want to differentiate user permissions based on roles, thanks to the
Access Control component of FIWARE, you can define the various roles, possibly with role
hierarchies, and define specific permissions for each role. This way you can determine
which type of users can access which specific resource in your backends. You can also
manage basic role permissions via the Identity Management user interface and they will be
pushed automatically to the Access Control component. For more advanced role
permissions, for instance combining user roles with other user attributes, resource attributes
(besides the URL), or environment attributes (e.g. time constraints), you would use the
Access Control interface directly. In any case, you can assign roles to your users with the
Identity Management interface.
So the two components that you have to know in order to manage access control in your
services are:
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 24
4.4.1 Keyrock GE
Identity Management covers a number of aspects involving users' access to networks,
services and applications, including secure and private authentication from users to
devices, networks and services, authorization & trust management, user profile
management, privacy-preserving disposition of personal data, Single Sign-On (SSO) to
service domains and Identity Federation towards applications. The Identity Manager is the
central component that provides a bridge between IdM systems at connectivity-level and
application-level.
Thanks to this GE you can create users and organizations in FIWARE. You can also
register new applications in order to use the OAuth2 protocol described above and to
manage roles and permissions in them.
4.4.2 Access Control GE
The Access Control GE provides XACML-standard-compliant authorization services. The
OASIS XACML (eXtensible Access Control Markup Language) standard [1] is the only
existing standard access control language written in XML as of writing. According to the
OASIS XACML FAQ, “it provides an extremely flexible language for expressing access
control that can use virtually any sort of information as the basis for decisions. It is a
functional superset of other familiar access control schemes, such as permissions, ACLs,
RBAC, etc. It is particularly designed to support large-scale environments where resources
are distributed and policy administration is Federated.”
In short, this component proves to be useful when dynamic attribute-based access control
matters, i.e. when authorization depends on attributes in the context of the access request,
such as the requester (subject)'s attributes, the requested action, the requested resource,
and possibly environment attributes. The component may be used in various ways to
control access to critical resources such as other FIWARE GEs or applications deployed by
developers from use case projects. It provides APIs to developers - or security policy
authors in general - for managing ABAC (Attribute-Based Access Control) and/or RBAC
(Role-Based Access Control) policies, and getting authorization decisions for these policies.
It may be combined with the FIWARE Security Proxy playing the role of PEP (Policy
Enforcement Point), in which case the Access Control GE plays the role of PDP (Policy
Decision Point) in XACML terms.
If you are interested in more details about how to handle access control in FIWARE check
out:
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 25
How to create your identity in FIWARE How to implement OAuth2 in your applications How to send requests to a FIWARE GE How to secure your backend service How to manage Access Control in FIWARE
4.4.3 How to create your identity in FIWARE
The first step to start using identity components in FIWARE is to create an account in
FIWARE Account. In order to do that you have to access the Account Portal and click in
“Sign Up” option:
Then follow the instructions filling your data and waiting for the confirmation mail.
Once you have an account you can start creating organizations and applications.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 26
4.4.4 How to implement OAuth2 in your applications
The first step to start managing authorization in your application is to register the application
in FIWARE Account. In order to do that you have to click on “Register” option of the Account
Portal
Then follow the steps with the data of your application. Once registered, you have to
implement OAuth2 protocol in your application. The message flow between your web
application and IDM account server should be:
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 27
In order to implement this flow you can use an OAuth2 library. Here you will find
implementations for PHP, Cocoa, iOS, Java, Ruby, Javascript, Python … We have
developed a Node.js example that you can download and test following these instructions:
1. Software requirements:
nodejs
npm
2. Install the dependencies:
npm install
3. Configure OAuth2 credentials (ClientID and Client Secret) in config.js file. You will find
them in your IDM account:
4. Start the example server
sudo node server
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 28
5. Connect to http://localhost to try the example
* Connect to http://localhost/logout to delete session cookies once you have logout in IDM
portal
4.4.5 How to send requests to a FIWARE GE
If your web app is going to use a FIWARE GE service you have to authenticate the
requests.
Once you have obtained an Oauth2 access‐token you have to include it in your client
requests in order to authenticate them. This way the GE will check your token with the IDM
and decide if your requests are valid. The architecture of this flow is:
So for include the access‐token in your requests you have to do:
GET /GE_URL_path HTTP/1.1
Host: GE_hostname
X-Auth-Token: access_token
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 29
For instance, using our OAuth2 example explained previously, you can get the
access_token in this code fragment
4.4.6 How to secure your backend service
You can also develop backend applications using FIWARE Account authentication. To do
so you should use our FIWARE PEP Proxy that allows you to validate the requests before
sending them to your server:
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 30
You can download the proxy here. The instructions to install and use it are:
Installation:
1. Software requirements:
nodejs
npm
Note: Both can be installed from (http://nodejs.org/download/)
2. Clone Proxy repository:
git clone https://github.com/ging/fi‐ware-pep‐proxy.git
3. Install the dependencies:
cd fi‐ware‐pep‐proxy/
npm install
4. Configure the app host in config.js file.
//Hostname to forward authenticated requests
config.app_host = 'www.google.es';
//Port where the HTTP server is running
config.app_port = '80';
5. Start the proxy server
sudo node server
How to use
Requests to the proxy should be made with a special HTTP Header like in the previous
example: X‐Auth‐Token. This header contains the OAuth access token obtained from
FIWARE IDM GE.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 31
GET / HTTP/1.1
Host: proxy_host
X-Auth-Token:z2zXk...ANOXvZrmvxvSg
To test the proxy you can generate this request running the following command:
curl -‐header "X-Auth‐Token:z2zXk...ANOXvZrmvxvSg" http://proxy_host
Once authenticated, the forwarded request will include additional HTTP headers with user
info:
X-Nick-Name: nickname of user in IDM
X-Display‐Name: display name in IDM
In the FIWARE eLearning platform you will find a webinar and a detailed tutorial explaining
you how to create accounts, applications and manage authorization in your applications.
4.4.7 How to manage Access Control in FIWARE
You can define custom roles for your applications via the IdM interface, then permissions for
each roles.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 32
When you define new role permissions, the IdM commits the resulting access control policy
to the FIWARE Access Control GE. From now on, for every access request, the FIWARE
Proxy protecting your application requests an authorization decision (XACML) to the IdM
GE which forwards it with extra user info to the Access Control GE. The Access Control GE
replies with Permit or Deny Decision (XACML) to the IdM GE which in turn forwards the
decision to the Proxy. The Proxy blocks the application request if and only if the decision is
Deny.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 33
4.4.8 References
[1] OASIS XACML Specifications
https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=xacml#XACML20
4.5 Publishing (Open) Data in FIWARE
This section explains how users (both end users and developers) can publish data in
CKAN, the Open Data publication GE, as well as how they can publish offerings containing
this data in WStore so as to set access control policies over it and/or monetize it. You
should note that at the time of writing this document, FIWARE offers two different types of
data: context information and datasets. Each type of data should be published in a different
way and for this reason you’ll find two differentiated sections. A third section is devoted to
explain the benefits of publishing your data in the WStore and the way of doing it.
If you are interested in more details about how to publish open data, check out:
How to Publish (Open) Datasets in CKAN
How to Publish Context Information as (Open) Data in CKAN How to Offer Datasets (including Context Information) through the WStore
o Offering Datasets Directly Through the CKAN Interface
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 34
o Offering Datasets Through the WStore interface o Acquiring Datasets Offered in the WStore
4.5.1 How to Publish (Open) Datasets in CKAN
To upload your own datasets (open data) to CKAN, you must access the FIWARE Data
portal. Once that you are logged in, you should go to the “Datasets” section and click on the
“Add Dataset” button.
In the first step, you will have to provide some basic information such as the name, the
description or the tags of your dataset. In addition, you will be asked for some additional
information:
Visibility: if you choose “Public”, all the users (even those that are not logged in) will
be able to access the dataset. Otherwise, only some selected users will be able to
access the dataset.
Searchable: you can choose if you want your dataset to be published in the queries
performed by the FIWARE Data portal users.
o This field is only enabled when “Visibility” is set to “Private”.
o If you create a public dataset, it will be always searchable.
Allowed Users: the list of users that can access your dataset.
o This field is only enabled when “Visibility” is set to “Private”.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 35
In the second step you will be asked to upload the data itself. You can provide a link or
upload a file. Any type of file is alowed, but if you want to generate an automatic API to
access your data, you must upload a CSV file.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 36
In the last step you are asked to provide some metadata. This metadata is not a must so
you can avoid this section if you want.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 37
4.5.2 How to Publish Context Information as (Open) Data in CKAN
Publishing and consuming open data is a cornerstone for the development of applications
and the creation of an innovation ecosystem. Through the mechanisms described in the
section Development of context-aware applications, the Context Broker can be used to publish
and consume context information. In particular, this information can be indeed open data
and consumed through the queries and subscriptions APIs (NGSI10). This way, it is
possible to publish real time or dynamic data, typically well structured, and offered it as
open data through the reutilization by developers. For instance, it is possible to offer in real
time data from sensors or systems to leverage the creation of new applications.
However, the Context Broker only provides the latest snapshot of the context information in
a given moment, but there are many cases where it is also required to store and publish
historical information of the context data generated over time. This is one of the usages of
the Open Data publication GE, CKAN, in FIWARE.
CKAN is an open source solution for the publication, management and consumption of
open data, usually, but not only, through static datasets. CKAN allows to catalogue, upload
and manage open datasets and data sources, while supports searching, browsing,
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 38
visualizing or accessing open data. CKAN is the Open Data publication platform that is most
widely used by many cities, public authorities and organizations.
You may take advantage of the connectors supported by the Context Broker that
automatically generate historic records generated each time there is a change in the context
information and make those records available for upload on the Open Data publication GE.
The data is then stored in a Datastore, and can be downloaded and queried through REST
APIs.
In order to achieve this behaviour it is necessary to deploy and configure Cygnus, a piece of
software complementary to the Context Broker GE. The instructions to install Cygnus can
be found here.
Once Cygnus has been installed, it is required to configure it. In a nutshell, there are three
steps: configure CKAN storage, create the desired subscriptions in the Context Broker and
run the process.
This sink persists the data in a datastore in CKAN. Datastores are associated to CKAN
resources and as CKAN resources we use the entityId-entityType string concatenation. All
CKAN resource IDs belong to the same dataset (also referred as package in CKAN terms),
which name is specified with the default_dataset property (prefixed by organization
name) in the CKAN sink configuration.
In order to configure CKAN storage, the file cygnus.conf has to be edited, specifying the
CKAN sink, the sink channel (where to read the notifications from), the CKAN’s user API
key, CKAN instance detail (IP, port, etc.), and Context Broker instance endpoint. All the
details can be found at:
https://github.com/telefonicaid/fiware-connectors/blob/master/flume/README.md
Once the storage has been configured, it is required to run the process with, for instance,
the following command:
$ APACHE_FLUME_HOME/bin/flume-ng agent --conf APACHE_FLUME_HOME/conf -f
APACHE_FLUME_HOME/conf/cygnus.conf -n cygnusagent -
Dflume.root.logger=INFO,console
Once the connector is running, it is necessary to tell Orion Context Broker about it, in order
Orion can send context data notifications to the connector. This can be done on behalf of
the connector by performing the following curl command (specifying the endpoint where
Cygnus is listening:
(curl localhost:1026/v1/subscribeContext -s -S --header 'Content-Type:
application/json' --header 'Accept: application/json' -d @- | python -
mjson.tool) <<EOF
{
"entities": [
{
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 39
"type": "Room",
"isPattern": "false",
"id": "Room1"
}
],
"attributes": [
"temperature"
],
"reference": "http://host_running_cygnus:5050/notify",
"duration": "P1M",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"pressure"
]
}
],
"throttling": "PT5S"
}
EOF
Once the process starts storing data, the dataset and resource will appear in CKAN and it
will be possible to browse and download data from CKAN portal, or query it through the
Datastore API. More information at:
http://docs.ckan.org/en/ckan-2.2/datastore.html#the-datastore-api
For instance, the following query would return the first 5 results of a dataset.
GET CKAN_HOST/api/action/datastore_search?resource_id=5a2ed5ca-2024-48d7-
b198-cf9d95c7374d&limit=5
4.5.3 How to Offer Datasets (including Context Information) through the WStore
FIWARE Store is one of the portals deployed in FIWARE Lab, which is the main showcase
of the project. Therefore, it is the best way, together with the Open Data GE and its
reference implementation (CKAN), for advertising your datasets and data APIs, and make
them available for prospective users. However, this is not the only advantage of publishing
the data in the FIWARE Store since you will also get the possibility of:
Managing the users that can access your data: This is very important since in some cases you have confidential information that cannot be spread.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 40
Forcing the users to accept some terms and conditions for using your data: in some cases, you will want the users of your data to accept some terms for using it (i.e. you want to offer your data for free but you have to ensure that this data is not used with commercial purposes).
Charging the users for accessing your data: there are some conditions (conflicts of interest, security, competence...) that force you to publish data with cost. In that cases, you can use CKAN and the Store together to charge the users for using your data. You are able to choose different payment models:
o Single payment: Users pay once. o Subscription: Users are forced to pay periodically (daily, monthly,...) for
using your data. In addition, users will be able to cancel the subscription but they won’t be able to access the data anymore.
o Pay per use: Users pay based on the amount of information consumed.
Once that your data (datasets or context information) is published in CKAN, you are able to
publish it in the FIWARE Store. You can do it in two different ways: through the CKAN
interface or through the Store. The first option is the easiest one but is more constrained
since you can only provide an offering with a single resource: the dataset itself. On the other
hand, if you choose the second option, you will be able to offer the dataset with other
resources (such as widgets...) that can add some extra value to your data.
4.5.3.1 Offering Datasets Directly Through the CKAN Interface
The first thing that you have to do is to click on the “Manage” button that you will see when
you access your dataset:
Then, you should select the “Publish” tab. In this tab you will be asked to provide some
information of the offering (the name, the description...). If you want to publish a private
dataset, you will be asked for a price. We remind you that if you want to create a free
offering, you can set the “Price” field to zero. Once that you have filled in the fields, you
have to click the “Publish Offering” button to finish the process.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 41
4.5.3.2 Offering Datasets Through the WStore interface
As mentioned before, you can also publish the datasets in the Store by using the methods
provided by the Store itself. To do so, first of all you have to enter the FIWARE Store. Once
in the Store, you should click in the “My Offerings” button, in the left side bar. To register
your dataset as a resource, click on the “Provider Options” button and then in the “Register
Resource”. You must note that an offering is composed by zero or more resources.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 42
In the new form you will have to complete some fields:
Name: The name of your dataset (it mustn’t match with the name provided in CKAN).
Version: The version of the dataset (typically: 1.0)
Description: Description of the dataset
Content Type: dataset
How to provide the resource: Provide resource URL Resource URL: the URL that you get when you create your dataset in the FIWARE
Data Portal (typically:
https://data.lab.fiware.org/dataset/{dataset_name}). You can check
this URL by accessing your dataset in FIWARE Data Portal. Open Resource: open offerings (those that cannot be acquired) can only contain
open resources. However, non-open offerings can contain open resources. (Typically you should disable this checkbox).
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 43
When you have registered your dataset as a resource, you will be able to provide it in the
Store. To do so, you should create an offering. You can do it by clicking the “Provider
options” button and then the one titled “Create Offering”.
A new dialog will be displayed. This dialog will guide throughout the process of creating a
new offering. In the first step, you will be asked to fulfil some basic information such as the
name, the version or the logo. You should also note that you are asked for a Notification
URL. This URL is the one that will be called when a user acquires a dataset. The Data
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 44
Portal needs to be notified every time a user acquires a dataset in order to allow this user to
access the dataset. In this option you should choose “Provide a notification URL” and
introduce the following one: “https://data.lab.fiware.org/api/action/dataset_acquired”. Please
note: if you don’t fill this field properly, users won’t be able to access acquired dataset.
In the following step you will be asked for the description, the pricing model and the legal
terms. Using this form, you are only able to choose two pricing models: single payment or
free offerings. If you want to set a more complex pricing model, you should provide a
custom USDL. Relating to the legal conditions, you must note that all users will be forced to
accept these conditions before acquiring the offering. If you don’t want users to accept legal
terms, you can leave these fields in blank.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 45
The next step asks you to select some applications. In this case you can avoid this section.
The most important section is the next one, where you will have to select the resources that
are bounded to the offering. Here you must select the resource created in the previous step.
Additionally, you can add some other resources such as WireCloud widgets that can add
extra value to your dataset. For example, you can add WireCloud widgets to plot graphs
based on the data.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 46
At this point, the offering is already created but is not published yet. To publish the offering,
we should click on the “Provided” button placed on the left side bar. In this section, you
have to look for the offering created previously. Then click on it and add some tags. To do
that, click on the “+” button placed below the “Tags” heading.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 47
You can add all the tags that you want, but if you want your dataset to appear in the
“Datasets” section of the Store, you must add the “dataset” tag.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 48
Finally, you must click in the “Publish” button to complete the process.
4.5.3.3 Acquiring Datasets Offered in the WStore
Customers can access the FIWARE Store to acquire one or more datasets. To do so, they
have to enter the Store and look for datasets in the “Data” section. This section can be
found in the left side bar.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 49
Once in the Data section, user will see all the offerings that contain data. Users can choose
the one that fits their requirements and acquire it by clicking in the appropriate button (the
one with the price or with the label “Free”).
By clicking this button, a new dialog will be displayed. If the offering has some legal
conditions attached, the user will be forced to accept them before completing the process.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 50
Finally, the user will be asked to enter his/her address.
Once that the offering has been acquired, the user will be able to access the attached
datasets. To do so, the user should access the offering and click on the “Resources” button.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 51
A new dialog will be displayed with all the resources contained in the offering. Among these
resources, the user will find the dataset:
4.6 Big Data analysis of historic context information
Similarly to what has been described in section Publication of context information as Open
Data, Cygnus software allows to store all the selected data published in the Context Broker
in an HDFS based storage. This allows having a long term historic database of context
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 52
information that can be used for later analysis, for instance implementing map & reducing
algorithm or performing queries over big data through Hive.
Similarly to what has been described in the section Publication of context information as
Open Data, the Cygnus component can be configured to gather data from the Context
Broker and store it in HDFS. The configuration in this case should include the Cosmos
Namenode endpoints, the service port, user’s credential, the Cosmos API used (webhdfs,
infinity, httpfs), the type of attribute and endpoint of Hive server.
Once the Context Data has been stored, it is possible to use the Big Data GE to process it
either with a map & reduce application or with Hive. It is of course also possible to process
in the Big Data GEs other big datasets, either by themselves or in combination with context
information.
A typical example would be to analyse massive information gathered from sensors in a city
over a long period of type. All the data would have been gathered through Context Broker
and Cygnus and stored in Big Data for a long period of time. In order to analyse the data, a
few steps should be followed (the examples in this whitepaper are based on a global and
shared instance of Cosmos Big Data GE):
Browse the Cosmos portal (http://cosmos.lab.fiware.org/cosmos-gui/). Use an already
registered user in FI-LAB to create a Cosmos account. The details of your account will be
given once registered, typically:
Cosmos username: if your FI-LAB username is <my_user>@mailprovider.com, your cosmos username will be <my_user>. This will give you a Unix-like account in the Head Node of the global instance, being your user space /home/<my_user>/.
Cosmos HDFS space: Apart from your Unix-like user space in the Head Node, you will have a HDFS space located at the entire cluster, it will be/user/<my_user>/
Now you should be ready to login into the Head Node of the global instance of Cosmos in
FI-LAB, simply using your FI-LAB credentials:
[remote-vm]$ export COSMOS_USER=<my_user> // this is not strictly
necessary, junt in order the example commands can be copied&pasted
[remote-vm]$ ssh [email protected]
Once logged, you can have access to your HDFS space by using the Hadoop file system
commands:
[head-node]$ export COSMOS_USER=<my_user> // this is not strictly
necessary, junt in order the example commands can be copied&pasted
[head-node]$ hadoop fs -ls /user/$COSMOS_USER // lists your HDFS space
[head-node]$ hadoop fs -mkdir /user/$COSMOS_USER/new_folder // creates a
new directory called "new_folder" under your HDFS space
...
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 53
Apart from using the context data stored, you can upload your own data to your HDFS
space using the Hadoop file system commands. This can be only done after logging into the
Head Node, and allows uploading Unix-like local files placed in the Head node:
[head-node]$ echo "long time ago, in a galaxy far far away..." >
unstructured_data.txt
[head-node]$ hadoop fs -mkdir /user/$COSMOS_USER/input/unstructured/
[head-node]$ hadoop fs -put unstructured_data.txt
/user/$COSMOS_USER/input/unstructured/
However, using the WebHDFS/HttpFS RESTful API will allow you to upload files existing
outside the global instance of Cosmos in FI-LAB. The following example uses HttpFS
instead of WebHDFS (uses the TCP/14000 port instead of TCP/50070), and curl is used as
HTTP client (but your applications should implement your own HTTP client):
[remote-vm]$ curl -i -X PUT
"http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/$COSMOS_USER/input_dat
a?op=MKDIRS&user.name=$COSMOS_USER"
[remote-vm]$ curl -i -X PUT
"http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/$COSMOS_USER/input_dat
a/unstructured_data.txt?op=CREATE&user.name=$COSMOS_USER"
[remote-vm]$ curl -i -X PUT -T unstructured_data.txt --header "content-
type: application/octet-stream"
http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/$COSMOS_USER/input_data
/unstructured_data.txt?op=CREATE&user.name=$COSMOS_USER&data=true
As you can see, the data uploading is a two-step operation, as stated in the WebHDFS
specification: the first invocation of the API talks directly with the Head Node, specifying the
new file creation and its name; then the Head Node sends a temporary redirection
response, specifying the Data Node among all the existing ones in the cluster where the
data has to be stored, which is the endpoint of the second step. Nevertheless, the HttpFS
gateway implements the same API but its internal behaviour changes, making the
redirection to point to the Head Node itself.
If the data you have uploaded to your HDFS space is a CSV-like file, i.e. a structured file
containing lines of data fields separated by a common character, then you can use Hive to
query the data:
[head-node]$ echo "luke,tatooine,jedi,25" >> structured_data.txt
[head-node]$ echo "leia,alderaan,politician,25" >> structured_data.txt
[head-node]$ echo "solo,corellia,pilot,32" >> structured_data.txt
[head-node]$ echo "yoda,dagobah,jedi,275" >> structured_data.txt
[head-node]$ echo "vader,tatooine,sith,50" >> structured_data.txt
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 54
[head-node]$ hadoop fs -mkdir /user/$COSMOS_USER/input/structured/
[head-node]$ hadoop fs -put structured_data.txt
/user/$COSMOS_USER/input/structured/
A Hive table can be created, which is like a SQL table. Log into the Head Node, invoke the
Hive CLI and type the following in order to create a Hive table:
[head-node]$ hive
hive> create external table <my_user>_star_wars (name string, planet
string, profession string, age int) row format delimited fields terminated
by ',' location '/user/<my_user>/input/structured/';
These Hive tables can be queried locally, by using the Hive CLI as well:
[head-node]$ hive
hive> select * from <my_user>_star_wars; // or any other SQL-like
sentence, properly called HiveQL
Or remotely, by developing a Hive client (typically, using JDBC, but there are some other
options for other non-Java programming languages) connecting tocosmos.lab.fi-
ware.org:10000.
Several pre-loaded MapReduce examples can be found in every Hadoop distribution. You
can list them by ssh'ing the Head Node and commanding Hadoop:
[head-node]$ hadoop jar /usr/lib/hadoop-0.20/hadoop-examples.jar
For instance, you can run the word count example (this is also known as the "hello world" of
Hadoop) by typing:
[head-node]$ hadoop jar /usr/lib/hadoop-0.20/hadoop-examples.jar wordcount
/user/$COSMOS_USER/input/unstructured/unstructured_data.txt
/user/$COSMOS_USER/output/
Please observe the output HDFS folder is automatically created.
The MapReduce results are stored in HDFS. You can download them to your Unix user
space within the Head Node by doing:
[head-node]$ hadoop fs -getmerge /user/$COSMOS_USER/output
/home/$COSMOS_USER/count_result.txt
You can also download any HDFS file to you home user in the Head Node by doing:
[head-node]$ hadoop fs -get
/user/$COSMOS_USER/structured/structured_data.txt /home/$COSMOS_USER/
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 55
If you want to download the HDFS file directly to a remote machine, you must use the
WebHDFS/HttpFS RESTful API:
[remote-vm]$ curl -i -L "http://cosmos.lab.fi-
ware.org:14000/webhdfs/v1/user/$COSMOS_USER/structured/structured_data.txt
?op=OPEN&user.name=$COSMOS_USER"
4.7 Creating application dashboards
Wirecloud is a web mashup platform aimed at empowering end users, without programming
skills, to easily create fully-fledged application dashboards built up from widgets, operators
and other pre-existing mashups. These building blocks have been made available
beforehand by developers (and even other end users in the case of pre-built mashups) in a
shared catalogue.
Wirecloud offers two different perspectives of use depending on whether you are a
developer or an end user:
As a developer, Wirecloud allows you to use cutting-edge web technologies for easily developing mashable web components, namely widgets and operators, and connecting them to backend services (e.g. support for pub/sub, “connectors” to a number of FIWARE GEs, etc.) and data sources (e.g. data APIs). Once created, Wirecloud fosters their shareability and reuse by end users.
As an end user, Wirecloud allows you to develop application dashboards without the need of programming skills (e.g. infomediaries, domain experts, etc.). These dashboards are created visually by mashing up widgets and operators, i.e. integrating heterogeneous data, application logic, and UI web components to create value added “instant” applications.
The following screenshots pertain to an application dashboard created in Wirecloud by a
domain expert to manage and operate lamp posts and reported issues (e.g. breakdowns,
sabotages) in the context of a smart city. The widgets and operators mashed up to produce
the dashboard were created and shared beforehand by developers and are connected to
backend resources by making the most of other FIWARE GEs such as the Orion Context
Broker and the Object Storage GEris thanks to specialized APIs and libraries provided by
Wirecloud.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 56
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 57
Wirecloud also offers support for mobile devices, especially for tablets, so that some of the
widgets in an application mashup can be enhanced to support native features of the device
and/or transformed into a native component to improve their performance when shown in a
mobile platform. As an example, the map in the app dashboard shown in the following
screenshot has been automatically replaced by a native component with better performance
and with access to the GPS features of the tablet.
If you are interested in more details about how to use WireCloud to (a) create your
application dashboard from pre-existing widgets and operators or (b) develop these widgets
and operators check out:
How to find widgets and other components for your dashboard How to create your application dashboard from these building blocks How to interconnect widgets and connect them to backend resources How to share, sell or make available your new dashboard How to develop new widgets and operators How to use other FIWARE GEs from your widgets and operators
4.7.1 How to find widgets and other components for your dashboard
The first step to create an application dashboard using Wirecloud is the discovering of the
components you are going to use. You can obtain those component in several ways, but
one of the easiest way is accessing the Wirecloud catalogue. This catalos is seamlessly
integrated with the FIWARE Marketplace. This marketplace is in turn made up of stores,
where they can offer and deal with Application Mashup Components (MACs), namely
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 58
widgets and operators, but also other existing mashups, they can combine to create value
added application dashboards connected to backend services and datasets. On the
marketplace, you can quickly search, find and compare widgets, operators, and even pre-
built mashups. We differentiate the marketplace from a store. While a store is owned by a
store owner who has full control over a limited widget, operator and mashup portfolio and
offerings, a marketplace is a platform for many stores to make their offerings available to a
broader audience and enable consumers to search and compare widgets, operators and
pre-built mashups and find the store, where to buy. The final business transaction (buying)
is done at the store and the whole back office process is handled by the store.
The following figure shows a screenshot of the Wirecloud catalogue, where you can see the
FIWARE Lab marketplace and the different stores made available through it:
The Wirecloud community has already made available an ever-growing catalogue of
widgets, operators and pre-built mashups that help you to create your application
dashboards. Some of them take advantage of other GEris like Kurento, CKAN or Orion. As
an example, the Kurento catalogue includes widgets for creating and supporting
videoconference features in your application dashboards. Others make use of existing open
source libraries to provide layered maps, interactive charts, tabular views, etc.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 59
4.7.2 How to create your application dashboard from these building blocks
Once obtained the initial set of components, you can start creating your dashboard by
adding some of the widgets into your workspace area:
You do not need to add those widgets in a particular order as you can always move and
resize them as you want. You can also minimize, remove, rename, and configure, etc. those
widgets.
4.7.3 How to interconnect widgets and connect them to backend resources
The wiring editor allows you to easily connect widgets in a mashup to create a fully-fledged
dashboard with RIA functionality. These widgets can be visually connected in the wiring
editor to the back-end services or data sources through an extendable set of operators,
including filters, aggregators, adapters, etc.
The wiring editor comes with a recommendation system for connections that help you to
identify compatible endpoints that could be connected.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 60
3
Once you have wired all the widgets/operators that compose the dashboard.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 61
4.7.4 How to share, sell or make available your new dashboard
Once you are finished with your dashboard, you can either describe and share it with other
people by uploading it to the Wirecloud catalogue, or offer it in a store of the FIWARE
Marketplace. You can even embed the resulting dashboard in other web pages or sites
outside Wirecloud by using the HTML code provided by the platform. For further information
regarding how to create an offering you can refer to the Store documentation. The following
figures illustrate how to obtain the HTML code needed to embed your dashboard in external
sites:
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 62
4.7.5 How to develop new widgets and operators
Widgets can be split up into three different components:
Descriptor (config.xml), which is a declarative description of the widget. This file contains, among other things, references to the rest of resources of a widget.
Code, composed of HTML, JavaScript and CSS files containing the definition and the behaviour of the widget.
Static resources, such as images, documentation and other static resources.
The following figure shows the structure of files for a widget, which is distributed as a zip
package of such structure with .wgt extension.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 63
There are three types of operators:
Data sources operators: Operators that provide information that can be consumed by other widgets/operators. For example, an operator that retrieves some type of information from a web service.
Data targets operators: Operators that are provided information and use it to do some tasks. For example, an operator that receives some information and push it to a web service.
Data transformation operators: This type of operators can be very useful since they can transform data in order to make it usable by widgets or operators that expect data structure to be slightly different.
Operators use the same structure as widgets. The only difference is that the descriptor file
(config.xml) does not link to an initial HTML document. Instead, it directly links to the list of
javascript files.
4.7.6 How to make AJAX requests to external services
We will use the Weather Underground service to show how to make AJAX requests to third
party services. This service provides a rest API (documented at
http://www.wunderground.com/weather/api/d/docs), but it is impossible to access this API
using normal AJAX request (using XMLHttpRequest) due browsers applying the “Same
Origin Policy” to javascript code. Fortunately, WireCloud provides the
MashupPlatform.http.makeRequest method for dealing with this problem. A possible way to
access to this API is by using the following code:
var getForecastByCoord = function getForecastByCoord(coord, onSuccess, onError) { var url; if ((typeof onSuccess !== 'function') || (typeof onError !== 'function')) { throw new TypeError(); }
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 64
url = 'http://api.wunderground.com/api/' + API_KEY + '/conditions/forecast/q/'; url += coord.lat + ',' + coord.lon; url += '.json'; MashupPlatform.http.makeRequest(url, { method: 'GET', onSuccess: function (response) { var forecast_data; forecast_data = JSON.parse(response.responseText); if (forecast_data.error) { onError(); } else { onSuccess(forecast_data); } }, onError: function () { onError(); } }); };
The getForecastByCoord function makes the appropriated request to Weather Underground
and passes the result to the onSuccess callback.
4.7.7 How to receive events
Input endpoints must be declared into the widget template before it can be used by the
javascript code of the widget. To do so, open config.xml and add an InputEndpoint element
into the Platform.Wiring section. The final result should look like:
... <wiring> <inputendpoint name="coord" type="text" label="Show forecast by coord" description="Shows the weather forecast for a given location (a latitude
longitude coordinate)." friendcode="location" /> </wiring> ...
This is how to declare the input endpoint when using RDF (turtle):
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 65
... wire:hasPlatformWiring [ a
<http://WireCloud.conwet.fi.upm.es/ns/widget#PlatformWiring>; wire:hasInputEndpoint [ a
<http://wirecloud.conwet.fi.upm.es/ns/widget#InputEndpoint>; rdfs:label "Show forecast by coord"; dcterms:description "Shows the weather forecast for a given location
(a latitude longitude coordinate)."; dcterms:title "coord"; wire:friendcode "location"; wire:type "text" ] ]; ...
Once declared the input endpoint in the widget template, you can register a callback for this
endpoint making use of the MashupPlatform.wiring.registerCallback method. In addition to
registering the input endpoint, we need to process event data before using it and to notify
the user that the forecast data for the given location is being requested. This can be
accomplished by using the following code:
var searchByCoordListener = function searchByCoordListener(event_data) { var tmp, coord; tmp = event_data.split(','); coord = { lat: tmp[1], lon: tmp[0] }; startLoadingAnimation(); getForecastByCoord(coord, function (forecast_data) { updateWeatherForecast(forecast_data); stopLoadingAnimation(); }, function () { clearWeatherForecast(); stopLoadingAnimation(); }); }; MashupPlatform.wiring.registerCallback("coord", searchByCoordListener);
4.7.8 How to send events
As we did with the input endpoint, we need to declare the new output endpoint in the
weather widget's template. This is the final result of the Wiring section after adding it:
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 66
... <wiring> <inputendpoint name="coord" type="text" label="Show forecast by coord" description="Shows the weather forecast for a given location (a latitude
longitude coordinate)." friendcode="location" /> <outputendpoint name="location_coord" type="text" label="Forecast location" description="This event is launched when the user clicks on the location name
of current forecast." friendcode="location" /> </wiring> ...
This is how to declare the output endpoint when using RDF (turtle):
... wire:hasPlatformWiring [ a
<http://wirecloud.conwet.fi.upm.es/ns/widget#PlatformWiring>; wire:hasInputEndpoint [ a
<http://wirecloud.conwet.fi.upm.es/ns/widget#InputEndpoint>; rdfs:label "Show forecast by coord"; dcterms:description "Shows the weather forecast for a given location
(a latitude longitude coordinate)."; dcterms:title "coord"; wire:friendcode "location"; wire:type "text" ] ]; wire:hasOutputEndpoint [ a
<http://wirecloud.conwet.fi.upm.es/ns/widget#OutputEndpoint>; rdfs:label "Forecast location"; dcterms:description "This event is launched when the user clicks on
the location name of current forecast."; dcterms:title "location_coord"; wire:friendcode "location"; wire:type "text" ]; ...
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 67
After adding the output endpoint to the widget description, we can send data through it
using the MashupPlatform.wiring.pushEvent method. The following code adds an event
listener to the location title that sends the location of the current forecast.
4.7.9 How to use other FIWARE GEs from your widgets and operators
Wirecloud offers support for a number of GEs including the Context Broker and the Object
Storage GE. What follows is an example of how a developer can access Orion Context
Broker (reference implementation of the FIWARE's Pub/Sub Context Broker GE) instances
from widgets and operators running on Wirecloud. For more detailed information on how to
use this and other FIWARE GEs from Wirecloud, please refer to Wirecloud tutorials.
First of all, widgets and operators wishing to use the javascript bindings provided by
Wirecloud for accessing the FIWARE NGSI Open RESTful API in order to seamlessly
interoperate with the Orion Context Broker must add the NGSI feature as a requirement into
their description files (config.xml files).
The following is an example of a widget description using the XML flavour of the WDL:
<?xml version='1.0' encoding='UTF-8'?> <widget xmlns="http://wirecloud.conwet.fi.upm.es/ns/macdescription/1" vendor="CoNWeT"
name="observation-reporter" version="1.0"> <details> <title>Observation Reporter</title> <authors>aarranz</authors> <email>[email protected]</email> <image>images/catalogue.png</image> <smartphoneimage>images/smartphone.png</smartphoneimage> <description>Creates a new observation</description> <doc>http://www.envirofi.eu/</doc> </details> <requirements> <feature name="NGSI"/> </requirements> <wiring/> <contents src="index.html" contenttype="text/html" charset="utf-8"
useplatformstyle="true"/> <rendering height="20" width="5"/> </widget>
The RDF/xml flavour of the same widget description is:
<?xml version="1.0" encoding="utf-8"?> <rdf:RDF xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:wire="http://wirecloud.conwet.fi.upm.es/ns/widget#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 68
xmlns:usdl="http://www.linked-usdl.org/ns/usdl-core#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:ns1="http://purl.org/goodrelations/v1#" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:vcard="http://www.w3.org/2006/vcard/ns#" > <wire:Widget
rdf:about="http://wirecloud.conwet.fi.upm.es/ns/widget#CoNWeT/observation-
reporter/1.0"> <vcard:addr> <vcard:Work rdf:nodeID="Nb17ce611aa2645e488515f86eb855e53"> <vcard:email>[email protected]</vcard:email> </vcard:Work> </vcard:addr> <usdl:utilizedResource> <usdl:Resource rdf:about="index.html"> <wire:codeCacheable>True</wire:codeCacheable> </usdl:Resource> </usdl:utilizedResource> <wire:hasPlatformWiring> <wire:PlatformWiring rdf:nodeID="Neecb97db81ed40859b8c04e935a9a9cc"/> </wire:hasPlatformWiring> <wire:displayName>Observation Reporter</wire:displayName> <wire:hasiPhoneImageUri rdf:resource="images/smartphone.png"/> <usdl:versionInfo>1.0</usdl:versionInfo> <usdl:hasProvider> <ns1:BusinessEntity rdf:nodeID="N9a6bf56577c741ac806997a80281afff"> <foaf:name>CoNWeT</foaf:name> </ns1:BusinessEntity> </usdl:hasProvider> <wire:hasImageUri rdf:resource="images/catalogue.png"/> <wire:hasPlatformRendering> <wire:PlatformRendering rdf:nodeID="N713e5ea11dce4750a592c754c748def7"> <wire:renderingHeight>20</wire:renderingHeight> <wire:renderingWidth>5</wire:renderingWidth> </wire:PlatformRendering> </wire:hasPlatformRendering> <wire:hasRequirement> <wire:Feature rdf:nodeID="N3cb336bd9b6243ecbf345c80442498f9"> <rdfs:label>NGSI</rdfs:label> </wire:Feature> </wire:hasRequirement> <dcterms:title>observation-reporter</dcterms:title> <dcterms:description>Creates a new observation</dcterms:description> <dcterms:creator> <foaf:Person rdf:nodeID="Ndb72cb5a7f3844b29b72f304baaa14a7"> <foaf:name>aarranz</foaf:name> </foaf:Person>
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 69
</dcterms:creator> </wire:Widget> </rdf:RDF>
One of the most important operations provided by the context broker is the support for
subscriptions. By using subscriptions our dashboard can obtain "real time" notifications
about the status of the entities of interest. Subscriptions are very similar to queries, the main
difference between queries and subscriptions is that queries are synchronous operations,
whilst subscriptions are asynchronous. Moreover, the Orion Context Broker will send a first
notification containing the data that would be returned for the equivalent query operation.
This way, you will know that there is no gap between the current values and the notified
changes.
Subscriptions are created through the createSubscription method. The following example
explains how to be notified about the changes of the position of the vans we are managing:
var entityIdList = [ {type: 'Van', id: '.*', isPattern: true} ]; var attributeList = null; var duration = 'PT3H'; var throttling = null; var notifyConditions = [{ type: 'ONCHANGE', condValues: ['current_position'] }]; var options = { flat: true, onNotify: function (data) { // called when the context broker sends a new notification }, onSuccess: function (data) { ngsi_subscriptionId = data.subscriptionId; } }; var ngsi_connection = new NGSI.Connection(ngsi_server[, options]); ngsi_connection.createSubscription(entityIdList, attributeList, duration, throttling, notifyConditions, options);
In this example, the call to createSubscription will make the context broker invoke the
onNotify function each time the 'current_position' attribute of the entities of type 'Van' is
changed. You must take into account that the Orion Context Broker evaluates patterns at
runtime, so using patters you will be able to receive notification about new entities provided
that the notify conditions are meet.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 70
This subscription will be expiring after 3 hours, time from which the context broker will stop
sending notifications. Anyway, widgets/operators can renew those subscriptions by using
the updateSubscription method, even if they have expired. Subscriptions can be cancelled
using the cancelSubscription method making the context broker release any info about the
subscription. In any case, WireCloud will cancel any subscription automatically when
widgets/operators are unloaded.
As an example, what follows is the value of the data parameter passed to the onNotify
callback when using the flat option:
{ "elements": { "van2": { "id": "van2", "type": "Van", "current_position": "43.47258, -3.8026643" }, "van4": { "id": "van4", "type": "Van", "current_position": "43.471214, -3.7994885" } }, "subscriptionId": "53708768286043030c116e2c", "originator": "localhost" }
Wirecloud tutorial on how to use Orion Context Broker from widgets and operators further
explain how to create connections, make queries, deal with pagination, create entities and
update their attributes, use geolocation capabilities, etc. It also includes a complete
example. Please also refer to the NGSI javascript API documentation for a more detailed
description of the API calls used along the tutorial.
4.8 Real-time processing of media streams
Your application may also need to integrate multimedia information. Multimedia information
comprises audio and video data, which is commonly used for the exchange of complex
information among humans (i.e. video conferencing, video-clip sharing, multimedia instant
messaging, etc.) In addition to this, in the last few years, cameras and microphones are
often used as advanced sensors that, combined with computer vision and other analysis
techniques, may generate rich information useful in different application areas including
eHealth, smart cities, security and defence, entertainment, etc.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 71
Incorporating into your application such features may be done using the Real-time
Multimedia Stream processing GE (reference implementation Kurento). In particular, you
may be interested in the following usage patterns
Multimedia communications.
You application may take advantage of this pattern whenever the introduction of human-to-
human communications on it makes sense. That communication may take place following
different information topologies such as person-to-person (i.e. traditional call model),
person-to-people (i.e. from one sender to several receivers), people-to-people (i.e. group
model where all participants can send and receive the information). The communication can
be complemented with other features provided by the enabler such as media recording (i.e.
storing in a persistent way the media stream of one or several people) and media playing
(i.e. integrating into the communication clips previously recorded).
Multimedia sensor based on computer vision.
This pattern is based on using computer vision or speech analysis techniques to transform
a device camera into an advanced sensor. Kurento provides, off-the-shelf, some basic
computer vision capabilities for detecting faces, detecting crowds, detecting car plates,
detecting movement, etc. However, you can create your own analysis modules and plug
them into Kurento in a simple and seamless way. Note that most of these analyses lead to
generation of data that takes the form of events (e.g. detected a specific face, detected a
car plate, detected an intruder, etc.) You may want to manage these events as Context
Information. You can do this by means of updates on context information using operations
exported by the Context Broker GE.
Multimedia enrichment basing on augmented reality.
This pattern is useful when combined with multimedia communications to generate
applications where the capabilities of the people receiving multimedia information are
augmented by the system. Augmented reality basically consists on adding layers of
information on top of the basic media stream captured by the camera. This makes possible
for the receiver to see information that is beyond the reality being communicated. This may
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 72
be useful in different scenarios including, for example, entertainment scenarios (e.g. adding
costumes to people in a conference), in learning scenarios (e.g. augmenting the teaching
lessons with animations or texts), in health (e.g. augmenting the communication with a
patient with biometric data), in security (e.g. augmenting information on the identity of the
people captured by a camera), in industrial applications (e.g. adding instructions on how to
manage or repair a complex machine), in advertising (e.g. adding customized ads to the
media stream depending on context or on other available data), etc.
If you are interested in learning more about the Real-time Multimedia Stream Processing
GE and its reference implementation, Kurento, you may find useful the following
information:
What’s WebRTC and what’s a media server
What’s the architecture of a Kurento enabled application
How to develop a Kurento enabled application
4.8.1 What’s WebRTC and what’s a media server
WebRTC is an open source technology that enables web browsers with Real-Time
Communications (RTC) capabilities via JavaScript APIs. WebRTC is currently under
standardization at the IETF and W3C and has the support of the most important companies
in the area of Internet and telecommunications. WebRTC has been conceived as a peer-to-
peer technology where browsers can directly communicate without the mediation of any
kind of infrastructure. This model is enough for creating basic applications but features such
as group communications, media stream recording, media broadcasting or media
transcoding are difficult to implement on top of it. For this reason, many applications require
using a media server.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 73
Conceptually, a WebRTC media server is just a kind of “multimedia middleware” (it is in the
middle of the communicating peers) where media traffic pass through when moving from
source to destinations. Media servers are capable of processing media streams and offering
different types including groups communications (distributing the media stream one peer
generates among several receivers, i.e. acting as Multi-Conference Unit, MCU), mixing
(transforming several incoming stream into one single composite stream), transcoding
(adapting codecs and formats between incompatible clients), recording (storing in a
persistent way the media exchanged among peers), etc.
Kurento, the Real-time Processing Stream Oriented GE, has been conceived as a WebRTC
capable media server. At the heart of the Kurento architecture there is a media server called
the Kurento Media Server (KMS). Kurento Media Server is based on pluggable media
processing capabilities meaning that any of its provided features is a pluggable module that
can be activated or deactivated. Moreover, developers can seamlessly create additional
modules extending Kurento Media Server with new functionalities which can be plugged
dynamically.
Kurento Media Server provides, out of the box, group communications, mixing, transcoding,
recording and playing. In addition, it also provides advanced modules for media processing
including computer vision, augmented reality, alpha blending and much more. Besides,
Kurento Media Server is capable of receiving video streams from most types of IP cameras
supporting the RTSP or HTTP protocols.
4.8.2 What’s the architecture of a Kurento enabled application
Kurento Media Server capabilities are exposed by the Kurento API to application
developers. This API is implemented by means of libraries called Kurento Clients. Kurento
offers two clients out of the box for Java and JavaScript. If you have another favourite
language, you can still use Kurento using directly the Kurento Protocol. This protocol allows
to control Kurento Media Server and it is based on Internet standards such as WebSocket
and JSON-RPC. The picture below shows how to use Kurento Clients in three scenarios:
Using the Kurento JavaScript Client directly in a compliant WebRTC browser
Using the Kurento Java Client in a Java EE Application Server
Using the Kurento JavaScript Client in a Node.js server
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 74
Kurento Client’s API is based on the concept of Media Element. A Media Element holds a
specific media capability. For example, the media element called WebRtcEndpoint holds the
capability of sending and receiving WebRTC media streams, the media element called
RecorderEndpoint has the capability of recording into the file system any media streams it
receives, the FaceOverlayFilter detects faces on the exchanged video streams and adds a
specific overlaid image on top of them, etc. Kurento exposes a rich toolbox of media
elements as part of its APIs.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 75
4.8.3 How to develop a Kurento enabled application
From the application developer perspective, Media Elements are like Lego pieces: you just
need to take the elements needed for an application and connect them following the desired
topology. In Kurento jargon, a graph of connected media elements is called a Media
Pipeline. Hence, when creating a pipeline, developers need to determine the capabilities
they want to use (the media elements) and the topology determining which media elements
provide media to which other media elements (the connectivity). The connectivity is
controlled through the connect primitive, exposed on all Kurento Client APIs. This primitive
is always invoked in the element acting as source and takes as argument the sink element
following this scheme:
sourceMediaElement.connect(sinkMediaElement)
For example, if you want to create an application recording WebRTC streams into the file
system, you’ll need two media elements: WebRtcEndpoint and RecorderEndpoint. When a
client connects to the application, you will need to instantiate these media elements making
the stream received by the WebRtcEndpoint (which is capable of receiving WebRTC
streams) to be feed to the RecorderEndpoint (which is capable of recording media streams
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 76
into the file system). Finally you will need to connect them so that the stream received by
the former is fed into the later:
WebRtcEndpoint.connect(RecorderEndpoint)
To simplify the handling of WebRTC streams in the client-side, Kurento provides an utility
called WebRtcPeer. Nevertheless, the standard WebRTC API (getUserMedia,
RTCPeerConnection, and so on) can also be used to connect to WebRtcEndpoints. For
further information please visit our Kurento tutorials.
4.9 Providing an advanced User Experience (UX)
You may want to incorporate advanced features in your web-based user interface, e.g.,
Augmented Reality or 3D visualization features. Then we recommend you to take a look to
the FIWARE GEs that have been defined in the Advanced Web-based User Interface
chapter. These components have recently been incorporated in FIWARE so we are also
working on their integration with other FIWARE GEs. For example, you soon will be able to
fill information linked to POIs (Points Of Interest) visible in your user interface using
FIWARE Advanced Web-based User Interface GEs with context information provided by the
Context Broker GE.
Furthermore, your web based user interface can be a real-time collaborative application
utilizing the Synchronization GE. It features automatic synchronization of your application or
UI data between multiple participants. This synchronization is already integrated with the
3D-UI GE so that your 3D web application is a multiuser space out of the box. This allows
both the creation of collaborative applications such as project work spaces and meeting
tools, and realtime multiplayer games.
The following diagram illustrates Advanced Web UI architecture and its GEs.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 77
The diagram shows how all the GEs from the WebUI chapter fit and can be used together.
It is also possible and often suitable to use the GEs individually. For example, if you make a
single user traditional form based GUI application for a mobile phone it can still use the POI
GE for storing and retrieving geopositioned information over standard HTTP. Or if you wish
to add 3D elements to your web page, without multiuser functionality nor extra data
backends, you can just use the 3D-UI GE by itself.
You can use FILAB to test and develop on the advanced Web UI technologies. It is also
possible to install them on your own machines and use web hosting services. For the
Synchronization GE for multiuser collaborative applications there is a commercial hosting
provider Meshmoon.com which is suitable for production use in business. Meshmoon is
also free to try: you can create an own collaborative 3D scene in a minute!
Below we show discuss two different examples of using the Enablers WebUI
4.9.1 3D-UI: Interactive 3D graphics and Augmented Reality in any Web browser
The 3D-UI GE allows you to add interactive 3D elements to a web page, or deliver a full
blown 3D application on Web technology. Under the hood it utilizes WebGL for hardware
accelerated graphics rendering. However the application developer is provided with a
friendly high-level API for describing the 3D graphics instead. You just add the 3D elements
in a so-called scene graph and the underlying rendering engine takes care of the drawing.
The GE is provided as a Javascript library to be used in your web application.
You can either easily load existing 3D models from files or create them with Javascript or
XML embedded in the HTML page. Interactivity is programmed in high level Javascript
similarly to all Web development.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 78
When using externally made 3D files, typically meshes for the geometry with associated
material definitions and texture images, there are certain file formats that you can use to
load them to FIWARE 3D-UI implementations. You can use normal modelling applications
such as 3D Studio Max, Maya, Blender or Sketchup etc. to author to these formats. There
are also multiple both free and for-free model repositories from where you can get suitable
ready-made models. In 3D these files are often called ‘assets’, and the export & import tools
are an ‘asset pipeline’ and the file collections ‘asset stores’. We document the details of the
supported asset pipelines in the GE manuals.
We provide two alternatives for this easy high level Web 3D authoring: Three.js and
XML3d.js. They are similar in purpose and scope but differ in implementation and usage. A
brief overview of both follows, with more details in dedicated sections later and further in the
manuals.
With Three.js everything is done in Javascript: you use the API to create the scene and add
the objects there. Only a single <canvas> element is put to the HTML document and even
that is done automatically by the library. Three.js is the most popular 3D solution on the
Web and there are huge amounts of examples and demos made with it. It is also used in
several serious commercial services such as Microsoft’s Photosynth2 and Sketchup’s Web
viewer. Yet three.js is just a personal project lead by the original individual developer in a
very informal manner. It’s very successful and active open source project with contributions
(mostly small and occasional) from hundreds of developers. As a part of FIWARE we help
users of this GE implementation by providing a tested stable version, documentation and
improvements to the library where needed.
XML3D.js is the reference implementation of the XML3D specification to add 3D support to
HTML with a specific set of new tags. This way you can add 3D elements to a web page
without writing any Javascript. Similar to the pre-existing <img> and <video> tags, <mesh>
works to add a 3D object to the view. The 3D elements are added to the DOM and can be
manipulated via it like other HTML elements. XML3D.js also includes an implementation of
XFlow, a new approach for declarative data processing. The library also features advanced
geometry loading to deal with huge amounts of details (millions of polygons) with streaming
and level of detail techniques (BLAST asset bundle with the POP-buffers geometry format).
XML3D.js has been lately adopted in some projects in the industry as well.
Finally, the WebTundra bundle integrates 3D-UI with the Synchronization GE for creating
real-time collaborative applications using WebGL together with WebSockets. WebTundra is
built using a Model-View-Controller architecture where the rendering and networking are
separate modules built on an abstract core scene API (the model). Currently in the 1.0
release (in FIWARE 3.3) WebTundra includes and defaults to a Three.js view
implementation. A XML3D.js view to WebTundra was also proven feasible in an experiment.
Currently WebTundra includes an independent partial XML3D implementation: you can
declare a 3D scene view in HTML in XML3D and it gets displayed by the Three.js view.
However WebTundra does currently not populate the 3D scene to the DOM for performance
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 79
reasons (when dealing with large 3d scenes). Use of WebTundra for networked multiuser
3d applications is also covered in a dedicated section.
For an independent comprehensive review of 3D graphics on the net, featuring both three.js
and xml3d.js but also technologies, see:
Evans, Alun, et al. "3D Graphics on the Web: a Survey." Computers & Graphics 41 (2014): 43-61.
http://www.sciencedirect.com/science/article/pii/S0097849314000260
4.9.2 XML3D: Interactive 3D graphics and Augmented Reality via DOM extensions
XML3D is an extension to HTML5 for declarative 3D content represented as a scene graph
like structure inside the DOM. All nodes within this graph are also nodes in the web sites
DOM tree representation and can be accessed and changed via JavaScript like any other
common DOM elements as well. On these DOM nodes, HTML events can be registered
similar to known HTML elements. Resources for mesh data can be stored externally in any
kind of external format (e.g. JSON, XML or binary) and referenced by URL.
This allows assembling your 3D application easily and directly as part of your website. 3D-
UI-XML3D comes with a large set of examples to show its basic usage. XML3D is
implemented as polyfill implementation and provided in a single JavaScript file that, when
linked to the website, provides access to the full feature set of XML3D.
With XML3D comes Xflow, a declarative data flow processing approach that allows complex
computations on your 3D scene. Xflow is directly implemented in xml3d.js and highly
optimized towards operations on XML3D scenes. Applications range from computing object
transformations in key frame animations to skinning of 3D models for skeletal animations.
Outside the application field of 3D scene modification, Xflow is also a great tool to perform
sophisticated image processing tasks.
4.9.2.1 How to set up a first XML3D scene
A set of example applications is deployed with your instance of XML3D when you setup
your instance of XML3D via the FI-Lab recipe. The same set of example is also available
online in the xml3d-examples github repository:
http://xml3d.github.io/xml3d-examples/
Setting up a 3D scene with XML3D is simple. All you need is a web server to which the
website can be deployed. If you install the 3D-UI-XML3D-GEri directly from the recipe, an
instance of an Apache2 server is already deployed.
To enable XML3D in your website, just add a link to the xml3d.js script file to your website’s
HTML.
A detailed tutorial of how to set up 3D scenes is given in the XML3D eLearning courses and
the 3D-UI-XML3D-GEi User and Programmer’s guide.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 80
4.9.2.2 How to use Xflow for AR applications with XML3D and the Augmented Reality GE
Xflow, which is part of xml3d.js, can not only be used to modify 3D scenes based for
example on vertex data. It can also be employed to perform image processing
computations, both on pixel data of static images and on data of video streams.
The latter builds the foundation for marker tracking in Augmented Reality (AR) applications.
Like for the 3D scene examples, a set of examples for image processing is deployed with
your instance of the 3D-UI-XML3D GE right away. This includes example applications that
overlay a video stream with an XML3D object. For the application, it makes no difference
whether the image data comes from a pre-computed video, or from an actual webcam video
stream.
The use-case of Xflow for marker tracking in video streams is implemented in the
Augmented Reality Generic Enabler. This Generic Enabler provides a simple interface to
define markers that should be tracked and to change camera or object transformations
based on a tracked marker in the scene.
4.9.3 3D graphics on the Web with Three.js
Gallery of applications: http://threejs.org/ Examples catalogue: http://threejs.org/examples/ Is integrated in WebTundra with Synchronization GE
o Also XML3D support Can be used also standalone for single user applications (normal web pages with
3d)
4.9.4 Multiuser 3D with WebTundra (Games, Virtual Worlds, Collaborative
applications)
WebTundra GE is a Javascript library that uses three.js and Synchronization GE to
implement a Tundra client and WebGL rendering. It takes care of asset loading, mapping
three.js objects to Tundra EC:s, handling networked changes to these in real time For a
preview about what you can do with three.js, look in http://threejs.org/examples/
For creating a WebTundra application, start from an existing three.js WebGL application.
Replication of objects and changes to their state are handled by the network-aware Entity-
Component system. For each network-aware three.js 3D geometry object, an entity with
Placeable and Mesh components is created. Mesh component loads the 3D asset off the
web and creates a corresponding three.js mesh in the scene. Changing Placeable attributes
like position, scale, rotation are synchronized to other clients via the server (and changed in
the corresponding three.js object).
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 81
For setting up, you need to have a server to connect to; either FI-LAB or own machine, we
also have a server running you can use right away. Set up your local development
environment and download WebTundra, or develop a client on a FI-LAB vm - we have an
installation recipe:
https://forge.fi-ware.org/scmrepos/svn/testbed/trunk/cookbooks/GESoftware/webtundra/recipes/
4.10 Hosting your application on a FIWARE Cloud
All the application components running in a centralized data center can be provisioned and
managed using the FIWARE Cloud capabilities. The preferred way of consuming these
capabilities is via one of the existing FIWARE Lab nodes, although the reference
implementation of the underlying GEs can be also deployed in your own data center
(requiring corresponding hardware and skills).
The foundation of the FIWARE Cloud is an Infrastructure as a Service platform based on
OpenStack, comprising the Compute (Nova), Storage (Cinder), Network (Neutron) and
Image (Glance) services.
The easiest way to start using the FIWARE Cloud is to open the FIWARE Cloud Portal, and
create one or more Virtual Machine instances in one of the FIWARE Lab nodes. The
catalogue of Virtual Machine images includes general purpose operating systems (such as
Ubuntu Server 14.04) as well as pre-packaged virtual appliances providing capabilities of
individual GEs (e.g., Orion Context Broker or CEP). Moreover, you can allocate persistent
storage volumes and attach them to VM instances to keep the persistent data of your
application. Once you have your virtual machines up and running, you can log into the
individual VMs (typically via SSH), and configure your application.
Once you need to deploy your application in production and/or at scale, you can leverage
the advanced orchestration capabilities provided by FIWARE Cloud, associated with
provisioning and lifecycle management of application blueprints. Blueprint is a composition
of virtual infrastructure resources, software components deployed in VMs and inter-relations
between them, provisioned and managed in an automated manner. For example, a
blueprint can represent a 3-tier application, comprising a Web tier, an application tier and a
database tier. Each tier may potentially have a different ‘base’ Virtual Machine image, with a
particular set of software components installed on top of the ‘base’ operating system,
selected from a dedicated software configuration management catalogue (based on Chef
standard). It is also possible to accompany a blueprint tier with a set of auto-scaling rules
that would trigger adding or removing VMs to/from a tier to fit the changing resource
demand of the application (based on ongoing monitoring of certain metrics).
FIWARE Cloud also provides Object Storage facility (based on OpenStack Swift), that can
be used to store and retrieve ‘blob’ objects and associated metadata.
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 82
Being fully compatible with the OpenStack standard, all the above capabilities can be also
consumed via a RESTful API and via a command line, although this usage is intended for
advanced users and operators.
Also, one of our goals is to contribute OpenStack enhancements developed in FIWARE
back to the OpenStack community, so that they become broadly available through multiple
commercial vendors.
You can find more details on how to use the different FIWARE Cloud capabilities in the
following sub-sections:
How to provision and manage your virtual infrastructure on FIWARE Cloud How to use Object Storage capabilities of FIWARE Cloud How to use Blueprints to manage complex applications hosted on FIWARE Cloud How to invoke FIWARE Cloud capabilities via an API
4.10.1 How to provision and manage your virtual infrastructure on FIWARE Cloud
FIWARE Cloud provides advanced IaaS capabilities based on OpenStack, including
Compute (Nova), Storage (Cinder), Network (Neutron) and Image (Glance) services. The
services are available both via OpenStack APIs and via the FIWARE Cloud Portal.
OpenStack Compute (Nova) service can be used to provision Virtual Machine instances
by specifying the virtual machine configuration ‘flavor’ (from a catalogue of pre-defined
flavors configured by the administrator) and virtual machine image, which comprise a pre-
packaged Operating System and a software stack managed by OpenStack Image
(Glance) service. FIWARE Lab holds a rich catalogue of Virtual Machine images, including
base Operating Systems (such as Ubuntu or Red Hat) as well as images comprising pre-
packaged installations of the various Generic Enabler implementations. It is possible to
automatically customize the guest Operating System configuration on the first boot of the
Virtual Machine, as well as to control the power state of the Virtual Machine (power on/off),
suspend/resume, take snapshots, etc. The availability of some of the functions depend on
the capabilities of the underlying virtualization layer. OpenStack Storage (Cinder) service
can be used to provision persistent storage volumes, allocated from one of the storage
pools configured by the administrator. These volumes can be then attached to a Virtual
Machine instance, to hold persistent data of an application. OpenStack Network (Neutron)
service is used to provision and manage virtual networks, which can be used to define
connectivity domains and attach Virtual Machine instances to them, as well as virtual
routers, used to control connectivity between virtual networks and with the Internet.
More details on how to work with the FIWARE Cloud infrastructure via the FIWARE Cloud
Portal can be found at http://edu.fi-ware.org/mod/scorm/view.php?id=90
and https://www.youtube.com/watch?v=mb7CdMnZh7Q
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 83
4.10.2 How to use Object Storage capabilities of FIWARE Cloud
The FIWARE Object Storage allows storing and retrieving ‘blob’ objects via OpenStack
Swift RESTful API (or via the FIWARE Cloud Portal). Objects are organized in containers
(aka folders), and can include arbitrary key-value metadata. The FIWARE Object Storage
provides high resiliency by keeping 3 copies of each object. Moreover, the underlying
implementation (based on OpenStack Swift) can scale up to many PBs of data (while the
disk space available in the experimental FIWARE Lab environment is typically more
limited).
More details on accessing the FIWARE Object Storage can be found in the following
tutorial: https://www.youtube.com/watch?v=M0plEExFQgI
4.10.3 How to use Blueprints to manage complex applications hosted on FIWARE
Cloud
When progressing from small experimental environments to a more extensive use of the
cloud infrastructure, it is often required to automate the provisioning and life cycle of
complex applications comprising multiple resources -- such as multi-tier applications with
several inter-dependent Virtual Machines, each running a particular software stack and
configured to interact with the other Virtual Machines. FIWARE Cloud provides advanced
capabilities to manage application blueprints, comprising one or more application tiers.
Each tier may consist of an elastic group of Virtual Machine instances, with auto-scaling
rules (enforced by the Policy Manager GE) that determine when Virtual Machines will be
added or removed to/from the group, typically based on the load on the corresponding tier
(monitored via a dedicated instrumentation provided by the Monitoring GE). Moreover,
each tier can define the software packages that should be configured within the guest
Operating System as part of the Virtual Machine instance provisioning. The software
packages and their configuration are managed by the Software Deployment and
Configuration GE, which is based on (and compatible with) Opscode Chef -- a widely used
open source software configuration management tool. The combination of these GEs,
surfaced in a holistic manner via the PaaS Management GE (and available in the FIWARE
Cloud Portal), provides unique value proposition currently not available in other OpenStack-
based clouds. It is worth noticing that the FIWARE team is working with the OpenStack
community to contribute the necessary enhancements to make our technology broadly
available.
More details on working with blueprints (via the FIWARE Cloud Portal) can be found at
http://edu.fiware.org/course/view.php?id=47
and https://www.youtube.com/watch?v=MdZS_0jt6Vo
Future Internet Core Platform
D.11.1.1: FIWARE Technical Whitepaper 84
4.10.4 How to invoke FIWARE Cloud capabilities via an API
All the FIWARE Cloud capabilities outlined above can be invoked via OpenStack APIs, with
unique extensions currently available only in FIWARE. A typical interaction would start by
authenticating against the Identity Management GE, and getting a token that can be used in
consequent calls to the individual services. For standard OpenStack APIs, one can use an
existing client library - one of the many maintained in the open source community.