adv fiware workshop starring docker and virtualisation
TRANSCRIPT
Johann Höchtl15. April 2016
Österreichische Adressdatenauf FIWAREFIWARE Workshop – FIWARE kennenlernen und nutzen
15 April 2016 – ATOS IT SolutionsSiemensstraße 92, 1210 Wien
15. April 2016 2
Motivation
Image CC https://flic.kr/p/b3VC6X
1. What is FIWARE?(and what is my money spent on?)
2. Save money by not sending it to Heroku <INSERT_CLOUD_PROVIDER>
3. Learn sth. newon a real world challenge
15. April 2016 3
15. April 2016 5
15. April 2016 6
15. April 2016 7Image CC https://flic.kr/p/7ajM9d
How the EU and giants like Telefonica or IBM are wasting
€100 millionand entrepreneurs’ time
https://novobrief.com/fiware-european-union-telefonica-ibm-waste-of-time/
15. April 2016 8
FIWARE Facts & Funding(http://de.slideshare.net/chicco785/fiware-primer-learn-fiware-in-60-minutes/10, November 2015)
Budget for FIWARE/FIWARE Lab (phases 1 & 2)126 M€ = (89 + 37) M€
countries
270 150companiespartners
400 M€ = (300 + 100) M€Budget (RTD + funding for entrepreneurs)
>1.000SMEs and start-ups
27
15. April 2016 9
What is FIWARE?(Credit http://tinyurl.com/fiware-open-ecosystem )
Open StandardPlatform
ecosystemopen globalsustainable
OpenStack-based Cloud & Library of Generic Enablers
support developers to create Internet
applications and startups based on FIWARE
establish worldwide links that help FIWARE to
thrive beyond European borders
15. April 2016 10
FIWARE Generic Enablers
Publish/Subscribe Context Broker - Orion Context Broker
Complex Event Processing (CEP) - Proactive Technology Online
Identity Management - KeyRock
PEP Proxy – Use the FIWARE account for authentication
Docker
15. April 2016 11
Openstack – Overview & FIWARE Usage
NOVA
IMAGE SERVICE
OBJECT STORE
VOLUME SERVICE
IDENTITY SERVICE
GLANCE SWIFT CINDER
KEYSTONE
OPST0005
COMPUTE NETWORKING
NEUTRON HEAT
Orchestration
CEILOMETER
Telemetry
DASHBOARD
HORIZON
15. April 2016 12
Example: Smart City Domain(Credit: http://de.slideshare.net/flopezaguilar/introduction-to-fiware-open-ecosystem/44)
CKAN
Big Data
Context Broker
Accounting & Paym
en t &
Billing
IDM
& Aut h
Short-term
historic data
BigData Processing
Data Quering/Actio
n, Publish/Subsc
r
Open Data publishing
Real-time processing
BIETL
RULESDEFINITION
TOOL
OPEN DATA PORTALS
Service orchestra
torContext
Adapters
CEP
IoT Backend Device
Managementmeasures / commands
IoT Broker & Config Management
(from sensors to things)
IoT/Sensor
Open Data
actuators
Media streams
Real Time
Media Stream
Processing
City Services
OPERATIONAL
DASHBOARD
KPI GOVERNANCE
GISInventory
Specific Enablers
Generic Enablers
15. April 2016 13
FIWARE GE Future Scenario
FIWARE Catalogue
creation of dedicated GE instances
Use of Global shared GE instance
FIWARE provider A
Applications
FIWARE provider B
Applications
15. April 2016 14
Getting go - Register
https://account.lab.fiware.org/
1 get account
2 get (almost) nothing
https://ask.fiware.org/question/6/do-i-need-to-be-part-of-an-accelerator-project-in-order-to-use-fiware/
15. April 2016 15
Apply for Community Account
15. April 2016 16
Computation Flavours
https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/FIWARE_Lab:_Upgrade_to_Community_Account#Quotas_and_Flavours
Flavour ID Flavour Name Memory User Disk Virtual CPUs
1 m1.tiny 512mb 1gb 1
2 m1.small 2048mb 20gb 1
3 m1.medium 4096mb 40gb 2
4 m1.large 8192mb 80gb 4
15. April 2016 17
FIWARE Lab Nodes
15. April 2016 18
Apply for community account
Description: Dockerify Postgresql
Status: To Do
Components: (Others)
User full name: jhoechtlUser account to be upgraded: [email protected] submission name:Preferred FIWARE Lab Node: TrentoNumber of VMs: 1.0# public IPs: 1.0total # vCPUs: 2.0total RAM: 4.0total harddisk: 40.0Object Storage:
5.12. 7.12.
In order to obtain a "Community account" on FIWARE Lab, the applicant has to described the experimentation he/she plans to do and how it is related with the FIWARE ecosystem. Based on this description the community account is granted or not (mainly it depends on the innovative idea proposed and how this idea leverages on FIWARE).
9.12.Sent description
14.12.Nothing happened; Inquiry
15.12.Scheduled
15. April 2016 19
Apply for community account15.12.
On Trento Node in order to get a floating IP, you should provide us a copy of the identity card/passport of the main user or responsible of the project.Please contact us at: [email protected] to step forward with the procedure.
17.12.
FloatingIP has also been released after identity verification procedure completed.User has been notified via email with details.
15. April 2016 20
Docker on FIWARE Lab Instancehttps://myprivate42.wordpress.com/2016/02/26/docker-on-fiware/
OpenStack-based IaaS to provide computing and storage capacity across multiple data centers
Docker based PaaS to deploy and compose scalable applications
15. April 2016 21
Digression: Docker Why and What(Credit http://de.slideshare.net/dotCloud/docker-intro-november)
AppA
Hypervisor (Type 2)
Host OS
Server
GuestOS
Bins/Libs
AppA’
GuestOS
Bins/Libs
AppB
GuestOS
Bins/Libs
VM
GuestOS
GuestOS
App A’
Docker
Host OS
ServerBins/Lib
s
App A
Bins/Libs
App B
App B’
App B’
App B’
Container
FIWARE (Openstack) Instance
Containers are isolated, but share OS and, where appropriate, bins/libraries
Faster deployment, less overhead, easier migration
15. April 2016 22
Infrastructure Support
Digression: Docker Why and What(Credit http://de.slideshare.net/dotCloud/docker-intro-november)
Tool Supporthttps://docs.docker.com/machine/get-started-cloud/
Amazon Web Services, Digital Ocean, Google Compute Engine, IBM Softlayer, Microsoft Azure, Microsoft Hyper-V, OpenStack, Oracle VirtualBox, Rackspace, VMware Fusion, VMware vCloud Air, VMware vSphere, exoscale
1. Abstract Infrastructure from Platform
2. Continue working in your known environment
3. Prevent vendor lock-in: Standardized container format OCI and supervision
http://de.slideshare.net/Docker/docker-birthday-3-intro-to-docker-slides
15. April 2016 23
Open Container Initiative
Alternatives to Docker
+
15. April 2016 24
FIWARE Openstack and Docker
https://wiki.openstack.org/wiki/Docker
https://docs.docker.com/machine/drivers/openstack/
15. April 2016 25
Functional overlap Openstack vs. DockerOpenstack (FIWARE) Docker
Deployment nova boot <instance> docker run <container>
Orchestration Heat docker swarm; Kubernetes
Storage Volume (Cinder) & Objects (Swift) Data-only container
Logical service grouping Blueprints Docker compose
Service Orchestration with Kubernetes
15. April 2016 26
BEV Address DataExcerpt of Address register (Adressregister), Go To Link
Up to now, two releases: 15.07.2015, 08.10.2015
8 CSV files, ~ 550MB
File Size Line count
ADRESSE.csv 288223907 2335801
ADRESSE_GST.csv 72282856 2503351
GEBAEUDE.csv 199002301 2336346
GEBAEUDE_FUNKTION.csv 11031600 344737
GEMEINDE.csv 51779 2101
ORTSCHAFT.csv 514326 17259
STRASSE.csv 5134464 131081
ZAEHLSPRENGEL.csv 314096 8826
15. April 2016 27
BEV Address DataGemeinde
(GKZ)
● Gemeindename
Ortschaft(OKZ)
● Ortsname
Strasse(SKZ)
● Strassenname
Adresse(ADRCD)
● PLZ● Hausnrtext● Hofname● RW, HW● Bestimmungsart
GEBAEUDE(ADRCD, SUBCD)
Municipality Names
Street names
Districts and Sub-entities
ZIP-Codes
Geo-Codes of Addresses and Buildings
15. April 2016 28
BEV Address Data1. Analyse Data
2. Import Data into relational Database
3. Provide full text search via Web interface
4. URIfy Address Data – provide a public unique identifier for each address in object
15. April 2016 29
BEV Address DataObservations …
● RW and HW are in Datum Austria → not suitable for general mapping on Google Maps or Openstreetmap
● Irregular naming of city districts:
"1010";"Wien";"Wien,Innere Stadt";"Kärntner Straße";"90001";"17223"
"8043";"Graz";"Graz,11.Bez.:Mariatrost";"Mariatroster Straße"
"9020";"Klagenfurt am Wörthersee";"Klagenfurt,07.Bez.:Viktringer Vorstadt";"Bahnhofstraße"
● Abbreviations
geg., Bez., Parz.
15. April 2016 30
BEV Address DataObservations and countermeasures
● RW and HW are in Datum Austria → not suitable for general mapping on Google Maps or Openstreetmap
→ Use PostGIS to convert coordinates:
UPDATE ADRESSE SET MGIAUSTRIAGK = ST_SetSRID(ST_MakePoint(RW, HW), epsg);ALTER TABLE ADRESSE ADD COLUMN LATLONG geometry(POINT);UPDATE ADRESSE SET LATLONG = ST_Transform(MGIAUSTRIAGK, 4326);
● Siemensstraße 92?
RW: 6967.7 HW: 347724.57 POINT(16.4219750234186 48.2678291034249)
http://www.openstreetmap.org/?mlat=48.2678291034249&mlon=16.4219750234186
15. April 2016 31
BEV Address Data - Observations● Irregular naming of city districts & abbreviations
→ Configure full text search with synonym and thesaurus dictionary:
Innere:1 01.bez erster innere13.bez: dreizehnter 13 13.bez#KlagenfurtViktring:13 12.bez dreizehnter Viktring
● Query “Wien, erster Bezirk”→ "1010";"Wien";"Wien,Innere Stadt";"An der Hülben"
● Query “Wien, innere stadt”→ "1010";"Wien";"Wien,Innere Stadt";"An der Hülben"
● Query “Graz, erster bezirk”→ "8010";"Graz";"Graz,01.Bez.:Innere Stadt";"Hans-Sachs-Gasse"
15. April 2016 32
BEV Address Data – Import Data into DB● There is no relational data base system available on FIWARE
→ Solution: Use Docker to provision open source database PostGIS
15. April 2016 33
Problems encountered● Official documentation didn't match reality:
– http://simple-docker-hosting-on-fiware-cloud.readthedocs.org/en/latest/manuals/install/
1. Custom security groups didn't work out of the box
2. FIWARE Operating system images required modified options for provisioning
3. Ports for Docker orchestration and management on FIWAREnot open by default
→ local FIWARE node support necessary
FIWARE email support was generally responsive and helpful
15. April 2016 34
BEVADDRESS on FIWARE – Web Interface● Full text search for addresses with auto completion
● Return unique URI for address
● Ability to display on Google Maps / OSM
● Search Address within certain radius after click on map
● Analysis: Longest street in Austria, Streets spanning districts, streets with “funny” names, etc., etc.,
● Also see https://opendata.host/de (requires API-key)
15. April 2016 35
BEVADDRESS on FIWARE – You are invited!Modified PostGIS Database optimized for BEVADDRESS full text search
https://github.com/the42/bevaddress-dataload
Scripts to setup PostGIS and load address data
https://github.com/the42/bevaddress-dataload
User Interface
TBD → Your help wanted!
15. April 2016 36
FIWARE CritiqueDocumentation is spread all over
http://help.lab.fiware.org/
11. - 13. 4. 2016
15. April 2016 37
CritiqueDocumentation is spread all over
http://de.slideshare.net/fi-ware
http://edu.fiware.org/
https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Main_Page
https://readthedocs.org/search/?q=FIWARE
http://catalogue.fiware.org/
https://ask.fiware.org/questions/
http://stackoverflow.com/questions/tagged/fiware
15. April 2016 38
CritiqueLacking middleware
15. April 2016 39
CritiqueLacking middleware
How to get there?
15. April 2016 40
CritiqueInnovation friendly?
Lego advertisment 1981 Lego advertisment 2013
15. April 2016 41
Reliability and Issues
Trento Node
15. April 2016 42
Reliability and Issues –Events in the life of a FIWARE app
26.11.2015 The FIWARE Lab will schedule a maintenance time slot on December, 10th from 09:00 till 11:00 am
04.12.2015 Cygnus 0.11.0 has been released. As usual, it is highly recommended you upgrade you current installation.
01.02.2016 A new Orion Context Broker release (0.27.0) has just been released,
18.02.2016 We are experimenting some network problem in Spain2 and probably you cannot access to your resources
29.02.2016 A new Orion Context Broker release (0.28.0) has just been released,
03.03.2016 Tomorrow (Fridata, March 4) the Cosmos service in FIWARE Lab will be affected by a maintenance task; certain hardware must be replaced in the hosting machines
10.03.2016 The Budapest2 node is experiencing some issues with the blade system.
29.03.2016 A new Orion Context Broker release (1.0.0) has just been released
11.04.2016 next Monday, 11 April, a maintenance task has been scheduled in the Cosmos Global Instance of FIWARE Lab.
15. April 2016 43
FIWARE Futurehttps://ask.fiware.org/question/460/remaining-lifetime-of-fiware-lab-and-commercial-instances/
How much longer will FIWARE Lab exist (and be accessible to us)?
Given the above, FIWARE Lab will be operational following the interests of its users and stakholders. Current plans say at least end of 2020.
15. April 2016 44
Findings
● Overall stability not yet enough to base a business model on
● Confusing accumulation of components, startup accelerator landing pages, documentation, infrastructure administration
● FIWARE node communication helpful and timely
15. April 2016 45
FIWARE is the European cloud
Time to start experimenting
CC https://flic.kr/p/qVdaZS
47
Donau-Universität Krems.Die Universität für Weiterbildung.
Johann HöchtlCenter for E-Governance
@myprivate42
at.linkedin.com/in/johannhoechtl
github.com/the42
myprivate42.wordpress.com/
CC-BY 4.0