Middleware for Indoor Location-Based Services

Daniele Miorandi U-Hopper & ThinkIN

daniele.miorandi@u-hopper.com

1

Why me?

2

Went all the way from research to innovation &

business

3

Background

• Wireless networking background (PhD in Telco Engineering)

• 10 years in research (130+ papers, 4 patents, 20+ projects acquired)

4

Background (2)• Executive VP R&D at U-Hopper since

2012

• Coordinating R&D activities of the company (focus: big data analytics)

• Leading strategic innovation projects

• Founder & Chief Research Officer at ThinkIN since 2015

• Leading algorithms design for indoor location-based services

5

Background (3)• Started working on indoor LBS products since

2012

• All the way from algorithm design to full-scale implementation & commercialisation

• Hands-on experience

• Led the design & implementation of the open source i-locate toolkit (more later on)

6

What are indoor LBSs?

7

Definition

Indoor location-based services (LBSs) make use of the knowledge of the position of

entities (people and assets) in indoor spaces to deliver value to their users

8

Why are they relevant?

9

93% vs 7%

• According to US EPA we spend 93% of our time indoor

• For the 7% we spend outdoor we have a number of LBSs (think just of Google Maps)

• What do we have for the remaining 93% of our time?

10

Why now?

11

Unique combination of three factors

• Indoor positioning tech becoming mature

• Sub-meter accuracy possible, coarse-grained location cheap

• Standards for indoor spaces representation

• IndoorGML by OGC (http://www.opengeospatial.org/standards/indoorgml)

• Standards ensuring interoperability among vendors & integrators

• InLocation Alliance (http://inlocationalliance.org/)

12

Are they fundamentally different from outdoor

LBS?

13

Yes

14

In detail• Indoor spaces are very different from outdoor

spaces

• Outdoor can be represented as 2D, indoor is 3D (or 2.5D)

• Indoor you have building, rooms etc. Outdoor you don’t

• Indoor positioning techniques are inherently noisy and inaccurate

15

Is there a real market for indoor LBSs?

16

Yes

17

Market data

• 4.72 $B in 2016

• CAGR of 37.4%

• Estimate to reach 23.13 $B in 2021

18Source: Markets&Markets - http://www.marketsandmarkets.com/Market-Reports/indoor-positioning-navigation-ipin-market-989.html

Key messages

• Indoor LBS market is blooming

• There are plenty of opportunities

• And space for doing both high-impact research and delivering innovation

19

What are the application domains where indoor LBSs are taking off?

20

Hot verticals• Retail

• Profile shoppers behaviour in-store

• Context-aware marketing

• Industry

• Real-time location service

• Asset tracking & management (incl. logistics and warehouses)

• Workflow optimization21

• Healthcare

• Workflow optimization

• Asset tracking

• Patient monitoring

• Government

• Indoor navigation

Are there concrete and understandable use cases with an actual application potential?

22

Case #1: Indoor navigation• Take me to a given office

• Across outdoor and indoor spaces

• Navigate me also indoor (turn-by-turn instructions)

• Could be useful in:

• Government offices

• Large hospitals

• Shopping malls

• ….

23

Case #2: (Portable) Asset management• Access all information about assets in

your organization

• Including the actual location of portable ones

• E.g., a defibrillator in a hospital

• Real-time search

• For usage

• For maintenance24

Case #3: People tracking

• Monitor the movement of fragile patients at home or in a semi-controlled environment (nursing home)

• Couple with geofencing for alerting risk situations (e.g., exiting the building or waking up at night)

25

Case #4: Workflow Optimization

• Track the movement of workforce and assets in a factory floor

• Translate movement patterns into execution status of industrial workflows

• Real-time dynamic optimization and ex-post analysis of execution efficiency

26

Case #5: Safety of Personnel

• Track in real-time the position of personnel in hazardous environments (e.g., oil refinery or offshore rig)

• Alert in case of entering safety-critical areas

• Track and guide in case of evacuation alarm or mustering

27

Case #6: Location-based content delivery

• You walk in a museum

• As you approach an artwork you get delivered multimedia content explaining the context in which it was created

28

Who are the key market players?

29

Positioning tech

Estimote - kontakt.io - Quuppa - Zebra - Cisco - OpenRTLS

30

Retail solutions

RetailNext - Walkbase - RetailerIN - Euclid - Tyco Retail Solutions

Healthcare solutions

Senion -TeleTracking - Locatible - GE Healthcare - Nively

Indoor Mapping

Google - Micello - OpenStreetMap -IndoorAtlas

Industry solutions

SkyeTech - OmniID - Extronics - Engica - ThinkIN

Watch out…

31

Key Enabling Technologies

32

Think about google maps…• Render a map

• Position yourself on said map

• Search for a place & show that place on the map

• Includes resolving the place name to a position

• Compute a route from A to B

• Multiple transportation means, even combined (multimodality)

• Navigate from A to B along the route

33

KETs for indoor LBSs• Indoor Positioning

• Maps

• Geocoding

• Geofencing

• Routing

• Analytics

34

Positioning: Qs

• What indoor positioning technologies are available?

• How do they compare with one another?

• Are they sufficiently stable?

35

Positioning: Existing technologies

• Based on radio technologies

• Proximity: beacons, RFID

• Location: WiFi, BLE, UWB, ZigBee

• Based on cameras

36

Positioning: Existing technologies (2)

• The practitioner’s view: cluster in two main categories

• Sub-meter accuracy:

• BLE (Quuppa)

• UWB (whatever based on Decawave chip)

• Coarse-grained (room-level accuracy):

• Beacons

• WiFi (with trilateration)

• ZigBee

37

Positioning: Existing technologies (3)

38

High AccuracyLow Accuracy

Low TCO

High TCOQuuppa

UWB

Beacons

WiFi

CameraZigBee

Positioning: Existing technologies (4)

• Additional approaches:

• Based on variations in the Earth’s magnetic field

• Dead reckoning

• Visible light communication-based

• FM radio-based

• etc.etc.

39

Positioning: stability• Indoor position is intrinsically noisy

• Fundamentally different from outdoor positioning (where GPS signal - maybe with EGNOS - provides good enough accuracy/reliability in 99% for use cases)

• Requires a lot of post-processing

• No silver bullet

• No out-of-the-box solution

40

Maps: Qs

• How to represent indoor spaces?

• What about standards?

• Are there sufficient indoor maps available?

• Are there open data repositories of maps?

41

Maps: How to represent indoor spaces

• Various approaches are possible

• For a good overview: Worboys, M.F., Modeling indoor space (keynote). Third ACM SIGSPATIAL International Workshop on Indoor Spatial Awareness (ISA 2011), November, Chicago, IL. 2011.

• Semantic models represent the types of entities in indoor space, as well as their properties and relationships (—> ontology)

• Topological models: focus on connectivity properties of a space

• Geometrical models: focus on geometry of indoor spaces (e.g., CAD)

• Hybrid: topological with geometrical features embedded in the description

42

Maps: standards

• Various standards have been proposed for the representation of indoor spaces

• The key standardization body in this field is the Open Geospatial Consortium (OGC, http://www.opengeospatial.org/ogc)

• Our focus: indoorGML

43

Maps: indoorGML• IndoorGML = open data model & XML schema for indoor spatial information

• Concepts:

• Space is structured as cells (cell~room)

• Geometry of cells can be described either directly, through external representation (CityGML) or can be omitted

• From geometry (primal space) to topology (dual space) through Poincaré duality

• Multi-layer representation of connectivity (walking user, wheelchair, robot, drone etc.)

• Anchor node: connection with outdoor graphs (e.g., OSM)

44

Interconnecting indoor - outdoor

• Entrance of the building is a special node

• Anchor point where outdoor and indoor networks are connected

45

Special links - vertical connectors

• For each floor a graph is constructed

• The graphs are interconnected through vertical links representing elevators or stairs

46

Maps: Availability• How many indoor maps available out there?

• In the range of thousands (estimate)

• The point is accessibility

• Indoor is NOT outdoor (!)

• A building is not a public space

• Access depends on the owner/manager

• In some cases (e.g., governmental buildings) there may be security reasons to prevent making data openly accessible

47

Maps: Open Data?

• Fragmented landscape (in total few hundreds):

• From i-locate portal: http://portal.i-locate.eu/

• From OSM community: http://wiki.openstreetmap.org/wiki/Indoor_Mapping

• From OpenStationMap: http://openstationmap.org/

48

Geocoding: Qs

• How to translate description of spaces to coordinates?

• What about the other way round (from coordinates to description)?

49

Geocoding: As• Geocoding for outdoor spaces: commercial/open source solutions

already out there

• Need to augment it for indoor spaces

• Similar functioning, can be implemented using, e.g., PostGIS extension to PostgreSQL

• And then combine outdoor + indoor results (indoor are 3d!)

• Same for reverse geocoding

50

Geofencing: Qs

• How to handle matching of indoor position data with a space-time rule (enter an area, exit an area, stay in an area for a given time)?

• How to make it scalable?

51

Geofencing: As• At the abstract level:

• Understand whether a point (=position of an entity) is inside a region (defined as a generic polygon)

• In case it is and it was not before, fire an event

• No major differences wrt outdoor, but:

• Finer-level granularity (room? close to an object in a room?)

• Need to cope with noisy position data

52

Geofencing: As• Various commercial solutions available

• Some opensource solutions, but hard to scale

• Processing-intensive —> big data streaming architecture

• Imagine 10,000 geofences and data about 1M entities transmitting their position every 1s….

• For a good intro look at John Murray’s approach (using MongoDB features, http://www.johnmurray.io/)

53

Routing: Qs

• How to route in indoor spaces?

• How to route across outdoor and indoor spaces?

• What are the differences to outdoor spaces only?

54

Routing: As• Requires a graph representation (connectivity graph) of the indoor space

• Natively supported if space represented as indoorGML

• Possibility of supporting different transportation means (walking, wheelchair etc.)

• Outdoor-to-indoor routing: just connect the two graphs through anchor node

• Differences from outdoor routing: 3D!

• Need to account for floor changes (lift or stairs)

• Lot of subtleties (e.g., what about half floors?)

55

Analytics: Qs

• Do I need specific data processing pipelines for producing analytics related to the occupation of indoor spaces?

• How to make it scalable?

56

Analytics: As

• Data processing pipelines used for computing outdoor analytics need to be tailored to deal with the specific features of indoor environments

• In particular, noisy positioning data

• Presence of physical barriers

• Use of contextual information for data cleaning

57

Analytics: examples

• Tracking assets

• Utilization

• Where used

• and by whom

58

Analytics: examples (2)• Tracking people

• Visits over time

• Dwell time in a given area

• Heatmaps

• Frequency

• Duration

• Common paths

59

Analytics for indoor spaces

• Computing analytics for indoor spaces is a processing-intensive process

• Can be implemented using `standard’ big data stacks based on open-source stuff (kafka+spark+redis+cassandra)

• Algorithms for data processing and scalable KPIs computation are an active research field

60

How are indoor LBSs structured?

61

Is there a reference architecture for indoor

LBS?

62

No.

Too much diversity?

63

Why a middleware?

64

No matter if you are a smart hacker…

.. or a Web entrepreneur…

..or a community

…

..with a clever idea for a new application enabled by indoor positioning

This will be your

expression when you

start building it!

At the moment..• Applications developed using a silo-like approach

• Integrated all the way down to the positioning system

• App developer are required to have understanding of domain specific issues (geocoding? WMS? handling noisy data?)

• —> Inhibiting innovation in the field

• —> High entry barrier for new players

70

Basically everybody is re-inventing the wheel

wasting time

and money!

• http://www.i-locate.eu/

• “Indoor/outdoor location and asset management through open geodata"

• EU project, funded under the CIP/PSP programme

• Open by default (code, maps, data, papers etc.)

• Relevance: developed an open-source toolkit for allowing app developers to quickly build & deploy indoor LBS

• Coupled with a portal for hosting maps and indoorGML representations

75

• Consortium comprising

• Led by Trilogis (IT), including high-tech SMEs (U-Hopper, ZigPos, IndSoft, Epsilon, GeoSys, Fida Solutions), innovation firms (Technoport, UrbaSofia, GSIG, C3L, Gist), research institutions (TUE, FBK) as well as end users (Alba Iulia Hospital, Brasov Municipality, Velletri Municipality, Rijeka Municipality, Tremosine Municipality, APSS, Bruckenthal Museum, Municipality Baia Sprie, Genova Municipality, Mitera Hospital)

• 14 pilots across 8 countries

• Covering a variety of use cases spanning outdoor and indoor spaces

76

What are the key middleware functionality

required?

77

Key functionality required

78

• Retrieve the position of an entity indoor

• Search for an indoor place & show that place on the map

• Includes resolving the place name to a position

• Compute a route from A to B

• Navigate from A to B along the route

• Create geofences

Is there anything from GIS that can be reused?

79

Indoor GIS• A lot of concepts and technical enablers can be taken

from the GIS field

• Yet, indoor information is inherently different

• Requires knowledge related to:

• Signal processing

• Indoor-specific standards (indoorGML)

• Big data

80

How do I build indoor LBSs? Are there open-source

framework I can (re-)use?

81

https://gitlab.com/groups/ilocate

82

A toolkit for

building indoor LBs

released under a permissive open source license (Apache v.2)

and enabling out-of-the-box two types of indoor LBSs:

#1: Self-app• Know where you are (outdoor/indoor)

• Compute route to intended destinations (outdoor/indoor)

• Turn-by-turn navigation to intended destinations (outdoor/indoor)

As added-value service (more later)

#2: Asset tracking

• Track the position of portable equipment in (near) real-time

• Plus geofencing, asset maintenance etc.etc.

i-locate toolkit design principles

1. Loosely coupled components

2. All is REST

3. Data is king

4. G&G (Grab&Go)

88

Platform

Locationdata

Application

Application

Application LBS LBS (Open)APIs

Toolkit

i-locate toolkit architecture

Proxy

• Localization is done server-side

• The proxy:

• Combining data from different positioning technologies (sensor fusion)

• Using them to estimate current position

• Makes higher-level components positioning technology agnostics

Proxy

• Unique access point for locating entities

• Currently supported technologies: • Quuppa • eeRTLS • WiFi (through outdoor localization + Combain + passive PI-Radar) • GPS • QR codes • Beacons • EGNOS (through external device)

• Implemented in PHP, using YII framework • Easily extensible

91

Configuration

• Allows to read/write specific attributes of tracked entities • E.g., battery level, RSSI etc. • REST interface

• GETilocate/configura3on/getLocaliza3onSystems• PUTilocate/configura3on/put/{localiza3onSystem_id}/{obj_id}

• Requires to be deployed locally on a gw or local server able to connect to the gw over REST

• Supported Indoor Localization Systems: • Quuppa • eeRTLS

• Implemented in Java

92

Communication bus• Based on the MQTT protocol

• Lightweight pub/sub system for IoT • OASIS standard • Using the Mosquitto broker implementation

• All location updates dispatched through mqtt broker

• Additional plugin developed for handling authorization for subscriptions

93

Monitoring• Aimed at sysadmins: check the status of services &

support troubleshooting

• Based on the Elastic (former: ELK) stack • Shippers read logs from VMs (or: containers) hosting

services and send to a centralized logstash server • Logstash server processes logs and stores them in

an ElasticSearch DB • A Kibana dashboard is attached to the DB for

visualizing logs • Can be easily configured to define which data to log

94

Security & Privacy

• Provides self-registration, authentication, validation & authorization functionality

• Authorization based on policies designed around a RBAC scheme

• Based on openAM opensource framework

95

OGC Spatial• Provides access to geographical information in a standardized,

interoperable way

• OGC standard • WMS, Web Map Service • WFS, Web Feature Service

• Makes i-locate data accessible by the most common GIS client

• Based on open source engine (geoserver)

• Includes geoserver functionality

96

Spatial solver• Provides an interface to access the i-

locate Open Repositories

• Includes tools and functions to filter and process geodata

• Based on PostGIS, includes RestFUL APIs

• Able to process also external datasets

97

Geofencing

• Generate alerts when tracked entities move in or out of a given region

• Push and pull notifications

• [Proprietary tech by Trilogis]

• [Check John Murray site for alternative open source implementations]

98

Location analytics

• Provides statistics on the usage of indoor spaces

• Based on proprietary ThinkIN platform (thinkin.io) • Open APIs and wrapper (data ingestion) based on

Apache Kafka

99

Routing• Based on the OpenTripPlanner (OTP) open-source platform for multimodal

routing

• It supports multiple indoorGML graphs and outdoor OpenStreetMap data

100

Routingservice

Routingalgorithm

Navigation graph

IndoorGraphs

OutdoorGraphs

indoorGML

OpenStreetMap

Multimodal routingAvoidance settingEtc.Start/endlocations

(latitude/longitude/level)

Travelplan(withturn-by-turnnavigationinformation)

Crowdsourcing

101

• Providessupportforgatheringuser-generatedgeographicalinformation

• BasedonUHproprietaryCIVICFLOWplatform(http://www.civicflow.com/)

Asset Management

• Connector to Box3 asset management service by Trilogis

• Integration of the assets representation and geographical information

• Compliant with ISO 55000 (asset representation) and supporting indoorGML

102

i-locate - Indoor/outdoor LOCation and Asset management Through open gEodata (GA 621040)

File: D.3.1 - i-Locate toolkit v1.0.docx D.3.1 Page: 63 i-locate toolkit v1.0

Figure 24: Web Client

The web client is composed by two main parts. A frontend, composed by web client itself, and a backend, consisted by two component, application server hosting the client and the engine for the asset management. The web client is a solution based on Terra3 webgis provided by Trilogis and is mainly based on Javascript and OpenLayer libraries. The Asset Management engine is based on Box3 application provided by Trilogis and is a stand alone solution developed in .NET technologies running on a dedicate machine.

The template comes in the form of a prototype which supports:

x webgis funcionalities (pan, zoom, identity), x visualization of position of assets, x interaction with Asset Management engine.

The web client interoperates actually with the following toolkit components:

x Indoor Localization: for retrieving indoor position of asset; x OGC Spatial: for accessing reverse geocoding functionality; x Upload/Download: to dynamically download indoor map of the reference building. x Geofencing: to receive notification on the status of the asset x Spatial Solver: to perform some spatial operation x Asset Management: to interact with the third part engine for Asset Management

The web client template is available here: http://02.cloud.i-locate.eu:8080/terra3/

• S u p p o r t a s s e t management use case

• WebGIS functionalities • R e a l T i m e a s s e t s

position • Extensible Framework

Web Client template

103

Mobile App Template• Support citizen guidance use case (indoor/outdoor)

• Developed using Titanium Appcelerator SDK (cross-platform support)

• Template to be personalised for matching specific use case requirements

• Supports:

• Locating user on a map (indoor/outdoor)

• Search for a place

• Compute route

• Display route on the map

• Turn-by-turn navigation

104

Are we forgetting something?

• We need a way to retrieve (indoor) maps

• We need a way to create/manage/retrieve indoorGML representations

• (And yes, we also need to retrieve outdoor maps)

• —> The i-locate portal

106

The i-locate portal• An infrastructure able to handle indoor/outdoor GIS

• Maps

• IndoorGML

• Based on ‘standard’ GIS tools:

• Geoserver/PostGIS/PostgreSQL

• Available as open source as well

107

Hands-on

108

Portal Demo

109

Toolkit Demo (REST APIs)

110

http://bit.ly/miorandiMiddleware2016

111

What’s in the Postman collection

• Get a map (portal)

• Give me my position (proxy)

• Compute the position of an entity (proxy)

• Resolve an indoor address (geocoder)

• Compute route from A to B (routing)

• …plus a number of convenience calls

112

The i-locate app (demo)

113

https://gitlab.com/ilocate/ilocate-app

114

Training Material

http://www.gisig.eu/platform/course/index.php?categoryid=15

115

Videos and lectures freely available (registration required) at: