geoevent server: creating connectors & processors using ... · send a text message send an...

GeoEvent Server: Creating Connectors and Processors Using

the GeoEvent SDK

Ming ZhaoReal-Time GIS Developer, Esri

mzhao@esri.com

Javier Delgadillo

Real-Time GIS Developer, Esri

jdelgadillo@esri.com

Agenda

Real Time GIS Overview

Connectors

Transports

Adapters

Processors

Summary

1

2

3

4

5

6

1Real Time GIS Overview

DesktopWeb Device

live & historic

aggregates & features

map & feature service

• Ingest high velocity real-time IoT

data into ArcGIS.

• Perform continuous analytics on

IoT events as they are received.

• Store IoT observations in a

spatiotemporal big data store.

• Visualize high velocity &

volume IoT data:

- as an aggregation

- or as discrete features.

• Notify about IoT patterns of

interest.

• Adjust behavior of things in our

environment through actuation.

stream service

live features

ArcGIS

Enterprise

GeoEvent

Server

ingestion

analytics

ArcGIS as an IoT PlatformArcGIS Enterprise with real-time & big data capabilities

GeoAnalytics

Server

spatiotemporal

big data store

storage analytics

visualization

actuation

Ingestion of real-time dataGeoEvent Server: input connectors

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

you can create

your own

connectors

Poll an ArcGIS Server for Features

Ou

t o

f th

e B

ox

Esri

Gallery

ActiveMQ

Part

ner

Gallery

Amazon IoT

Azure IoT

CompassLDE

enviroCar

exactEarth AIS

FAA (ASDI)

GNIP

Networkfleet

OSIsoft

Valarm

Waze

Zonar

AIS

Common Alerting Protocol

Cursor-on-Target

Exploitation Support Data

GTFS

Instagram

Kafka

KML

MQTT

NMEA 0183

RabbitMQ

Sierra Wireless (RAP)

Trimble (TAIP)

Twitter

Poll an external website for GeoJSON, JSON, or XMLHTTP

Receive Features, GeoJSON, JSON, or XML on a REST endpointREST

Receive GeoJSON or JSON on a WebSocketWS

Receive RSS

Receive Text from a TCP or UDP Socket

Subscribe to an external WebSocket for GeoJSON or JSONWS

Watch a Folder for new CSV, GeoJSON, or JSON Files.csv

AIS

you can create

your own

connectors

configure a new input connector by pairing a transport & adapter together

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Feature-JSON

Connectors Transport Adapter

GeoJSON

Receive Feature-JSON from Kafka

Receive GeoJSON on a REST endpoint

Feature-JSON

Adapters

GeoJSON

JSON

RSS

Text

XML

CAP

Cursor-on-Target

GeoMessage

Instagram

NMEA

Sierra Wireless (RAP)

Trimble (TAIP)

Twitter

VMF

Es

riG

all

ery

Waze

Feature Service

Transports

File

HTTP

HTTP+BasicAuth

HTTP+OAuth

TCP

UDP

Waze

WebSocket

Esri

Gallery Amazon IoT

Azure IoT

Kafka

IRC

Kafka

MQTT

Kafka

HTTP

RabbitMQ

Twitter

Input Connector =

Adapter + TransportIngestion of real-time data

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

DisseminationGeoEvent Server: output connectors

you can create

your own

connectors

Ou

t o

f th

e B

ox

Add or Update a Feature

Esri

Gallery

Amazon IoT

Azure IoT

ActiveMQ

Cursor-on-Target

Hadoop

Kafka

MongoDB

MQTT

RabbitMQ

Twitter

Publish Text to a UDP Socket

Push GeoJSON or JSON to an external WebsiteHTTP

Push GeoJSON or JSON to an external WebSocketWS

Push Text to an external TCP Socket

Send a Text Message

Send an Email

Send an Instant Messageim

Send Features to a Stream Service

Write to a CSV, GeoJSON, or JSON File .csv

Add a Feature to a spatiotemporal big data store

Update a feature in a spatiotemporal big data store

you can create

your own

connectors

Disseminationconfigure a new output connector by pairing an adapter & transport together

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Message Formatter SMTP

Connectors TransportAdapter

Feature-JSON Kafka

Send an Email

Publish Feature-JSON to Kafka

Big Data Store

Transports

Feature Service

File

HTTP

HTTP+BasicAuth

HTTP+OAuth

SMTP

Stream Service

TCP

UDP

WebSocket

XMPP

Cache

Adapters

Feature-JSON

GeoJSON

JSON

Message Formatter

RSS

Text

WebSocket

ActiveMQ

Amazon IoT

Azure IoT

Hadoop

IRC

Kafka

Esri

Gallery

MongoDB

MQTT

RabbitMQ

Cursor-on-Target

Esri

Gallery

SMS

TCP-Squirt

Twitter

Output Connector =

Adapter + Transport

you can create

your own

processors

Streaming analytics, policies & orchestration• GeoEvent Server: processors

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Buffer Creator

Convex Hull Creator

Difference Creator

Envelope Creator

Field Calculator

Field Enricher

Field Mapper

Geotagger

Incident Detector

Intersector

Projector

Simplifier

Symmetric Difference

Track Gap Detector

Field Reducer Union Creator

Ou

t o

f th

e B

ox

Add XYZ

Esri

Gallery

Bearing

Ellipse

Event Volume Control

Extent Enricher

Field Grouper

GeoNames Lookup

Range Fan

Reverse Geocoder

Service Area Creator

Symbol Lookup

Track Idle Detector

Unit Converter

Visibility

Motion Calculator Query Report

Extending GeoEventSoftware Development Kit (SDK)

• You can create your own custom connectors and processors using the GeoEvent

Software Development Kit (SDK)

Your processors

Your

connectors

Your

connectors

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

GeoEvent Server SDK

- api: JavaDoc content associated with GeoEvent Server SDK

- repository: Local maven repository

- samples: Sample connectors (and processors)

- GeoEvent Developer Guide

- JavaDoc available by opening api/index.html in a browser (Chrome, Firefox, IE, Safari)

Extending GeoEventSoftware Development Kit (SDK)

2 Connectors

ConnectorsWhat Does a Connector Do?

• Connectors are used to create Inputs and Outputs, hiding the technical details

• It might be very specific

• Get latest earthquakes from USGS

• Or more general

• Connect to an RSS feed

Creating InputsConfigured Using Connectors

GeoEvent Server

National Weather Service

US Geological Survey

Live Traffic Services

Ou

tpu

ts

GeoEvent ServicesStorms

Earthquakes

Traffic

Receive Text from TCP

Connectors

Inputshttp://

Receive RSS

Cre

ate

In

pu

t

Poll External Website for JSON

Cre

ate

In

pu

t

ConnectorsConfiguring a Connector

• The connector helps the user by

▪ Providing default values

▪ Re-label properties to be appropriate to the context

▪ Move properties under an “advanced” area to discourage modification

▪ Completely hide properties that the user should not see

ConnectorHow Does it Help

• By choosing a Connector, the user implicitly selects components from the

GeoEvent Server that knows:

▪ HOW to move data (Transport)

▪ WHAT the data looks like (Adapter)

Example Input

TCP Transport Text Adapter

Input (Receive text from a TCP Socket)

Start

StopByte [ ]

GeoEventGeoEvent Services

Exploring a Connector

Demo

3 Transports

TransportWhat makes up a Transport

Transport Definition • Defines transport metadata and properties

Transport • Implements transport application logic

Transport Service • Provides a Transport Definition

• Instantiates new Transports

Transport BehaviorTransport

• Transports are given

- Properties that define behavior

- A “ByteListener” where bytes should be sent

• Transport is started by the server and it sends bytes to the receiver

• Transport is stopped by the server and it stops sending bytes

Transport LifecycleTransport

Transports have a lifecycle that determines if they are producing data.

STOPPED

STOPPING

STARTED

STARTING

ERROR

STOPPED

Server calls start() Transport reads

properties and starts

Server calls stop()Transport closes

resources

Internal error occurs

STARTED

Outbound TransportTransport

• Outbound Transports accept arrays of bytes from the Adapter and transmit them.

• Occasionally the destination for the bytes depends on content in the GeoEvent.

- The Transport has the option of “looking back” at the GeoEvent that generated the bytes, and

using it to route the data.

Property DefinitionProperties

• Transports and Adapters request properties through their “Definition” class

• Each requested property has a

- Name, Description, Type (String, Integer, Float, …), Default Value …

Advanced Topics – More on PropertiesProperties

• Properties can be Mandatory or Optional.

• Some properties are dependent on others

▪ Example: “Compression Algorithm” depends on “Compressed = True”

• Some properties only accept values from a list of Allowed Values

▪ “Compression Algorithm” : [ Run Length Encoding, Zip, LZW ]

My Sample Input

Advanced Topics – More on PropertiesProperties

• Properties can also be created programmatically in a Java Class.

• Allowed values were strings in previous release. Replaced by objects in

10.3.1. As a result, you can display allowed values with localized labels.

Exploring a Custom Transport

Demo

4 Adapters

AdapterWhat makes up an Adapter

Adapter • Implements adapter application logic

Adapter Definition • Defines adapter metadata and properties

Adapter Service • Provides an Adapter Definition

• Instantiates new Adapters

Adapter BehaviorAdapters

• Adapters are given

- Properties that define behavior

- A “GeoEventListener” where the GeoEvents should be sent

• Adapters are DATA DRIVEN

- No start/stop calls

- The adapter is handed a byte array and pushes any generated GeoEvents to the Listener

Adapter Code WalkthroughAdapter Properties

Adapter Code WalkthroughAdapter Properties

Adapter Code WalkthroughParsing Data

5 Processors

Real-Time GISArcGIS GeoEvent Extension for Server: continuous analytics on GeoEvents as they are received

You can create

your own

processors.

GeoEvent Extension

Inp

uts

Ou

tpu

ts

GeoEvent Services

Buffer Creator

Convex Hull Creator

Difference Creator

Envelope Creator

Field Calculator

Field Enricher

Field Mapper

Geotagger

Incident Detector

Intersector

Projector

Simplifier

Symmetric Difference

Track Gap Detector

Field Reducer Union Creator

Ou

t o

f th

e B

ox

Add XYZ

Esri

Gallery

Bearing

Ellipse

Event Volume Control

Extent Enricher

Field Grouper

GeoNames Lookup

Range Fan

Reverse Geocoder

Service Area Creator

Symbol Lookup

Track Idle Detector

Unit Converter

Visibility

Motion Calculator Query Report

Extending GeoEvent

• Software Development Kit (SDK)

• You can create your own custom connectors and processors using the GeoEvent

Software Development Kit (SDK)

Your processors

Your

connectors

GeoEvent Server

Inp

uts

Ou

tpu

ts

GeoEvent Services

Your

connectors

ProcessorLifecycle Review

GeoEvent Server

GeoEvent Service

Processor

Validation

Shutdown Initialization

Creation

Processing…

Anatomy of a ProcessorWhat makes up a Processor?

• Implements processor lifecycle

Configuration • Defines processor, it’s dependencies and metadata

Processor Definition • Defines processor metadata, properties

and GeoEvent Definition(s)

Processor

Processor Service • Instantiates processor definition, processor and

applies configuration

Processor Service

• Creates processor definition

• Creates processor instance

• Create new Java class

• Extend GeoEventProcessorServiceBase class

• Create a Processor• Create a Processor

Processor DefinitionProperties and GeoEvent Definition(s) in Code

• Define collection of Properties that are required at run-time• Provide Metadata about your Processor• BundleLoggerFactory can log localized messages if so desired

• Similar to log4j commons-logging

Identification

1. Name

2. Domain

3. Version

Processor DefinitionMetadata

Processor DefinitionMetadata

Description

4. Label

5. Description

6. Contact

Information

1. Creation

ProcessorLifecycle

2. Initialization

3. Validation

1. Creation

ProcessorLifecycle

2. Initialization

3. Validation

1. Creation

ProcessorLifecycle

2. Initialization

3. Validation

4. Processing

ProcessorLifecycle

5. Shutdown

6. Service Start / Stop

4. Processing

ProcessorLifecycle

5. Shutdown

6. Service Start / Stop

Processor ServiceConfiguration

• Declare processor service as a service implementing

GeoEventProcessorService interface from GeoEvent Server SDK

Demo

Troubleshooting / DemoWhat’s going wrong?

Debugging

• Open fie C:\Program Files\ArcGIS\Server\GeoEvent\etc\ArcGISGeoEvent.cfgand un-comment lines (as Administrator!!)

• Continue numbering scheme (actual number will depend on version of GeoEvent)

Troubleshooting

• Configure Debugging for Remote Java Application

Real-Time GIS: Creating Connectors using the GeoEvent SDKSummary

• Connectors – Recipe for creating inputs/outputs

- Transport – Moves raw data in/out of the GeoEvent Server

- Adapter – Converts raw data to GeoEvents and back

- Properties – Used to configure a transport, adapter, or processor for a specific use case. Allows the

end user to configure component.

• Processors

- GeoEvent allows you to develop and deploy custom processors

- Can take advantage of Esri Java Geometry libraries

- Extent Enricher available on GitHub: https://github.com/Esri/extent-enricher-for-geoevent

• GeoEvent SDK – Tools that help you to extend the GeoEvent capabilities and to build your

own custom applications.

- Samples

- GeoEvent Developer Guide

Where to learn more?Resources

• To learn more, visit the ‘Get Started’ area of the GeoEvent Server page:

- http://links.esri.com/geoevent

- Introduction

- Big Data Store

- Resiliency

- Multiple-Machine Sites

- Notifications

- Stream Services

- RSS, HTTP, Files

- REST Admin API

• Ask questions on the GeoEvent Forum:

- http://links.esri.com/geoevent-forum

Please Attend Our Other Sessions!

• GeoEvent Server: An Introduction Tue, 2:30-3:30 pm, Primrose B

• GeoEvent Server: Applying Real-Time Analytics Tue, 5:30pm-6:30 pm, Primrose A

• Real-Time and Big Data GIS: Best Practices Wed, 10:30-11:30 am, Primrose B

Fri, 1:00-2:00 pm, Catalina/Madera

• ArcGIS and the Internet of Things (IoT) Wed, 2:30-3:30 pm, Primrose B

• Real-Time and Big Data GIS: Leveraging the Wed, 4:00-5:00 pm, Primrose A

Spatiotemporal Big Data Store

• Developing Real-Time Web Apps with the ArcGIS API Tue, 9:00-9:30 am, Demo Theater 1: Oasis 1-2

for JavaScript

• Real-Time GIS: Road Ahead Thu, 4:00-5:00 pm, Pasadena/Sierra/Ventura

• GeoEvent Server: Creating Connectors and Fri, 8:30-9:30 am, Mesquite B

Processors Using the GeoEvent SDK

Please Take Our Survey!

Download the Esri Events app

and find your event

Select the session you

attended

Scroll down to the

“Feedback” section

Complete Answers,

add a Comment,

and Select “Submit”

Esri.com Proceedings page

http://proceedings.esri.com

Where users can find PDF'ed

slides after DevSummit

Questions / Feedback?

Ming ZhaoReal-Time GIS Developer, Esri

mzhao@esri.com

Javier Delgadillo

Real-Time GIS Developer, Esri

jdelgadillo@esri.com