actionable insights - thompson

© 2014 IBM Corporation

Actionable InsightsLeveraging your Connectivity, Big Data and Predictive Analytics infrastructure to drive top line revenue

Ben ThompsonChief Architect IBM Integration Bus

July 2014


Capture what’s happening in

real-time

Generate model of future

Predict most likely outcome

Proactively optimize business

Tap into relevant data with context

.

.

Unleash real-time dataflowing throughout enterprise

Actionable InsightsWhat’s the Big Idea?


An Example of Actionable Insight

MQTT DICOMPACS

Predicting Deciding

Integrating

ImagingModality

PatientReport

Monitoring

ElectronicMedical Record

AlertDoctor

ODBCJDBC

SMS


Industrial Process Control – Machinery Failure

OPC OPC

Predicting Deciding

Integrating

PowerConsumption

Monitoring

Temperature

SCADA SAPBAPI

VibrationRPM

Order Part


Solutions at ScaleWhat Big Data means for …

Connected Appliances

Connected Cars

Smartphones

Internet TVsHome Hubs

Smart Meters

Home health devices

� Connectivity & Integration

� The Internet of Things

� Analytics


MQTT A transport for driving the Internet of Things

Lossy or Constrained

Network

Lossy or Constrained

Network

Real-World Aware Business Processing

High volumes of data/events

1999 Invented by Dr. Andy Stanford-Clark (IBM), Arlen Nipper (now Cirrus Link Solutions)

2011 - Eclipse PAHO MQTT open source project

2004 MQTT.org open community

2013 – MQTT Technical Committee formed

Cimetrics, Cisco, Eclipse, dc-Square,

Eurotech, IBM, INETCO Landis &

Gyr, LSI, Kaazing, M2Mi, Red Hat,

Solace, Telit Comms, Software AG,

TIBCO, WSO2

• MQTT is a lightweight publish-subscribe protocol

with reliable bi-directional message delivery

• Open Source

• Standards


Passing Data from Things into the Enterprise The Power of MQTT and IBM MessageSight

• MQTT’s very compact wire format, results in lower

network costs than an HTTP equivalent

• Lightweight footprint – protocol will run on low power

devices

• Clients: C = 80kb; Java = 100kb JavaScript = 80kb

• Recovery, store and forward, and publish/subscribe

are all provided by the MQTT implementations, and

don’t have to be coded into application logic

• Simple set of verbs, easy for developers to learn

• Easy integration with Systems of Record

Lower development costsLower development costs

Lower running costsLower running costs

• Near real-time push of information

• Minimal battery usage

• Store and forward messaging

• Exactly once delivery (where required)

• MQTT’s Event-Driven design point means that a

single server can support a million connected users or

devices

• Publish/Subscribe allows additional functionality to

be added without change to existing application code

More Flexibility and ScaleMore Flexibility and Scale

Improved User ExperienceImproved User Experience


Action HTTP MQTT

Get single piece of data 302 bytes 69 bytes (<4 times)

Send single piece of data 320 bytes 47 bytes (<6 times)

Get 100 pieces of data 12600 bytes 2445 bytes (<5 times)

Send 100 pieces of data 14100 bytes 2126 bytes (<6 times)

Characteristics HTTP MQTT

Style Document-centric, request/response Data-centric, publish/subscribe

Verbs GET/POST/POST/DELETEcomplex spec

Pub/Sub/Unsubsimple protocol, easy to learn

Message size Large message, lots of data in headers 2 bytes in minimum header

Quality of Service None, requires custom coding in application

3 levels:best-effort, at-least-once, exactly once

Data distribution No distribution mechanism (1-to-1 only) 1-to-none, 1-to-1, 1-to-n

Deliver Relevant InformationOptimizing network with event-driven notification


• Analytics is the discovery and communication of meaningful

patterns in data

• Predictive analytics uses statistical techniques to build a model

that describes key relationships in data

• Predictive models are applied to new observations to estimate

the likelihood or values of unknown (usually future) events

Predictive AnalyticsDiscovering trends in real-time data in flight

15 Petabytes of big

data generated daily

95% of Mobile traffic

is data by 2015

15b devices

connected by 2020

420m wearable

health monitors by 2014

Big Data from

Internet of Things


Analysing the Past, Present and FutureDiscovering trends in real-time data in flight

PAST• Applying analytical techniques to past, archived events

• Correlation & Filtering

• Advanced Queries – “where is my transaction?”

• Data Analyser and Observer!

PRESENT• Analysing current in-flight events

• Correlation, aggregation, metrics

• Calculation of KPIs, real-time dashboard display

• Reporter and Observer!

FUTURE• Predictive Analytics

• Invoke predictive models, trained on past data

• Trigger actions based on predicted outcomes

• Participant!


Monitoring the Past and Present with IIBAccounting & Statistics, Monitoring and Record & Replay


� Message flow statistics - One record is created for each message flow in a server:– Message flow, Server (Execution Group) and Node (Broker) name and UUID – Type of data collected (snapshot or archive) – Processor and elapsed time spent processing messages – Processor and elapsed time spent waiting for input – Number of messages processed – Minimum, maximum, and average message sizes – Number of threads available and maximum assigned at any time – Number of messages committed and backed out – Accounting origin

� Thread statistics - One record is created for each thread assigned to the message flow:– Thread number (this has no significance and is for identification only) – Processor and elapsed time spent processing messages – Processor and elapsed time spent waiting for input – Number of messages processed – Minimum, maximum, and average message sizes

� Node statistics - One record is created for each node in the message flow: – Node name and Node type (for example MQInput) – Processor time spent processing messages – Elapsed time spent processing messages – Number of times the node is invoked – Number of messages processed – Minimum, maximum, and average message sizes

� Terminal statistics - One record is created for each terminal on a node: – Terminal name and Terminal type (Input or Output)– Number of times that a message is propagated to the terminal

Accounting & StatisticsReal-time publication of summarized system performance data


Publication +

Subscription

Monitoring your IntegrationsPublication of actual payload data for later analysis


Business Transaction Monitoring versus Business Activity Monitoring“Archive available for later queries” versus “Real-time view relating to pre-defined KPIs”

� Both monitoring capabilities have their pros and cons:

– Real-time view gives you quicker insight but no post-event searching

– Archived view slower to produce insight but complex post-event searching is easy


Data AnalysisIterative Build-time Analysis of large XML documents

� Create a Data Analysis project, select a set of sample XML documents for analysis, and IIB will generate a Data Analysis Model. Views and filters are provided to navigate through the complex content in a variety of ways.

� Revealed elements whose content relates to a known code set translation (defined in a glossary) are highlighted. Create Target Model (drag and drop items from the Data Analysis Model to the Target Model).

� Make further edits to theTarget Model (either for output messages or output to a database). Generate graphical maps which will convert input instance XML documents into instance XML documents which conform to the Target Model. Generate maps for inserts into a database.

� Use the generated subflow and associated resources in the normal way within the IIB Toolkit


� Right-click menu from a “Focus Element” in the Data Analysis Model offers highlighting options.

� Choosing Highlight All Coexisting Elements: The percentage in the square brackets [nn%] shows the percentage of the instance documents containing the Focus Element which also include the element in question.

66.7% of those instance documents containing

TopLevel_Element2 also contain TopLevel_Element1

Data Analysis ToolsHighlight co-existing elements


Descendant “0”

(Volume Element itself!)

Descendant “1” Elements(e.g. Appendix)

Descendant “2” Elements(e.g. Bibliography)

Descendant “3” Elements(e.g. Bibliography)

Descendant “4” Elements(e.g. Name, Author)

Data Analysis ToolsHighlighting the Min and Max Depth of Descendants


Manufacturing Industry ScenariosDiscovering trends in real-time data in flight

Rig

Mine

Factoryππππr2 h


A Pattern for MessageSight Integration


� Provide business insight during integration data fl ows– e.g. intelligent decision making; score then action in-flight request based on a business rule– User creates (e.g.) if-then-else rules– The bus acts on these rules in flow, e.g. for business level routing

� New Decision Service node– Identifies inputs to business rules from in-flight data

• e.g. the customers order from whole request• e.g. the item price from key fields…

– Invokes the built-in rule engine– Captures rules output for downstream processing

� Create rules directly inside Integration Bus toolki t– Significant rules authoring facility built-in– Automatic package & deploy with integration assets– Dynamically reconfigure business rule– Optionally refer to business rules on external ODM decision server– Exploit separate full ODM Decision Center for BRMS scenarios

� Embedded rules engine for high performance– Rule is executed in the same OS process as integration data flow– Rule update notification ensures consistent rule execution– Optional governance of rules through remote ODM Decision Center

Decision Management


Applying Analytics to In-flight Data

� Analytics node for model based decision making– Find & express patterns in data with analytics models– Analytics equivalent to Business Decision node

• Pluggable engine for e.g. R, SPSS, SAS…– 2 key scenarios are “model score” and “model trend”– e.g. %buy additional item, SKU lower than expected

� Define the model in tools– This is a high value skill; understand & express behaviour– Use historic dataset; this is typically offline scenario– Both built-in tooling and external model import/reference

� Deploy/Change the Model– Model is encoded into integration flow logic– Deployed with integration solution– Analytics policy for dynamic change without redeploy– Optionally packaged as part of Shared Library Support

� Using the model in real time– Act on these models in integration flow – Scoring: Synchronous use of model score real-time data– Observing: Compare models in real-time for divergence

� Key, related considerations– Shared Libraries required with dynamic linkage

• All Applications using library “see” re-deploy


Analytics Node

� Demand is growing for analytics to be a real-time activity� As data flows through the enterprise, IIB has visibility to score it

against a predictive model� Data Scientist Role

– Prepares a model based on an analytics engine. – For example R, SPSS, SAS

� Integration Developer Role– Formats a data stream and applies it to a model

� Analytics Node– R Scalar variable types: double, integer, character (string),

logical (Boolean)– Data frames can be considered like database tables, consisting

of labelled and typed columns and unlimited rows� Configuration of input and output parameters

– XPath expressions point to locations in the input and output trees

– Direction of Parameter allows a single properties table to control tree copying and return results from the scoring process

Score


Healthcare Industry ScenariosDiscovering trends in real-time data in flight

� Operational

– KPI’s

– Retrospective view of performance

� Clinical insights

– Real Time Analytic Processing

– Interventive care from insight into longitudinal care records

� Cognitive Analytics

– Assisted treatment/diagnosis

Data Baby!


� Almost 25% of the population is over 65, and that number is growing

� Medical advances mean people are living longer

� Services for the elderly account for almost 50% of the social services budget

� Many more elderly people are choosing to remain at home, even when

they are alone

� Ensure their safety and provide needed services but the city had to find a

cost effective way to know when its people needed help

� A mesh-network of sensors that monitor the home environment—

temperature, CO2, water leaks, etc.—of elderly citizens living alone

� Additional home remote medical interaction with medical professionals,

saving trips to the doctor

� It all works with a little help from “angels” (relatives or friends of the user)

who are alerted if there is a problem

� A new model of social and health service that operates on existing budgets

and resources, even as the elderly population increases

� Provides a technological, but still human, system of care via the remote

“angels”—the user can be independent, but not feel alone

� Social service and health staff can concentrate on people who really need a

physical presence with them, while those in the monitoring program

maintain an excellent quality of life

http://www.youtube.com/watch?v=kDvW8R4BL0I


Waste ManagementCombining the Internet of Things, Big Data, Analytics and Mobile!

• Weight and type of waste

• Excess of waste

• Optimization of the collection path

• Exception management (bins in

wrong places, need of additional

bins, replacement of bins etc.)

• Send/receive working orders

to/from SAP

Central Acquisition System

Field Management System

SAP

DB2

IIB

WAS (J2EE app.)

MQTT client

Worklight Application

GPSBPM ODM

MessageSight

HTTP(s)

MQTT

GPRS/3G

RFID reader


Slope aware power train

optimization

Flooding/Slippery risk aware

Driving alert

100

Dynamic/Variable Speed Limit

alert & speed control

Bus

Signal status aware speed

control going thru crossingHeight/load limit aware fleet

driving alert & detouring

Accident/congestion aware

detouring & navigation

Dynamic parking space

availability navigation

Passenger crowd aware bus

dynamic speed management

Environment pollution surveillance

traffic fencing control & fleet alert

!

!

Co2

!

:-)

!

��

!

Low Bridge

The Connected CarLocation Awareness: tracking where things are and how things move!

actionable insights - thompson

Data & Analytics

ibm corporation mqtt

ibm corporation solutions

ibm corporation capture

power of mqtt

ibm messagesight mqtts

wso2 mqtt

big data

mqtt implementations