tibco advanced analytics meetup (taam) november 2015

TIBCO Advanced Analytics Meetup : Q4 2015Michael O’ConnellChief Data [email protected]@MichOConnell

November 2015

• TIBCO Analytics & Data Science (Michael)• Data Analysis Pipeline

• Visual Analytics & Dashboards (Catalina Herrera)• Building Dashboards on the fly

• Predictive Analytics (Peter Shaw, Anna Nowakowska)• Simple Data Functions (R and TERR)

• Customer Analytics (Churn & Value)

• Data Access & APIs (Andrew Berridge, Mathew Lee)• Iron Python

• Advanced GeoAnalytics

• Wrap-Up / Questions (Michael)• Resources, Training, Links

Increase Productivity

Grow Revenue

Value

Reduce Risk

ROI

TIBCO Advanced Analytics Meetup (TAAM) Agenda

© Copyright 2000-2015 TIBCO Software Inc.

Data Access & Prep

Exploratory Data Analysis Features Visual

DashboardModel & Predict

Deploy Champion

Model

Test & Learn

Channel

Social

Loyalty

Campaign

Filter

Map

Merge

Shape

Propensity

Affinity

Improve Guided -------- Deploy -------- In-LineExplore Data

Aggregate

Prepare DataBusiness Case


Grow Revenue

EnsembleForest

RegressionAdditive ModelsSegment

Visualize

Pricing

Promotion

ChallengerModels

At Rest

In Motion

Value Theses

Reduce Risk

ROI

Value

Dashboard Updates

Data a Insight a Action


Spotfire Platform


SpotfireDesktop


Spotfire Platform

Immediate Long-Term Competitive AdvantageValue to the Organization

TIBCO is the only analytics platform that provides business value across this Analytics Spectrum

Self-service Dashboards API and CEP

Predictive and Prescriptive Analytics

Measure Diagnose Predict Optimize Operationalize Automate

Analytics Maturity

Insight to Action



Self-service Dashboards


Analytics Maturity

Predictive and Prescriptive Analytics API and CEP

Insight to Action





Analytics Maturity

API and CEP Predictive and

Prescriptive Analytics

Insight to Action



Self-service Dashboards API and CEP



Analytics Maturity

Insight to Action

Spotfire Visual Analytics - Interactive Brush-Linked


Spotfire Visual Analytics + R Extensions

3D rotate SurfacePolar

Contour Network Funnel


Spotfire + JS Extensions

Sankey Venn

ChordDonut

Gantt

WaterfallDials


Dashboards and Themes – KPIs

Dashboards and Themes – Company Specific

Mock-Up Only

Fictitious data

Dashboards and Themes – Color

Map Charts

Map Layer

Marker Layer Feature Layer

Image LayerWMS Layer

Map Layers


Roads

Labels

Borders

Standard map

Standard map - light

Basic map

Basic map - light

Base Map Options


• Color• Shape • Size

• Relative amounts • Size

Marker Layer Feature Layer

• Color

Marker or Feature Layer

• Tooltip

• Labels

Map Elements


Building Dashboards

Catalina Herrera

Jaspersoft Pixel-Perfect Embedded Reports

Analytic Workspaces & Analytic Fabric

APIs

Sear

ch, S

harin

g et

c.

Business Analysts Report Developers

Data Discovery Analytics Dashboards Reports Embedding

Analytic WorkspacesAnalytic environments

Analytic FabricConnectivity layer

Data Professionals

25%

Business Professionals

75%

BusinessApplications

Communication& Collaboration

Platforms






Analytics Maturity

Predictive and Prescriptive Analytics APIs and CEP

From Dashboards to Value

Big Data Analytics


Data a Insight a Action

Data Access & Prep

Exploratory Data Analysis Features Visual

DashboardModel & Predict

Deploy Model

Test & Learn

Channel

Churn

Loyalty

Campaign

Filter

Map

Merge

Shape

Propensity

Affinity

Improve Guided -------- Deploy -------- In-LineExplore Data

Aggregate

Prepare DataBusiness Case


Grow Revenue

EnsembleForest

RegressionAdditive ModelsSegment

Visualize

Pricing

Promotion

ChallengerModels

At Rest

In Motion

Value Theses

Reduce Risk

ROI

Purchase

Dashboard Updates

Fast DataBig Data© Copyright 2000-2015 TIBCO Software Inc.

TIBCO’s Enterprise Runtime for R (TERR)

• TIBCO has rewritten R as a Commercial Compute Engine • Latest statistics scripting engine: S a S-PLUS® a R a TERR• Runs R code including CRAN packages

• Engine internals rebuilt from scratch at low-level• Redesigned data objects, memory management• High performance + Big Data

• TERR is licensed from TIBCO• TERR Installs with Spotfire Analyst / Desktop and other TIBCO products (CEP, Stat Services)• Spotfire Server can manage all TERR / R scripts, artifacts for reuse • Standalone Developer Edition: www.TIBCOmmunity.com • Supported by TIBCO


Spotfire and TERR local TERR on server

Spotfire-TERR – Local and Server

• Build models on data using local TERR engine embedded in Spotfire

• Build models on big data directly in TERR on server and display results in Spotfire

• Run TERR as parallel sessions on Hadoop cluster, controlled and visualized in Spotfire

Data Source TERR TSSS

Spotfire

Results

ODBCJDBCSDCFile

DataFunction

Larger Data

Modeling

Spotfire

LocalTERR

ODBCJDBCSDCFile

Data

Data Source

Both Spotfire and TERR can load data from any ODBC or JDBC compliant source or from Spotfire Data Connections (SDC) or Spotfire Information Links stored in the Spotfire library. © Copyright 2000-2015 TIBCO Software Inc.

Date

R P

acka

ges

1/1/2002 1/1/2003 1/1/2004 1/1/2005 1/1/2006 1/1/2007 1/1/2008 1/1/2009 1/1/2010 1/1/2011 1/1/2012 1/1/2013

5000

4500

4000

3500

3000

2500

2000

1500

1000

500

0

Number of R- or SAS posts to Stack Overflow by week. (copyright by r4stats.com)

Number of contributed packages on CRAN(http://cran.r-project.org/)

> 6,000 Packages !

R Community


126 R User Groups in US and EMEA 6,000 User-Contributed Packages Very Active R Community

Contextual Analytics- Right click- Forecasting

Contextual Analytics- Menu - Machine Learning

Simple Predictive Analytics in Spotfire


Extensible Predictive Analytics – Expressions

Lines by TERR( contourLines(x,y) )

Color by TERR( kmeans(x1,x2) )

Interactive Spotfire Analytics with R/TERR- Expression – on the fly R code- Points: kmeans(), …- Lines: contourLines(), spline(), …- Expression Functions – sharing


Analysis Workflows – Data Functions

Interactive Spotfire Analytics with R- Data Function- Robust Cluster Analysis- Any Analysis in R / CRAN

Variables driving segments- Random Forest

Revenue by product- Color by segment

Analysis Workflows – Route Optimization

Predictive Analytics Dashboards

Mock-Up Only

Fictitious data

Predictive Analytics

Customer Analytics Workflows

Anna NowakowskaPeter Shaw






Analytics Maturity

Event Processing

From Dashboards to Value

BIG DATAAT REST

FAST DATAIN MOTION

Insight to Action

Spotfire Data Access

DATASOURCES

XMLRDBMSFlatFiles CubesSpread-

sheetsHadoop & Big Data

stores

AnalyticalDWs e.g.Exadata

Event Data Streams

ActiveSpaces

In-MemoryLoad data from source in to memory

In-DatabaseLeave data in DBDynamically load and discard data to visualize

On-DemandDynamically swap data in and out of memory.

SQLMDX

10100110

Custom GUI-driven data access via SDK

SiebeleBusiness

Local data sources

AccessExcel STDF

Drag-and-drop

MySQL

SQL ServerOracle

Information Services(join, transform, reusable,

parameterized, dynamic query for in-memory use)

Databases

JDBC/ODBC

HadoopSFDC

PostgreSQL

TeradataNetezza

Etc.XML

RDBMS

FlatFiles

Spread-sheets

WebServices

OracleE-Business

RDBMSRDBMSRDBMS

SAP BWSAP R/3 DATA

FABRIC

Salesforce

ODBCOLE DB

SqlClient

Direct connection

OracleTeradataAsterMS SSAS

Teradata

Direct Query(dynamically query and retrieve data

for visualization and analysis)Databases

MySQLEtc.

OBIEE

NetezzaHadoop


Spotfire Data Access

Winner of 2014 Strata Cloudera AwardFor Best Advanced Analytics Application

Hadoop & Spark

TERR on the nodes of Hadoop Cluster

TERR in Action

• Hadoop cluster compute• TIBCO Cloud Compute Grid• TIBCO Streambase• TIBCO Business Events• KNIME• Lavastorm• Rstudio• Teradata• TIBCO Statistics Services• TIBCO Spotfire

Hadoop & Spark

Non Geographic Data – Map Chart Layers


Non Geographic Data – Interactive Spotfire Charts

Airplane Seating Chart

Football Field Seating Chart


Spotfire Interactive Charts created by TERR script

Spotfire API’s

Spotfire MapChart & GeoAnalytics

Andrew BerridgeMathew Lee

Optimize pricing Check for

fraud

Make offer to customer

Restock inventory

Reroute transport

Give customer service

Proactively maintain machines

Fast Data Analytics – Interventions


Spotfire – Getting Started

Spotfire Education

Contact – Follow me on Twitter

Thank you!Michael O’Connell, PhDChief Data ScientistTIBCO [email protected]@MichOConnellhttp://about.me/moconnell+1-919-7401560


mailto:[email protected]

mailto:[email protected]

http://about.me/moconnell

Creating a TERR data function in

Spotfire

Peter Shaw



• Example shown here: Customer Churn

• Quick look at data

• Create data function in RStudio to draw curve on scatter plot

• Linear

• Quadratic

• Apply model to new data: Current customers.

• Create new column: Probability of churn

• Use this to drill-down to find at-risk customers

Outline


R code

For this demo, the data is read into the R environment directly.Here Rstudio is configured to use the TERR engine as will be used in the data function.Alternatively, the data function can use a save() command to save any incoming objects to a disk file, and then these can be loaded into the interactive R environment.


Data function development: linear model


Quadratic model


Streamlined data function for prediction


• Thanks to Ujval Kamath and Ian Cook for developing early form of this demo

Thanks

TIBCO Advanced Analytics Meetup

Product Update Including APIs

Andrew Berridge, Michael O’ConnellTIBCO Data Science



65

Product Updates and APIs

• IronPython and APIs• Why Use IronPython with the Spotfire APIs?• Custom Sort Order API Example• IronPython Authoring Improvements in Recent Versions of Spotfire

• Spotfire Updates• Styling and Themes• Annotation and Collaboration

Agenda


66

Two Books of Interest

A Picture is Worth a Thousand Tables• Graphics in Life-Sciences• Many useful lessons and analysis

examples for other industries

TIBCO Spotfire – A Comprehensive Primer• Covers how to use Spotfire• Practical real-world examples• A big section on IronPython, including documentation, examples

and code snippets• Covers predictive, geoanalytics and many other areas of Spotfire


67

Why Use IronPython with the Spotfire APIs?Apply Customizations and Automate Multiple Tasks With a Single Click!• Apply customizations:

• Change axes

• Create new visualizations

• Set colorings, update color rules

• Produce analyses customized for users

• Extend features of Web Player (Consumer)• Roll up multiple user actions into a single click

• Work with data:• Set custom sort orders

• Set filters, mark data with “where” clauses

• Load data, refresh data

Using IronPython Automation Services extension, automate IronPython scripts with Automation Services


68

Custom Sort Order API

Introduced in Spotfire 7.0• Important and not well known!

• In the past, it was not possible to programmatically set a custom sort order

• This could lead to unexpected sorting:• If not all values are known during analysis authoring;

• Then the data is refreshed, any new values will not be sorted accordingly

• API now allows sort order to be set (e.g. via an algorithm or external lookup

table)

• Can automate this via Automation Services Run Script extension; or via an

OnLoad IronPython script (triggered from JavaScript)


69

Significant Improvements Since Spotfire 6.0

• Improved debugging

• Line numbers are shown when exceptions are thrown (when using the script

editor)

• New dialog for managing scripts, accessed from Edit->Document

Properties menu

• One click to “Trust All” scripts (IronPython or JavaScript)

IronPython Authoring Improvements


70

Demo!

IronPython Authoring Improvements


71


Spotfire Desktop Client only GUI for altering the sort order…

Here we see fruits sorted in size order

Refresh the data, for example, to add “Plum”, then it will automatically appear at the bottom of the sort order

New API allows us to set a custom sort order


72


Demo!


73

Annotate Visualizations; Comment on and Discuss Visualizations

• New ability to attach annotations to visualizations

• Annotations can be styled and placed anywhere on the visualization

• Collaboration allows conversations between users

• Conversations are recorded alongside the visualizations

Spotfire 7.5 Updates – Collaboration and Annotation


74

Spotfire 7.5 Updates – Collaboration and Annotation

Demo!


Spotfire 7.0 Updates – Theming and StylingSpotfire can now be themed and styled to suit YOUR design!

• Every aspect of the visual styling is customizable

• Set colors and fonts

• Background images, styles and colors

• Borders, visual layout

• Apply the style of one DXP file to another


Spotfire 7.0 Updates – Theming and StylingExamples


77

Demo!

Spotfire 7.0 Updates – Theming and Styling


78

Spotfire 7.5 Updates – Infographics

79


Thank you!

tibco advanced analytics meetup (taam) november 2015

Data & Analytics