Structuring Data from Unstructured Things

Sean Lorenz, Founder & CEO, Senter@seanlorenz | @SenterIoT

WE CURRENTLY LIVE IN THE

INTERNET OF THING ERA

IS THIS AS GOOD AS THE IOT GETS?

WE WANT ADAPTIVE APPS

FROM CROSS-MANUFACTURER DATA

APIs ARE THE LIFEBLOOD OF THE IOT

WHAT DO YOU DOWITH ALL THAT DATA?

NO MORE UNTAPPEDDATA LAKES

NO MORE BADIoT DASHBOARDS

MORE DATA STRUCTURING & SENSEMAKING

BUILDING DATA-DRIVEN IOT APPS

1 2 3 4INGEST ORGANIZE PREDICT ACCESS/STORE

2ORGANIZE

timeseries images text sparse binary sparse analog

INGEST - PROCESS DIFFERENT DATA TYPES

ORGANIZE - THE SOLUTION??

ORGANIZE - Python Data Analysis Library (pandas)

• A fast and efficient DataFrame object for data manipulation;

• Tools for reading and writing data between in-memory data structures and different formats;

• Intelligent data alignment and integrated handling of missing data

• Columns can be inserted and deleted from data structures for size mutability;

• High performance merging and joining of data sets;

• Time series-functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging. Even create domain-specific time offsets and join time series without losing data

http://pandas.pydata.org/

ORGANIZE - Python Data Analysis Library (pandas) http://pandas.pydata.org/

ORGANIZE - Handling Time Series Data

A few things to remember:

• Schema design that minimizes memory, disk I/O

• How often do you aggregate the data?

• Read/write to a database needs to be fast, reliable, scalable, adaptable

• Dealing with uneven time period data inputs

• How much of the raw data do you keep?

• Appending existing vs. creating new DataFrames

ORGANIZE - 3 Examples

Twitter, Fitbit, Temperature

timeseries images text sparse binary sparse analog

Deep RNN & LSTM coding of electrical activity to categorize activity peaks

Deep RBM coding of facial anomaly detection from security cameras

Deep RBM coding of CRM keywords & phrases for concept clustering

Sparse PCA & LASSO of ERP system data for delivery probability

Sparse Bayesian coding of IoT sensor data for smart trigger event notifications

PREDICT - NOT ALL ALGORITHMS ARE CREATED EQUALLY

PREDICT - Google TensorFlow + LSTM RRN + time series data

ACCESS/STORE - So, so, so many options….

multimodal sensor fusion w/ cognitive deep learning

IoT home and health, phone app, & EHR data

The Hub for Adaptive Connected Home Health

+

Loca

tion

Sta

tes

Act

ivit

y Pa

tter

ns

Hz

Band

s

O2

Pred

icti

on

Med

Clu

ster

s

Med

Usa

ge

Pred

icti

on

Patient Health States

PREDICTIVE HOME HEALTH IoT EXAMPLE

CARE PLAN ACTION

Thanks.