where is the big data industry going? from structure:data 2013

DATA

“What Data Can’t Do”

David Brooks Op-Ed, February 13th 2013

“Big Data has trouble with Big Problems”

- David Brooks

Advertising

Advertising Prediction

DATA:SCIENCE

Jeff Hammerbacher! DJ Patil!

Data Science

Kosinski et al PNAS 2013

Liking Curly Fries on Facebook is top predictor of intelligence

DATA:SCIENCE

DATA:intelligence

Iraq:2007

War IED

Open Source Data Collection

Iraq Fr

eque

ncy

Attack Size

xmin

α = 2.31

Iraq Fr

eque

ncy

Attack Size

xmin

α = 2.31 PREDICT?

Insurgent model: Nature 2009

Zhao et al, Phys. Rev. Lett. 2009

Impact of increasing troop numbers

DATA:SCIENCE

DATA:intelligence vs.

Data:SCIENCE Data:Intelligence

Improvement 10% 10x

Data:SCIENCE Data:Intelligence

Improvement 10% 10x

Model goal Predict/Optimize Create/Change

Data:SCIENCE Data:Intelligence

Improvement 10% 10x

Model goal Predict/Optimize Create/Change

Decision Algorithm Human

Data:SCIENCE Data:Intelligence

Improvement 10% 10x

Model goal Predict/Optimize Create/Change

Decision Algorithm Human

Data Big/Clean Small/Messy

Data:SCIENCE Data:Intelligence

Improvement 10% 10x

Model goal Predict/Optimize Create/Change

Decision Algorithm Human

Data Big/Clean Small/Messy

Communication Equations Stories

Data:SCIENCE Data:Intelligence

Improvement 10% 10x

Model goal Predict/Optimize Create/Change

Decision Algorithm Human

Data Big/Clean Small/Messy

Communication Equations Stories

Problem Tactical Strategic

Powerpoint Google Excel + +

STRATEGIC TOOLKIT

“What is the most effective way to allocate capital to spur growth in

K-12 education technology”!

**…and how will I know if we’re successful!

“What are the dominant narratives about

climate change in India”!

**…and how does it vary between age groups!

“What are my competitors doing with advanced

flexible display technology”!

**…should I compete or partner with them?!

“What is the structure of the

insurgent groups in Syria”!

**…what is the likely impact of peace keepers!

Five Heuristics

For Using Data to Solve Big Problems

1. Data needs to be designed for human interaction

Human Machine

Human Centered UI

2. Understand limits of human processing.

900 ms

Source: Nanex

3. Data is messy, incomplete and biased.

…Deal with it…

4. Data needs theory

The future will look like the past…

Build model:Understand

5. Data needs stories… stories need data

Equa%on  governing  Insurgent  Dynamics  

Mathematical

Myth:Hydra

BIG DATA & BIG PROBLEMS

DATA:intelligence

Human + Machine

Interface

Human + Machine