small sensors big data barry smyth

122
SMALL SENSORS. BIG DATA. FROM CLARITY TO INSIGHT IN THE WORLD OF THE SENSOR WEB Barry Smyth, INSIGHT Centre for Data Analytics @barrysmyth, [email protected] Tuesday 1 October 13

Upload: university-college-dublin-ucd-research

Post on 25-Apr-2015

450 views

Category:

Technology


2 download

DESCRIPTION

 

TRANSCRIPT

Page 1: Small Sensors Big Data Barry Smyth

SMALL SENSORS. BIG DATA.FROM CLARITY TO INSIGHT IN THE WORLD OF THE SENSOR WEB

Barry Smyth, INSIGHT Centre for Data Analytics@barrysmyth, [email protected]

Tuesday 1 October 13

Page 2: Small Sensors Big Data Barry Smyth

In a typical lifetime ...

Tuesday 1 October 13

Page 3: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 4: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 5: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 6: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 7: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 8: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 9: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 10: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 11: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 12: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 13: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 14: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 15: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 16: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 17: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 18: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 19: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 20: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 21: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 22: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 23: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 24: Small Sensors Big Data Barry Smyth

1 billion breaths 2.5 billion heart beats 100 million litres oxygen 100 trillion cells 50,000 litres water 70 tonnes food 70 million calories 3 years toilet 1 billion km 1 year traffic 50,000 kms walking 0.5 million kWh 250 tonnes coal 50 tonnes waste 12 years work 500 days sick 150,000 yawns 28 years asleep 2 years reading 9 years of TV

Tuesday 1 October 13

Page 25: Small Sensors Big Data Barry Smyth

1 = 10bytes18exabyte

Tuesday 1 October 13

Page 26: Small Sensors Big Data Barry Smyth

1 = 10bytes18exabyte

1000,000,000,000,000,000

Tuesday 1 October 13

Page 27: Small Sensors Big Data Barry Smyth

1 = 10bytes18exabyte

20,000 x all of the printed material in the

US Library of Congress.Or all of the words spoken by humans. Ever!

Tuesday 1 October 13

Page 28: Small Sensors Big Data Barry Smyth

1 = 10bytes18exabyte

6 !hours

But, we now create this much information every

Tuesday 1 October 13

Page 29: Small Sensors Big Data Barry Smyth

Connecting atoms & bits.

From algorithms to data.

Tuesday 1 October 13

Page 30: Small Sensors Big Data Barry Smyth

A PARADIGMSHIFT

algorithmdata

algorithm

dataTuesday 1 October 13

Page 31: Small Sensors Big Data Barry Smyth

VSIT’S NOT (JUST) ABOUT THE DATA

N = all

Messy & Diverse

Reusable

Correlation

N = small

Clean & Uniform

Disposable

Causation

Tuesday 1 October 13

Page 32: Small Sensors Big Data Barry Smyth

computation

sensorsdev

data

Tuesday 1 October 13

Page 33: Small Sensors Big Data Barry Smyth

THE WORLD’S FIRSTSUPERCOMPUTER

Brainchild of Seymour Cray, in 1964 the closet-sized CDC 6600 was the biggest, baddest computer of the age.

5,500 kgs, 480 kb RAM, 3M FLOPs, $60m

Tuesday 1 October 13

Page 34: Small Sensors Big Data Barry Smyth

MOORE’S MAGICAL LAWS

In 1965, Intel co-founder, Gordon Moore, noted a doubling of “computing power” every 1-2 years and predicted that this would continue for at least 10 years...

... this became known as Moore’s Law.

Tuesday 1 October 13

Page 35: Small Sensors Big Data Barry Smyth

Tuesday 1 October 13

Page 36: Small Sensors Big Data Barry Smyth

A SELF-FULFILLING PROPHECY?

CDC 6600

[1964]

IBMPC[1981]

iPHONE 5S[2013]

0.05FLOPs/$

200FLOPs/$

8MFLOPs/$

x4,000 x40,000

Tuesday 1 October 13

Page 37: Small Sensors Big Data Barry Smyth

TO PUT THIS INTO PERSPECTIVE ...

The iPhone 5S is about 60,000 times more powerful that the Apollo 11’s guidance computer.

Tuesday 1 October 13

Page 38: Small Sensors Big Data Barry Smyth

IF MOORE’S LAW APPLIED TO CARS?

“If the auto industry had moved at the same speed ... ...your car today would cruise comfortably at a million miles an hour and probably get a half a million miles per gallon of gasoline. But it would be cheaper to throw your Rolls Royce away

Tuesday 1 October 13

Page 39: Small Sensors Big Data Barry Smyth

RAYKURZWEIL

“A computer that once fit in a building, when I was a student, now fits in my pocket and is one thousand times more powerful despite being a million times less expensive.”

Tuesday 1 October 13

Page 40: Small Sensors Big Data Barry Smyth

MEMORY, DISK SIZE, BANDWIDTH, PIXELS,...

All subject to Moore’s Law like improvements over the past 30 years...

... except for battery power / energy density.

Tuesday 1 October 13

Page 41: Small Sensors Big Data Barry Smyth

Tuesday 1 October 13

Page 42: Small Sensors Big Data Barry Smyth

THE RISE OF THESENSORWEB

Tuesday 1 October 13

Page 43: Small Sensors Big Data Barry Smyth

UBIQUITOUS COMPUTING

Mark Weiser’s 1988 vision for a Post-PC world saw computing evolve from a terminal-based paradigm to one in which computing and computation would simply disappear into the fabric of our world.

Tabs, Pads, Boards ⇒ Smart Dust, the Internet of Things, Wearable Computing

Tuesday 1 October 13

Page 44: Small Sensors Big Data Barry Smyth

THE EMERGINGSENSOR WEB

Tabs

Pads

Boards

Smart Sensors

The Internet of Things

Wearable Computing

Tuesday 1 October 13

Page 45: Small Sensors Big Data Barry Smyth

A MATERIALS SCIENCE DETOUR

Chemistry & Physics ⇒ Novel Materials & Structures

Common Materials ⇒ Next Generation Sensors

Tuesday 1 October 13

Page 46: Small Sensors Big Data Barry Smyth

NOKIA’S MORPH CONCEPT DEVICE

Tuesday 1 October 13

Page 47: Small Sensors Big Data Barry Smyth

CHALLENGES OF PHYSICAL SENSING

Conventional sensors (thermistors, flow meters, photoreceptors).

Biofouling & Calibration.

Robustness, Reliability, Energy & Communications.

Cost, Cost, Cost.

Tuesday 1 October 13

Page 48: Small Sensors Big Data Barry Smyth

SWEATSENSING

Microfluidic, Lab-on-a-Chip, Wearable.

pH sensitive dye &photo-detector.

Accurate, continuous, realtime

Athletic performance ⇒ Cystic Fibrosis

Tuesday 1 October 13

Page 49: Small Sensors Big Data Barry Smyth

UNIVERSAL MOBILE SENSING PLATFORM

Tuesday 1 October 13

Page 50: Small Sensors Big Data Barry Smyth

UNIVERSAL MOBILE SENSING PLATFORM

C A M E R A

MICROPHONES P E E D

L I G H T

O R I E N T A T I O N

HUM IDITY

T E M P E R A T U R E

LOCATION

TOUCH

MOTION

DIRECTIONF I N G E R P R I N T S

Tuesday 1 October 13

Page 51: Small Sensors Big Data Barry Smyth

Connectivityhigh-speed data

Mobilitylocation-aware

Poweralways on

Tuesday 1 October 13

Page 52: Small Sensors Big Data Barry Smyth

THE QUANTIFIED SELF MOVEMENT

A data-rich approach to everyday living.

Gordon Bell (Microsoft) and the My Life Bits project ⇒ Digitizing everyday life.

SenseCam

Tuesday 1 October 13

Page 53: Small Sensors Big Data Barry Smyth

7 YEARS3 MONTHS2 WEEKS 1 PERSON12M PHOTOS1TB

Tuesday 1 October 13

Page 54: Small Sensors Big Data Barry Smyth

Activitiesclassificationsummarisation

Lifestylebehaviourspreferences

Eventssegmentation

clustering

Tuesday 1 October 13

Page 55: Small Sensors Big Data Barry Smyth

Tuesday 1 October 13

Page 56: Small Sensors Big Data Barry Smyth

THE DISRUPTION OF HEALTHCARE

Always-on personal sensing, 24/7/365

The Creative Disruption of Healthcare

Activity and exercise, sleep and moods, food, blood glucose, heart rate, pulse ox, lung function, ...

Tuesday 1 October 13

Page 57: Small Sensors Big Data Barry Smyth

THERE’SAN APPFOR THAT

Tuesday 1 October 13

Page 58: Small Sensors Big Data Barry Smyth

EXERCISE & FITNESS

Runkeeper iPhone/Android

Running, Walking, Biking, ...

Age, gender, weight, ...

Location, pace, duration, climb, calories, heart rate,...

Tuesday 1 October 13

Page 59: Small Sensors Big Data Barry Smyth

TRACKING SLEEP

Basic ‘sleep tracking’based on motion.

Duration vs Movement

Sleep Quality (≈ time/move)

Sleep Notes / Wakeup Moods

Comparative AnalyticsTuesday 1 October 13

Page 60: Small Sensors Big Data Barry Smyth

MOOD & FOCUS

The Melon Headband

Uses EEG to track brain activity to assess ‘focus’.

Tagging, location, and activity information helpsusers to better assess what impacts their focus.

Tuesday 1 October 13

Page 61: Small Sensors Big Data Barry Smyth

FOOD & NUTRITION

Meal logging and nutritionalanalysis.

Manual vs Semi-Automatic.

Calorie goals and diet plans.

Integrated weight tracking.

Tuesday 1 October 13

Page 62: Small Sensors Big Data Barry Smyth

HEART RATE SENSING

Using smartphone camera with your finger. No external sensor required.

Detecting colour changes due to capillary blood-flow.

Tagging, comparative analytics etc

Tuesday 1 October 13

Page 63: Small Sensors Big Data Barry Smyth

BLOOD GLUCOSE

External blood glucose sensorautomatically syncs readings with app.

Readings tagged withmealtime, exercise etc.

Analysis and visualization of trends, logs, stats.

Tuesday 1 October 13

Page 64: Small Sensors Big Data Barry Smyth

MOBILE SPIROMETRY

Using a mobile phone microphone to evaluate lung function.

FVC, FEV, PEF measures.

Audio ⇒ Features ⇒ Machine Learning.

Mean 5.1% error wrt clinical spirometry ⇒ suitable for home-based monitoring.

Tuesday 1 October 13

Page 65: Small Sensors Big Data Barry Smyth

MOBILE SPIROMETRY

Using a mobile phone microphone to evaluate lung function.

FVC, FEV, PEF measures.

Audio ⇒ Features ⇒ Machine Learning.

Mean 5.1% error wrt clinical spirometry ⇒ suitable for home-based monitoring.

Tuesday 1 October 13

Page 66: Small Sensors Big Data Barry Smyth

SENSORS& SPORTS

Profs Brian Caulfield & Niall Moyna (@ CLARITY)

Player Health vs Performance Analysis

Rugby, Athletics, Cycling, Equestrian, Archery, Boxing, GAA, ...

Tuesday 1 October 13

Page 67: Small Sensors Big Data Barry Smyth

AUTOMATIC TACKLE CLASSIFICATION

GPS +Accelerometer

Tuesday 1 October 13

Page 68: Small Sensors Big Data Barry Smyth

CONSUMER-DRIVEN HEALTHCARE?

Towards preventative, sensor-based, data-driven healthcare.

Sparse checkups ⇒ 24/7/365 Sensing

The data is ours to share ...

Apps vs Prescriptions?

Tuesday 1 October 13

Page 69: Small Sensors Big Data Barry Smyth

ALWAYS ONMOBILESENSING

Tuesday 1 October 13

Page 70: Small Sensors Big Data Barry Smyth

SCALINGTuesday 1 October 13

Page 71: Small Sensors Big Data Barry Smyth

vertical scalingTuesday 1 October 13

Page 72: Small Sensors Big Data Barry Smyth

vertical scalingTuesday 1 October 13

Page 73: Small Sensors Big Data Barry Smyth

horizontal scalingTuesday 1 October 13

Page 74: Small Sensors Big Data Barry Smyth

horizontal scalingTuesday 1 October 13

Page 75: Small Sensors Big Data Barry Smyth

PARTICIPATORYSENSINGTuesday 1 October 13

Page 76: Small Sensors Big Data Barry Smyth

ASTHMOPOLIS SMART INHALER

Tuesday 1 October 13

Page 77: Small Sensors Big Data Barry Smyth

PARTICIPATORY SENSING

Tuesday 1 October 13

Page 78: Small Sensors Big Data Barry Smyth

HACKING YOUR COMMUTE

GPS & Navigation Assistants

Map Apps Rule the World

TomTom, Garmin, Google, Apple, Nokia, ...

Tuesday 1 October 13

Page 79: Small Sensors Big Data Barry Smyth

CROWDSOURCED MAPPING (WAZE)

Free smartphone app.

Real-time sensing of users’location, time, speed etc.

x millions of users

= social mapping + traffic flow, alerts, hazards, ...

Tuesday 1 October 13

Page 80: Small Sensors Big Data Barry Smyth

Tuesday 1 October 13

Page 81: Small Sensors Big Data Barry Smyth

Tuesday 1 October 13

Page 82: Small Sensors Big Data Barry Smyth

CITIZEN SENSING PUBLIC TRANSPORT

Roadify (iPhone App)

Status updates for publictransport experiences.

Train, bus, subway, ferry,parking, ...

Opinions ⇒ Alerts, Recommendations, Delays, ...

Tuesday 1 October 13

Page 83: Small Sensors Big Data Barry Smyth

TURNING PEOPLE INTO SENSORS

Participatory/Citizen Sensing

Big, messy data ⇒ real-time insights.

The smartphone as a mobile sensor platform...

... and the willingness of people to contribute to data to causes that matter

Tuesday 1 October 13

Page 84: Small Sensors Big Data Barry Smyth

FROM REALTO VIRTUALSENSORS

Tuesday 1 October 13

Page 85: Small Sensors Big Data Barry Smyth

MINING THE DATA EXHAUST

From Real to Virtual Sensors

Page Views, Read Times, Mouse Movements, Search Queries, Result Clicks, Social Connections, Share, Comments, Likes, Posts, Emails, IMs, ...

Tuesday 1 October 13

Page 86: Small Sensors Big Data Barry Smyth

THE ORIGINAL BIG DATA COMPANY

Mining relevance & reputation from links.

Search logs as sensor data.

Tuesday 1 October 13

Page 87: Small Sensors Big Data Barry Smyth

PAGERANK GOOGLE’S BIG IDEA

The importance of a page as a ranking signal.

Estimating importance from in-links ...

... and PageRank was clever way to count in-links to

accurate estimate importance.

Tuesday 1 October 13

Page 88: Small Sensors Big Data Barry Smyth

GOOGLE’S BIGGER IDEA

$40 billion

Google’s real Bigger Idea was that it’s search engine could sense our intentions through our queries and click ...

... and that it could match this demand with real-time supply through its search adverts.

Tuesday 1 October 13

Page 89: Small Sensors Big Data Barry Smyth

SEARCH LOGS AS SENSOR DATA

“... Web search ... can be likened to a large-scale distributed network of sensors for identifying potential side effects of drugs. There is a potential public health benefit in listening to such signals, and integrating them with other sources of information.”

“Web-Scale Pharmacovigilance: Listening to Signals from the Crowd” J Am Med Inform Assoc. (2013)

Tuesday 1 October 13

Page 90: Small Sensors Big Data Barry Smyth

SENSING DRUG SIDE-EFFECTS

82MQueries

6MUsers

Tuesday 1 October 13

Page 91: Small Sensors Big Data Barry Smyth

SENSING FLU TRENDS

Identified trigger terms correlated we known past outbreaks. Tracked real-time occurrence of these terms, location by location to deliver accurate* regional outbreak maps that correlated well with verifiedCDC data.

Tuesday 1 October 13

Page 92: Small Sensors Big Data Barry Smyth

TURNING BROWSERS INTO BUYERS

Understanding user preferences.

Making personalized suggestions.

Tuesday 1 October 13

Page 93: Small Sensors Big Data Barry Smyth

itemsus

ers

Tuesday 1 October 13

Page 94: Small Sensors Big Data Barry Smyth

itemsus

ers

Correlations between the ratingspatterns of users denote user similarity ...

People like you have also liked ...

Tuesday 1 October 13

Page 95: Small Sensors Big Data Barry Smyth

itemsus

ers

Conversely correlations between the ratings patterns of items denote item similarity ...

If you liked X then you might like Y...

Tuesday 1 October 13

Page 96: Small Sensors Big Data Barry Smyth

MINING USER-GENERATED REVIEWS

Tuesday 1 October 13

Page 97: Small Sensors Big Data Barry Smyth

USER-GENERATED REVIEWS

+‘vesstaff

locationbed

servicebreakfast

-‘vesnoiseelevatorscarpethealth clubpublic transport

Chicago Hotels

Tuesday 1 October 13

Page 98: Small Sensors Big Data Barry Smyth

OPINION AMPLIFICATION

Twitter, FaceBook as a source ofreal-time opinions.

Raw Text ⇒ Sentiment ⇒ Opinion

These days Twitter data has been used to predict election outcomes, box office success, and musical talent ...

Tuesday 1 October 13

Page 99: Small Sensors Big Data Barry Smyth

Participatory sensing as collective intelligence

Human Intelligence + Brute-Force Computation

TOWARDS COLLECTIVE INTELLIGENCE

Tuesday 1 October 13

Page 100: Small Sensors Big Data Barry Smyth

DEALING WITH EMAIL SPAM

Back in 2000 Yahoo had a problem ...

Bots registering free email accounts for the purpose of bulk spam.

How to recognise real people from the spambots?

Luis Von Ahn

Manuel Blum

Tuesday 1 October 13

Page 101: Small Sensors Big Data Barry Smyth

Yahoo! Mail CAPTCHATuesday 1 October 13

Page 102: Small Sensors Big Data Barry Smyth

250mCAPTCHASPER DAY

150kPERSON-HOURSPER DAY

7mPERSONHOURS

45CAPTCHADAYS!

Tuesday 1 October 13

Page 103: Small Sensors Big Data Barry Smyth

What if we coulddo something more with all of this

‘CAPTCHA time’?

Tuesday 1 October 13

Page 104: Small Sensors Big Data Barry Smyth

Tuesday 1 October 13

Page 105: Small Sensors Big Data Barry Smyth

99.1%word-levelaccuracy

1.2bnCAPTCHASin year 1

440mwords

17mbooks

Tuesday 1 October 13

Page 106: Small Sensors Big Data Barry Smyth

GAMES WITH A PURPOSE

In 2003 there were 9bn hours of solitaire played on PCs...

... and these days there are around 70m hours of FarmVille played every week!

It only took about 20m hours of human effort to build the Panama Canal!

Tuesday 1 October 13

Page 107: Small Sensors Big Data Barry Smyth

FOLD.IT - MOLECULAR GAME PLAY

Tuesday 1 October 13

Page 108: Small Sensors Big Data Barry Smyth

HOW WELL DOES IT ALL WORK?

In 2011, players of Foldit helped to decipher the crystal structure of the Mason-Pfizer monkey virus (M-PMV) retroviral protease, an AIDS-causing monkey virus.

Players “produced” an accurate 3D model of the enzyme in just 10 days! This structure had eluded scientists for some 15 years.

Khatib, F. et al. (2011). "Crystal structure of a monomeric retroviral protease solved by protein folding game players". Nature Structural & Molecular Biology 18 (10): 1175

Tuesday 1 October 13

Page 109: Small Sensors Big Data Barry Smyth

BIG DATA OR BIG BROTHER?

Tuesday 1 October 13

Page 110: Small Sensors Big Data Barry Smyth

THE END OF THE AGE OF PRIVACY?

“Technology is neither good nor bad, nor is it neutral”

Public by Default.

The Price of Free?

Ownership of Personal Data?

Tuesday 1 October 13

Page 111: Small Sensors Big Data Barry Smyth

THE END OF ANONYMITY

The Case of AOL Searcher No. 4417749.

20M anonymized queries, 600k users as research data (AOL, 2006).

User No. 4417749 = 62 year old Thelma Arnold of Lilburn, Ga.

Tuesday 1 October 13

Page 112: Small Sensors Big Data Barry Smyth

THE PANOPTICON STATE?

Zamyatin’s dystopian glass-walled future of government surveillance.

NSA Prism programme.

Tuesday 1 October 13

Page 113: Small Sensors Big Data Barry Smyth

A shift in the data ownership model ⇒ a new asset class for personal data?

Owned by the individual ⇒ shared with services.

Cloud storage (e.g. DropBox) as a shareable repository of personal data...

CONTROLLING PERSONAL DATA

Tuesday 1 October 13

Page 114: Small Sensors Big Data Barry Smyth

THE BIG DATA WORLD OF THE SENSOR WEB

Tuesday 1 October 13

Page 115: Small Sensors Big Data Barry Smyth

N = ALLMESSYCORRELATION

Tuesday 1 October 13

Page 116: Small Sensors Big Data Barry Smyth

THE OPTION-VALUE OF BIG DATADATA-DRIVEN EVERYTHINGPOWER TO THE PEOPLE

Tuesday 1 October 13

Page 117: Small Sensors Big Data Barry Smyth

THE OPTION-VALUE OF BIG DATA

Reuse & Recycle

From Primary to Secondary Uses of Data

The Unintended Consequences of Data

Tuesday 1 October 13

Page 118: Small Sensors Big Data Barry Smyth

DATA-DRIVEN EVERYTHING

Social Science, Linguistics, Anthropology, Cultural Studies, Journalism, Political Science, Humanities ...

All impacted by Big Data Thinking...

Tuesday 1 October 13

Page 119: Small Sensors Big Data Barry Smyth

GOOGLE’S N-GRAM VIEWER

Acerbi A, Lampos V, Garnett P, Bentley RA (2013) The Expression of Emotions in 20th Century Books. PLoS ONE 8(3)

Tuesday 1 October 13

Page 120: Small Sensors Big Data Barry Smyth

DATA-DRIVEN EVERYTHING

Michel J-P, Shen YK, Aiden AP, Veres A, Gray MK, et al. (2011) Quantitative analysis of culture using millions of digitized books. Science 331: 176–182

Lieberman E, Michel J-P, Jackson J, Tang T, Nowak MA (2007) Quantifying the evolutionary dynamics of language. Nature 449: 713–716

Richards, Daniel Rex. "The content of historical books as an indicator of past interest in environmental issues." Biodiversity and Conservation (2013): 1-9.

Lampos, Vasileios, et al. "Analysing Mood Patterns in the United Kingdom through Twitter Content." arXiv preprint arXiv:1304.5507 (2013).

Tuesday 1 October 13

Page 121: Small Sensors Big Data Barry Smyth

POWER TO THE PEOPLE

Personal Data & Personal Analytics

People as Sensors in Participatory Sensing

Human Computation & Collective Intelligence

Tuesday 1 October 13

Page 122: Small Sensors Big Data Barry Smyth

Creating a Data-Driven SocietyTuesday 1 October 13