mongodb and the internet of things

MongoDB and the Internet of Things

July 28, 2015

Chris Biow Sr. Solutions Architect chris.biow@mongodb.com

芒果数据 Power Your IoT App with MongoDB

芒果数据

ABOUT SPEAKER

@chris_biow Search Engines Post-relational Databases

AGENDA

IoT Overview & Use Cases Architecture & Challenges Agility & Scalability with MongoDB Powered by MongoDB Case Study

WHAT’S IoT? BIG DATA

IoT CLOUD

It’s a BUZZWORD! TOP BUZZWORDS!

Internet 4.0

The Evolution of Internet

Source: http://postscapes.com/what-exactly-is-the-internet-of-things-infographic

What’s in IoT?

CONNECTED COW by VITAL HERD

E-pill ingested into stomach Transmits heart rate, temp, chemical composition Notifies farmer when abnormality is detected Health management 94 Million Cows in US, Billions of savings

MyJohnDeere

Source: Cisco.

Source: GSMA.

Source: IDC.

TECHNOLOGY STACK

Hardware Platform: Arduino, Raspberry Pi, Intel Edison

Wireless Transport: Zigbee, Z-Wave, WIFI, GPRS, Bluetooth-LE

Communication Protocol: MQTT, CoAP, XMPP, AMQP, RESTful

Middleware and Storage: Application servers, Database Servers

Value Delivery: Business Analytics, User Access & Control

CHALLENGES

Hardware Platform

Wireless Transport

Communication Protocol

Middleware and Storage

Value Delivery

Sensor interface not standard

Long last, efficient connectivity

Variable data format Enormous data volume

CASE STUDY – Airplane Tracking MH-370

Variable Data Structure

location: [ 38.2031, -120.4904 ] , speed: 750, altitude: 29384, engine: fuel_level: 78% , temperature: 89, EPR: xx N-value: { N1: xxx, N2: xxx, N3: xx} …

ADS-C, HFDL, ASDI, EUROCONTROL, ACARS

Multiple sources

Multiple forms

SAMPLE DESIGN 1

EVENT_ID PLANE_ID TIMESTAMP LAT LONG ENGINE TEMP

FUEL LEVEL

… SPEED

100001 3902 1437297148810 38.2031 -124.4904

100002 3902 1437297149213 750

Modeling all metrics as columns in one relational table

Huge table, lots of wasted space caused by empty values Frequent schema change and data migrations when adding new metrics

SAMPLE DESIGN 2

EVENT_ID METRIC_NAME METRIC_VALUE

100001 LAT 38.2031

100001 LONG -124.4904

100002 SPEED 750

Store variable metrics in an EAV table

EVENT_ID PLANE_ID TIMESTAMP

100001 3902 1437297148810 METRIC_VALUE needs be defined as TEXT field Index implication for METRIC_VALUE field Multiple self joins necessary

Enormous Data Volume

A single flight, per minute interval: 3 * 60 * 100 = 18K data points/flight 100,000 flights per day: 1.8 Billion, 1.8TB per day 21,000 QPS

Managing IoT data is hard …

25

LET

POWER YOUR NEXT IoT SOLUTION

26

Expressive Query Language

Strong Consistency

Secondary Indexes

Flexibility

Scalability

Performance

Relational NoSQL Nexus Architecture

AGILITY SCALABILITY

AGILITY

Start coding now, without month long ER design.

Changing schema as you go without penalty.

Flexible schema models variable structure with ease

location: (-84.2391, 34.1039) speed: 750 engine: fuel_level: 100 , temperature: 88.48

DATA MODEL

1

3

2

1 Variable data structure Sparse Indexes Dynamic Schema

2

3

Find all planes within 20km of New York

QUERY EXAMPLE

OPTIMIZE

With document model

A time series is a sequence of data points, typically consisting of successive measurements made over a time interval. Examples of time series are ocean tides, counts of sunspots, and the daily closing value of the Dow Jones Industrial Average. --wikipedia

BUCKETING OPTIMIZATION of TIME SERIES DATA

{ plane_id: "3209", hour: ISODate("2014-07-03T16:00:00.000Z"), metrics: { engine_fuel: { "0"： 99， "1"： 98.5， "2"： 98， ... "59": 69 }, avg: 81.4 } }

{ plane_id: "3209", ts: ISODate("2014-07-03T16:00:00.000Z") metrics: { engine_fuel: 99 } }, { plane_id: "3209", ts: ISODate("2014-07-03T16:01:00.000Z") metrics: { engine_fuel: 98.5 } }, { plane_id: "3209", ts: ISODate("2014-07-03T16:02:00.000Z") metrics: { engine_fuel: 98 } }

. . . { plane_id: "3209", ts: ISODate("2014-07-03T16:59:00.000Z") metrics: { engine_fuel: 69 } }

60:1

•  Less docs – space savings •  Write performance - less index entries •  Queryable & better analytics support

SCALABILITY Shared-nothing, scales horizontally, linearly

Auto-balance ensures a balanced cluster

SHARDED CLUSTER config

config

config

CHOOSING A SHARD KEY FOR SENSORS

Cardinality - LARGE

Write distribution - EVEN

Query isolation – ISOLATED

CHOOSING A SHARD KEY

Cardinality Write distribution Query isolation Reliability Index locality

Cardinality Write Distribution

Query Isolation Reliability Index

Locality

_id Doc level One shard Scatter/gather All users affected Good

hash(_id) Hash level All Shards Scatter/gather All users affected Poor

asset_id Many docs All Shards Targeted Some assets affected Good

asset_id, ts Doc level All Shards Targeted Some assets affected Good

芒果数据

Production in 16 grams (e.g. under a beer keg)

43

WE CAN HELP MongoDB Enterprise Advanced The best way to run MongoDB in your data center

MongoDB Ops Manager The easiest way to run MongoDB in your datacenter

Production Support In production and under control

Development Support Let’s get you running

Consulting We solve problems

Training Get your teams up to speed.