mongodb and the internet of things
TRANSCRIPT
![Page 1: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/1.jpg)
MongoDB and the Internet of Things
July 28, 2015
Chris Biow Sr. Solutions Architect [email protected]
![Page 2: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/2.jpg)
芒果数据 Power Your IoT App with MongoDB
![Page 3: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/3.jpg)
芒果数据
ABOUT SPEAKER
@chris_biow Search Engines Post-relational Databases
![Page 4: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/4.jpg)
AGENDA
IoT Overview & Use Cases Architecture & Challenges Agility & Scalability with MongoDB Powered by MongoDB Case Study
![Page 5: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/5.jpg)
WHAT’S IoT? BIG DATA
IoT CLOUD
It’s a BUZZWORD! TOP BUZZWORDS!
![Page 6: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/6.jpg)
Internet 4.0
The Evolution of Internet
![Page 7: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/7.jpg)
Source: http://postscapes.com/what-exactly-is-the-internet-of-things-infographic
What’s in IoT?
![Page 8: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/8.jpg)
![Page 9: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/9.jpg)
![Page 10: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/10.jpg)
![Page 11: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/11.jpg)
CONNECTED COW by VITAL HERD
E-pill ingested into stomach Transmits heart rate, temp, chemical composition Notifies farmer when abnormality is detected Health management 94 Million Cows in US, Billions of savings
![Page 12: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/12.jpg)
MyJohnDeere
![Page 13: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/13.jpg)
Source: Cisco.
![Page 14: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/14.jpg)
Source: GSMA.
![Page 15: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/15.jpg)
Source: IDC.
![Page 16: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/16.jpg)
TECHNOLOGY STACK
Hardware Platform: Arduino, Raspberry Pi, Intel Edison
Wireless Transport: Zigbee, Z-Wave, WIFI, GPRS, Bluetooth-LE
Communication Protocol: MQTT, CoAP, XMPP, AMQP, RESTful
Middleware and Storage: Application servers, Database Servers
Value Delivery: Business Analytics, User Access & Control
![Page 17: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/17.jpg)
![Page 18: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/18.jpg)
CHALLENGES
Hardware Platform
Wireless Transport
Communication Protocol
Middleware and Storage
Value Delivery
Sensor interface not standard
Long last, efficient connectivity
Variable data format Enormous data volume
![Page 19: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/19.jpg)
CASE STUDY – Airplane Tracking MH-370
![Page 20: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/20.jpg)
Variable Data Structure
location: [ 38.2031, -120.4904 ] , speed: 750, altitude: 29384, engine: fuel_level: 78% , temperature: 89, EPR: xx N-value: { N1: xxx, N2: xxx, N3: xx} …
ADS-C, HFDL, ASDI, EUROCONTROL, ACARS
Multiple sources
Multiple forms
![Page 21: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/21.jpg)
SAMPLE DESIGN 1
EVENT_ID PLANE_ID TIMESTAMP LAT LONG ENGINE TEMP
FUEL LEVEL
… SPEED
100001 3902 1437297148810 38.2031 -124.4904
100002 3902 1437297149213 750
Modeling all metrics as columns in one relational table
Huge table, lots of wasted space caused by empty values Frequent schema change and data migrations when adding new metrics
![Page 22: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/22.jpg)
SAMPLE DESIGN 2
EVENT_ID METRIC_NAME METRIC_VALUE
100001 LAT 38.2031
100001 LONG -124.4904
100002 SPEED 750
Store variable metrics in an EAV table
EVENT_ID PLANE_ID TIMESTAMP
100001 3902 1437297148810 METRIC_VALUE needs be defined as TEXT field Index implication for METRIC_VALUE field Multiple self joins necessary
![Page 23: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/23.jpg)
Enormous Data Volume
A single flight, per minute interval: 3 * 60 * 100 = 18K data points/flight 100,000 flights per day: 1.8 Billion, 1.8TB per day 21,000 QPS
![Page 24: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/24.jpg)
Managing IoT data is hard …
![Page 25: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/25.jpg)
25
LET
POWER YOUR NEXT IoT SOLUTION
![Page 26: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/26.jpg)
26
Expressive Query Language
Strong Consistency
Secondary Indexes
Flexibility
Scalability
Performance
Relational NoSQL Nexus Architecture
![Page 27: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/27.jpg)
AGILITY SCALABILITY
![Page 28: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/28.jpg)
AGILITY
Start coding now, without month long ER design.
Changing schema as you go without penalty.
Flexible schema models variable structure with ease
![Page 29: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/29.jpg)
location: (-84.2391, 34.1039) speed: 750 engine: fuel_level: 100 , temperature: 88.48
DATA MODEL
1
3
2
1 Variable data structure Sparse Indexes Dynamic Schema
2
3
![Page 30: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/30.jpg)
Find all planes within 20km of New York
QUERY EXAMPLE
![Page 31: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/31.jpg)
OPTIMIZE
With document model
A time series is a sequence of data points, typically consisting of successive measurements made over a time interval. Examples of time series are ocean tides, counts of sunspots, and the daily closing value of the Dow Jones Industrial Average. --wikipedia
![Page 32: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/32.jpg)
BUCKETING OPTIMIZATION of TIME SERIES DATA
{ plane_id: "3209", hour: ISODate("2014-07-03T16:00:00.000Z"), metrics: { engine_fuel: { "0": 99, "1": 98.5, "2": 98, ... "59": 69 }, avg: 81.4 } }
{ plane_id: "3209", ts: ISODate("2014-07-03T16:00:00.000Z") metrics: { engine_fuel: 99 } }, { plane_id: "3209", ts: ISODate("2014-07-03T16:01:00.000Z") metrics: { engine_fuel: 98.5 } }, { plane_id: "3209", ts: ISODate("2014-07-03T16:02:00.000Z") metrics: { engine_fuel: 98 } }
. . . { plane_id: "3209", ts: ISODate("2014-07-03T16:59:00.000Z") metrics: { engine_fuel: 69 } }
60:1
• Less docs – space savings • Write performance - less index entries • Queryable & better analytics support
![Page 33: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/33.jpg)
SCALABILITY Shared-nothing, scales horizontally, linearly
Auto-balance ensures a balanced cluster
![Page 34: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/34.jpg)
SHARDED CLUSTER config
config
config
![Page 35: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/35.jpg)
CHOOSING A SHARD KEY FOR SENSORS
Cardinality - LARGE
Write distribution - EVEN
Query isolation – ISOLATED
![Page 36: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/36.jpg)
CHOOSING A SHARD KEY
Cardinality Write distribution Query isolation Reliability Index locality
Cardinality Write Distribution
Query Isolation Reliability Index
Locality
_id Doc level One shard Scatter/gather All users affected Good
hash(_id) Hash level All Shards Scatter/gather All users affected Poor
asset_id Many docs All Shards Targeted Some assets affected Good
asset_id, ts Doc level All Shards Targeted Some assets affected Good
![Page 37: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/37.jpg)
芒果数据
![Page 38: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/38.jpg)
![Page 39: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/39.jpg)
![Page 40: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/40.jpg)
![Page 41: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/41.jpg)
![Page 42: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/42.jpg)
Production in 16 grams (e.g. under a beer keg)
![Page 43: MongoDB and the Internet of Things](https://reader034.vdocument.in/reader034/viewer/2022042716/55cb202fbb61eb42638b4678/html5/thumbnails/43.jpg)
43
WE CAN HELP MongoDB Enterprise Advanced The best way to run MongoDB in your data center
MongoDB Ops Manager The easiest way to run MongoDB in your datacenter
Production Support In production and under control
Development Support Let’s get you running
Consulting We solve problems
Training Get your teams up to speed.