ibm_iot_architecture_and_capabilities
TRANSCRIPT
© 2015 IBM Corporation
IBM Analytics
IBM Internet of Things - Architecture and Capabilities
Pradeep Natarajan - Manager, IoT Solutions and Services April 30th, 2015
Agenda
• IoT Architectural Components Overview • Informix for the Intelligent Gateway • Informix for the Cloud • IoT Partners and Platforms
2
3
Informix for the Intelligent Gateway
4
Internet of Things Topography
Streams
Deep Analytics Zone
Device/Sensors
Smart Gateways
Sensor Analytics Zone
4
• Gateways can reduce the cost of the backend cloud • Reduces cloud storage by filtering/aggregating/analyzing data locally • Reduces cloud CPU requirements by precomputing values
• Reduces latency since actions can be taken immediately • Intelligent gateways can detect and respond to local events as they
happen rather than waiting for transfer to the cloud
• Some users are not comfortable putting all their data in the cloud • Gateways allow customers to capture and get value from their sensors
without sending data to the cloud
• Protocol Consolidation • Cloud does not need to implement the 100’s of IoT protocols
Over time more and more of the processing will move from the cloud to gateway devices
How Do Gateways Help IoT Solutions?
5
What are the Requirements for a Gateway Database?• The database management system must:
▪ Have a small install footprint, less than 100 MB ▪ Run with low memory requirements – less than 256 MB ▪ Use lossless compression or other techniques to minimize
storage space ▪ Have built-in support for common types of IoT data like time series
and spatial/GIS data ▪ Simple application development supporting both NoSQL and SQL ▪ Driverless, easy access to the data ▪ Require absolutely no administration ▪ Ideally should be able to network multiple gateways together to
create a single distributed database
6
The database must be powerful enough to ingest, process and analyze data in real-time
IBM Informix: The Ideal Database for Gateways
Simple to use ▪ Hands-Free operation – No administration ▪ Supports popular interfaces such as REST and Mongo as well
as ODBC/JDBC ▪ Handles SQL and JSON data in the same database
Performance ▪ One of a kind support for Time Series and Spatial data ▪ Stream data continuously into the database ▪ Run analytics as data arrives ▪ Dynamically add and update analytics when needed ▪ Storage is typically 1/3 the size compared to other vendors
Invisible
Agile
7
Informix is the only database management system perfectly suited to run in Gateways
Sensor Data is Time Series Data• What is a Time Series?
▪ A logically connected set of records ordered by time
• What are the Key Strengths of Informix TimeSeries? ▪ Much less space required
• Typically about 1/3 the space required by other vendors ▪ Queries run orders of magnitude faster
• Unique optimized storage means codes paths are shorter and more data fits in memory
▪ Purpose built streaming data loader for sensor data • Automatically run analytic and/or aggregate functions on new data
▪ Can store structured (SQL) or unstructured (JSON) data for quick application development • REST/ODBC/JDBC/JSON interfaces available
▪ 100’s of functions predefined • Programming APIs available to create your own analytics
8
Traditional Table Approach
Informix TimeSeries Approach
Meter_ID Time KWH Voltage ColN1 1-1-11 12:00 Value 1 Value 2 ……… Value N2 1-1-11 12:00 Value 1 Value 2 ……… Value N3 1-1-11 12:00 Value 1 Value 2 ……… Value N… … … … ……… …1 1-1-11 12:15 Value 1 Value 2 ……… Value N2 1-1-11 12:15 Value 1 Value 2 ……… Value N3 1-1-11 12:15 Value 1 Value 2 ……… Value N… … … … ……… …
Meter_ID Series
1 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
2 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
3 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
4 [(1-1-11 12:00, value 1, value 2,…, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]
…
Traditional Sensor data storage vs Informix TimeSeries Storage
9
IoT Requirements for SpatioTemporal Data
• Many IoT applications have a spatial component to them
▪ Vehicles, cell phones, even pets… tracking is common
• In these cases both location and time is important ▪ Show me the vehicles that have
passed by location X in the last hour
▪ Where has my car been over the last few hours?
• Informix allows you to combine Time series and Spatial data in the same query
10
Informix Handles Structured and Unstructured Data• There is no common format for sensor data
▪ Vendor 1 sends just temperature, Vendor 2 sends temperature plus pressure
• Informix is the only product to handle structured and unstructured data simultaneously in the same database ▪ Store meta data about your devices (e.g. manufacturer) as JSON ▪ Store device history data (e.g. temperature) as SQL time series
• Informix translates between structured and unstructured on the fly ▪ Allows SQL apps to access the unstructured data and Mongo apps to access
the structured data ▪ Structured and unstructured data can be combined in a single query
• “Give me all the temperature data for sensors made by manufacturer X between Jan 1 and Feb 1”
11
Data Access Options
12
MongoDB Client
REST Client
SQLI Client
DRDA Client
Informix DBMS
Informix NoSQL Listener
Informix
• NoSQL ↔ SQL Translation • REST, MongoDB Protocol
Support
• SQLI, DRDA Protocol Support • Relational, Collection, Time
Series, and Spatial Data Support
Spatial
Time Series
JSON Collection
Relational Table
A REST client is any program capable of
making a HTTP request
Data Access Options
13
MongoDB Client
REST Client
SQLI Client
DRDA Client
Informix DBMS
Informix NoSQL Listener
Informix
• NoSQL ↔ SQL Translation • REST, MongoDB Protocol
Support
• SQLI, DRDA Protocol Support
• Relational, Collection, Time Series, and Spatial Data Support
Spatial
Time Series
JSON Collection
Relational Table
You can use the MongoDB PHP drivers
are available for MongoDB with the
NoSQL Listener
What is Node-Red
• nodered.org • A visual tool for wiring the Internet of
Things • Node.js application. Uses building
blocks of reusable code to control flow of data
14
• “Flows” are created to process data.
• Example: Take data in, transpose, process and store in a database and/or publish to a message broker
• Collaboration among node developers
Node-Red Benefits
• Rapid Development • Simple use with JSON • Simple REST • Simple MQTT • Contributor Nodes – simple to use other services
15
Node-RED
16
IoT Developers - Get Started!
• Gateway starter - https://ibm.biz/BdXr2W • Code samples - https://ibm.biz/BdX4QV • Github - https://github.com/IBM-IoT/
17
18
Informix for the Cloud and Operational Zone
19
What are the IoT Requirements for the Cloud?
• Requirements - similar to gateways (but for different reasons): • Potentially 1000’s of servers means zero administration is a must • Data volume adds up very quickly so low storage overhead is required • Data flows into the cloud continuously and must be processed in real-
time • Must be able to handle time series, spatial, and NoSQL data natively
• Additional requirements • Must be able to scale-out • Must be available as a service
The database must be able to ingest, process and analyze data in real-time
20
Why use Informix in the “Operational Zone”?
Simple to use • Hands-Free operation • Supports REST and Mongo APIs as well as ODBC/JDBC • Stores SQL and JSON database in the same database
Highly Available • Close to zero down time • Partition or Hash your data across servers in the cloud • Dynamically add/remove additional servers
Performance • Continuous High Performance Analytics • Specialized support for Time Series and Spatial data
Invisible
Agile
Resilient
21
Shards: Scale-out your Database across Servers or Gateways
• Distribute data among servers by range or hash partitioning
• Each shard can have an associated secondary server for high availability
• Run queries across all shards or a subset of the shards
• Only shards that could qualify are searched
• Shards are searched in parallel • Ignores shards that are offline
Shards in a
22
IoT Analytics - Operational and Big Data Analytics
• Operational Analytics • Needed when results are required in (near) real-time
• Real-time monitoring, situational detection, correlation of events, e.g. • Dynamic advertising based on customer movement • Real-time equipment failure prediction
• Operational analytics are required in gateways as well as the cloud • Gateways need to aggregate, filter, monitor for conditions
• Analytics must run efficiently while new data is being loaded • Must be able to dynamically add and update analytics in the cloud
and gateways
• Big Data analytics are required when you have the time to do “Deeper/Wider” analysis
• Latency between data arrival and results not an issue • Efficient ETL process from the operational repository is a must
23
TCP/IP
Bulk Loader
SQL Queries (from apps)
Informix Warehouse Accelerator
Compressed DB partition
Query Processor
Data Warehouse
Informix SQL
Query Router Results
Informix Warehouse Accelerator: • Connects to Informix via TCP/IP & DRDA • Analyzes, compresses, and loads to memory
• Copy of (portion of) warehouse • Processes routed SQL query and
• returns answer to Informix
Informix Warehouse Accelerator (IWA) In-Memory Analytics
Informix: • Routes SQL queries to accelerator • User need not change SQL or apps. • Can always run query in Informix
• Too short an est. execution time
24
TimeSeries Service on BlueMix
Every IoT deployment will need to store time series data and possibly spatial data
• Provides REST, JSON, ODBC/JDBC based services to:
• Store, retrieve, and join time series, spatial, relational, and NoSQL data
• Built-in time series analytics
• Multi-tenancy support
• Scales out automatically as needed
• No Administration needed
• Quickly and simply add new sensors
• Interface for continuously loading sensor data
• Adapters for MQTT and MessageSight
IoT Foundation Service on BlueMix
BlueMix Services
Docker Engine
• Open source project
• Supported on every major Linux distro (MS Windows in 2015)
• Client-server architecture with daemon deployed on physical or virtual host
• Uses Linux kernel cgroups and namespaces for process resource management and isolation
• Uses copy-on-write filesystem for git-like image change management
25
A portable, lightweight application runtime and packaging tool built on top of kernel container primitives
Build Ship Run
Docker Engine
Containers
26
App A’
Host OSServer
Bins/LibsApp A App B App B’
App B’
App B’
Container
What is a Container • An isolated user space within a running Linux OS • Shared kernel across containers • Direct device access • All packages and data in an isolated run-time, saved as a
filesystem. • Resource management implemented with cgroups • Resource isolation through namespaces
Run
Bins/Libs
Docker Hub
• Enable sharing and collaboration of Docker Images • Private and public repositories of images • Certified base images by ISVs
27
Ship
Docker Hub
$ docker push $ docker pull
Informix on Docker Hub
https://registry.hub.docker.com/u/ibmcom/informix-innovator-c/
• IBM Informix Innovator-C • 12.10.FC4
28
Docker Hub
$docker pull ibmcom/informix-innovator-c
IoT Partners & Platforms
29
IBM Informix and Intel Deliver Leading Edge Solutions for IoT
▪ Informix support for Intel’s new Quark processor, now supporting Intel family, from Quark to Xeon.
▪ Quark port in the free Informix Developer Edition
▪ Informix Developer Edition available as part of standard Intel Gateway developer SDK
Combining IBM and Intel’s strength at the Edge, Gateway and in the Cloud provides an intelligent, e2e IoT solution for rapid time to market.
Intel® Quark™ SoC / Intel® Atom™ / Intel® Xeon™ Windriver McAfee Security Supports OSGI/TR69 Informix Database
30
Summary• IBM Informix - best fit for IoT architecture
• IoT gateway • IoT cloud analytics
• Supported on a wide array of platforms • Best in class embeddability • Native support for sensor data - TimeSeries & Spatial data • Native support for unstructured (JSON) data • Ease of application development - REST access • High availability and dynamic scaling
31
IBM Analytics © 2015 IBM Corporation33
Legal Disclaimer
• © IBM Corporation 2015. All Rights Reserved. • The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in
this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.
• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.
• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server). Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:UNIX is a registered trademark of The Open Group in the United States and other countries.
• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.
• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.