iot ( m2m) - big data - analytics: emulation and demonstration
TRANSCRIPT
IOT (M2M) - BIG DATA - ANALYTICS
CHAKER ALLAOUI
EMULATION & DEMONSTRATION
GPS Sensor Philips HUE MQTT
Geo-Location Temperature Commands
Node-Red
Topic - Broker Humidity Data Subscriber
IoT PLATFORM
Publisher
Features Monitoring
Table of Contents
PARTIE II : Emulation
PARTIE III : DemonstrationARCHITECTURE IoT
PARTIE I : Presentation
Cars Trafics
Data-Transfert
Commands
AnalyticsBig DataM2M
Examples
PARTIE I : PRESENTATION
5
The Internet of Things (IoT) is an emerging global Internet-
based information architecture facilitating the exchange
of goods and services. The IoT has the purpose of
providing an IT-infrastructure facilitating the exchange of
“things” in a secure and reliable manner, i.e. its function is
to overcome the gap between objects in the physical
world and their representation in information systems. The
IoT will serve to increase transparency and enhance the
efficiency of global supply chain networks.
In systems engineering, a 'system monitor (SM) is a process
within a distributed system for collecting and storing state
data. This is a fundamental principle supporting
Application Performance Management. The argument
that system monitoring is just a nice to have, and not
really a core requirement for operational readiness,
dissipates quickly when a critical application goes down
with no warning, configuration data for the monitor,
configuration data for the system being monitored.
IoT Platform
6
Features
The Internet of Things is the network of physical objects or "things" embedded with electronics,
software, sensors, and connectivity to enable objects to exchange data with the manufacturer,
operator and/or other connected devices based on the infrastructure of International
Telecommunication Union's Global Standards Initiative. The Internet of Things allows objects to be
sensed and controlled remotely across existing network infrastructure, creating opportunities for more
direct integration between the physical world and computer-based systems, and resulting in
improved efficiency, accuracy and economic benefit. Each thing is uniquely identifiable through its
embedded computing system but is able to interoperate within the existing Internet infrastructure.
Experts estimate that the IoT will consist of almost 50 billion objects by 2020. The term “Internet of
Things” was coined by British entrepreneur Kevin Ashton in 1999. Typically, IoT is expected to offer
advanced connectivity of devices, systems, and services that goes beyond machine-to-machine
communications (M2M) and covers a variety of protocols, domains, and applications. The
interconnection of these embedded devices (including smart objects), is expected to usher in
automation in nearly all fields, while also enabling advanced applications like a Smart Grid, and
expanding to the areas such as Smart city. Things, in the IoT, can refer to a wide variety of devices
such as heart monitoring implants, biochip transponders on farm animals, electric clams in coastal
waters, automobiles with built-in sensors, or field operation devices that assist fire-fighters in search and
rescue. These devices collect useful data with the help of various existing technologies and then
autonomously flow the data between other devices. Current market examples include smart
thermostat and washer/dryers. https://en.wikipedia.org/wiki/Internet_of_Things
7
IoT History
http://electronicdesign.com/embedded/understanding-protocols-behind-internet-things
8
Monitoring (1)
Protocol: There are many tools for collecting system data from hosts and devices using the SNMP
(Simple Network Management Protocol). Most computers and networked devices will have some
form of SNMP access. Interpretation of the SNMP data from a host or device requires either a
specialized tool (typically extra software from the vendor) or a Management information base (MIB), a
mapping of commands/data references to the various data elements the host or device provides.
The advantage of SNMP for monitoring is its low bandwidth requirements and universal usage in the
industries. Unless an application itself provides a MIB and output via SNMP, then SNMP is not suitable
for collecting application data. Other protocols are suitable for monitoring applications, such as
CORBA (language/OS-independent), JMX (Java-specific management and monitoring protocol), or
proprietary TCP/IP or UDP protocols (language/OS independent for the most part).
Data access: refers to the interface by which the monitor data can be utilized by other processes. For
example, if the system monitor is a CORBA server, clients can connect and make calls on the monitor
for current state of an element, or historical states for an element for some time period. The system
monitor may be writing data directly into a database, allowing other processes to access the
database outside the context of the system monitor. This is dangerous however, as the table design
for the database will dictate the potential for data-sharing. Ideally the system monitor is a wrapper for
whatever persistence mechanism is used, providing a consistent and 'safe' access interface for others
to access the data.
Mode: The data collection mode of the system monitor is critical. The modes are: monitor poll, agent
push, and a hybrid scheme. https://en.wikipedia.org/wiki/System_monitoring
9
Monitoring (2)
Monitor poll: In this mode, one or more processes in the monitoring system actually poll the system
elements in some thread. During the loop, devices are polled via SNMP calls, hosts can be accessed
via Telnet/SSH to execute scripts or dump files or execute other OS-specific commands, applications
can be polled for state data, or their state-output-files can be dumped.
The advantage of this mode is that there is little impact on the host/device being polled. The host's
CPU is loaded only during the poll. The rest of the time the monitoring function plays no part in CPU
loading. The main disadvantage of this mode is that the monitoring process can only do so much in its
time. If polling takes too long, the intended poll-period gets elongated.
Agent push: In agent-push mode, the monitored host is simply pushing data from itself to the system
monitoring application. This can be done periodically, or by request from the system monitor
asynchronously. The advantage of this mode is that the monitoring system's load can be reduced to
simply accepting and storing data. It doesn't have to worry about timeouts for SSH calls, parsing OS-
specific call results, etc.
The disadvantage of this mode is that the logic for the polling cycle/options are not centralized at the
system monitor, but distributed to each remote node. Thus changes to the monitoring logic must be
pushed out to each node. Also, in agent-based monitoring, a host cannot inform that it is completely
"down" or powered off, or if an intermediary system (such as a router) is preventing access to the
system.
Hybrid mode: The median mode between 'monitor-poll' and 'agent-push' is a hybrid approach, where
the system configuration determines where monitoring occurs, either in the system monitor or agent.
10
Node-Red
Internet of Things solutions often require pulling together hardware devices, APIs
and online services in new and interesting ways. Time spent writing the boilerplate
code to talk to these different systems, such as accessing a Serial port or complete
an OAuth flow against Twitter, is not time spent on creating the real value of the
solution. We need tools that make it easier for developers at all levels to bring
together the different streams of events, both physical and digital, that make up
the Internet of Things. Node-RED provides a browser-based UI for creating flows of
events and deploying them to its light-weight runtime. With built in nodeJS, it can
be run at the edge of the network or in the cloud. The node package manager
(npm) ecosystem can be used to easily extend the palette of nodes available,
enabling connections to new devices and services.
Node-Red in its simplest form is an open source visual editor for wiring the internet
of things produced by IBM. What does this mean. Well for someone lazy like me it
means I can spend more time making stuff "talk" to each other than worrying
about all of the interfacing code I will need to write. The system contains "Nodes"
which look simply to be icons that you drag and drop on to the canvas and wire
together. Each Node offers different functionality which can range from a simple
debug node to be able to see what's going on in your flow, through to a
Raspberry Pi node which allows you to read and write to the GPIO pins of your Pi.
http://nodered.org
11
Example 1: Mail Data Collect
12
Example 2: Twitter Data Collect
13
Example 2: Mail and Twitter Node-Red Flow
14
Example 3: MQTT broker data in Node-Red
15
Example 3: MQTT broker Node-Red Flow
16
Example 3: MQTT broker HiveMQ Stats
Node-RED provides a browser-based flow editor that
makes it easy to wire together flows using the wide
range nodes in the palette. Flows can be then
deployed to the runtime in a single-click. JavaScript
functions can be created within the editor using the
a rich text editor. A built-in library allows you to save
useful functions, templates or flows for re-use.
The light-weight runtime is built on nodeJS, taking full
advantage of its event-driven, non-blocking model.
This makes it ideal to run at the edge of the network
on low-cost hardware such as the Raspberry Pi as
well as in the cloud.
With over 120,000 modules in Node's package
repository, it is easy to extend the range of palette
nodes to add new capabilities.
The flows created in Node-RED are stored using JSON
which can be easily imported and exported for
sharing with others.
An online flow library allows you to share your best
flows with the world.
PARTIE II : EMULATION
18
Philips Hue Emulation
Philips hue combines brilliant LED light with intuitive technology. Then puts it in the
palm of your hand.
Together, the bulbs, the bridge and the app will change the way you use light.
Forever. Experiment with shades of white, from invigorating blue/white to cozy
yellow/white. Or play with all the colors in the spectrum.
Hue can wake you up. Help protect your home. Relive your favorite memories.
Improve your mood. Even keep you informed about the weather.
The wireless LED light bulbs bring hue to life. They deliver clear and crisp light. All
shades of white. And every color in the spectrum.
They don’t just look great. They’re practical too. They dim. They flash. They pulse.
They do pretty much anything you want. And they screw straight into your current
light fittings. Naturally.
Hue light bulbs are available in E26, E27, GU10, PAR16 and BR30 formats. Or for
something a bit different, try LivingColors Bloom, Iris and LightStrips.
The heart of the hue system. The bridge is literally a bridge between your app and
the bulbs. Linked to wi-fi via your router, it can connect up to 50 bulbs at a time.
Not to mention hue’s other accessories, like hue tap.
The bridge also connects your system to the wider world. With it you can control
your lights remotely or link them up to the rest of the web, newsfeeds or even your
own inbox. It’s really rather smart. http://www2.meethue.com/en-us
19
Philips Hue Emulation - Features
The key to controlling hue lies in the palm of your hand. Use the app to change colors, brightness,
functions and everything else in between. However you want it.
Control your bulbs one at a time or all together. Find the right intensity and shade of white for the task
at hand. Pick that perfect tone to match the moment. Or select a color from a photo and recreate it
in your room. You don’t have to be at home to control your lights. The Myhue portal is an online
control panel that connects you to hue, from anywhere in the world.
Make it look like someone’s home when you’re relaxing on the beach. Or just switch your lights off if
you’ll be home later than you thought. The portal also saves your personalized scenes and Light
Recipes. And it even keeps your bridge software up-to-date. Lighting is about more than just
illuminating the dark. It doesn’t take an expert to notice bad light. And the right light can transform
entire spaces. Even change the way you feel. With hue, you get full control over the light in your life.
Set the mood. Change the ambience. Feel different. This is what hue can do. With tunable white light
and a full spectrum of color, all at the tap of an app, you can set the perfect tone and enhance a
moment. All in an instant. Pick a photo – from the app or your own library - to use as your palette. Then
simply drag the color picker across the image and your bulbs will match the color instantly. Easy.
So now you can relive that glorious sunset. Or bring the calm of the ocean straight into your living
room. Whatever you feel like. Like your creation. Then save it as a ‘scene’ to use again. Really like it?
Then show off a bit and share it with the hue community.
Scenes can be anything from a single bulb’s color, to a whole house bathed in light. And everything in
between. It’s all up to you.
20
Philips Hue Emulation - Emulator
21
Philips Hue Emulation – GET Status Request
22
Philips Hue Emulation – PUT Request of color settings
23
Philips Hue Emulation – PUT Request Response
24
Philips Hue Emulation – Change color results
25
Sensor Emulation
IBM IoT Foundation
A new service just went live which addresses the key challenge of bringing the
world of sensors and controllers with that of big data and analytics. When
combined with the IBM Bluemix platform, IBM Internet of Things Foundation
provides simple, but powerful, and scalable application access to devices and
their data. You can rapidly build applications, visualization dashboards, mobile
apps, and applications that can feed your back-end enterprise with IoT data.
IBM Internet of Things Foundation is a fully managed, cloud-hosted service that
makes it simple to derive value from Internet of Things (IoT) devices. We start with
your device, be it a sensor, a gateway or something else. Using our recipes you
can get it connected and start sending data securely up to the cloud using the
open, lightweight MQTT messaging protocol. From there, you can setup and
manage your devices using your online dashboard or our secure APIs, so that your
apps can access live and historical data fast.
You are now ready to start creating applications using your device data. You
could do this within our IBM Bluemix platform, another cloud or your own servers. If
you aren’t convinced that these things are sending data via our service, then run
the two on separate machines or mobile devices. To do this all you need to do is
enter the device identifier from one to the other. http://www-03.ibm.com/software/products/en/internet-of-things-foundation
26
Sensor Emulation - Features
Connect: Register and connect a wide variety of devices to the service.
Uses industry-standard MQTT, MQ Telemetry (MQTT) protocol (OASIS ratified).
Remotely monitor the connectivity of devices.
Securely connect many popular development boards and devices to the cloud using device recipes.
Extend to new devices with open-source device code from Eclipse Paho.
Collect: Collect and manage a time-series view of data.
See what is happening on your devices with near-real time IoT data visualization.
Store your IoT device events in a time-series database for analysis.
Build dashboards and gain access to historical data through APIs.
Control and secure data currently trapped on instrumented devices.
Assemble: Visually assemble events from the IoT into logic flows.
Use Node-Red in Bluemix for easy drag-and-drop flow assembly.
Analyze data and act on insights in near real time, applying them where they matter most.
Optimize business results with near real-time decision making.
Manage: Pay for what you use with purchase options available through Bluemix—pay through the IBM
Cloud marketplace or as a term subscription.
Hosted on IBM SoftLayer®, 24x7 operation, managed and monitored by IBM.
Powered by proven IBM MessageSight and IBM Informix® technologies.
27
Sensor Emulation – IBM Apps and MAC Adresse
28
Sensor Emulation - Device Validation
29
Sensor Emulation – Node Red Apps
30
Sensor Emulation - Collect Data
31
Sensor Emulation - IBM virtualization
32
Sensor Emulation – IBM Interface Commands
33
Sensor Emulation – Logs for HDFS System
34
MQTT Emulation
If you aren’t convinced that these things are sending data via our service, then run
the two on separate machines or mobile devices. To do this all you need to do is
enter the device identifier from one to the other. With the simulator sending data
all is set to now feed the Node-RED flow in your starter application and the
instructions walk you through this change. MQTT (formerly MQ Telemetry Transport)
is a publish-subscribe based "light weight" messaging protocol for use on top of the
TCP/IP protocol. It is designed for connections with remote locations where a
"small code footprint" is required and/or network bandwidth is limited. The Publish-
Subscribe messaging pattern requires a message broker. The broker is responsible
for distributing messages to interested clients based on the topic of a message.
Andy Stanford-Clark and Arlen Nipper of Cirrus Link Solutions authored the first
version of the protocol in 1999.
The specification does not specify the meaning of "small code foot print" or the
meaning of "limited network bandwidth". Thus, the protocol's availability for use
depends on the context. In 2013 IBM submitted MQTT v3.1 to the OASIS
specification body with a charter that ensured only minor changes to the
specification could be accepted. MQTT-SN is a variation of the main protocol
aimed at embedded devices on non-TCP/IP networks, such as ZigBee.
Historically, the 'MQ' in 'MQTT' came from IBM's MQ message queuing product
https://en.wikipedia.org/wiki/MQTT
35
MQTT Emulation – MQTT Definition
line. However, queuing per se is not required to be supported as a standard feature in all situations.
MQTT stands for MQ Telemetry Transport. It is a publish/subscribe, extremely simple and lightweight
messaging protocol, designed for constrained devices and low-bandwidth, high-latency or unreliable
networks. The design principles are to minimize network bandwidth and device resource requirements
whilst also attempting to ensure reliability and some degree of assurance of delivery. These principles
also turn out to make the protocol ideal of the emerging “machine-to-machine” (M2M) or “Internet of
Things” world of connected devices, and for mobile applications where bandwidth and battery
power are at a premium.
Alternative protocols include the Advanced Message Queuing Protocol, the IETF Constrained
Application Protocol and XMPP.
There are several MQTT brokers available such as IBM Message Sight. They vary in their feature set and
some of them implement additional features on top of the standard MQTT functionality.
In the real world, there are a number of projects that implement MQTT.
Facebook Messenger. Facebook has used aspects of MQTT in Facebook Messenger. However, it is
unclear how much of MQTT is used or for what; Moreover it is to be noted that this is a phone
application not a sensor application.
IECC Scalable Delta Rail's latest version of their IECC Signaling Control System uses MQTT for
communications within the various parts of the system and other components of the signaling system.
It provides the underlying communications framework for a system that is compliant with the CENELEC
standards for safety-critical communications.
36
MQTT Emulation – Java Publisher Example
37
MQTT Emulation – Java Subscriber Example
38
MQTT Emulation – Stats with MQTT SPY
39
MQTT Emulation – Messages Broker
40
MQTT Emulation - HiveMQ Stats
41
GPS Emulation
Creating software for the Internet of Things seems to get easier every day, as new
development tools make popular programming languages work with more and
more connected devices. One of the latest additions is Cylon.JS, an open-source
JavaScript framework designed for controlling robots, embedded sensors and IoT
devices. JavaScript is already hugely familiar to web developers and other
programmers because it runs in just about every web browser and is used to make
websites interactive (think of the chat boxes on social media sites that let you send
and receive instant messages with friends, without reloading the page). It’s also
fairly easy to learn the basics, even for amateurs. Cylon.JS is built on nodeJS, a
JavaScript framework used for many web apps that provides common ground for
IoT applications—such as the ability to simultaneously send, receive and process
many messages from many sources. Named for the science fiction cyborgs of
Battlestar Galactica, Cylon.JS supports a wide variety of drones, robotic toys,
wearables, smart home devices, IoT prototyping and embedded computing
modules, and cloud-based analytics and messaging platforms. It lets programmers
create powerful interactions between users, devices and services with just a few
lines of code. For instance, you can set up a Pebble smartwatch to act as a
controller for “robots” (connected devices) like a Nest thermostat or Phillips Hue
lightbulb. Or you can build your own projects by wiring physical sensors to
development boards like Arduino or Tessel. http://postscapes.com/javascript-for-robots-a-iot-cylonjs
42
GPS Emulation – Cylon.JS Framework
For example, the cylon-api-http plugin offers a RESTful HTTP interface, along with server-sent events
that can be used to subscribe to device events.
Another API plugin cylon-api-socketio supports an interface for clients based on the popular socket.io
module. All API plugins interface directly with the MCP to fetch robots and information about them.
They are also capable of executing commands on Robots and Devices.
A Cylon robot is a collection of devices and connections, with the necessary glue to allow them to
coordinate together. A User will instantiate one or more robots through the MCP, and either start them
all simultaneously or individually. Robots can also have custom commands (to coordinate multiple
devices/connections) and can communicate with each other through the MCP.
Devices and Connections are an abstraction layer over drivers and adaptors, respectively. These
allow for a layer of indirection between Drivers/Adaptors and the Robot itself, and ensure the
interfaces can remain consistent.
Drivers and Adaptors are the meat of Cylon, and allow Cylon to communicate with devices and
services, and issue commands/receive events from them. These are implemented as part of Cylon
submodules (cylon-sphero, etc), and are platform specific.
Adaptors are in charge of connecting to platforms, no matter the medium. For example, the cylon-
sphero Adaptor class communicates over Bluetooth, while the cylon-leapmotion Adaptor talks to a
WebSocket server.
The Adaptor classes ensure that the Drivers are able to directly communicate with the connected
platform.
43
GPS Emulation – nodeJS Apps
44
GPS Emulation- Node Red Flow
45
GPS Emulation – Node Red Data Transfert and Debug
46
GPS Emulation Process
47
GPS Emulation – Maps Representation
48
Trafic Cars Emulation
IBM IoT Foundation
The Connected Car starter kit for Bluemix allows you to easily model real-time
traffic in a city. The kit comes with a nodeJS vehicle simulator, an HTML5
visualization web app, and a control client that all communicate through the IBM
Internet of Things (IoT) Foundation service for real-time messaging.
The diagram to the right provides an overview of how the kit works. Simulated
vehicles are registered with IoT Foundation, and API keys are generated for each
application in the kit.
The authorization tokens allow for messaging between devices, applications, and
services.
The starter kit can be set up and deployed in Bluemix with the following steps: Set
up your IoT Foundation account: register your simulated vehicles and generate an
API key for your HTML5 applications.
Create a placeholder application and IoT/Geospatial services in Bluemix.
Download and configure the starter kit to match your Bluemix and IoT
environment.
Deploy the starter kit to Bluemix using the Cloud Foundry CLI
Use Node-RED (or build your own application) to extend the starter kit
http://m2m.demos.ibm.com/trafficsimulator.html
49
Trafic Cars Emulation – YAML Config
50
Trafic Cars Emulation – Settings Config
51
Trafic Cars Emulation – nodeJS Apps
52
Trafic Cars Emulation – Cloud-foundry CLI Commands
The Internet of Things already connects billions of
devices, with forecasts predicting steep growth rates
in the coming years. Many of these devices, such as
smartphones and connected vehicles, are mobile.
Awareness of the location of on-the-move devices
opens up exciting new application opportunities.
Support for these new applications requires highly
scalable services that can analyze high volumes of
data in real time. With the Geospatial Analytics service
in IBM Bluemix, you can monitor moving devices from
the Internet of Things. The service tracks device
locations in real time with respect to one or more
geographic regions. Geospatial Analytics can be used
as a building block in applications that support several
use cases. For example, a retail business might want to
monitor for potential customers nearby its stores and
send them promotions via push notifications or tweets.
Geospatial Analytics could also be used to detect
connected motorists entering areas that they might
want to avoid due to an accident, weather…
53
Trafic Cars Emulation - IoT IBM Interface
54
Trafic Cars Emulation – Payload Messages
55
Trafic Cars Emulation – Node Red integration
56
Trafic Cars Emulation – Node Red Data Transfert and Debug
57
Trafic Cars Emulation- Process Stats
PARTIE III : DEMONSTRATION
IOT BIG DATA
ANALYTICS
DEVICES PROTOCOLE HARDWARE
ARCHITECTURE « IOT (M2M) - BIG DATA – ANALYTICS »
M2M
IoT (M2M) – Big Data – Analytics : Architecture
The initial contribution of the Eclipse SmartHome project will be a large
subset of the current code of openHAB (open Home Automation Bus). In
particular: Source code for all mentioned interfaces and services of the
framework. Source code of a couple of extensions (openHAB bindings,
actions and persistence services) Documentation of the architecture
and its extension points. Build scripts and sample files.
To enable others to use code from openHAB in their own solutions and
products, we wanted to put the code base on a solid and trustworthy
foundation. The legal aspects of many open-source projects are often
pretty fuzzy, even if they have a permissive license. But for companies
using such project in their own products, there is always the risk that the
copyright of the code is not clear and that the code might be
contaminated by patents. To reduce such risks, it is very beneficial to
have a rigid intellectual property management and clear contribution
processes - this is one of the things that a foundation such as Eclipse
provides. We therefore decided to contribute the core framework of
openHAB to the Eclipse Foundation, which became the new Eclipse
SmartHome project.
60
Software IOT
http://kaikreuzer.blogspot.com/2014/06/openhab-20-and-eclipse-smarthome.html
61
Software IoT - History
The Eclipse SmartHome project is a proposed open source project under the Eclipse Technology Top-
Level Project. This proposal is in the Project Proposal Phase (as defined in the Eclipse Development
Process) and is written to declare its intent and scope. We solicit additional participation and input
from the Eclipse community. Please send all feedback to the Eclipse Proposals Forum.
Since the emergence of broadband internet connections, smartphones and tablets the smart home
market shows a remarkable upsurge. This has led to a very fragmented market, which makes it difficult
for customers to "bet on the right horse". In fact, there is not one system, protocol or standard that
could possibly fulfill all potential requirements. There is hence a need for platforms that allow the
integration of different systems, protocols or standards and that provide a uniform way of user
interaction and higher level services. The goals of the Eclipse SmartHome project can be summarized
as: Provide a flexible framework for smart home and ambient assisted living (AAL) solutions. This
framework focuses on the use cases of this domain, e.g. on easy automation and visualization
aspects. Specify extension points for integration possibilities and higher-level services. Extending and
thus customizing the solution must be as simple as possible and this requires concise and dedicated
interfaces. Provide implementations of extensions for relevant systems, protocols or standards. Many of
them can be useful to many smart home solutions, so this project will provide a set of extensions that
can be included if desired. They can also be in the shape of a general Java library or an OSGi bundle,
so that these implementations can be used independently of the rest of the project as well.
Provide a development environment and tools to foster implementations of extensions. The right
tooling can support the emergence of further extensions and thus stimulates future contributions to
62
Software IoT - Features
the project. Create a packaging and demo setups. Although the focus is on the framework, it needs
to be shown how to package a real solution from it, which can be used as a starting point and for
demo purposes. The Eclipse SmartHome project is a framework that allows building smart home
solutions that have a strong focus on heterogeneous environments, i.e. solutions that deal with the
integration of different protocols or standards. Its purpose is to provide a uniform access to devices
and information and to facilitate different kinds of interactions with them. This framework consists out
of a set of OSGi bundles that can be deployed on an OSGi runtime and which defines OSGi services
as extension points. The stack is meant to be usable on any kind of system that can run an OSGi stack
- be it a multi-core server, a residential gateway or a Raspberry Pi.
Currently planned initial features and extension points of this framework include:
Type System: A basic but extensible type system for smart home data and commands that provides a
common ground for an abstracted data and device access.
Data Registry: Keeps track of data point instances and their values.
Event Mechanism: Passes data and operations asynchronously in a loosely coupled way between
components. The Infrastructure to exchange data and commands with external systems and devices,
e.g. through regular polling or background threads. Useful binding implementations such as HTTP, NTP
or TCP will be part of the project. An implementation of a flexible rule engine that allows changing
rules during runtime. Provides triggering upon events or on a temporal basis.
Extensible through rule actions that are made available to all rules. Simple but powerful textual
representation of rules Declarative User Interfaces.
63
openHAB Framework – Architecture (1)
The openHAB project is split into two parts:
openhab-runtime: This is the package, which you will actually run on your server and which does the
"real" work.
* penhab-designer:
This is a configuration tool for the openhab-runtime. It comes with user-friendly editors to configure
your runtime, to define your UIs and to implement your rules.
* openHAB Runtime:
The openHAB runtime is a set of OSGi bundles deployed on an OSGi framework (Equinox). It is
therefore a pure Java solution and needs a JVM to run. Being based on OSGi, it provides a highly
modular architecture, which even allows adding and removing functionality during runtime without
stopping the service. Here is an overview of the main bundles and how they depend on each other:
Communication
openHAB has two different internal communication channels:
an asynchronous event bus
a stateful repository, which can be queried
The Event Bus
The event bus is THE base service of openHAB. All bundles that do not require stateful behaviour
should use it to inform other bundles about events and to be updated by other bundles on external
events.
There are mainly two types of events:
64
openHAB Framework – Architecture (2)
Commands which trigger an action or a state change of some item/device.
Status updates which inform about a status change of some item/device (often as a response to a
command). All protocol bindings (which provide the link to the real hardware devices) should
communicate via the Event Bus. This makes sure that there is a very low coupling between the
bundles, which facilitates the dynamic nature of openHAB.
As a technical foundation, the OSGi EventAdmin service is used by openHAB. This is a light-weight
pub-sub implementation, which perfectly meets the requirements.
It is important to note that openHAB is not meant to reside on (or near) actual hardware devices
which would then have to remotely communicate with many other distributed openHAB instances.
Instead, openHAB serves as an integration hub between such devices and as a mediator between
different protocols that are spoken between these devices. In a typical installation there will therefore
be usually just one instance of openHAB running on some central server. Nonetheless, as the OSGi
EventAdmin service can also be used as a remote service, it is possible to connect several distributed
openHAB instances via the Event Bus.
Item Repository
Not all functionality can be covered purely by stateless services. Therefore openHAB also offers the
Item Repository, which is connected to the Event Bus and keeps track of the current status of all items.
The Item Repository can be used whenever needed to access the current state of items. E.g. a user
interface needs to display the current state of items in the moment of the user access. Also the
automation logic execution engine needs to always be informed about the current states.
65
openHAB Framework – Architecture (3)
The Item Repository avoids requiring each bundle to cache states themselves for their internal use. It
also makes sure that the state is in sync for all those bundles and it provides the possibility to persist
states to the file system or a database, so that they are even kept throughout a system restart.
The following diagram shows how these communication channels are used:
Sitemap
openHAB comes with a generic textual configuration for its user interfaces: The Sitemap. The Sitemap
is a tree structure of widgets, which define the different pages of a UI and their content. Widgets can
be associated to items, for which they should show the status and/or control elements.
The definition of the Sitemap is quite abstract by design; it is supposed to be a suitable UI model for
different kinds of user interfaces, so that the user does not have to configure each of them in case he
sets up multiple UIs. If a UI has further requirements on top of the sitemap, it is still possible to introduce
additional configuration options which are then specific for the UI in question.
Item UI Providers
Item UI providers offer a dynamic way to configure the UI, so that not everything must be stored
statically in the Sitemap. An item UI provider can for example define, what widget should be used for
an item, if none is specified and can dynamically define icons and labels (which might depend on
the current state of the item) for the widgets.
An important use case for this feature is the dynamic display of item groups - all that there is stored in
the Sitemap is the information that a certain group should be displayed; the page is then dynamically
assembled at runtime with whatever items are available at that time.
66
openHAB Framework – Commands (1)
67
openHAB Framework – Commands (2)
68
openHAB Framework – Commands (3)
69
openHAB Framework – Commands (4)
70
openHAB Framework – Webservices (1)
71
openHAB Framework – Webservices (2)
72
openHAB Framework – Webservices (3)
73
openHAB Framework – Webservices (4)
OpenHAB is aimed to provide a simple drop-in server where components can be connected and
driven from a user interface, whilst allowing a general mechanism to control all components. Although
many of these devices may have their own mobile applications, controlling each one of them
requires opening a mobile application and they cannot be integrated together. Using OpenHAB as
the main hub, the devices can all be controlled with one mobile application or website, and scripts
can be set up to drive individual components.
74
openHAB Designer
75
openHAB Designer – Geent Template Example
The openHAB Designer is an Eclipse RCP application for configuring the openHAB runtime. It comes
with editors for configuration files like the sitemap. Its advantage over simple text editors is the full IDE
support, like syntax checking, auto completion, highlighting and content assist. It is also meant to
implement and deploy rules for automatic actions.
Rather than reinventing the wheel, SiteWhere depends on many
supporting open source technologies to accomplish its goals. Most of
the technologies used have been tested for years in production
environments and have been continuously upgraded thanks to many
iterations of improvements. The following open source components are
used by SiteWhere:
Tomcat provides the core server in which SiteWhere runs. SiteWhere is
deployed as a web archive (WAR file) that runs when the Tomcat server
is started.
Spring delivers the core configuration framework that bootstraps
SiteWhere and allows it to be configured and extended. Rather than
using a hard-coded set of components, SiteWhere has a core set of
service provider interfaces and default implementations of the
interfaces. By using the same interfaces in custom classes and plugging
them in via Spring, a third parties can extend SiteWhere without touching
the core code.
MongoDB: MongoDB is a NoSQL database that can be used to store
SiteWhere data. It is very quick and easy to install, has great
performance, and does not require the computing resources of HBase.
76
Software M2M
http://docs.sitewhere.org/1.0.4/overview.html#what-is-sitewhere
77
Sitewhere - Software M2M (1)
SiteWhere is an open source M2M integration platform – a system that choreographs data coming
from and going to a web of connected devices. It provides the following functionality:
Supplies a server based on proven technologies that acts as a controller for the processing of device
data. The server can be installed on a local machine or run in the cloud and is built on technologies
that are designed to scale to billions of processed device events per day.
Provides long-term persistence of data sent from devices. Historical device event data is valuable and
SiteWhere offers a platform where data is never deleted, no matter the volume of events.
Offers service provider interfaces (SPIs) that provide a core object model for the platform and allow
third parties to extend and customize the system to work with new technologies.
Delivers an advanced device provisioning system that allows control of the full lifecycle of registering
devices, sending commands based on hardware type, receiving data responses and aggregating
them. The system is based on a core set of interfaces that allow new communications protocols and
encoding schemes to be added and configured easily.
Provides a model for standard types of data generated by devices. For instance, location data is
stored such that it can be geospatially indexed and accessed by queries like ‘which device events
happened near this given location in the given time span’. Device measurement data is stored such
that it can be analyzed, faceted, and graphed. Alerts for exceptional conditions can be captured
and indexed so that an audit trail is available after the fact.
Associates devices with external assets such as people or physical items. For instance, a badge may
be associated with the person wearing it. A tracking device can be associated with the piece of
78
Sitewhere - Software M2M (2)
heavy equipment it is attached to. The asset information is offered via an asset management
framework that allows external systems to drive the information. For instance, the list of people that
badges can be assigned to can come from an existing LDAP data store. SiteWhere tracks the
assignment of devices to assets over time and ties events to the assignment so at a later date you can
query ‘what were the locations of the badge when it was assigned to this person’.
Integrates with third party integration frameworks such as Mule ESB, allowing event data to trigger
complex interactions such as adding data to Salesforce or notifying alert conditions using text
messages generated by Twilio.
Provides an HTML5 administrative application that allows all of the system data to be viewed and
manipulated in a way that makes it easy to understand the data. The admin app uses REST services
provided by the core platform to interact with the data. Third party applications can use the same
REST services to interact with SiteWhere data without having to use the admin application.
Offers a Java client that can interact with most of the provided REST services. For external systems with
existing Java code, interfacing with SiteWhere is as simple as a few lines of code.
Protects information by limiting access to data based on a proven user management system. The
system can be configured to use external sources of identity data such as LDAP data stores.
Generates detailed analytics information from gathered device data and provides factilities for
executing advanced faceted queries on the data.
79
Sitewhere - MongoDB DataBase
80
Sitewhere – WebServices (1)
81
Sitewhere – WebServices (2)
Devices must communicate with each other (D2D). Device data then
must be collected and sent to the server infrastructure (D2S). That server
infrastructure has to share device data (S2S), possibly providing it back to
devices, to analysis programs, or to people. From 30,000 feet, the
protocols can be described in this framework as:
• MQTT • XMPP • DDS • AMQP
Each of these protocols is widely adopted. There are at least 10
implementations of each. Confusion is understandable, because the
high-level positioning is similar. In fact, all four claim to be real-time
publish-subscribe IoT protocols that can connect thousands of devices.
And it’s true, depending on how you define “real time,” “things,” and
“devices.” Nonetheless, they are very different indeed! Today’s Internet
supports hundreds of protocols. The IoT will support hundreds more. It’s
important to understand the class of use that each of these important
protocols addresses.
82
Protocoles & Positioning Systems
83
Protocle MQTT
MQTT (MQ Telemetry Transport) is a publish-subscribe based "light weight" messaging protocol for use
on top of the TCP/IP protocol. It is designed for connections with remote locations where a "small
code footprint" is required and/or network bandwidth is limited. The Publish-Subscribe messaging
pattern requires a message broker. The broker is responsible for distributing messages to interested
clients based on the topic of a message. Andy Stanford-Clark and Arlen Nipper of Cirrus Link Solutions
authored the first version of the protocol in 1999. The specification does not specify the meaning of
"small code foot print" or the meaning of "limited network bandwidth". Thus, the protocol's availability
for use depends on the context. In 2013 IBM submitted MQTT v3.1 to the OASIS specification body with
a charter that ensured only minor changes to the specification could be accepted. MQTT-SN is a
variation of the main protocol aimed at embedded devices on non-TCP/IP networks, such as ZigBee.
Historically, the 'MQ' in 'MQTT' came from IBM's MQ message queuing product line. However, queuing
per se is not required to be supported as a standard feature in all situations.
Alternative protocols include the Advanced Message Queuing Protocol, the IETF Constrained
Application Protocol and XMPP.
MQTT Broker: There are several MQTT brokers available such as IBM Message Sight. They vary in their
feature set and some of them implement additional features on top of the standard MQTT
functionality. See MQTT.org for more information. In the real world, there are a number of projects that
implement MQTT. Facebook Messenger. Facebook has used aspects of MQTT in Facebook Messenger.
However, it is unclear how much of MQTT is used or for what; Moreover it is to be noted that this is a
phone application not a sensor application. https://en.wikipedia.org/wiki/MQTT
84
Protocle GSM
GSM (Global System for Mobile Communications, originally Group Special Mobile), is a standard
developed by the European Telecommunications Standards Institute (ETSI) to describe protocols for
second-generation (2G) digital cellular networks used by mobile phones, first deployed in Finland in
July 1992. As of 2014 it has become the default global standard for mobile communications - with over
90% market share, operating in over 219 countries and territories.
2G networks developed as a replacement for first generation (1G) analog cellular networks, and the
GSM standard originally described a digital, circuit-switched network optimized for full duplex voice
telephony. This expanded over time to include data communications, first by circuit-switched
transport, then by packet data transport via GPRS (General Packet Radio Services) and EDGE
(Enhanced Data rates for GSM Evolution or EGPRS).
Subsequently, the 3G++ developed third-generation (3G) UMTS standards followed by fourth-
generation (4G) LTE Advanced standards, which do not form part of the ETSI GSM standard.
"GSM" is a trademark owned by the GSM Association. It may also refer to the (initially) most common
voice codec used, Full Rate. GSM Layer 1: FDMA/TDMA is the air interface(radio), also called Um
interface. For more on this interface refer GSM tutorial in tutorials section. At MS, FDMA/TDMA is used
which is also followed at BTS, BTS takes this format from MS and convert it to 64kbps digital format for
the digital link and interfaces with BSC. BSC communicates with MSC in the same format. GSM Layer 2:
Layer 2 is the data link layer, which does following three main functions.
- Establish, maintain and tear down the link - Flow control - Error detection - Work on the Layer-3 frames
https://en.wikipedia.org/wiki/GSM
85
GPS - Global Positioning System
The Global Positioning System (GPS) is a space-based satellite navigation system that provides
location and time information in all weather conditions, anywhere on or near the earth where there is
an unobstructed line of sight to four or more GPS satellites. The system provides critical capabilities to
military, civil, and commercial users around the world. The United States government created the
system, maintains it, and makes it freely accessible to anyone with a GPS receiver.
The US began the GPS project in 1973 to overcome the limitations of previous navigation systems,
integrating ideas from several predecessors, including a number of classified engineering design
studies from the 1960s. The U.S. Department of Defense (DoD) developed the system, which originally
used 24 satellites. It became fully operational in 1995. Bradford Parkinson, Roger L. Easton, and Ivan A.
Getting are credited with inventing it.
Advances in technology and new demands on the existing system have now led to efforts to
modernize the GPS system and implement the next generation of GPS Block IIIA satellites and Next
Generation Operational Control System (OCX). Announcements from Vice President Al Gore and the
White House in 1998 initiated these changes. In 2000, the U.S. Congress authorized the modernization
effort, GPS III.
In addition to GPS, other systems are in use or under development. The Russian Global Navigation
Satellite System (GLONASS) was developed contemporaneously with GPS, but suffered from
incomplete coverage of the globe until the mid-2000s. There are also the planned European Union
Galileo positioning system, India's Indian Regional Navigation Satellite System, and the Chinese BeiDou
Navigation Satellite System. https://en.wikipedia.org/wiki/Global_Positioning_System
Raspberry PI – Mobile - SIM CARD –
GPS TRACKER – LG SMART TV –
GOOGLE MAPS – LED LIGHTING –
CARS TRACKER – OPENHAB
FRAMEWORK – OSGI MODULE –
SITEWHERE - THERMOMETER
86
Plateforme IoT
87
Application : materials
88
YOUTUBE CHANNEL : INNOVAIOT
Application : Movie Demonstration
INNOVA_IOT
INNOVAIOT
INNOVAIOT
Big data analytics is the process of examining large data sets containing
a variety of data types. Big data, unknown correlations, market trends,
customer preferences and other useful business information. The
analytical findings can lead to more effective marketing, new revenue
opportunities, better customer service, improved operational efficiency,
competitive advantages over rival organizations and other benefits.
To analyze such a large volume of data, big data analytics is typically
performed using specialized software tools and applications for
predictive analytics, data mining, text mining, forecasting and data
optimization. Collectively these processes are separate but highly
integrated functions of high-performance analytics. Using big data tools
and software enables an organization to process extremely large
volumes of data that a business has collected to determine which data
is relevant and can be analyzed to drive better business decisions in the
future.
89
Big Data and Analytics
90
Big Data Analytics – Uses cases
But behavioral analytics can be more than just tracking people. It’s also about understanding the
interactions and dynamics between processes, machines and equipment, even macroeconomic
trends, which may yield new learning into operational risks and opportunities. When combined with
rich insights about consumer behavior, companies gain a more nuanced, detailed and accurate
picture of where markets are headed. And that gives them a huge advantage in directing to
proactive investments to seize the best opportunities. The largest of those opportunities maybe
market-facing or customer-centric, but supply chain improvements or other operational
enhancements first discovered by behavioral analytics may also lead to worthwhile performance
gains. Behavioral analytics has real applications beyond the realm of marketing and customer
intelligence. There is sensor data that can track traffic patterns, for instance, leading to “smarter cities.
It’s possible to know if cold storage chains have been broken or medications have been
compromised during shipments. Such detailed and up-to-the-second views of goods moving across
global supply chains are ultimately another flavor of behavioral analytics. As with big data more
generally, the issue isn’t whether companies have the data they need to do behavioral analytics. It’s
more that they need to determine the optimal strategies to collect, manage and analyze in a
scalable and accessible way the right data to gain better insights into how people – not to mention
processes, equipment and other assets – behave.
It is important to note that big data leaders have long since deployed those strategies and
established strong behavior analytics capabilities. It’s likely not a coincidence while such companies
also tend to be market leaders. http://bigdata.teradata.com/US/Big-Ideas/Behavioral-Analytics
Jaspersoft is a commercial open source software vendor focused on business
intelligence, including data visualization, reporting, and analytics. Jaspersoft
provides commercial and open source software, support, services, and licensing
around the JasperReports, JasperReports Server, Jaspersoft Studio, iReport, and
Jaspersoft ETL products. Jaspersoft's main related product is JasperReports Server,
a Java web application that provides advanced report server capabilities such as
report scheduling, permissions, ad hoc reporting, dashboards, and multi-tenancy.
It is offered in a community open source edition as well as under several
commercial editions with broad support for various databases and data sources,
including NoSQL and other big data datasources.
Jaspersoft was originally called Panscopic, and was founded by Al Campa, CEO,
and Raj Bhargava, VP of Products in 2001. Panscopic raised $23M from Doll
Capital, Discovery Ventures, Morgenthaler Ventures, and Partech. In 2004
Panscopic teamed up with Teodor Danciu, acquired the intellectual property of
JasperReports, and changed the name of the company to Jaspersoft. Brian
Gentile became CEO in 2007. Teodor Danciu began work on JasperReports in
June 2001, the sf.net project was registered in September 2001and JasperReports
0.1.5 was released November 3, 2001. The company released JasperReports
Version 1.0, July 21, 2005. The code was originally licensed under a copyleft
JasperReports License.
91
JasperSoft – Analytics and Reporting
https://en.wikipedia.org/wiki/Jaspersoft
92
JasperSoft – MongoDB Analytics
93
JasperSoft – Data Sources
94
JasperSoft – MongoDB Query Langage
95
Elasticsearch
Elasticsearch is a highly scalable open-source full-text search and analytics
engine. It allows you to store, search, and analyze big volumes of data quickly and
in near real time. It is generally used as the underlying engine/technology that
powers applications that have complex search features and requirements.
Here are a few sample use-cases that Elasticsearch could be used for:
You run an online web store where you allow your customers to search for
products that you sell. In this case, you can use Elasticsearch to store your entire
product catalog and inventory and provide search and autocomplete
suggestions for them. You want to collect log or transaction data and you want to
analyze and mine this data to look for trends, statistics, summarizations, or
anomalies. In this case, you can use Logstash (part of the Elasticsearch, Logstash,
Kibana stack) to collect, aggregate, and parse your data, and then have
Logstash feed this data into Elasticsearch. Once the data is in Elasticsearch, you
can run searches and aggregations to mine any information that is of interest to
you. You run a price alerting platform which allows price-savvy customers to
specify a rule like "I am interested in buying a specific electronic gadget and I
want to be notified if the price of gadget falls below $X from any vendor within the
next month". In this case you can scrape vendor prices, push them into
Elasticsearch and use its reverse-search (Percolator) capability to match price
movements against customer queries. https://www.elastic.co
96
Elasticsearch - Features
Real-Time Data: How long can you wait for insights on your fast-moving data. With Elasticsearch, all
data is immediately made available for search and analytics.
Real-Time Analytics: Combining the speed of search joins with the power of analytics changes your
relationship with your data. Interactively search, discover, and analyze to gain insights that improve
your products or streamline your business.
Distributed: Elasticsearch allows you to start small and scale horizontally as you grow. Simply add more
nodes, and let the cluster automatically take advantage of the extra hardware.
High Availability: Elasticsearch clusters are resilient — they will detect new or failed nodes, and
reorganize and rebalance data automatically, to ensure that your data is safe and accessible.
Multitenancy: A cluster may contain multiple indices that can be queried independently or as a
group. Index aliases allow filtered views of an index, and may be updated transparently to your
application.
Full-Text Search: Elasticsearch builds distributed capabilities on top of Apache Lucene to provide the
most powerful full- text search capabilities available in any open source product. Powerful, developer-
friendly query API supports multilingual search, geolocation, contextual did-you-mean suggestions,
autocomplete, and result snippets.
Document-Oriented: Store complex real world entities in Elasticsearch as structured JSON documents.
All fields are indexed by default, and all the indices can be used in a single query.
Schema-Free, Developer-Friendly, RESTful API, Per-Operation Persistence, Apache 2 Open Source
License, Conflict Management…
97
Elasticsearch – Indexation Example
98
Elasticsearch – MongoDB River Indexation
99
Elasticsearch – MongoDB Replication
100
Elasticsearch – MongoDB Index Status
101
Elasticsearch – MongoDB Collection
102
Elasticsearch –MongoDB Collection Search with index
103
Elasticsearch – index Stats
104
Elasticsearch – Cluster Stats
105
Apache Hadoop
Apache Hadoop is an open-source software framework written in Java for
distributed storage and distributed processing of very large data sets on computer
clusters built from commodity hardware. All the modules in Hadoop are designed
with a fundamental assumption that hardware failures (of individual machines, or
racks of machines) are commonplace and thus should be automatically handled
in software by the framework. The core of Apache Hadoop consists of a storage
part (Hadoop Distributed File System (HDFS)) and a processing part (MapReduce).
Hadoop splits files into large blocks and distributes them amongst the nodes in the
cluster. To process the data, Hadoop MapReduce transfers packaged code for
nodes to process in parallel, based on the data each node needs to process. This
approach takes advantage of data locality nodes manipulating the data that
they have on hand to allow the data to be processed faster and more efficiently
than it would be in a more conventional supercomputer architecture that relies on
a parallel file system where computation and data are connected via high-speed
networking. The base Apache Hadoop framework is composed of the following
modules: Hadoop Common – contains libraries and utilities needed by other
Hadoop modules; Hadoop Distributed File System (HDFS) – a distributed file-system
that stores data on commodity machines, providing very high aggregate
bandwidth across the cluster; Hadoop YARN and Hadoop MapReduce
programming model. https://en.wikipedia.org/wiki/Apache_Hadoop
106
Apache Hadoop – User Interface
107
Apache Hadoop – HDFS Example
The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity
hardware. It has many similarities with existing distributed file systems. However, the differences from
other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be
deployed on low-cost hardware. HDFS provides high throughput access to application data and is
suitable for applications that have large data sets. HDFS relaxes a few POSIX requirements to enable
streaming access to file system data. HDFS was originally built as infrastructure for the Apache Nutch
web search engine project.
108
Apache Hadoop – Log File example for Querying
109
Apache Hadoop – Log Files in HDFS Format
110
Apache Hadoop – Log Files Status in Hadoop
111
Apache Hadoop – Apache Hive combine
The Apache Hive data warehouse software facilitates querying and managing large datasets residing
in distributed storage. Hive provides a mechanism to project structure onto this data and query the
data using a SQL-like language called HiveQL. At the same time this language also allows traditional
map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or
inefficient to express this logic in HiveQL.
112
Apache Hive – Hive Query Langage
DROP TABLE: delete the table and the file of data, if the table already exists.
CREATE EXTERNAL TABLE: create a new external table in Hive. The external tables store only the
definition of table in Hive. The data stay in their location of origin and in their size of origin.
ROW FORMAT: indicate to Hive the mode of formatting of the data. In this case, the fields of every
newspaper are separated by a space.
STORED AS TEXTFILE LOCATION: indicate to Hive the location of the data (the directory example /
data) and specifies that they are stored under the shape of text. The data can be in a single file or
repartees on several files in the directory.
SELECT: select and count all the lines the column of which t10 contains the value ‘host=chaker’.
113
Apache Hive– Request results (1)
114
Apache Hive– Request results (2)
115
Apache Hive– Request results (3)
116
Hadoop and Hive in Big Data
Apache Hive is a data warehouse system build on top of Hadoop to query Big Data. Hive originated
at Facebook and was open sourced in August 2008. The challenge Facebook had to address is one
faced by many companies since then. Eventually data growth in a company challenges the
capabilities of deployed RDBMS or NoSQL systems. Reports and analytics start to take minutes, then
hours, and eventually overlap with other queries and the whole system grinds to a halt. Another
common scenario companies start processing big data with Hadoop discovers the value of making
the data accessible beyond the development team capable of writing complex map-reduce jobs.
The term Big Data is freely used in this context. A colleague defined Big Data jokingly as anything
beyond one million rows – Microsoft Excels row limit. The underlying point is that Big Data is a point of
view and can be generalized as the point where simple solutions and deployed technology fail.
The subsequent question is if scaling and investing heavily in it is the most economical solution.
Commercial large-scale data warehouse solutions are very expensive. Furthermore, some of the data
collected today, e.g. poorly structured or highly denormalize data, can be impractical to manage
with these systems. The Hadoop ecosystem regularly is utilized to scale data processing in a feasible
manner. Hadoop becomes either a replacement or a batch process addition to the existing
infrastructure for data analysis, extraction, loading, transformation, reporting, and machine learning.
The downside, which Facebook encountered, was that data stored in Hadoop is inaccessible to
business users. There are higher-level languages like Pig, Cascading, Crunch or Scalding. They are,
however, geared towards software developers that want to avoid the verbosity of pure Java map-
reduce development.
Green Internet of Things
The term "Internet of Things" emanated to describe a number of technologies and
research disciplines that enable global connectivity over the world-wide physical
objects. Enabling technologies like RFID, sensor networks, biometrics, and
nanotechnologies are now becoming very common, bringing the Internet of Things
into real implementations addressing varying applications, including smart grid, e-
health, intelligent transportation, etc. They foreshadow an exciting future that closely
interconnects our physical world via green networks. Green networks in IoT will
contribute to reduce emissions and pollutions, exploit environmental conservation and
surveillance, and minimize operational costs and power consumption. The Green
Internet of Things (G-IoT) is predicted to introduce significant changes in our daily life
and would help realizing the vision of “green ambient intelligence”. Within a few years
we will be surrounded by a massive amount of sensors, devices and “things”, which
will be able to communicate via IP, act “intelligently”, and provide green support for
users in managing their tasks. These new smart objects will also be context-aware and
able to perform certain functions autonomously, calling for new forms of green
communication between people and things and between things themselves, where
power consumption is optimized and bandwidth utilization is maximized. This
development would not only be relevant to researchers, but also to corporations and
individuals alike. It is hence the aim of our workshop to focus on both design and
implementation aspects in green networks, or networks that can be utilized in
providing green systems through IoT enabling technologies. At the same time, our
workshop call is open for topics relevant to IoT in general.http://research.cs.queensu.ca/G-IoT12
Keywords
IoT – M2M Big Data Analytics
Internet of Things, Machine 2 Machine, MQTT, Broker, Message
Elasticsearch, Apache Hadoop, Apache Hive, MongoDB
JasperSoft Studio, MongoDB, Apache Cassandra, MySQL, JSON
Node-Red, nodeJS, Webservices, Maps, google maps, GPS Tracker, cars trafic, thermostat connecté, adresse MAC, Device registration
Elasticsearch index, MongoDB river, Marvel sense, Elasticsearch HQ
Kibana analytic, Logs analytics
HiveMQ, Apache Tomcat, MQTT Spy, Cylon.JS, Cloud-foundry Clicommands, openHAB, openHABDesigner, Greent Template, Sitewhere
MapReduce, Apache Hive, Hive Query Langage, MongoDB river, HDFS, MongoDB Query Langage, Replication, Collection, elasticsearchindex, cluster
Hadoop UI, Monitoring, Protocol, Data access, Monitor poll, Hybrid mode
Node-Red Flow, API Keys, API Token, Collect Data, Data source, Twitter, Mail, Webservices, Maps, google maps, GPS Tracker, cars trafic, thermostat connecté, adresse MAC, Device registration, MQTT Publisher, MQTT Subscriber
Mqtt Broker, Publisher, Subscriber, GET Request, PUT Request, JSON, YAML
HiveMQ, Apache Tomcat, MQTT Spy, Cylon.JS, Cloud-foundry Clicommands, openHAB, openHABDesigner, Greent Template, Sitewhere
Links
Apache Haddop : https://hadoop.apache.org
IBM Bluemix : https://console.ng.bluemix.net
Elasticsearch : https://www.elastic.co
openHab : http://www.openhab.org
Sitewhere : http://www.sitewhere.org
Jaspersoft : http://www.jaspersoft.com
Node-red : http://nodered.org
MongoDB : https://www.mongodb.org
Apache Tomcat : http://tomcat.apache.org
Apache Hive : https://hive.apache.org
nodeJS : https://nodejs.org
Visit my profile in LinkedIn
Visit my website
http://tn.linkedin.com/in/chakerallaoui
http://allaoui-chaker.github.io
Contact
Idea Create Refine