fiware big data_chatrooms_demo_v1
TRANSCRIPT
Introduction
2
• Big Data– What is it?– How much data is
there?– Data growing forecast
• How to deal with Big Data– What is MapReduce– Hadoop architecture
• Cosmos Big Data– What is cosmos– Cosmos architecture– What can be done
with Cosmos
• Cluster operation– Using the RESTful API– Using the Python CLI
• How to exploit the data– I/O
• Hadoop file systemcommands
• WebHDFS/HttpFSRESTful API
– Querying• Local Hive CLI• Remote Hive client
– MapReduceapplications• Java programming• Oozie
• Cosmos place in FIWARE– General IoT platform– Real-time context data
persistence
• Roadmap
Example-driven explanation (1)
3
Wirecloud(Alice)
Orion Cygnus CosmosWirecloud
(Bob)
subscribe(“mychat”)
subscribe(“mychat”)
update(“mychat”,”hi bob”)
notify(“mychat”,“hi bob”)
notify(“mychat”,“hi bob”)
subscribe(*)
write(“mychat”,”hi bob”)
update(“mychat”,”hi alice”)
notify(“mychat”,”hi alice”)
notify(“mychat”,“hi alice”)
write(“mychat”,”hi alice”)
mychathi bob
mychathi bobhi alice
run(“mychat”)
run(“mychat”)
Example-driven explanation (2)
4
Wirecloud(Charles)
Orion Cygnus Cosmos
subscribe(“mychat”)
read(“mychat”,[“hi bob”,”hi alice”])
mychathi bobhi alice
subscribe(“other”)
Wirecloud(David)
subscribe(“other”)
update(“other”,”hi charles”)
notify(“other”,”hi charles”)
notify(“other”,“hi charles”)
write(“other”,”hi charles”)
otherhi
charles
run(“mychat”)
run(“other”)
update(“mychat”,”hi all”)
notify(“mychat”,“hi all”)
notify(“mychat”,“hi all”)
write(“mychat”,”hi all”)
mychathi bobhi alice
hi all
run(“other”)
Wirecloud
5
• Builds user interfaces based on basicbuilding blocks– Widgets (maps, text boxes, tables…)
– Operators (widgets without graphics)
• The building blocks can be wired
• Able to read from– Orion Context Broker
– Cosmos HDFS
– CKAN
Orion Context Broker
6
• Manages context information– Context information is that of interest for an application
• Implements the OMA NGSI model– Entities having each one several attributes– E.g. entity=mychat, attributes=msg
• Regarding an entity’s attribute, it only stores the lastvalue– mychat.msg=hi bob (t0)– mychat.msg=hi alice (t1)– mychat.msg=hi all (t2)
• Synchonous operation– UpdateContext and QueryContext
• Asynchronous operation– SubscribeContext and NotifyContext
Cygnus connector
7
• Builds the historic regarding Orion entities– chat.msg [(t0,hi bob),(t1,hi alice),(t2,hi all)]
• Subscribes to Orion for the entity’s attributes aimedto be persisted– Orion sends notifications when a subscribed entity’s
attribute changes
• Persistence can be made based on:– HDFS files (Hadoop, Cosmos)– MySQL tables– CKAN resources– Mongo documents (upcoming)
• Regarding HDFS, a RESTful API is used for writing(WebHDFS)
Cosmos Big Data
8
• Provides Big Data storage capabilities– Based on Hadoop Distributed File System, running on top of a
shared cluster– Cluster lifecycle: forever– Data isolation capabilites
• Provides Big Data computing capabilities– Based on Hadoop MapReduce engine, running on top of
private clusters– In the future: Spark clusters, other…– Clusters lifecycle: created, used and destroyed– Access to the shared HDFS
• Hadoop ecosystem– Hive/Shark for querying the data– Oozie for remote scheduling of tasks– WebHDFS/HttpFS for RESTful I/O API
Hands-on: creating the environment (1)
9
• Create an account in Cosmos (log by using your FIWARE credentials)http://cosmos.lab.fi-ware.org/cosmos-gui/
• Create a new fresh VM at FIWARE cloudhttp://cloud.lab.fi-ware.org
• Create the FIWARE yum repo (/etc/yum.repos.d/fiware.repo)[testbed-fi-ware]
name=Fiware Repository
baseurl=http://repositories.testbed.fi-ware.org/repo/rpm/x86_64/
gpgcheck=0
enabled=1
• Create the MongoDB yum repo (/etc/yum.repos.d/mongodb.repo)[mongodb]name=MongoDB Repositorybaseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/gpgcheck=0enabled=1
Hands-on: creating the environment (2)
10
• Install and run MongoDB (orion uses it)$ yum install mongodb-org$ mkdir –p /data/db$ nohuo mongod &
• Install and run Orion Context Broker$ yum install contextBroker$ service contextBroker start
• Install Cygnus connector$ yum install cygnusEdit the configuration files (see next slide)$ service cygnus start
Hands-on: creating the environment (3)
11
• agent_chatrooms.conf (showing relevant conf)chatrooms.sources = http-source
chatrooms.sinks = hdfs-sink
chatrooms.channels = hdfs-channel
chatrooms.sources.http-source.port = 5050
chatrooms.sources.http-source.handler.notification_target = /notify
chatrooms.sources.http-source.handler.default_service = hamburg
chatrooms.sources.http-source.handler.default_service_path = chatrooms
chatrooms.sinks.hdfs-sink.channel = hdfs-channel
chatrooms.sinks.hdfs-sink.cosmos_host = 130.206.80.46
chatrooms.sinks.hdfs-sink.cosmos_port = 14000
chatrooms.sinks.hdfs-sink.cosmos_default_username = frb
chatrooms.sinks.hdfs-sink.cosmos_default_password = xxxxxxxxxxxxx
chatrooms.sinks.hdfs-sink.hdfs_api = httpfs
chatrooms.sinks.hdfs-sink.attr_persistence = column
chatrooms.sinks.hdfs-sink.hive_host = 130.206.80.46
chatrooms.sinks.hdfs-sink.hive_port = 10000
chatrooms.sinks.hdfs-sink.krb5_auth = false
• Cygnus_instance_chatrooms.conf (showing relevant conf)CYGNUS_USER=cygnus
CONFIG_FOLDER=/usr/cygnus/conf
CONFIG_FILE=/usr/cygnus/conf/agent_chatrooms.conf
AGENT_NAME=chatrooms
LOGFILE_NAME=cygnus_chatrooms.log
ADMIN_PORT=8081
Hands-on: creating the environment (4)
12
• Subscribe to Orion on behalf of Cygnus(curl $ORION_HOST:$ORION_PORT/v1/subscribeContext -s -S --header 'X-Auth-Token: '$TOKEN'' --
header 'Content-Type: application/json' --header 'Accept: application/json' -d @-) <<EOF
{
"entities": [
{
"type": "ChatMessageLastOnly",
"isPattern": "true",
"id": ".*"
}
],
"reference": "http://$CYGNUS_HOST:$CYGNUS_PORT/$CYGNUS_TARGET",
"duration": "P5Y",
"notifyConditions": [
{
"type": "ONCHANGE",
"condValues": [
"msg"
]
}
]
}
EOF
Hands-on: using the scripts
13
• Download the scripts from herehttps://github.com/frbattid/fiware-chatrooms-demo
• Content– orion_append_context.sh
• Creates a chatroom entity with initial message
– orion_update_context.sh• Updates the message of an existent chatroom
– orion_query_context.sh• Gets the last message of an existent chatroom
– cosmos_read.sh• Reads from Cosmos the whole conversation regarding a chatroom
• Other– orion_subscribe_context.sh
• Makes a subscription to all the chatrooms
– Idm_get_token.sh• Gets an authentication token, if needed
Further reading
14
• Cosmos@FIWARE catalogue– http://catalogue.fiware.org/enablers/bigdata-
analysis-cosmos
• Cosmos presentation• http://es.slideshare.net/FranciscoRomeroBuen
o/fiwarecosmosv7tech
• Cygnus@github– https://github.com/telefonicaid/fiware-
connectors