Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Solving real-time IoT challenges:
Couchbase Connect 2016Mohanraj Umapathy,
Verizon Labs, Director – IoT Platform
Couchbase at Verizon Labs
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Agenda
• IoT overview
• Verizon solutions for IoT/ThingSpace
• Verizon Labs real-time data pipeline
• Benefits of Couchbase
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
A new definition of IoT
3
IoT is transforming businesses through an aware and autonomous network
that provides actionable insight and better business intelligence.
AwareThe asset must be able
to sense something.
AutonomousThe asset must send data
somewhere automatically.
ActionableThe data must
inform decisions.
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
4
1. IDC, Worldwide Internet of Things Spending by Vertical Market 2015–2018 Forecast, IDC #256255, June 2015.
Market adoption
Growing faster in some
industries than others
• Manufacturing, utilities and
transportation
are leading.
• Retail, healthcare and
consumer are catching up
fast.1
IoT worldwide
installed base
• 10.3 billion in 2014
• 29.5 billion in 20201
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or
distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement. 5
How are you going to make sure your
IoT implementation succeeds?
Put it together correctly • Successful IoT applications require
integrated system design, security
and management.
Keep it running smoothly • A complete solution involves ongoing
development, deployment, integration
and device management.
Harvest the reward efficiently • To extract business value, you need to
cost-effectively store, analyze and make
sense of collected data.
Overcoming the challenges
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
ThingSpace IoT Ecosystem
6
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
ThingSpace IoT
ThingSpace platformVerizon network
End-to-end device integration and management
ApplicationsDevices
Connectivity
management
Device
management
Reporting and
analytics
Security and
compliance
Ze
ro-t
ou
ch
on
bo
ard
ing A
PIs
an
d d
eve
lop
er
po
rtal
Wired access
LTE/5G
Traditional and
IoT cores
7
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Real-Time Intelligent Data Pipeline
Data Ingestion Insights
Device
Sensor
Cloud
Streaming Data
Event Processing Pipeline
Visualization
AI Devices
Map
Analytics
Rules
Action
Storage
Machine Learning
Transformation
Filter
Relationship
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Verizon Labs IoT/ThingSpace & Couchbase
Problem
Solved
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Breakoutwith Mohan Umapathy
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
• Introduction
• Why Couchbase?
• Solving IoT challenges in active-active data centers
• Lesson learned with Couchbase in production
• POC: Couchbase, Mesos, Docker and Spark – 3M ops/sec
Agenda
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Introduction
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Mohanraj UmapathyVerizon Labs, Director – IoT Platform
Mohan leads the Internet of Things (IoT) Platform
team that develops highly available and scalable
IoT platform services at Verizon Labs.
Prior to joining Verizon Labs, Mohan was Director
at Qualcomm leading the IoT Platform effort.
Mohan spent most of his career developing fault-
tolerant distributed messaging systems and rule
engines for IoT use cases.
Mohan enjoys playing chess and sports that his
kids sign up for.
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or
distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement. 14
• Mohanraj Umapathy
• Yifeng Chen
• Abhishek Jamloki
• Steven Le
Team Credits
• To be updated by
Couchbase
• Alex Ma
• Couchbase support team
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
ThingSpace IoT ecosystem
ThingSpace platformVerizon network
End-to-end device integration and management
ApplicationsDevices
Connectivity
management
Device
management
Reporting and
analytics
Security and
compliance
Ze
ro-t
ou
ch
on
bo
ard
ing A
PIs
an
d d
eve
lop
er
po
rtal
Wired access
LTE/5G
Traditional and
IoT cores
15
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Why Couchbase?
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Verizon Labs IoT/ThingSpace & Couchbase
Problem
Solved
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Couchbase hardware & software
Type Specification
CPU E5-2697v3
Memory 256GB
Disk 2TB SSD (RAID 1)
OS Centos 6.7
Couchbase Server 3.1.1
JVM Oracle JRE 1.7u 64-bit
18
Specs
• Do not tune if Couchbase meets your
performance requirements
• Consult your OS and Couchbase
professional for tuning parameters
• Benchmark, benchmark and
benchmark!
Rules
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Couchbase performance
19
Number
of Nodes Size
Stable
Rate (s)
Stable CAS
Rate (s)
Max Rate
(s)
Average
CPU IOPS Rate
3 2.5 KB 150,000 47,000 250,000 30% 7000
Operation Events Events > 100ms
Read 1 billion 7012 (0.0007%)
Write 200 million 2620 (0.0013%)
Total 1,200,000,000 9622
Total Doc Memory Resident <5ms
100M 100% 99.5%
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Solving device report challenges
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Use case requirement
Complex and dynamic IoT device report
• Multi-tenancy and grouping of IoT devices
• Arbitrary search and sort on more than 35 device fields
Scaling to support devices from several thousand enterprise customers
Reduce time to market (TTM) for new device features
Always on, Active-Active DCs and 99.999%
21
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Our Challenges
Exponential growth of IoT devices in our system
• Device report from portal has unpredictable response time
• Query from RDBMS does not meet our latency requirement
Difficult to maintain legacy stored procedure
• >5000+ lines
• Join queries
RDBMS high replication latency
Time to market (TTM) and capacity projection
22
High Latency
Legacy
Exponential growth
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or
distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement. 23
Solution Overview
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Datagrid
Datagrid cluster
• Distributed, memory-first and modular data-centric system
• Built on top of Couchbase, ElasticSearch and etc
• Near-real time or real time data analytics depending on queries.
Datagrid client
• CAS and query data from Datagrid clusters
Sync Service
• Incremental data synchronization between DBMS and Datagrid cluster
24
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Datagrid
25
Product Version
Couchbase (CB) 3.1.1
ElasticSearch (ES) 1.7.1
CB-ES Plugin 2.1.1
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Lessons learned in production
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Minimize human touchpoints
• DevOps Automation Framework
• Vagrant, Docker and Ansible/Puppet
• End-to-end cluster installation and validation via REST API
27
DevOps automation
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Alert and monitoring
28
Basic metric
per node
Cluster level Node level Bucket level
Host CPU
Process CPU
Disk I/O
Network I/O
Node failures
File system
Size (number of nodes)
Active tasks
Number of connections
Health (up/down)
CPU usage (sys/usr/iowait)
RAM usage
Major page faults
Disk space usage
Swap usage (should be
always 0)
Disk IOPS
Disk IO latency
Get ops rate
CAS ops rate
Set ops rate
Ram quota usage
Active document resident ratio
XDCR level (2 XDCR queues X2 DCs):
• Outgoing queue size
• Outgoing replication rate
• Incoming replication rate
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Understand your workflow
Watch and tune your clusters to suit your workflow:
Compaction
Add/Remove Nodes
Security Updates
29
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Couchbase, Mesos, Docker, Spark and AlluxioProof of Concepts
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Real-Time Intelligent Data Pipeline
Data Ingestion Insights
Device
Sensor
Cloud
Streaming Data
Event Processing Pipeline
Visualization
AI Devices
Map
Analytics
Rules
Action
Storage
Machine Learning
Transformation
Filter
Relationship
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Basic streaming dataflow
32
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Data pipeline demo
Processing 500,000 accounting events/sec to find aggregated IoT device usages.
• POC using Datagrid, built on top of Couchbase (14 nodes), Spark (60 nodes),
Kafka (6 nodes), Mesos, Docker
• Couchbase cluster sustains 3.5 millions operations/sec
• Video: https://www.youtube.com/watch?v=7k4qypwYiRs
33
Type Rate/s Nodes CPU per node Memory per node
Device Traffic
Simulator
500,000 5 8 8GB
Spark 500,000 60 4 16GB
Couchbase 3,500,000 14 20 128GB
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Performance improvement
Achieved 50% higher processing rate with Spark on Alluxio, an open source memory
distributed storage
34
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Performance improvement
35
Achieved 50% higher
processing rate with Spark on
Alluxio, an open source
memory distributed storage
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Our next steps
Evaluating Couchbase 4.5.x
N1QL
Memory-optimized indexes
Integrated full-text search
Partial updates and reads
Extra use cases with Couchbase
36
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
We’re hiring
37
Verizon Website: http://www.verizon.com/about/careers
My email: [email protected]
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement.
Thank you.
38