cisco connect toronto 2017 - model-driven telemetry

© 2016 Cisco and/or its affiliates. All rights reserved. 1

CiscoConnect

Model-Driven TelemetryViktor OsipchukTechnical Marketing Engineer

12 October 2017


Be sure to make right measurements before implementing automation @scale

What Automation Without Visibility Looks Like


Source: Google @ Bay Area OpenDaylight Meetup 06/16


Telemetry as the basis for networks @scale

https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45687.pdf

5© 2016 Cisco and/or its affiliates. All rights reserved.

Current challenges


syslog

SNMP

CLI

Too Slow

Incomplete

Network-Specific

Hard to Operationalize

Why Network Visibility Is Hard


What Happens When You Push SNMP Too Hard

• 10 second poll / push• 3 pollers / telemetry receivers• 30 minute measurement intervals

• 288 100Gig E Interfaces (Line Rate)• SNMP: IF-MIB (query by row)


Telemetry fundamentals


Three Enablers for Telemetry

Push Not Pull

Analytics-Ready Data

Data-Model Driven


Push Beats SNMP Pull for Speed & Scale

• 10 second poll / push• 3 pollers / telemetry receivers• 30 minute measurement intervals

• 288 100Gig E Interfaces (Line Rate)• SNMP: IF-MIB (query by row)


“OSI model” of Telemetry

Data store

Data model

Producer

Exporter

Collector

Native (raw) data inside a router’s database

Raw data mapped to a model (YANG native, OpenConfig, etc)

Sending requested data in model format to the “Exporter” at defined intervals

Encoding and delivering data to the collector(s) destination(s)

Information collection for processing (e.g., data monitoring, automation, analytics)

“Data”“Layer”Te

lem

etry

end

-to-e

nd


YANG Is A Modeling Language

module ietf-interfaces {import ietf-yang-types {

prefix yang; }container interfaces {

list interface { key "name"; leaf name {

type string;}leaf enabled {

type boolean; default "true";

}…

Edited for

Brevity

Self-contained top-level hierarchy of nodes

Import or define data types

Leaf nodes for simple data

Lists for sequence of entries

Containers group related nodes

Other YANG Features

• RO or RW • Optional nodes• Choice• Augment

• When• Arbitrary XML• RPC• etc


Models are Available in Github

You Should Do This

• Telemetry only cares about operational (*-oper.yang) models.

• 143 oper YANG models published for XR 6.1.1• 151 oper YANG are for XR 6.1.2• 177 oper YANG are for XR 6.2.1• 180 oper YANG for XR 6.2.2• 198 oper YANG for XR 6.3.1https://github.com/YangModels/yang/tree/master/vendor/cisco/xr

https://github.com/YangModels/yang/tree/master/vendor/cisco/xr



Data store

Data model

Producer

Exporter

Collector





Information collection for processing (e.d., data monitoring, automation, analytics)


lem

etry

end

-to-e

nd


Finding the Data You Want To Stream

$ pyang -f tree Cisco-IOS-XR-infra-statsd-oper.yang--tree-path infra-statistics/interfaces/interface/latest/generic-counters

telemetry model-driven

sensor-group SGROUP1

sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-

statistics/interfaces/interface/latest/generic-counters


What Will Be Pushed With That Config

{"Timestamp": 1480547974706,"Keys": {

"interface-name": "MgmtEth0/RP0/CPU0/0"},"Content": {

"applique": 0,"availability-flag": 0,"broadcast-packets-received": 25035,"broadcast-packets-sent": 0,"bytes-received": 165321050,"bytes-sent": 233917498,"carrier-transitions": 3,"crc-errors": 0,"framing-errors-received": 0,"giant-packets-received": 0,"input-aborts": 0,"input-drops": 62,"input-errors": 0,"input-ignored-packets": 0,"input-overruns": 0,"input-queue-drops": 0,

"last-data-time": 1480547974,"last-discontinuity-time": 1479244159,"multicast-packets-received": 457,"multicast-packets-sent": 0,"output-buffer-failures": 0,"output-buffers-swapped-out": 0,"output-drops": 0,"output-errors": 104,"output-queue-drops": 0,"output-underruns": 0,"packets-received": 373156,"packets-sent": 311583,"parity-packets-received": 0,"resets": 0,"runt-packets-received": 0,"seconds-since-last-clear-counters": 0,"seconds-since-packet-received": 0,"seconds-since-packet-sent": 0,"throttled-packets-received": 0,"unknown-protocol-packets-received": 0

}

Repeated for all interfaces



Data store

Data model

Producer

Exporter

Collector







lem

etry

end

-to-e

nd


Configuring Destination


destination-group DGROUP

address family ipv4 192.168.1.1 port 2104

---- and/or ----

address family ipv6 2001:db8::1 port 2104

encoding self-describing-gpb

protocol tcp

Specify where you want to send your data

Specify how you want your data to look like

Specify how you want your data to be delivered


Basic Concept: Encoding

Encoding (or “serialization”) translates data (objects, state) into a format that can be transmitted across the network. When the receiver decodes (“de-serializes”) the data, it has an semantically identical copy of the original data.

DATA

DATA

“Decode”

“Encode”

IOS XR platforms Encodings

• Compact GPB• Key-Value GPB• JSON (6.3.1)


GPB Encoding

Design Goals

• Simplicity• Performance• Forward/Backward

Compatibility

Non-Goals

• Human-Readable• Self-Describing• Text-based

Google Protocol Buffers (GPB)

Call them “protobufs”

for short

“Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.”


Telemetry Has Two GPB Encoding Options

data_gpb {row {

timestamp: 1485794640469keys: "\n\026GigabitEthernet0/0/0/0"content:

"\220\003\010\230\003\001\240\003\002\250\003\000\260\003\000\270\003\000\300\003\000\310\003\000\320\003\300\204=\330\003\000\340\003\000\350\003\000\360\003\377\001"}

2X fasterOperationally more complex (but not relative to SNMP!)

data_gpbkv {timestamp: 1485793813389fields {name: "keys"fields { name: "interface-name" string_value:

"GigabitEthernet0/0/0/0" }}fields {name: "content"fields { name: "input-data-rate" uint64_value: 8 }fields { name: "input-packet-rate" uint64_value: 1 }

<<< 9 lines are skipped >>> fields { name: "input-load" uint32_value: 0 }fields { name: "reliability" uint32_value: 255 }

}}...

3X largerNative models: still need heuristics for key names

GPB – “compact” GPB – “self-describing”


Dial-Out

• TCP & gRPC (from 6.1.1)• UDP (from 6.2.1)

Dial-In

• gRPC only (from 6.1.1)

Transport Options

Collector

Data

SYNSYN-ACK

ACK Collector

Data

SYNSYN-ACK

ACK


gRPC: Like REST But Different

Runs over HTTP/2Optimize for page load timeServer push, header compression, multiplexing, TLSRFC 7540 (May 2015)Preserves most HTTP1.1 syntax

Defines Services (“RPCs”) Encodes Using Google Protocol Buffers (“GPB” or “protobufs”)

Services and MessagesAuto-generate code in many languages

http://www.grpc.io/docs/#hello-grpc

http://www.grpc.io/docs/#hello-grpc



Data store

Data model

Producer

Exporter

Collector







lem

etry

end

-to-e

nd


A Telemetry Subscription


subscription Sub1

sensor-group-id SGROUP1 sample-interval 30000

destination-id DGROUP1

*Omit Destination Group For gRPC Dial-In


Telemetry on Cisco products


Cisco XR Telemetry overview

Classic XR ASR9k Evolved XR ASR9k NCS5500 NCS6kMDT support 6.1.1 6.1.1 6.1.1 6.1.3

Data modelsYANG

(native, OC, IETF) Link for models

YANG (native, OC, IETF)

Link for models


Link for models


Link for models

Transport(Control

protocols)

TCP (dial-out), UDP (dial-out)*

gRPC (dial-in, dial-out), TCP (dial-out), UDP (dial-out)*

gRPC (dial-in, dial-out), TCP (dial-out), UDP

(dial-out)*

TCP (dial-out), UDP (dial-out)*

Encoding GPB / GPB-KV / JSON** GPB / GPB-KV / JSON** GPB /

GPB-KV / JSON**GPB /

GPB-KV / JSON**

Collectors Pipeline*** Pipeline*** Pipeline*** Pipeline***

* UDP support from 6.2.1** JSON support from 6.3.1*** Open-sourced and ready to use: https://github.com/cisco/bigmuddy-network-telemetry-pipeline





https://github.com/cisco/bigmuddy-network-telemetry-pipeline


NX OS IOS-XE

MDT support 7.0(3)I6(1) 16.6.1*

Data models Data Management Engine, NX-API, YANG (native, OC, IETF)

YANG (native**, IETF) Link for models

Transport(Control protocols)

gRPC* (dial-out), UDP** (dial-out), HTTP***(dial-out)

Netconf (for YANG), GNMI(16.8.1), gRPC (16.9.1)

Formats GPB/JSON XML, GPB (16.9.1)

Collectors Pipeline TBD

Min sample interval 5 sec 1 sec

Max # of dial-out destinations 5 TBD

Cisco NXOS/XE Telemetry high-level overview

* gRPC supports GPB only * supported on Catalyst 3650/3850/9300/9500, ASR1000, ISR4000** UDP from 7.0(3)I7(1), supports both, GPB and JSON ** Native models are different from YANG models in XR*** HTTP suppors JSON only

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/programmability/guide/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x_chapter_011000.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/prog/configuration/166/b_166_programmability_cg/model_driven_telemetry.html

https://github.com/YangModels/yang/tree/master/vendor/cisco/xe

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/programmability/guide/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x_chapter_011000.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/prog/configuration/166/b_166_programmability_cg/model_driven_telemetry.html


Event-Driven Telemetry overview

Model-Driven Telemetry

100 interfaces UP / 0 interfaces DOWN






Event-Driven Telemetry

Router X



Router X

Tim

e

Tim

e


Event-Driven Telemetry configurationA configuration example:

telemetry model-drivensensor-group interface

sensor-path Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface!subscription interface

sensor-group-id interface sample-interval 0 ## ”0” means EDT!

Please, go ahead

And tell us what you want to have!

IOS XR 6.3.1 includes support for:

1. Interface changes• Sensor path: Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interface• Sensor path: Cisco-IOS-XR-ipv6-ma-oper:ipv6-network/nodes/node/interface-data/vrfs/vrf/global-briefs/global-brief (*)

2. RIB changes• Sensor path: Cisco-IOS-XR-ip-rib-ipv4-oper:rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-name/routes/route• Sensor path: Cisco-IOS-XR-ip-rib-ipv6-oper:ipv6-rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-

name/routes/route3. Syslog updates

• Sensor path: Cisco-IOS-XR-infra-syslog-oper:syslog/messages/message

More to come soon !


Some Commonly Used Models and SubtreesData Model

Interface Oper State sensor-path Cisco-IOS-XR-pfi-im-cmd-oper:interfaces/interface-xr/interfaceInterface Data Rate sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/data-rateInterfaces Stats sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-statistics/interfaces/interface/latest/generic-countersOptics Ports Info sensor-path Cisco-IOS-XR-controller-optics-oper:optics-oper/optics-ports/optics-port/optics-InfoUptime Info sensor-path Cisco-IOS-XR-shellutil-oper:system-time/uptimeCPU State sensor-path Cisco-IOS-XR-wdsysmon-fd-oper:system-monitoring/cpu-utilizationMemory Info sensor-path Cisco-IOS-XR-nto-misc-oper:memory-summary/nodes/node/summaryProcesses Memory sensor-path Cisco-IOS-XR-procmem-oper:processes-memory/nodesLLDP Info sensor-path Cisco-IOS-XR-ethernet-lldp-oper:lldp/nodes/node/neighbors/summaries/summary

IPv4 RIB Infosensor-path Cisco-IOS-XR-ip-rib-ipv4-oper:rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-name/routes/route

IPv6 RIB Infosensor-path Cisco-IOS-XR-ip-rib-ipv6-oper:ipv6-rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-name/routes/route

BGP IPv4 Routes Infosensor-path Cisco-IOS-XR-ip-rib-ipv4-oper:rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-name/protocol/bgp/as/information

BGP IPv6 Routes Infosensor-path Cisco-IOS-XR-ip-rib-ipv6-oper:ipv6-rib/vrfs/vrf/afs/af/safs/saf/ip-rib-route-table-names/ip-rib-route-table-name/protocol/bgp/as/information

BGP ipv4 Neighbor Info sensor-path Cisco-IOS-XR-ipv4-bgp-oper:bgp/instances/instance/instance-active/default-vrf/neighbors/neighborMPLS-TE Tunnels Summary Info sensor-path Cisco-IOS-XR-mpls-te-oper:mpls-te/tunnels/summaryRSVP Interface Info sensor-path Cisco-IOS-XR-ip-rsvp-oper:rsvp/interface-briefs/interface-briefNCS5500 NPU Stats sensor-path Cisco-IOS-XR-fretta-bcm-dpa-npu-stats-oper:dpa/stats/nodes/nodeNCS5500 NPU Resources Info sensor-path Cisco-IOS-XR-fretta-bcm-dpa-hw-resources-oper:dpa/stats/nodes/node/hw-resources-datas/hw-resources-data


Your infrastructure should be readyData NCS5516

numbersgRPC/KVGPBbytes*

gRPC/KVGPBBW (mbps)

gRPC/GPBbytes*

gRPC/GPBBW (mbps)

UDP/JSONbytes*

UDP/JSONBW (mbps)

Interface Oper State* 2160 5105157 0.6806876 1089581 0.145277467 5774492 0.769932267Interface Data Rate 2160 795737 0.3182948 188765 0.075506 1003445 0.401378

Interfaces Stats 2160 2144454 0.8577816 377912 0.1511648 2520016 1.0080064Optics Ports Info 576 5369650 0.477302222 740693 0.065839378 5438332 0.483407289

Uptime Info 1 176 0.0002816 143 0.0002288 373 0.0005968CPU State 18 419895 0.335916 66315 0.053052 380407 0.3043256Memory Info 18 5045 0.004036 1448 0.0011584 6984 0.0055872Processes Memory 538 691910 0.061503111 222950 0.019817778 937089 0.0832968LLDP Info 574 231640 0.185312 87911 0.0703288 300228 0.2401824IPv4 RIB Info* 650,022 1065910551 71.0607034 246565671 16.4377114 1279423829 85.29492193IPv6 RIB Info* 14876 27311283 3.6415044 6427000 0.856933333 32784283 4.371237733BGP IPv4 Routes Info 650,000 593 0.000158333 303 0.0080466667 790 0.0002188BGP IPv6 Routes Info 12800 590 0.000157333 300 0.00008 784 0.000209067BGP ipv4 Neighbor Info 2 48784 0.0390272 6841 0.0054728 48904 0.0391232MPLS-TE Tunnels Summary Info 1003 5755 0.004604 704 0.0005632 5865 0.004692RSVP Interface Info 5 1919 0.0015352 532 0.0004256 2465 0.001972NCS5500 NPU Stats 96 119816982 1.59755976 19158017 0.255440227 129493262 1.726576827NCS5500 NPU Resources Info 96 256502 0.2052016 47352 0.0378816 266352 0.2130816

TOTAL Bandwidth ~ 80 mbps ~ 18 mbps ~ 95 mbps

Bytes are per a single collection* Go with EDT starting from IOS XR 6.3.x


Collectors and analytics platforms


Kafka

Different Collection Models

Logstash

ElasticSearch

KibanaPanda

BYO

Custom Open Source, Customizable

Proprietaryor OS-based

Commercial Stack

Prometheus /InfluxDB

Grafana

Applications

Storage

Collection


Pipeline: An Open Source Collector

Pipeline

Kapacitor

Output to file, TSBD, Kafka…Ingest, transform, filterSelf-monitoring, horizontally scalable


TSDB basic comparisonData InfluxDB Prometheus Elasticsearch

Website https://influxdata.com/ https://prometheus.io/ https://www.elastic.co/products/elasticsearchCategory Real-time Analytics Monitoring System Real-time SearchSupported Measurements metrics, events metrics metrics, events

Sharding and Replication Manual Manual (supports federation) AutomaticHigh Availability (HA) Double writing 2 servers Double writing 2 servers ClusteringUnderlying Technology Golang Golang JavaOperational Complexity Low (medium with HA) Low MediumStorage Backend Custom Custom DocumentSupported Data Types int64, float64, bool, and string float64 string, int32, int64, float32, float64, bool, nullBytes per point after compression 2.2 1.3 22Metric Precision nano second milli second milli secondRecording type events fixed interval eventsWrite Performance - Single Node 470k metrics / sec (custom HW) 800k metrics / sec 30k metrics / secWrite Performance - 5 Node Cluster - - 120k metrics / sec (calculated)Query Performance (1 host, 12hr by 1m) 3.78 ms (min), 8.17 (avg) tbd 13.23 ms (min), 28.6 (avg)Query Performance Medium to Fast Moderate ModerateQuery Language InfluxQL (SQL like) PromQL Query DSLData Model metric names, fields, labels metric names, labels metric names, labelsQuery Language Functionality 4/5 5/5 3/5Query Language Usability 5/5 4/5 3/5Community Size large large largeLicense MIT Apache 2 Apache 2Latest Version v1.3.5 v2.0.0-beta.2 v5Maturity Stable Stable Stable

Full table: https://tinyurl.com/jsd4esyGood to read: https://tinyurl.com/ybaw4ww6

DB ranking: https://tinyurl.com/ya8rrrjpInfluxDB vs Elasticsearch: https://tinyurl.com/y7yxjf6v

InfluxDB vs OpenTSDB: https://tinyurl.com/y8ofbjyyInfluxDB vs Cassandra: https://tinyurl.com/y83vv9ys

https://influxdata.com/

https://prometheus.io/

https://www.elastic.co/products/elasticsearch

https://tinyurl.com/jsd4esy

https://tinyurl.com/ybaw4ww6

https://tinyurl.com/ya8rrrjp

https://tinyurl.com/y7yxjf6v

https://tinyurl.com/y8ofbjyy

https://tinyurl.com/y83vv9ys


TSDB: systems requirements

Pipeline

Grafana uses TSDB for the work, server requirements are small (250MB of RAM, 1CPU – the answer from the developer: https://tinyurl.com/ybdhexqu)

Prometheus doesn’t give exact requirements, but in many discussions shows 64GB of RAM and and 32 CPU: https://tinyurl.com/yd7l38ed)

Pipeline is a lightweight program, written in Go (with concurrency). You can also start several instances [on a single server] if needed.

InfluxDB requirements:https://tinyurl.com/yd9f3ntj

Server CPU,cores

RAM, GB IOPS

Low load [5k writes, 100k series, 5 queries per second] 2-4 2-4 500

Moderate load [250k writes, 1m series, 25 queries per second] 4-6 8-32 500-1000

High load [>250k writes, >1m series, >25 queries per second] 8+ 32+ 1000+

https://tinyurl.com/ybdhexqu)

https://tinyurl.com/yd7l38ed

https://tinyurl.com/yd9f3ntj


Real life deployments

https://promcon.io/2017-munich/slides/monitoring-cloudflares-planet-scale-edge-network-with-prometheus.pdf


MDT in real time

Demo


ASR 9006

Demo topology

BE 14 BE 13

BE 26BE 56

BE 15BE 46

BE 36 BE 12

BE 16

NCS 5502

NCS 5011 NCS 5501

NCS 5501 NCS 5502

Pipeline

Kapacitor

100GE link10GE link

Telemetry stream


Key Takeaways

• Speed & Scale Require Visibility• It’s Not Hard to Beat SNMP• Data Models Are Your Friends• A Big Data Platform Is In Your Future


ResourcesTutorials, Blogs, VoDs

https://xrdocs.github.io/telemetry/http://blogs.cisco.com/sp/the-limits-of-snmphttp://blogs.cisco.com/sp/boring-is-the-new-awesomehttp://blogs.cisco.com/sp/why-you-should-care-about-model-driven-telemetryhttps://youtu.be/tIN8BjHwpNs (NANOG 67: 10 Lessons from Telemetry)

YANGhttps://github.com/YangModels/yang/tree/master/vendor/cisco (Cisco YANG models)http://blogs.cisco.com/getyourbuildon/yang-opensource-tools-for-data-modeling-driven-management(YANG open source tools)https://developer.cisco.com/site/ydk/ (YDK intro)

Telemetry Tools :https://github.com/cisco/bigmuddy-network-telemetry-pipelinehttps://github.com/cisco/bigmuddy-network-telemetry-stacks

Demos and Labhttps://dcloud-cms.cisco.com/demo/mdt-ios-xr-611-v1 (dCloud)https://youtu.be/F_S9-ctNFe0 (demo on NCS 5508)

http://www.cisco.com/c/en/us/td/docs/iosxr/Telemetry/Telemetry-Config-Guide.html

http://blogs.cisco.com/sp/the-limits-of-snmp

http://blogs.cisco.com/sp/boring-is-the-new-awesome

http://blogs.cisco.com/sp/why-you-should-care-about-model-driven-telemetry

https://youtu.be/tIN8BjHwpNs

https://github.com/YangModels/yang/tree/master/vendor/cisco

http://blogs.cisco.com/getyourbuildon/yang-opensource-tools-for-data-modeling-driven-management

https://developer.cisco.com/site/ydk/

https://github.com/cisco/bigmuddy-network-telemetry-collector

https://github.com/cisco/bigmuddy-network-telemetry-stacks

https://dcloud-cms.cisco.com/demo/mdt-ios-xr-611-v1

https://youtu.be/F_S9-ctNFe0

Thank you.