model driven telemetry

© 2016 Cisco and/or its affiliates. All rights reserved. 1

Model-Driven Telemetry#ConnectCAJimmy FanizziSystems Engineer – Global Service Provider

November 28, 2017

ConnectCisco


Look Twice Before You Leap.

-- Charlotte Bronte

What Automation Without Visibility Looks Like


Source: Google @ Bay Area OpenDaylight Meetup 06/16

5© 2016 Cisco and/or its affiliates. All rights reserved.

Current challenges


syslogSNMP

CLI

Too Slow

Incomplete

Network-Specific

Hard to Operationalize

Why Network Visibility Is Hard


SNMP Polling Is Hard on Everybody

Request-ID 1: Sent, No Response

Request-ID 2: Sent, No Response

ManagersNetworkRouters


What Happens When You Push SNMP Too Hard

• 10 second poll / push• 3 pollers / telemetry receivers• 30 minute measurement intervals

• 288 100Gig E Interfaces (Line Rate)• SNMP: IF-MIB (query by row)


Telemetry fundamentals


Three Enablers for Telemetry

Push Not Pull

Analytics-Ready Data

Data-Model Driven


Push Means Don’t Wait To Be Asked

T1, interface statsT2, interface stats

T1, interface statsT2, interface stats

• Collect Once, Send Many• Wait for a Period of Time• Repeat


Push Beats SNMP Pull for Speed & Scale

• 10 second poll / push• 3 pollers / telemetry receivers• 30 minute measurement intervals

• 288 100Gig E Interfaces (Line Rate)• SNMP: IF-MIB (query by row)


“OSI model” of Telemetry

Data store

Data model

Producer

Exporter

Collector

Native (raw) data inside a router’s database

Raw data mapped to a model (YANG native, OpenConfig, etc)

Sending requested data in model format to the “Exporter” at defined intervals

Encoding and delivering data to the collector(s) destination(s)

Information collection for processing (e.g., data monitoring, automation, analytics)

“Data”“Layer”

Tele

met

ry e

nd-to

-end



Data store

Data model

Producer

Exporter

Collector





Information collection for processing (e.d., data monitoring, automation, analytics)


Tele

met

ry e

nd-to

-end



Data store

Data model

Producer

Exporter

Collector







Tele

met

ry e

nd-to

-end


YANG Is A Modeling Language

module ietf-interfaces {import ietf-yang-types {

prefix yang; }container interfaces {

list interface { key "name"; leaf name {

type string;}leaf enabled {

type boolean; default "true";

}…

Edited for

Brevity

Self-contained top-level hierarchy of nodes

Import or define data types

Leaf nodes for simple data

Lists for sequence of entries

Containers group related nodes

Other YANG Features

• RO or RW • Optional nodes• Choice• Augment

• When• Arbitrary XML• RPC• etc


Models are Available in Github

You Should Do This

• Telemetry only cares about operational (*-oper.yang) models.

• 143 oper YANG models published for XR 6.1.1• 151 oper YANG are for XR 6.1.2• 177 oper YANG are for XR 6.2.1• 180 oper YANG for XR 6.2.2• 198 oper YANG for XR 6.3.1https://github.com/YangModels/yang/tree/master/vendor/cisco/xr


Finding the Data You Want To Stream

$ pyang -f tree Cisco-IOS-XR-infra-statsd-oper.yang--tree-path infra-statistics/interfaces/interface/latest/generic-counters

telemetry model-driven

sensor-group SGROUP1

sensor-path Cisco-IOS-XR-infra-statsd-oper:infra-

statistics/interfaces/interface/latest/generic-counters


What Will Be Pushed With That Config

{"Timestamp": 1480547974706,"Keys": {

"interface-name": "MgmtEth0/RP0/CPU0/0"},"Content": {

"applique": 0,"availability-flag": 0,"broadcast-packets-received": 25035,"broadcast-packets-sent": 0,"bytes-received": 165321050,"bytes-sent": 233917498,"carrier-transitions": 3,"crc-errors": 0,"framing-errors-received": 0,"giant-packets-received": 0,"input-aborts": 0,"input-drops": 62,"input-errors": 0,"input-ignored-packets": 0,"input-overruns": 0,"input-queue-drops": 0,

"last-data-time": 1480547974,"last-discontinuity-time": 1479244159,"multicast-packets-received": 457,"multicast-packets-sent": 0,"output-buffer-failures": 0,"output-buffers-swapped-out": 0,"output-drops": 0,"output-errors": 104,"output-queue-drops": 0,"output-underruns": 0,"packets-received": 373156,"packets-sent": 311583,"parity-packets-received": 0,"resets": 0,"runt-packets-received": 0,"seconds-since-last-clear-counters": 0,"seconds-since-packet-received": 0,"seconds-since-packet-sent": 0,"throttled-packets-received": 0,"unknown-protocol-packets-received": 0

}

Repeated for all interfaces



Data store

Data model

Producer

Exporter

Collector







Tele

met

ry e

nd-to

-end


Configuring Destination


destination-group DGROUP

address family ipv4 192.168.1.1 port 2104

---- and/or ----

address family ipv6 2001:db8::1 port 2104

encoding self-describing-gpb

protocol tcp

Specify where you want to send your data

Specify how you want your data to look like

Specify how you want your data to be delivered


Basic Concept: Encoding

Encoding (or “serialization”) translates data (objects, state) into a format that can be transmitted across the network. When the receiver decodes (“de-serializes”) the data, it has an semantically identical copy of the original data.

DATA

DATA

“Decode”

“Encode”

IOS XR platforms Encodings

• Compact GPB• Key-Value GPB• JSON (6.3.1)


GPB Encoding

Design Goals

• Simplicity• Performance• Forward/Backward

Compatibility

Non-Goals

• Human-Readable• Self-Describing• Text-based

Google Protocol Buffers (GPB)

Call them “protobufs”

for short

“Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.”


Telemetry Has Two GPB Encoding Options

data_gpb {row {

timestamp: 1485794640469keys: "\n\026GigabitEthernet0/0/0/0"content:

"\220\003\010\230\003\001\240\003\002\250\003\000\260\003\000\270\003\000\300\003\000\310\003\000\320\003\300\204=\330\003\000\340\003\000\350\003\000\360\003\377\001"

}

2X fasterOperationally more complex (but not relative to SNMP!)

data_gpbkv {timestamp: 1485793813389fields {

name: "keys"fields { name: "interface-name" string_value:

"GigabitEthernet0/0/0/0" }}fields {

name: "content"fields { name: "input-data-rate" uint64_value: 8 }fields { name: "input-packet-rate" uint64_value: 1 }

<<< 9 lines are skipped >>> fields { name: "input-load" uint32_value: 0 }fields { name: "reliability" uint32_value: 255 }

}}...

3X largerNative models: still need heuristics for key names

GPB – “compact” GPB – “self-describing”


Dial-Out

• TCP & gRPC (from 6.1.1)• UDP (from 6.2.1)

Dial-In

• gRPC only (from 6.1.1)

Transport Options

Collector

Data

SYNSYN-ACK

ACK Collector

Data

SYNSYN-ACK

ACK


gRPC: Like REST But Different

Runs over HTTP/2Optimize for page load timeServer push, header compression, multiplexing, TLSRFC 7540 (May 2015)Preserves most HTTP1.1 syntax

Defines Services (“RPCs”) Encodes Using Google Protocol Buffers (“GPB” or “protobufs”)

Services and MessagesAuto-generate code in many languages

http://www.grpc.io/docs/#hello-grpc



Data store

Data model

Producer

Exporter

Collector







Tele

met

ry e

nd-to

-end


A Telemetry Subscription


subscription Sub1

sensor-group-id SGROUP1 sample-interval 30000

destination-id DGROUP1

*Omit Destination Group For gRPC Dial-In


Telemetry on Cisco products


Cisco XR Telemetry overview

Classic XR ASR9k Evolved XR ASR9k NCS5500 NCS6kMDT support 6.1.1 6.1.1 6.1.1 6.1.3

Data modelsYANG

(native, OC, IETF) Link for models

YANG (native, OC, IETF)

Link for models


Link for models


Link for models

Transport(Control

protocols)

TCP (dial-out), UDP (dial-out)*

gRPC (dial-in, dial-out), TCP (dial-out), UDP (dial-out)*

gRPC (dial-in, dial-out), TCP (dial-out), UDP

(dial-out)*

TCP (dial-out), UDP (dial-out)*

Encoding GPB / GPB-KV / JSON** GPB / GPB-KV / JSON** GPB /

GPB-KV / JSON**GPB /

GPB-KV / JSON**

Collectors Pipeline*** Pipeline*** Pipeline*** Pipeline***

* UDP support from 6.2.1** JSON support from 6.3.1*** Open-sourced and ready to use: https://github.com/cisco/bigmuddy-network-telemetry-pipeline


NX OS IOS-XE

MDT support 7.0(3)I6(1) 16.6.1*

Data models Data Management Engine, NX-API, YANG (native, OC, IETF)

YANG (native**, IETF) Link for models

Transport(Control protocols)

gRPC* (dial-out), UDP** (dial-out), HTTP***(dial-out)

Netconf (for YANG), GNMI(16.8.1), gRPC (16.9.1)

Formats GPB/JSON XML, GPB (16.9.1)

Collectors Pipeline TBD

Min sample interval 5 sec 1 sec

Max # of dial-out destinations 5 TBD

Cisco NXOS/XE Telemetry high-level overview

* gRPC supports GPB only * supported on Catalyst 3650/3850/9300/9500, ASR1000, ISR4000** UDP from 7.0(3)I7(1), supports both, GPB and JSON ** Native models are different from YANG models in XR*** HTTP suppors JSON only

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/programmability/guide/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_Programmability_Guide_7x_chapter_011000.html

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/prog/configuration/166/b_166_programmability_cg/model_driven_telemetry.html


Some Commonly Used Models and Subtrees

32BRKSPG-2333


Your infrastructure should be ready

Bytes are per a single collection* Go with EDT starting from IOS XR 6.3.x

Data NCS5516numbers

gRPC/KVGPBBW (mbps)

gRPC/GPBBW (mbps)

UDP/JSONBW (mbps)

InterfaceOperState* 2160 0.6806876 0.145277467 0.769932267InterfaceDataRate 2160 0.6365896 0.151012 0.802756InterfacesStats 2160 1.7155632 0.3023296 2.0160128OpticsPortsInfo 576 0.477302222 0.065839378 0.483407289UptimeInfo 1 0.0002816 0.0002288 0.0005968CPUState 18 0.335916 0.053052 0.3043256MemoryInfo 18 0.004036 0.0011584 0.0055872ProcessesMemory 538 0.061503111 0.019817778 0.0832968LLDPInfo 574 0.185312 0.0703288 0.2401824IPv4RIBInfo * 650,022 47.37393978 10.95847427 56.86328129IPv6RIBInfo * 14876 3.6415044 0.856933333 4.371237733BGPIPv4RoutesInfo 650,000 0.0004744 0.0002424 0.000632BGPIPv6RoutesInfo 12800 0.000472 0.00024 0.0006272BGPipv4NeighborInfo 2 0.0390272 0.0054728 0.0391232MPLS-TETunnelsSummaryInfo 1003 0.004604 0.0005632 0.004692RSVPInterfaceInfo 5 0.0015352 0.0004256 0.001972NCS5500NPUStats 96 1.59755976 0.255440227 1.726576827NCS5500NPUResourcesInfo 96 0.2052016 0.0378816 0.2130816

~57mbps ~13mbps ~68mbps


Collectors and analytics platforms


Kafka

Different Collection Models

Logstash

ElasticSearch

KibanaPanda

BYO

Custom Open Source, Customizable

Proprietaryor OS-based

Commercial Stack

Prometheus /InfluxDB

Grafana

Applications

Storage

Collection


Pipeline: An Open Source Collector

Pipeline

Kapacitor

Output to file, TSBD, Kafka…Ingest, transform, filterSelf-monitoring, horizontally scalable


MDT in real time

Demo


Topology

tgn-01Pagent Traffic Generator

xrv-01XRv9k

rcv-01IOS

Gi0/0/0/0 Gi 0/0/0/1

Traffic

Pipeline

TelemetryQoS Data Policy-map output


How does the stack work?

PipelineTransforms telemetry data for DB storage

InfluxDBStores telemetry data

GrafanaData visualization and alerting

Python Flask App to trigger Ansible playbooks

AnsibleAutomation Engine

Cisco Network Service Orchestrator

IOS XR - core data source

NSO


The power of a modular tool chain

Data transformation

Data storage

Data visualization and alerting tool Webhook end point

Automation engine

Service orchestrator

IOS XR - core data source

NSO

= New Service Offering


NSO

telemetry model-drivendestination-group devaddress family ipv4 10.85.204.18 port 5432encoding self-describing-gpbprotocol tcp!!sensor-group QoSsensor-path Cisco-IOS-XR-qos-ma-oper:qos/nodes/node/policy-

map/interface-table/interface/output!

!subscription Sub1sensor-group-id QoS sample-interval 30000destination-id dev!

How does the stack work? – IOS XR

Pipeline address

Sensors

Subscription


Key Takeaways

• Speed & Scale Require Visibility• It’s Not Hard to Beat SNMP• Data Models Are Your Friends• A Big Data Platform Is In Your Future


ResourcesTutorials, Blogs, VoDs

https://xrdocs.github.io/telemetry/http://blogs.cisco.com/sp/the-limits-of-snmphttp://blogs.cisco.com/sp/boring-is-the-new-awesomehttp://blogs.cisco.com/sp/why-you-should-care-about-model-driven-telemetryhttps://youtu.be/tIN8BjHwpNs (NANOG 67: 10 Lessons from Telemetry)

YANGhttps://github.com/YangModels/yang/tree/master/vendor/cisco (Cisco YANG models)http://blogs.cisco.com/getyourbuildon/yang-opensource-tools-for-data-modeling-driven-management(YANG open source tools)https://developer.cisco.com/site/ydk/ (YDK intro)

Telemetry Tools :https://github.com/cisco/bigmuddy-network-telemetry-pipelinehttps://github.com/cisco/bigmuddy-network-telemetry-stacks

Demos and Labhttps://dcloud-cms.cisco.com/demo/mdt-ios-xr-611-v1 (dCloud)https://youtu.be/F_S9-ctNFe0 (demo on NCS 5508)

Thank you.

model driven telemetry

Technology