1

6TiSCH

(IPv6-based) Deterministic WSN

for the Industrial InternetPascal Thubert

Cisco Systems

January 14th, 2015

2

Deterministic Networking

Industrial Internet

Technical choices for the Industrial Internet

6TiSCH in a nutshell

6TiSCH Architecture deep dive

6TiSCH Applications

3

Deterministic Networking

4

New level of (Deterministic?) guaranteesA differentiator for high-end forwarding engines

Sharing physical resources with classical networking

For traffic known a priori (control loops…)

Time Synchronization and global Schedule

5

End-to-End latency enforced by timed pause at station

Periodic trains along a same path and same schedule

Collision avoidance on the rails guaranteed by schedule

5

6

A bus every T. minutes, Stop A to Stop B in X. minutes

Worst end-to-end delivery time < (T + X)

Every packet in an Airbus 380 takes a bus across the fabric

6

7

Two classes of bleeding-edge customers, Industrial and Audio/Video.

Both have moved into the digital world, and some are using packets, but now they all

realize they must move to Ethernet, and most will move to the Internet Protocols.

1. Industrial: process control, machine control, and vehicles.At Layer 2, this is IEEE 802.1 Time-Sensitive Networking (TSN).

Data rate per stream very low, but can be large numbers of streams.

Latency critical to meeting control loop frequency requirements.

2. Audio/video: streams in live production studios.At Layer 2, this is IEEE 802.1 Audio Video Bridging (AVB).

Not so many flows, but one flow is 3 Gb/s now, 12 Gb/s tomorrow.

Latency and jitter are important, as buffers are scarce at these speeds.

3. Vehicule, SmartGrid: streams in live production studiosNorm Finn draft-finn-detnet-problem-statement-01 7

8

Back-of-the-envelope calculations:

1. Industrial:Automotive factory floor: 1000 networks • 1000 packets/s/network •

100,000 s/day = 1011 packets/day.

Machine fails safe when 2 consecutive packets are lost.

At a random loss ratio of 10–5, 10–10 is chance of 2 consecutive losses.

1011 packets/day • 10–10 2-loss ratio = 10 production line halts/day.

In extreme cases, lost packets can damage equipment or kill people.

2. Audio video production: (not distribution)1010 b/s • 10 processing steps • 1000 s/show = 1014 bits = 1010 packets.

Waiting for ACKs and retries = too many buffers, too much latency.

Lost packets result in a flawed master recording, which is the user’s end product.

Norm Finn draft-finn-detnet-problem-statement-01 8

9

1. Zero congestion loss.This requires reserving resources along the path. (Think, “IntServ” and “RSVP”) You cannot guarantee

anything if you cannot say, “No.”

This requires hardware in the form of buffers, shapers, and schedulers. Overprovisioning not useful: its packet loss curve has a tail.

Circuits only scale by aggregation in to larger circuits. ( MPLS? Others?)

0 congestion loss goes hand-in-hand with finite guaranteed latency, also of importance to the users.

2. Seamless redundancy.1+1 redundancy: Serialize packets, send on 2 (or more) fixed paths, then combine and delete extras.

Paths are seldom automatically rerouted.

0 congestion loss means packet loss is failed equipment or cosmic rays.

Zero congestion loss satisfies some customers without seamless redundancy. The reverse is not true in a converged network—if there is congestion on one path, congestion is likely on the other path, as well.

draft-finn-detnet-problem-statement-01Norm Finn 9

10

It’s all about

Real boxes and links

Real buffers and queues

Real packets

Real Time

10

It’s not aboutClassical Layers

Standard bodies

QoS and stat mux

11

• Single nailed-up path

Sender

(Talker)

NIC

NIC

Flow ID

Receiver

(Listener)

Per-Flow State

11

12

Replication & Elimination of individual packets

Sender

(Talker)

NIC

NIC

Flow ID

Receiver

(Listener)

Per-Flow State

12

13

Control (southbound)

interface

Controller

Service (northbound)

interface

Topology,

resources

Sender

(Talker)

NIC

NIC

Receiver

(Listener)

13

14

Control (southbound)

interface

User/app/ser

vice

Per-Flow State

TSpec

User/app/ser

vice

Controller

Service (northbound)

interface

?

Reservation

request

Sender

(Talker)

NIC

NIC

Flow ID

Receiver

(Listener)

14

15

Sender

(Talker)

User/app/ser

vice

NIC

NIC

Per-Flow State

TSpec

User/app/ser

vice

Controller

Flow ID

?

Service (northbound)

interface

Receiver

(Listener)

?

?

UNI

interface

Control (southbound)

interface

UNI

interface 15

16

Same as normal networking, but with the following features for critical data streams:

1. Time synchronization for network nodes and hosts to better than 1 µs.

2. Software for resource reservation for critical data streams (buffers and schedulers in network nodes and bandwidth on links), via configuration, management, and/or protocol action.

3. Software and hardware to ensure extraordinarily low packet loss ratios, starting at 10–6 and extending to 10–10 or better, and as a consequence, a guaranteed end-to-end latency for a reserved flow.

4. Convergence of critical data streams and other QoS features (including ordinary best-effort) on a single network, even when critical data streams are 75% of the bandwidth.

Norm Finn draft-finn-detnet-problem-statement-01 16

17

• IEEE Std 802.1BA-2011 “Audio Video Bridging (AVB) Systems”A profile of a number of standards, picking out required options, special initialization parameters, etc., required for AVB-compliant bridges and end stations.

An “AVB Device” is a device conforming to 802.1BA.

• IEEE Std 802.1AS-2011 “Timing and Synchronization for Time-Sensitive Applications in Bridged Local Area Networks”

A plug-and-play profile of IEEE 1588, including master clock selection, link discovery, and automatic creation of a tree to distribute the clock signal.

• IEEE Std 802.1Qat-2010 “Stream Reservation Protocol (SRP)”The software protocol for making stream reservations

Has been rolled into 802.1Q as Clauses 34 and 35 of IEEE Std 802.1Q-2011.

• IEEE Std 802.1Qav-2009 “Forwarding and Queuing Enhancements for Time-Sensitive Streams”A special credit-based hardware shaper for bridges and end stations that gives better latency guarantees than the usual shapers.

Has been rolled into 802.1Q as Clause 34 of IEEE Std 802.1Q-2011.

• IEEE Std 1722-2011 “Layer 2 Transport Protocol for Time Sensitive Applications in a Bridged Local Area Network”

A Layer 2 transport protocol carrying a time-to-display stamp on each packet.

• (All 802 standards (not 1722) are free 6 months after publication at http://standards.ieee.org/about/get.)

Norm Finn draft-finn-detnet-problem-statement-01 17

18

• P802.1AS-REV* “Timing and Synchronization”Revision of 802.1AS making it clear that it can run on a router as easily as on a bridge.

• P802.1Qbu* “Frame Preemption”Amends 802.1Q to support 802.3br

• P802.3br “Interspersed Express Traffic”One level of transmission preemption – interrupts transmission of an ordinary frame to transmit an “express” frame, then resumes the ordinary.

802.3 document, not an 802.1 document.

• P802.1Qbv* “Enhancements for Scheduled Traffic”Runs the 8 port output queues of a bridge on a rotating schedule.

• P802.1Qca* “Path Control and Reservation”Enhances 802.1 ISIS to create multiple paths through a network.

• P802.1CB* “Seamless Redundancy”Defines the sequence-split-recombine method for reliability improvement.

Stand-alone document. NOT an amendment to 802.1Q.

• P802.1Qcc* “Stream Reservation Protocol (SRP) Enhancements and Performance Improvements”For more streams, faster convergence, less chattiness, and maybe more.

* For the necessary password, Google “p802.1 username password”

Norm Finn draft-finn-detnet-problem-statement-01 18

19

Industrial Internet

20

Challenge: harness innovationMore efficient operations

New and/or improved experience

Beyond Control and AutomationOptimize processes (by 1%?)

Leveraging IT, Live big data and Analytics

21

Control Loops

global optimization of routes for jitter an latency – thus computed by a PCE -,

flow (over) provisioning, duocast for 1 hop (1Hz and 4Hz, no routing) with static multipath hard slot allocation.

Control Loop plan B

self-healing -thus distributed- routing, RPL

Supervisory Flows

no goal: requires determinism up into the backbone

Management

a separate topology – a RPL instance - that does not break.

OF?, root selection?

Alerts

bursty, unexpected, on-demand slot allocation, prioritization

22

Large Scale Monitoring / Live Big Data

stuff like corrosion? low cost scalability – this distributed routing, soft slots?

Device (Limited) Mobility

E.g. Cranes spanning area (ship)

with linear and circular motion that are mobile within a range

Mobile Worker

additionally provision slots ready for mobile workers) does not need deterministic since human

Wide Subnets

thousands of devices– thus a backbone, multiple BBRs for 1 LLN –

within a subnet – to avoid renumbering though not always the case, sometimes isolation is wanted

Non-Production Episodes

fast and autonomic behavior – again distributed routing

Coexistence with Legacy Devices

a common management above, coarse level makes sense (channels backlisting but no more…single admin is good)

23

• Not Process Control but “Missing Measurements”

Reliability and availability are important, which implies

Scheduling and admission control

• Scalability

10’s of thousands of new devices

• Deployment cost factor is key

For Emerson this is the second layer of automation:

Typically missing are the measurements you need to

monitor the condition of the equipment--temperature,

pressure, flow and vibration readings you can use to

improve site safety, prevent outages and product losses,

and reduce maintenance costs of equipment such as

pumps, heat exchangers, cooling towers, steam traps and

relief valves.

24

15M to 115M Analytics related connections*

Classical Monitoring only doubles

Analytics related M2M connections surge

* Source: ABI Research

25

* Source: ABI Research

Big Data and

Analytics

“Without adequate analytics clout available, and the right practices to take advantage of it, the companies

rolling out M2M solutions will be destined to be stuck in the lower realms of applications: monitoring, reporting,

and simple rules-based actions.” *

Large Scale

monitoring

6TiSCH

26

Industrial

Internet

Control Data

Office Data

Business Data

Nb of Devices (order of magnitude)

Pro

cess C

riti

cali

ty

1 000250 10 000 1 000 000

ISA100.11a

WiFi

WirelessHART

6TISCH / PCE

LPWA

6TISCH / RPL

CG-Mesh

27

Maintenance and operation represent 75% of the Total equipment cost

Dow

ntim

e

Maintenance & operation COST

Corrective maintenance

Preventive

Maintenance

Predictive

Maintenance

Actionable Prescriptions

Deployment of Wireless sensors is seen as an efficient solution

28

Data: Capillary Wireless Acquisition of large amounts of live Big Data

Information: Fog DiM to add descriptive meta-tags, allowing for compression and correlation

Knowledge: Prediction from self-learned models and Knowledge Diffusion

Wisdom: Machine Learnt Expertise, auto-Generating Prescriptions and Actionable Recommendations

(Data)

ACQUISITION

(Knowledge)

DIFFUSION

(Wisdom)

OPEN LOOP

(Information)

AGGREGATION

6TiSCH Data in

Motion / Fog

Learning

Machines

/ Cloud

29

• IIC website http://www.iiconsortium.org

• New York Times http://bit.ly/IIC-03272014

• IEEE http://flip.it/zbfnU

• Huge tutorial http://bit.ly/iic-319

• McRock's Industrial Internet of Things Report 2014

• AT&T, CISCO, GE, IBM AND INTEL FORM INDUSTRIAL INTERNET CONSORTIUM TO

IMPROVE INTEGRATION OF THE PHYSICAL AND DIGITAL WORLDS March 27, 2014

30

Customers after next % point of operational optimization:

Requires collecting and processing of live “big data”, huge amounts of missing measurements by widely distributed sensing and analytics capabilities.

Achievable by combination of the best of IT and OT technologies together, forming the IT/OT convergence, aka Industrial Internet.

Architectural approach, standards, Industry adoption needed to embrace radical changes happening in IT networking technologies. Products are following.

Secured-by-default model required throughout network lifecycle.

6TiSCH extends Deterministic Wireless Industrial Networking technologies to also reach higher scales at lower costs (but then, guarantees as well).

Operational technology (OT) is hardware and software that detects or causes a change through the

direct monitoring and/or control of physical devices, processes and events in the enterprise.

31

Technical choices for theIndustrial Internet

32

Link State requires full & constant topology awareness

=> 6TiSCH / RPL chose Distance Vector

Wireless scales but power constraints and fuzzy topology

=> Different risks, value in diversity

TDMA requires timesync, CSMA-CA needs idle time

Different usages, deterministic vs. best effort

Reactive mixes routing and forwarding (eg IPv6 ND)

Centralized optimizes, Distributed scales economically

=> Different usages, deterministic vs. best effort

Link State ?

Wired?

CSMA?

Reactive ?

Centralized

?

Single

Hop ?Spatial reuse

=> Wired backhaul a good idea in any case

33

Reaching farther outNew usages / types of devices

Global Coverage from Near Field to Satellite via 3/4G

BUT

Lack of trust in Industrial vs. WiredMultiple Interferers, ISM band crowdedIssues with IPv6 for Scalability and Mobility

New level of cost effectiveness

Deploying wire is slow and costly

Low incremental cost per device

34

Higher speed does not necessarily mean more energyEnergy budget: Power vs. Time

LP WiFi a contender to 802.15.4

802.11ah designing for 2 hops

Interesting developments in LowPower-WideAreaultra narrow band (e.g. SigFox 100KHz)

802.15.4K (e.g. OnRamp Wireless)

LOng RAnge (Cycleo / Semtech, 2.4GHz

35

16

ch

an

ne

l o

ffse

ts

e.g. 31 time slots (310ms)

A

BC

DE

FG

H

I

J

• Schedule => direct trade-off between throughput, latency and power consumption.

• A collision-free communication schedule is typical in industrial applications.

• But requires network synchronization, and de-sync means long isolation

36

Channel Hopping : retry around interference,

round robin strategy

Time Slotted (or Synchronized) : Deterministic: through TDM, Synchronized + Time formatted in SlotFrame(s)

Tracks: below IP, can be orchestrated by a third party like virtual circuits

Slotted: benefits of slotted aloha vs. aloha => reduce chances of collisions

Battery operation: when traffic profile is known, devices only wake upon need

37

Ethernet

Complexity

DV

Single

Hop ?

CSMA-CACSMA-CD

Wired

Single

Hop ?

6TiSCH

Reactive

Distributed

?

CG-Mesh LTE

ISA100.11aYES NO

WiFi

38

Direct Communication when possible yields:

• Simpler design

• Lower OPEX

• Multipath issues and uneven transmission quality

Low Power TSCH mesh is a complex technology adapted to:

• Range extension with Spatial reuse of the spectrum

• Centralized Reservation for deterministic critical data streams.

• Separation of resources for classical IP routing (RPL)

Norm Finn draft-finn-detnet-problem-statement-01 38

39

6TiSCH in a nutshell

40

IEC based on HART 7.1.

TDMA

fixed time slots (10ms)

Mesh only

Shipped YE-2008.

Vendor driven

Emerson, E&H, ABB, Siemens

IEC based on 2011 revision

TDMA+CSMA

Var. time slots

Star, mesh and hybrid topology

IPv6, 6LoWPAN, TCP-friendly

Shipped mid-2010

Mostly user driven

Honeywell, Yokogawa, Invensys

IEC 62601

WIA-PA

IEC 62591 IEC 62734

ISA100.11a

Alternate from China

Star, mesh and hybridtopology

Standardization workstarted in 2006.

41

6TiSCH

WiHART WIA-PAISA100Common Network

Management and Control

HART WIA-PAISA100

MGT /

CTRL.

APPLI.HART WIA-PAISA100

Better Co-ExistenceTRIPLE OPEX + interferences

Wia-PA ISA100 NETW. Wireless

HART

42

• Industrial requires standard-based products

• Must support equivalent features as incumbent protocols

• Must provide added value to justify migration

• 6TiSCH value proposition

Design for same time-sensitive MAC / PHY (802.15.4e TSCH)

Direct IPv6 access to the device (common network mgt)

Distributed routing & scheduling for scalability (for monitoring)

Large scale IPv6 subnet for mobility (50K +)

43

Active IETF WG, 6 WG docs in or close to last call

Define an Architecture that links it all together

Align existing standards

(RPL, 6LoWPAN, PANA, RSVP, PCEP, MPLS) over 802.15.4e TSCH

Support Mix of centralized and distributed deterministic routing

Design 6top sublayer for L3 interactions

Open source implementations (openWSN…)

Multiple companies and universities participating

Cisco Confidential 44© 2013-2014 Cisco and/or its affiliates. All rights reserved.

6TiSCH Client stack

Centralized route and

track computation

and installation

Management and

Setup

Discovery

Pub/Sub

Authentication for

Network Access

Wireless ND

(NPD proxy)

Time Slot

scheduling and

track G-MPLS

forwarding

Distributed route and

track computation and

installation

Distributed route and

track computation and

installation

IEEE 802.15.4e TSCH

6LoWPAN HC

IPv6

RPL

6top

TCP UDP ICMP CCAMP

PCEP/PCC CoAP/DTLS AAA 6LoWPAN ND

}

45

Limit of IP direct connectivity

ISA100.11a ||

WirelessHART

ISA100.11a ||

WirelessHARTSensor

Wireless Control

Loop

Actuator

PLC

Root AP

Management

Wireless Controller

NAC

/Firewall

- Control loop limited to subnet

- No end-to-end IP connectivity

- No single view management

- No Cisco presence in control loop

Mesh AP +

Manager +

App GW

VPN Concentrator

46

6TiSCHSensor

Actuator

Backbone

Router

PCE

Intelligent device

Management

Wireless Controller

Backbone

Router

Limit of IPv6 subnet(s)Limit of IPv6 subnet

End to end IPv6 routing

NAC

/Firewall

VPN Concentrator

47

6TiSCH

6TiSCHSensor

Actuator

Backbone

Router

Management

Wireless Controller

Backbone

Router

Single Multilink IPv6 Subnet with free inner mobility (same subnet == no renumbering)

Virtual Service Engine

(virtual PLC, PCE …)

+ IPv6 ND registrar

+ Network Function

Virtualization (NFV)

NAC

/Firewall

VPN Concentrator

IPv6 ND registration and proxy operation

48

So WHY ?

• From a technology point of view

Scheduled network + PCE

Emulate Industrial WSNs including TSN aspects and limitations (mobility, scalability)

Peristaltic Scheduling + Distributed routing and scheduling (RPL) + Efficient IPv6 ND (WiND)

Large scale monitoring for Industrial Internet in Co-Existence with Industrial WSNs

• From a user point of view

Common Network management (over IPv6)

Open standards (IETF, IEEE, IEC)

Reduced OPEX (though we suggest end-to-end pcpl & multiple apps as opposed to single protocol)

• From a market point of view

IPv6 end-to-end enable control loop virtualization

Support dynamic scheduling and deterministic on a same network

Single administration, lower OPEX

49

Deterministic IPv6 over IEEE802.15.4e TimeSlotted Channel Hopping (6TiSCH)

The Working Group will focus on enabling IPv6 over the TSCH mode of the IEEE802.15.4e standard. The scope of the WG includes one or more LLNs, each one connected to a backbone through one or more LLN Border Routers (LBRs).

Active drafts

http://tools.ietf.org/html/draft-ietf-6tisch-terminology

http://tools.ietf.org/html/draft-ietf-6tisch-tsch

http://tools.ietf.org/html/draft-ietf-6tisch-architecture

http://tools.ietf.org/html/draft-ietf-6tisch-minimal

http://tools.ietf.org/html/draft-ietf-6tisch-6top-interface

http://tools.ietf.org/html/draft-ohba-6tisch-security

http://tools.ietf.org/html/draft-ietf-6tisch-coap

50

Converged Plant NetworkHigh availability

Flow Isolation

Guaranteed Bandwidth

IP based Control NetworkAutonomic, zero touch commissioning

Time Sensitive Networking for critical apps

Packet Reliability

IPv6-based Wireless Field NetworkDeterministic, Autonomic, Secure

Large Scale for Monitoring (RPL)

Backward Compatible (with ISA100 or HART)

10’s

of K

10’s

100’s

Control

network

Converged plant net

Wireless Field network

51

6TiSCHArchitecture deep dive

52

53

1. Scans all channels to detect

Extended Beacon

2. Synchronizes

3. Parses beacons for schedule

4. Performs Join Process (AAA)

5. Renumbers and DADs

6. Route setup (eg RPL DAO)

6TiSCH forms a single multilink

subnet, no loss of sync, no

renumbering

Discovery steps; new/moving device:

54

Neighbors are discovered at L2 from 15.4e MAC Extended BeaconTime parent selection based on a join priority field in the EBJoin priority as a distance but no real loop avoidance

=> time loops and

=> node isolation

When a new node B joins the networkit needs to wait for an EB to align itself to the sequence of slots

EBs are sent using TSCHNode cannot know in advance the channel to listen

it may take a long time for B before catching an EB

The problem exacerbates for bootstrapping networks with many hops

After a node joinssynchronization should be maintained using data/ack and keepalive handshakes

55

One RPL instance dedicated to time parenting

RPL Rank used as join priority, set once once joined that timing instance

0xFF (Infinite) as long as not joined that instance => not a suitable time parent

RPL provides a better loop avoidance yet may let temporary loops

If DAGMaxRankIncrease is non-zero

8.2.2.4. Rank and Movement within a DODAG Version

If multiple control messages are lost

Loops will be detected reactively on traffic

3.2.7. Data-Path Validation and Loop Detection

11.2. Loop Avoidance and Detection

Q: Proscribe DAGMaxRankIncrease > 0 ?

56

56

Multiple uncoordinated DODAGs with independent roots (differing DODAGIDs)

DODAGs partition the network

No connectivity between DODAGs

to distribute time:

e.g. roots are GPS-based time sources

Nodes may jump to alternate DODAG

A single DODAG with a single root

May be used for time synchronization

Time may drift with distance if large mesh

A single DODAG with a virtual root that coordinates LLN sinks (with the same DODAGID) over a backbone network

May be used for time synchronization

Time can be synchronized over the backbone

e.g. Precision Time Protocol (IEC 61588, IEEE 1588v2)

6TiSCH

LLN

6TiSCH LLN

6TiSCH

LLN

Common time

source

Floating

root

57

58

Centralized

God’s view optimization

Multipath redundancy

Deterministic (optimized)

Virtualization

Distributed

Autonomic & Mobile

Highly available (DARPA)

Deterministic

Scalability

59

Distance Vector + stretch

Peer only with parents

DV + Non-storing mode

Lazy Update & datapath valid.

Constrained instances, TID

Req coupling with LISP/NEMO

Dynamic topologies

Peer selection

Constrained Objects

Fuzzy Links

Routing, local Mobility

Global Mobility

Low Power Lossy Nets Addressed in RPL ?

60

In the context of routing, a Directed Acyclic Graph (DAG) is formed by a collection of vertices (nodes) and edges (links).

Each edge connecting one node to another (directed) in such a way that it is not possible to start at Node X and follow a directed path that cycles back to Node X (acyclic).

A Destination Oriented DAG (DODAG) is a DAG that comprises a single root node.

Here a DAG that is partitioned in 2 DODAG

Clusterhead

5

4

4

0

1

3

1 1

2

2

2

2

2

3

3

3

3

3

3

2

4

4

5

0

6

5

4

61

In Green: A’s subDAG.

Impacted if A’s connectivity isbroken

Domain for routing recovery

In Red: B’s fanout DAG

(or reverse subDAG)

Potential SPAN on B’s DAO

Thus potential return paths

Fanout must be controlled to limitintermediate states

Clusterhead

5

4

4

0

1

3

1 1

2

2

2

2

2

3

3

3

3

3

3

2

4

4

5

0

6

5

4

A

B

62

e.g. ARC of siblings

Allows more alternates

ARCs are walked with no routingprogress

Forwarding over DAG AND ARCs

Reduces congestions of upper links of the DAG

Still LORA for P2P

IGP subarea (bidirectional)

Link selected and oriented by TD

Potential link

Clusterhead

5

Clusterhead0

1

11

2

2

2

2

2

3

3

3

3

3

3

2

3

5

4

4

4

4

63

Either

routes in the RIB / VRF or

G-MPLS forwarding state

Indexed by tuple (IPv6, InstanceID)

Steps:1. A needs a route to B

2. A selects a local InstanceID

3. A requests a computation from PCE

4. PCE returns route steps (and timeslots)

5. A install routing/forwarding states alongreturned route or use source routing

Track

PCEP request Potential link

Clusterhead

5

Clusterhead0

1

11

2

2

2

2

2

3

3

3

3

3

3

2

3

5

4

4

4

4

PCE

BA

64

The RPL instance ID allows different routing optimizations, constraints and policies.

D Local Instance ID 0..64

0 1 2 3 4 5 6 7

0 Global Instance ID 0..128

0 1 2 3 4 5 6 7

1

The RPL instance ID is encoded in 1 octet. The first bit indicates whether Global or Local.

“A local RPLInstanceID is autoconfigured by the node that owns the DODAGID and it MUST be unique for that DODAGID. The DODAGID used to configure the local RPLInstanceID MUST be a reachable IPv6 address of the node, and it MUST be used as an endpoint of all communications within that Local instance.”

Inside a packet: “If the 'D' flag is set to 1, then the destination address of the IPv6 packet MUST be the DODAGID. If the 'D' flag is cleared, then the source address of the IPv6 packet MUST be the DODAGID.”

6TiSCH extends RPL’s language of DODAGID to route (reservation) endpoint.

65

128 global instances per network

Indexed by tuple (IPv6, InstanceID)

Running as Ships-in-the-night

1 instance = 1 VRF = 1 « L3 vlan »

Serving different Objective Functions

Using different metrics

Can be shared between RPL and PCE

RPL along a DODAG

PCE adding orthogonal shortcuts

Clusterhead0

1

11

2

2

2

22

3

3

3

3

3

3

2

3

5

4

4

4

4

Clusterhead

Constrained instance

Default instance

Potential link

A

66

Clusterhead

5

Clusterhead0

1

11

2

2

2

22

3

3

3

3

3

3

2

3

5

4

4

4

4

64 local Instances

per IPv6 source address

Indexed by tuple (IPv6, InstanceID)

Used by RPL or PCE

RPL: for P2P applications

RPL: to index RSVP path

PCE: Serves as Track ID,

included in PCEP request

from the source device

Track –PCE)

RSVP reservation for RPL

Potential link

67

RPL is an extensible proactive IPv6 distance vector protocol

Supports MP2P, P2MP and P2P between devices (leaves) and a root (border router)

P2P reactive extension

RPL specifically designed for “Lossy” networks

Agnostic to underlying link layer technologies (802.15.4, PLC, Low Power Wireless)

RFC 6206: The Trickle Algorithm

RFC 6550: RPL: IPv6 Routing Protocol for LLNs

RFC 6551: Routing Metrics Used for Path Calculation in LLNs

RFC 6552: Objective Function Zero for the Routing Protocol for LLNs

RFC 6553: RPL Option for Carrying RPL Information in Data-Plane Datagrams

RFC 6554: An IPv6 Routing Header for Source Routes with RPL

RFC 6719: MRHOF Objective Function with hysteresis

RPL is pronounced

“Ripple”

68

69

A

B

X Y

U

V

1TX

1TX

1RX

1RX

shaping

QoS

RED

1+1 TX

Packets

are

serialized

70

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

UYA X

71

1TX

1TX

1RX

1RX

Dest MAC set to

broadcast 0xFFFF

DSCP Deterministic

Dest MAC restored by

6top at final destination

Dest MAC not changed

DSCP Deterministic

2TX 2RX

A

B

X Y

U

V

72

1TX

1TX

1RX

1RXAdditional retries

Using L3 bundle

Dest MAC set to Y

DSCP Deterministic

Transmission failure

Over track slots

Retries exhausted over

Track L2 bundle

2TX 2RX

A

B

X Y

U

V

If arrival in time

Dest MAC reset to

broadcast 0xFFFF

DSCP Deterministic

Placed back on track

Dest MAC set to

broadcast 0xFFFF

DSCP Deterministic

1TX 1RX

73

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

UYA X

74

ISA100.11a /

WIHART

15.4e TSCH 15.4e TSCH 15.4e TSCH 15.4e TSCH

15.4 PHY 15.4 PHY 15.4 PHY15.4 PHY

CoAP

UDP

IPv6

6LoWPAN-HC

6top

CoAP

UDP

IPv6

6LoWPAN-HC

6top

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LoWPAN-HC

6top

15.4 PHY

15.4e TSCH

ISA100.11a /

WIHART

UDP

IPv6

6LoWPAN-HC

6top

CoAP

Dest MAC set to

broadcast

0xFFFF

Dest MAC not

changed

Dest MAC restored

by 6top

75

1TX

1TX

1RX

1RX

Dest MAC restored

by 6top

Dest MAC set to X

DSCP not Deterministic

2TX 2RX

P

B

X Y

U

V

A

Q

Packet to be routed via Y

Dest MAC set to Y

Placed of deterministic track

Packet extracted from track due

to dmac == Y and/or DSCP and

then routed to a next hop != V

No frame in slot

Associated to

deterministic

76

A

B

X Y

U

V

1TX1RX

1RX

Next

fragment

follows

n TX

First

fragment

installs

states

77

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

15.4 PHY

15.4e TSCH

CoAP

UDP

IPv6

6LP-HC

6top

UYA X

1st1st

NextNext

78

79

Channelo

ffset

Up to 1

6 c

hannel

slotOffset, 0 .. N

The matrix repeats itself over time. CDU is defined by 6TiSCH and represents the global characteristics of the

network, channel unused, duration of the timeslot and number of timeslots per iteration.

6TiSCH defines a new global concept that is called a Channel distribution/usage (CDU) matrix; a

Channel distribution/usage (CDU) matrix is a matrix of so-called “cells” with an height equal to the

number of available channels (indexed by ChannelOffsets), and a width in timeslots that is the period of

the network scheduling operation (indexed by slotOffsets).

Timeslot width, typically 10-15ms each in 802.15.4e

Cell (slotOffset,

channelOffset)

80

A Slotframe is a MAC-level abstraction that is internal but common to all nodes and contains a series of

timeslots of equal length and priority. It is characterized by a slotframe_ID, and a slotframe_size. Multiple

slotframes can coexist in a node’s schedule, i.e., a node can have multiple activities scheduled in different

slotframes, based on the priority of its packets/traffic flows. The timeslots in the Slotframe are indexed by

the SlotOffset; the first timeslot is at SlotOffset 0.

Channelo

ffset

Up to 1

6 c

hannel

slotOffset, 0 .. N

TimeSlotSlotframe

CDU

matrixCell (slotOffset,

channelOffset)

Timeslot width, typically 10-15ms each in 802.15.4e

81

A Chunk is a well-known list of cells, well-distributed in time and frequency, within the CDU matrix; a chunk

represents a portion of the CDU. The partition of the CDU in chunks is globally known by all the nodes in the

network to support the appropriation process, which is a negotiation between nodes within an interference

domain. A node that manages to appropriate a chunk gets to decide which transmissions will occur over the

cells in the chunk within its interference domain. Ultimately, a chunk represents some amount of bandwidth

and can be seen as the generalization of a transmission channel in the time/frequency domain.

Chunk 1

Chunk 2

...

Chunk 8

CDU matrix, 6 channels, 7 timeslots, 8 chunks, represented from ASN= 0

82

The chunks are defined at the epochal time but the 802.15.4e operation is an iterative process that repeats

over and over. Typically, the effective channel for a given transmission is incremented by a constant that is

prime with the number of channels, modulo the number of channels at each iteration. It results that the

channel of a given transmission changes at each iteration and the matrix virtually rotates.

To illustrate that rotation, we represent above where the transmission would happen in the second

iteration, for a CDU matrix of 8 channels if we increment the effective channel by 3 at each iteration:

Chunk 1

Chunk 2

...

Chunk 10

Effective channel offsets for chunks 1 .. 10 at the reference epoch and after one iteration

83

slotOffset, 0 .. N

High PriorityRCV XMIT RCVSlotframe 1

slotOffset, 0 .. M

Low PriorityRCV RCV RCV XMIT XMITSlotframe 2

Will receive

Will receive if nothing to xmit

Will xmit

Will receive

Will receive

Decision to transmit / rcv

made at the beginning of

time slot based on

existing frames in queues

and slotframe priorities

84

11 12 13

25242322

3534333231

464544434241

85

86

Parent schedule

Child 1 schedule

Child 2 schedule

Child 3 schedule

Xmit mcast rcv rcv xmit rcv xmit

rcv rcv xmit

rcv xmit rcv

rcv xmit

Slotframe

Say a parent

appropriates the

pink chunk from

the CDU matrix

above. Now the

parent gets to

decide when the

pink cells are

used and by

which node.

A device maintains a schedule that dictates its

transmissions and receptions inside the slotframe.

The RPL parent

allocates timeslots

between itself and

its children. The

parent takes the

timeslots off a

chunk that it has

appropriated.

87

88

6TiSCH finally defines the peer-wise concept of a bundle, that is needed for the communication between

adjacent nodes.

A Bundle is a group of equivalent scheduled cells, i.e. cells identified by different [slotOffset,

channelOffset], which are scheduled for a same purpose, with the same neighbor, with the same flags, and

the same slotframe.

The size of the bundle refers to the number of cells it contains. Given the length of the slotframe, the size

of the bundle translates directly into bandwidth, either logical, or physical. Ultimately a bundle represent a

half-duplex link between nodes, one transmitter and one or more receivers, with a bandwidth that amount

to the sum of the timeslots in the bundle. Bundles thus come in pairs, an incoming and an outgoing.

A bundle is globally identified by (source MAC, destination MAC, trackID)

timeslot

bundle

89

We use a well-known constant as trackId for a L3 link, that I’ll refer as NULLT.

So an IP link between adjacent nodes A and B comprises 2 bundles, (macA, macB, NULLT) and (macB,

macA, NULLT).

timeslot

bundle

timeslot

bundle

outgoingIncoming

IP layer

AA

B

90

Along a that has a segment …A->B->C… , there are 2 bundles in node B,

incoming = (macA, macB, trackId) and outgoing = (macB, macC, trackId).

timeslot

bundle

timeslot

bundle

outgoingIncoming

C

B

A

91

92

Multiple L3

Multicast

messages

RA

Protections: MLD snooping for SNMA (limited)

and RS. Cisco only: IPv6 FHS (SISF) in WLC

1. MAC address flooded over

spanning tree for L2

switching

2. Device sends multiple

multicasts

3. L2 fabric handles as

broadcast

4. Broadcast clogs the wireless

access at low access speed

VM, NFV,Wireless or IoT device moves:

Multiple L2

Broadcast

messages

93

Authoritative Registrar(s)MIPv6 HA, 6LBR, interface to external services

Intermediate Registrars6LR, NEAR, Optionally ND proxy

Backbone RoutersRPL root, ND proxy

Legacy IPv6 devices

Authoritative

Registrar

Authoritative

Registrar / 6LBR

Intermediate

Registrar / 6LR

Intermediate

Registrar /

opt ND proxy

Backbone router

(ND proxy)

94

6LR Wireless

Router

IPv6 registrar

(ND proxy)

6LR Wireless

Router

IPv6 registrar

(ND proxy)

6LBR/RPL Root 6LBR/RPL Root

6LR Wireless

Router

6LR Wireless

Router

Backbone

Gate

wa

y

Gate

wa

y

95

Multicast AvoidanceRegistration for DAD

Binding (location, owner, MAC) to IPv6 address

Registrar hierarchy for lookups and policy enforcement

Scalable Backbone OperationL2 routing (IS-IS) + proxy-ND in the backboneRegistration + L3 Routing in

attached networks (RPL RFC 6550)

Optional MAC address proxying to avoid MAC@ floods

New ND methods between registrars and other devices (e.g. LISP)

IETF draftsdraft-thubert-6lowpan-backbone-router

draft-chakrabarti-nordmark-6man-efficient-nd

draft-thubert-6man-wind-sail

96

6LR 6LBR 6BBR Router/ServerLP Node

Any

IPv6 Radio

Mesh Ethernet

NA (~O)

NS (ARO)

NS (ARO)

DAR(ARO), DAO

Router/ServerRouter/Server

Ethernet

inside

Radio (1 Hop)

NS lookup

NS DAD

NS (ARO)

Create proxy state

Classical NDRPL, others…6LoWPAN ND Efficient ND

NA (ARO)

DAC(ARO)

NA (ARO)

ML Subnet

97

A new Efficient ND, aka Wireless ND

Multicast Avoidance

Registration for DAD

Binding (location, owner, MAC) to IPv6 address

Registrar hierarchy for lookups and policy enforcement

OperationL2 routing (IS-IS) + proxy-ND in the backboneRegistration + L3

Routing in attached networks (RPL)

Optional MAC address proxying to avoid MAC@ floods

New ND methods between registrars and other devices (e.g. LISP)

98

Used to resolve conflicts

Need In ND: TID to detect movement ->eARO

Need In RPL: Object Unique ID if we use RPL for DAD

0 1 2 3

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| Type | Length = 2 | Status | Reserved |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

|Res|P|N| IDS |T| TID | Registration Lifetime |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

| |

+ OUID ( EUI-64 or equivalent ) +

| |

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

99

Routers within subnet have a connected route

installed over the subnet backbone.

PCE probably has a static address in which

case it also has a connected route

Connected

Route to

subnet

100

Gateway to the outside participate to some IGP

with external network and attracts all extra-

subnet traffic via protocols over the backbone

Default

Route

In RIB

101

Directly upon NS(ARO) or indirectly upon DAR

message, the backbone router performs DAD on

behalf of the wireless device.

DAR

NS

(ARO)

DADNS DAD

(ARO)

102

NA(ARO) or DAC message carry succeful

completion if DAD times out.

NA(Override) is optional to clean up ND cache

stale states, e.g. if node moved.

DAC

NA

(ARO)

Optional

NA(O)

103

The BR maintains a route to the WSN node for

the DAO Lifetime over instance VRF. VFR may

be mapped onto a VLAN on the backbone.

RPL

DAO

Host

Route

Optional

NA(O)

104

The BR maintains a route to the WSN node for

the DAO Lifetime over instance VRF that is

continued with RPL over backbone.

RPL

DAO

RPL DAO

Host

Route

105

DAD option has:

Unique ID

TID (SeqNum)

Defend with NA if:

Different OUID

Newer TID

NS DAD

(ARO)NA (ARO)

NS

(ARO)

106

DAD option has:

Unique ID

TID (SeqNum)

Defend with NA if:

Different OUID

Newer TID

DAR

NA

(ARO)

DAD

107

RPL

DAO

Optional

NA(ARO)

Host

Route

DAD option has:

Unique ID

TID (SeqNum)

Defend with NA if:

Different OUID

Newer TID

NA (ARO) with

older TID (loses)

108

Packet

NS

lookup

NA ARO option has:

Unique ID

TID (SeqNum)

NA (ARO)

109

NS

lookup

Mixed mode ND

BBR proxying over

the backbone

NA (ARO)

Packet

110

6TiSCHApplications

111

Control loops

global optimization of routes for jitter an latency – thus computed by a PCE -,

flow (over) provisioning, duocast for 1 hop (1Hz and 4Hz, no routing) with static multipath hard slot allocation.

control loop plan B

self-healing -thus distributed- routing, RPL

supervisory flows

no goal: requires determinism up into the backbone

management

a separate topology – a RPL instance - that does not break.

OF?, root selection?

alerts

bursty, unexpected, on-demand slot allocation, prioritization

112

Monitoring of lots of lesser importance

stuff like corrosion? low cost scalability – this distributed routing, soft slots?

Cranes spanning area (ship)

with linear and circular motion that are mobile within a range

no goal some will need deterministic for coordination between cranes

Mobile worker

additionally provision slots ready for mobile workers) does not need deterministic since human

Large plants

thousands of devices– thus a backbone, multiple BBRs for 1 LLN –

within a subnet – to avoid renumbering though not always the case, sometimes isolation is wanted

Non-production episodes

fast and autonomic behavior – again distributed routing

Coexistence with legacy devices

a common management above, coarse level makes sense (channels backlisting but no more…single admin is good)

113

Deterministic

Wi-Fi or Ethernet

EnterprisePlant Control

Network

114

(road, street) traffic control

increase of bandwidth allocated as more traffic in the road

Smart urban parking

redundancy of routes an over-provision as RF degrades with cars on top of the sensors

Green zones. Monitor moisture in gardens, trigger watering.

Toll, micro-paiment

Gargabe detection / collection path optimization

Citylights monitoring

large scale meshes

115

an entity operating an "umbrella" network on which different types of traffic flow, possibly coming from customers of this entity.

When a new customer asks to be granted access to the network, the network operator can predict with pretty good granularity the impact this new traffic on the remaining bandwidth of the network, and on the power consumption of individual motes, which allows the operator to grant/deny, and probably charge differently

116

Resource management

Heating, Cooling and Lighting

Air quality, water leaks etc…

Food / Drink vending machines

Room occupancy

Security and Safety

(e.g. locate people in case of fire)

Exit signs monitoring

TSCH for improved availability

redundancy as RF degrades when

there are changes on the environment,

doors, moving metallic apparel, etc…

117

Less wire / copper Lower cost of goods

Easier assembly => lower manufacturing cost

Lower weight => lower gas consumption

Less maintenance / repair

Easier addition of second mount devices

Connected vehiculeRemote maintenance

Fleet managements

118

asset tracking operations on the move (again mobility, and dynamics).

monitoring e.g. temperature of freezers, intrusion sensors, …

119

Domotics & Home Automation/ Monitoring & Security

Energy and water use:

energy and water supply consumption monitoring to obtain advice on how to save cost and resources.

Intrusion Detection Systems:

Detection of window and door openings and violations to prevent intruders

Brute force & scalability to defeat tempering

120

The Vision

121

Trillion devices in the InternetThe core technologies will not change

Leakless Autonomic Fringe

Leakless Route Projection

Million devices in a SubnetNew models for the subnet protocols

IPv6 ND, ARP, Service Discovery …

Fiber + Copper + Radio Backbone

802.11 + 802.15.4 + … Fringe

Unhindered MobilityLocation / ID Separation

10’s

of K

Thank you.

123

The Problems

124

10s of Ks devices with multihop LLN access

No broadcast in the LLNs

No renumbering (quasi permanent IPv6 address)

Continuous reachability as a LLN device moves in Subnet

Radio conditions change cause network reorganisation

Already use cases such as handheld, cranes, small vehicles

Backward Compatibility with classical IPv4 and v6 devices

Support of discovery protocols throughout the subnet

125

Cost Optimized in the backbone

Any to any traffic

Multipath and load balancing a bonus

Power Optimized in the LLNs

Most traffic flows to or from the LLN Backbone Router

Control traffic must be limited vs. Data traffic

Broadcast or scalable multicast in the backbone

Multicast or Controlled dissemination in the LLN

126

LLNs Time synchronization via the backbone

to keep all LLNs in sync and

allow movement from one LLN to the next

Slow deterministic LLNs (process control)

No need for Deterministic backbone

Until scale and congestion loss becomes an issue

Upcoming faster deterministic wireless (factory automation)

deterministic loops across networks and server OS

127

End-to-End time-synchronizationSynchronization via the backbone a plus

Standard time exchange, adapted precision.

Sync and schedule with the Deterministic backbone

MTU size a complex issue across networksIdeally harmonize MTUs across multilink subnet or

at least detect discrepancies

Link Local lookups should be emulatedProxy operation (per protocol)

Genralized hash based multicast