THI NA TO DA UN MATERIALETTI US009832085B1

( 12 ) United States Patent Malboubi

( 10 ) Patent No . : US 9 , 832 , 085 B1 ( 45 ) Date of Patent : Nov . 28 , 2017

( 54 ) SYSTEM FOR ESTIMATING UNKNOWN ATTRIBUTES OF INTEREST IN THE UNDER - DETERMINED INVERSE PROBLEM AND A PROCESS OF ACCOMPLISHING THE SAME

. . . . . . . . . . . . . . . 7 , 836 , 201 B2 * 11 / 2010 Kotrla . . . . . . . . . . . . . . . . H04L 41 / 0896

709 / 203 8 , 125 , 911 B2 * 2 / 2012 Patel H04L 12 / 5691

370 / 241 8 , 248 , 925 B2 * 8 / 2012 Allan . . . . . . . . . . . . . . . . . . . . . H04L 45 / 00

370 / 229 ( Continued ) ( 71 ) Applicant : Mehdi Malboubi , San Ramon , CA

( US ) OTHER PUBLICATIONS

( 72 ) Inventor : Mehdi Malboubi , San Ramon , CA ( US )

( * ) Notice : Subject to any disclaimer , the term of this patent is extended or adjusted under 35 U . S . C . 154 ( b ) by 129 days .

M . Malboubi . L . Wang , C . Chuah , and P . Sharma , “ Intelligent sdn based traffic ( de ) aggregation and measurement paradigm ( istamp ) , " IEEE INFOCOM , 2014 .

( Continued )

Primary Examiner — Guang Li ( 74 ) Attorney , Agent , or Firm — Law Office of Michael O ' Brien

( 21 ) Appl . No . : 15 / 002 , 376 ( 22 ) Filed : Jan . 20 , 2016

Related U . S . Application Data ( 60 ) Provisional application No . 62 / 105 , 437 , filed on Jan .

20 , 2015 . ( 57 ) ABSTRACT

( 51 ) Int . Ci . H04L 12 / 26 ( 2006 . 01 )

( 52 ) U . S . CI . CPC . . . . . . . H04L 43 / 045 ( 2013 . 01 ) ; H04L 43 / 0829

( 2013 . 01 ) ; H04L 43 / 0852 ( 2013 . 01 ) ; H04L 43 / 0888 ( 2013 . 01 )

( 58 ) Field of Classification Search ??? . . . . . . . . . . . . . . . . HO4L 45 / 38 USPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 / 252 See application file for complete search history .

A method for solving an under - determined inverse problem or network inference / tomography problem in per - flow size , delay , loss and throughput inference in a computer network , through a system is presented . The method includes the following steps , which are not necessarily in order . First , establishing the computer network having a plurality of nodes wherein the per - flow size , the delay , the loss and the throughput inference are unknown . An original observation or routing matrix determines how flows are appeared on the links and construct the measurements . Next , performing a learning phase to obtain an optimal observation matrix or pseudo - optimal observation matrix . After that , performing a computer controller adaptive measurement and inference phase to estimate the set of unknowns using the measure ment quantities , and a function of one of the set consisting of : the optimal observation matrix , the original observation matrix , or both .

( 56 ) References Cited

U . S . PATENT DOCUMENTS 6 , 785 , 240 B1 *

7 , 746 , 784 B2 *

8 / 2004 Cao . . . . . . . . . . . . . . . . . . . . . . H04L 43 / 045 370 / 238

6 / 2010 de Heer . . . . . . . . . . . . . . . . . . G06F 9 / 505 370 / 229 15 Claims , 17 Drawing Sheets

MEASUREMENT AND INFERENCE PHASE

MEASURING THE SET OF QUANTITIES WHICH ARE THE COMBINATIONS OF UNKNOWNS

DENTIFYING A PARTIAL SET OF MOST INFORMATIVE UNKNOWNS THAT MUST BE

RE - MEASURED AND UPDATED 544

546 RELEASURING A PARTIALSET

OF UNKNOWNS

548 — RE - COMPUTING AND UPDATING THE OPTIAL OBSERVATION MATRIXTO

FORMAN UPDATED OPTIMAL OBSERVATION MATRIX

ESTIMATING THE SET OF UNKNOWNS USING THE MEASUREMENT QUANTITIES , ANDA

FUNCTION OF ONE OF THE SET CONSISTING OF THE OPTIMAL OBSERVATION MATRIX THE ORIGINAL OBSERVATION MATRIX ,

OR BOTH

US 9 , 832 , 085 B1 Page 2

( 56 ) References Cited U . S . PATENT DOCUMENTS

. . . . . . . .

8 , 711 , 676 B2 * 4 / 2014 Bhat . . . . . . . . . . . . . . . . . . H04L 41 / 0668 370 / 216

9 , 083 , 627 B2 * 7 / 2015 Vasseur H04L 45 / 125 9 , 641 , 452 B2 * 5 / 2017 Lu . . . . . . HO4L 47 / 788 9 , 646 , 253 B2 * 5 / 2017 Kankar . . . . . . . . . . . . . . . . . . GOON 5 / 048

2007 / 0177506 A1 * 8 / 2007 Singer . . . . . . . . . . . . . . . . . . . . . HO4L 45 / 38 370 / 235

2012 / 0127029 A1 * 5 / 2012 Rachlin . . . . . . . . . . . . . . . . . . . G01S 19 / 40 342 / 357 . 23

2012 / 0191843 A1 * 7 / 2012 Ding . . . . . . . . . . . . . . . . . . . GO6F 11 / 3006 709 / 224

2013 / 0262661 A1 * 10 / 2013 Malboubi . . . . . . . . . . . H04L 41 / 142 709 / 224

A . Medina and et al . , “ Traffic matrix estimation : existing techniques and new directions , ” in ACM SIGCOMM , 2002 . R . M , Y . Zhang , W . Willinger , and L . Qiu , “ Spatio - temporal compressive sensing and internet traffic matrices , ” IEEE / ACM Transactions on Networking , vol . 20 , pp . 662 - 676 , 2012 . Q . Zhao , Z . Ge , J . Wang , and J . Xu , “ Robust traffic matrix estimation with imperfect information : Making use of multiple data sources , ” ACM - SIGMETRICS , 2006 . J . Cao , D . Davis , S . Wiel , and B . Yu , “ Time varying network tomography : Router link data , ” JNL of American Stat . Association , 2000 . G . Huang , A . Lall , C . - N . Chuah , and J . Xu , “ Uncovering global icebergs in distributed streams : Results and implications , ” J . Net work Syst . Manage , vol . 19 , No . 1 , pp . 84 - 110 , 2011 . A . Tootoonchian , M . Ghobadi , and Y . Ganjali , " Opentm : traffic matrix estimator for openfiow networks , ” 2010 . N . van Adrichen , C . Doerr , and F . Kuipers , “ OpenNet Mon : Network monitoring in openflow sdn networks , ” IEEE , INFOCOM , 2014 . M . Yu , L . Jose , and R . Miao . " Software defined traffic measurement with opensketch , " ACM - USENIX , 2013 L . Jose . M . Yu , and J . Rexford , “ Online measurement of large traffic aggregates on commodity switches , " ACM - Hot - ICE , 2011 . M . C . A . P . Lakhina . E . K . C . M . Diot , and N . Taft , “ Structural analysis of network traffic flows , " ACM Sigmetrics , 2004 . A . Israel and T . E Greville in Generalized Inverses : Theory and Applications , Springer , 2003 . H . Nguyen and P . Thiran , “ Network loss inference with second order statistics of end - to - end flows , " ACM - IMC , 2007 . M . Basseville and I . Nikiforov in Detection of Abrupt Changes : Theory and Application , Prentice - Hall , 1993 . “ Geant network : . ” http : / / totem . info . ucl . ac . be / dataset . html . “ Miniet , " At : http : / / mininet . org / .

OTHER PUBLICATIONS M . Moshref , M . Yu , R . Govindan , and A . Vandat , “ Dream : Dynamic resource allocation for software - defined measurement , ” ACM SIGCOMM ( HozSDN ) , 2014 . A . Soule , K . Salamatian , and N . Taft , “ Combining filtering and statistical methods for anomaly detection , " ACM SIGCOMM , 2005 . M . Roughan , M . Thorup , and Y . Zhang , “ Traffic engineering with estimated traffic matrices , " ACM - IMC , 2003 . “ Cisco netflow , ” At : http : / / en . wikipedia . org / wiki / Netflow . “ Sampled netflow , ” At : http : / / www . cisco . com / en / US / docs / ios / 12 \ 0s / feature / guide / 12s \ sanf . html . A . Kumar and J . Xu , “ Sketch guided sampling using on - line estimates of flow size for adaptive data collection , ” in INFOCOM , 2006 . M . Roughan , “ A case study of the accuracy of snmp measure ments , " JECE , vol . 2010 , pp . 33 : 1 - 33 : 7 , Jan . 2010 . * cited by examiner

nnnnnnnnnnnnaannnnnnnnnnnnnnnnnnnnnDDDDDDonnnnnnnnnnnnnnnnnnnnnn n nnnnnnnnnnnnnnnnnnnndDDDDDonnnnnnnnnnnnnnnnnnnnnnODDDDDannnnnnnnnnnnnnnnnnnnn

n nnnnnnnnnnnnnnnnnnnnnDDDDDonnnnnnnnnnnnnnnnnnnnnnooonnnnnnnnnnnnnnnnnnn

US 9 , 832 , 085 B1

FIG . 1

SINJOV WNS

WOO 8970000

29WW9019 98150

* *

*

80 60 00

Box 12 2

sex

081

#

FP

381

yung

? ?? .

»

8 .

* * * * * * * *

? ??? u c

? ? ?

« ?? ?

?

? ?? ?

? ?

?

? ?? ?? ?? ???

?

? ?

?

? ?? ?? ??

?

? wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww w

W

111 TTTTTTTFFFF

2

PACKETS

.

"

PACKETS

wwwww

1

OSP with

. COMWWWWWYYYYYYYYYYYYYY . . . . . OKWKWWXX

En

mange

www .

* * * * * * *

2

STATISTICS

FLOW MEASUREMENTS PREFIX KEYS OF PER - | PER - FLOW

.

be

2

* * * *

OUTPUT

o

o

INCOMING

com

Pony

440

* *

1 . La La La La

* R

* *

Sheet 1 of 17

YYYYYYYYYYYYYYY

po *

1

*

1

www

wwwwwwwwwwwwwwwwww wwwwwww

SOLSIMS L ABAXI add 379M1 MO1

F

a

g ta

OUR

wwwwwwwwwwwwwwwwwwwww

92 Phone

wwwwwwwwwwwwwwwwwwwwww w wwwwww

wwwwwwwww

wwwgoogo wher

yong

propinay

ho

093

881

marge de la sing

42

*

*

99 * 5 *

* * * * * * * ???????????????????? * *

? ????????????????????

GAAN

LEKKKKKKK

xxxxxxx

?

xxxxxx

??? ,

???????

? ???????????????????

? ??????Xx

?????

14 mm

Nov . 28 , 2017

& STATISTICS NETWORK MEASUREMENTS

SNMP MANAGEMENT SYSTEM

* We woon

ww

wewe

MESSAGES & CONTROL ( NMC ) NETWORK MEASUREMENT

ex u

.

OEEEEEEEEEEEEEEE

CONTROLLER

- - 01

* * * *

*

* *

* *

mm -

I

= 2K KK KK Kw w w w

B B B KRK www

* *

- *

w

*

*

-

* w

* *

- WW * * * * * *

w

-

w WA KX sx ww ww

KK KK KK

w w

w

-

-

zk KK KK kw

w

w

do

wwwwwwww wwwwwwww

wwwwwwwww wwwwwwwwwwwwwwww

wwwwwwwwwwwwwww Awwwwwww

Awwwwwwwwwwwww www

OPTIMAL - COHERENT INFERENCE ALGORITHN

ADAPTIVE PER - FLOW MEASUREMENT

-

atent

- -

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

m

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

=

n

2

-

=

H

x

u

u

-

1

U . S . Patent Nov . 28 , 2017 Sheet 2 of 17 US 9 , 832 , 085 B1

SNMP LINK - LOAD MEASUREMENT VECTOR YT - PEKKERKEKKAAAAAAA

mm wwwwwwwwww wwwwwwwwwwwwwwwwww wwwww EK O

O

yi

CHANGE DETECTION wwwwwwww W WWWW WY WYM - OOO OOO

- - wwwwwwwwwwwwwwwwwwwww

wwwmmmm wwwmm EEEEEEE D TA . . . . .

D

Y COHERENT INFERENCE Y

YYYY mc ADAPTIVE TM

UPDATING SNMP OOM

IDENTIFICATION . . . . . . EEEEEEEIIIIIIIII www wwwwwwwmmm R

1 K

ADAPTIVE PER - FLOW MEASUREMENT ( AFM )

OPTIMAL - COHERENT INFERENCE ALGORITHM ( OCIA )

FIG . 2

U . S . Patent Nov . 28 , 2017 Sheet 3 of 17 US 9 , 832 , 085 B1

mo

T

ON FIG . 3 MEASUREMENT & INFERENCE PHASE ( MIP ) COHERENCE TIME

-

-

-

THEMEN LEARNING PHASE

U , S , Patent _ Nov . 28 , 2017 Sheet 4 0f17 Us 9 , 832 , 085 B1

215

STORAGE CONTROLLER SYSTEM MEMORY

OUTPUT DEVICES

MEASUREMENT DEVICES PROCESSOR P { ;

DEVICES NETWORK ? 260 23 ' ]

FiG , 4

U . S . Patent

LEARNING PHASE

m

nnnnnnnnnnnnnnnnnnnnn

. . . . . . .

KKKKKKK

. . . . .

MEASURING A SET OF QUANTITIES WHICH ARE COMBINATIONS OF UNKNOWNS

nnnnnnnnnnnnnnnnnnnnn

TRANSMITTINGA VECTOR OF KNOWN SYMBOLS AT MULTIPLE TRANSMITTERS

- 5062

Nov . 28 , 2017

??????? ??????

. . . . . LEKKKKKK

504

MEASURING AT LEAST SOME OF A SET OF UNKNOWNS

home

5 / 17

. : :

MEASURING THE VECTOR OF KNOWN SYMBOLS AT MULTIPLE RECEIVERS ; AND THEREBY CREATING VECTORS OF MEASUREMENTS

- 5064

Sheet 5 of 17

- 506

COMPUTING AN OPTIMAL OBSERVATION MATRIX USING BOTH SETS OF THE ABOVE MEASUREMENTS

EEEEEETTTTTTTTTTTTTT

OLOREEEEE D

???????????????

m

COMPUTING THE OPTIMAL OBSERVATION MATRIX FOR EACH VECTOR OF KNOWN SYMBOLS AND SAVING THE VECTOR OF KNOWN SYMBOLS AND THE PLURAUTY OF OPTIMAL OBSERVATION MATRIXES AT A RECEIVER DEVICE

- 5066

. . .

KKKKKKK

w

wwwwww . . . . . . KKKKKKK w wwww SURRRRRRRUUUUUuuuuuuuu uuuuuuu9999 R

u wwww11100RRRRR

FIG . 5B

FIG . 5A

US 9 , 832 , 085 B1

U . S . Patent Nov . 28 , 2017 Sheet 6 of 17 US 9 , 832 , 085 B1

LEARNING PHASE

wwwwww www

CREATING A LINK LOAD VECTOR wwwwwwwwwwwww www W wwwwwwwww wwwwwwwwwwwwwwww

510 CREATING A VECTOR REPRESENTATION OF A TRAFFIC MATRIX BY MEASURING AND STORINGA SIZE OF A FLOW FROM EACH NODE TO

EVERY OTHER NODE FOR A SPECIFIED TIME PERIOD

wwww

2

FORMING AN OPTIMAL OBSERVATION MATRIX BY TAKING A MATRIX MULTIPLICATION OF THE MEASURED LINK LOAD VECTOR ,

A TRANSPOSE OF THE VECTOR REPRESENTATION OF THE MEASURED TRAFFIC MATRIX , AND A PSEUDO INVERSE OF A MATRIX

MULTIPLICATION OF THE VECTOR REPRESENTATION OF THE MEASURED TRAFFIC MATRIX AND THE TRANSPOSE OF THE

VECTOR REPRESENTATION OF THE MEASURED TRAFFIC MATRIX . KKKKKKKKKKKKKKANA . KKK ??KKKKKK??????? ??????????????????? ? ???

FIG . 5C

U . S . Patent

ch

wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww ww

wwwwwwwwwww

w

wwwwwwwwwwwwwwwww

n nnnnnnnnnnnnnnnnnnnn nnnnnnnnnnnnnnnnnnnnnn

MEASURING A LOAD ON A SET OF THE PLURALITY OF LINKS CREATING A PLURALITY OF LINK LOAD MEASUREMENTS WHERE LINK LOADS ARE LINEAR COMBINATIONS OF UNKNOWN NETWORK FLOWS

crime 5082

INSTALLING A SET OF RULES IN ONE OF A SET CONSISTING OF : A STORAGE SYSTEM , A MEMORY SYSTEM , OR BOTH OF ONE OF THE SET CONSISTING OF : NETWORK SWITCHES , NETWORK ROUTERS , OR BOTH

PWwwWwwW

- 5102

Nov . 28 , 2017

wwwwwwwwwwwwwwww wwwwwwwwwwwwwwwwww FORMING THE MEASURED LINK LOAD VECTOR FROM THE PLURAUTY OF LINK LOAD MEASUREMENTS

5084

POLLING COUNTERS IN THE STORAGE SYSTEM THAT CREATE FLOW - SIZE MEASUREMENTS AS FLOWS ARE MEASURED AND NETWORKED PACKETS ARE ROUTED

5104

Wwwwwwwwww

Sheet 7 of 17

nnnnnnnnnnn

wwwwww

WWW

. . . . . OKKKKKKKYYYYYYYYYYYYYY . . . . . .

O

W YYYYY

FYYYYYY

LEKKKKKKYYYYYYYYYYYYYY . . . . . LOKKKKKKYYYYYYYYYYYYYYY . . . . . KWKI

TRANSFERRING THE MEASURED LINK LOAD VECTOR TO A NETWORK CONTROLLER CONNECTED TO THE COMPUTER NETWORK

150861

TRANSFERRING THE MEASUREMENTS INTOA MEMORY , AND THE NETWORK CONTROLLER

9019 mm

www www

FIG . 5D

FIG . 5E

US 9 , 832 , 085 B1

U . S . Patent Nov . 28 , 2017 Sheet 8 of 17 US 9 , 832 , 085 B1

MEASUREMENT AND INFERENCE PHASE

??? ???? ?????? ????? ??? ??? ???? ????? ???

MEASURING THE SET OF QUANTITIES WHICH ARE THE COMBINATIONS OF UNKNOWNS 542 - PIPEREKIRI EKIPIPIL

PURURDUR RRURURDURURUQURURRERAVA

544 coram IDENTIFYINGAPARTIAL SET OF MOST

INFORMATIVE UNKNOWNS THAT MUST BE RE - MEASURED AND UPDATED

546 RE - MEASURING A PARTIAL SET

OF UNKNOWNS

548 RE - COMPUTING AND UPDATING THE OPTIMAL OBSERVATION MATRIX TO

FORMAN UPDATED OPTIMAL OBSERVATION MATRIX

wwwwwwwwwwwww wwwwwwwwwwwwwwww

ESTIMATING THE SET OF UNKNOWNS USING THE MEASUREMENT QUANTITIES , AND A

FUNCTION OF ONE OF THE SET CONSISTING OF : THE OPTIMAL OBSERVATION MATRIX , THE ORIGINAL OBSERVATION MATRIX ,

OR BOTH

MANAMAMAMA X XXXXX

FIG . 6A

U . S . Patent Nov . 28 , 2017 Sheet 9 of 17 US 9 , 832 , 085 B1

5420

TRANSMITTINGAN UNKNOWN VECTOR OF SYMBOLS FROM MULTIPLE TRANSMITTERS

- 5422

homomommmmmmmmmmmmmmmmmmmmmmmmmmmmm

CREATING A MEASUREMENT VECTOR FROM MULTIPLE MEASUREMENTS AT MULTIPLE RECEIVERS

ESTIMATING VECTORS OF SYMBOLS USING THE MEASUREMENT VECTOR AND THE OPTIMAL

OBSERVATION MATRIX CREATING ESTIMATED VECTORS OF SYMBOLS

5426

COMPUTINGA DISTANCE BETWEEN THE ESTIMATED VECTORS OF SYMBOLS , AND THE VECTORS OF KNOWN SYMBOLS

5428

DECODINGA MINIMUM DISTANCE VECTOR OF KNOWN SYMBOLS AS AN ULTIMATE ESTIMATE OF THE UNKNOWN VECTOR OF

SYMBOLS 6420 VU

nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn nnnnnnnnnnnnnn nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

FIG . 6B

U . S . Patent Nov . 28 , 2017 Sheet 10 of 17 US 9 , 832 , 085 B1

M

en MEASURING SNMP LINK LOADS AS MEASURED QUANTITIES WHICH ARE LINEAR COMBINATIONS

OF A SIZE OF UNKNOWN NETWORK FLOWS - 5502

MEASURING A SET OF UNKNOWN NETWORK FLOW SIZES USING COUNTERS ASSOCIATED WITH ONE OF

THE SET CONSISTING OF : FLOW - TABLE ENTRIES , TCAM ENTRIES OR BOTH

COMPUTING THE OPTIMAL OBSERVATION MATRIX ASA REPLACEMENT OF THE ORIGINAL ROUTING MATRIX

5506 wwwwwwwwwww

sro

INFERRINGA SET OF UNKNOWN FLOWS USING SPECIFIC ESTIMATION TECHNIQUES AND USING

SNMP LINK LOAD MEASUREMENTS , AND THE FUNCTION OF ONE OF THE SET CONSISTING OF THE OPTIMAL OBSERVATION MATRIX , THE ORIGINAL ROUTING

MATRIX , OR BOTH

FIG . 6C

atent Nov . 28 , 2017 Sheet 11 of 17 US 9 , 832 , 085 B1

ESTABLISHING AN ITERATION TIME 52 STARTING ATIMER 155 annoncen ooooo o oooooo

ro . . . . 9977322

mom 556 IF TIMER IS LESS THAN

ITERATION TIME ? sammarsementara STOP 574

YES CREATING AN ADJUSTED LINK LOAD VECTOR 1558

2222222 1 * * * LLLLLLLLS . . .

xxxx YYYYYYYYYYY Y YYYYYY

TRANSMITTING AND STORING THE LINK LOAD VECTOR TO THE CONTROLLER ARRANAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA A AAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAA

nnnnnnnnnnnnnn W WWWWWWWWWWWWW

562 YES DETECTING A SIGNIGIFANT Parametre

CHANGE IN THE MEASURMENT QUANITITIES ?

RESTARTING THE LEARNING PHASE 570

NO . . . KKKKKKKKK

DENTIFYA PARTIAL SET OF THE MOST INFORMATIVE FLOWS THAT MUST BE REMEASURED AND UPDATED

www w wwwwwwwwwwwww wwwwwwwwwwwwwwwww W wwwwwwwwwwwwwwwwww wwwwwwwwwwwwwwwww wwwwwwwwwwwwwwww w wwwwww

MEASURING AND STORING A PARTIAL SET OF THE MOST INFORMATIVE FLOWS BY INSTALLING AN ADJUSTED SET OF RULES AND POLLING ADJUSTED COUNTERS 1 - 576

nnnnnnnnnnnnnnnnn w wwwwwww WWWWWW W wwnnnnnnnn

TRANSMIT ADJUSTED MEASUREMENTS INTO THE CONTROLER AND MEMORY mm - 578 ??????? ???????? YYYYYY . . . . . YY KYYYYYYYYYYYYYY . . . . .

wwwwwwwwwwww FORMING AN ADJUSTED OPTIMAL OBSERVATION MATRIX BY TAKING A MATRIX MULTIPLICATION OF THE ADJUSTED LINK LOAD VECTOR , A TRANSPOSE OF THE ADJUSTED VECTOR REPRESENTATION OF THE TRAFFIC MATRIX AND A PSEUDO - INVERSE OF A MATRIX MULTIPLICATION OF THE ADJUSTED VECTOR REPRESENTATION OF THE TRAFFIC MATRIX AND THE TRANSPOSE OF THE ADJUSTED

VECTOR REPSENTATIONOF THE TRAAFFIC MATRIX

580

TO CREATING AN ESTIMATE OF THE VECTOR REPRESENTATION OF THE TRAFFIC MATRIX BY TAKING THE

MATRIC MULTIPLICATION OF THE PSEUDO - INVERSE OF THE ADJUSTED OPTIMAL OBSERVATION MATRIC AND THE ADJUSTED LINK LOAD VECTOR

wwwnnnnnnnnnnnnnnnn n nnnnnnnnnnnnnnn nom nnnnnnnnnnnnn nnnnnnnnnnnnnnnnnnnn n nnnnnnnnnnnnnnnnnnn mann

FIG . 6D

U . S . Patent

558

560

MEASURING A LOAD ON A SET OF THE PLURALITY OF UNKS CREATING A PLURAUTY OF LINK LOAD MEASUREMENTS WHERE LINK LOADS ARE LINEAR COMBINATIONS OF UNKNOWN NETWORK FLOWS

(

5582

INSTALLING A SET OF RULES IN ONE OF A SET CONSISTING OF : A STORAGE SYSTEM , A MEMORY SYSTEM , OR BOTH OF ONE OF THE SET CONSISTING OR NETWORK SWITCHES , NETWORK ROUTERS , OR BOTH

5602

Nov . 28 , 2017

WA MAMA

LLLLLLLLLLLLLLLL

LLLLLLLLLLLLLLLLLLLLLL

FORMING THE MEASURED LINK LOAD VECTOR FROM THE PLURALITY OF LINK LOAD MEASUREMENTS

rm5584

POLLING COUNTERS IN THE STORAGE SYSTEM THAT CREATE FLOW - SIZE MEASUREMENTS AS FLOWS ARE MEASURED AND NETWORK PACKETS ARE ROUTED

5604

Sheet 12 of 17

nnnnnnnnnnnnnnnnnnn

nnnnnnnnnnnnn

KRKA

- 5586

TRANSFERRING THE MEASURED LINK LOAD VECTOR TO A NETWORK CONTROLLER CONNECTED TO THE COMPUTER NETWORK

TRANSFERRING THE MEASUREMENTS INTOA MEMORY AND THE NETWORK CONTROLLER

- 5606

FIG . 6E

FIG . 6F

US 9 , 832 , 085 B1

atent Nov . 28 , 2017 Sheet 13 of 17 US 9 , 832 , 085 B1

Mann

INPUT X = 0 , L2 , La FOR 48 = 1 , 4 - 2 , measure yle and yt

2799mm

? ? ?????????????????????? KKKYYYYYYYYYYY

. COMPUTE THE ANGLE BETWEEN y? AND y?c ' BY B

wwwwwwwwwwwwwwwwwww syyich CALCULATING AT = Arccos AND STORE IT Well lyte * 112

wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww

NOF V - 5624 www t > Llp - 1

wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww

COMPUTE Hp IL Ept = 1 E 5626 uri = t - 1 + 1 . . RRRRRRR

AND ET LE 0 j = 1 - 44 - L5 + 1

Ret = sign ( 10g 1 , TP = Ret - Ret ? f E 1

- 5628 IF THERE IS

A RISE OR FALL BETWEEN - 1 and

+ 1 in Tet ?

NO

$ = $ + 1 INDENTIFY 16 * = 1 - 1 as Xth CHANGE TIME

KKKKKKKKXXXXXX KKKKKKKXXXXXXX

= + 1 ????????Y???? . . . OK

FIG . 7

U . S . Patent Nov . 28 , 2017 Sheet 14 of 17 US 9 , 832 , 085 B1

$ THIS FLOW CHART IS FOR ONE COHERENCE TIME FROM ( 48 + 1 ) 70 ( 48 * 21 )

INPUT CHANGE TIMES 10H + AND T # OF TCAM ENTRIES ) REQUIRED LINK - LOAD VECTORS y FOR , 8 + 1 < < tc8 + 1

L ang nnnnnnnnnnnnnn

USET ENTRIES OF TCAM TO MEASURE THE MOST CURRENT VECTOR REPRESENTATION OF TRAFFIC MATRIX XE NAOR MULTIPLE MEASURMENT INTERNALS

484 mc

FOR ANY GIVEN TIME I , WHERE 68 + 1 < t * t * * * FIND T ENTIES OF XL : INDICATED BY SET TOT , WHICH MAXIMIZES THE ANGLE BETWEEN Y? ANO yück - USE T ENTRIES OF TCAM TO MEASURE THE FLOWS , IDENTIFIED BY SET TOT , AT CURRENT TIME IAS XIOL - UPDATE THE ENTRIES OF Xme , IDENTIFIES by to BY NEW MEASURMENTS XEOL . THAT IS Xmoltot = X

486

FIG . 8

U . S . Patent Nov . 28 , 2017 Sheet 15 of 17 US 9 , 832 , 085 B1

THIS FLOW CHART IS FOR ONE COHERENCE TIME FROM ( 18 + 1 ) 70 ( 08791

INPUT : ROUTING MATRIX H 242 # OF ICAM ENTRIES LTX TOE

FOR ANY GIVEN TIME 1 , WHERE te T À E LU OXO

IMA 1 - CONSTRUCT pt = Xmc XL AND YEY ! Hi , To ' y xant L - 244 - CONSTRUCT Hol = yliput in tort ) * AND ESTIMATE * * IN THE SAME MANNER AS Wo = 4etty - CONSTRUCT WO ' AS Wot - v * uxembourg

MANNED MANERA

FIG . 9

DODO

US 9 , 832 , 085 B1

FIG . 10 DISCRETE - TIME ( 0 )

400

300

002 _

009 _

009

_

002 _

001 _

nnnnnn m

nnnnnnnnnnn

nnnnnnnnnnnn . .

ma nr . . . . . connnnnnnnnnnnnnnnn

Annnnnnnnnaaa

nnnnnnnnn

e & A

NMSE 080

Sheet 16 of 17

nnaangaan

ea

KK -

-

. w

w

KX KX

KKK w w

KK

- K

X

XX XX

K

*

w

ww

o WWWWW

> XXX

quod

KR )

K

- O

>

KK KK KK

-

w

KARU ww >

?? ?? .

NMSE ( T )

furter

opoggup

o

Nov . 28 , 2017

( 1 ) LISAN

y es SWN - - -

?KKKK

WaSHIN

( 0 ) ZSUN

e

pohy pro toto

wyspy

#

?????????????????????????????????????????????????????????????? u uw wew = 2

2

Roo m * * *

the * * * * * * . . they

got www

www .

x wwwww

W WY Yo than

* *

U . S . Patent

U . S . Patent

Frrrr . . 1111111TTTS

AALLLLLLLLLLL22 23

ALLLLLLLS . . . . . . 2 222

LLLLLLS . . . . . . 2 2 2 2 2

Dynam *

wanamth

NMSE , OR O

Nov . 28 , 2017

MEASUR

&

OUAC LIVIUR

.

w

* *

w

w

www

?????? ?

* * *

* * wa to Law

o

O ( 1 ) AND NMSELO

NMSE O

LIIIIII

.

Sheet 17 of 17

A

WOMEN

Hem p

: : 3 whom wer

EE

w

Yut

raguas

w

ww

: No wwmmwme

L

O

0

O

m

ennnnnnooooonnnnnnnnnn

A

nnonce comment Joo

o oooonnnnnn

g

400

300

DISCRETE - TIME ( 1 )

500

50

FIG . 11

US 9 , 832 , 085 B1

US 9 , 832 , 085 B1

SYSTEM FOR ESTIMATING UNKNOWN In some embodiments , the under - determined inverse ATTRIBUTES OF INTEREST IN THE problem is a traffic matrix estimation problem and a set of

UNDER - DETERMINED INVERSE PROBLEM unknown network flows are estimated by first , measuring AND A PROCESS OF ACCOMPLISHING THE SNMP link loads as measured quantities which are linear

SAME 5 combinations of a size of unknown network flows . Then , measuring a partial set of unknown network flow sizes using

RELATED APPLICATION counters associated with one of the set consisting of : flow table entries , TCAM entries or both . Next , computing the

This application claims priority to provisional patent optimal observation matrix as a replacement of the original application U . S . Ser . No . 62 / 105 , 437 filed on Jan . 20 , 2015 , 5 10 routing matrix . After that , inferring a set of unknown flows the entire contents of which is herein incorporated by using SNMP link load measurements , and using specific

estimation techniques that can include a function of one of reference . the set consisting of the optimal observation matrix , the BACKGROUND original routing matrix , or both .

15 A method for determining a size of network flows in a computer network through a system that can include the The embodiments herein relate generally to systems that following steps which are not necessarily in order . First , control flow in systems . providing a network having a plurality of nodes connected

Prior to embodiments of the disclosed invention there was by a plurality of links , wherein nodes denote network no efficient way to solve an under - determined inverse prob - 20 devices and / or elements including computer hosts , physical lem . As a result , there was no way to produce optimized switches , physical routers , virtual switches , and virtual flows in an underdetermined system . Embodiments of the routers . Next , performing a learning phase by iteratively disclosed invention solve this problem . performing the following steps until all link loads and a set

of unknown network flow sizes have been measured , and an SUMMARY 25 optimal observation matrix is computed . First , creating a

measured link load vector by performing , first , measuring a A method for solving an under - determined inverse prob - load on a set of the plurality of links creating a plurality of

lem or network inference / tomography problem in per - flow link load measurements where link loads are linear combi size , delay , loss and throughput inference in a computer nations of a set of unknown network flows . Then , forming network , through a system given an original observation 30 the measured link load vector from the plurality of link load matrix , includes the following steps , which are not neces - measurements . After that , transferring the measured link sarily in order . First , establishing the computer network load vector to a network controller connected to the com having a plurality of nodes wherein the per - flow size , the puter network . Following that , creating a vector represen delay , the loss and the throughput inferences are unknown ; tation of a measured traffic matrix by measuring and storing wherein an original routing matrix determines how flows 35 a size of a flow from each node to every other node for a from each node to every other node are appeared on links . specified time period , by performing the following steps . Next , performing a learning phase by first measuring a set of First , installing a set of rules in one of a set consisting of : a quantities which are combinations of unknowns creating a storage system , a memory system , or both of one of the set first set of measurements . Then , measuring at least some of consisting of : physical / virtual network switches , physical / a set of unknowns creating a second set of measurements . 40 virtual network routers , or both . Next , polling counters in the After that , using the first set of measurements and the second storage system that create flow - size measurements as flows set of measurements , an optimal observation matrix or are measured and network packets are routed . After that , pseudo - optimal observation matrix is computed . Following transferring the measurements into a memory , and the net that , performing a computer controller adaptive measure work controller . Following that , forming the optimal obser ment and inference phase by first measuring the set of 45 vation matrix by taking a matrix multiplication of the quantities which are the combinations of the unknowns measured link load vector , a transpose of the vector repre creating measurement quantities . Then , identifying a partial sentation of the measured traffic matrix , and a pseudo set of most informative unknown network flows that must be inverse of the matrix multiplication of the vector represen re - measured and updated . After that , re - measuring a partial tation of the measured traffic matrix and the transpose of the set of unknowns . Next , re - computing and updating the 50 vector representation of the measured traffic matrix . optimal observation matrix to form an updated optimal In some embodiments this method can include , perform observation matrix . Following that , estimating the set of ing an adaptive measurement and inference phase by itera unknowns using the measurement quantities , and a function tively and adaptively performing the following steps until of one of the set consisting of : the optimal observation adjusted link loads are measured , a set of most informative matrix , the original observation matrix , or both . 55 unknown network flows to be re - measured and updated is

In some embodiments the method can include determin - identified , the set of most informative unknown network ing a starting time of the learning phase and a duration of the flows are measured and the vector representation of a traffic learning phase by detecting sequential significant changes in matrix is updated to form an adjusted vector representation measurements during the adaptive measurement and infer - of the traffic matrix , an adjusted optimal observation matrix ence phase . Then , determining a set of most informative 60 is computed , and an unknown vector representation of the unknown network flows using at least one learning algo traffic matrix is estimated . The steps include : first , estab rithm ; wherein the at least one learning algorithm includes lishing an iteration time . Next , starting a timer . After that , for a multi - bandit algorithm . a timer value less than the iteration time , performing the

In some embodiments , the method can include generating following steps . First , creating an adjusted link load vector the optimal observation matrix with a regularized or non - 65 by performing the following steps . First , measuring an regularized optimization technique including a least square adjusted load on the set of the plurality of links creating a estimation technique with or without constraints . plurality of adjusted link load measurements . Next , forming

US 9 , 832 , 085 B1

the adjusted link load vector from the plurality of adjusted FIG . 3 shows a schematic view of one embodiment of the link load measurements . After that transferring the adjusted present invention ; link load vector to the network controller . Following that , FIG . 4 shows a schematic view of one embodiment of the creating the adjusted vector representation of the traffic present invention ; matrix by storing and updating an adjusted size of the flow 5 FIG . 5A shows a schematic view of one embodiment of from a set of nodes to another set of nodes identified by the the present invention ; set of most informative unknown network flows by identi - FIG . 5B shows a schematic view of one embodiment of fying the set of most informative unknown network flows the present invention ; that must be re - measured and updated . Then , installing an FIG . 5C shows a schematic view of one embodiment of adjusted set of rules , for measuring identified most infor ost in for 10 the present invention ; mative flows , in one of a set consisting of : the storage FIG . 5D shows a schematic view of one embodiment of

the present invention ; system , the memory system , or both of one of the set FIG . 5E shows a schematic view of one embodiment of consisting of : the network switches , the network routers , or the present invention ; both . In some embodiments , the network switches can be 15 15 FIG . 6A shows a schematic view of one embodiment of either physical switches , virtual switches or both . In some the present invention ; embodiments , the network routers can be either physical FIG . 6B shows a schematic view of one embodiment of routers , virtual routers or both . After that , polling adjusted the present invention ; counters in the one of a set consisting of : the storage system FIG . 6C shows a schematic view of one embodiment of and the memory system that create adjusted measurements 20 the present invention : of the most informative flows as adjusted flows are mea - FIG . 6D shows a schematic view of one embodiment of sured and the network packets are routed . Following that , the present invention ; transferring the adjusted measurements into the memory and FIG . 6E shows a schematic view of one embodiment of the network controller . Then , forming the adjusted vector the present invention ; representation of the traffic matrix by updating the size of the 25 FIG . 6F shows a schematic view of one embodiment of most informative flows . Following that , forming the the present invention ; adjusted optimal observation matrix by taking the matrix FIG . 7 shows a schematic view of one embodiment of the multiplication of the adjusted link load vector , the transpose present invention ; of the adjusted vector representation of the traffic matrix , FIG . 8 shows a schematic view of one embodiment of the and the pseudo inverse of the matrix multiplication of the 30 present invention ; adjusted vector representation of the traffic matrix and the FIG . 9 shows a schematic view of one embodiment of the transpose of the adjusted vector representation of the traffic present invention ; matrix . After that , creating an estimation of the vector FIG . 10 shows a graphical output of one embodiment of representation of the traffic matrix by taking the matrix the present invention ; multiplication of the pseudo inverse of the adjusted optimal 35 FIG . 11 shows a graphical output of one embodiment of observation matrix and the adjusted link load vector . the present invention ;

In some embodiments , this method can include , detecting a significant change in the measurements . Then , restarting DETAILED DESCRIPTION OF CERTAIN the learning phase . Finally , identifying the set of the most EMBODIMENTS informative flows by finding a set of flows which maximize 40 an angle between a current SNMP link - load vector and a By way of example , and referring to FIG . 1 , a controller SNMP link - load vector at a time of the significant change . 10 is communicatively coupled to an operating network 20 In some embodiments , the storage system can be one of the using a network communication system including network set consisting of : TCAM , flow - table entries , or both . The measurement controlling messages 14 . Operating network network controller can be connected to switches and routers 45 20 further comprises software defined network devices and on the network through one of the set consisting of : a secure non - software defined network devices . Network measure channel , a protocol , or both . ment controlling messages 14 can dynamically configure or

In some embodiments , an under - determined inverse prob reconfigure the software defined measurement network 12 as lem is an inverse problem wherein the number of measure - well as poll required network measurements and statistics ments is less than the number of unknowns that must be 50 16 . Operating network 20 and software defined measure estimated . An over - determined inverse problem is the ment network 12 further comprise a plurality of network inverse problem wherein the number of measurements is devices and elements including open flow switches 18 , and larger than the number of unknowns that must be estimated simple network management protocol ( SNMP ) agents . Con A square inverse problem is an inverse problem wherein the troller 10 , which can either be a physical component or a number of measurements is equal to the number of 55 virtualized controller , further comprises SNMP management unknowns that must be estimated . system 36 and network configuration protocol . Software

defined measurement network 12 further comprises a plu BRIEF DESCRIPTION OF THE FIGURES rality of open flow switches 18A , 18B , 18C , 18D , and 18E

in an operating network 20 . The plurality of open flow The detailed description of some embodiments of the 60 switches 18A , 18B , 18C , 18D , and 18E are either mainly or

invention is made below with reference to the accompanying partially for network traffic measurement and network figures , wherein like numerals represent corresponding parts packet forwarding and routing . In some embodiments , the of the figures . plurality of open flow switches can be one of the set

FIG . 1 shows a schematic view of one embodiment of the consisting of physical switches , virtual switches or both . present invention ; 65 Controller 10 further comprises an adaptive per flow mea

FIG . 2 shows a schematic view of one embodiment of the surement module 22 and an optimal - coherent inference present invention ; module 24 . The adaptive per flow measurement module 22

US 9 , 832 , 085 B1

indicates the set of most informative flows 44 in the oper FIG . 3 represents the behavior of a network flow over ating network 20 that can be directly measured . The optimal - multiple coherence times in the software defined measure coherent inference module 24 processes the measurements ment network 12 . Change time to indicates the time of the and statistics 16 and provides accurate estimates of network lth major change in the network traffic and the 1th coherence flows 42 . The controller 10 , then sends network measure - 5 time is the interval between 1th and ( 1 + 1 ) th major change ment controlling messages 14 to the software defined mea - times in network flows 42 . That is , T . ' = { te + 1 , . . . , t . 2 + 1 - 1 } . surement network 12 . In some embodiments , required per Controller 10 assumes that at the tth measurement interval flow measurements 32 and aggregated flow measurements 40 with measurement interval 40 , two sets of network 32 are measured by NetFlow protocol , sFlow protocol , and measurements are provided . First the SNMP management probing packets . system 36 interacts with SNMP agents 38 and provides the

The network measurement controlling messages 14 indi - SNMP link load measurements Y ' . Second , under hard cate which flow in the operating network 20 should be resource constraint of the size of ternary content addressable directly measured at different times or locations by re - memory 26 in flow tables of open flow switches 18A , 18B , configuring either a ternary content addressable memory 26 , 1 18C , 18E where the sum of all ternary content addressable a flow table or both . The ternary content addressable memories in switches T < < n , all T entries in ternary content memory 26 ( TCAM ) models the TCAM entries of the addressable memory 26 are used to track and measure all plurality of open flow switches 18A , 18B , 18C , 18D , and flows and / or the set of most informative flows 44A , 44B and 18E . The flexibility of open flow is used to install TCAM 44C and to provide T per flow measurements 32 . wildcard matching rules or prefix keys 28 and collect 20 Returning to FIG . 2 , adaptive per flow measurement associated statistics 30 . Associated statistics 30 can include module 22 comprises two methods : change detection per flow measurements 32 or aggregated measurement of method 46 and adaptive traffic matrix updating method 48 . multiple flows in each measurement interval 40 . The longest At a first measurement interval 40 in the beginning of the prefix matching forwarding table mechanism in open flow learning phase 50 , a first SNMP link - load vector Y ' c is switches 18 facilitates the implementation of a per - flow 25 created . At t " " measurement interval 40 , a second SNMP measurement process where prefix keys 28 of the per - flow link - load vector Y ' is created . Additionally , the SNMP link measurements 34 are stored in the ternary content address load vector Y ' is compared with the first SNMP link - load able memory 26 or flow tables and corresponding counters vector Yt? with change detection method 46 . The detection measure the size of flows . This process is called de - aggre - of the 1 " significant change in Y ' s indicates the beginning of gation . 30 a new coherence time and the beginning of a learning phase

In some embodiments , controller 10 further comprises a 50 . At the new coherence time to , a learning phase 50 SNMP management system 36 , alternately , SNMP manage - begins . In a learning phase 50 , all available T entries of the ment system 36 is communicatively coupled to controller ternary content addressable memory 26 in the software 10 . Network measurement controlling messages 14 are a defined measurement network 12 are used to measure the mean of the interaction of the Simple Network Management 35 most current traffic matrix 62 . In a computer controlled Protocol ( SNMP ) management system 36 and SNMP agents measurement and inference phase 54 , that is , over 1 " " coher 38 to measure SNMP link - loads . In some embodiments , ence time T . , the T most informative flows 44A , 44B and controller 10 and SNMP management system 36 are collo - 44C are directly measured and T flows 42 are updated in the cated in one machine and in other embodiments , they are most current traffic matrix 62 . located in separate machines and they can poll the statistic 40 Controller 10 further utilizes the optimal - coherent infer counts periodically or in different measurement intervals the ence module 24 . Optimal - coherent inference module 24 uses frequency of which is limited by practical constraints . the most current traffic matrix 62 and the SNMP link load Regardless , a focus is on traffic matrix 52 measurement and vector 56 to identify an optimal observation matrix 58 of inference where ternary content addressable memory 26 SNMP link load vectors 56 , denoted by H , . At each entries of all of the open flow switches 18 of the software 45 measurement interval 40 , the per flow measurements 32 and defined measurement network 12 is modeled as a large flow the SNMP link - loads are used to coherently estimate the table . traffic matrix 60 or X . ( equivalently W ) using the fol

FIG . 2 shows another view of operating network 20 . At lowing optimization formulation represented in the equation each measurement interval 40 or I there are m SNMP link below where to denotes the set of directly measured load measurements defined by the vector Ymut : = { y , ' } ; _ ; " 50 per - flow counts , and n = N ( N - 1 ) flows which form the vector representation of traffic matrix 52 , here Xnx1 * : = { x , f } ; = 1 " . The SNMP link load vector 56 or Y ' is linearly related to the traffic matrix 52 , XOAN here X ' as Y ' = HX ' where jth 1 in ith row of the binary routing matrix H represents the contribution of the jth flow x , in the 55 ith link load y ; " . If H is not binary , then jth value in ith row of denotes the set of flows that must be estimated , and matrix H represents the contribution of the jih attribute x in the ith quantity yi . The set of origin - destination flows in network exhibit a constant behavior in the sense that the Xi = XoUXO dynamic of network flows 42 are predictive over time 60 intervals with different durations , called as coherence time . The notion of coherence time is the time interval within which network flows 42 demonstrate , on average , a consis s . t . X or is known and Y ' = H , X ' and Xorn Xo = 0 tent and predictable behavior . Additionally , coherence time is defined as the time between the occurrences of two 65 consecutive major changes in the behavior of the traffic Under hard constraints of limited resources in software matrix 52 which is quantified by an appropriate metric . defined measurement network 12 this framework is simple ,

Si = 1

W = min | | 0 | | TOT

US 9 , 832 , 085 B1

10

flexible and efficient with very high estimation accuracy tubes ( CRT ) or liquid crystal displays ( LCD ) . Some embodi which can be deployed on the plurality of open flow ments include devices such as a touchscreen that functions switches 18A , 18B , 18C , 18D , and 18E . Moreover , the as both input and output devices . communication overhead between controller 10 , and the Bus 205 also couples electronic system 200 to a network plurality of open flow switches 18A , 18B , 18C , 18D , and 5 240 through a network adapter ( not shown ) . In this manner , 18E is low because the required measurements for the the computer can be a part of a communications network , or fine - grained estimation of network flows 42 are aggregated a network of computers ( such as a local area network SNMP link loads which are readily and reliably available in ( “ LAN ” ) , a wide area network ( “ WAN ” ) , or an intranet ) , or ISP networks . Further , only a few most beneficial flows are a network of networks ( such as the Internet ) . Any or all required to be directly measured . components of electronic system 200 may be used in con FIG . 4 conceptually illustrates an electronic system 200 with which some embodiments of the invention are imple junction with the invention .

Bus 205 couples electronic system 200 to controller 250 mented . The electronic system 200 may be a computer , and measurement devices 260 . In some embodiments con phone , PDA , or any other sort of electronic device . Such an

electronic system includes various types of computer read - 15 troll pread . 15 troller 250 can adaptively determine a set of required able media and interfaces for various other types of com quantities and unknowns of interest that must be measured puter readable media . Electronic system 200 includes a bus at important instances of time . Measurement devices 260 205 , processing unit ( s ) 210 , a system memory 215 , a read can be used to measure a set of required quantities and only 220 , a permanent storage device 225 , input devices unknowns of interest . 230 , output devices 235 , a network 240 , a controller 250 , 20 These functions described above can be implemented in and measurement devices 260 . digital electronic circuitry , in computer software , firmware

The bus 205 collectively represents all system , peripheral , or hardware . The techniques can be implemented using one and chipset buses that communicatively connect the numer or more computer program products . Programmable proces ous internal devices of the electronic system 200 . For sors and computers can be packaged or included in mobile instance , the bus 205 communicatively connects the pro - 25 devices or embedded systems . The processes may be per cessing unit ( s ) 210 with the read - only 220 , the system formed by one or more programmable processors and by one memory 215 , and the permanent storage device 225 . or more set of programmable logic circuitry . General and

From these various memory units , the processing unit ( s ) special purpose computing and storage devices can be 210 retrieves instructions to execute and data to process in interconnected through communication networks . order to execute the processes of the invention . The pro - 30 Some embodiments include electronic components , such cessing unit ( s ) may be a single processor or a multi - core as microprocessors , storage and memory that store computer processor in different embodiments . program instructions in a machine - readable or computer The read - only - memory ( ROM ) 220 stores static data and instructions that are needed by the processing unit ( s ) 210 readable medium ( alternatively referred to as computer and other modules of the electronic system . The permanent 35 " 25 readable storage media , machine - readable media , or storage device 225 , on the other hand , is a read - and - write machine - readable storage media ) . Some examples of such memory device . This device is a non - volatile memory unit computer - readable media include RAM , ROM , read - only that stores instructions and data even when the electronic compact discs ( CD - ROM ) , recordable compact discs ( CD system 200 is off . Some embodiments of the invention use R ) , rewritable compact discs ( CD - RW ) , read - only digital a mass - storage device ( such as a magnetic or optical disk 40 versatile discs ( e . g . , DVD - ROM , dual - layer DVD - ROM ) , a and its corresponding disk drive ) as the permanent storage variety of recordable / rewritable DVDs ( e . g . , DVD - RAM , device 225 . DVD - RW , DVD + RW , etc . ) , flash memory ( e . g . , SD cards ,

Other embodiments use a removable storage device ( such mini - SD cards , micro - SD cards , etc . ) , magnetic and / or solid as a floppy disk or a flash drive ) as the permanent storage state hard drives , read - only and recordable Blu - Ray® discs , device 225 . Like the permanent storage device 225 , the 45 ultra density optical discs , any other optical or magnetic system memory 215 is a read - and - write memory device . media , and floppy disks . The computer - readable media may However , unlike storage device 225 , the system memory store a computer program that is executable by at least one 215 is a volatile read - and - write memory , such as a random processing unit and includes sets of instructions for per access memory . The system memory 215 stores some of the forming various operations . Examples of computer pro instructions and data that the processor needs at runtime . In 50 grams or computer code include machine code , such as is some embodiments , the invention ' s processes are stored in produced by a compiler , and files including higher - level the system memory 215 , the permanent storage device 225 , code that are executed by a computer , an electronic com and / or the read - only 220 . For example , the various memory ponent , or a microprocessor using an interpreter . units include instructions for processing appearance altera Optimal - coherent inference module 24 contains the fol tions of displayable characters in accordance with some 55 lowing steps . First , at the 1th time change t . ' , the optimal embodiments . From these various memory units , the pro - observation matrix 58 of SNMP link load vectors 56 is cessing unit ( s ) 210 retrieves instructions to execute and data identified as H . . . This can be done by first pooling the to process in order to execute the processes of some embodi - SNMP link load vectors 56 at t = t , using SNMP management ments . system 36 and measuring the most current traffic matrix 62

The bus 205 also connects to the input and output devices 60 at tut . . Next , a minimum norms solution is computed using 230 and 235 . The input devices enable the person to com the optimal observation matrix 58 for t = ' + 1 : 1 : 1 2 + 1 – 1 . Both municate information and select commands to the electronic of these stages involve underdetermined inverse problems system . The input devices 230 include alphanumeric key because all mxn elements of the optimal observation matrix boards and pointing devices ( also called " cursor control 58 must be measured using m + n measurements of SNMP devices ” ) . The output devices 235 display images generated 65 link load vector 56 and traffic matrix 52 at t = and m + n < mx by the electronic system 200 . The output devices 235 n . In the second stage , m per - flow measurements 32 is less include printers and display devices , such as cathode ray than n flows 42 .

US 9 , 832 , 085 B1 10

In Stage 1 : 1 ) = argmin | | Y ' – H * X ' ll2 at = 1 hon = argmaxd? = hon

argmax ( hiX - hon X + ) ? for i € { 1 , . . . , m } and te { 1 , . . . , Tss } hon In Stage 2 : Wo = argmin | X¥ | l2 s . 1 . Yt = H * X * for 1 € = T ! 5

On

In Stage 3 : W = H * y where H ' * : = ( H : " ( HH : , 1 ) Here , h ; denotes the ith row of matrix H , X ' denotes the th vector representation of the traffic matrix 52 , and h , denotes a row of matrix Hom . This maximization can be performed ,

10 Where | indicates the pseudo - inverse operator , T indi in an iterative way , using the least square estimation

approach , including the Least Mean Squares ( LMS ) algo cates the complex conjugate transpose of the matrix , and rithm , defined in the equation below : Wodenotes the estimate of the vector representation of the traffic matrix 52 or X ' at time t . hon ( i + 1 ) = hon ( j ) – 2udxt "

The optimal observation matrix 58 in Stage 1 can be 15 Here , there is iteration over parameter j until convergence . X® is the complex conjugate transpose of X ' . u is set to a computed as H ' = Y ' X * ( X®X® ) * . In some embodiments , the small positive real number less than one . ho is normalized general form of optimal observation matrix 58 or H . can be by its norm | | ho | | at each iteration j . Accordingly , by design used as HG = ( Y * X * ( X + X + ) + QX ' ) where Q is an arbitrary ing m , rows , an optimal observation matrix 58 or Hon with

mxm matrix and A 18 the space of the rows perpendicular 20 size m , xn can be designed . A function of How , denoted by mxm matrix and X is the space of the rows perpendicular 20 size to X ' . The inverse or pseudo - inverse of a matrix can be H f . can be used to collect a new set of measurements computed using different techniques including the singular YM = HonX ' where X ' is the unknown vector representation value decomposition method . of the traffic matrix 52 for t > Tss . An estimate of Xf , can be

In the learning phase , Minimum norm estimations using inferred by computing the matrix multiplication of the Ht = YtX ( x x ) # has several properties . First , estimations 25 pseudo - inverse of Hon and Yn . are identifiable , that is , W = H9Yf = X . Second , measure This process can include the following steps of learning ments are compatible , that is , Y = H X + . Third , estimations phase 50 . First , measuring a set of vector of quantities ,

are noise immune because even if measurement vector Yt is which are combinations of unknowns , over several period of times and creating a first set of measurements . Next , mea noisy , then W = HQg1Y + = X ' . 30 suring a set of vector of unknowns over several period of

In some embodiments , Wo can be computed using both times and creating a second set of measurements . Finally , the original observation matrix H and the optimal observa computing one of the set consisting of an optimal observa tion matrix 58 or H . , including the minimum - norm estima tion matrix 506 and pseudo - optimal observation matrix by tion technique as follows : using the first set of measurements and the second set of

measurements , and the least square estimation approach , including the Least Mean Squares ( LMS ) algorithm .

FIG . 5A shows one embodiment of learning phase 50 in more detail . Learning phase 50 comprises the following steps , which are not necessarily in order . First , at step 502 ,

40 measuring a set of quantities which are combinations of unknowns creating a first set of measurements . Next , at step In some embodiments , optimal - coherent inference mod 504 , measuring at least some of a set of unknowns creating ule 24 can be alternately computed using regularized esti a second set of measurements . After that , at step 506 , using mation techniques including the minimization of the addi the first set of measurements and the second set of measure tion of | | Y4 - H * X | | and a fraction of | | H | | . The unknown in this wn in this 45 ments to form the optimal observation matrix 58 or pseudo minimization problem that must be estimated is H ' , and optimal observation matrix . In some embodiments , this can operator norm | | : | | is appropriately defined based on the be done with a regularized optimization technique including

application . a least square estimation technique with constraints . The In some embodiments , computing the optimal observation regularized optimization technique can include the least

matrix 506 can alternately include the following steps . 50 square estimation technique without the constraints . Alter Given a set of original observations , a set of new measure nately , this can be done with a non - regularized optimization ments is provided by designing the optimal observation technique including a least square estimation technique with matrix 506 which provides a set of most informative mea constraints . The non - regularized optimization technique can surements to the set of given observations . This set of new include the least square estimation technique without the

1 55 constraints . Additional combinations of these can exist . observations is complementary to the set of given original Turning to FIG . 5B , in some applications such as an observations and tries to minimize the estimation error as under - determined multiple - input multiple output ( MIMO ) much as possible . For the application in network inference , communication system , step 506 can be shown in more let Yss be a set of Tss vectors of original given link - loads Y detail . In the learning phase 50 , for all possible vectors of where Y ' is an mx1 vector for t = 1 , . . . , Tss . At each time 60 , . . : : : $ s . At each time 60 communication symbols , the optimal observation matrix 58 interval t , Y ' = HX ' where H is the original routing matrix can be computed as follows . First , at step 5062 , transmitting with size mxn , and X ' is the vector representation of the a vector of known symbols at multiple transmitters . Next , at traffic matrix 52 . X ' is assumed to be known over Tss periods step 5064 , measuring the vector of known symbols at of time t = 1 , . . . , Tss in the learning phase 50 . Then , a row multiple receivers , and thereby creating vectors of measure of the optimal observation matrix 58 or Hon with n columns 65 ments . After that , at step 5066 , computing the optimal can be designed by maximizing distance da ? , defined in the observation matrix 58 for each vector of known symbols equation below : creating a plurality of optimal observation matrixes 58 .

7 tr

* 6 = ( #

US 9 , 832 , 085 B1 12

Finally , saving the vector of known symbols and the plu mum distance vector of known symbols as an ultimate rality of optimal observation matrix 58 at a receiver device . estimate of the unknown vector of symbols .

Turning to FIG . 5C , learning phase 50 can be completed In FIG . 6C , at step 550 , estimating the set of unknowns by performing the following steps . First , at step 508 , creat using the measurement quantities , further includes the fol ing a measured link load vector . Then , at step 510 , creating 5 lowing steps which are not necessarily in order . First , at step a vector representation of a measured traffic matrix 52 by 5502 , measuring SNMP link loads as measured quantities measuring and storing a size of a flow from each node to which are linear combinations of a size of unknown network every other node for a specified time period . After that , at flows 42 . Then , at step 5504 , measuring a set of unknown step 512 , forming the optimal observation matrix 58 by network flow sizes 30 using counters associated with one of taking a matrix multiplication of the measured link load 10 the set consisting of : flow - table entries , TCAM entries or vector , a transpose of the vector representation of the mea - both . After that , at step 5506 , computing the optimal obser sured traffic matrix 52 , and a pseudo inverse of the matrix vation matrix 58 as a replacement of the original routing multiplication of the vector representation of the measured matrix . Following that , at step 5508 , a set of unknown flows traffic matrix 52 and the transpose of the vector representa - 15 are inferred using specific estimation techniques and using tion of the measured traffic matrix 52 . SNMP link load measurements , and the function of one of

Turning to FIG . 5D , in some embodiments , step 508 — the set consisting of the optimal observation matrix 58 , the creating a measured link load vector can be accomplished original routing matrix , or both . with the following steps , which are not necessarily in order . In FIG . 6D , one embodiment of computer controlled First , at step 5082 , measuring a load on a set of the plurality 20 measurement and inference phase 54 , includes the following of links creating a plurality of link load measurements where steps , which are not necessarily in order . First , at step 552 , link loads are linear combinations of unknown network establishing an iteration time . Next , at step 554 , starting a flows 42 . Next , at step 5084 , forming the measured link load timer . Following that , at step 556 , for a timer value less than vector from the plurality of link load measurements . After the iteration time , performing the following steps . If not , that , at step 5086 , transferring the measured link load vector 25 then stop the process at step 574 . to a network controller 10 that is connected to the computer If the process continues , at step 558 , creating an adjusted communications network . link load vector . Then , at step 560 , transmitting and storing

Turning to FIG . 5E , in some embodiments , step 510 the adjusted link load vector to the controller 10 . After that , creating a vector representation of a measured traffic matrix at step 562 , detecting a significant change in the measure 52 by measuring and storing a size of a flow from each node 30 ment quantities . If so , after that , at step 570 , restarting the to every other node for a specified time period can be learning phase 50 . accomplished with the following steps which are not nec If not , then , at step 572 , identifying a partial set of most essarily in order . First , at step 5012 , installing a set of rules in one of a set consisting of : a storage system , a memory informative flows 44 . Following that , at step 576 , measuring system , or both of one of the set consisting of : network 35 s and storing a partial set of most informative flows by switches , network routers , or both . After that , at step 5014 , installing an adjusted set of rules and polling counters . Next , polling counters in the storage system that create flow - size at step 578 , transmitting adjusted measurements to the measurements as flows are measured and network packets controller 10 and memory . After that at step 580 , forming the are routed . Following that , at step 5016 , transferring the adjusted optimal observation matrix 58 by taking the matrix measurements into a memory , and the network controller 10 . 40 multiplication of the adjusted link load vector , the transpose

Turning to FIG . 6A , in some embodiments , computer of the adjusted vector representation of the traffic matrix 52 , controlled adaptive measurement and inference phase 54 and the pseudo inverse of the matrix multiplication of the includes the following steps , which are not necessarily in adjusted vector representation of the traffic matrix 52 and the order . First , at step 542 , measuring the set of quantities transpose of the adjusted vector representation of the traffic which are the combinations of the unknowns creating mea - 45 matrix 52 . Following that , at step 582 , creating an estimation surement quantities . Next , at step 544 , identifying a partial of the vector representation of the traffic matrix 52 by taking set of most informative unknown network flows 42 that must the matrix multiplication of the pseudo inverse of the be re - measured and updated . Following that , at step 546 , adjusted optimal observation matrix 58 and the adjusted link re - measuring a partial set of unknowns . After that at step load vector . 548 , re - computing and updating the optimal observation 50 In FIG . 6E , step 558 , creating an adjusted link load vector , matrix 58 to form an updated optimal observation matrix 58 . can include the following steps , which are not necessarily in Then , at step 550 , estimating the set of unknowns using the order . First , at step 5582 measuring a load on a set of the measurement quantities , and a function of one of the set plurality of links creating a plurality of link load measure consisting of : the optimal observation matrix 58 , the original ments where link loads are linear combinations of unknown observation matrix , or both . 55 network flows 42 . After that , at step 5584 , forming the

In FIG . 6B , step 5420 , in some embodiments , the com - measured link load vector from the plurality of link load puter controlled measurement and inference phase 54 measurements . Following that , at step 5586 , transferring the includes the following steps , which are not necessarily in measured link load vector to a network controller 10 con order . First , at step 5422 , transmitting an unknown vector of nected to the computer network . symbols from multiple transmitters . Next , at step 5424 , 60 In FIG . 6F , step 585 , creating the adjusted vector repre creating a measurement vector from multiple measurements sentation of the traffic matrix 52 , can be accomplished with at multiple receivers . Following that , at step 5426 , estimat - the following steps , which are not necessarily in order . First , ing vectors of symbols using the measurement vector and at step 5822 installing a set of rules in one of a set consisting the optimal observation matrices creating estimated vectors of : a storage system , a memory system , or both of one of the of symbols . Then , at step 5428 , computing a distance 65 set consisting of : network switches , network routers , or between the estimated vectors of symbols , and the vectors of both . Next , at step 5824 , polling counters in the storage known symbols . After that , at step 5430 , decoding a mini system that create flow - size measurements as flows are

US 9 , 832 , 085 B1 14

10

0 = arccos s KIN

measured and network packets are routed . After that , at step entries of ternary content addressable memory 26 are used to 5826 transferring the measurements into a memory and the measure the Xme ( : = X " ) in one or more network controller 10 .

The adaptive per flow measurement module 22 has two main components , the change detection method 46 and 5 adaptive traffic matrix updating method 48 . The change in the network traffic is measured by computing the angle between two SNMP link - load vectors , defined in the equa measurement epochs ; in this case , to? = { 1 , . . . , n } and tion : TO ' l = n .

Input : number of TCAM entries T , 1th change time te ' , ( 1 + 1 ) th change time t ? + 1 at step 472

( Y " , y? ) , Output : At each epoch t , the updated traffic matrix 52 or L where I e T and 0 se * | | Y | 12 | | Y4 | | 2 X and TO Initialization : Use T entries of TCAM to measure

Xme " : = Xmeat step 484 Where Y ' and Yt : denote the SNMP link - load vectors at While t ' + 1stst 2 + 1 , do at step 486

the beginning of the 1th coherence time and at the current Find T entries of the updated traffic matrix 52 or Xmet time t , respectively . That is , e ' quantifies the deviation of two indicated by the set to? , which maximizes the angle SNMP link - load vectors at two different times Yt and Yto as 20 between Yt and Yté . an indication of a major change in the network traffics . Use T entries of the TCAM to measure the indicated flows Accordingly and to have a causal change detection method , at the current time t as Xcom at any time t , the average of O ' ' s over two periods of time , Xme * [ TO ] = XZO ' , which in MATLAB language means determined by two windows w , and w , " with lengths Ly and replacing all elements of Xme with indices in the set to Ly , is compared where w " : = { t - L , - L , + 1 , . . . , t - L , } and 25 by Xco . w , ' : = { t - L , + 1 , . . . , t } . The transition between two consecu - End while tive relative comparisons is reported as a change time te As shown in FIG . 9 , for t . ' + 1ststel + 1 the T flows 42 which Turning to FIG . 7 , The pseudo - code of this process step 562 , maximize 6 , are directly measured . These per flow measure detecting a significant change in the measurement quantities ments 34 have the most contribution in equalizing / coincid is as follows : 30 ing the two SNMP link - load vectors Y ' and Y . ; in this case ,

Input : 0 = { 0 } + 1 , w ' , w , ' at step 5622 TO? = { 11 , . . . . . , ir } where i , € { 1 , . . . , n } and TO?I = T . Output : The set of detected change times Sc : = { te Accordingly , at each epoch t , also , T elements of traffic

matrix 52 vector Xm are updated by the recently measured te ? , . . . } Initialization : 1 = 0 , S . = 0 flows 42 . The following process represents the main steps in

35 implementing the optimal - coherent inference module 24 in While t > L , + L , + 1 do at step 5624 a sample coherence time T ? Input : Routing matrix H , # of TCAM entries T , T . ' , Xme ' ,

TO ' at step 242 E ai and Output : W . for tet ?

i = t - Lj - Ly + 1 40 Fort from ( t . ' + 1 ) to ( t ? + 1 – 1 ) with step size + 1 , do at step 244

Y = Xme \ X . com and YT = Y - H ( : , TO? ) Xyou where operator : is the colon operator in MATLAB language and it means all rows , and H ( : , TO ) denotes a sub - matrix of H

If there is a rise or fall between - 1 and + 1 in T , ' at step 43 constructing from all rows of H with columns identified by the set to ' . 5628 then H = Y Y ( YU ) * and estimate Y * t in the same manner 1 = 1 + 1 , t ' = - 1 , S . * ( 1 ) = t ' at step 5630 as W . = ( H . 9 ' Y End if W t = * * UXO End while 50 End for This method is not based on any statistical assumption Here , to make the best use of the most - current Xme ' , first , about 0 , and thus , a more change detection method 46 can optimal - coherent inference module 24 cancels out the direct

be used to enhance its performance . Without loss of gener - per - flow statistics 52 . which have been accurately measured . ality and to simplify the current worst - case implementation and provide the required Y and Y for optimal - coherent of the change detection method 46 , 0 , is computed as the 55 inference module 24 . Then , the optimal observation matrix angle between Y , and Y , for te [ 2 , . . . , T . ] where To is the 58 of SNMP link - loads H . is computed . Finally , the mini duration of traffic in each data set . mum - norm solution * is estimated and traffic matrix 52 or

Turning to FIG . 8 , to adaptively update the most current W at time t is constructed . traffic matrix 62 or Xm , which is required for the optimal . In all algorithms , the matrix and vector manipulations are coherent inference module 24 , the flexibility of the software 60 match with the syntax of the commands of MATLAB defined measurement network 12 is used to allocate the language . limited available ternary content addressable memory 26 FIG . 8 and FIG . 9 represent the flowchart of algorithms in entries for measuring the most informative flows 44 that lth coherence time T from ( T + 1 ) to ( T 2 + 1 – 1 ) , that is , leads to the best possible estimation accuracy using the T ' = { t . ' + 1 , . . . , t + 1 - 1 } . optimal - coherent inference module 24 . The following 65 FIG . 10 and FIG . 11 show the effectiveness of embodi method summarizes the adaptive traffic matrix updating ments of the disclosed invention with real traffic traces from method 48 where by detecting the 1th change time to all T Geant network considered . FIG . 10 shows the performance

1 = Hot D , E , = I j = t - Lr + 1

Rs = sign ( 108 ) . 7 ) = R ' ; – RË ' at step 5626

15

mm

US 9 , 832 , 085 B1 16

of a non - adaptive case which is a special case of the adaptive embodiment , method , and examples , but by all embodi case with only one learning phase in the beginning . FIG . 11 ments and methods within the scope and spirit of the shows the performance of an adaptive case where there are invention as claimed . multiple learning phases at the important time of the detec tion of significant changes . What is claimed is :

The framework below provides different metrics for 1 . A method for solving an under - determined inverse evaluating the performance of the network controller 10 . problem or network inference / tomography problem in per

flow size , delay , loss and throughput inference in a computer network , through a system ; given an original observation matrix , the method comprising : NMSE = NMSE = 7Ž NMSE ( ) where NMSE ( ) : - | | X ’ – Willz NMSE ( t ) where NMSE ( 1 ) : = | | X¥ | l2 ( a ) establishing the computer network having a plurality

of nodes wherein the per - flow size , the delay , the loss and the throughput inferences are unknown ; wherein an

NMSE denotes normalized mean squared error and mea original routing matrix determines a contribution of sures the accuracy of flow size estimation technique . each flow size on each link - load ; Accordingly , NMSEmn denotes the error using the regular ( b ) performing a learning phase by : minimum norm estimation method where Xm ' = ' Y = ( HT ( 1 ) measuring a set of quantities which are combina ( HH ? ) - ) Y ' , and NMSE , denotes the estimation error using tions of unknowns creating a first set of measure the optimal - coherent inference module 24 . The NMSEm " , 20 ments ; NMSE , " denote the NMSE in the adaptive case , and NMS ( 2 ) measuring at least some of a set of unknowns Emna and NMSE , na denote the NMSE in non - adaptive creating a second set of measurements ; case . ( 3 ) using functions of the first set of measurements and As used in this application , the term “ a ” or “ an ” means “ at functions of the second set of measurements , com

least one " or " one or more . ” 25 puting an optimal observation matrix or pseudo As used in this application , the term “ about ” or “ approxi optimal observation matrix ;

mately ” refers to a range of values within plus or minus 10 % ( c ) performing a computer controlled adaptive measure of the specified number . ment and inference phase by : As used in this application , the term " substantially ” ( 1 ) measuring the set of quantities which are the

means that the actual value is within about 10 % of the actual 30 combinations of the unknowns creating measure ment quantities ; desired value , particularly within about 5 % of the actual

desired value and especially within about 1 % of the actual ( 2 ) identifying a partial set of most informative unknown network flows that must be re - measured desired value of any variable , element or limit set forth and updated ;

herein . ( 3 ) re - measuring a partial set of unknowns ; All references throughout this application , for example ( 4 ) re - computing and updating the optimal observation patent documents including issued or granted patents or matrix to form an updated optimal observation equivalents , patent application publications , and non - patent matrix ; literature documents or other source material , are hereby ( 5 ) estimating the set of unknowns using the measure incorporated by reference herein in their entireties , as though 40 ment quantities , and a function of one of the set individually incorporated by reference , to the extent each consisting of : the optimal observation matrix , the reference is at least partially not inconsistent with the original observation matrix , or both ; disclosure in the present application ( for example , a refer wherein , the under - determined inverse problem is a traffic ence that is partially inconsistent is incorporated by refer matrix estimation problem and a set of unknown net ence except for the partially inconsistent portion of the 45 work flows are estimated by : reference ) . a ) measuring SNMP link loads as measured quantities

A portion of the disclosure of this patent document which are linear combinations of a size of unknown contains material which is subject to copyright protection . network flows ; The copyright owner has no objection to the facsimile b ) measuring a set of unknown network flow sizes using reproduction by anyone of the patent document or the patent 30 counters associated with one of the set consisting of : disclosure , as it appears in the Patent and Trademark Office flow - table entries , TCAM entries or both ;

c ) computing the optimal observation matrix as a replace patent file or records , but otherwise reserves all copyright rights whatsoever . ment of the original routing matrix ; and

d ) inferring a set of unknown flows using specific esti Any element in a claim that does not explicitly state mation techniques and using SNMP link load measure " means for ” performing a specified function , or " step for ” ments , and the function of one of the set consisting of performing a specified function , is not to be interpreted as a the optimal observation matrix , the original routing “ means ” or “ step ” clause as specified in 35 U . S . C . $ 112 , 16 . matrix , or both . In particular , any use of “ step of " in the claims is not 2 . The method of claim 1 . further comprising : ( d ) deter intended to invoke the provision of 35 U . S . C . $ 112 , 116 . 60 mining a starting time of the learning phase and a duration

While the foregoing written description of the invention of the learning phase by detecting sequential significant enables one of ordinary skill to make and use what is changes in measurements during the computer controlled considered presently to be the best mode thereof , those of adaptive measurement and inference phase . ordinary skill will understand and appreciate the existence of 3 . The method of claim 1 , further comprising : ( f ) deter variations , combinations , and equivalents of the specific 65 mining the set of most informative unknown network flows embodiment , method , and examples herein . The invention using at least one learning algorithm ; wherein the at least should therefore not be limited by the above described one learning algorithm includes a multi - bandit algorithm .

35

US 9 , 832 , 085 B1 18

4 . The method of claim 1 , further comprising generating 13 . The method of claim 12 , wherein , the system is an the optimal observation matrix using one of the set consist - under - determined multi - input and multi - output communica ing of : tion system wherein :

( a ) a regularized optimization technique including a least a ) in the learning phase , for all possible vectors of square estimation technique with constraints ; and 5 communication symbols , the optimal observation ( b ) the regularized optimization technique including the matrix is computed as follows : least square estimation technique without the con 1 ) transmitting a vector of known symbols at multiple straints . transmitters ; 5 . The method of claim 1 , further comprising generating 2 ) measuring the vector of known symbols at multiple

the optimal observation matrix using one of the set consist - 10 receivers ; creating vectors of measurements ; ing of : 3 ) computing the optimal observation matrix for each ( a ) a non - regularized optimization technique including a vector of known symbols and a corresponding vector least square estimation technique with constraints ; and ( b ) the non - regularized optimization technique including of measurements , creating a plurality of optimal

the least square estimation technique without the con - 15 observation matrixes ; straints . 4 ) saving the vector of known symbols and the plurality

6 . The method of claim 1 , generating an estimation of of optimal observation matrixes at a receiver device ; unknowns using a specified estimation technique using one b ) in the computer controlled adaptive measurement and of the set consisting of : inference phase :

( a ) a regularized optimization technique including a least 20 1 ) transmitting an unknown vector of symbols from square estimation technique with constraints ; and multiple transmitters ;

( b ) the regularized optimization technique including the 2 ) creating a measurement vector from multiple mea least square estimation technique without the con surements at multiple receivers ; straints . 3 ) estimating vectors of symbols using the measure

7 . The method of claim 1 , generating an estimation of 25 ment vector and all optimal observation matrices unknowns using a specified estimation technique using one creating estimated vectors of symbols ; of the set consisting of : 4 ) computing a distance between the estimated vectors ( a ) a non - regularized optimization technique including a of symbols , and the vectors of known symbols ;

least square estimation technique with constraints ; and 5 ) decoding a minimum distance vector of known ( b ) the non - regularized optimization technique including 30 symbols as an ultimate estimate of the unknown the least square estimation technique without the con vector of symbols . straints . 14 . A system useful for displaying a solution to an 8 . The method of claim 1 , wherein , measurements of the

set of unknowns , in both learning phase and computer under - determined inverse problem in per - flow size , delay , controlled adaptive measurement and inference phases , are 35 10 25 loss and throughput inference in a computer network ; the approximated using : 1 ) mathematical , statistical , heuristic and experimental models , 2 ) auxiliary information , and 3 ) ( a ) the computer network having a plurality of nodes information from network topology , distance between where the per - flow size , the delay , the loss and through nodes , the size of queues , buffers in nodes , and any com put estimates inference are unknown ; bination of these . 40 ( b ) a computer store configured to store data for at least

9 . The method of claim 1 , wherein , a function of the two or more nodes ; optimal observation matrix is used for routing a set of ( c ) a display configured to display the per - flow size , the network flows . delay , the loss and throughput estimates in the com

10 . The method of claim 1 , wherein , in the learning phase , puter network ; several vectors of measurement quantities and unknown 45 ( d ) a computer controller connected to each of the plu attributes of interest are available , and the optimal observa rality of nodes , wherein the computer controller is tion matrix or pseudo - optimal observation matrix is calcu coupled to the computer store and programmed to : lated using a least mean squares algorithm ; wherein , in the ( 1 ) perform a learning phase by : computer controlled adaptive measurement and inference ( i ) measuring a set of quantities which are combi phase , a set of new measurements are provided using a 50 nations of unknowns creating a first set of mea function of the optimal observation matrix ; wherein the surements ; unknowns of interest is estimated using a specific estimation ( ii ) measuring at least some of a set of unknowns technique . creating a second set of measurements ;

11 . The method of claim 1 , ( iii ) using functions of the first set of measurements wherein , the under - determined inverse problem is con - 55 and functions of the second set of measurements ,

structed by converting one of the set consisting of : a and computing an optimal or pseudo - optimal square inverse problem , over - determined inverse prob observation matrix ; lem or both to an under - determined inverse problem , ( 2 ) perform an adaptive measurement and inference

wherein , the optimal observation matrix is computed and phase by : used for the estimation of unknowns of interest in one 60 ( i ) measuring the set of quantities which are the of the set consisting of : a square inverse problem , an combinations of the unknowns creating measure over - determined inverse problem or both . ment quantities ;

12 . The method of claim 1 , wherein , the under - determined ( ii ) identifying a partial set of most informative inverse problem is a system identification in the under unknown network flows that must be re - measured determined multi - input and multi - output systems , and a 65 and updated ; realization of parameters of a model of the system are ( iii ) re - measuring the partial set of most informative estimated by computing an optimal observation matrix . unknown network flows ;

19

by :

US 9 , 832 , 085 B1 20

( iv ) re - computing and updating an optimal observa ( ii ) measure at least some of a set of unknowns creating tion matrix to form an updated optimal observa a second set of measurements ; tion matrix ; ( iii ) use functions of the first set of measurements and

( v ) estimating the set of unknowns using the mea functions of the second set of measurements , and surement quantities , and a function of one of the 5 computing an optimal or pseudo - optimal observation set consisting of : the optimal observation matrix , matrix ; an original observation matrix , or both ; and ( 2 ) perform an adaptive measurement and inference phase ( 3 ) display the unknowns ;

wherein , the under - determined inverse problem is a traffic ( i ) measuring the set of quantities which are the com matrix estimation problem and a set of unknown net - 10 binations of the unknowns creating measurement work flows are estimated by : quantities ; a ) measuring SNMP link loads as measured quantities ( ii ) identifying a partial set of most informative which are linear combinations of a size of unknown network flows ; unknown network flows that must be re - measured

b ) measuring a set of unknown network flow sizes using 15 and updated ; counters associated with one of the set consisting of : ( iii ) re - measuring a partial set of most informative flow - table entries , TCAM entries or both ; unknowns ;

c ) computing the optimal observation matrix as a replace ( iv ) re - computing and updating an optimal observation ment of the original routing matrix ; and matrix to form an updated optimal observation

d ) inferring a set of unknown flows using specific esti - 20 matrix ; mation techniques and using SNMP link load measure ( v ) estimating the set of unknowns using the measure ments , and the function of one of the set consisting of ment quantities , and a function of one of the set the optimal observation matrix , the original routing consisting of : the optimal observation matrix , an matrix , or both . original observation matrix , or both ; and

15 . A computer - readable memory adapted for use by a 25 ( 3 ) display the unknowns ; computer network in determining a solution to an under wherein , the under - determined inverse problem is a traffic determined inverse problem in per - flow size , delay , and matrix estimation problem and a set of unknown net throughput inference , the computer - readable memory used work flows are estimated by : to direct a computer of the network to perform the steps of : a ) measuring SNMP link loads as measured quantities ( a ) the computer network having a plurality of nodes 30 which are linear combinations of a size of unknown where the per - flow size , the delay , the loss and through network flows ; put estimates inference are unknown ; b ) measuring a set of unknown network flow sizes using ( b ) a computer store configured to store data for at least counters associated with one of the set consisting of : two or more nodes ;

( c ) a display configured to display the per - flow size , the 35 flow - table entries , TCAM entries or both ; delay , the loss and throughput estimates in the com c ) computing the optimal observation matrix as a replace puter network ; ment of the original routing matrix ; and

( d ) a computer controller connected to each of the plu d ) inferring a set of unknown flows using specific esti rality of nodes , wherein the computer controller is mation techniques and using SNMP link load measure coupled to the computer store and programmed to : ments , and the function of one of the set consisting of 40

( 1 ) perform a learning phase by : the optimal observation matrix , the original routing ( i ) measure a set of quantities which are combinations matrix , or both . of unknowns creating a first set of measurements ; * * * * *