1 outline of contents ① issues, criteria and topics. ② motivations behind this line of research...

1

Outline of Contents① Issues, Criteria and Topics.

② Motivations behind This Line of Research

③ Survey of Typical Networks

④ My Research Work and Contributions

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

2

① Issues / Criteria / Topics

Topology and Analysis Routing and Communication Mapping and Simulation Algorithm and Computation VLSI Design and Construction

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

3


Topology and Analysis degree,diameter,average distance, bisection

bandwidth,connectivity,symmetry,recursiveness, scalability,Hamiltonian path

Routing and Communication Mapping and Simulation Algorithm and Computation VLSI Design and Construction

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

4


Topology and Analysis Routing and Communication

Pattern: routing,broadcasting,multicasting,gossip Evaluation:Easy routing,deadlock-free,delay,traffic

density, Control Strategy: centralized/distributed,

deterministic/adaptive,minimal/non-minimal, Switching: circuit/packet, wormhole, virtual cut-through

Mapping and Simulation Algorithm and Computation VLSI Design and Construction

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

5

Topology and Analysis Routing and Communication Mapping and Simulation

To compare the computing power G H Smallest possible dilation and congestion

Algorithm and Computation VLSI Design and Construction


embedding

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

6


1 2 3 4 5 6 7 8

1

8

2

7

3

6

4

5

1 2 3 4 5 6 7 8

12

3

45

6

7

8

Dilation=2, congestion=2

Dilation=7, congestion=2

How to embed a ring into a line:

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

7


Topology and Analysis Routing and Communication Mapping and Simulation Algorithm and Computation

PRAM model is unpractical. Basic algorithms:

sorting,searching,permutation,matrix multiplication,bit reversal, graph algorithm,iteration method,symbolic computing.

VLSI Design and Construction

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

8


Topology and Analysis Routing and Communication Mapping and Simulation Algorithm and Computation VLSI Design and Construction

Wafer-scale integration Layout design: area and wire length,wire

area,crossing number,node cost and modularity Applicable to the board design

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

9

② Motivations

A branch of combinatorics Structures for building multiprocessor

and multicomputer More than structural:

ASIC:implementing special parallel algorithm on VLSI/Wafer scale.

P2P overlay topologies Data Center Networking SoC or NoC for Many-core

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

10

② Big Names F. Thomson Leighton (Theory) http://theory.lcs.mit.edu/~ftl/ Lionel M. Ni (wormhole) http://www.cps.msu.edu/~ni/ Arnold L. Rosenberg (Butterfly) http://www.cs.umass.edu/~rsnbrg/ Burkhard Monien (Embedding) http://www.uni-paderborn.de/fachbereich/AG/monien/ Laxmi N. Bhuyan (Multi-stage) http://www.cs.tamu.edu/faculty/bhuyan/ Kenneth E. Batcher (Bitonic) http://nimitz.mcs.kent.edu/~batcher/index.html Ivan Stojmenovic (Honeycomb) http://www.csi.uottawa.ca/~ivan/ Ke Qiu (Star and Pancake) http://dragon.acadiau.ca/~kqiu/home.html Wei Zhao (Routing) http://www.cs.tamu.edu/faculty/zhao/ S. Lennart Johnsson (Hypercube) http://www.cs.uh.edu/~johnsson/ Satoshi Fujita(gossip) http://www.se.hiroshima-u.ac.jp/~fujita/ William J. Dally (k-ary n-cube) http://www.ai.mit.edu/people/billd/ S. Yalamanchili (Engineering ct) http://users.ece.gatech.edu/~sudha/ C.E. Leiserson (Parallel Algorithms) http://supertech.lcs.mit.edu/~cel/ Kai Hwang (Benchmarking) http://ceng.usc.edu/~kaihwang/

http://theory.lcs.mit.edu/~ftl/

http://www.cps.msu.edu/~ni/



Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

11

② A Special Journal on Interconnection Networks

D Frank Hsu (Fordham University) Bruce M Maggs (Carnegie Mellon )Jean-Claude Bermond (CNRS/INRIA/UNSA)Tse-Yun Feng (Penn State University)Yoji Kajitani (Tokyo Institute of Tech)F Tom Leighton (MIT)Guo-Jie Li (Chinese Academy of Sciences)Burkhard Monien (University of Paderborn)Howard Jay Siegel (Purdue University)Tom Stern (Columbia University)

Among the editorial board:

Website: http://journals.worldscinet.com/join

12

③ Survey of Network Topology

The following discussion of the properties of interconnection networks is based on a collection of nodes that communicate via links. In an actual system the nodes can be either processors, memories, or switches. Two nodes are neighbors if there is a link connecting them.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

13

③ Network Parameters

Channel width: Number of wires that are used per channel (i.e. the number of bits

that can be transmitted simultaneously on one channel).

Channel direction: The direction in which the messages can be transmitted. Unidirectional : can send messages in just one direction Bi-directional : support two-way communication over the same

channel.

Bisectional width (BW): It is defined as the number of channels that are cut when the

network is divided into two equal parts.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

14


Node degree (): Number of channels connecting a node to its neighbors. In practice the degree of a topology has an effect on cost, since the

more links a node has the more logic it takes to implement the connections.

Network diameter (D): Maximum distance between any two nodes in the network.

Cost Effectiveness: x D, Problem of Dense graph: Given the fixed degree and diameter, how

to design a graph which can contain as many nodes as possible? Average distance:

= Vji n

jid

,2

),(

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

15


Number of links (l): The total number of channels in the network.

Symmetry: A network is said to be symmetric if it looks the same

from every node. (E.g. Hypercube, crossbar) Recursiveness

A large-size network consists of two or many small-size networks of the same kind, such as tree and hypercube. This property renders the network suitable for solving divide-and-conquer algorithm

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

16

③ Crossbar switch

P0 –> M1P1 –> M3P2 –> M2P3 –> M0

Crossbar switch

N*N switches (N processors N memory modules). The switch configures itself dynamically to connect a processor to a memory

module. No contention -- Supports N! permutations. Costly, hard to scale, wastes switches for most patterns.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

17

③ Crossbar switch

Easy for broadcasting and also for any permutation. As long as each processor wants to communicate with a different memory there will be no contention -- Supports N!

permutations. If two or more processors need to access the same

memory, however, one will be blocked until the switch reconfigures itself.

A crossbar has a short diameter - information needs to pass through only one switching element on a path from one edge to another.

Poor scalability -- If there are N processors and N memories, there are N2 interior switches. Adding another processor or memory means adding another N interior nodes.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

18

③ Multistage Network

P1

P2

P3

PP-1

P1

P2

P3

PP-1

Stag

e 1

Stag

e 2

Stag

e 3

Stag

e k

Systems built with these topologies have processors on one edge of the network, memories or processors on another edge, and a series of switching elements at the interior nodes.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

19

③ Multistage Network : Omega Network

p processors and log2 p stages Each stage consists of a perfect shuffle

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

20

③ Multistage Network : Butterfly

ButterFly Network

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

21

③ Multistage Network

Built from small (e.g., 2X2 crossbar) switch nodes, with a regular interconnection pattern.

In order to send information from one edge to another, the interior switches are configured to form a path that connects nodes on the edges.

The information then goes from the sending node, through one or more switches, and out to the receiving node.

The size and number of interior nodes contributes to the path length for each communication, and there is often a ``setup time'' involved when a message arrives at an interior node and the switch decides how to configure itself in order to pass the message through.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

22

③ Multistage Network: Problems

Generalized MINs: for in Inputs and jn outputs, use i×j switches at n stages.

Bens networks: with 2logn-1 stages, it becomes a nonblocking network, allowing all permutations.

Fault tolerance: form all switches at a stage as a ring.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

23

③ Line, Ring and Fully Connected

Linear Array: the simplest topology. Fully Connected: Direct connection between every pair of processors, Highest cost, Similar to crossbar in some properties. Ring: Each node in the ring is connected to only two other nodes.Chordal Ring: A compromise between Ring and Fully Connected Network.

Fully connected: Degree = N-1, Diameter = 1, BW = (N/2)2, Links = N*(N-1)/2, Symmetric. Ring: Diameter = N/2, Degree=2, BW = 2, Symmetric

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

24

③ Mesh and Torus

Mesh: Deg = 2,3,4, Diameter = 2*Sqrt(N), Bisect= Sqrt(N), Easy to build, scalable .k-ary n-cube: Generalization of mesh-like Networks

Mesh Topology 3D Torus2D Torus

2D torus: Meshes with ``wraparound'' connections, e.g. the node at the top of the grid has an ``up'' link that connects to the node at the bottom of the grid (also left to right).

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

25

③ Mesh and Torus: Problems

Diameter of Asymmetric Networks: linear array, mesh, tree, shuffle exchange.

Torus and Midmiew: how to use the wraparound links to obtain the optimal diameter?

Layout of torus: Reduce the physical wire length.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

26

③ Hypercube

Can embed Hamiltonian cycle, mesh, tree, etc. Low latency, high bandwidth, but costly (high number of links). Hard to build (layout the chip and wires). Hard to scale up (As degree increases, number of I/O ports increases). Solution: CCC

For dimension D, Degree = D, Diameter = D, Bisect = 2(d-1) Nodes = 2D, Links = D*2(D-1)

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

27

③ Hypercube : assign node IDs

The nodes are numbered so that two nodes are adjacent if and only if the binary representations of their IDs differ by one bit. For example, nodes 011 and 010 are immediate neighbors.

011

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

28

③ Hypercube : Properties

Hamiltonian Cycle 2n-node hypercube contains I×J mesh, where I×J = 2n

Contains 2 binary tree with 2n-1 － 1 nodes Optimal in fault-tolerance Variants of hypercube:folded hypercube, hierarchical

hypercube,incomplete hypercube,CCC, shuffle-change. Y. Saad and M.H. Schultz, Topological Properties of

Hypercube, IEEE TC, July 1988. Generalized Hypercube, IEEE TC, April 1984.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

29

③ CCC: Solution to Hypercube :

F.P. Preparata, et al, The Cube-Connected Cycles: A Versatile Network for Parallel Computation, CACM, May 1981,G. Chen, et al, Tight Layouts of CCC, IEEE TPDS, Feb. 2000.G. Chen, et at, Layout of CCC without Long Wires, Computer Journal, in 2005.

000 001

100

110 111

011010

101

(000,0)

(000,2) (000,

1)

1

2

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

30

③ Tree and Star

Tree and Star

Tree: Degree = 1 (leaves), 2 (root), 3 (interior nodes), Diameter = 2logN, Bisect = 1. Tree/Star bottleneck: Expect at root

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

31

③ Tree

The tree has nodes of degree 1 (leaves), 2 (root), 3 (interior nodes). So it is not symmetric.

Short diameter: For depth k, the number of nodes is 2k - 1, and the diameter is

2k (or ~ 2 log N, at the same order as Hypercube). For example, a processor with 262,144 nodes would have

diameter 512 in a mesh but only 36 in a tree.

Bisection bandwidth is 1. It suffers from a serious bottleneck

-- Solution : Fat Tree

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

32

③ Fat Tree

Fat-tree layout

Fat-Tree

Expand link bandwidth at each higher level

BB

BB/8

B/4

B/2

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

33

③ Fat Tree

Leiserson's fat-trees [Lei85] – built in CM-5. The Fat Tree network provides uniform bandwidth between

any two end-points on a net. It does this by doubling the number of "up" paths as one goes "up" the tree from a processor in a leaf to the root node.

Given a fat tree of this type with h levels of switches, Number of processor nodes = 2h

Number of switch nodes = 2h- 1 Greatest distance between processor nodes = 2h

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

34

③ Packet Routing in Fat Tree

Note that a message going from PE2 to PE5 may choose any one of four paths from the lower left router to the one at the root. This means that all four PE's attached to the lower left router have a path available to them to reach another node.

4 port

8 port

PE2 PE5

four paths

PE1 PE3 PE10PE9PE8PE7PE6 PE16PE14PE13PE12PE11PE4 PE15

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

35

③ Comparison -- (N node, n=log 2 (N))

Degree Diameter Bisection No. of Links(d) (D) (B) (L)

1D mesh 2 N-1 1 N-1

2D mesh 4 2(N1/2 - 1) N1/2 2N- N1/2

star N-1 2 2 N-1

Ring 2 N / 2 2 N

2D torus 4 2(N1/2 / 2) N1/2 2N

Hypercube n n N/2 n N/2

Complete connected N-1 1 (N/2)2 N(N-1)/2

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

36

③ Example MPP Networks

Name Number Topology Bits Clock Link Bisect. YearnCube/ten 1-1024 10-cube 1 10 MHz 1.2 640 1987iPSC/2 16-128 7-cube 1 16 MHz 2 345 1988MP-1216 32-512 2D Mesh 1 25 MHz 3 1,300 1989Delta 540 2D Mesh 16 40 MHz 40 640 1991CM-5 32-2048 fat tree 4 40 MHz 20 10,240 1991CS-2 32-1024 fat tree 8 70 MHz 50 50,000 1992Paragon 4-1024 2D mesh 16 100 MHz 200 6,400 1992T3D 16-1024 3D Torus 16 150 MHz 300 19,200 1993

No standard MPP topology!

MBytes/second

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

37

③ Scalable Topology

Scalability Refers to the increase in the complexity of

communication (time) as more nodes are added. In a highly scalable topology more nodes can be added

without severely increasing the amount of logic required to implement the topology and without increasing the diameter.

Example : Doubling the number of nodes in a hypercube increases the degree by only 1 link per node, and likewise increases the diameter by only 1 path. An opposite example is linear array.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

38

③ Hypercube Problems

Embedding the links: For a 64K node (d=16) hypercube machine, there would be

512K (16x64K/2) links. With current technology, it is difficult to scale a hypercube

beyond about 4K nodes, with about 24K links.

Operation cost: As the number of dimensions increases, the nodes must do

more work to keep up with the incident message traffic. In fact, because most processors can handle only one I/O

transaction at a time, many hypercube algorithms operate on the principle of serializing processing by dimension. i.e. all pairs of nodes in one dimension communicate, then the next dimension, etc. See Ascend/Descend Algorithms

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

39

③ Symmetric Topology

Symmetric Rings, fully connected networks, and hypercubes

are all node symmetric. Trees and stars are not. A tree has three different

types of nodes, namely a root node, interior nodes, and leaf nodes, each with a different degree. A star has a distinguished node in the center which is connected to every other node.

When a topology is node asymmetric a distinguished node can become a communications bottleneck.

Importance of Symmetry: Node symmetry renders identical software at every node;edge symmetry avoids hot traffic spot in the network.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

40

Routing algorithms—XY Routing

P Sx,Sy source processor

P Dx,Dy destination processor

Shortest path – “manhattan distance”, abs(Sx-Dx) + abs(Sy-Dy)

Algorithm: 1.Reduce distance along X

dimension until 0 2.Reduce distance along Y

dimension until at P Dy,Dx

P 0,0 :source

P 3,2 :destination

P 3,2

P 0,0

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

41

③ Routing algorithms—XY Routing

Begin receive(M);If D=I accept(M)Elseif D • x >I • x

sendright(M) Elseif D • x <I • x sendleft(M)Elseif D • y >I • y

senddown(M)Elseif D • y >I • y sendup(M)Endif;

end

In a mesh, S I D More Information

M[S,D] M[S,D]

S(0,0)

D(3,2)

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

42

E-Cube Routing

E-Cube RoutingPs source processor Pd destination processor

Shortest path -- hamming distance between Ps and Pd

Routing taken in the following manner: 1. Exclusive-or the source and

destination processor numbers 2. Going from the Least

Significant Bit (lsb) to Most Significant Bit (MSB). Each position a “1” exists in the result of the exclusive, an edge is taken.

P000 source P110 destination

110

43

④ My Work and contributions Novel Interconnection Networks:

Shuffle-Ring, Shuffle-Cube, Wall Mesh.

Layout Design: MIDIMEW, CCC.

Routing in special Networks:

Shuffle-Exchange, Unidirectional Networks

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

44

④ New Topology: Shuffle-Ring

Definition: anan-1…a1a0 is connected to

shuffle an-1an-2…a0an

ring anan-1…a1 ( a0 ±1 ）

Properties:

Simplicity

Constant degree

Keep all hypercube properties

Compact layout

•G. Chen et al, Shuffle-Ring: Overcoming the Increasing Degree of Hypercube, Proc. 2nd Int. Symp. on High-Performance Computer Architecture(HPCA-2), California, Feb. 1996, 130-138.

•G. Chen el al, Shuffle-Ring: A New Constant-degree Network, International Journal of Foundations of Computer Science, March 1998, 77-98

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

45

④ New Topology: Shuffle-Cube

Properties:

Constant degree

Keep all hypercube properties

Compact layout

•G. Chen, et al, CTSN: A New Fault-Tolerant Network, Proceedings of 13th International Conference on Parallel and Distributed Computing Systems(PDCS'2000), Las Vegas, Nevada, August 2000, 517-522.

0 1

2 3

4 5

6 7

c d

e f

8 9

a b



Cube anan-1…ai … a0 (i<k)

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

46

④ New Topology: Wall Mesh

Definition: ( x, y ) is connected to

( x ±1, y ) horizontally

and ( x, y ±1 ) vertically.

Properties:

Constant degree=3 or 4,

Equivalent to mesh in Computing Power

Logarithmic diameter

Easy layout

•G. Chen,et al, The Wall Mesh, Computer Architecture'97: Selected Papers of the 2nd Australasian Conference, Springer, 1997, 217-230.

•陈贵海等，墙式网孔，计算机学报， 2000年 4 月， 374-381 页。

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

47

④ New Topology: Isomorphism

•G. Chen et al, Comments on ``A New Family of Cayley Graph Interconnection Networks of Constant Degree Four'', IEEE Transactions on Parallel and Distributed Systems, December 1997, 1299-1300.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

48

④ Layout Design: MIDIMEW

•G. Chen, et al, Laying Out Midimew Networks with Constant Dilation, Lecture Notes in Computer Science(854), September 1994, 773-784.

•G. Chen et al, Optimal Layouts of Midimew Networks, IEEE Transactions on Parallel and Distributed Systems, September 1996, 954-961.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

49

④ Layout Design: Cube-Connected Cycles

G. Chen, et al, A Compact Layout of Cube-Connected Cycles, Proc. of 4th Int. Conf. on High Performance Computing, India, Dec. 1997, 422-427. G. Chen, et al, Tight Layouts of CCC, IEEE Transactions on Parallel and Distributed Systems, Feb. 2000, 182-191.G. Chen, et at, Layout of CCC without Long Wires, Computer Journal, Nov. 2001, Vol. 44, No. 5.

000 001

100

110 111

011010

101

(000,0)

(000,2) (000,

1)

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

50

④ Routing: Shuffle-Exchange Networks

•G. Chen, et al, Shortest-Path Routing in Shuffle-Exchange Networks, Lectures in Operations Research, August 1998, 142-153. •陈贵海等，洗牌交换网中的最短路由算法，计算机学报， 2001 年 1 月。•G. Chen, et al, An Algorithm for Optimal Routing in Shuffle-Exchange Networks, to appear in IEEE Transactions on Computers.



Exchange an-1an-2…a1a0

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

51

④ Routing: Unidirectional Networks

G. Chen, at al, A Distance-Vector Routing Protocol for Networks with Unidirectional Links, Computer Communications, Feb. 2000, 418-424. •G. Chen, et al, An Improved Routing Protocol for Networks with Unidirectional Links, ICPP, Spain, Sept. 2001.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

Questions?

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

53

Assignment 11) Prove any graph/network has a longest path?

2) Choose one of the following

2.1) Write the routing algorithm for complete binary tree?

(Suppose nodes are numbered from top to down, from left to right, and beginning from root.)

2.2) Prove that hypercube has optimal fault tolerance.

Send your solutions to TA via Email before Nov.17, 2012.

Sh

an

gh

ai Jia

o T

on

g U

niv

ersity

20

11

54

Reading materials 1) Y. Saad and M. H. Schultz, Topological Properties of Hypbercube, IEEE

Transactions on Computers ， Vol. 24, No. 5

2) L.N. Bhuyan and D.P. Agrawal, Generalized Hypercube and Hyperbus Structures for a Computer Network, IEEE Transactions on Computers, Vol. 33, No. 5, 1984

3) C. E. Lerserson, Fat-tree: Universal network for Hardware-Efficient Supercomputing, IEEE Transactions on Computers, pp. 892-901, 1994

1 outline of contents ① issues, criteria and topics. ② motivations behind this line of research...

Documents