Networking @ BCCStefan WahlNiels BakkerElisa Jasinska

Maxim Salomon

Planned Network

Realized Network

Detailed Network G60

Alboinkontor(Speedbone)

Lützowstrasse 105/106

BCC

BGP Uplinks

• Used Uplinks

• AS174 CogentCo

• AS248 23C3

• AS286 KPN Eurorings

• AS5430 Freenet (IPv6)

• AS8881 Versatel

– AS9033 ECIX DUS

• AS12732 D-Hosting

– AS25074 Mesh

BGP Changesduring Congress

• RIPE Tools http://www.ris.ripe.net/bgplay/

• Marks für 89.22.0.0/16• Starting announcement

• Versatel 2006-12-24 22:21

• KPN 2006-12-25 21:29

• CogentCo 2006-12-26 15:00

• Prepending as174 2006-12-27 14:42

Bandwidth

• Real Bandwidth• CogentCo 0400Mbps

• KPN Eurorings 1000Mbps

• Versatel 1000Mbps

• D-Hosting 0600Mbps

• Average• Out 1.6Gbps (>50%)

• In 0.6Gbps (<20%)

KPN Eurorings

CogentCo

Versatel

D-Hosting

Total Uplink

Issues

• Uplink per ISP > 1 GE physical Port

• Balancing load over IP Uplink

• Getting bigger Ports

• Core Router/Switch in BCC

• No redundancy

• Core Routers Off Site

• Simpler Setup

• Taking down the network takes a few days

• Insurance

Outlook

• CAMP 2007

• Fiber to the Camp (FttC)

• GE (10GE) IP Uplink

• Distribution Layer

BCC NetworkAgenda

• Network Diagrams

• Catering for Special Interest Groups

• Hardware Used

• Outages and Other Trouble

• Pretty Graphs

Layer 9

<visitor>||

wireless or cable||

<internet>

Layer 3

RX8m10

Upstream

Hack center89.22.16...46.0/23

2001:4ce8:23c3:1016..1047::/64

B-Level89.22.48...62.0/23

2001:4ce8:23c3:1048..1063::/64

C-Level89.22.64...94/23

2001:4ce8:23c3:1064..1095::/64

D-Level / Special Interest89.22.0...15.0/24

2001:4ce8:23c3:1000...1016::/64

Wireless89.22.96.0/21

2001:4ce8:23c3:1096::/6489.22.104,108.0/22

2001:4ce8:23c3:1104,1108::/64

VersatelAS8881

Internet

10G5x1G11x1G

8x1G10G

1G

?xG

10G+1G

1G

5G

Layer 2

RX8m10

10G

5x1G

10G

Cisco3xC35242xC3548

2948

HP

16x2626A-Level

E300

S50

Colo

server park

2948

HP

11x2626B-Level

11x1G

Cisco8xC3524

S50 2948

HP

11x2626C-Level

server park

10G

E30010G

1G

wall sockets

B-Level

wall sockets

C-Level

wall sockets

D-Level

DFDF

Layer 1

LS

A87 A85

D57

C57 C91B90.01

Hackcenter A Hackcenter B

Saal 1

Saal 2

Saal 3

Saal 4

C-Ring

B-Ring

Infortresen

Video Studio

Konferenzleitung

89.22.16.0/2089.22.32.0/20

89.22.48.0/20 89.22.64.0/20

WLAN : 89.22.96.0/19

89.22.0.0/20

2948

2626

16x2626

Workshop

Helpdesk

16xU

TP

2948

2626

16x2626

ER

LX

2xM

MD

F

?xU

TP

8x3524

MM

DF

<96xU

TP

MM

DF

15xU

TP

12xU

TP

<131xU

TP

E300

<96xU

TP

MM

DF

12xMMF

2xM

MD

F

6xM

MD

F

8xM

MD

F

2xM

MD

F

16xU

TP

16xU

TP

12xU

TP

16xU

TP

?xU

TP

2948

2626

13x2626

NOC

4xMMDF

MMDF

Orga

DF

104xU

TP

6 A03

N/C16

wlsw

N/A

outl

B0.248

4 B01.01

6 A06

A812

6 A44

4 A33

A102

6 A08

4 A07

A056

A046

A02.014

A01

A02

room

28

12

n

4xU

TP

wlswoutl

10 N/C

B11

A08

room

2

2

nB081 N/A

B1012

8 B82 N/A

N/A25 B95

N/C29

B034

56 B02

6 B04

5 B01.03

6 B01.02

B01.014

B051

N/Abeam.1

wlswoutl

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

B01.031

7 acces.

10 B83

fassad.10

14 B96

8 B94

B9212

16 B90

2 B87

B812

B80.014

B804

2 B19

B01

B18

room

2

36

n

259xU

TP

6 B06

bünhe4

2 C83

C0150

19 B09

50 C02

13 B08

3 B07

B05.012

C1922

wlswoutl

4 fassad.

26 NC

8 acces.

C864

C852

C842

2 C54

B05

C51

room

4

17

n

214xU

TP

N/C38

N/A52 D90

N/AC9066

C046

wlswoutl

C03

room

6

n

130xU

TP

(6 u

sable

)

12xM

MF

(15 u

sable

)

24xU

TP

12xM

MF

20xU

TP

(18 u

sable

)

24xM

MF

(10 u

sable

)

24xU

TP

6xM

MF

(12 u

sable

)

24xU

TP

D546

D014

N/C10

1 hofseit.

1 alex.st.

dach2

kuppel34

6 D57

D524

D504

wlswoutl

C90.01

room

4

n

76xU

TP

12xMMF

12xM

MF

A85

D57D57 D57

12xM

MF

A87

24xM

MF

B90.01

D57

12xM

MF

C57

12xM

MF

C91

D57 D57 D57 D57 D57 D57

24xU

TP

A85

24xU

TP

B90.01

24xU

TP

C57

24xU

TP

C90.01,C91

72xU

TP

D01..hofseit.

A87 A85 B90.01 C57 C91D57

E300

RX8

2x3524

2948

2626

16x2626

server park

?xU

TP

m10

SM

DF

MM

DF

MMDF

VersatelAS8881

SM

DF

MM

DF

1xMMDF

wirelessUpstream

(Stefan)

Engel

2x3548

10xU

TP

S50

S50

console

4xM

MD

F

SM

DF

console

server

Special Interest Groups

• Colocation

• Freifunk (OLSR)

• Sputnik (RFID)

• cbase fiber & wireless link

Colocation Area

Server Park

Patches D57

Patches C57

Patches C57

Patches A85

Network A87

Network A87 - CWDM

S50 The Movie

Core Router

Links

• 3 x 10GE-SR

• 1 x 10GE-ER

• 14 x 1000baseSX

• 1 x 1000baseLX

• ± 100 1000baseT

• ± 3,000 10baseT/100baseTX ports

• 1 x 10base2 coax

Issues & Outages

• Late Hardware Deliveries

• Force10 • Servers

Outages & Issues

• Core Router

• Broken Switch Fabric Module

• High CPU load on linecards

• Out of IPv4 next-hop memory space

SFMcentral-services#power-on snm 2Power on SNM2.rw_power_on_snm: write 000700ff to RW_MBRIDGE_CARD_POWER_OFF_REG

*** SNM 2 POST diags *** FE 0 access passed; FE 0 Serdes PRBS Loopback passed FE 0 multicast distribution table passed[...]FE200 slot 0 fe 2 Serdes 60 prbs loopback failed FE 1 multicast distribution table passed

FE 2 access passed;Error:rw_power_on_snm: SFM POST diags failed for SNM 2central-services#SYSLOG: Dec 28 22:28:44:<189>central-services, BGP Peer 2001:4ce8::3 DOWN (Hold Timer Expired)

LP CPUSSH@central-services#show cpu lp SLOT #: LP CPU UTILIZATION in %: in 1 second: in 5 seconds: in 60 seconds: in 300 seconds: 1: 25 25 23 24 2: 19 22 22 19 3: 7 9 9 11 4: 1 1 1 1 5: 3 4 3 2 6: 3 3 3 4 7: 1 1 1 1 8: 4 3 3 3

SSH@central-services#

Nexthop

INFO: Out of nexthop entries for path count 1 on slot 2.

Nexthop

SSH@central-services#rconsole 1Connecting to slave CPU 1/1... (Press Ctrl-Shift-6 X to exit)

rconsole-1/1@LP>enaNo password has been assigned yet...rconsole-1/1@LP#sh ip nexthop

Paths Total Free In-use 1 2816 0 2816 2 512 0 512 4 512 0 512 8 256 0 256

Nexthop

• Workaround: Kill ARP sponge to free up resources on all linecards

• Conversation with supplier is ongoing

rconsole-1/1@LP#sh ip next

Paths Total Free In-use 1 2816 894 1922 2 512 510 2 4 512 508 4 8 256 240 16

Graphs! - Core Traffic

Graphs! - IPv6

Graphs! - A Level

1003 Mbps incoming1675 Mbps outgoing

Peak Traffic IPv6: 6.6%

Wireless LAN 23C3Agenda

• 22C3 Recall

• Preparation & Design

• Topology

• Performance & Security

• Monitoring for Station Troubleshooting

• Statistics

• Outlook 2007 & Thanks...

Wireless LAN 22C3 Recall

• 36 Single Mode Access Points

• Low density 802.11a

• 2400 Aruba Switch = 512 Max Users

• User peak 509 != Average 350

• Performance Issues (QoS)

• Rogue Access Points

Wireless LAN 23C3Preparation & Design

• Location Planning

• Autocad

• Backbone Design

• Separation

• QoS

• Snort, Shaping

• Security

• ACL, Fuzzying, Load tests

Wireless LAN 23C3 Topology

Wireless LAN 23C3 Setup802.11 Performance

• Calibration

• Channel Setup

• Load Balancing

• Traffic Shaping

Wireless LAN 23C3Security

• Access Lists

• L2 Frame Monitoring

• Rogue AP prevention

• Gateway protection

Monitoring for Troubleshooting

Wireless LAN 23C3Statistics

Wireless LAN 23C3 Statistics

• 1-2d Max Assocs 1678, Average 631, Current was 1399

• 3-4d Max Assocs 1423, Average 633, Current was 633

• 1-2d Max in 156.5 Mb/s, Max out 30.9 Mb/s

• 3-4d Max in 163.1 Mb/s, Max out 32.1 Mb/s

Wireless LAN 23C3Outlook

• Protect users with a bridge filter?

• Provide 5 GHz Cards?

USE 802.11a 5 GHz!!!

Thanks to Aruba Networks, Luiz

Thank You!