buffer-to-buffer credits exchangesbuffer credits, … credits exchangesbuffer credits, exchanges,...
Post on 09-May-2018
232 Views
Preview:
TRANSCRIPT
Buffer-to-Buffer Credits ExchangesBuffer-to-Buffer Credits, Exchanges, and Urban Legends
Lou Ricci IBMLou Ricci, IBMHoward L. Johnson, Brocade
8 August 2011 (3:00pm – 4:00pm)Session 9931Session 9931
Room Europe 7
L l St ffLegal Stuff
• Notice• IBM may have patents or pending patent applications covering subject matter
described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A.
• Any references in this information to non-IBM Web sites are provied for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.
• Trademarks• The following terms are trademarks of the International Business MachinesThe following terms are trademarks of the International Business Machines
Corporation in the United States, other countries, or both: FICON® IBM® RedbooksTM System z10TM z/OS® zSeries® z10TM
• Other Company, product, or service names may be trademarks or service marks of othersothers.
Ab t tAbstract
• Performance in a FICON network is influenced by the underlying flow y y gcontrol mechanisms of Fibre Channel. In this session, we examine how Buffer-to-Buffer credits flow from the channel to the control unit. We also look at how exchanges are used in FICON applications and how they change with the introduction of zHPF. During both examinations, we explore the role of the FICON Director in managing Buffer-to-Buffer credits and exchanges over a cascaded network. Throughout the session we debunk the various FICON “UrbanThroughout the session, we debunk the various FICON “Urban Legends” featuring credits and exchanges. Take the opportunity to learn from two of the FICON industry’s leading experts in channel and fabric development and join our sessionfabric development and join our session.
A dAgenda
• Buffer CreditsWhat are the and ho do the ork?• What are they and how do they work?
• How do you fill the pipe?• What if you can’t fill the pipe?• What’s wrong with multiple senders and one receiver?• What happens when the pipes are different sizes?
What’s it like in the real world?• What s it like in the real world?• How do cascades Directors work?
• Exchanges• What are they and how do they work?
• What’s an Exchange?g• How many exchanges are needed?• Can they be "reused?”• Can you have too many exchanges?
• Error Sensitivity• Is FICON more sensitive to errors than FCP?• Is FICON more sensitive to errors than FCP?
• How sensitive are FICON frames to loss or corruption?• What recovery actions are taken?• What are the differences with FCP?
BUFFER CREDITSWhat are they and how do they work?
Wh t i B ff t B ff C dit?What is Buffer-to-Buffer Credit?
• The greater the BB Credit….The greater the BB Credit….• A. The faster frames can be sent• B. The farther apart the two ports can be• C. The larger the frames can be• D. None of the above
Wh t i B ff t B ff C dit?What is Buffer-to-Buffer Credit?
• The greater the BB Credit….The greater the BB Credit….• A.• B. The farther apart the two ports can be• C.• D.
Fl C t lFlow Control
• Related to the devices’ ability to receive and process frames• Manages when frames are coming faster than they can be processed• Manages when frames are coming faster than they can be processed• Dropped frames occur when frames are arriving too fast to be processed
• Frames can only be transmitted when the receiver is ready• Credit establishment communicates the number of frames a device can receive at a timeCredit establishment communicates the number of frames a device can receive at a time• The credit value is exchanged at login• Transmission stops when credit runs out• The receiver indicates when it is ready to receive more frames
Credit exchange at Fabric Login
Host says, “I can receive 40 frames.”
Storage says, “I can receive 16 frames.”
Switch says, “I can receive 8 frames.”
B ff C ditBuffer Credit
• At initialization, the two ports establish creditEach b ffer credit corresponds to a frame (regardless of si e)• Each buffer credit corresponds to a frame (regardless of size)
• Each side can support different values• Credit Count
• If a port doesn’t have credit, it can’t send a frameCredit Count has reached zero• Credit Count has reached zero
• Mechanism limits frame drops
Host thinks, “Good I can send S it h thi kGood, I can send 8 frames without thinking about it.”
Storage thinks, “Good, I can send 8 frames without thinking about it.”
Switch thinks, “OK, I can send 40 frames that way and 16 frames this way, but I have to think about it.”
Credit Count88
Credit Count8
Credit Count40
Credit Count16
Credit accounting after Fabric Login
R i R d (R RDY)Receiver Ready (R_RDY)
• R_RDYU d f li k l l fl t l
• Frame transmissionBB C dit i d t d• Used for link level flow control
• Called buffer-to-buffer credit (BB Credit)
• R_RDY is not a frame• It is a “primitive” so it doesn’t consume
• BB Credit is decremented• Once for each frame transmitted
• When BB Credit = 0• Transmission stops
• Frame received• It is a primitive so it doesn t consume a buffer • R_RDY is sent
• Causes transmitter to increment BB Credit
Credit Accounting during Transmission
Storage says “ICredit Count
1
Host says, “I can send 8 frames without stopping.”
Storage says, I can send 8 frames without stopping.”
1Credit Count
40
Credit Count5
Credit Count16R_RDY
stopping.Switch says, “I can 40 frames that way and 16 frames the other way without stopping.”
R_RDY
Urban Legend:B ff C dit t Z P blBuffer Credits at Zero are a Problem
• Buffer credit determines DISTANCETh di t t d b t d till i t i f ll li k f t• The distance two nodes can be apart and still maintain full link frame rate
• Buffer credit is the number of FRAME buffers• A port provides for it’s NEAREST neighbor for RECEIVING framesp p g• Does NOT have to be symmetrical
• Buffer credit is a FRAME count• Not a data SIZE• Not a data SIZE• A 1 byte frame consumes 1 buffer credit• A 2K byte frame consumes 1 buffer credit
• Number of credits needed is determined by:• Raw Link Speed• Speed of light thru a fiber• Distance between two adjacent nodesDistance between two adjacent nodes
BUFFER CREDITSExample: A full pipe
Initial Conditions
Channel “Perfect” Switch Control Unit
20
Number of B-B Credits the switch advertised to the channel during link init
NOTE: In these animations, both the frames and R_RDY’s are numbered. This is for illustrative purposes only. In reality, neither the frames nor the R_RDY’s are numbered. The arrival of an R_RDY only informs the receiver that A frame has been forwarded, now WHICH frame has been forwarded.
20
19
11
18
12 12
17
3 123 12
16
3 14 23 14 2
15
5 3 14 25 3 14 2
14
5 3 16 4 25 3 16 4 2
13
7 5 3 16 4 27 5 3 16 4 2
12
78 5 3 16 4 278 5 3 16 4 2
11
9 78 5 36 4 2 19 78 5 36 4 2 1
1
9 78 5 36 410
10
129 78 5 36 410 12
1 2
9
9 78 56 41011 3 129 78 56 41011 3 12
31 2
8
9 78 56101112 3 14 29 78 56101112 3 14 2
1 2 3 4
7
9 78 610111213 5 3 14 29 78 610111213 5 3 14 2
1 2 3 4 5
6
9 781011121314 5 3 16 4 29 781011121314 5 3 16 4 2
1 2 3 4 5 6
5
9 8101112131415 7 5 3 16 4 29 8101112131415 7 5 3 16 4 2
1 2 3 4 5 6 7
4
910111213141516 78 5 3 16 4 2910111213141516 78 5 3 16 4 2
1 2 3 4 5 6 7 8
4
1011121314151617 9 78 5 3 16 4 21011121314151617 9 78 5 3 16 4 2
2 3 4 5 6 7 8 9
4
1112131415161718 9 78 5 3 16 4 2101112131415161718 9 78 5 3 16 4 210
3 4 5 6 7 8 9 10
4
1213141516171819 9 78 5 36 4 210111213141516171819 9 78 5 36 4 21011
4 5 6 7 8 9 1011 1
THIS PAGE INTENTIONALLY LEFT BLANK
BUFFER CREDITSExample: A not so full pipe
Suppose the switch is too far away from the channel for the B-B credit it pp yadvertised to the channel
20
19
11
18
12 12
17
3 123 12
16
3 14 23 14 2
15
5 3 14 25 3 14 2
14
5 3 16 4 25 3 16 4 2
13
7 5 3 16 4 27 5 3 16 4 2
12
78 5 3 16 4 278 5 3 16 4 2
11
78 5 3 16 4 29 78 5 3 16 4 29
10
78 5 3 16 4 2910 78 5 3 16 4 2910
9
78 5 3 16 4 291011 78 5 3 16 4 291011
8
78 5 3 16 4 29101211 78 5 3 16 4 29101211
7
78 5 3 16 4 2910111213 78 5 3 16 4 2910111213
6
78 5 3 16 4 291011121314 78 5 3 16 4 291011121314
5
78 5 3 16 4 29101112131415 78 5 3 16 4 29101112131415
4
78 5 3 16 4 2910111213141516 78 5 3 16 4 2910111213141516
1
3
78 5 3 16 4 291011121314151617 78 5 3 16 4 291011121314151617
1 2
2
78 5 3 16 4 29101112131415161718 78 5 3 16 4 29101112131415161718
1 2 3
1
78 5 3 16 4 2910111213141516171819 78 5 3 16 4 2910111213141516171819
1 2 3 4
0
78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920
1 2 3 4 5
0
78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920
1 2 3 4 5 6
0
78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920
1 2 3 4 5 6 7
0
78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920
1 2 3 4 5 6 7 8
0
78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920
1 2 3 4 5 6 7 8 9
0
78 5 3 16 4 291011121314151617181920 78 5 3 16 4 291011121314151617181920
1 2 3 4 5 6 7 8 9 10
0
78 5 36 4 291011121314151617181920 78 5 36 4 291011121314151617181920
1 2 3 4 5 6 7 8 9 1011 1
0
78 5 36 491011121314151617181920 78 5 36 491011121314151617181920
1 2 3 4 5 6 7 8 9 101112 1 2
0
78 56 491011121314151617181920 78 56 491011121314151617181920
1 2 3 4 5 6 7 8 9 10111213 1 2 3
0
78 5691011121314151617181920 78 5691011121314151617181920
1 2 3 4 5 6 7 8 9 1011121314 1 2 3 4
0
78 691011121314151617181920 78 691011121314151617181920
1 2 3 4 5 6 7 8 9 101112131415 1 2 3 4 5
0
789101112131415161718192021 7891011121314151617181920
2 3 4 5 6 7 8 9 101112
21
13141516 1 2 3 4 5 6
0
8910111213141516171819202122 891011121314151617181920
3 4 5 6 7 8 9 101112
2122
1314151617 1 2 3 4 5 6 7
0
91011121314151617181920212322 91011121314151617181920212322
4 5 6 7 8 9 101112131415161718 1 2 3 4 5 6 7 8
0
1011121314151617181920212324 22 1011121314151617181920212324 22
5 6 7 8 9 10111213141516171819 1 2 3 4 5 6 7 8 9
0
212324 2225 1112131415161718192021
1 2 3 4 5 6 7 8 9 10
2324 2225
6 7 8 9 1011121314151617181920
11121314151617181920
0
212324 222526 12131415161718192021
2 3 4 5 6 7 8 9 10
2324 222526
7 8 9 1011121314151617181920
121314151617181920
11
0
212324 22252627 131415161718192021
3 4 5 6 7 8 9 10
2324 22252627
8 9 1011121314151617181920
1314151617181920
1112
0
212324 2225262728 1415161718192021
4 5 6 7 8 9 10
2324 2225262728
9 1011121314151617181920
14151617181920
111213
0
212324 222526272829 151617181920212324 222526272829
1011121314151617181920
151617181920
5 6 7 8 9 10 11121314
0
212324 22252627282930 1617181920212324 22252627282930
11121314151617181920
1617181920
6 7 8 9 10 1112131415
0
212324 2225262728293031 17181920212324 2225262728293031
121314151617181920
17181920
7 8 9 10 111213141516
0
212324 222526272829303132 181920212324 222526272829303132
1314151617181920
181920
8 9 10 11121314151617
0
212324 22252627282930313233 1920212324 22252627282930313233
14151617181920
1920
9 10 1112131415161718
0
212324 2225262728293031323334 20212324 2225262728293031323334
151617181920
20
10 111213 151617181914
0
212324 222526272829303132333435 212324 222526272829303132333435
1617181920 11121314151617181920
0
212324 22252627282930313233343536 212324 22252627282930313233343536
181920 2117 121314151617181920
0
212324 2225262728293031323334353637 212324 22252627282930313233343536
181920 13141516171819202122
37
0
212324 222526272829303132333435363738 212324 22252627282930313233343536
1920 14151617181920212223
3738
0
212324 22252627282930313233343536373839 212324 22252627282930313233343536
20 15161718192021222324
373839
0
212324 2225262728293031323334353637383940 212324 22252627282930313233343536
1718192021 25222324
37383940
16
0
212324 2225262728293031323334353637383940 212324 22252627282930313233343536
17181920212223242526
37383940
0
212324 2225262728293031323334353637383940 212324 22252627282930313233343536
181920222324252627
37383940
21
0
212324 2225262728293031323334353637383940 212324 22252627282930313233343536
192022232425262728
37383940
21
0
212324 2225262728293031323334353637383940 212324 22252627282930313233343536
202223242526272829
37383940
21
0
212324 2225262728293031323334353637383940 212324 22252627282930313233343536
222324252627282930
37383940
21
0
2324 2225262728293031323334353637383940 2324 22252627282930313233343536
222324252627282930
37383940
21 31 21
0
232425262728293031323334353637383940 2324252627282930313233343536
3132
37383940
22232425262728293021 2221
THIS PAGE INTENTIONALLY LEFT BLANK
BUFFER CREDITSExample: Multiple Senders and One Receiver
Suppose there are multiple senders to one receiver
20
Suppose there are multiple senders to one receiver
Each sender attempts to send at 100% link speed
20
20
1919
1
1
19
1818
12
12
18
1717
3 12
3 12
17
1616
3 14 2
3 14 2
16
1515
5 3 14 2
5 3 14 2
15
1414
5 3 16 4 2
5 3 16 4 2
14
1313
7 5 3 16 4 2
7 5 3 16 4 2
13
1212
78 5 3 16 4 2
78 5 3 16 4 2
12
1111
78 5 36 4 29
1
1
78 5 3
1
6 4 29
11
1010
78 5 36 49102
1
78 5 36 4910
11
21
101
99
78 56 4910113
1 2
78 56 491011
112
32
19
31
88
78 56910121134
1 2
78 569101211
1122
341 2
841 2
7745
78 6910111213
1 2 3
11223
34
78 6910111213
1 27
45
1 2
66456
7891011121314
1 2 3
11223
45
3
7891011121314
1 2 3
656
1 2 3
55567
89101112131415
1 2 3 4
1122334
45
89101112131415
1 2 3
5567
1 2 3
4 845678
910111213141516
1 2 3 4
11223344
56
910111213141516
1 2 3 4
4678
1 2 3 4
4 946789
1011121314151617
52 3 4
112233445
56
1011121314151617
1 2 3 4
36789
1 2 3 4
3 910
36789
1112131415161718
52 3 4
1122334455
67
1112131415161718
52 3 4
378910
52 3 4
3 1011
3789
10
1213141516171819
5 63 4
1223344556
67
1213141516171819
52 3 4
27891011
52 3 4
2 101112
2789
10
1314151617181920
5 63 4
2233445566
78
1314151617181920
5 63 4
289101112
5 63 4
2 111213
289
10
1415161718192021
5 6 74
2334455667
1415161718192021
5 63 478
15 63 4 8
91011121313
1 11121314
189
10
1516171819202122
5 6 74
3344556677
5 6 7489
1516171819202122
15 6 74 9
101112131414
1 12131415
19
1011
1617181920212223
5 6 7 8
3445566778
89
1617181920212223
5 6 74
091011121314
5 6 74
1415
0 1213141516
09
1011
1718192021222324
5 6 7 8
4455667788
910
1718192021222324
5 6 7 8
0101112131415
5 6 7 8
1516
0 1314151617
0101112
1819202122232425
96 7 8
4556677889
910
18192021222324
5 6 7 8
0101112131415
5 6 7 8
151617
18
0 131415161718
0101112
19202122232425
96 7 8
5566778899
1011
19202122232425
96 7 8
0111213141516
96 7 8
161718
19
0 141516171819
0111213
20212223242526
9 107 8
56677889910
202122232425
101196 7 8
0111213141516
96 7 8
19
161718
1920
0 141516171819
0111213
212223242526
9 107 8
667788991010
2122232425
11129 107 8
26
0121314151617
9 107 8
19
1718
20
2021
0 151617181920
0121314
222324252627
9 10 118
6778899101011
1112
2223242526
9 107 8
0121314151617
9 107 8
19
1718
2021
202122
0 151617181920
0121314
2324252627
9 10 118
77889910101111
12
23242526
9 10 118
27
0 131415161718
9 10 118
1918
202122
212223
0 161718192021
0131415
2425262728
9 10 11 12
788991010111112
12
24252627
9 10 118
0 131415161718
9 10 118
1918
2021222323
21222324
0 161718192021
0131415
25262728
9 10 11 12
8899101011111212
1314
25262728
9 10 11 12
0
19
1415161718
9 10 11 12
192021222324
22232425
0 171819202122
0141516
26272829
1310 11 12
89910101111121213
1314
25262728
9 10 11 12
0
19
1415161718
9 10 11 12
192021222324
22232425
0 171819202122
0
26272829
1310 11 12
141516
991010111112121313
141
26272829
1310 11 12
0
19
15161718
20
1310 11 12
202122232425
23242526
0 181920212223
0151617
27282930
13 1011 12
9101011111212131314
26272829
1310 11 121410
1310 11 12
19
15161718
20202122232425
THIS PAGE INTENTIONALLY LEFT BLANK
BUFFER CREDITSExample: Different sized pipes
F t PiFat Pipe / Skinny Pipe
20
8 Gig Link 4 Gig Link
19
1
18
12
17
3 12
16
3 14 2
15
5 3 14 2
14
5 3 16 4 2
13
7 5 3 16 4 2
12
78 5 3 16 4 2
11
9 78 5 36 4 2 1
102
9 78 5 36 410 1
1
93
19 78 56 41011
1
2
843
129 78 56101112
1 2
74543
1239 78 610111213
1 2
6456
1239 781011121314
1 2 3
54565
7
12349 8101112131415
1 2 3
844565
78
1234910111213141516
1 2 3 4
9356789
123451011121314151617
1 2 3 4
910
356789
123451112131415161718
52 3 4
1011
278910
6 123451213141516171819
52 3 4
101112
278910
6 234520 13141516171819
5 63 4
111213
1891011
6 234572021 141516171819
53 4 6
11121314
1891011
6 3457202122 1516171819
5 74 6
12131415
09101112
6 3457820212223 16171819
5 74 6
1213141516
09101112
6 457820212223 171819
5 7 86
24
1314151617
010111213
9 6 457820212223 181924
5 7 86
131415161718
010111213
9 6 57823 20212223 1924
97 86
19
141516171819
011121314
10 9 6 57823 2021222324
97 86
1920
141516171819
011121314
10 9 6782325 21222324
97 8 10
2021
151617181920
012131415
11 10 9 6782325 222324
97 8 10
202122
151617181920
012131415
11 10 9 7826 2325 2324
9 118 10
2223
161718192021
013141516
12 11 10 9 7826 2325 24
9 118 10
222324
161718192021
013141516
12 11 10 9 826 232527
9 11 1210
2324
171819202122
014151617
13 12 11 10 9 826 232527
9 11 1210
232423
171819202122
014151617
13 12 11 10 92628 2527
1311 1210
232423
181920212223
015161718
2628 2527
1311 1210
14 13 12 11 10 9
23242325
181920212223
015161718
14 13 12 11 1025 2628 27
1311 12 14
THIS PAGE INTENTIONALLY LEFT BLANK
BUFFER CREDITSExample: Real Life?
More real life example: Two senders sending at 30% 50% link rate to one receiver
20
More real life example: Two senders sending at 30% - 50% link rate to one receiver
20
20
1919
1
1
19
1818
12
12
18
1717
3 12
12
18
1616
3 14 2
12
18
1616
3 14 2
3 12
17
1616
3 14 2
3 14 2
16
1616
3 14 2
3 14 2
16
1515
3 14 25
3 14 2
16
1515
3 14 25
3 14 2
16
1515
34 25
1
34 2
1
1
16
1515
3452
1
34
11
2
5
115
1
14143
2
456
1
112
2
3456
114
1
1414
5634
21
56
1122
1 2
34
141 2
14144
56
21 3
11223
3
456
1 214
1 2
14144
56
21 3
11223
4
3
31 2
56
1431 2
1414 4
56
21 3
1122334
4
56
31 2
7
1331 2
14145
6
421 3
11223344
4
567
31 213
431 2
1313 5
6
421 3
7
112233445
567
431 213
431 2
1414
67
542 3
112233445
567
431 213
431 2
1313
678
542 3
12233445
67
5
543214
5432
1313 6
78
543
9
223344556
6
7
5432
8
135432
1212 6543
78910
23344556
543
78
6
614
543 6
1313
78910
654
33445566
78
543 614
543 6
1212
7891011
654
3445566
78
54 615
54 6
1313
7891011
65
445566
8
7
754 615
754 6
1313
7891011
65
45566
8
7
75 6
9
1575 6
1414
891011
76
5566
8
77
75 6
9
1575 6
1313
891011
76
12
56677
8
75 6
9
1575 6
1313
91011
876
12
56677
8
8
76
9
1676
14149
1011
87
12
667788
876
9
16876
THIS PAGE INTENTIONALLY LEFT BLANK
BUFFER CREDITSExample: Cascaded Directors
8208
7
1
820
1
6
12
820
12
5
3 12
820
3 12
4
3 14 2
820
3 14 2
3 819
15 34 2 1
1
5 34 2
2 818
125 36 4 12
1 2
5 36 4
1 817
3 127 56 4
31 2
3 127 56 4
0 816
3 14 278 56
1 2 3 4
3 14 278 56
0 815
5 3 14 29 78 6
2 3 4 5
5 3 14 29 78 6
0 814
5 3 16 4 278910 5 3 16 4 278910
3 4 5 6
0 813
7 5 3 16 4 2891011 7 5 3 16 4 2891011
4 5 6 7
0 812
78 5 3 16 4 29101112 78 5 3 16 4 29101112
5 6 7 8
0 811
9 78 5 3 16 4 210111213 9 78 5 3 16 4 210111213
6 7 8 9
0 810
9 78 5 3 16 4 21011121314 9 78 5 3 16 4 21011121314
7 8 9 10
0 79
12131415 9 78 5 36 4 21011 112131415
8 9 1011
9 78 5 36 4 21011
1
1
0 68
78 5 36 49101112 1213141516 78 5 36 49101112
1 2
1213141516
9 101112
0 57
14151617 3 1278 56 491011121314151617
10 111213
3 1278 56 4910111213
1 2 3
0 46
78 569101112131415161718 3 14 278 5691011121314
1 2 3 4
15161718
11121314
3 14 2
0 35
5 34 278 6910111213141516171819 5 34 278 69101112131415
1 2 3 4 5
16171819
12131415 1
0 24
5 36 47891011121314151617181920 5 36 478910111213141516
1 2 3 4 5 6
17181920
13141516 1 2
0 13
7 56 489101112131415161718192021 7 56 4891011121314151617
1 2 3 4 5 6 7
18192021
14151617 1 2 3
0 02
78 56910111213141516171819202122 78 569101112131415161718
1 2 3 4 5 6 7 8
19202122
15161718 1 2 3 4
0 01
78 691011121314151617181920212322 78 6910111213141516171819
1 2 3 4 5 6 7 8 9
20212322
16171819 2 3 4 5
0 00
7891011121314151617181920212324 22 78910
1 2 3 4 5 6 7 8 9 10
11121314151617181920212324 22
17181920 3 4 5 6
0 00
891011121314151617181920212324 2225 891011
2 3 4 5 6 7 8 9 10
121314151617181920
11
212324 2225
181920 21 4 5 6 7
THIS PAGE INTENTIONALLY LEFT BLANK
BUFFER CREDITSOther Neat Stuff
H h dit d I d?How much credit do I need?
• Good “Rule of thumb”
Number of credits needed = 1 + Link speed in Gb/s * Distance in kmFrame Size in KBFrame Size in KB
Example: 20 km at 1 Gb/s 110
4Gb32 Credits
Example: 20 km at 1 Gb/s1 + 1 * 20 = 11
25060708090
100en
t Dat
a R
ate
Example: 10 km at 4 Gb/s1 + 4 * 10 = 21
24 20 36 52 68 84 100 116 132 148
Distance (Km)
10203040
Per
ce
2 Distance (Km)
H “l ” i f ?How “long” is a frame?
• Traveling at the speed of light, a frame can be very long• At 1G, the length of a frame is about 4-kilometers.• At 2G, the length of a frame is about 2-kilometers.• At 4G, the length of a frame is about 1-kilometer.• At 8G, the length of a frame is about 500-meters.• At 16G, the length of a frame is about 200-meters.
H “f t” i f ?How “fast” is a frame?
• Speed of light in fibreSpeed of light in fibre• 200,000 km/second• 5-microseconds/km
• Transmission Rates in fibre• At 1G, a frame is sent in about 20-microseconds.• At 2G, a frame is sent in about 10-microseconds.t G, a a e s se t about 0 c oseco ds• At 4G, a frame is sent in about 5-microseconds.• At 8G, a frame is sent in about 2.5-microseconds.• At 16G a frame is sent in about 1-microsecond• At 16G, a frame is sent in about 1-microsecond.
EXCHANGES
Th t f t i iThe parts of a transmission
• Frame• Building block of an Fibre Channel connection• Contains the information to be transmitted• The address of the source and destination• Control information
• Sequence• A set of one or more related FramesA set of one or more related Frames• Transmitted unidirectionally from one port to an other
• Exchange• An Exchange is one or more nonconcurrent sequences• An Exchange is one or more nonconcurrent sequences• A single operation• May be unidirectional or bidirectional
Fib Ch l FFibre Channel Frame
The basic building block is the FRAME
SOF Header Payload CRC EOF
4 24 0 – 2048(2112)
4 4
Source/Destination link addressesExchange ID
Protects Frame
Identifies end of a frame
Identifies beginning of a frame
Protocol TypeStart or End of SequenceStart or End of Exchange
Identifies end of a frame
g g
Fi IU E lFicon IU Examples
SOF Header Ficon CRC EOF
1 Frame IU to transfer a Read CCW
SOF Header Header CRC EOF
SOF Header Ficon Header
CCW Data (2016 bytes) CRC EOF
3 Frame IU to transfer 4K of data
Header
SOF Header CCW Data (2048 bytes) CRC EOF
SOF Header CCW Data (32 bytes)
Ficon CRC CRC EOF
Urban Legend:FICON f E h Th FCPFICON uses fewer Exchanges Than FCP
• Fibre Channel Architecture defines an Exchange asg• “A mechanism for identifying and managing an operation between
two ports“
• All IUs (a.k.a. Sequences) that make up a single I/O operation are part of an Exchange
• In Ficon, each concurrent I/O operation uses two Exchanges• One unidirectional Exchange for IUs from the Channel to the CU• A different unidirectional Exchange for IUs from the CU to the• A different unidirectional Exchange for IUs from the CU to the
Channel
• The PAIR is commonly know as a “Ficon Exchange”• The PAIR is commonly know as a Ficon Exchange
S d IUSequences and IUs
• Each Upper Layer Protocol (ULP) defines the contents and format of pp y ( )it’s own Information Units (IUs)• Commands• Data• Status• Control• Etc
• Ficon IUs can be up to 8K (8192) in size• 8160 (8K-32) bytes of data
• 32 bytes contain Ficon Header informationy
• 4 frames are needed for the largest IU• The collection of frame(s) that make up a IU are called a Sequence
• A Sequence may be as small as a single Frame• A Sequence may be as small as a single Frame
H E h d I d?How many Exchanges do I need?
• Little’s Law states:• The number of “things” in a system can be determined by multiplying the
average arrival rate of those “things” by the average time each “thing” stays in the system.
• Applied to Ficon:• The average number of Exchanges active at any given time =
A erage I/O rate * A erage response timeAverage I/O rate * Average response time
• Example: 5000 Ficon I/Os / Second on a given channel with .4ms service time1 needs 2 Active Exchanges (pairs) at any given timeservice time needs 2 Active Exchanges (pairs) at any given time
1 The amount of time the I/O is active in the channel
ERROR SENSITIVITY
Urban Legend:FICON i iti th FCPFICON is more sensitive than FCP
• Is Ficon More Sensitive to Errors than FCP?• Reasons for Link Errors are the same
• Is a Ficon frame more likely to get lost, damaged or corruptedIs a Ficon frame more likely to get lost, damaged or corrupted than FCP?• The probability is the same• Frames are framesFrames are frames
• When a Ficon frame gets lost, damaged or corrupted, is the recovery action different from FCP?recovery action different from FCP?• Both protocols retry when errors occur
• FCP by the Device Driver• Ficon by IOS/ERPy
S Wh t th Diff ?So What are the Differences?
• z Operating Systemsz Operating Systems tend to provide more detailed messagesg
• Ficon does provide additional debug data and actions• RNID
Source: FC-FS-3 INCITS/T11 Draft Standard v0 92• Link Error Status Blocks• Extensive State Change
Processing
Source: FC FS 3 INCITS/T11 Draft Standard v0.92
See www.t11.org
Processing
SUMMARYThank you
SSummary
• Buffer Credits• Distance• Flow Control
• Exchanges• Unidirectional• Bidirectional
• Error Sensitivity• Error Sensitivity• Recovery• Reporting
SHARE, Orlando, August 2011Buffer-to-Buffer Credits, Exchanges, and Urban Legends
THANK YOU!
Buffer to Buffer Credits, Exchanges, and Urban LegendsSession 9931
REFERENCES
S k Bi hSpeaker Biography
• Lou RicciLou Ricci• IBM
• 32-years• 24 years in channel development• 24-years in channel development• An inventor of FICON
• FICON Firmware Team Leader• Contact Information
• lricci@us.ibm.com
S k Bi hSpeaker Biography
• Patty DrieverPatty Driever• IBM• System z I/O and Networking Technologist
• Contact Information• pgd@us.ibm.com
S k Bi hSpeaker Biography
• Howard L. JohnsonHoward L. Johnson• BROCADE• Technology Architect, FICON
27 t h i l d l t d t• 27 years technical development and management
• Contact Information• howard johnson@brocade comhoward.johnson@brocade.com
BONUS SLIDES
S d f Li htSpeed of Light
• 299 792 458 meters per second• In a vacuum• In a vacuum• National Institute of Standards and Technology
• http://www.nist.gov/pml/wmd/metric/length.cfm• 300 000 000 meters per second
• Common approximation• Wikipedia
• http://en.wikipedia.org/wiki/Speed_of_light• 1.44 – 1.46
• Refractive index of 1300 nanometer fiber• Encyclopedia of Laser Physics and Technologyy p y gy
• http://www.rp-photonics.com/refractive_index.html• 1.5
• Common approximation• Wikipedia
• http://en wikipedia org/wiki/Optical fiber#Materialshttp://en.wikipedia.org/wiki/Optical_fiber#Materials• 200 000 000 meters per second
• Calculated speed of light in single-mode fiber• V = C/N• 200 000 000 = 300 000 000 / 1.5
E d t E d C ditEnd to End Credit
• Device to Device Flow ControlDevice to Device Flow Control• Between source and destination
• Not the linksSi il t b ff t b ff fl t l• Similar to buffer-to-buffer flow control• At N_Port Login• Report available receive buffers (EE_Credit)• Transmitter counts buffers transmitted (EE_Credit_CNT)• Receiver acknowledges frame (ACK)
• ACK 1 (a single data frame in a sequence) – most common• ACK n (several (N) consecutive data frames in a sequence)• ACK 0 (all data frames in a sequence) – not used
Vi t l Ch lVirtual Channels
• Technology to allocate BB Credits to particular data flowsTechnology to allocate BB_Credits to particular data flows• Class F traffic has one data flow• Assigned with Zoning by using special Zone names
THIS SLIDE INTENTIONALLY LEFT BLANK
top related