Cisco Press800 East 96th StreetIndianapolis, IN 46240 USA

Cisco Express Forwarding

Nakia Stringfield, CCIE No. 13451Russ White, CCIE No. 2635Stacia McKee

ii

Cisco Express Forwarding

Nakia Stringfield, Russ White, Stacia McKee

Copyright© 2007 Cisco Systems, Inc.

Published by:Cisco Press800 East 96th Street Indianapolis, IN 46240 USA

All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission from the publisher, except for the inclusion of brief quotations in a review.

Printed in the United States of America 2 3 4 5 6 7 8 9 0

Second Printing August 2013

ISBN-10: 1-58705-852-9

ISBN-13: 978-1-58705-852-3

Warning and Disclaimer

This book is designed to provide information about Cisco Express Forwarding (CEF). Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness is implied.

The information is provided on an “as is” basis. The authors, Cisco Press, and Cisco Systems, Inc., shall have nei-ther liability nor responsibility to any person or entity with respect to any loss or damages arising from the informa-tion contained in this book or from the use of the discs or programs that may accompany it.

The opinions expressed in this book belong to the authors and are not necessarily those of Cisco Systems, Inc.

Trademark Acknowledgments

All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capital-ized. Cisco Press or Cisco Systems, Inc., cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.

Feedback Information

At Cisco Press, our goal is to create in-depth technical books of the highest quality and value. Each book is crafted with care and precision, undergoing rigorous development that involves the unique expertise of members from the professional technical community.

Readers’ feedback is a natural continuation of this process. If you have any comments regarding how we could improve the quality of this book, or otherwise alter it to better suit your needs, you can contact us through email at feedback@ciscopress.com. Please make sure to include the book title and ISBN in your message.

We greatly appreciate your assistance.

iii

Corporate and Government SalesCisco Press offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales.

For more information please contact: U.S. Corporate and Government Sales 1-800-382-3419 corpsales@pearson-techgroup.com.

For sales outside the U.S. please contact: International Sales international@pearsoned.com.

Publisher Paul BogerCisco Representative Anthony WolfendenCisco Press Program Manager Jeff BradyAssociate Publisher David DusthimerExecutive Editor Brett BartowManaging Editor Patrick KanouseDevelopment Editor Dayna IsleySenior Project Editor San Dee PhillipsCopy Editor Written Elegance, Inc.Technical Editors Neil Jarvis, LJ WobkerTeam Coordinator Vanessa EvansBook and Cover Designer Louis AdairComposition Mark ShirarIndexer Tim WrightProofreader Molly Proue

iv

About the AuthorsNakia Stringfield, CCIE No. 13451, is a network consulting engineer for Advanced Services at Cisco in Research Triangle Park, North Carolina, supporting top financial customers with network design and applying best practices. She was formerly a senior customer support engineer for the Routing Protocols Technical Assistance Center (TAC) team, troubleshooting issues related to CEF and routing protocols. Nakia has been with Cisco for more than six years, previously serving as a technical leader for the Architecture TAC team. She has given training courses on CEF operation and troubleshooting for inter-nal employees. Nakia also worked for a year with IBM Global Services LAN Support in Research Tri-angle Park, North Carolina. Nakia attended North Carolina State University and completed her bachelor of science degree in electrical engineering in 1996. She also earned a master of science in computer net-working and computer engineering from North Carolina State University in 2000.

Russ White, CCIE No. 2635, is a member of the Routing Protocol Design and Architecture Team at Cisco, Research Triangle Park, North Carolina. He is a member of the Internet Engineering Task Force (IETF) Routing Area Directorate, a cochair of the Routing Protocols Security Working Group in the IETF, a regular speaker at Networkers, a member of the Cisco Certified Internetwork Expert (CCIE) Content Advisory Group, a member of the core team developing the new Cisco Design certification, a regular contributor to the Internet Protocol Journal, and the coauthor of six other books about routing and routing protocols, including Optimal Routing Design, from Cisco Press. Russ primarily works in the development of new features and design architectures for routing protocols.

Stacia McKee is a customer support engineer and technical leader of the Routing Protocols (RP) Tech-nical Assistance Center (TAC) team at Cisco in Research Triangle Park, North Carolina. This team focuses on providing postsales support of IP routing protocols, Multiprotocol Label Switching (MPLS), quality of service (QoS), IP multicast, and many other Layer 3 technologies. Stacia has been with Cisco for more than six years, previously serving as a technical leader of the Architecture TAC team and mem-ber of the WAN/Access TAC team. She has created and presented training on packet switching, router architecture, and troubleshooting for internal employees. Stacia has also been a technical editor and reviewer of Cisco.com technical documentation, mainly in router and IOS architecture and IP routing protocols technologies. She works closely with the IP Routing and IP Services groups within the Cisco Network Software and Systems Technology Group (NSSTG) on customer problems and early field tri-als. In 2000, Stacia completed her bachelor of science degree in computer information systems, bache-lor of science degree in business administration, and bachelor of arts degree in computer science at the College of Charleston in Charleston, South Carolina.

v

About the Technical ReviewersNeil Jarvis has been a software engineer in the networking industry since 1990. He is currently employed by Cisco Systems as a distinguished engineer, responsible for the architecture and develop-ment of switching control and data plane software, including Cisco Express Forwarding (CEF). He was a technical contributor and editor of a number of IEEE 802 standards, including 802.1 (bridging) and 802.5 (token ring). He was IEEE 802.1 vice-chair for a number of years. Neil graduated with a master’s degree in microelectronic systems engineering from UMIST (Manchester, England) in 1989 and now lives with his wife in Edinburgh, Scotland.

LJ Wobker, CCIE No. 5020, holds a bachelor of science degree in computer science from North Caro-lina State University in Raleigh, North Carolina. He started his networking career running cables as a college intern in the Cisco Research Triangle Park TAC lab and has worked in TAC, Advanced Services, and software development. For the last five years, LJ has been a technical marketing engineer, support-ing the Cisco 12000 and CRS-1 series routers.

vi

DedicationsNakia Stringfield:

I would like to dedicate this book to my wonderful, supportive husband, Kwame Stringfield, and to our beautiful daughter, Kyra. Most of all, thanks go to God for favor and challenging opportunities. Thanks to my parents, Robert and Annette; my family; my pastors; Dr. Frank and JoeNell Summerfield; and my friends for their many prayers and for believing in me.

Russ White:

I would like to dedicate this book to my two daughters, Bekah and Hannah, as well as to my beautiful wife, Lori. I would like to thank God for the opportunities and skills to work on routers, routing, and books.

Stacia McKee:

I would like to dedicate this book in memory of my former colleague and dearest friend, Parag Avinash Kamat (July 19, 1977–August 19, 2004). May his memory live on forever. I would like to thank my wonderful husband, Michael McKee, and my parents, Richard and Sidney Froom, for their love, patience, and support while completing this project. I also thank God for all His blessings in my life.

vii

AcknowledgmentsThis book would not have been possible without the help of many people whose various comments and suggestions helped to formulate this project. First, we would like to give special recognition to Richard Froom for providing crucial direction and valuable feedback for this book. We also want to thank the technical reviewers for this book, Neil Jarvis and LJ Wobker.

Finally, we want to thank Brett Bartow, Chris Cleveland, and Dayna Isley, as well as the other people at Cisco Press, for working with us, keeping us on track, and getting this book published.

viii

This Book Is Safari EnabledThe Safari® Enabled icon on the cover of your favorite technology book means the book is available through Safari Bookshelf. When you buy this book, you get free access to the online edition for 45 days.

Safari Bookshelf is an electronic reference library that lets you easily search thousands of technical books, find code samples, download chapters, and access technical information whenever and wherever you need it.

To gain 45-day Safari Enabled access to this book:

• Go to http://www.ciscopress.com/safarienabled

• Complete the brief registration form

• Enter the coupon code R7CH-25PD-7T4V-4VDV-RYMJ

If you have difficulty registering on Safari Bookshelf or accessing the online edition, please e-mail customer-service@safaribooksonline.com.

ix

Contents at a GlanceIntroduction xvi

Part I Understanding, Configuring, and Troubleshooting CEF 3

Chapter 1 Introduction to Packet-Switching Architectures 5

Chapter 2 Understanding Cisco Express Forwarding 51

Chapter 3 CEF Enhanced Scalability 81

Chapter 4 Basic IP Connectivity and CEF Troubleshooting 103

Part II CEF Case Studies 135

Chapter 5 Understanding Packet Switching on the Cisco Catalyst 6500 Supervisor 720 137

Chapter 6 Load Sharing with CEF 163

Chapter 7 Understanding CEF in an MPLS VPN Environment 217

Part III Appendix 257

Appendix A Scalability 259

Index 264

x

ContentsIntroduction xvi

Part I Understanding, Configuring, and Troubleshooting CEF 3

Chapter 1 Introduction to Packet-Switching Architectures 5

Routing and Switching 5Understanding Broadcast and Collision Domains 5

Broadcast and Collision Domains 6Broadcast and Collision Domains in Routing 7

Layer 3 Switching 8

Understanding Router Pieces and Parts 9Interface Processors 10Central Processing Unit 11Memory 11Backplanes and Switching Fabrics 11

Shared Memory 11Crossbar Switching Fabric 13Bus Backplanes 14

Cisco IOS Software: The Brains 17Memory Management 17

Memory Pools 17Memory Regions 18Packet Buffers 20

Interaction with Interface Processors 28

Processes and Scheduling 28Process Memory 28Process Scheduling 29

Understanding the Scheduler 29Process Life Cycle 29Process Priorities 32Scheduling Processes 32Process Watchdog 34Special Processes 35

Putting the Pieces Together: Switching a Packet 35Getting the Packet off the Network Media 35

Inbound Packets on Shared Media Platforms 36Inbound Packets on Centralized Switching Routers with Line Cards 37Inbound Packet Handling on Distributed Switching Platforms 38

Switching the Packet 39Switching the Packet Quickly During the Receive Interrupt 39Process-Switching the Packet 41

Transmitting the Packet 44

xi

Hardware and Software show Commands 45

Summary 48

Chapter 2 Understanding Cisco Express Forwarding 51

Evolving Packet-Switching Methods 51Process Switching 51Fast Switching 52What Is CEF? 53

CEF Tables 54Forwarding Information Base (FIB) 54The Adjacency Table 60Relating the CEF Tables 61

CEF Table Entries 62FIB Entries 62

Attached FIB Entry 63Connected FIB Entry 63Receive FIB Entry 63Recursive FIB Entry 64Default Route Handler FIB Entry 66ADJFIB FIB Entry 66Learned from IGPs 67

Generic FIB Entries 67Interface-Specific FIB Entries 68

FIB Entries Built for a Multiaccess Network Interface 68FIB Entries Built on a Point-to-Point Network Interface 69FIB Entries Built on a 31-Bit Prefix Network Interface 69

Special Adjacencies 69Auto Adjacencies 70Punt Adjacency 70Glean Adjacency 71Drop Adjacency 72Discard Adjacency 73Null Adjacency 73No Route Adjacencies 74Cached and Uncached Adjacencies 74Unresolved Adjacency 75

Switching a Packet with CEF 75

The CEF Epoch 77

Configuring CEF/dCEF 77

Summary 78

References 79

xii

Chapter 3 CEF Enhanced Scalability 81

Fundamental Changes to CEF for CSSR 82Data Structures 82Switching Path Changes 84

Changes to show Commands 86show ip cef 86show ip cef interface 86show ip cef summary 87show cef state capabilities 88

New show ip cef Commands 89show ip cef tree 89show ip cef internal 90show ip cef switching statistics 91

New show cef Commands 91

CEF Event Logger 94

CEF Consistency Checker 97Passive Checkers 97Active Checkers 97Consistency-Checking Process 98

New CEF Processes 100FIB Manager 100Adjacency Manager 100Update Manager 100

Summary 101

Chapter 4 Basic IP Connectivity and CEF Troubleshooting 103

Troubleshooting IP Connectivity 103Accurately Describe the Problem 104Scoping the Network Topology 105Reviewing the OSI Model for Troubleshooting 106

Troubleshooting Physical Connectivity 106Troubleshooting Layer 2 Issues 107

Verifying the ARP Table 108Verifying the Routing Table 111

Using IOS Ping with the Record Option to Rule Out CEF 115Troubleshooting the CEF FIB Table 116

Verifying the CEF Configuration 117Confirming the IP CEF Switching Path 119Using CEF Accounting Counters to Confirm the Switching Path 123Verifying the CEF Switching Details 123

xiii

Verifying the Adjacency Table 126Hardware-Specific Troubleshooting 128

Troubleshooting Punt Adjacencies 129

Understanding CEF Error Messages 131

Troubleshooting Commands Reference 131

Summary 133

References 133

Part II CEF Case Studies 135

Chapter 5 Understanding Packet Switching on the Cisco Catalyst 6500 Supervisor 720 137

CEF Switching Architecture on the Cisco Catalyst 6500 137Understanding Software-Based CEF and Hardware-Based CEF 137Centralized and Distributed Switching 138

Troubleshooting CEF on the Catalyst 6500 SUP720 Platforms 139Simple Checking of Connectivity 139Systematic Checking of Connectivity 141Troubleshooting Load Sharing 148

Summary 149

References 149

Chapter 6 Load Sharing with CEF 163

Benefits of Load Sharing 163

Load Sharing with Process Switching and Fast Switching 164

Comparing CEF Per-Packet and CEF Per-Destination Load Sharing 168Understanding Per-Destination Load Sharing 168Understanding Per-Packet Load Sharing 169

Minimizing Out-of-Order Packets 169Configuring CEF Per-Packet Load Sharing 170

CEF Architecture and Load Sharing 171

CEF Load Sharing Across Parallel Paths 173CEF Per-Destination Example 173CEF Per-Packet Example 180

Per-Packet Load Sharing on Hardware-Based Platforms 184

CEF Per-Packet Load Sharing on the Cisco GSR Platform 185

CEF Load-Sharing Troubleshooting Examples 186

xiv

CEF Per-Destination Load Sharing Overloading One Link 186CEF Per-Packet Load Sharing Causing Performance Issues 198Troubleshooting a Single-Path Failure with CEF Load Sharing 200CEF Traffic-Share Allocation 202CEF Polarization and Load-Sharing Algorithms 210

Original Algorithm 212Universal Algorithm 212Tunnel Algorithm 213Hardware Platform Implementations 213

Summary 214

References 215

Chapter 7 Understanding CEF in an MPLS VPN Environment 217

An Internet Service Provider’s Simple MPLS VPN Design 217

Understanding the CEF and MPLS VPN Relationship 219Case 1: Label Disposition 221Case 2: Label Imposition 222Case 3: Label Swapping 224Troubleshooting an MPLS VPN 224

CEF Considerations When Troubleshooting MPLS VPN Across Various Platforms 225

Cisco 7200 Router with an NPE-G2 226Cisco 7500 Router 226Cisco Catalyst 6500 with a Supervisor 2 227Catalyst 6500 with a Supervisor 720 3BXL 228Cisco 12000 Series Router 231Cisco 10000 Series Router 236

CEF and MPLS VPN Load-Sharing Considerations 237PE-CE Load Sharing: CE Multihomed to Same PE 237PE-CE Load Sharing: Site Multihomed to Different PEs 243Load Sharing Between P and P Devices 252CEF and MPLS VPN Load-Sharing Platform Dependencies 253

Summary 253

References 254

Part III Appendix 257

Appendix A Scalability 259

Index 264

xv

Icons Used in This Book

Command Syntax ConventionsThe conventions used to present command syntax in this book are the same conventions used in the IOS Command Reference. The Command Reference describes these conventions as follows:

• Boldface indicates commands and keywords that are entered literally as shown. In actual con-figuration examples and output (not general command syntax), boldface indicates commands that are manually input by the user (such as a show command).

• Italics indicate arguments for which you supply actual values.

• Vertical bars (|) separate alternative, mutually exclusive elements.

• Square brackets [ ] indicate optional elements.

• Braces { } indicate a required choice.

• Braces within brackets [{ }] indicate a required choice within an optional element.

PC Terminal CatalystSwitch

MultilayerSwitch

Network Cloud Line: Ethernet Line: Serial File Server

Router

xvi

IntroductionHow does a router switch a packet? What is the difference between routing a packet and switching a packet? What is this CEF feature that is referred to in Cisco documentation and commonly found in Cisco IOS commands? This book answers these questions through comprehensive discussions of Cisco Express Forwarding (CEF).

CEF is a term used to describe one of the mechanisms used by Cisco IOS routers and Cisco Catalyst switches to forward packets. Other packet-switching mechanisms include process switching and fast switching. CEF is found in almost all Cisco IOS routers and Catalyst switches. However, documentation of the topic is scarce. From a technical support perspective, CEF is a widely misunderstood topic whose implementation varies significantly on multiple Cisco platforms. Cisco engineers, Cisco partners, and customers need material on CEF to properly deploy, maintain, and troubleshoot their networks.

CEF offers the following benefits:

• Improved performance—CEF is less CPU-intensive than fast-switching route caching. More CPU processing power can be dedicated to Layer 3 services such as quality of service (QoS) and encryption.

• Scalability—CEF offers full switching capacity at each line card when distributed CEF (dCEF) mode is active.

• Resilience—CEF offers unprecedented levels of switching consistency and stability in large dynamic networks. CEF can switch traffic more efficiently than typical demand-caching schemes.

Goals and MethodsThis book addresses common misconceptions about CEF and packet switching across various plat-forms. The goal is to help end users understand CEF and know how to troubleshoot, regardless of whether a CEF or another problem is occurring in the network. Little information collectively addresses these concerns because CEF is proprietary. This book helps you understand CEF better by using the fol-lowing methods:

• Explaining CEF basics

• Supplying troubleshooting scenarios that enhance your ability to recognize common mistakes

• Providing best practices for configuration

Who Should Read This BookThe focus audience of this book is networking professionals who require an understanding of Cisco packet-forwarding architecture and who are tasked with troubleshooting routing and switching issues in a Cisco network environment. This book is an invaluable guide for those seeking to gain an understand-ing of how CEF works and how to troubleshoot CEF issues on various hardware platforms.

xvii

How This Book Is OrganizedAlthough this book could be read from cover to cover, it is designed to be flexible and allows you to eas-ily move between chapters and sections of chapters to cover just the material that you need to trouble-shoot an immediate problem or to understand a concept.

Cisco Express Forwarding is divided into two parts. The first part of the book provides an overview of packet-switching architectures and CEF operation and advanced features. It also covers the enhanced CEF structure and general troubleshooting. The second part of the book focuses on particular case stud-ies. Because CEF is a widely misunderstood technology, the case studies focus on a list of the common topics that have been problematic for customers and those supporting Cisco networks. The case studies review and expand on material from the previous parts of the book and provide more in-depth analysis of real networking topologies and troubleshooting steps.

Part I, “Understanding, Configuring, and Troubleshooting CEF” includes the following chapters:

• Chapter 1, “Introduction to Packet-Switching Architectures”—This chapter explains packet-switching architecture and terminology. It also explains utilization of memory and buffers.

• Chapter 2, “Understanding Cisco Express Forwarding”—This chapter deals with the basics of CEF architecture and operation. It defines CEF terminology and history.

• Chapter 3, “CEF Enhanced Scalability”—This chapter discusses the enhanced CEF struc-ture and its purpose.

• Chapter 4, “Basic IP Connectivity and CEF Troubleshooting”—This chapter deals with general troubleshooting in a software-switching environment. Software switching has typically been used on routers.

Part II, “CEF Case Studies,” deals with special CEF case studies covering the following common scenarios:

• Chapter 5, “Understanding Packet Switching on the Cisco Catalyst 6500 Supervisor 720”—This chapter helps you understand the impact of CEF and learn how packet switching works on a Cisco Catalyst 6500 SUP720.

• Chapter 6, “Load Sharing with CEF”—This chapter discusses load sharing with CEF. It covers the purpose, configuration, and troubleshooting of common problems.

• Chapter 7, “Understanding CEF in an MPLS VPN Environment”—This chapter explains the impact of CEF in an MPLS VPN environment.

The book concludes with Appendix A, “Scalability,” which discusses CEF design considerations that could impact network scalability.

xviii

The Future of CEF and Packet SwitchingAlthough this book provides solid information for software handling and hardware handling, it does not provide a detailed description of implementation on all Cisco platforms and related technologies. Hard-ware design changes rapidly, and packet handling on one platform could easily consume the entire book.

This book does not address Parallel Express Forwarding (PXF), which is used on devices such as Cisco 10000 series routers, Cisco 7600 series Optical Service Modules (OSMs), and Cisco 7300 series rout-ers. PXF leverages a combination of parallel processing and pipelining techniques to the CEF algo-rithms for faster throughput and optimal flexibility through ASIC technology. Because PXF is highly dependent on the platform and specific ASIC technology, it is not covered in this book.

Hardware switching will continue to be optimized for performance advantages. Introduction of distrib-uted CEF (dCEF) on Cisco 7500 series routers was a start down this path years ago to offload packet switching from the central processor to the Versatile Interface Processor (VIP) line card. Then progres-sion occurred to hardware-based localized switching on Cisco 6500s with Distributed Forwarding Cards (DFCs), FlexWans, and OSMs.

Cisco recently introduced IOS Software Modularity, which provides subsystem In-service Software Upgrades and Process Fault Containment to the Cisco Catalyst 6500 series switches.

As you continue to learn more about Cisco Express Forwarding, you may find the following resources helpful:

• Bollapragada, V., R. White, and C. Murphy, Inside Cisco IOS Software Architecture, Indianapolis,Indiana: Cisco Press; 2000. Provides a detailed treatment of Cisco 7500 routers and Cisco 7200 routers.

• Cisco, Parallel Express Forwarding on the Cisco 10000 Series, www.cisco.com/en/US/products/hw/routers/ps133/products_white_paper09186a008008902a.shtml.

• Cisco, Cisco 7600 Series Router Q & A, www.cisco.com/en/US/products/hw/routers/ps368/products_qanda_item09186a008017a32b.shtml.

• Cisco, PXF Information for Cisco 7304 Routers, www.cisco.com/en/US/products/hw/routers/ps352/prod_maintenance_guide09186a008057410a.html.

• Cisco, Cisco Catalyst 6500 Series Switches with IOS Software Modularity Make IT Managers More Productive and Further Improve Network Reliability, http://newsroom.cisco.com/dlls/2005/prod_082905.html.

• Cisco, Cisco Catalyst 6500 with Cisco IOS Software Modularity, www.cisco.com/en/US/products/hw/switches/ps708/products_promotion0900aecd80312844.html.

This page intentionally left blank

This chapter covers the following topics:

• Fundamental changes to CEF for CCSR

• Changes to show commands

• New show ip cef commands

• New show cef commands

• CEF event logger

• CEF consistency checker

• New CEF processes

C H A P T E R 3

CEF Enhanced ScalabilityCisco Express Forwarding (CEF) is the most widely used forwarding mechanism on IP networks, so why change what works? There are several reasons:

• To improve CEF scaling and convergence times, with the end goal being the ability to handle up to 1 million prefixes in the forwarding table (so that you can sleep well if your network has reached the 900,000-route mark, and you were worried about the next phase of network growth)

• To make the interfaces between CEF, the routing table, access lists, Multiprotocol Label Switching (MPLS), and the various hardware forwarding engines more consistent and more defined

• To improve memory utilization

• To provide a more consistent mechanism to add new features to the switching path

• To provide for CEF Management Information Base (MIB) support

• To improve the performance of MPLS traffic engineering (TE) switching

• To merge the IP version 4 (IPv4) CEF tables and the IPv6 CEF tables, and their associated infrastructure and control interfaces

Modifications to CEF that occurred in Cisco IOS Release 12.2S largely involve internal changes that aren't obvious through output at the console, but improve the rate at which new features can be introduced, decrease the amount of work required to make CEF work with new hardware, and increase the quality of the code in the switching path.

This chapter starts by discussing fundamental changes to CEF to implement CEF Enhanced Scalability (also called CSSR), including new data structures that are slightly more complex than the ones we describe in Chapter 2, “Cisco Express Forwarding.” The output from show commands has changed somewhat in this newer version of CEF; that's the next item on the menu, followed by a very useful new feature, the CEF event logger, and finally, a high-level overview of some new CEF processes.

82 Chapter 3: CEF Enhanced Scalability

Fundamental Changes to CEF for CSSRTwo primary changes were made to fundamental CEF operation for CSSR:

• The data structures making up the CEF tables

• The switching path packets take when switched by CEF

The following sections cover these two areas of fundamental change.

Data StructuresBefore CSSR, CEF held data using three different types of data structures: the Forwarding Information Base (FIB), the loadinfo, and the adjacency. These data structures contained some amount of overlapping information, which was copied between them when some types of CEF entry were created or modified. To save memory when storing forwarding information, CSSR added a new, fourth type of data structure, which contains a single copy of the replicated information from the older data structures. Figure 3-1 illustrates CEF before CSSR.

Figure 3-1 CEF Data Structures

In this case, the route to 10.1.2.0/24 uses 10.1.1.1 as its next hop. The 10.1.1.1 network is not, itself, directly connected but is reachable through some other next hop. This is called a routing recursion, a common occurrence in many networks. As Figure 3-1 shows, in this case, the path information for 10.1.1.0/24 and 10.1.2.0/24 are identical, because both routes

fibtype

next hop

destination: 10.1.1.0/24

path information 2

path information X

path information 1

loadinfo

hash bucket 1

hash bucket 3

hash bucket X

hash bucket 2

forwarding info

adjacency table

fibtype

destination: 10.1.2.0/24

next hop: 10.1.1.1

path information 2

path information X

path information 1

hash

Duplicated

informationThe route to

10.1.2.0/24 dependson the route to10.1.1.0/24

Fundamental Changes to CEF for CSSR 83

are reachable through the same adjacency table entries. Figure 3-2 shows how path recursion arises in a network.

Figure 3-2 Route Recursion in a Network

In Figure 3-2, Routers B and C are running Open Shortest Path First (OSPF) on all their interfaces. This means that Router C has a route, learned through OSPF, for the 10.1.1.0/24 network, which connects Routers A and B.

Router A is also configured to advertise 10.1.2.0/24 to Router B through Border Gateway Protocol (BGP) across an external BGP (eBGP) peering session. Router B is readvertising 10.1.2.0/24 through BGP, across an internal BGP (iBGP) peering session, to Router C. When Router B is readvertising 10.1.2.0/24 through BGP to Router C, it leaves the next hop toward the destination set to the IP address of the BGP it learned the route from—in this case, Router A—or an address on the 10.1.1.0/24 network.

For Router C to forward traffic to a destination on 10.1.2.0/24, it looks up the destination and finds it is reachable through a next hop on the 10.1.1.0/24 network. This is the route recursion on the network that Figure 3-1 shows in the CEF table. Router C uses the path to 10.1.1.0/24 to reach destinations on 10.1.2.0/24.

Because BGP normally does not reset the next hop on routes received from eBGP peers, route recursion is normal in large-scale networks using BGP on top of some other routing protocol, such as most service provider networks.

To reduce the amount of memory CEF uses in networks of this type, CSSR separates the path information out from the remainder of the FIB information, including the destination

router bgp 65000 network 10.1.2.0 mask 255.255.255.0 neighbor 10.1.1.2 remote-as 65001 ....

router bgp 65001 neighbor 10.1.1.1 remote-as 65000 neighbor 10.1.3.2 remote-as 65001 ....router ospf 100 network 0.0.0.0 0.0.0.0 area 0

router bgp 65001 neighbor 10.1.3.1 remote-as 65001 ....router ospf 100 network 0.0.0.0 0.0.0.0 area 0

A

10.1.3.0/2410.1.1.0/24

10.1

.2.0

/24

eBGP iBGP

OSPF

B C

84 Chapter 3: CEF Enhanced Scalability

and prefix information. This allows recursive routes to share the same path information, reducing memory requirements, as shown in Figure 3-3.

Figure 3-3 CEF Data Structures After CSSR

As you can see from Figure 3-3, only one copy of the path information is stored in this new structure design. This reduces duplicated information in the CEF table, especially in networks with the largest number of routes, which use BGP and hence have a lot of routes reachable through recursion.

Switching Path ChangesCSSR also made one change to the CEF switching path. In pre-CSSR CEF, the switching path was a monolithic unit; each feature was checked as a packet was switched through the router in software, as Figure 3-4 illustrates.

fibtype

destination: 10.1.1.0/24

next hop

next hop: 10.1.1.1

path information 2

path information 1

path information X

Pointer to Path Information

destination: 10.1.2.0/24

fibtype

Pointer to Path Information

loadinfo

hash bucket 1

hash bucket 3

hash bucket X

hash bucket 2

adjacency table

forwarding infoThe route to10.1.2.0/24 dependson the route to10.1.1.0/24

hash

Fundamental Changes to CEF for CSSR 85

Figure 3-4 CEF Switching Path

In pre-CSSR CEF, each feature in the CEF switching path, including Network Address Translation (NAT), packet filtering, and Web Cache Communications Protocol (WCCP), is checked as the packet is switched, regardless of whether the feature is enabled. If the feature is not enabled, the check is simple—just one or two lines of code—but for each feature added to the CEF switching path, the CEF code itself must be changed and maintained.

CSSR changed this process, creating an insertion point in the switching path, as illustrated in Figure 3-5.

Figure 3-5 Feature Insertion in CSSR

No

Packetprocessed bydevice driver

.... Processing

Switch Packet

C Processing

Is C enabled?

.... Processing

Is .... enabled?

Yes Yes Yes

No No No

Packetprocessed bydevice driver

Is A enabled?

A Processing B Processing .... Processing

Is B enabled? Is .... enabled?

Yes Yes Yes

No No

Feature A

Packetprocessed bydevice driver

Packetprocessed bydevice driver

Switch Packet

Packetprocessed bydevice driver

Packetprocessed bydevice driver

Switch Packet

With feature A inserted

86 Chapter 3: CEF Enhanced Scalability

Normally, packets are switched along the path from the inbound device driver, through the CEF switching code, to the outbound device driver. No checks are made to determine whether additional features require processing on the packet.

If a feature is required, a new node is inserted in the switching path. As the packet is switched, control is passed to this additional node in the switching chain and then back to the CEF process. Multiple nodes can be inserted either before or after the CEF switch itself; they are chained, so each one occurs before or after some other feature in the switching path.

This change in the forwarding path not only eliminates the requirement to check for each feature in the path being enabled, but it also allows new features to be added, or old features to be removed, without modifying the CEF code responsible for switching packets.

Changes to show CommandsA number of commands have been added or changed in CSSR. The following sections describe each change or added show command.

show ip cefExample 3-1 shows the output of the show ip cef command in CEF and CSSR. The output is much shorter than in pre-CSSR CEF, because some information has been taken out of the command. The version, epoch, packet count, and byte count have all been removed, because this information is available elsewhere. The wording of the output has been shortened as well.

show ip cef interfaceMost of the information in the header of the show ip cef [interface] detail command has been removed, because it is available elsewhere; this includes the number of routes, number of leaves, the type of load sharing configured, the number of resets, and the timer configuration. The display of the CEF table entries in the second part of the output is similar

Example 3-1 Output of the show ip cef Command

CEF-router#show ip cef 10.1.1.0 detail10.1.1.0/24, epoch 0 via 10.1.2.1, 0 dependencies, recursive next hop 10.1.2.1, Ethernet0/0 via 10.1.2.0/24 valid glean adjacency

CSSR-router#show ip cef 10.1.1.0 detail10.1.1.0/24, epoch 0 recursive via 10.1.2.1 attached to FastEthernet0/0

Changes to show Commands 87

to the output format for show ip cef, with the same changes as noted previously. You can see these changes in Example 3-2.

show ip cef summaryA number of fields have been removed from the show ip cef summary output as well, because the information is available in the output of other show commands. The output of show ip cef summary is identical to the header section of show ip cef [interface] detail,described in the preceding section, and the changes are identical. Example 3-3 provides a sample output for show ip cef summary.

Example 3-2 Output of the show ip cef detail Command

CEF-router#show ip cef [interface] detailIP CEF with switching (Table Version 66), flags=0x0 46 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 1 46 leaves, 23 nodes, 30360 bytes, 85 inserts, 39 invalidations 0 load sharing elements, 0 bytes, 0 references universal per-destination load sharing algorithm, id 51DB3C24 3(0) CEF resets, 0 revisions of existing leaves Resolution Timer: Exponential (currently 1s, peak 1s) 0 in-place/0 aborted modifications refcounts: 3167 leaf, 3112 node Table epoch: 0 (46 entries at this epoch) Adjacency Table has 5 adjacencies10.1.2.0/24, version 43, epoch 0, attached, connected0 packets, 0 bytes via Ethernet0/0, 0 dependencies valid glean adjacency....

CSSR-router#show ip cef [interface] detailIPv4 CEF is enabled and runningVRF Default: 31 prefixes (31/0 fwd/non-fwd) Table id 0, version 31, 1 resets Database epoch: 0 (31 entries at this epoch) 10.1.2.0/24, epoch 0, flags attached, connected attached to FastEthernet0/010.1.2.1/32, epoch 0, flags adjfib NetFlow: Origin AS 0, Mask Bits 0 attached to FastEthernet0/010.1.1.0/24, epoch 0 recursive via 10.1.2.1 attached to FastEthernet0/0

88 Chapter 3: CEF Enhanced Scalability

show cef state capabilitiesThe output of the show cef state command has changed dramatically in CSSR, as shown in Example 3-4. The new output shows three sections: one for the route processor (RP instance), one for IPv4 (IPv4 CEF Status), and one for IPv6 (IPv6 CEF Status). Much of the information included in the older version of the output has been removed, because it is available in the output of other show commands.

Example 3-3 Output of the show ip cef summary Command

CEF-router#show ip cef summary IP CEF with switching (Table Version 66), flags=0x0 46 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 1 46 leaves, 23 nodes, 30360 bytes, 85 inserts, 39 invalidations 0 load sharing elements, 0 bytes, 0 references universal per-destination load sharing algorithm, id 51DB3C24 3(0) CEF resets, 0 revisions of existing leaves Resolution Timer: Exponential (currently 1s, peak 1s) 0 in-place/0 aborted modifications refcounts: 3167 leaf, 3112 node Table epoch: 0 (46 entries at this epoch)

CSSR-router#show ip cef summary IPv4 CEF is enabled and runningVRF Default: 31 prefixes (31/0 fwd/non-fwd) Table id 0, version 31, 1 resets Database epoch: 0 (31 entries at this epoch)

Example 3-4 Output of the show ip cef state capabilities Command

CEF-router#show cef state capabilitiesCEF Status [RP] CEF enabled/running dCEF disabled/not running CEF switching enabled/running CEF default capabilities: Always CEF switching: no Always dCEF switching: no Default CEF switching: yes Default dCEF switching: no Drop multicast packets: no OK to punt packets: yes NVGEN CEF state: no fastsend() used: yes CEF NSF capable: no RPR+/SSO standby capable: no IPC delayed func on SSO: no FIB auto repair supported: yes LCs not running at init time: no Hardware forwarding supported: no

New show ip cef Commands 89

New show ip cef CommandsThree new commands have been added under show ip cef, including show ip cef tree,show ip cef internal, and show ip cef switching statistics. These three commands are covered in the following sections.

show ip cef treeExample 3-5 shows the output of the show ip cef tree command.

Hardware forwarding in use: no Load-sharing pr. packet supported: yes

CSSR-router#show cef stateCEF Status: RP instance common CEF enabledIPv4 CEF Status: CEF enabled/running dCEF disabled/not running CEF switching enabled/running universal per-destination load sharing algorithm, id 5E850505IPv6 CEF Status: CEF disabled/not running dCEF disabled/not running original per-destination load sharing algorithm

Example 3-5 Output of the show ip cef tree Command

CSSR-router#show ip cef treeVRF Default tree information: MTRIE/RTREE storing IPv4 addresses 31 entries (31/0 fwd/non-fwd) Forwarding tree: Forwarding lookup routine: IPv4 mtrie 8-8-8-8 optimized 36 inserts, 5 deletes 8-8-8-8 stride pattern short mask protection enabled for <= 4 bits without process suspension 31 leaves (868 bytes), 20 nodes (20800 bytes) 23236 total bytes leaf ops: 36 inserts, 5 deletes leaf ops with short mask protection: 3 inserts, 2 deletes per-prefix length stats: lookup off, insert off, delete off refcounts: 2150 leaf, 2085 node node pools: pool[C/8 bits]: 20 allocated (0 failed), 20800 bytes Non-Forwarding tree: 29 inserts, 29 deletes 0 nodes using 0 bytes

Example 3-4 Output of the show ip cef state capabilities Command (Continued)

90 Chapter 3: CEF Enhanced Scalability

The output of show ip cef tree includes the following:

• A section per the virtual routing and forwarding (VRF) table. In the output in this example, there is only one VRF, the Default, so only one section is shown. If more than one VRF existed on this router, there would be a section for each VRF configured.

• The type of addresses, IPv4 or IPv6, being stored in the tree.

• The number of entries in the tree.

• The type of tree used in this VRF, which indicates the stride. In Example 3-5, the stride is 8-8-8-8.

• The number of nodes and leaves in the tree, as well as the number of tree operations, such as inserts and deletes.

• Information about the amount of memory used by each tree.

show ip cef internalAnother new command added is show ip cef internal, as shown in Example 3-6. If you run the show ip cef internal command, you might see some entries prefixed by the tilde symbol (~). These are nonforwarding entries, which are stored in a separate tree from the forwarding entries. Nonforwarding entries result from CEF receiving information about specific destinations from multiple sources and only using one of the provided forwarding entries.

Example 3-6 Output of the show ip cef internal Command

CSSR-router #show ip cef internalIPv4 CEF is enabled and runningVRF Default: 31 prefixes (31/0 fwd/non-fwd) Table id 0, version 31, 1 resets Database epoch: 0 (31 entries at this epoch) 0.0.0.0/32, version 0, epoch 0, flags receive, RIB, refcount 4 sources: CEF path 638F83F0, path list 638F3350, share 1, flags receive ifnums: (none) path_list contains no resolved destination(s). HW IPv4 notified. receive output chain: receive (10)10.1.2.0/24, version 4, epoch 0, flags attached, connected, RIB, refcount 4 sources: RIB feature space: IPRM: 0x0004800C path 638F82A0, path list 638F3230, share 1, flags attached ifnums: (none) path_list contains at least one resolved destination(s). HW IPv4 notified. attached to FastEthernet0/0, adjacency glean output chain: glean

New show cef Commands 91

The show ip cef internal command essentially shows each CEF table entry, with all the available information about the entry. Useful fields include the source of the CEF table entry or the process that installed the entry, the type of entry, interfaces, and features installed on the switching path for each entry. This is a lot of information, and it probably won't be useful in many troubleshooting situations.

show ip cef switching statisticsCSSR adds a new command, show [ip|ipv6] switching statistics [feature]. This new showcommand provides information on each packet that CEF switched. For any packets punted to the process-switching path, the output of show ip cef switching will tell you why they were punted. Example 3-7 provides sample output for this new command.

New show cef CommandsAnother series of commands under show cef have also been added as part of CSSR. These commands contain generic information about CEF, rather than information related to IP switching of CEF.

As shown in Example 3-8, the output of the show cef fib and show cef loadinfo commandsprovides information about the number of entries allocated in each table and memory failures encountered when allocating new entries. This information is useful when troubleshooting a router with low memory or a memory leak.

Example 3-7 Output of the show ip cef switching statistics Command

CSSR-Router# show ip cef switching statistics Path Reason Drop Punt Punt2HostRP RIB Packet destined for us 0 253 0RP RIB Total 0 253 0 RP LES Packet destined for us 0 253 0RP LES Total 0 253 0 RP PAS Packet destined for us 0 506 0RP PAS TTL expired 0 0 160RP PAS Total 0 506 160 All Total 0 1012 160

Example 3-8 Output of the show cef fib and show cef loadinfo Commands

CSSR-router#show cef fib 31 allocated IPv4 entries, 0 failed allocations0 allocated IPv6 entries, 0 failed allocations

CSSR-router#show cef loadinfo0 allocated loadinfos, 0 failed allocations

92 Chapter 3: CEF Enhanced Scalability

In Example 3-9, the output of show cef memory shows each type of memory CEF uses and how much of it is in use. This is useful information for troubleshooting memory leaks or a router with a memory allocation problem.

The new command show cef table provides a summary of each CEF table configured on the router. Example 3-10 shows two CEF tables configured: a single table for IPv4 forwarding information and a single table for IPv6 forwarding information. The number of prefixes and the table version number are given. Any VRF within each table is listed in the table.

Example 3-11 shows the output of show cef timers, which provides a somewhat graphical display of the timers used to maintain the CEF tables. Timers in Cisco IOS Software are related by a parent/child relationship; when the parent timer expires (wakes up), all the child timers are marked as expired as well. This allows multiple overlapping events to be controlled independently but resynchronized, or easily restarted at the same time, when certain events occur.

Example 3-9 Output of the show cef memory Command

CSSR-router#show cef memory Memory in use/allocated Count --------------------------------------------------------------------- ADJ: DROP adjacency : 368/424 ( 86%) [1] ADJ: Discard adjacency : 368/424 ( 86%) [1] .... CEF: FIBHWIDB : 7592/8320 ( 91%) [13] CEF: FIBIDB : 2600/3328 ( 78%) [13] CEF: FIBSWSB control : 576/1024 ( 56%) [8] ....

Example 3-10 Output of the show cef table Command

CSSR-router#show cef table1 active IPv4 table out of a maximum of 10000VRF Prefixes Version Memory FlagsDefault 31 31 25584 1 active IPv6 table out of a maximum of 1VRF Prefixes Version Memory FlagsDefault 0 0 72

Example 3-11 Output of the show cef timers Command

CSSR-router#show cef timersCEF background process Expiration Type| 18.196 (parent) | 18.196 FIB checkers: IPv4 scan-rib-ios scanner | 18.196 FIB checkers: IPv4 scan-ios-rib scanner | 18.196 FIB checkers: IPv6 scan-ios-rib scanner | 18.468 FIB checkers: IPv4 scan-hw-sw scanner

New show cef Commands 93

Most of the timers relate to checking the CEF tables for consistency periodically or with throttling certain types of responses, such as Address Resolution Protocol (ARP) requests, so that they don't occur too often.

Finally, an entire chain of new commands allow you to examine the path information that has been added, including show cef path, show cef path list, and show cef path list walk.Each one shows the same information, with increasing amounts of detail, as Example 3-12 shows.

| 18.468 FIB checkers: IPv4 scan-sw-hw scanner Platform counter polling is not enabledIPv4 CEF background process Expiration Type| 0.160 (parent) | 0.160 adjacency update hwidb | 0.196 ARP throttle | 3.192 fibidb queue

Example 3-12 Output of the show cef path Command

CSSR-router#show cef path39 allocated IPv4 paths, 0 failed allocations0 allocated IPv6 paths, 0 failed allocations 39 Total Paths, 1 Recursive Paths, 0 Unresolved Paths

CSSR-router#show cef path list38 path lists (11 in shared path list hash table, 27 in special list)0 failed allocations hash table: [ 2] path list 638F2870, 1 path, 1 output chain, 1 lock [ 4] path list 638F3230, 1 path, 1 output chain, 1 lock [ 8] path list 638F30B0, 1 path, 1 output chain, 1 lock [ 9] path list 638F2F30, 1 path, 1 output chain, 1 lock [ 10] path list 638F2DB0, 1 path, 1 output chain, 1 lock [ 11] path list 638F2C30, 1 path, 1 output chain, 1 lock [ 12] path list 638F28D0, 1 path, 1 output chain, 1 lock [ 12] path list 638F2930, 2 paths, 1 output chain, 1 lock [ 12] path list 638F2AB0, 1 path, 1 output chain, 1 lock [ 31] path list 638F2510, 1 path, 1 output chain, 3 locks [ 46] path list 638F2690, 1 path, 1 output chain, 3 locks

CSSR-router#show cef path list walkCSSR-router##show cef path list walk38 path lists (11 in shared path list hash table, 27 in special list)0 failed allocations hash table: [ 2] path list 638F2870, 1 path, 1 output chain, 1 lock

Example 3-11 Output of the show cef timers Command (Continued)

continues

94 Chapter 3: CEF Enhanced Scalability

CEF Event LoggerCisco IOS Software components include event loggers. An event logger is a process that runs constantly, collecting much of the same information that various types of debug output provide, but without the overhead and without having to be explicitly enabled.

The event logger allows you to gather the information required to troubleshoot a problem regardless of whether you explicitly enabled debugging when the problem occurred, as long as you catch the log soon after the problem occurs. Event logs generally have a fixed size, which means that they will only hold a specific number of events before discarding the oldest event to replace it with the most recent one. The size of the IP CEF event log is set using the ip cef table event-log command; the default size is 16,000 events.

The CEF event logger records events in the CEF table, such as the insertion and deletion of CEF entries, as shown in Example 3-13.

[ 4] path list 638F3230, 1 path, 1 output chain, 1 lock [ 8] path list 638F30B0, 1 path, 1 output chain, 1 lock [ 9] path list 638F2F30, 1 path, 1 output chain, 1 lock [ 10] path list 638F2DB0, 1 path, 1 output chain, 1 lock [ 11] path list 638F2C30, 1 path, 1 output chain, 1 lock [ 12] path list 638F28D0, 1 path, 1 output chain, 1 lock [ 12] path list 638F2930, 2 paths, 1 output chain, 1 lock [ 12] path list 638F2AB0, 1 path, 1 output chain, 1 lock [ 31] path list 638F2510, 1 path, 1 output chain, 3 locks [ 46] path list 638F2690, 1 path, 1 output chain, 3 locks hash table path lists: path list 638F2870, flags 21, 2 locks ifnums: (none) path_list contains no resolved destination(s). HW IPv4 notified. 1 path path 638F76D0, path list 638F2870, share 1, flags attached ifnums: (none) path_list contains no resolved destination(s). HW IPv4 notified. attached to Null0, adjacency Null0 1 output chain chain[0]: Null0 path list 638F3230, flags 29, 2 locks ifnums: (none) path_list contains at least one resolved destination(s). HW IPv4 notified.....

Example 3-13 Output of the show ip cef event Command

CSSR-router#show ip cef event% Command accepted but obsolete, unreleased or unsupported; see documentation. 00:00:09.380: [Default] *.*.*.*/*'00 New FIB table [OK]

Example 3-12 Output of the show cef path Command (Continued)

CEF Event Logger 95

The first line indicates that this is an unsupported command, which means you will not find much documentation about this command on Cisco.com or in any manuals. This is because this command will eventually be obsolete, replaced by commands under the monitorevent-log chain, as shown in the following example.

The columns in this output contain the following information:

• Timestamp—This is the time at which the event occurred.

• VRF—The name of the VRF in which the event occurred is contained in brackets.

• Prefix and Prefix Length—The prefix that was inserted or removed, or that some other action was taken on is contained in this column.

• Action—The action taken is contained in this column.

• Result—The result of the action is contained in this column.

The action column can contain a large number of values, including the following:

• Events concerning the enabling or running of the CEF process, such as FIB enabled, FIB running, distributed FIB (dFIB) enabled, and dFIB running

• Error-handling events, such as “Handling malloc failed”

00:00:11.112: [Default] 0.0.0.0/32'00 FIB insert [OK]00:00:11.112: [Default] 255.255.255.255/32'00 FIB insert [OK]00:00:11.112: [Default] 224.0.0.0/24'00 FIB insert [OK]00:00:11.112: [Default] 224.0.0.0/4'00 FIB insert [OK]00:00:11.332: [Default] 224.0.0.0/4'00 FIB remove (flagged) [OK]00:00:11.332: [Default] 224.0.0.0/4'00 FIB remove (deleted) [OK]00:00:11.332: [Default] 224.0.0.0/4'00 FIB insert [OK]00:00:11.584: [Default] 0.0.0.0/32'00 FIB remove (flagged) [OK]00:00:11.584: [Default] 0.0.0.0/32'00 FIB remove (deleted) [OK]00:00:11.584: [Default] 224.0.0.0/24'00 FIB remove (flagged) [OK]00:00:11.584: [Default] 224.0.0.0/24'00 FIB remove (deleted) [OK]00:00:11.584: [Default] 224.0.0.0/4'00 FIB remove (flagged) [OK]00:00:11.584: [Default] 224.0.0.0/4'00 FIB remove (deleted) [OK]00:00:11.584: [Default] 255.255.255.255/32'00 FIB remove (flagged) [OK]00:00:11.584: [Default] 255.255.255.255/32'00 FIB remove (deleted) [OK]00:00:11.584: [Default] *.*.*.*/*'00 Flush FIB table (4/0ms) [OK]00:00:11.584: [Default] 0.0.0.0/32'00 FIB insert [OK]00:00:11.584: [Default] 255.255.255.255/32'00 FIB insert [OK]00:00:11.584: [Default] 224.0.0.0/24'00 FIB insert [OK]00:00:11.584: [Default] 224.0.0.0/4'00 FIB insert [OK]00:00:11.588: [Default] 10.1.2.0/24'00 FIB insert [OK]00:00:11.588: [Default] 10.1.2.21/32'00 FIB insert [OK]00:00:11.588: [Default] 10.1.2.0/32'00 FIB insert [OK]00:00:11.588: [Default] 10.1.2.255/32'00 FIB insert [OK]00:00:11.588: [Default] 10.1.2.0/24 NBD up [OK] ....

Example 3-13 Output of the show ip cef event Command (Continued)

96 Chapter 3: CEF Enhanced Scalability

• CEF process events, such as “Scanner process created” and “Scanner event loop enter”

• IPv4 FIB table entry events, such as FIB insert and FIB delete

• IPv6 FIB table entry events, such as FIB insert and FIB delete

• Line card events driven by the route processor

• Adjacency table events, such as interface up or down, protocol up or down, and others

Another CEF event trace log is also accessible through the show monitor event-trace cefcommand, as shown in Example 3-14.

As you can see from the output shown in Example 3-13, this information is more useful for code-level debugging than CEF-level debugging.

Example 3-14 Output of the show monitor event-trace cef Command

CSSR-router#show monitor event-trace cef ? all Show all the traces in current buffer back Show trace from this far back in the past clock Show trace from a specific clock time/date events CEF Events from-boot Show trace from this many seconds after booting interface CEF Interface Events ipv4 CEF IPv4 Events ipv6 CEF IPv6 Events latest Show latest trace events since last display merged Show entries in all event traces sorted by time

CSSR-router##show monitor event-trace cef all cef_events: 00:00:03.172: Inst unknown -> RP00:00:03.172: SubSys fib_ios_chain init00:00:09.372: SubSys fib init00:00:09.376: SubSys ipv4fib init00:00:09.384: SubSys ipv4fib_ios init00:00:09.432: SubSys fib_ios init.... cef_interface: 00:00:09.440: <empty> (sw 3) Create new00:00:09.440: <empty> (sw 3) SWIDBLnk FastEthernet0/0(3)00:00:09.440: Fa0/0 (sw 3) NameSet 00:00:09.440: <empty> (hw 1) Create new00:00:09.440: <empty> (hw 1) HWIDBLnk FastEthernet0/0(1)00:00:09.440: Fa0/0 (hw 1) NameSet 00:00:09.440: Fa0/0 (sw 3) State down -> up00:00:09.440: <empty> (sw 4) Create new00:00:09.440: <empty> (sw 4) SWIDBLnk FastEthernet1/0(4)....

CEF Consistency Checker 97

CEF Consistency CheckerThe CEF consistency checker verifies that the Update Manager (discussed in the section “New CEF Processes,” later in this chapter) is maintaining the local FIB tables on each line card correctly. Two consistency checkers are included with CSSR: active and passive.

Passive CheckersPassive checkers run constantly, in the background, unless you disable them using the [no]cef table consistency-check <ipv6|ipv4> command. During passive checking, the following items occur each minute:

• Each line card sends one interprocess communications (IPC) message containing CEF consistency checking information by default, although more can be configured.

• The route processor sends one IPC message containing CEF consistency check information to each line card.

• The route processor compares 1000 prefixes in the Routing Information Base (RIB) with their CEF entries to make certain that the CEF table matches the RIB. This is 60,000 prefixes per hour.

The configuration command cef table consistency-check <af> type <scan -ios -rib>[count <count>] [period <seconds>] controls the number of prefixes examined in each passive check and the time between passive checks.

To control the recording of error messages when an inconsistency is found, use the configuration command cef table consistency-check <af> error-message. In both of these commands, <af> is the address family, such as IPv4 or IPv6, you would like to configure.

Active CheckersAn active consistency check is initiated at the console, using the test cef enable command,followed by test cef table consistency [detail], as shown in Example 3-15.

Example 3-15 Output of the test cef enable and test cef table consistency Commands

CSSR-router#test cef enableThe use of TEST CEF commands will severely impact network performanceand stability and should be used with extreme caution. For safety,execute the “test cef disable” command to disable this capability whenit is no longer required.

CSSR-router#test cef table consistency detail full-scan-rib-ios: Checking IPv4 RIB to FIB consistencyfull-scan-rib-ios: FIB checked 8 prefixes, and found 0 missing.full-scan-ios-rib: Checking IPv4 FIB to RIB consistencyfull-scan-ios-rib: Checked 8 FIB prefixes in 1 pass, and found 0 extra.Error: Failed to run IPv6 full-scan-rib-ios checker

continues

98 Chapter 3: CEF Enhanced Scalability

An active check on a table of 150,000 prefixes can take between 5 and 60 seconds.

Consistency-Checking ProcessThe consistency-checking process contains two phases:

• Building, transmitting, and comparing the FIB table information

• Handling a detected inconsistency

The following list and Figure 3-6 illustrate the first part of this process:

1 The CEF consistency checker on the route processor builds a consistency check message by walking the local CEF table. For each entry, a description of the entry, including a checksum, is inserted into an IPC message.

2 When the IPC message is full, it is transmitted to all the line cards.

3 The CEF consistency checker compares the information received in the consistency check IPC packet with the same entries in the local CEF tables, including comparing the checksum computed locally with the checksum computed on the route processors.

4 If any entry does not match, the line card creates a new IPC message containing the local information about this entry and transmits it to the route processor.

Figure 3-6 CEF Consistency-Checking Process

Error: Failed to run IPv6 full-scan-ios-rib checkerNo IPv4 inconsistencies found, check took 00:00:00.004No IPv6 inconsistencies found, check took 00:00:00.000

Example 3-15 Output of the test cef enable and test cef table consistency Commands (Continued)

10.1.1.0/24 10.1.1.0/24

10.1.2.0/24 10.1.2.0/24

10.1.3.0/24 10.1.3.0/24

10.1.4.0/24 10.1.4.0/24

10.1.5.0/24 10.1.5.0/24

10.1.6.0/24 10.1.6.0/24

RP CEF Table LC CEF Table

1

24

3

CEF Consistency Checker 99

When the route processor receives the IPC message from the line card, it reexamines the local CEF tables, looking for a mismatch. If the data is still mismatched, the route processor will build a new IPC packet with the correct information and transmit it to the line card. If, after three checks (which allows time for any pending updates to be completed), the line card and route processor tables continue to be inconsistent, the line card is marked inconsistent on the route processor.

After a line card is marked inconsistent, CEF can auto-repair the CEF tables, if cef table consistency-check <af> auto-repair is configured. To auto-repair, CEF waits 10 seconds to allow all current consistency checks to finish. At the end of this time, the CEF epoch is incremented. This, in turn, causes the route processor to walk through the local CEF tables, generating updates to every line card for every entry. As these updates are generated, old FIB table information is purged. A hold-down timer prevents multiple auto-repairs from running concurrently.

To check the current state of the CEF consistency checkers, use the show cef table consistency-check command, as shown in Example 3-16.

Example 3-16 Output of the show cef table consistency-check Command

CSSR-router#show cef table consistency-check Consistency checker master control: enabled IPv4: Table consistency checker state: scan-rib-ios: disabled 0/0/0/0 queries sent/ignored/checked/iterated scan-ios-rib: disabled 0/0/0/0 queries sent/ignored/checked/iterated full-scan-rib-ios: enabled [1000 prefixes checked every 60s] 0/0/0/0 queries sent/ignored/checked/iterated full-scan-ios-rib: enabled [1000 prefixes checked every 60s] 0/0/0/0 queries sent/ignored/checked/iterated Checksum data checking disabled Inconsistency error messages are disabled Inconsistency auto-repair is enabled (10s delay, 300s holddown) Inconsistency auto-repair runs: 0 Inconsistency statistics: 0 confirmed, 0/16 recorded IPv6: Table consistency checker state: scan-ios-rib: disabled 0/0/0/0 queries sent/ignored/checked/iterated full-scan-rib-ios: enabled [1000 prefixes checked every 60s] 0/0/0/0 queries sent/ignored/checked/iterated full-scan-ios-rib: enabled [1000 prefixes checked every 60s] 0/0/0/0 queries sent/ignored/checked/iterated Checksum data checking disabled Inconsistency error messages are disabled Inconsistency auto-repair is enabled (10s delay, 300s holddown) Inconsistency auto-repair runs: 0 Inconsistency statistics: 0 confirmed, 0/16 recorded

100 Chapter 3: CEF Enhanced Scalability

New CEF ProcessesBeyond the data structure and switching path changes, which impact CEF switching directly, CSSR also added the following new processes to CEF to better manage the CEF data structures and CEF operation:

• FIB Manager

• Adjacency Manager

• Update Manager

The following sections describe each manager.

FIB ManagerThe FIB Manager manages the prefix path lists and loadinfo maps, including managing multilevel load sharing (through recursions). The FIB Manager also dynamically allocates new memory chunks as needed and handles statistics and feature data.

Adjacency ManagerThe Adjacency Manager, as its name implies, manages the adjacency tables. This includes managing the interface state, enabling and disabling protocols, and maintaining of a per-interface tree.

Update ManagerThe Update Manager keeps track of which entries in the FIB tables need to be updated. Updates are pulled from the line cards to the route processor through the Update Manager, which allows the line cards to regulate the rate at which new FIB information is transferred.

Summary 101

SummaryThis chapter introduced you to CEF Enhanced Scalability (CSSR). Cisco IOS Release 12.2S updated the data structures that make up the CEF tables and the switching path packets taken when switched by CEF.

This chapter also described changes to the show commands and new show and show ip cefcommands in CSSR.

CSSR also includes an event logger and consistency checker. Information gathered by the event logger aids in troubleshooting. The CEF consistency checker verifies that the Update Manager is maintaining the local FIB tables on each line card correctly. The consistency checkers included with CSSR can be either active or passive.

CSSR also added a FIB Manager, an Adjacency Manager, and an Update Manager to better manage the CEF data structures and CEF operation.

I N D E X

Numerics31-bit prefix network interface, FIB entries, 69

Aactive consistency checkers, 97–98Adjacency Manager, 100adjacency table, 60

auto adjacencies, 70cached adjacencies, 74–75discard adjacencies, 73drop adjacencies, 72epoch, 77glean adjacencies, 71not route adjacencies, 74null adjacencies, 73punt adjacencies, 70–71relationship to other address-mapping tables,

60–61uncached adjacencies, 74–75unresolved adjacencies, 75

adjacency table, verifying, 126–128ADJFIB FIB entries, 66aliases, 19ARP table, verifying, 108–111articulating IP connectivity problems, 104–105attached FIB entries, 63auto adjacencies, 70

Bbenefits of load sharing, 153–154blackholing, 262broadcast domains, 5–6

in routing, 7–8buffer pools

managing, 21–27types of, 20

bus backplane, 14–16

Ccached adjacencies, 74–75CE (customer edge) routers, 207CEF

accounting, enabling, 123configuring, 77–78hardware implementations, 203–204load distribution table, 162load sharing, troubleshooting, 176–188per-destination load sharing, 158

example of, 163–170per-packet load sharing, 159

configuring, 160–161example of, 170–173out-of-order packets, minimizing, 159performance issues, troubleshooting,

188–189traffic-share allocation, 192–199

CEF EXEC error messages, 131CEF polarization, 210–212cef table consistency-check command, 97CEF table, verifying, 116–125CEF/MFI (CEF/MPLS Forwarding

Infrastructure), 51centralized switching, 138Cisco 10000 series routers

MPLS VPN, troubleshooting, 226Cisco 12000 series routers

MPLS VPN, troubleshooting, 221–226Cisco 6500 router, troubleshooting MPLS VPN

on router with Supervisor 2, 217–218on router with Supervisor 720, 218–221

Cisco 7200 with NPE-G2, troubleshooting MPLS VPN, 216

Cisco 7500 router, troubleshooting MPLS VPN, 216–217

Cisco Catalyst 6500 switchescentralized switching, 138distributed switching, 138SUP720, 137

CEF, troubleshooting, 139–141connectivity, verifying, 141–148load sharing, troubleshooting, 148–149PFC3, 138

256

Cisco GSR (Gigabit Switch Router), per-packet load sharing, 175–176

Cisco IOS Software, 17input queues, 42interaction with interface processors, 28memory management, 17

memory pools, 17–18memory regions, 18packet buffers, 20–27

output queues, 42processes, 28

life cycle of, 29–32priorities, 32scheduling, 32–34special types of, 35watchdog timer, 34

collision domains, 5–6in routing, 7–8

commandscef table consistency-check, 97debug arp, 110external overload-signalling, 252ip cef table loadinfo force, 252mls ip cef load-sharing full, 148ping, 115–116, 191remote login module, 146show adjacency, 127show adjacency detail, 182show arp, 109, 132show buffers, 20–21, 47show cef drop, 124show cef fib, 91show cef interface, 160, 185–186show cef loadinfo, 91show cef memory, 92show cef not-cef-switched, 130show cef path, 94show cef state capabilities, 88–89show cef table, 92show cef table consistency-check, 99show cef timers, 92show interfaces, 106, 167–172, 177–190show interfaces stat, 119show ip cache, 120show ip cef, 86, 117, 123, 131–132, 161,

164–166, 169, 173, 178–179show ip cef event, 94–95

show ip cef interface, 86–87show ip cef internal, 90–91show ip cef summary, 87–88show ip cef switching statistics, 91show ip cef tree, 89–90show ip route, 112, 132, 143, 157, 181show ip route summary, 250show memory, 18, 45show mls cef adjacency entry, 145show mls statistics, 149show monitor event-trace cef, 96show processes, 31, 48show processes memory, 29, 35show region, 19, 46test cef enable, 97–98test cef table, 97–98

configuringCEF, 77–78

per-packet load sharing, 160–161dCEF, 77–78

connected FIB entries, 63connectivity, 103

troubleshootingadjacency table, verifying, 126–128ARP table, verifying, 108–111CEF table, verifying, 116–125issues, articulating, 104–105Layer 2, 107–108network topology, scoping, 105on SUP720, 139–141physical connectivity, 106–107routing table, verifying, 111–115

verifying on SUP720, 141–148consistency checking process, 98–99

active, 97–98passive, 97

convergence, 262–263CPU (central processing unit), 11crossbar switching fabric, 13–14CSSR

CEF operation, changes in, 82data structures, 82, 84switching path, 84, 86

processes added to CEF, 100

Cisco GSR (Gigabit Switch Router)

257

Ddata structures, CEF modifications for CSSR,

82–84dCEF (Distributed CEF ), 53

configuring, 77–78dead processes, 35debug arp command, 110debug commands, troubleshooting MPLS VPN,

215default route handler FIB entries, 66demand-based switching caches, 52design considerations for switching tables,

handling overlapping prefixes, 59–60direct memory access, 10discard adjacencies, 73distributed switching, 138

inbound packets, 38DRAM (dynamic random-access memory, 11drop adjacencies, 72dynamic buffer pools, 20

EeiBGP multipath feature, configuring, 233–234EIGRP (Enhanced IGRP), CEF traffic-share

allocation, 192–199enabling CEF accounting, 123epoch, 77Error Message Decoder, 131event logging, 94–96examples

of CEF per-destination load sharing, 163–170of CEF per-packet load sharing, 170–173

external overload-signalling command, 252

Ffast switching, 52, 154–158FIB (Forwarding Information Base), 54

epoch, 77flags, 62–66

entries learned from IGPs, 67for 31-bit prefix network interface, 69

for multiaccess network interface, 69for point-to-point network interface, 69generic entries, 67–68interface-specific entries, 68

longest-match lookups, 58mtries, 54

performance, 56stride patterns, 54, 57

structure of, 57FIB Manager, 100flags (FIB), 62–66

entries learned from IGPs, 67for 31-bit prefix network interface, 69for multiaccess network interface, 69for point-to-point network interface, 69generic enties, 67–68interface-specific entries, 68

flat memory model, 29

G-Hgeneric FIB entries, 67–68glean adjacencies, 71

hardware, required packet switching componentsbus backplane, 14–16CPU, 11interface processors, 10memory, 11switching fabric, 11–14

hardware abstraction, 18hardware-based CEF, 137–138hardware-based platforms, per-packet load

sharing, 174hardware-switching adjacency table,

troubleshooting, 129

Iidle state (processes), 31IMA (inverse multiplexing over ATM), 154inbound packets

on centralized switching routers with line cards, 37

inbound packets

258

on distributed switching platforms, 38on shared media platforms, 36

Init process, 35input queues, 42interface buffers, 20interface processors, 10

interaction with Cisco IOS Software, 28interface-specific FIB entries, 68ip cef table loadinfo force command, 262IP connectivity, troubleshooting, 103

adjacency table, verifying, 126–128ARP table, verifying, 108, 110–111CEF table, verifying, 116–117, 119, 121,

123–125issues, articulating, 104–105Layer 2 issues, 107–108network topology, scoping, 105physical connectivity, 106–107routing table, verifying, 111–115

ISP networks, 207

Llabel disposition, 210–212label imposition, 209, 212–213label swapping, 210, 214Layer 2 connectivity, troubleshooting, 107–108Layer 2 header rewrite string, 53Layer 3 switching, 8LFIB (Label Forwarding Information Base), 209life cycle of processes, 29–32line cards

consistency checking process, 98–99memory, verifying, 251

load distribution table, 162load sharing

benefits of, 153–154CEF polarization, 200–202in MPLS VPN environment

between P and P devices, 242CE multihomed to same PE, 237–233platform dependencies, 243site multihomed to different PEs, 233–241

troubleshooting, 148–149

lookups, longest match, 58LSP (Label Switch Path)tunnels, 209

Mmanaging buffer pools, 21–24, 26–27memory, 11

constraints, 249–251on line cards, verifying, 251process memory, 28

memory management (Cisco IOS Software), 17memory pools, 17–18memory regions, 18packet buffers, 20–27

memory pools on Cisco IOS Software, 17–18buffer pools, managing, 21–27memory regions, 18memory subregions, 19packet buffers, 20

minimizing out-of-order packets, 159mls ip cef load-sharing full command, 148MPLS VPNs

ISP networks, 207label disposition, 210–212label imposition, 209, 212–213label swapping, 210, 214LFIB, 209LSP tunnels, 209PE-CE load sharing

between P and P devices, 242CE multihomed to same PE, 227–233platform dependencies, 243site multihomed to different PEs, 233–241

troubleshooting, 214debug commands, 215on Cisco 10000 series routers, 226on Cisco 12000 series routers, 221–226on Cisco 6500 routers with Supervisor 2,

217–218on Cisco 6500 routers with Supervisor 720,

218–221on Cisco 7200 router with NPE-G2, 216on Cisco 7500 router, 216–217show commands, 215

VRF tables, 208

inbound packets

259

mtries (multiway tries), 54performance, 56stride patterns, 54, 57

multiaccess network interface, FIB entries, 69

N-Onew state (processes), 30not route adjacencies, 74null adjacencies, 73

out-of-order packetsminimizing, 159preventing, 40–41

output interface, 53output queues, 42overlapping prefixes, switching table design

considerations, 59–60overloaded links, troubleshooting CEF

per-destination load-sharing, 176–188

Ppacket buffers, 20packet switching, 51

during receive interrupt, 39–40inbound packets

on centralized switching routers with line cards, 37

on distributed switching platforms, 38on shared media platforms, 36

packets, transmitting, 44process switching, 41–44required components, 9

bus backplane, 14–16CPU, 11interface processors, 10memory, 11switching fabric, 11–14

passive consistency checkers, 97PE (provider edge)routers, 207per-destination load sharing, 158

per-packet load sharing, 159configuring, 160–161on Cisco GSR platform, 175–176on hardware-based platforms, 174

PFC3 (Policy Feature Card 3), 138physical connectivity, troubleshooting, 106–107ping command, 115–116, 191platform dependencies for MPLS VPN load

sharing, 243point-to-point network interface, FIB entries, 69preventing out-of-order packets, 40–41priorities assigned to Cisco IOS Software, 32private buffer pools, 20process memory, 28process scheduling, 29

process life cycle, 29–32process priorities, 32

process switching, 51–52, 154–158processes

scheduling, 32–34special types of, 35watchdog timer, 34

processor hogs, 34process-switching packets, 41–44public buffer pools, 20punt adjacencies, 70–71

troubleshooting, 129–131

Rready state (processes), 31receive FIB entries, 63–64receive interrupt, 38receive rings, 10recursive FIB entries, 64–65recursive lookups, 64–65regions, aliases, 19remote login module command, 146required components for packet switching, 9

bus backplane, 14–16CPU, 11interface processors, 10memory, 11switching fabric, 11–14

ring buffers, 10

ring buffers

260

routingbroadcast domains, 7–8collision domains, 7–8packet switching

bus backplane, 14–16CPU, 11memory, 11required components, 9–10switching fabric, 11–14

routing recursions, 82–84routing tables, verifying, 111–115

Sscalability, memory constraints, 249–251Sched process, 35scheduler, 29scheduling processes, 32–34SDRAM (synchronous dynamic random-access

memory), 11shared media platforms, inbound packets, 36shared memory architectures, 12–13show adjacency command, 127show adjacency detail command, 131, 192show arp command, 109, 132show buffers command, 20–21, 27, 47show cef drop command, 124show cef fib command, 91show cef interface command, 160, 185–186show cef loadinfo command, 91show cef memory command, 92show cef not-cef-switched command, 130show cef path command, 94show cef state capabilities command, 88–89show cef table command, 92show cef table consistency-check command, 99show cef timers command, 92show commands, troubleshooting

MPLS VPN, 225show interfaces command, 106, 167–172, 177–180show interfaces stat command, 119show ip cache command, 120show ip cef command, 86, 117, 123, 131–132, 161,

164–166, 169–170, 173, 178–179show ip cef event command, 94–95

show ip cef interface command, 86–87show ip cef internal command, 90–91show ip cef summary command, 87–88show ip cef switching statistics command, 91show ip cef tree command, 89–90show ip route command, 112, 132, 143, 157, 181show ip route summary command, 260show memory command, 18, 45show mls cef adjacency entry command, 145show mls statistics command, 149show monitor event-trace cef command, 96show processes command, 31, 48show processes memory command, 29, 35show region command, 19, 46single-path failures, troubleshooting, 190–192software-based CEF, 138static buffer pools, 20stride patterns, 54, 57structure of FIB, 57subregions, 19SUP720 (Supervisor 720), 137

CEF, troubleshooting, 139connectivity problems, 139–141

connectivity, verifying, 141–148load sharing, troubleshooting, 148–149PFC3, 138

switching, required components, 9bus backplane, 14–16CPU, 11interface processors, 10memory, 11switching fabric, 11–14

switching fabric, 11bus backplane, 14–16crossbar switching fabric, 13–14shared memory architectures, 12–13

switching path, CEF modifications for CSSR, 84–86

switching tables, handling overlapping prefixes, 59–60

syntax for aliases, 19

routing

261

Ttest cef enable command, 97–98test cef table command, 97–98traffic-share allocation, 192–199transmit interrupt, 44transmit rings, 10transmitting packets during packet switching

process, 44–45troubleshooting CEF, 103

IP connectivityadjacency table, verifying, 126–128ARP table, verifying, 108–111CEF table, verifying, 116–125issues, articulating, 104–105Layer 2 issues, 107–108network topology, scoping, 105physical connectivity, 106–107routing table, verifying, 111–115

load-sharingon SUP720, 148–149overloaded links, 176–188

MPLS VPN, 214debug commands, 215on Cisco 10000 series routers, 226on Cisco 12000 series routers, 221–226on Cisco 6500 routers with Supervisor 2,

217–218on Cisco 6500 routers with Supervisor 720,

218–221on Cisco 7200 routers with NPE-G2, 216on Cisco 7500 routers, 216–217show commands, 215

on Catalyst 6500 SUP720 platform, 139connectivity, 139–141

per-packet load sharing, performance issues, 188–189

punt adjacencies, 129–131single-path failures, 190–192

TTIB (Toaster Tag FIB), 217tunnel algorithm, 213

U-Vuncached adjacencies, 74–75unresolved adjacencies, 75Update Manager, 100

consistency checkersactive, 97–98passive, 97

verifyingadjacency table, 126–128ARP table, 108–111CEF table, 116–125connectivity on SUP720, 141–148memory on line cards, 261routing table, 111–115

VIPs (Versatile Interface Processors), 216VRF (virtual routing and forwarding) tables, 208

W-X-Y-Zwatchdog timer, 34

watchdog timer