There is no explicit means or method to actually troubleshoot switch supervisor redundancy. Instead, the troubleshooting process entails understanding the redundancy method implemented, and how it is supposed to function, i.e. what the norm is. From there, you can identify any anomalies with the switchover process. For the most part, if a supervisor redundancy mode is not functioning the way it should be, you should seek the assistance of the TAC. The following section describes the supported supervisor redundancy methods.
Understanding Switch Supervisor Redundancy
Cisco Catalyst 4500 and 6500 series switches support two supervisor modules or engines within the switch chassis to allow for high availability. When the switch boots up, the first supervisor that boots up is referred to as the primary or active supervisor engine and the second supervisor module is referred to as the standby or redundant supervisor engine. When the primary or active supervisor is up, it controls all switch functions, including management operations, data plane operations, and control plane operations. In the event that the active supervisor engine fails or is removed, the standby supervisor assumes this responsibility. The standby supervisor engine assumes primary supervisor engine status when one of the following events occurs:
- The primary supervisor engine fails or crashes
- The primary supervisor engine is rebooted
- The administrator forces a manual failover from active to standby
- The primary supervisor engine is physically removed
- Clock synchronization between the supervisor engines fails (SSO)
Cisco IOS software supports three modes for redundant supervisor engine implementations:
- Route Processor Redundancy (RPR)
- Route Processor Redundancy Plus (RPR+)
- Stateful Switchover (SSO)
With RPR, when the switch boots up, the RPR process runs between the two supervisor engines and the first supervisor to complete the boot process becomes the active supervisor engine and the second supervisor becomes the standby supervisor engine. When using RPR, the standby supervisor engine is only partially booted and initialized, meaning that not all switch subsystems become operational. For example, on the standby, the MSFC and PFC are not active.
Clock synchronization occurs between Primary and Backup every 60 seconds, and the startup configuration and configuration registers are synchronized between supervisors. When the active supervisor engine fails, the standby supervisor engine becomes operational and the following occurs within the switch:
- All switching modules are reloaded and powered up again
- Remaining subsystems on the MSFC are brought up
- ACLs are reprogrammed into supervisor engine hardware
Because the standby supervisor engine is not fully initialized, any failover from the active to the standby supervisor engine results in a disruption of network traffic as the standby supervisor engine goes through the steps listed above and assumes the active Supervisor role. This entire process takes generally takes between 2 and 4 minutes.
RPR+ improves on RPR and provides failover generally within 30 to 60 seconds. When RPR+ mode is used, the redundant supervisor engine is fully initialized and configured, but is not fully operational. When the redundant supervisor engine first initializes, the startup-config file is copied from the active supervisor engine to the redundant supervisor engine, which overrides any existing startup configuration file on the redundant supervisor engine, allowing the supervisor engines to become synchronized.
When switch configuration changes occur during normal operation, redundancy performs an incremental synchronization from the active supervisor engine to the redundant supervisor engine. RPR+ synchronizes user-entered CLI commands incrementally line-by-line from the active supervisor engine to the redundant supervisor engine.
Even though the redundant supervisor engine is fully initialized, it only interacts with the active supervisor engine to receive incremental changes to the configuration files. The console on the redundant supervisor engine is locked and you cannot enter CLI commands on the redundant supervisor engine. When the active supervisor engine fails, the redundant supervisor engine finishes initializing without reloading other switch modules and the following events occur on the switch:
- Traffic is disrupted until the Redundant Supervisor Engine completes the takes over
- The switch maintains any static routes across the switchover
- The switch does not maintain any dynamic routing protocol information
- The switch clears the FIB Tables on switchover
- The switch clears the CAM Tables on switchover
- State information, such as active TCP sessions, is not maintained on switchover
When implementing RPR+, it is important to ensure that the supervisor modules are similar, i.e. the same model, memory, and version for example, and that they are running the same Cisco IOS software version. If these are different, the switch will revert to RPR mode instead of RPR+.
SSO is the preferred redundancy mode for supervisor engines. Similar to RPR and RPR+, SSO establishes one of the supervisor engines as active while the other supervisor engine is designated standby. Unlike RPR and RPR+ however, with SSO, the redundant supervisor engine is fully booted and initialized and then SSO synchronizes the two supervisors.
With SSO, supervisor engines must be synchronized so that the redundant supervisor engine is always ready to assume control in the event that the active supervisor engine fails. Configuration information and data structures are synchronized between the supervisor engines at startup and whenever changes to the active supervisor engine configuration occur.
Unlike RPR and RPR+ redundancy, SSO maintains state information between the redundant supervisor engines. This includes forwarding information in the FIB, as well as adjacency entries, which ensures that Layer 2 traffic is not interrupted and the switch can still forward Layer 3 traffic after a switchover from the active to the redundant supervisor engine. When using SSO, the following events cause a switchover:
- A hardware failure on the active supervisor engine
- Clock synchronization failure between supervisor engines
- A manual switchover
During SSO switchover, all system control and routing protocol execution is transferred from the active supervisor to the standby supervisor engine within 0 to 3 seconds. Non-Stop Forwarding works with in conjunction with SSO to minimize the amount of time a network is unavailable to its users following a switchover while continuing to forward IP packets. NSF is primarily used to ensure the continued forwarding IP packets following a supervisor engine switchover and is supported by BGP, OSPF, EIGRP, IS-IS, as well as CEF.
NSF allows routing protocols to detect a switchover and take the necessary action to continue forwarding network traffic. NSF allows routing protocols to recover route information from the NSF-capable peer devices instead of waiting for the FIB to be rebuilt before the switch can actually begin forwarding traffic. This allows for high availability and resiliency during supervisor engine switchover. When NSF is implemented, routing protocols depend on CEF to continue forwarding packets during switchover while they build the Routing Information Base (RIB) tables. After the routing protocols have converged, CEF updates the FIB table and removes stale route entries. CEF also updates the switch modules with the new FIB information. Cisco NSF is configured on a per-routing protocol basis.