Understanding and Troubleshooting HSRP

As is the case with all protocols and technologies, in order to troubleshoot something, you must have a solid understanding of how it works. In this section, we are going to revisit HSRP fundamentals, reinforcing the material covered in the SWITCH exam, and then conclude the section by discussing some common HSRP problems and how to troubleshoot them.

Hot Standby Router Protocol Overview

Hot Standby Router Protocol is a Cisco-proprietary First Hop Redundancy Protocol that allows physical gateways that are configured as part of the same HSRP group to share the same virtual gateway address. Network hosts residing on the same subnet as the gateways are configured with the virtual gateway IP address as their default gateway. Multiple HSRP groups can be configured under the same interface, allowing for load sharing of traffic between the gateways.

While operational, the primary gateway forwards packets destined to the virtual gateway IP address of the HSRP group. In the event that the primary gateway fails, the secondary gateway assumes the role of primary and forwards all packets sent to the virtual gateway IP address. However, it is important to remember that this only happens if the secondary gateway is configured to preempt. If preemption is not configured, then the secondary gateway will never assume the role of the active gateway.

Cisco IOS software supports two versions of HSRP: version 1 and version 2. By default, when Hot Standby Router Protocol is enabled in Cisco IOS software, version 1 is enabled. The following section lists and describes the operational differences between these two versions:

1. HSRPv1 restricts the number of configurable HSRP groups to 255, whereas the version 2 group numbers have been extended from 0 to 4095
2. HSRPv1 routers communicate by sending messages to Multicast group address 224.0.0.2 using UDP port 1985.  HSRPv2 routers communicate by sending messages to Multicast group address 224.0.0.102 using UDP port 1985
3. The version 2 packet format uses a Type/Length/Value (TLV) format. HSRP version 2 packets received by an HSRP version 1 router will have the type field mapped to the version field by HSRP version 1 and subsequently ignored
4. Although HSRPv1 advertises timer values, these values are always to the whole second as it is not capable of advertising or learning millisecond timer values. HSRPv2 is capable of both advertising and learning millisecond timer values
5. Version 2 provides improved management and troubleshooting by including a 6-byte Identifier field which is populated with the physical router interface MAC address and is used to uniquely identify the source of HSRP active Hello messages. In version 1, these messages contain the Virtual MAC address as the source MAC, which means it is not possible to determine which HSRP router actually sent the HSRP Hello message
6. In HSRP version 1, the Layer 2 address that is used by the virtual IP address will be a virtual MAC address composed of 0000.0C07.ACxx, where xx is the HSRP group number in Hexadecimal value and based on the respective interface. HSRP version 2, however, uses a new MAC address range 0000.0C9F.F000 to 0000.0C9F.FFFF for the virtual gateway IP address

Troubleshooting Hot Standby Router Protocol

The majority of HSRP issues are due to router and switch misconfigurations. For this reason, it is important to have an intimate understanding of the operation of this protocol, so as to be able to quickly identify any HSRP issues. In addition to misconfigurations, device resource utilization and Physical and Data layer issues can also affect the operation of HSRP. While it is not possible to delve into specifics on all possible HSRP issues, the following section describes the common HSRP problem scenarios and provides recommended solutions for them:

1. Gateway Logging Continuous HSRP State Changes
2. HSRP Gateways Not Reflecting the Correct State
3. HSRP Does Not Detect Peer Router
4. HSRP Causes MAC Violation on a Secure Switch Port

One of the most common problems experienced in networks running HSRP is continuous state changes. If you recall, from the SWITCH guide, we learned that when HSRP is enabled on an interface, the gateway interface goes through a series of states. These states are:

1. Disabled
2. Init
3. Listen
4. Speak
5. Standby
6. Active

The most common messages seen after HSRP has been configured are transitions between the active and standby state. During the speak phase, the standby gateway exchanges messages with the active gateway. Upon completion of this phase, the primary gateway transitions to the active state, and the backup gateway transitions to the standby state.  The standby state indicates that the gateway is ready to assume the role of active gateway if the primary gateway fails, and the active state indicates that the gateway is ready to actively forward packets. Continuous state transitions between active and standby result in the following log messages:

Such error messages indicate a situation in which a standby HSRP router did not receive three successive HSRP hello packets from its HSRP peer and therefore assumes the role of active router. There are several possible causes for the loss of HSRP packets between the peers. The most common reasons for these errors include the following:

1. Physical Layer Problems
2. Data Link Problems
3. Excessive Traffic
4. Gateway Resource Issues

Physical layer issues, e.g. flapping interfaces, errors, etc, may prevent HSRP packets from being successfully sent between the peers. You can look for link transitions by filtering log output to include only link up/down messages and then troubleshooting the identified links using the show interfaces command as was described earlier in this guide. For example, you can check the logs on the local gateway and if there is nothing there indicating link failure, proceed and check the logs of the peer. Keep in mind that the devices might be connected to different switches, so it is important to verify all links in the path between the gateways.

Layer 2 problems, such as Spanning Tree issues, can cause a loss of HSRP messages between the peers. STP loops can cause Broadcast storms, duplicated frames, and MAC table inconsistency. All of these problems affect the entire network, and especially HSRP. HSRP error messages can be the first indication of an STP issue. As stated in the previous chapter, when troubleshooting STP, it is important to understand the network topology, which includes, the location of the root bridge, backup root bridge (if applicable) blocking or redundant ports, and forwarding ports.

And finally, HSRP state changes are often due to High CPU utilization on the gateway. You can use the show processes cpu command to troubleshoot and identify CPU utilization issues. Filter the output of this command to show processes that are utilizing the CPU. If the issue is a result of high CPU utilization, troubleshoot the problem using the appropriate tools, which may include putting a sniffer on the network and the trace the system that is causing this utilization.

A commonly encountered issue following HSRP implementation is that the standby gateway never becomes the active gateway following the primary gateway failure or that the primary gateway never re-assumes its role after it is back online. These issues are almost always attributed to device misconfigurations. As stated earlier in the previous section, by default HSRP does not preempt. This means that even if the standby HSRP priority is higher than that of the primary, if preemption is not configured, it will never become active.

Similarly, if the active gateway fails and is restored, if preemption is not configured, it will not re-assume its previous role even though it has a higher priority than the standby gateway.
Consider the following output which shows the output of the show standby command:

From the output above, we can determine that Cat-6500-1 is the active gateway for HSRP group 1. This switch has been configured with a priority of 105 for the group, and is also configured to preempt. In the event that the switch or SVI fails and is restored, it will become the active gateway again because it is configured to preempt. Continuing with the example, the following shows the output of the show standby command on the current standby gateway:

From an outright failure standpoint, e.g. if the entire switch fails or the switch SVI goes down, the configuration will work because if the active gateway is no longer present, the standby gateway simply becomes the active gateway. No preemption is required in such situations. The issue arises when the active gateway does not fail, but decrements its priority based on additional configuration such as interface tracking, for example. Assume, for example, that the configuration on the gateway is modified to include tracking as follows:

Following this configuration, the GigabitEthernet0/1 interface on the active gateway is disabled. The show standby command on this device now shows the following:

Following this, the show standby command on the standby router now shows the below:

Notice that even with the high priority of 100, versus that on the active router which is 95 after the tracking configuration decremented the configured priority by 10 (default), the second gateway does not become the active gateway for the group because it has not been configured to preempt. This is a common mistake that is made when configuring HSRP.

NOTE: When configuring HSRP, ensure that preemption is configured on both the primary as well as the standby gateway. You can verify if preemption is configured by looking for coup messages in the output of the debug standby command as illustrated in the following output:

If preemption is not configured, the output of the debug standby would show the following:

In the debug output above, the local HSRP gateway first receives a Hello from the remote HSRP gateway that indicates a priority value of 95. The local HSRP advertises its priority value of 100. Next, the priority on the remote gateway is changed to 105 as illustrated in the received Hello. However, because preemption is not enabled, there is no state change because the remote HSRP gateway does not send out a coup message. Given this, the local gateway remains the active gateway even though it has a lower HSRP priority value than that of the remote or peer device.
There are two primary reasons why an HSRP gateway does not recognize its peer. The first is due to a lack of connectivity between the two devices and the second is due to device misconfigurations. When an HSRP gateway does not recognize its peer, the output of the show standby command displays a message similar to the following:

The router output in this section indicates that the gateway is configured for HSRP but does not recognize its HSRP peers. In order for this to occur, the router must fail to receive HSRP hellos from the neighbor router. This can be viewed using the debug standby command as follows:

Following the output of the debugs, the local gateway sends out a Hello sourced from its local IP address of 10.0.0.1 advertising a priority value of 105 and reflecting the VIP address of 10.0.0.254. The local gateway doesn’t hear anything from its peer and transitions to the standby state. Again the local gateway hears nothing from its peer and transitions to the active state. Also notice that there are no received HSRP packets from any other devices.

When troubleshooting this issue, first verify Physical layer connectivity between the gateways. Use the show interfaces command on the local router to verify Physical layer status. Also keep in mind that Data Link issues, e.g. Spanning Tree and VLAN issues can also cause connectivity issues between the gateways. If there is connectivity between the two gateways, then check for HSRP misconfigurations on the devices.

In networks using port security, it is not uncommon for HSRP to cause MAC violations due to misconfigurations on the switches. By default, when port security is enabled on a switch port, only a single secure MAC address is permitted. When port security is configured on the switch ports that are connected to the HSRP enabled routers, it causes a MAC violation, since you cannot have the same secure MAC address on more than one interface. A security violation will occur on a secure port in one of these situations:

1. If the maximum number of secure MAC addresses is added to the address table, and a station whose MAC address is not in the table attempts to access the interface
2. If an address that is learned or configured on one secure interface is seen on another secure interface in the same VLAN

By default, a port security violation causes the switch interface to become error-disabled and to shut down immediately, which blocks the HSRP status messages between the routers. Two solutions can be applied to resolve this issue. The first is to configure HSRP using the standby use-bia interface configuration command which forces the gateways to use the interface MAC instead of the virtual MAC for HSRP. The second alternative is to simply disable port security on the ports connected to devices running HSRP.