In the previous section, we discussed the use of three CLI commands that can be used for the troubleshooting of Physical Layer issues. This section describes some common approaches to identifying and troubleshooting intra-VLAN connectivity issues. Some of the more common causes of intra-VLAN connectivity issues include:
- Duplex Mismatches
- Bad NIC or Cable
- Congestion
- Hardware Issues
- Software Issues
- Resource Oversubscription
- Configuration Issues
Duplex mismatches can result in very slow network performance and connectivity. While some improvements in auto negotiation have been made, and the use of auto negotiation is considered valid practice, it is still possible for duplex mismatches to occur. An example of when this might happen is when the NIC is set to 100/Full and the switch port is auto negotiating. In this case, the NIC will retain its 100/Full setting, but the switch port will be set to 100/Half. Another example would the inverse; i.e. the NIC is set to auto negotiate while the switch port is set to 100/Full. In that case, the NIC would auto negotiate to 100/Half while the switch retained its static 100/Full configuration resulting in a duplex mismatch.
It is therefore good practice to manually specify speed and duplex settings for 10/100 Ethernet connections, where feasible, to avoid duplex mismatches with auto negotiation. Duplex mismatches can not only affect users directly connected to the switch but also network traffic that traverses inter-switch links that have mismatched duplex settings. The port interface speed and duplex settings can be viewed using the show interfaces command.
NOTE: Because Catalyst switches support only full duplex for 1Gbps links, this is commonly not an issue for GigabitEthernet connections.
There are multiple counters in Cisco IOS software that can be used to identify a potentially bad NIC or cabling issue. NIC or cabling issues can be identified by checking the values of certain counters in different show commands. For example, if the switch port counters show an incrementing number of frames with a bad CRC or FCS errors, this can most likely be attributed to either a bad NIC on the workstation or machine or a bad network cable.
Network congestion can also cause intermittent connectivity issues in the switched network.
The first sign that your VLAN is overloaded is if the Rx or Tx buffers on a port are oversubscribed. Additionally, excessive frame drops on a port can also be an indication of network congestion. A common cause of network congestion is due to underestimating aggregate bandwidth requirements for backbone connections. In such cases, congestion issues can be resolved by configuring Etherchannels or adding additional ports to existing Etherchannels. While network congestion is a common cause of connectivity issues, it is also important to know that the switch itself can experience congestion issues, which can have a similar impact on network performance.
Limited switch bandwidth can result in congestion issues, which can severely impact network performance. As you may recall, in the SWITCH guide we learned that in LAN switching, bandwidth refers to the capacity of the switch fabric. Therefore if the switch fabric is on 5Gbps and you attempt to push 7Gbps worth of traffic through the switch, the end result is a packet loss and poor network performance. This is a common issue in oversubscribed platforms where the aggregate capacity of all ports can exceed the total backplane capacity.
Hardware problems can also cause connectivity issues in the switched LAN. Examples of such issues include bad ports or bad switch modules. While you could troubleshoot such issues by looking at physical indicators such as LEDs, if possible, such issues are sometimes difficult to troubleshoot and diagnose. In most cases, you should seek the assistance of the TAC when you suspect potentially faulty hardware issues.
Software bugs are even more difficult to identify because they cause deviation, which is hard to troubleshoot. In the event that you suspect a software bug may be causing connectivity issues, you should contact the TAC with your findings. Additionally, if error messages are printed on the console or are in the logs, you can also use some of the online tools available from Cisco to implement a workaround or get a recommendation for a version of software in which the issue has been resolved and verified.
As with any other hardware device, switches have limited resources, such as physical memory. When these resources are oversubscribed, this can lead to severe performance issues. Issues such as high CPU utilization can have a drastic impact on both switch and network performance. Resource troubleshooting for IOS switches is described in the following chapter.
And finally, as with any other technology, incorrect configurations may also cause connectivity issues either directly or indirectly. For example, the poor placement of the root bridge may result in slow connectivity for users. Directly, integrating or adding an incorrectly configured switch into the production network could result in an outright outage for some or all users. The following sections describe some common VLAN-related issues, their probable causes and the actions that can be taken to remedy them.
Troubleshooting Dynamic VLAN Advertisement
Cisco Catalyst switches use VTP to dynamically propagate VLAN information throughout the switched domain. VTP is a Cisco proprietary Layer 2 messaging protocol that manages the addition, deletion, and renaming of VLANs for switches in the same VTP domain.
There are several reasons why a switch might not be able to dynamically receive any VLAN information when added to the VTP domain. Some common causes include the following:
- Layer 2 Trunking Misconfigurations
- Incorrect VTP Configuration
- Configuration Revision Number
- Physical Layer Issues
- Software or Hardware Issues or Bugs
- Switch Performance Issues
NOTE: For brevity, only trunking, VTP configuration, and the configuration revision number are described in the following section. Physical layer troubleshooting was described in the previous section. Software or hardware issues and bugs and switch performance issues will be described in the following chapter.
In order for switches to exchange VLAN information using VTP, a trunk must be established between the switches. Cisco IOS switches support both ISL and 802.1Q trunking mechanisms. While some switches default to ISL, which is a Cisco proprietary trunking mechanism, the current Cisco IOS Catalyst switches default to 802.1Q. When provisioning trunking between switches, it is considered good practice to manually specify the trunking encapsulation protocol. This is accomplished using the switchport trunk encapsulation [isl|dot1q] interface configuration command when configuring the link as a trunk port.
There are several commands that you can use to troubleshoot trunk connectivity issues. You can use the show interfaces command to verify basic port operational and administrative status. Additionally, you can append the trunk or errors keyword to perform additional troubleshooting and verification. The show interfaces [name] counters trunk command can be used to view the number of frames transmitted and received on trunk ports.
The output of this command also includes encapsulation errors, which can be used to verify 802.1Q and ISL and trunking encapsulation mismatches as illustrated in the following output:
| Cat-3550-1#show interfaces fastethernet 0/12 counters trunkPort TrunkFramesTx TrunkFramesRx WrongEncap Fa0/12 1696 32257 0 |
Referencing the output above, you can repeat the same command to ensure that both the Tx and Rx columns are incrementing and perform additional troubleshooting from there. For example, if the switch is not sending any frames, the interface might not be configured as a trunk, or might be down or disabled. If the Rx column is not incrementing, then it may be that the remote switch might not be configured correctly.
Another command that can be used to troubleshoot possible Layer 2 trunk misconfigurations is the show interfaces [name] trunk command. This output of this command includes the trunking encapsulation protocol and mode, the native VLAN for 802.1Q, the VLANs that are allowed to traverse the trunk, VLANs that are active in the VTP domain, as well as VLANs that are pruned. A common issue with VLAN propagation is that the upstream switch has been configured to filter certain VLANs on the trunk link using the switchport trunk allowed vlaninterface configuration command. Following is the output of the show interfaces [name] trunk command:
| Cat-3550-1#show interfaces trunkPort Mode Encapsulation Status Native vlan Fa0/12 desirable n-802.1q trunking 1 Fa0/13 desirable n-802.1q trunking 1 Fa0/14 desirable n-isl trunking 1 Fa0/15 desirable n-isl trunking 1Port Vlans allowed on trunk Fa0/12 1-4094 Fa0/13 1-4094 Fa0/14 1-4094 Fa0/15 1-4094Port Vlans allowed and active in management domain Fa0/12 1-4 Fa0/13 1-4 Fa0/14 1-4 Fa0/15 1-4Port Vlans in spanning tree forwarding state and not pruned Fa0/12 1-4 Fa0/13 none Fa0/14 none Fa0/15 none |
Another common trunking misconfigurations issue is native VLAN mismatches. When you are configuring 802.1Q trunks, the native VLAN must match on both sides of the trunk link; otherwise the link will not work. If there is a native VLAN mismatch, STP places the port in a port VLAN ID (PVID) inconsistent state and will not forward on the link. In such cases, an error message similar to the following will be printed on the console or in the log:
| *Mar 1 03:16:43.935: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 1 on FastEthernet0/11 VLAN2. *Mar 1 03:16:43.935: %SPANTREE-2-BLOCK_PVID_PEER: Blocking FastEthernet0/11 on VLAN0001. Inconsistent peer vlan. *Mar 1 03:16:43.935: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking FastEthernet0/11 on VLAN0002. Inconsistent local vlan. *Mar 1 03:16:43.935: %SPANTREE-2-RECV_PVID_ERR: Received BPDU with inconsistent peer vlan id 1 on FastEthernet0/12 VLAN2. *Mar 1 03:16:43.935: %SPANTREE-2-BLOCK_PVID_PEER: Blocking FastEthernet0/12 on VLAN0001. Inconsistent peer vlan. *Mar 1 03:16:43.939: %SPANTREE-2-BLOCK_PVID_LOCAL: Blocking FastEthernet0/12 on VLAN0002. Inconsistent local vlan. |
While STP troubleshooting will be described later in this section, this inconsistent state could be validated using the show spanning-tree command as illustrated below:
| Cat-3550-1#show spanning-tree interface fastethernet 0/11Vlan Role Sts Cost Prio.Nbr Type ——————- —- — ——— ——– —————————- VLAN0001 Desg BKN*19 128.11 P2p *PVID_Inc VLAN0002 Desg BKN*19 128.11 P2p *PVID_Inc |
If you have checked and validated that the trunk is indeed correctly configured and operational between the two switches, the next step would be to validate VTP configuration parameters. These parameters include the VTP domain name, the correct VTP mode, and the VTP password, if one has been configured for the domain using the show vtp status and show vtp passwordcommands, respectively. Below is the output of the show vtp status command:
| Cat-3550-1#show vtp status VTP Version : running VTP2 Configuration Revision : 0 Maximum VLANs supported locally : 1005 Number of existing VLANs : 8 VTP Operating Mode : Server VTP Domain Name : TSHOOT VTP Pruning Mode : Enabled VTP V2 Mode : Enabled VTP Traps Generation : Disabled MD5 digest : 0x26 0x99 0xB7 0x93 0xBE 0xDA 0x76 0x9C … [Truncated Output] |
When using the show vtp status command, check to ensure that the switches are running the same version of VTP. By default, Catalyst switches run VTP version 1. A switch running VTP version 1 cannot participate in a VTP version 2 domain. If the switch is incapable of running VTP version 2, then all VTP version 2 switches should be configured to run version 1 instead using thevtp version global configuration command.
NOTE: If you change the VTP version on the server, the change is propagated to client switches in the VTP domain automatically.
As was described in the SWITCH guide, VTP propagation is enabled for VTP client/server or server/server devices. If VTP is disabled on a switch, i.e. transparent mode, the switch will not dynamically receive VLAN information via VTP. However, be mindful of the fact that with version 2, transparent mode switches will forward received VTP advertisements out their trunk ports and act as VTP relays. This happens even if the VTP version is not the same.
The VTP domain name should also be consistent on the switches. And finally, the output of theshow vtp status command also includes the MD5 hash used for authentication purposes. This hash, which is derived from the VTP domain name and password, should be consistent on all switches in the domain. If the VTP passwords or domain names are different on the switches, the calculated MD5 has will also be different. If the domain name or password is different, theshow vtp status command will indicate an MD5 digest checksum mismatch as shown in the following output:
| Cat-3550-1#show vtp status VTP Version : running VTP2 Configuration Revision : 0 Maximum VLANs supported locally : 1005 Number of existing VLANs : 8 VTP Operating Mode : Server VTP Domain Name : TSHOOT VTP Pruning Mode : Enabled VTP V2 Mode : Enabled VTP Traps Generation : Disabled MD5 digest : 0x26 0x99 0xB7 0x93 0xBE 0xDA 0x76 0x9C *** MD5 digest checksum mismatch on trunk: Fa0/11 *** *** MD5 digest checksum mismatch on trunk: Fa0/12 *** … [Truncated Output] |
And finally, the configuration revision number can wreak havoc when using VTP. Switches use the configuration revision number to keep track of the most recent information in the VTP domain. Every switch in the domain stores the configuration revision number that it last heard from a VTP advertisement and this number is incremented every time new information is received. When any switch in the VTP domain receives an advertisement message with a higher configuration revision number than its own, it will overwrite any stored VLAN information and synchronize its own stored VLAN information with the information received in the advertisement message.
Therefore, if you are wondering why the switch that you integrated into the VTP domain is not receiving any VLAN information, it may be that the same switch had a higher configuration revision number and caused all other switches to overwrite their local VLAN information and replace it with the information received in the advertisement message from the new switch. To avoid such situations, always ensure that the configuration revision number is set to 0 prior to integrating a new switch into the domain. This can be done by changing the VTP mode or changing the VTP domain name on the switch. The configuration revision number is included in the output of the show vtp status command.
Troubleshooting Loss of End-to-End Intra-VLAN Connectivity
There are several possible reasons for a loss of end-to-end connectivity within a VLAN. Some of the most common causes include the following:
- Physical Layer Issues
- VTP Pruning
- VLAN Trunk Filtering
- New Switches
- Switch Performance Issues
- Network Congestion
- Software or Hardware
NOTE: For brevity, only trunking, VTP pruning, trunk filtering, and the integration of new switches into the domain will be described in this section. Software or hardware issues and bugs and switch performance issues are described in the following chapter. Physical layer troubleshooting was described in the earlier in this chapter.
VTP pruning removes VLANs from the VLAN database of the local switch when no local ports are a part of that VLAN. VTP pruning increases the efficiency of trunks by eliminating unnecessary Broadcast, Multicast and unknown traffic from being flooded across the network.
While VTP pruning is a desirable feature to implement, incorrect configuration or implementation can result in a loss of end-to-end VLAN connectivity. VTP pruning should only be enabled in client/server environments. Implementing pruning in a network that includes transparent mode switches may result in a loss of connectivity. If one or more switches in the network are in VTP transparent mode, you should either globally disable pruning for the entire domain or ensure that all VLANs on the trunk link(s) to the upstream transparent mode switch(es) are pruning ineligible, i.e. they are not pruned, using the switchport trunk pruning vlan interface configuration command under the applicable interfaces.
In addition to VTP pruning, the incorrectly filtering VLANs on switch trunk links can result in a loss of end-to-end VLAN connectivity. By default, all VLANs are allowed to traverse all trunk links; however, Cisco IOS software allows administrators to selectively remove (or add) VLANs to specific trunk links using the switchport trunk allowed vlan interface configuration command. Use the show interfaces [name] trunk and the show interfaces [name] switchportcommands to view pruned and restricted VLANs on trunk links. Following is the output of theshow interfaces [name] trunk command:
| Cat-3550-1#show interfaces trunkPort Mode Encapsulation Status Native vlan Fa0/1 on 802.1q trunking 1 Fa0/2 on 802.1q trunking 1Port Vlans allowed on trunk Fa0/1 1,10,20,30,40,50 Fa0/2 1-99,201-4094Port Vlans allowed and active in management domain Fa0/1 1,10,20,30,40,50 Fa0/2 1,10,20,30,40,50,60,70,80,90,254Port Vlans in spanning tree forwarding state and not pruned Fa0/1 1,10,20,30,40,50 Fa0/2 1,40,50,60,70,80,90,254 |
Following is the output of the show interfaces [name] switchport command on a port that has been statically configured as an 802.1Q trunk link:
| Cat-3550-2#show interfaces fastethernet 0/7 switchport Name: Fa0/7 Switchport: Enabled Administrative Mode: trunk Operational Mode: trunk Administrative Trunking Encapsulation: dot1q Operational Trunking Encapsulation: dot1q Negotiation of Trunking: On Access Mode VLAN: 1 (default) Trunking Native Mode VLAN: 1 (default) Administrative Native VLAN tagging: enabled Voice VLAN: none Administrative private-vlan host-association: none Administrative private-vlan mapping: none Administrative private-vlan trunk native VLAN: none Administrative private-vlan trunk Native VLAN tagging: enabled Administrative private-vlan trunk encapsulation: dot1q Administrative private-vlan trunk normal VLANs: none Administrative private-vlan trunk associations: none Administrative private-vlan trunk mappings: none Operational private-vlan: none Trunking VLANs Enabled: 3,5,7 Pruning VLANs Enabled: 2-8 Capture Mode Disabled Capture VLANs Allowed: ALLProtected: false Unknown unicast blocked: disabled Unknown multicast blocked: disabled Appliance trust: none |
As was described in the previous section, the integration of a new switch into the network can result in a loss of VLAN information in the management domain. This loss of VLAN information can result in a loss of connectivity between devices within the same VLAN. Ensure that the configuration revision number is reset prior to integrating a new switch into the LAN.