Cisco IOS switches support several commands that can be used to troubleshoot Layer 1 or at least, suspected, Layer 1 issues. However, in addition to being familiar with the software command suite, it is also important to have a solid understanding of physical indicators, i.e. LEDs, which can be used to link status or can indicate an error condition.
Troubleshooting Link Status Using Light Emitting Diodes (LEDs)
If you have physical access to the switch or switches, LEDs can be a useful troubleshooting tool. Different Cisco Catalyst switches provide different LED capabilities. Understanding the meaning of the LEDs is an integral part of Catalyst switch link status and system troubleshooting. Cisco Catalyst switches have front-panel LEDs that can be used to determine link status as well as other variables, such as system status. The supported LEDs, as well as what they indicated, are listed and described in Table 3-1:
| LED | Color or State | Description |
|---|---|---|
| Status | Green | The color green indicates that all diagnostics have passed and that the module is operational |
| Orange | The color orange indicates that the module is booting or running diagnostics. This color could also indicate an over temperature condition, i.e. a minor temperature threshold has been exceeded during environmental monitoring | |
| Red | The color red indicates that the module is resetting. The switch has just been powered on or the module has been hot inserted during the normal initialization sequence. The color red could also indicate an over temperature condition, i.e. a major temperature threshold has been exceeded during environmental monitoring. If the module fails to download code and configuration information successfully during the initial reset, the LED stays red; the module does not come online | |
| Off | If the status LED is off, this indicates that the module is not receiving power or had been powered off | |
| LINK | Green | The color green indicates that the port is active and the link is connected and operational |
| Orange | The color orange indicates that the port is disabled through the CLI command, i.e. the shutdown command is configured | |
| Flashing orange | A flashing orange LED indicates that the port is faulty and has been disabled by the system | |
| Off | If the link LED is off, this indicates that the port is not active or the link is not connected. It does not mean no cable is connected | |
| PHONE | Green | If the phone LED is green, this indicates that the voice daughter card is installed |
| Off | If the phone LED is off, this indicates that the voice daughter card is not detected or is not installed |
Another popular Catalyst switch, the Catalyst 4500 series modular switch, also has LEDs that can be used for link and system troubleshooting. The Ethernet module LEDs, as well as the meanings of these LEDs, are listed and described in Table 3-2:
| LED | Color or State | Description |
|---|---|---|
| Status | Green | The color green indicates that all diagnostic tests have passed |
| Red | The color red indicates that a test, other than an individual port test, has failed | |
| Orange | The color orange indicates system boot up, that self-test diagnostics running, or that the module is disabled | |
| Link | Green | The color green indicates that the port is operational and that a signal has been detected |
| Orange | The color orange indicates that the link has been disabled by software, i.e. the shutdown command has been issued | |
| Flashing orange | A flashing orange LED indicates that the link has been disabled due to a hardware failure | |
| Off | If the link LED is off, this means no signal has been detected. It does not necessarily mean that the port is not connected | |
| Port Status | Green | The color green indicates that the port is operational and that a signal has been detected |
| Orange | The color orange indicates that the link or port has been disabled by software | |
| Flashing orange | A flashing orange color indicates that the link has been disabled due to a hardware failure | |
| Off | If the LED is off, this indicates that no signal is detected |
In addition to understanding what the different LED colors mean, it is also important to have an understanding of what action to take to remedy the issue. For example, assume that you are troubleshooting a Catalyst 6500 series switch and you notice that the status LEDs on the supervisor engine or any switching modules is red or off. In such cases, it is possible that the module might have shifted out of its slot, or, in the event of a new module, was not correctly inserted into the chassis. In this case, the recommended action is to reseat the module. In some cases, it may also be necessary to reboot the entire system.
While a link or port LED color other than green typically indicates some kind of failure or other issue, it is important to remember that a green link light does not always mean that the cable is fully functional. For example, a single broken wire or one shutdown port can cause the problem where one side has a link light, but the other side does not. This could be because the cable encountered physical stress that causes it to be functional at a marginal level. In such cases, the CLI can be used to perform additional troubleshooting.
Using the Command Line Interface to Troubleshoot Link Issues
There are several CLI commands that can be used to troubleshoot Layer 1 issues on Cisco IOS Catalyst switches. Commonly used commands include the show interfaces, the show controllers, and the show interface [name] counters errors commands. In addition to knowing these commands, you are also required to be able to accurately interpret the output or information that these commands provide.
The show interfaces command is a powerful troubleshooting tool that provides a plethora of information, which includes the following information:
- The administrative status of a switching port
- The port operational state
- The media type (for select switches and ports)
- Port input and output packets
- Port buffer failures and port errors
- Port input and output errors
- Port input and output queue drops
Following is the output of the show interfaces command for a GigabitEthernet switch port:
| Catalyst-3750-1#show interfaces gigabitethernet 3/0/1 GigabitEthernet0/1 is down, line protocol is down (notconnect) Hardware is Gigabit Ethernet, address is 000f.2303.2db1 (bia 000f.2303.2db1) MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive not set Auto-duplex, Auto-speed, link type is auto, media type is unknown input flow-control is off, output flow-control is desired ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output never, output hang never Last clearing of “show interface” counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 no buffer Received 0 broadcasts (0 multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 0 input packets with dribble condition detected 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out |
Most Cisco Catalyst switch ports default to the ‘notconnect‘ state as illustrated in the first line of the output printed by this command. However, a port can also transition to this state if a cable is removed from the port or is not correctly connected. This status is also reflected when the connected cable is faulty or when the other end of the cable is not connected to an active port or device, e.g. if a workstation connected to the switch port is powered off.
NOTE: When troubleshooting GigabitEthernet ports, this port status may also be a result of incorrect GBICs being used between the two ends.
The first part of the output in the first line printed by this command, i.e.[interface] is up refers to the Physical layer status of the particular interface. The second part of the output, i.e. line protocol is down, indicates the Data Link layer status of the interface. If this indicates an ‘up‘, it means that the interface can send and receive keepalives. Keep in mind that it is possible for the switch port to indicate that the Physical layer is up, while the Data Link layer is down, for example, such as when the port is a SPAN destination port or if the local port is connected to a CatOS switch with its port disabled.
The input queue indicates the actual number of frames dropped because the max queue size was exceeded. The flushes column counts Selective Packet Discard (SPD) drops on the Catalyst 6000 series switches. SPD drops low priority packets when the CPU is overloaded in order to save some processing capacity for high priority packets. The flushes counter in the show interface command output increments as part of SPD, which implements a selective packet drop policy on the IP process queue of the router. Therefore, it applies to only process switched traffic. The different Cisco IOS switching methods will be described again later in this guide.
The total output drops indicates the number of packets dropped because the output queue is full. This is often seen when traffic from a multiple inbound high bandwidth links, e.g. GigabitEthernet links, being switched to a single outbound lower bandwidth link, e.g. a FastEthernet link. The output drops increment because the interface is overwhelmed by the excess traffic due to the speed mismatch between the inbound and outbound bandwidths.
In addition to the show interfaces command, the show interfaces [name] counters errorscommand can also be used to view interface errors and facilitate Layer 1 troubleshooting. Following is the output that is printed by the show interfaces [name] counters errorscommand:
| Catalyst-3750-1#show interfaces gigabitethernet 3/0/1 counters errorsPort Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize Gi3/0/1 0 0 0 0 0Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants Gi3/0/1 0 0 0 0 0 0 0 |
The following section describes some of the error fields included in the output of the show interfaces [name] counters command and what issues or problems non-zero values under these fields may indicate.
The Align-Err field reflects a count of the number of frames received that don’t end with an even number of octets and have a bad CRC. These errors are usually the result of a duplex mismatch or a physical problem, such as cabling, a bad port, or a bad NIC. When the cable is first connected to the port, some of these errors can occur. Also, if there is a hub connected to the port, collisions between other devices on the hub can cause these errors.
The FCS-Err field reflects the number of valid size frames with Frame Check Sequence (FCS) errors but no framing errors. This is typically a physical issue, such as cabling, a bad port, or a bad NIC. Additionally, a non-zero value under this field could indicate a duplex mismatch.
A non-zero value in the Xmit-Err field is an indication that the internal send (Tx) buffer is full. This is commonly seen when traffic from a multiple inbound high bandwidth links, e.g. GigabitEthernet links, being switched to a single outbound lower bandwidth link, such as a FastEthernet link, for example.
The Rcv-Err field indicates the sum of all receive errors. This counter is incremented when the interface receives an error like a runt, giant or FCS-Err, for example.
The Undersize field is incremented when the switch receives frames that are smaller than 64-bytes in length. This is commonly caused by a faulty sending device.
The various collisions fields indicate collisions on the interface. This is common for half-duplex Ethernet, which is almost non-existent in modern networks. However, these counters should not increment for full-duplex links. In the event that non-zero values are present under these counters, this typically indicates a duplex mismatch issue. When a duplex mismatch is detected, the switch prints a message similar to the following on the console or in the log:
%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/1 (not full duplex), with R2 FastEthernet0/0 (full duplex)
As will be described in the section pertaining to Spanning Tree protocol, duplex mismatches can cause STP loops in the switched network if a port is connected to another switch. These mismatches can be resolved by manually configuring the speed and duplex of switch ports.
The Carri-Sen (carrier sense) counter increments every time an Ethernet controller wants to send data on a half duplex connection. The controller senses the wire and checks if it is not busy before transmitting. A non-zero value under this field indicates the interface is operating in half-duplex mode. This is normal for half-duplex.
Non-zero values can also be seen under the Runts field due to a duplex mismatch or because of other Physical layer problems, such as a bad cable, port, or NIC on the attached device. Runts are received frames that are smaller than the minimum IEEE 802.3 frame size, which is 64 bytes for Ethernet, and with a bad CRC.
And finally, the Giants counter is incremented when frames are received that exceed the IEEE 802.3 maximum frame size, which is 1518 bytes for non-jumbo Ethernet, and have a bad FCS. For ports or interfaces connected to a workstation, a non-zero value in this counter is typically caused by a bad NIC on the connected device. However, for ports or interfaces that are connected to another switch, e.g. via a trunk link, this field will contain a non-zero value if 802.1Q encapsulation is used. With 802.1Q, the tagging mechanism implies a modification of the frame because the trunking device inserts a 4-byte tag and then re-computes the FCS.
Inserting a 4-byte tag into a frame that already has the maximum Ethernet size creates a 1522-byte frame that can be considered a baby giant frame by the receiving equipment. Therefore, while the switch will still process such frames, this counter will increment and contain a non-zero value. To resolve this issue, the 802.3 committee created a subgroup called 802.3ac to extend the maximum Ethernet size to 1522 bytes; however, it is not uncommon to see a non-zero value under this field when using 802.1Q trunking.
And finally, the show controllers ethernet-controller <interface> command can also be used to display traffic counter and error counter information similar to that printed by the show interfaces and show interfaces <name> counters commands. Following is the output of theshow controllers ethernet-controller <interface> command:
| Catalyst-3750-1#show controllers ethernet-controller gigabitethernet 3/0/1Transmit GigabitEthernet3/0/1 Receive 4069327795 Bytes 3301740741 Bytes 559424024 Unicast frames 376047608 Unicast frames 27784795 Multicast frames 1141946 Multicast frames 7281524 Broadcast frames 1281591 Broadcast frames 0 Too old frames 429934641 Unicast bytes 0 Deferred frames 226764843 Multicast bytes 0 MTU exceeded frames 137921433 Broadcast bytes 0 1 collision frames 0 Alignment errors 0 2 collision frames 0 FCS errors 0 3 collision frames 0 Oversize frames 0 4 collision frames 0 Undersize frames 0 5 collision frames 0 Collision fragments 0 6 collision frames 0 7 collision frames 257477 Minimum size frames 0 8 collision frames 259422986 65 to 127 byte frames 0 9 collision frames 51377167 128 to 255 byte frames 0 10 collision frames 41117556 256 to 511 byte frames 0 11 collision frames 2342527 512 to 1023 byte frames 0 12 collision frames 5843545 1024 to 1518 byte frames 0 13 collision frames 0 Overrun frames 0 14 collision frames 0 Pause frames 0 15 collision frames 0 Excessive collisions 0 Symbol error frames 0 Late collisions 0 Invalid frames, too large 0 VLAN discard frames 18109887 Valid frames, too large 0 Excess defer frames 0 Invalid frames, too small 264522 64 byte frames 0 Valid frames, too small 99898057 127 byte frames 76457337 255 byte frames 0 Too old frames 4927192 511 byte frames 0 Valid oversize frames 21176897 1023 byte frames 0 System FCS error frames 127643707 1518 byte frames 0 RxPortFifoFull drop frame 264122631 Too large frames 0 Good (1 coll) frames 0 Good (>1 coll) frames |
NOTE: The output above will vary slightly depending on the switch platform on which this command is executed. For example, Catalyst 3550 series switches also included a Discarded frames field which shows the total number of frames whose transmission attempt is abandoned due to insufficient resources. A large number in this field typically indicates a network congestion issue. In the output above, you would look at the RxPortFifoFull drop frame field which indicates the total number of frames received on an interface that are dropped because the ingress queue if full.