MLS troubleshooting requires troubleshooting at both the control and data planes. The control plane troubleshooting is the same as that performed on routers. MLS troubleshooting should follow a systematic approach, which involves checking the control plane, i.e. routing information, and then verifying the data or forwarding plane information. The following basic steps should be taken when troubleshooting Unicast Multilayer switching issues:
- Verify that IP routing information for the address is correct
- Verify that the next hop has a valid MAC address
- Verify that the FIB next hop is the same as the RIB next hop
- Verify the CEF adjacency table rewrite information
- Verify FIB and adjacency table population in TCAM
The first step is to verify that the destination address is present in the routing table. This step is performed using the show ip route command as follows:
| Cat-6500-1#show ip route 0.0.0.0 0.0.0.0 Routing entry for 0.0.0.0/0, supernet Known via “static”, distance 1, metric 0, candidate default path Routing Descriptor Blocks: * 10.10.10.1 Route metric is 0, traffic share count is 1 |
This first step is used to ensure that there is a route to the intended destination network and the route has a valid next hop address. If the route does not exist or the next-hop address is incorrect, troubleshooting of routing protocol, next-hop interfaces, or route configuration will be required. The second step is to verify that the next hop address has a valid next hop MAC address using theshow ip arp command. If the ARP entry for the next hop address is incomplete, then you will need to troubleshoot ARP. Following is the output of the show ip arpcommand for the next hop address shown in the previous output:
| Cat-6500-1#show ip arp | include 10.105.30.2
Internet 10.10.10.1 2 000f.20da.833d ARPA Vlan4 |
If the MAC address is incorrect, you need to verify whether or not another device owns that IP address. You can determine the MAC address of the next hop device using the show interfacescommand if it is a Cisco IOS device. In Cisco Catalyst 6500 series switches, you can use the Layer 2 Traceroute utility which is invoked using the traceroute mac <source mac> <destination mac> privileged EXEC command.
If the ARP entry is incomplete, it means that you did not get any replies from that host. In that case, you need to verify that the host is up and running. Following this, continue troubleshooting at the data plane to ensure that the same information is also present. Validate that the route entry in the FIB contains the same next hop address as in the first step by using theshow ip cefcommand. If there is a discrepancy between the two, a routing loop can be created. The output of this command for the same prefix shows the following entries:
| Cat-6500-1#show ip cef 0.0.0.0 0.0.0.0 detail 0.0.0.0/0, version 38, epoch 0, cached adjacency 10.10.10.1 0 packets, 0 bytes Flow: AS 0, mask 0 via 10.10.10.1, 0 dependencies, recursive next hop 10.10.10.1, Vlan4 via 10.10.10.1/32 (Default) valid cached adjacency |
And finally, verify that the CEF adjacency table contains the same rewrite information as the ARP table from Step 2 by using the show adjacency detailcommand as follows:
| Cat-6500-1#show adjacency detail | begin 10.10.10.1 IP Vlan4 10.10.10.1(7) 7834810780 packets, 1413540065564 bytes 000F20DA833D 000E39E29C000800 ARP 00:54:07 Epoch: 0 |
After performing these checks and correcting any identified issues, if you are still experiencing routing issues, you will need to verify the population of the FIB and adjacency table in TCAM using the show mls cef commands. TCAM troubleshooting should be performed under the supervision of a TAC engineer. TCAM is similar to CAM but allows for information to be looked up much faster. The TCAM is described in the SWITCH guide which is available online at www.howtonetwork.net.