Troubleshooting Converged Networks

Unlike traditional data networks, voice and video troubleshooting typically involves numerous components, which can make troubleshooting a complex process. For example, voice issues could be due to gateway, gatekeeper or call agent misconfigurations or errors. Troubleshooting these components requires familiarity with all of them. Fortunately, from a networking perspective, your troubleshooting will center on voice and video quality issues, which may arise from misconfigurations or errors on the infrastructure devices, i.e. the routers and switches that transport this traffic.

For the most part, almost all voice or video quality issues are caused by some type of network degradation. While poor network performance might not affect data traffic, it does have an adverse impact on delay-sensitive traffic such as voice and video. The sections that follow list some voice troubleshooting targets and how to troubleshoot problems in these areas.

Troubleshooting IP Telephony Implementations

There are several troubleshooting targets that you should focus on when troubleshooting voice issues. These troubleshooting targets include the following:

• Power over Ethernet
• Switch Port Configuration
• IP Services
• Packet Loss and Congestion
• Queuing

As stated earlier in this chapter, in most VoIP implementations, LAN switches provide power to the IP phones. For this reason, it is important to ensure that the switch can provide sufficient power to the number of phones that will be connected to it. Consider using the Cisco Power Calculator to calculate power-supply requirements for a specific PoE configuration. The tool can be used to calculate power requirements for Catalyst 6500, Catalyst 4500-E/4500, Catalyst 3750-E/3750, Catalyst 3560-E/3560 and Catalyst 2960 series switches, in addition to others.

When the switch does not have sufficient power to provide power to all devices that require it, most commonly because of incorrect power configurations, a message similar to the following will be printed on the console:

When using PoE, it is important to verify the power configuration prior to connecting devices to the switch. If you will be using non-Cisco IP phones or devices, consult the vendor documentation to determine the amount of power required by these devices and incorporate those numbers into your calculations. Use the show power suite of commands to troubleshoot power issues as was described earlier in this chapter.

Additionally, you can check the power amount drawn by one or more Cisco IP phones using theshow cdp neighbors command as illustrated in the following output:

Understanding the power a phone requires or draws can help with power budget calculations. For example, if you are running low on power, you can reduce the available amount of power for some phones, allowing others to operate. Cisco 7970 IP phones, for example, can draw 10.25W or 6.3W power, with the only difference being that at 6.3W the phone has reduced screen brightness. If you are running low on overall power, you could consider adjusting the power that is provided (based on documentation) to allow other devices to operate.

Switch port misconfigurations are a common source of problems for VoIP implementations. Use the show interfaces <name> switchport command to verify that the switch port is configured correctly. Following is a sample output of this command on a port that is configured as a Multi-VLAN Access Port:

Additionally, you can also ensure that the correct parameters have been specified by looking at the configuration of the port(s). Following is a sample configuration of an MVAP that has been correctly configured with both the data and voice VLAN:

Cat-6500-1#show running-config interface gigabitethernet 4/22

Building configuration...

Current configuration : 167 bytes

!

interface GigabitEthernet4/22

 switchport

 switchport access vlan 100

 switchport mode access

 switchport voice vlan 200

 no ip address

 spanning-tree portfast

end

IP services include those core services required by the IPT implementation, i.e. DHCP, TFTP, NTP, CDP, and, in some instances, DNS. It is important to ensure that these services are configured in the correct manner. CDP should be enabled on all ports connected to IP phones because this is how the phone and the switch communicate VLAN and PoE information.

If you are running Cisco IOS DHCP server, TFTP server or NTP server, verify that the network devices that will be providing these services are configured in the correct manner. For example, with Cisco IOS DHCP server, ensure that DHCP Options are correctly configured for each DHCP pool that will be provide IP addressing information to the IP phones.

Packet loss and congestion are a common cause of voice quality problems. Dropped packets and network congestion can cause delay and jitter, which adversely affect voice quality. Troubleshooting packet loss is a time consuming process. Each interface in the path of the VoIP call should be examined and drops and congestion should be eliminated.

There are various tools that you can use to determine whether you are experiencing packet loss in your network and where in the network the packets are getting dropped. If you have access to the Cisco IP phone, you can press the question mark (?) button twice during an active call to view call statistics, which include packet loss and jitter. This screen is illustrated in Figure 12-7:

Fig. 12-7. Verifying Packet Loss on Cisco IP Phones

If you do not have physical access to the IP phone, you can view the same statistics information via a web browser by typing in the IP address of the phone and navigating to the Streaming Statistics link, which allows you to view the call statistics for voice calls as shown in Figure 12-8:

Fig. 12-8. Verifying Packet Loss on Cisco IP Phones, contd.

Following these checks, if you do notice packet loss, use the appropriate router and switch commands to look for the following problem indicators and locations:

• Interface drops
• Buffer drops
• Policy-map drops
• Interface congestion
• Link congestion

When troubleshooting packet loss issue, you should keep in mind that it could be occurring at any layer of the OSI model. It is therefore important to check all possibilities at each hop in the path between source and destination. While time consuming, it is a necessary process. Additionally, consider using sniffers in your troubleshooting efforts as they can be very useful in identifying where in the network the packet loss is occurring. The use of sniffers was described earlier in this guide.

Queuing delay can be a significant contributor to jitter. When you have too much jitter end-to-end, you encounter voice quality problems.  This is common in networks that either have no QoS or are using incorrect QoS mechanisms for router and switch queue management. On routers, consider using strict priority queuing, e.g. LLQ, to ensure that the voice traffic is serviced and de-queued before other traffic types. On Cisco IOS Catalyst switches, ensure that switch ports are configured appropriately. For example, ensure that trust states are correctly configured, in addition to correct classification and marking.

When LLQ is set up, you can check policy-map drops by looking for packets that match the priority class. This shows how much traffic is matching the priority class and how much bandwidth is used. Ensure that the Cisco IOS queuing mechanism is placing the VoIP packets within the proper queues using the appropriate show commands to verify queuing.

Troubleshooting IP Video Implementations

There are several troubleshooting targets that you should focus on when troubleshooting video issues. These troubleshooting targets include the following:

• Packet Loss and Congestion
• Queuing
• Bandwidth Utilization
• IP Multicast

Like voice traffic, video is susceptible to packet loss and congestion which can and typically do result in a degradation of quality. Use the same troubleshooting techniques described in the previous section to identify and resolve packet loss and congestion issues. Queuing also results in a degradation of IP video quality. It is important to ensure that the IP video traffic is both classified and marked correctly at the Access layer.

Proper queuing mechanisms should be implemented so that this traffic is given preferential treatment, and therefore de-queued, before normal data traffic. However, it is important to understand that video should never be placed into the same queue as voice traffic. While LLQ should be used for voice, video traffic should simply be given a higher priority than normal data traffic, e.g. using a DSCP value of AF41. LLQ should not be used for video traffic when voice traffic exists on the same network.

While both IP voice and IP video traffic have the potential to consume a lot of bandwidth, the bandwidth requirements for all video is substantial compared to VoIP. For this reason, it is important to ensure that there is adequate bandwidth allocated for the video traffic. From a troubleshooting perspective, verify that policy-maps configured for video traffic have been assigned the correct bandwidth, based on the design and expected amount of video calls. You can also look at the policy-map drops to determine if traffic is exceeding what has been configured and if packets are dropped by the router using the show policy-map command.
Most video implementations use IP Multicast which results is a more efficient use of resources. However, this also introduces an additional troubleshooting element. A great tool to test Multicast operation is the Cisco IOS Multicast Routing Monitor (MRM). MRM is a management diagnostic tool that provides network fault detection and isolation in a large Multicast routing infrastructure. It is designed to notify a network administrator of Multicast routing problems in near real time, i.e. almost immediately.

MRM has three components that play different roles. These three components are the Manager, the Test Sender, and the Test Receiver. To test a Multicast environment using test packets, perhaps before an upcoming multicast event, you need all three components. While going into specifics on MRM implementation or configuration is beyond the scope of the TSHOOT exam, when implemented, if the Test Receiver detects an error, such as packet loss or duplicate packets, it sends an error report to the router configured as the Manager. The Manager immediately displays the error report. These errors can also be viewed using show commands.

Using this information, you can then troubleshoot the Multicast network using the appropriate commands. Using MRM to troubleshoot IP Multicast provides the following advantages:

• It allows you to find fault in Multicast routing in near real time, which allows you to find out about problems almost immediately
• It allows you to verify IP Multicast operation prior to implementing IP video. This prevents you having to wait for problem reports before determining that there is a problem with the Multicast implementation
• The printed information is easy to understand. This ease of use facilitates the troubleshooting process and makes MRM one of the simplest tools to use
• The MRM tool is very flexible and scalable. It can be used in small networks as well as in networks that have a large amount of users

In summation, while the troubleshooting processes for voice and video might seem both long and complex, you will find that having a solid fundamental understanding in how these technologies operate somewhat simplifies the troubleshooting process. Of course, from an overall solution perspective, there are several components, such as Cisco Unified Communications Manager, that will be outside of your scope of responsibility. Your function, however, should be to eliminate the network as a source of voice or video issues.