The Cisco IOS Generic Troubleshooting Toolkit – Part 2

Troubleshooting Hardware

The Cisco IOS diagnostic toolset include Cisco Generic Online Diagnostics (GOLD) and Cisco IOS Service Diagnostics. Running diagnostics is an inherent part of maintenance as well as troubleshooting. For example, diagnostics may be run on newly purchased hardware to ensure and validate functionality before it is integrated into the existing network. Additionally, diagnostics could also be run on hardware while troubleshooting a problem report, in an attempt to isolate issues.

Cisco GOLD defines a common framework for diagnostics operations across different platforms running Cisco IOS Software. The GOLD framework specifies the platform-independent fault-detection architecture for centralized and distributed systems. This includes the common diagnostics CLI and the platform-independent fault-detection procedures for boot-up and runtime diagnostics. When boot-up diagnostics detects a failure on a Cisco Catalyst 6500 Series switch, for example, the failing modules are shut down.

Should diagnostics fail, you can open up a trouble report or ticket with the Technical Assistance Center (TAC) to perform additional troubleshooting for the specified module(s) or initiate the process of replacing the faulty hardware.  By default, Catalyst 6500 series switches run minimal diagnostics at boot-up; however, this default diagnostics behavior can be changed using thediagnostic bootup level [minimal | complete] global configuration command to allow the switch to run complete (full) diagnostics at boot-up instead.

Once the switch has completed running boot-up diagnostic, you can use the show modulecommand to verify the status of the diagnostic testing as shown in the following output:

The show diagnostic result command can be used to view detailed testing information about the boot-up diagnostics for all modules as shown in the following output:

NOTE: You are not required to configure GOLD for the current TSHOOT certification exam.

In addition to boot-up diagnostics, you can also run on-demand diagnostics or even schedule diagnostic testing to be run at a specific time. These configurations, however, are beyond the scope of the current TSHOOT certification exam and are not described any further in this guide.

Troubleshooting Cisco IOS Services

In addition to GOLD, Cisco IOS Service Diagnostics allows for diagnostic testing and tools for common problems in Border Gateway Protocol (BGP), Open Shortest Path First (OSPF) Protocol, and Quality of Service (QoS), as well as tools to monitor and detect abnormal system resource utilization using scenario-specific troubleshooting scripts.

Unlike GOLD, Cisco IOS Service Diagnostics is a programmable diagnostics service that also leverages other Cisco IOS Software capabilities, specifically the Embedded Event Manager (EEM), Embedded Syslog Manager (ESM) and Tool Command Language (Tcl). As is the case with GOLD, the configuration of Cisco IOS Service Diagnostics is beyond the scope of the current TSHOOT certification exam and will not be described in further detail in this guide.

Filtering Command Output

While most engineers are familiar with the show command suite that is available in Cisco IOS software, not all of them are aware of the filtering capabilities integrated into the software. Filtering or restricting command output to only the information you will need to troubleshoot a particular problem not only demonstrates your level of knowledge about the capabilities of Cisco IOS software, it also saves time, which can result in a speedier resolution of the problem.

In Cisco IOS software, output filtering is performed by appending the pipe (|) symbol which then allows the output to be acted on according to the specified logic, which includes the begin,exclude, include, and section keywords. Following these keywords, the desired output can be matched against regular expressions. Table 2-3 lists and describes some of the commonly used regular expressions when filtering command output:

The begin keyword filters the output beginning at the specified phrase. The following example illustrates how to filter the output of the current (running) configuration such that the output begins at the Border Gateway Protocol (BGP) configuration that is implemented on the router:

The exclude keyword is used to exclude the specified phrases or regular expressions from the command output. The following example illustrates how the exclude keyword can be used to filter the output of the show ip interfaces brief command so that only interfaces that only interfaces that are not administratively shut down are displayed in the output of the command:

The include keyword includes the specified phrase or regular expression in the output of the command. If the specified phrase is part of a line, the entire line is displayed. The following example illustrates how to shown on information on errors for a WAN interface. Notice in the output that because the specified keyword is part of a line, the entire line is printed:

The section keyword prints only information that matches the specified section and no more. For example, if the show running-configuration | begin router bgp command was specified, the router would parse the configuration and begin at the statement router bgp. However, the router would also display all other configuration following this. So not only is the BGP configuration displayed, all other configuration after the BGP configuration is included. Using thesection keyword prevents this behavior as is illustrated in the following example:

As seen in the output above, only the relevant matched section of the configuration is printed. As stated earlier, regular expressions can also be used when filtering command output. The following example illustrates how to filter the running configuration on a router such that only lines that end with the number 0 (zero) are displayed:

And as a final example, the following illustrates how to filter the router output such that only lines that include 0/0 or 0/1 are printed:

NOTE: If you include a space between the second pipe and the 0/0 and 0/1 keywords, nothing will be matched by the command.

Redirecting Output

In addition to filtering command output, Cisco IOS software also allows output to be redirected to an external location or Flash memory. This is useful when executing commands that print a lot of information, such as the show tech-support command, for example. Output redirection can be performed using the append, redirect or tee keywords after the pipe.

The append keyword allows you to redirect and add the output of any show command to an existing file. You can use the show <command> | append ? command to view valid locations for the platform on which you are working. The following example shows the use of this command to view valid locations for a Cisco 2600XM series router:

In the output above, valid options for the router include appending to a file stored in NVRAM or on an FTP server. The following example shows how to append the output of the running configuration file to an existing file named r2-config that is located on FTP server 150.1.1.254:

Keep in mind that the example above assumes that both the router and FTP server have been configured correctly in the event that authentication is required. If credentials are required, include these in the command as shown in the example below which specifies an FTP username of NETADMIN with an FTP password of TSHOOT when connecting to the FTP server:

As previously stated, the redirect keyword is used to redirect the output of a command to a local or remote file location. Again, the options available will depend on the platform on which the command is executed. The following example illustrates the options that are available on a Cisco 2600XM series router running Cisco IOS version 12.4 Mainline:

The following example shows how to redirect the output of the show tech-support ospf detailcommand to a TFTP server with the IP address 150.1.1.254. The file will be saved on the TFTP server with the name r2-ospf-tshoot:

The following example illustrates how to save the same file to the local router Flash memory:

To view the contents of the file (if saved to Flash) you can use the more command as follows:

And finally, the tee keyword copies the output of any show command to a file and displays the same content on the terminal at the same time. The following output shows the available locations for copying the file while using the tee keyword on a Cisco 2600XM series router:

The following example illustrates how to view the contents of the show ip ospf neighbor detail command while simultaneously copying the same output to a TFTP server with the IP address 150.1.1.254. The file will be saved on the TFTP server as r2-ospf-output:

The following example illustrates how to save the same output to Flash while viewing it:

The more command can then be used to view the file that has been stored in Flash as follows: