While there is no single specific definition for the term, in general terms, troubleshooting can be thought of as the process of identifying or diagnosing a problem. Following diagnosis, a resolution is typically implemented to rectify or correct the problem. While the general idea behind troubleshooting is that it begins after a problem or an issue has been reported, i.e. it is a reactive process; it is important to understand that with effective proactive monitoring, it is possible to identify and resolve potential problems and resolve them before they impact users.
As was stated in the introduction, there is no single troubleshooting method that can be applied to all situations; hence the reason for the different troubleshooting methodologies which will be described in the following section. Despite the different approaches used in these methodologies, they all fit into to the same basic high-level three-step troubleshooting flow, which is comprised of the following phases:
The Problem Report
Problem Diagnosis
Problem Resolution
These three phases in the basic troubleshooting flow process are illustrated in below Figure 2-1:
Fig. 2-1. High-Level Steps in the Troubleshooting Flow
The problem report is what typically initiates the troubleshooting process. The objective of this phase is to define the problem. While most users report a problem when it occurs or is experienced, this is not always the case. In some cases, a problem could be reported hours or even days after the fact. Yet still, it is not uncommon for users to report a problem that has been ongoing for an extended period of time. As an example, a user could call into the help desk and report that an application has been experiencing intermittent connectivity issues for a while, and while they previously tolerated it, it has now degraded to the point where they would now like it to get resolved.
Following the problem report, the very first step of the troubleshooting process itself is problem diagnosis. The problem diagnosis phase is the most time consuming phase. This phase is comprised of five general, but integral, steps, all of which are critical to the overall success of the problem resolution effort. The five steps included in the problem diagnosis phase are:
Collecting information about the problem
Analyzing or examining the collected information
Eliminating possible causes
Hypothesizing or theorizing potential causes
Verifying the hypothesis or theory
The first step included in the problem diagnosis phase entails collecting or gathering as much information about the problem as is possible. This not only includes collecting as much information from the person reporting the problem, but also collecting data from the network itself. The efficiency with which the information is collected, as well as the overall quality of that information, greatly increases the chances of successfully identifying and resolving the problem.
Cisco provides a plethora of tools that can be used to gather or collect information from the network or from internetwork devices. As a network engineer, you should be intimately familiar with these tools (which are described later in this chapter) and their capabilities. While the number of available tools is large, data collection need not be overwhelming as in most cases, a small subset of the available tools can typically be used to solve most of the problems.
Once the pertinent information has been collected, it should be analyzed. There are numerous ways in which collected information can be analyzed. This may entail using automated tools, referencing the collectors experience or knowledge about the system or platform in question in order to distinguish between normal and abnormal system behavior, or comparing the information against previously collected baseline information gathered from monitoring tools.
When the information has been analyzed, some potential causes of the problem can be easily eliminated. For example, if a user reported intermitted network connectivity issues and following your information gathering and analysis you observe the port the user machine is connected to taking errors and flapping, you can potentially rule out an overall network issue should this issue only be happening only for that specific user.
After eliminating some potential causes, the next step is to hypothesize or theorize about what the potential problem could be. Continuing with the previous intermittent connectivity example, having eliminated the network as a potential cause, the next step would be to determining whether or not the switch port or the patch cable between the user machine and the switch port was bad (amongst other possible issues), for example.
The last step of the problem diagnosis phase is to verify your hypothesis or theory. This step may include running appropriate tests or diagnostics using the relevant tools to validate whether what you think to be problem is in fact the problem. Alternatively, you could also apply what you think to be the solution and then verify if that resolves the problem. For example, continuing with the user intermittent connectivity issue, you could replace the patch cable between the user machine and the switch port or move the user network connection to another switch port and see if that resolves the intermittent connectivity issue. For simplicity, the steps involved in the problem diagnosis phase are illustrated in Figure 2-2:
Fig. 2-2. Steps within the Problem Diagnosis Phase
While these steps present a structured approach to network troubleshooting, it is important to understand that the process flow may require going back and forth between different steps. Assume for example, that after testing your proposed solution to the problem the problem still exists. In such cases, you may need to go back and gather more information and proceed through the entire cycle again. It is important to keep this in mind because even following a structured troubleshooting approach does not always guarantee that you will be able to identify the root cause of the reported problem. While there are no guarantees of success, following a structured troubleshooting approach does increase the probability or likelihood of overall success in identifying and then ultimately resolving the issue or problem.
The primary objective of the problem diagnosis phase is to reach a hypothesis for the root cause of the problem. Following the completion of the problem diagnosis phase, the troubleshooting process moves to the problem resolution phase. In this final phase, the troubleshooter should notify and then confirm with the user that reported the problem that the problem has indeed been rectified. In addition to notifying the user that initiated the problem report, the troubleshooter also needs to inform any other invested parties. For example, if the problem had been escalated to management, they should be notified when the problem has been resolved and confirmation of the same has been received.
And finally, an additional but important task is to include all relevant documentation and notes that may be used as a reference in the future prior to closing out the trouble report. If changes were made to the network to resolve the problem, the network relevant documentation should be updated to include these changes.