IOS Maintenance and Monitoring Tools – Part 3

Logging

Logging messages and events both locally as well as to a syslog server is a core maintenance task. Syslog is a protocol that allows a host to send event notification messages across IP networks to event message collectors – also known as syslog servers or syslog daemons. In other words, a host or a device can be configured in such a way that it generates a syslog message and forwards it to a specific syslog daemon (server).

A syslog daemon or server is an entity that listens to the syslog messages that are sent to it. You cannot configure a syslog daemon to ask a specific device to send it syslog messages. In other words, if a specific device has no ability to generate syslog messages, then a syslog daemon cannot do anything about it. In the real world, corporations typically use Solarwinds (or similar) software for syslog capturing. Additionally, freeware such as the Kiwi syslog daemon is also available for syslog capturing.

Syslog uses User Datagram Protocol (UDP) as the underlying transport mechanism, so the data packets are unsequenced and unacknowledged. While UDP does not have the overhead included in TCP, this means that on a heavily used network, some packets may be dropped and therefore logging information will be lost. However, the Cisco IOS allows administrators to configure multiple syslog servers for redundancy. A syslog solution is comprised of two main elements: a syslog server and a syslog client.

The syslog client sends syslog messages to the syslog sever using UDP as the Transport Layer protocol, specifying a destination port of 514. These messages cannot exceed 1,024 bytes in size; however, there is no minimum length.  All syslog messages contain three distinct parts. These parts are the priority, the header, and the message.

The priority of a syslog message represents both the facility and severity of the message. This number is an eight bit number. The first 3 least significant bits represent the severity of the message (with 3 bits you can represent 8 different Severities) and the other 5 bits represent the facility. You can use these values to apply filters on the events in the syslog daemon.

NOTE: Keep in mind that these values are generated by the applications on which the event is generated, and not by the syslog server itself.

The values sent by Cisco IOS devices are listed and described below in Table 1-1:

In syslog, the facility is used to represent the source that generated the message. This source can be a process on the local device, an application, or even an operating system. Facilities are represented by numbers (integers). In Cisco IOS software, there are eight local use facilities that can be used by processes and applications (as well as the device itself) for sending syslog messages. By default, Cisco IOS devices use facility local7 to send syslog messages. However, it should be noted that most Cisco devices provide options to change the default facility level. In Cisco IOS software, the logging facility [facility] global configuration command can be used to specify the syslog facility. The options available with this command are as follows:

To send messages via syslog, you must perform the following sequence of steps on the device:

1. Globally enable logging on the router or switch using the logging on configuration command. By default, in Cisco IOS software, logging is enabled; however, it is only enabled to send messages to the console. The logging on command is a mandatory requirement when sending messages to any destination other than the console
2. Specify the severity of messages to send to the syslog server using the logging trap [severity] global configuration command. You can specify the severity numerically or using the equivalent severity name
3. Specify one or more syslog server destinations using the logging [address] or logging host [address] global configuration commands
4. Optionally, specify the source IP address used in syslog messages using the logging source-interface [name]. This is a common practice on devices with multiple interfaces configured. If this command is not specified, the syslog message will contain the IP address of the router or switch interface used to reach the server. If there are multiple interfaces for redundancy, this address may change when the primary path (interface) is down. Therefore, it is typically set to a Loopback interface

The following configuration example illustrates how to send all informational (level 6) and below messages to a syslog server with the IP address 192.168.1.254:

This configuration can be validated using the show logging command as illustrated below:

When configuring logging in general, it is important to ensure that the router or switch clocks reflect the actual current time which allows you to correlate the fault data. Inaccurate or incorrect timestamps on log messages make the fault and problem isolation using a filtration and correlation process very difficult and very time consuming. In Cisco IOS devices, the system clock can be configured manually or the device can be configured to synchronize its clock with a Network Time Protocol (NTP) server. These two options are discussed in the following sections.

Manual clock or time configuration is fine if you have only a few internetwork devices in your network. In Cisco IOS software, the system time is configured using the clock set hh:mm:ss [day & month | month & day] [year] privileged EXEC command. It is not configured or specified in global configuration mode. The following configuration example illustrates how to set the system clock to October 20 12:15 AM:

Alternatively, the same configuration could also be implemented on the router as follows:

Following this configuration, the show clock command can be used to view the system time:

One interesting observation of note is that when the system time is manually configured or set using the clock set command, it defaults to the GMT (UTC) time zone as can be above. In order to ensure that the system clock reflects the correct time zone, for those of us who are not in the GMT time zone, you must use the clock timezone [time zone name] [GMT offset]global configuration command. For example, the United States has 6 (six) different time zones, each with a different GMT offset. These time zones are Eastern Time, Central Time, Mountain Time, Pacific Time, Hawaii Time and Alaska Time.

In addition to this, some of the time zones use standard time and daylight savings time. Given this, it is important to ensure that the system time is correctly set (standard or daylight) on all devices when manually configuring the system clock. The following configuration example illustrates how to set the system clock to 12:40 AM on October 20 for the Central Standard Time (CST) time zone which is 6 (six) hours behind GMT:

Following this configuration, the system clock on the local router now shows the following:

NOTE: If you use the clock set command before the clock timezone command, the time that you specified using the clock set command will be offset by offset specified using the clock timezone command.  For example, assume that the configuration commands that are used in the example above were entered on the router as follows:

Because the clock set command is used first, the output of the show clock command on the router would show the system clock offset by 6 hours as specified using the clock timezonecommand. This behavior is illustrated in the following output on the same router:

NOTE: Cisco IOS routers and switches can be configured to automatically switch to summer time (daylight saving time) using the clock summer-time zone recurring [week day month hh:mm week day month hh:mm [offset]] global configuration command. This negates the need to have to manually adjust the system clock on all manually configured devices during standard and daylight saving time periods.

The second method setting or synchronizing the system clock is to use a Network Time Protocol (NTP) server as a reference time source. This is the preferred method in larger networks with more than just a few internetwork devices. NTP is a protocol that is designed to time-synchronize a network of machines. NTP is documented in RFC 1305 and runs over UDP.

An NTP network usually gets its time from an authoritative time source, such as a radio clock or an atomic clock attached to a time server. NTP then distributes this time across the network. NTP is extremely efficient; no more than one packet per minute is necessary to synchronize two machines to within a millisecond of one another.

NTP uses the concept of a stratum to describe how many NTP hops away a machine is from an authoritative time source. Keep in mind that this is not routing or switching hops, but NTP hops, which is a totally different concept. A stratum 1 time server typically has a radio or atomic clock directly attached, while a stratum 2 time server receives its time via NTP from a stratum 1 time server, and so on. When a device is configured with multiple NTP reference servers, it will automatically choose as its time source the machine with the lowest stratum number that it is configured to communicate with via NTP.

In Cisco IOS software, a device is configured with the IP addresses of one or more NTP servers using the ntp server [address] global configuration command. As previously stated, multiple NTP reference addresses can be specified by repeatedly using the same command. In addition to this, this command can also be used to configure security and other features between the server and client. These features will be described in detail later in this guide. The following configuration example illustrates how to configure a device to synchronize its time with an NTP server with the IP address 10.0.0.1:

Following this configuration, the show ntp associations command can be used to verify the communications between the NTP devices as illustrated in the following output:

The output of this command provides some very telling information of which only some will be relevant to the TSHOOT certification exam. First, the address field indicates the IP address of the NTP server as confirmed by the value 10.0.0.1 specified under this field. The ref clock field indicates the reference clock used by that NTP server. In this case, the IP address 127.127.7.1 indicates that the device is using an internal clock (127.0.0.0/8 subnet) as its reference time source.  If this field contained another value, such as 192.168.1.254, for example, that would be the IP address that the server was using as its time reference.

Next, the st field indicates the stratum of the reference. From the output printed above, we can determine that the 10.0.0.1 NTP device has a stratum of 5. The stratum on the local device will be incremented by 1 to a value of, as is shown below, because it receives its time source from a server with a stratum value of 5. If another device were synchronized to the local router, it would reflect a stratum of 7 and so forth. The second command that is used to validate the NTP configuration is the show ntp status command, the output of which is shown below:

The output of the show ntp status command indicates that the clock is synchronized to the configured NTP server (10.0.0.1). This server has a stratum of 5, hence the local device reflects a stratum of 6. An interesting observation when NTP is configured is that the local time still defaults to GMT as can be seen in the bolded section above. To ensure that the device displays the correct time zone, you must issue the clock time-zone command on the device.
After the system clock has been set, either manually or via NTP, it is important to ensure that the logs sent to the server contain the correct timestamps. This is performed using the service timestamps log [datetime | uptime] global configuration command. The datetime keyword supports the following self-explanatory additional sub-keywords:

The uptime keyword has no additional sub-keywords and configures the local router to include only the system uptime as the timestamp for sent messages. The following configuration example illustrates how to configure the local router to include the local time, millisecond information, as well as the time zone for all messages:

Following this configuration, the local router console would print the following message:

In addition, the syslog daemon on server 150.1.1.254 would also reflect the same as illustrated below in the Kiwi Syslog Manager screenshot in Figure 1-1:

Fig. 1-1. Configuring Log Timestamp