Logging

A syslog daemon or server is an entity that listens to the syslog messages that are sent to it. You cannot configure a syslog daemon to ask a specific device to send it syslog messages. In other words, if a specific device has no ability to generate syslog messages, then a syslog daemon cannot do anything about it. In the real world, corporations typically use SolarWinds (or similar) software for syslog capturing. Additionally, freeware such as the Kiwi Syslog daemon is also available for syslog capturing.

Syslog uses User Datagram Protocol (UDP) as the underlying transport mechanism, so the data packets are unsequenced and unacknowledged. While UDP does not have the overhead included in TCP, this means that on a heavily used network, some packets may be dropped and therefore logging information will be lost. However, Cisco IOS software allows administrators to configure multiple syslog servers for redundancy. A syslog solution is comprised of two main elements: a syslog server and a syslog client.

The syslog client sends syslog messages to the syslog sever using UDP as the Transport Layer protocol, specifying a destination port of 514. These messages cannot exceed 1024 bytes in size; however, there is no minimum length. All syslog messages contain three distinct parts: the priority, the header, and the message.

The priority of a syslog message represents both the facility and the severity of the message. This number is an 8-bit number. The first 3 least significant bits represent the severity of the message (with 3 bits, you can represent 8 different severities) and the other 5 bits represent the facility. You can use these values to apply filters on the events in the syslog daemon.

NOTE: Keep in mind that these values are generated by the applications on which the event is generated, not by the syslog server itself.

The values set by Cisco IOS devices are listed and described below in Table 40.1 (please memorise the levels and level names):

 

Table 40.1 –  Cisco IOS Software Syslog Priority Levels and Definitions

Level	Level Name	Syslog Definition	Description
0	Emergencies	LOG_EMERG	This level is used for the most severe error conditions, which render the system unusable.
1	Alerts	LOG_ALERT	This level is used to indicate conditions that need immediate attention from administrators.
2	Critical	LOG_CRIT	This level is used to indicate critical conditions, which are less critical than Alerts but still require administrator intervention.
3	Errors	LOG_ERR	This level is used to indicate errors within the system; however, these errors do not render the system unusable.
4	Warnings	LOG_WARNING	This level is used to indicate warning conditions about system operations that did not complete successfully.
5	Notifications	LOG_NOTICE	This level is used to indicate state changes within the system (e.g., a routing protocol adjacency transitioning to a down state).
6	Informational	LOG_INFO	This level is used to indicate informational messages about the normal operation of the system.
7	Debugging	LOG_DEBUG	This level is used to indicate real-time (debugging) information that is typically used for troubleshooting purposes.

 

In syslog, the facility is used to represent the source that generated the message. This source can be a process on the local device, an application, or even an operating system. Facilities are represented by numbers (integers). In Cisco IOS software, there are eight local use facilities that can be used by processes and applications (as well as the device itself) for sending syslog messages. By default, Cisco IOS devices use facility local7 to send syslog messages. However, it should be noted that most Cisco devices provide options to change the default facility level. In Cisco IOS software, the logging facility [facility] global configuration command can be used to specify the syslog facility. The options available with this command are as follows:

R1(config)#logging facility ?auth    Authorization systemcron    Cron/at facilitydaemon  System daemonskern    Kernellocal0  Local use local1  Local use local2  Local use local3  Local use local4  Local use local5  Local use local6  Local use local7  Local use lpr     Line printer system mail    Mail system news    USENET news sys10   System use sys11   System use sys12   System use sys13   System use sys14   System use sys9    System use syslog  Syslog itself user    User process uucp    Unix-to-Unix copy system

To send messages via syslog, you must perform the following sequence of steps on the device:

1. Globally enable logging on the router or switch using the logging on configuration command. By default, in Cisco IOS software, logging is enabled; however, it is only enabled to send messages to the console. The logging on command is a mandatory requirement when sending messages to any destination other than the console.
2. Specify the severity of messages to send to the syslog server using the logging trap [severity] global configuration command. You can specify the severity numerically or using the equivalent severity name.
3. Specify one or more syslog server destinations using the logging [address] or logging host [address] global configuration commands.
4. Optionally, specify the source IP address used in syslog messages using the logging source-interface [name]. This is a common practice on devices with multiple interfaces configured. If this command is not specified, then the syslog message will contain the IP address of the router or switch interface used to reach the server. If there are multiple interfaces for redundancy, this address may change when the primary path (interface) is down. Therefore, it is typically set to a Loopback interface.

The following configuration example illustrates how to send all informational (level 6) and below messages to a syslog server with the IP address 192.168.1.254:

R2(config)#logging onR2(config)#logging trap informationalR2(config)#logging 192.168.1.254

This configuration can be validated using the show logging command, as illustrated below:

R2#show logging Syslog logging: enabled (11 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)     Console logging: disabled     Monitor logging: level debugging, 0 messages logged, xml disabled,filtering disabled     Buffer logging: disabled, xml disabled, filtering disabled     Logging Exception size (4096 bytes)     Count and timestamp logging messages: disabled No active filter modules.     Trap logging: level informational, 33 message lines logged         Logging to 192.168.1.254(global) (udp port 514, audit disabled,  link up), 2 message lines logged, xml disabled,                filtering disabled

When configuring logging in general, it is important to ensure that the router or switch clocks reflect the actual current time, which allows you to correlate the fault data. Inaccurate or incorrect timestamps on log messages make the fault and problem isolation using a filtration and correlation process very difficult and very time consuming. In Cisco IOS devices, the system clock can be configured manually or the device can be configured to synchronise its clock with a Network Time Protocol (NTP) server. These two options are discussed in the following sections.

Manual clock or time configuration is fine if you have only a few internetwork devices in your network. In Cisco IOS software, the system time is configured using the clock set hh:mm:ss [day & month | month & day] [year] privileged EXEC command. It is not configured or specified in Global Configuration mode. The following configuration example illustrates how to set the system clock to October 20 12:15 am:

R2#clock set 12:15:00 20 october 2010

Alternatively, the same configuration could be implemented on the router as follows:

R2#clock set 12:15:00 october 20 2010

Following this configuration, the show clock command can be used to view the system time:

R2#show clock12:15:19.419 UTC Wed Oct 20 2010

One interesting observation of note is that when the system time is configured manually or set using the clock set command, it defaults to the GMT (UTC) time zone, as can be seen above. In order to ensure that the system clock reflects the correct time zone, for those who are not in the GMT time zone, you must use the clock timezone [time zone name] [GMT offset] global configuration command. For example, the United States has six different time zones, each with a different GMT offset. These time zones are Eastern Time, Central Time, Mountain Time, Pacific Time, Hawaii Time, and Alaska Time.

In addition, some of the time zones use Standard Time and Daylight Saving Time. Given this, it is important to ensure that the system time is set correctly (Standard or Daylight Saving) on all devices when manually configuring the system clock. The following configuration example illustrates how to set the system clock to 12:40 am on October 20 for the Central Standard Time (CST) time zone, which is six hours behind GMT:

R2#config tEnter configuration commands, one per line.  End with CNTL/Z.R2(config)#clock timezone CST -6R2(config)#endR2#clock set 12:40:00 october 20 2010

Following this configuration, the system clock on the local router now shows the following:

R2#show clock12:40:17.921 CST Wed Oct 20 2010

NOTE: If you use the clock set command before the clock timezone command, then the time that you specified using the clock set command will be offset by using the clock timezone command.  For example, assume that the configuration commands that are used in the example above were entered on the router as follows:

R2#clock set 12:40:00 october 20 2010R2#config tEnter configuration commands, one per line.  End with CNTL/Z.R2(config)#clock timezone CST -6R2(config)#end

Because the clock set command is used first, the output of the show clock command on the router would show the system clock offset by 6 hours, as specified using the clock timezone command. This behaviour is illustrated in the following output on the same router:

R2#show clock06:40:52.181 CST Wed Oct 20 2010

NOTE: Cisco IOS routers and switches can be configured to switch automatically to summertime (Daylight Saving Time) using the clock summer-time zone recurring [week day month hh:mm week day month hh:mm [offset]] global configuration command. This negates the need to have to adjust the system clock manually on all manually configured devices during Standard Time and Daylight Saving Time periods.

The second method of setting or synchronising the system clock is to use a Network Time Protocol (NTP) server as a reference time source. This is the preferred method in larger networks with more than just a few internetwork devices. NTP is a protocol that is designed to time-synchronise a network of machines. NTP is documented in RFC 1305 and runs over UDP.

An NTP network usually gets its time from an authoritative time source, such as a radio clock or an atomic clock attached to a time server. NTP then distributes this time across the network. NTP is extremely efficient; no more than one packet per minute is necessary to synchronise two machines to within a millisecond of one another.

NTP uses the concept of a stratum to describe how many NTP hops away a machine is from an authoritative time source. Keep in mind that this is not routing or switching hops, but NTP hops, which is a totally different concept. A stratum 1 time server typically has a radio or atomic clock directly attached, while a stratum 2 time server receives its time via NTP from a stratum 1 time server, and so on. When a device is configured with multiple NTP reference servers, it will automatically choose as its time source the machine with the lowest stratum number that it is configured to communicate with via NTP.

In Cisco IOS software, a device is configured with the IP addresses of one or more NTP servers using the ntp server [address] global configuration command. As previously stated, multiple NTP reference addresses can be specified by repeatedly using the same command. In addition, this command can also be used to configure security and other features between the server and the client. The following configuration example illustrates how to configure a device to synchronise its time with an NTP server with the IP address 10.0.0.1:

R2(config)#ntp server 10.0.0.1

Following this configuration, the show ntp associations command can be used to verify the communications between the NTP devices, as illustrated in the following output:

R2#show ntp associations  address     ref clock    st  when  poll reach  delay  offset  disp *~10.0.0.1  127.127.7.1  5   44    64   377    3.2    2.39    1.2 * master (synced), # master (unsynced), + selected, – candidate, ~ configured

The address field indicates the IP address of the NTP server as confirmed by the value 10.0.0.1 specified under this field. The ref clock field indicates the reference clock used by that NTP server. In this case, the IP address 127.127.7.1 indicates that the device is using an internal clock (127.0.0.0/8 subnet) as its reference time source. If this field contained another value, such as 192.168.1.254, for example, then that would be the IP address the server was using as its time reference.

Next, the st field indicates the stratum of the reference. From the output printed above, you can see that the 10.0.0.1 NTP device has a stratum of 5. The stratum on the local device will be incremented by 1 to a value of 6, as shown below, because it receives its time source from a server with a stratum value of 5. If another device was synchronised to the local router, it would reflect a stratum of 7 and so forth. The second command that is used to validate the NTP configuration is the show ntp status command, the output of which is illustrated below:

R2#show ntp status Clock is synchronized, stratum 6, reference is 10.0.0.1 nominal freq is 249.5901 Hz, actual freq is 249.5900 Hz, precision is 2**18 reference time is C02C38D2.950DA968 (05:53:22.582 UTC Sun Mar 3 2002) clock offset is 4.6267 msec, root delay is 3.16 msec root dispersion is 4.88 msec, peer dispersion is 0.23 msec

The output of the show ntp status command indicates that the clock is synchronised to the configured NTP server (10.0.0.1). This server has a stratum of 5, hence the local device reflects a stratum of 6. An interesting observation when NTP is configured is that the local time still defaults to GMT, as can be seen in the bolded section above. To ensure that the device displays the correct time zone, you must issue the clock time-zone command on the device.

After the system clock has been set, either manually or via NTP, it is important to ensure that the logs sent to the server contain the correct timestamps. This is performed using the service timestamps log [datetime | uptime] global configuration command. The [datetime] keyword supports the following self-explanatory additional subkeywords:

R2(config)#service timestamps log datetime ?localtime      Use local time zone for timestampsmsec           Include milliseconds in timestampshow-timezone  Add time zone information to timestampyear           Include year in timestamp<cr>

The [uptime] keyword has no additional subkeywords and configures the local router to include only the system uptime as the timestamp for sent messages. The following configuration example illustrates how to configure the local router to include the local time, millisecond information, and the time zone for all messages:

R2#configure terminalEnter configuration commands, one per line.  End with CNTL/Z.R2(config)#logging onR2(config)#logging console informationalR2(config)#logging host 150.1.1.254R2(config)#logging trap informational R2(config)#service timestamps log datetime localtime msec show-timezone

Following this configuration, the local router console would print the following message:

Oct 20 02:14:10.519 CST: %SYS-5-CONFIG_I: Configured from console by console Oct 20 02:14:11.521 CST: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 150.1.1.254 started – CLI initiated

In addition, the syslog daemon on server 150.1.1.254 would also reflect the same, as illustrated in the Kiwi Syslog Manager screenshot in Figure 40.1 below:

  

Figure 40.1 – Configuring Log Timestamps