Logging
Logging messages and events both locally as well as to a syslog server is a core maintenance task. Syslog is a protocol that allows a host to send event notification messages across IP networks to event message collectors – also known as syslog servers or syslog daemons. In other words, a host or a device can be configured in such a way that it generates a syslog message and forwards it to a specific syslog daemon (server).
A syslog daemon or server is an entity that listens to the syslog messages that are sent to it. You cannot configure a syslog daemon to ask a specific device to send it syslog messages. In other words, if a specific device has no ability to generate syslog messages, then a syslog daemon cannot do anything about it. In the real world, corporations typically use Solarwinds (or similar) software for syslog capturing. Additionally, freeware such as the Kiwi syslog daemon is also available for syslog capturing.
Syslog uses User Datagram Protocol (UDP) as the underlying transport mechanism, so the data packets are unsequenced and unacknowledged. While UDP does not have the overhead included in TCP, this means that on a heavily used network, some packets may be dropped and therefore logging information will be lost. However, the Cisco IOS allows administrators to configure multiple syslog servers for redundancy. A syslog solution is comprised of two main elements: a syslog server and a syslog client.
The syslog client sends syslog messages to the syslog sever using UDP as the Transport Layer protocol, specifying a destination port of 514. These messages cannot exceed 1,024 bytes in size; however, there is no minimum length. All syslog messages contain three distinct parts. These parts are the priority, the header, and the message.
The priority of a syslog message represents both the facility and severity of the message. This number is an eight bit number. The first 3 least significant bits represent the severity of the message (with 3 bits you can represent 8 different Severities) and the other 5 bits represent the facility. You can use these values to apply filters on the events in the syslog daemon.
NOTE: Keep in mind that these values are generated by the applications on which the event is generated, and not by the syslog server itself.
The values sent by Cisco IOS devices are listed and described below in Table 1-1:
Level | Level Name | Syslog Definition | Description |
---|---|---|---|
0 | Emergencies | LOG_EMERG | This level is used for the most severe error conditions which render the system unusable |
1 | Alerts | LOG_ALERT | This level is used to indicate conditions that need immediate attention from administrators |
2 | Critical | LOG_CRIT | This level is used to indicate critical conditions which are less than Alerts, but still require administrator intervention |
3 | Errors | LOG_ERR | This level is used to indicate errors within the system; however, these errors do not render the system unusable |
4 | Warnings | LOG_WARNING | This level is used to indicate warning conditions about system operations that did not complete successfully |
5 | Notifications | LOG_NOTICE | This level is used to indicate state changes within the system, e.g. a routing protocol adjacency transitioning to a down state |
6 | Informational | LOG_INFO | This level is used to indicate informational messages about the normal operation of the system |
7 | Debugging | LOG_DEBUG | This level is used to indicate real-time (debugging) information that is typically used for troubleshooting purposes |
In syslog, the facility is used to represent the source that generated the message. This source can be a process on the local device, an application, or even an operating system. Facilities are represented by numbers (integers). In Cisco IOS software, there are eight local use facilities that can be used by processes and applications (as well as the device itself) for sending syslog messages. By default, Cisco IOS devices use facility local7 to send syslog messages. However, it should be noted that most Cisco devices provide options to change the default facility level. In Cisco IOS software, the logging facility [facility] global configuration command can be used to specify the syslog facility. The options available with this command are as follows:
R1(config)#logging facility ? auth Authorization system cron Cron/at facility daemon System daemons kern Kernel local0 Local use local1 Local use local2 Local use local3 Local use local4 Local use local5 Local use local6 Local use local7 Local use lpr Line printer system mail Mail system news USENET news sys10 System use sys11 System use sys12 System use sys13 System use sys14 System use sys9 System use syslog Syslog itself user User process uucp Unix-to-Unix copy system |
To send messages via syslog, you must perform the following sequence of steps on the device:
- Globally enable logging on the router or switch using the logging on configuration command. By default, in Cisco IOS software, logging is enabled; however, it is only enabled to send messages to the console. The logging on command is a mandatory requirement when sending messages to any destination other than the console
- Specify the severity of messages to send to the syslog server using the logging trap [severity] global configuration command. You can specify the severity numerically or using the equivalent severity name
- Specify one or more syslog server destinations using the logging [address] or logging host [address] global configuration commands
- Optionally, specify the source IP address used in syslog messages using the logging source-interface [name]. This is a common practice on devices with multiple interfaces configured. If this command is not specified, the syslog message will contain the IP address of the router or switch interface used to reach the server. If there are multiple interfaces for redundancy, this address may change when the primary path (interface) is down. Therefore, it is typically set to a Loopback interface
The following configuration example illustrates how to send all informational (level 6) and below messages to a syslog server with the IP address 192.168.1.254:
R2(config)#logging on R2(config)#logging trap informational R2(config)#logging 192.168.1.254 |
This configuration can be validated using the show logging command as illustrated below:
R2#show logging Syslog logging: enabled (11 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled) Console logging: disabled Monitor logging: level debugging, 0 messages logged, xml disabled, filtering disabled Buffer logging: disabled, xml disabled, filtering disabled Logging Exception size (4096 bytes) Count and timestamp logging messages: disabledNo active filter modules.Trap logging: level informational, 33 message lines logged Logging to 192.168.1.254(global) (udp port 514, audit disabled, link up), 2 message lines logged, xml disabled, filtering disabled |
When configuring logging in general, it is important to ensure that the router or switch clocks reflect the actual current time which allows you to correlate the fault data. Inaccurate or incorrect timestamps on log messages make the fault and problem isolation using a filtration and correlation process very difficult and very time consuming. In Cisco IOS devices, the system clock can be configured manually or the device can be configured to synchronize its clock with a Network Time Protocol (NTP) server. These two options are discussed in the following sections.
Manual clock or time configuration is fine if you have only a few internetwork devices in your network. In Cisco IOS software, the system time is configured using the clock set hh:mm:ss [day & month | month & day] [year] privileged EXEC command. It is not configured or specified in global configuration mode. The following configuration example illustrates how to set the system clock to October 20 12:15 AM:
R2#clock set 12:15:00 20 october 2010 |
Alternatively, the same configuration could also be implemented on the router as follows:
R2#clock set 12:15:00 october 20 2010 |
Following this configuration, the show clock command can be used to view the system time:
R2#show clock 12:15:19.419 UTC Wed Oct 20 2010 |
One interesting observation of note is that when the system time is manually configured or set using the clock set command, it defaults to the GMT (UTC) time zone as can be above. In order to ensure that the system clock reflects the correct time zone, for those of us who are not in the GMT time zone, you must use the clock timezone [time zone name] [GMT offset]global configuration command. For example, the United States has 6 (six) different time zones, each with a different GMT offset. These time zones are Eastern Time, Central Time, Mountain Time, Pacific Time, Hawaii Time and Alaska Time.
In addition to this, some of the time zones use standard time and daylight savings time. Given this, it is important to ensure that the system time is correctly set (standard or daylight) on all devices when manually configuring the system clock. The following configuration example illustrates how to set the system clock to 12:40 AM on October 20 for the Central Standard Time (CST) time zone which is 6 (six) hours behind GMT:
R2#config t Enter configuration commands, one per line. End with CNTL/Z. R2(config)#clock timezone CST -6 R2(config)#end R2#clock set 12:40:00 october 20 2010 |
Following this configuration, the system clock on the local router now shows the following:
R2#show clock 12:40:17.921 CST Wed Oct 20 2010 |
NOTE: If you use the clock set command before the clock timezone command, the time that you specified using the clock set command will be offset by offset specified using the clock timezone command. For example, assume that the configuration commands that are used in the example above were entered on the router as follows:
R2#clock set 12:40:00 october 20 2010 R2#config t Enter configuration commands, one per line. End with CNTL/Z. R2(config)#clock timezone CST -6 R2(config)#end |
Because the clock set command is used first, the output of the show clock command on the router would show the system clock offset by 6 hours as specified using the clock timezonecommand. This behavior is illustrated in the following output on the same router:
R2#show clock 06:40:52.181 CST Wed Oct 20 2010 |
NOTE: Cisco IOS routers and switches can be configured to automatically switch to summer time (daylight saving time) using the clock summer-time zone recurring [week day month hh:mm week day month hh:mm [offset]] global configuration command. This negates the need to have to manually adjust the system clock on all manually configured devices during standard and daylight saving time periods.
The second method setting or synchronizing the system clock is to use a Network Time Protocol (NTP) server as a reference time source. This is the preferred method in larger networks with more than just a few internetwork devices. NTP is a protocol that is designed to time-synchronize a network of machines. NTP is documented in RFC 1305 and runs over UDP.
An NTP network usually gets its time from an authoritative time source, such as a radio clock or an atomic clock attached to a time server. NTP then distributes this time across the network. NTP is extremely efficient; no more than one packet per minute is necessary to synchronize two machines to within a millisecond of one another.
NTP uses the concept of a stratum to describe how many NTP hops away a machine is from an authoritative time source. Keep in mind that this is not routing or switching hops, but NTP hops, which is a totally different concept. A stratum 1 time server typically has a radio or atomic clock directly attached, while a stratum 2 time server receives its time via NTP from a stratum 1 time server, and so on. When a device is configured with multiple NTP reference servers, it will automatically choose as its time source the machine with the lowest stratum number that it is configured to communicate with via NTP.
In Cisco IOS software, a device is configured with the IP addresses of one or more NTP servers using the ntp server [address] global configuration command. As previously stated, multiple NTP reference addresses can be specified by repeatedly using the same command. In addition to this, this command can also be used to configure security and other features between the server and client. These features will be described in detail later in this guide. The following configuration example illustrates how to configure a device to synchronize its time with an NTP server with the IP address 10.0.0.1:
R2(config)#ntp server 10.0.0.1 |
Following this configuration, the show ntp associations command can be used to verify the communications between the NTP devices as illustrated in the following output:
R2#show ntp associationsaddress ref clock st when poll reach delay offset disp *~10.0.0.1 127.127.7.1 5 44 64 377 3.2 2.39 1.2 * master (synced), # master (unsynced), + selected, – candidate, ~ configured |
The output of this command provides some very telling information of which only some will be relevant to the TSHOOT certification exam. First, the address field indicates the IP address of the NTP server as confirmed by the value 10.0.0.1 specified under this field. The ref clock field indicates the reference clock used by that NTP server. In this case, the IP address 127.127.7.1 indicates that the device is using an internal clock (127.0.0.0/8 subnet) as its reference time source. If this field contained another value, such as 192.168.1.254, for example, that would be the IP address that the server was using as its time reference.
Next, the st field indicates the stratum of the reference. From the output printed above, we can determine that the 10.0.0.1 NTP device has a stratum of 5. The stratum on the local device will be incremented by 1 to a value of, as is shown below, because it receives its time source from a server with a stratum value of 5. If another device were synchronized to the local router, it would reflect a stratum of 7 and so forth. The second command that is used to validate the NTP configuration is the show ntp status command, the output of which is shown below:
R2#show ntp status Clock is synchronized, stratum 6, reference is 10.0.0.1 nominal freq is 249.5901 Hz, actual freq is 249.5900 Hz, precision is 2**18 reference time is C02C38D2.950DA968 (05:53:22.582 UTC Sun Mar 3 2002) clock offset is 4.6267 msec, root delay is 3.16 msec root dispersion is 4.88 msec, peer dispersion is 0.23 msec |
The output of the show ntp status command indicates that the clock is synchronized to the configured NTP server (10.0.0.1). This server has a stratum of 5, hence the local device reflects a stratum of 6. An interesting observation when NTP is configured is that the local time still defaults to GMT as can be seen in the bolded section above. To ensure that the device displays the correct time zone, you must issue the clock time-zone command on the device.
After the system clock has been set, either manually or via NTP, it is important to ensure that the logs sent to the server contain the correct timestamps. This is performed using the service timestamps log [datetime | uptime] global configuration command. The datetime keyword supports the following self-explanatory additional sub-keywords:
R2(config)#service timestamps log datetime ? localtime Use local time zone for timestamps msec Include milliseconds in timestamp show-timezone Add time zone information to timestamp year Include year in timestamp <cr> |
The uptime keyword has no additional sub-keywords and configures the local router to include only the system uptime as the timestamp for sent messages. The following configuration example illustrates how to configure the local router to include the local time, millisecond information, as well as the time zone for all messages:
R2#configure terminal Enter configuration commands, one per line. End with CNTL/Z. R2(config)#logging on R2(config)#logging console informational R2(config)#logging host 150.1.1.254 R2(config)#logging trap informational R2(config)#service timestamps log datetime localtime msec show-timezone |
Following this configuration, the local router console would print the following message:
Oct 20 02:14:10.519 CST: %SYS-5-CONFIG_I: Configured from console by console Oct 20 02:14:11.521 CST: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 150.1.1.254 started – CLI initiated |
In addition, the syslog daemon on server 150.1.1.254 would also reflect the same as illustrated below in the Kiwi Syslog Manager screenshot in Figure 1-1:
Fig. 1-1. Configuring Log Timestamp