Troubleshooting NAT

Before we delve into NAT troubleshooting, it is important to understand that while NAT provides the advantage of allowing private networks to communicate, it also has numerous limitations. These limitations are:

1. Breaking the end-to-end IP model
2. The need to maintain connection state issues
3. The inhabitation of end-to-end security
4. Applications that are not NAT-friendly
5. Address space collision
6. Ratio of internal and reachable IP addresses

IP (in general) was designed so that only network endpoints (i.e. hosts and servers) handle the connection. However when NAT is implemented, one or more intermediate NAT-enabled devices must terminate and re-originate each session that is translated. This breaks the end-to-end model of IP and may cause issues with some applications and protocols.

When NAT is used, NAT-enabled devices must maintain the connection state for each of the translations being performed. Depending on the number of translations, this can consume a significant amount of device resources, such as memory, and result in poor performance. In addition to this, because all packets must be processed, NAT can hinder network and device performance by introducing latency due to device processing delays. This results in longer round trip times between source and destination hosts which could cause severe performance problems for real-time applications such as voice and video.

NAT can also cause security headaches in identifying the sources of network breaches when traffic is coming from a NAT’d location. By hiding the true identity of the owner, NAT can make it very difficult to identify the true origin or address of an attacker. Another inhibition to end-to-end security is the incompatibility of NAT with cryptography and encryption. When some security algorithms, such as IP Security (IPsec) are employed, they cannot be used in conjunction with NAT because NAT changes the source address of packets before they are forwarded to their destination. This change causes the cryptography method to fail because it thinks that the packet has been tampered with along the way.

Not all applications work in the same manner. This means that some applications, for example applications that use proprietary protocols, etc, are not compatible with NAT. The two most common issues experienced with NAT are on applications that use embedded IP addresses or port numbers. When NAT changes these port numbers or IP addresses, as is the case with IPsec for example, the applications do not function as expected, or in some instances, cease functioning completely. This means that while NAT can be used to provide privately addressed devices access to public networks, it cannot be used in all instances.

Address space collision typically occurs when two or more organizations merge. Because RFC 1918 address space is commonly used internally in different organizations, with NAT enabling these privately addresses networks to communicate with the outside world; it is not uncommon for two organizations that merge to find out that they have the same address space. If re-IP addressing the network is not an option, and it usually is not because of the complexity involved, then ‘double-NAT’ is typically used to allow organizations with overlapping address space to communicate with each other. However, the downside to this is the additional complexity that is introduced into the environment.

And finally, NAT works well when there are a few internal hosts that must be accessed from external networks, such as the Internet. However, NAT can become an issue when multiple hosts need to be accessed from the Internet. Given the greatly depleted IPv4 address space, acquiring a large number of public addresses is not always possible. Because this is not always possible, it highlights another limitation of IPv4.

As is the case with DHCP troubleshooting, the most common sources of problems when using NAT are due to device misconfigurations. Common misconfigurations include the following:

1. Misconfigured NAT Access Control Lists
2. Asymmetric Routing
3. Misconfigured NAT Address Pool
4. Router Interfaces Missing NAT Commands

Misconfigured NAT ACLs are a common cause of NAT problems. You can use the show ip nat statistics command to determine the ACL that is used for translations, followed by the show ip access-lists command to verify that the ACL is configured correctly. Following is a sample output of the information printed by the show ip nat statistics command (highlighting the ACL used for NAT):

Asymmetric routing can also cause NAT problems wherein some protocols work, and others do not. Consider the topology in Figure 9-4 for example:

Fig. 9-4. Asymmetric Routing and NAT

Referencing Figure 9-4, NAT is enabled on R6. The inside interface (FastEthernet0/0) has been assigned the IP address 10.1.1.1/24. The outside interface (Serial0/0) has been assigned the IP address 200.1.1.1/24. NAT is not enabled the Serial0/1 interface. The routing table of R6 is illustrated in the diagram. This router uses the path via neighbor 200.1.1.2 to reach the 192.1.1.0/24 subnet. However, R6 prefers the path via neighbor 201.1.1.2 to reach the 172.1.1.0/24 and 150.1.1.0/24 subnets.

R1 uses the path via neighbor 192.1.1.1 to reach the 200.1.1.0/24 subnet. On R6, NAT has been configured to translate the address 200.1.1.5 to internal address 10.1.1.5, which is the address assigned to the FastEthernet0/0 interface of R7.

Next, R1 initiates a Telnet session to 200.1.1.5 (R7). This session is sourced from the Loopback 0 interface of R5 (150.1.1.254). The TCP SYN packet is sent via neighbor 192.1.1.1 and arrives at the Serial0/0 interface of R6. R6 translates the packet, which has a destination address of 200.1.1.5, to internal address 10.1.1.5 and forwards the packet out its LAN interface to R7. R7 sends a SYN ACK response to 150.1.1.254. This packet hits R6 which sends it out of Serial0/1, based on the routing table entries.

Because NAT is not enabled on Serial0/1, the source address of 10.1.1.5 is not changed. The packet is received by R1 with a source address of 10.1.1.5 and a destination address of 150.1.1.254. R1 is not aware of any TCP sessions to this source address and issues a RST to 200.1.1.5. The TCP session is terminated before it establishes. While this has been described as a NAT issue, the actual recommended solution would be to correct the asymmetric routing solution because NAT is operating in the manner it should be. Once the asymmetric routing issue has been resolved, R1 will be able to establish a Telnet session to R7.

Another common misconfiguration is an inadequate number of addresses in the NAT pool. When configuring NAT, there must be a one-to-one correlation between the public pool addresses and internal host addresses. If this is not the case, some host addresses will not be translated as the public pool will run out of addresses. In situations where the number of internal addresses exceeds the number of pool addresses, you should consider implementing Port Address Translation (PAT) instead. PAT allows multiple private IP addresses to a single public IP address by using different ports. This functionality is enabled by appending the overload keyword to theip nat inside source list <ACL> [interface|pool] <name> [overload] global configuration command in Cisco IOS software.

And finally, another common misconfiguration that also causes NAT problems is incorrectly designating interfaces as inside or outside, or even forgetting to designate the correct device interfaces as inside or outside. This is typically an issue on devices with multiple interfaces. Verify that the correct NAT configuration has been implemented by parsing the device configuration or by using the show ip nat statistics command, as illustrated below:

In addition to device misconfigurations, the following problems can also result in NAT issues:

1. Layer 1 and Layer 2 Issues
2. Layer 3 Issues
3. Device Resource Depletion
4. Incompatible Applications

Layer 1 and Layer 2 issues can also cause issues such as intermittent connectivity even when NAT is configured correctly on the device. Verify that everything at these layers is operating correctly using the appropriate commands.

As was described earlier in this section, Layer 3 issues such as asymmetric routing can and often do cause problems when NAT is implemented. In environments where multiple paths between source and destination exist, verify that routing is symmetric otherwise NAT will not function as expected. In addition to routing, also check for traffic filtering at Layer 3. This too can break NAT. Verify both local and transient device filtering using the appropriate suite of commands.

In some cases, the size of the NAT table can significantly increase, which consumes a lot of the resources, e.g. memory, available on the device. When this happens, the %NAT: System busy. Try later error message is printed on the console when a show command related to NAT or a show running-config or write memory command is executed. You can avoid this issue and protect the device using the ip nat translation max-entries <number> global configuration command which specifies the maximum number of NAT entries that are permitted in the NAT table of the router or switch.

And finally, as was stated earlier in this chapter, not all applications are NAT-friendly. In some cases, enabling NAT will break some applications and protocols, e.g. IPsec. It is important to understand the applications and protocols used in the network prior to implementing NAT otherwise you could spend endless hours troubleshooting something that will never work with Network Address Translation in the first place. In addition to IPsec, additional protocols that can also be affected by NAT because they use embedded IP addresses include FTP, Internet Relay Chat (IRC), Simple Network Management Protocol, H.323, Lightweight Directory Access Protocol (LDAP), and Session Initiation Protocol (SIP).

NOTE: Cisco IOS software supports a feature called NAT Transparency, which addresses the vast majority of incompatibilities between NAT and IPsec. However, it should be noted that it does not resolve all possible incompatibilities. NAT Transparency is beyond the scope of the TSHOOT exam and will not be described in any additional detail in this chapter or this guide.

In conclusion, given that NAT is a resource-hungry application, you should consider using the show commands described in this chapter when troubleshooting NAT. This includes using theshow running-config command to verify device configurations, as well as the show ip nat translation and show ip nat statistics commands to verify NAT operation. However, there may be situations where you have used these commands and still cannot resolve the NAT issue. In such cases, you can (with caution) use debugging commands to aid your troubleshooting efforts. Cisco IOS software supports the following options when debugging Network Address Translation. It should be noted that the options displayed below are based on Cisco IOS software version 12.4. Options will differ with other versions:

It should be noted that the majority of these options are beyond the scope of the TSHOOT exam. Commonly used debugs include the debug ip nat <ACL> command which restricts output to the address(es) permitted in the specified standard ACL, the debug ip nat command which displays information about each packet the device translates, and the debug ip nat detailed command which prints detailed information about each translated packet, including additional information such as protocol type and port numbers. The following is a sample output of the detailed information that is printed by the debug ip nat detailed command: