Continuing with the Enterprise Campus design, the process of Layer 3 design optimization, including managing oversubscriptions, CEF polarization, routing protocol design, First Hop Redundancy Protocols (FHRPs), and Layer 2 to Layer 3 boundary design, will be analyzed.
Managing Oversubscription
In any network environment, situations will seldom be found in which all of the users access the same servers at the same time. The concept of data oversubscription is usually applied at the Access and Distribution layers and implies understanding that not everyone will be accessing the bandwidth at the same time.
Data oversubscription recommendations often imply a 20:1 ratio for the access ports on the Access Layer to Distribution Layer uplinks. Some congestion might exist but this will not happen very often and usually can be solved by applying QoS techniques. If the network experiences frequent congestion, then the design does not have enough uplink bandwidth. When experiencing congestion, the goal is to increase the bandwidth, not to lower the oversubscription ratio.
The recommendation for the Distribution Layer to Core Layer link is a 4:1 ratio. If the bandwidth capacity on the Access Layer links is increased, for example to 1 Gbps (or multiples of 1 Gbps through aggregation techniques), then techniques for the Distribution Layer to Core Layer links will have to be investigated (e.g., additional EtherChannel links or 10 Gbps links).
When the bandwidth for the Distribution Layer to Core Layer link starts to grow, oversubscription at the Access Layer must be controlled and key design decisions must be made. To accommodate for greater needs at the Access Layer for the end-stations, the solution would seem to be to increase the number of uplinks between the Distribution Layer and the Core Layer, but the problem with this is that it adds extra peer relationships, which will lead to extra management overhead.
Another solution involves using EtherChannels, which reduces the number of peers because a single logical interface is created. This solution has some caveats. For example, when dealing with OSPF on an IOS-based device, the link failure event will increase the link cost and this will force traffic to be rerouted, which might lead to convergence issues. To mitigate this issue, consider manually configuring OSPF interface costs. If OSPF is running on a hybrid-based switch, the link cost will not increase because of the use of EtherChannels, and this can result in an overload on the other links in the bundle, as OSPF keeps partitioning the traffic equally along the channels with different bandwidths. When using EIGRP, the link costs will not be modified at link failures because this routing protocol considers end-to-end costs.
An important configuration aspect when using EtherChannels with LACP involves the “min-links” feature, which defines the minimum number of member ports that must be in the “link up” state and bundled in the EtherChannel for that interface to transition to the “up” state. The goal of using this feature is to prevent low-bandwidth LACP EtherChannels from becoming active.
Another technique to support the greater needs for bandwidth is to use 10 Gbps links. This is recommended instead of using EtherChannels because it has the advantage of not adding to routing complexity and not increasing the number of routing peers. In addition, routing protocols can deterministically find the best path between the Distribution Layer and the Core Layer.
Cisco Express Forwarding Polarization
CEF polarization, illustrated below in Figure 2.7, is part of the link load balancing methodology:
Figure 2.7 – CEF Polarization
CEF operates on all of the multilayer switches in the figure above and it has a deterministic behavior. As packets initiate from the left Access Layer switch and traverse the network toward the right Access Layer switch, they all use the same input value for the CEF hash and this implies using the same path. This has the negative effect of not utilizing some of the redundant links in the topology, which is known as CEF polarization.
This can be avoided by tuning the CEF algorithm across the infrastructure layers. The default input hash value in Layer 3 is based on the source and destination values. If this input is changed to Layer 3 plus Layer 4, adding port number values to the calculation, the output hash value will also change.
Cisco recommends using alternating hashes in the Core and Distribution Layers to avoid ignoring or underutilizing redundant links. In the Core Layer, the default hashes based on Layer 3 information (i.e., source and destination addresses) should be used, but in the Distribution Layer, Layer 3 plus Layer 4 information should be used as input for the CEF hashing algorithm. This can be accomplished by using the “mls ip cef load-sharing full” command in global configuration mode on all Distribution Layer switches. This will force the CEF algorithm to use Layer 3 plus Layer 4 information to calculate the hash values. Doing this will eliminate the static path effect in the CEF decision-making process and will help to better balance the traffic across equal cost redundant links in the network.
To achieve the best EtherChannel load balancing scenario, use two, four, or eight ports in the port-channel bundle. In addition, tune the hashing algorithm that is implemented to select the certain EtherChannel link that will transmit a packet. By default, Layer 3 source and destination information will be used, but an additional level of load balancing can be provided by implementing Layer 4 TCP port information as the input for the algorithm. This can be accomplished with the “port-channel load-balance src-dst-port” command in global configuration mode. This command forces the EtherChannel algorithm to add Layer 4 port information as the input and this is the recommended practice.
Routing Protocols
Routing protocols are usually used in the Distribution Layer to Core Layer and the Core Layer to Core Layer connections. An alternative would be moving Layer 3 routing implementation to the Access Layer. This solution is becoming more popular because it avoids the complexity of implementing STP and lowers re-convergence times in large-scale deployments.
Three methods can be used to quickly re-route around failed links and failed multilayer switches to provide load balancing with redundant paths:
- Build redundant triangle topologies for deterministic convergence (avoid using squares).
- Control unnecessary peering across the Access Layer by peering only on transit links.
- Summarize at the Distribution Layer to Core Layer for optimal results (limit the EIGRP query diameter and propagation of OSPF Link-State Advertisements).
Figure 2.8 – Square versus Triangle Topologies
Analyzing Figure 2.8 above, it is usually the Access Layer devices that have redundant links to the upstream multilayer switches. Avoid using square topologies because this will lead to a higher degree of routing protocol convergence if there is a link failure. In other words, OSPF or EIGRP will have to take more time to calculate new paths using their specific route calculation algorithms. When using the triangle design, each node is directly connected to every other mode, so the convergence time is lower.
It is important to build the network Distribution Layer and Core Layer design based on triangles so that there are equal cost paths to all of the redundant nodes. This will avoid non-deterministic convergence, depending on the timers that come with the Layer 3 routing protocols. Another important routing protocol design consideration is to limit unnecessary peering across the Access Layer.
Figure 2.9 – Limiting Unnecessary Peering
Having Layer 3 peering across the different Access Layer switches (i.e., having multiple hosts on different switches across switch blocks) adds memory load, routing protocol update overhead, and more complexity. In addition, if there is a link failure, the traffic may transit through a neighboring Access Layer switch to get to another VLAN member. The goal is to eliminate unnecessary routing peering adjacencies (see Figure 2.9 above) and to configure the ports toward the Layer 2 switches as passive interfaces to suppress routing update advertisements. If a Distribution Layer switch does not receive a routing update from a potential peer on one of these interfaces, it will not process the updates and will not form a neighbor adjacency across that interface. The command for accomplishing this is usually “passive-interface [interface number]” in the routing process configuration mode.
The third method is to summarize at the Distribution Layer (i.e., toward the Core Layer but not sending queries beyond that point). By summarizing to the Core Layer, the EIGRP and OSPF re-convergence processes are simplified so the re-routing process is optimized. For example, when an Access Layer link goes down, the return traffic at the Distribution Layer to that device will be dropped until there is a convergence of OSPF or EIGRP and traffic is routed across another link.
When summarization is used at the Distribution Layer, summarizing toward the Core Layer, the individual distribution nodes will not advertise this failure of the Access Layer switch or loss of connectivity to a single VLAN or subnet. The Core Layer does not know that it cannot send traffic to the Distribution Layer switch where the Access Layer link failed, so the summarization toward the Core Layer limits the number of peers the EIGRP router must query or the number of OSPF Link-State Advertisements (LSAs) that must be processed, and this speeds up the re-routing process. This process can be accomplished by using the “ip summary-address eigrp” command for EIGRP or using an area border boundary for OSPF. To implement an efficient summarization scheme, an excellent hierarchical network addressing design and a contiguous address space is necessary.
First Hop Redundancy Protocols
Another critical aspect of convergence and high availability in the network design is the ability to have redundancy on behalf of the end-user in the Access Layer to the first hop (or the default gateway). With Layer 2 switches at the Access Layer and with Layer 3 switches at the Distribution Layer, the default gateway is usually the interface on the Distribution Layer switch that connects to the Access Layer.
| Note: Only FHRPs are needed if operating at Layer 2 in the Access Layer and Layer 3 in the Distribution Layer. If Layer 3 is extended to Access Layer switches, then these switches will be the default gateways for the end-stations. |
To recover from a failure of the devices that are performing as the default gateway for the end-stations, there are three FHRPs to choose from:
- Hot Standby Router Protocol (HSRP): This is a Cisco proprietary protocol, the most common and preferred option when using Cisco equipment.
- Virtual Router Redundancy Protocol (VRRP): This is an IETF standard.
- Gateway Load Balancing Protocol (GLBP): This is the most recent protocol (again, a Cisco invention) and the most sophisticated FHRP.
HSRP and Cisco-enhanced VRRP are the most common and robust options. VRRP is usually used in a multi-vendor environment, when interoperability with other vendor devices is needed. HSRP and GLBP can help achieve a convergence of less than 800 ms, and the parameters can be tuned to achieve fast convergence if there is a node or link failure at the Layer 2 to Layer 3 boundary.
Hot Standby Router Protocol
Analyzing Figure 2.10 below, there are two gateway routers that connect to one Layer 2 switch that aggregates the network hosts:
Figure 2.10 – Hot Standby Router Protocol
Router 1 is a potential default gateway (10.10.10.1) and Router 2 is another potential default gateway (10.10.10.2). The two routers will be configured in an HSRP group and they will serve clients with a virtual default gateway address of 10.10.10.3, although it is not assigned to any router interface and is only a virtual address.
In this example, Router 1 is the active gateway, as it is the one that is forwarding traffic for the 10.10.10.3 virtual address, while Router 2 is the standby gateway. The two routers exchange HSRP messages to each other to check on each other’s status. For instance, if Router 2 no longer hears from Router 1, it realizes that Router 1 is down and it will take over as the active HSRP device (if preemption is enabled on Router 2). These devices are transparently providing access for the clients, as they are transparently serving the clients as default gateways.
Virtual Router Redundancy Protocol
The way VRRP works is similar to that of HSRP. The difference is that two routers are configured in a VRRP group. In this case, Router 1 is the master (instead of the active) and it forwards all of the traffic, while Router 2 is the slave (backup) (instead of the standby) since it can have more than one device. An example of a Vrrp configuration is illustrated in Figure 2.11 below:
Figure 2.11 – Virtual Router Redundancy Protocol
As is the case for HSRP, the VRRP group presents a virtual IP address to its clients. An interesting aspect about VRRP is that it can utilize, as the virtual IP address, the same address that is on the master device. For example, in the figure above, the virtual address is configured as 10.10.10.1, identical to the address on the Router 1 interface.
Gateway Load Balancing Protocol
GLBP is the most unique of the FHRPs, as it not only has the ability to achieve redundancy but also has the ability to accomplish load sharing. In addition, it is much easier to use more than two devices. Figure 2.12 below illustrates a GLBP configuration:
Figure 2.12 – Gateway Load Balancing Protocol
Consider an example in which there are three routers configured in a GLBP group, which will be assigned a virtual default gateway address (10.10.10.4) that is also configured on the clients. In this example, Router 1 is elected as the Active Virtual Gateway (AVG) and Routers 2 and 3 are in the Active Virtual Forwarder (AVF) state.
When the hosts request ARPs for the 10.10.10.4 MAC address, the AVG responds to the ARP requests and performs a round robin with the virtual MAC addresses of the AVF machines. Router 1 responds to the first ARP it receives with its own virtual MAC address, then it responds to the second ARP it receives with Router 2’s virtual MAC address and then with Router 3’s virtual MAC address. The AVG can then round robin the traffic over the available AVF devices. The round robin simplistic balancing approach can be changed within the configuration with other load balancing techniques for GLBP.
| Note: The AVG can also function as an AVF and it usually does so. |
FHRPs and Preemption
An important aspect when dealing with FHRPs is their preemptive behavior. This allows an HSRP router to assume the primary role when its FHRP priority is higher than the current active router. If preemption is not used and the primary router fails, the standby router will assume the active role and it will maintain this role even if the other router comes back up. Preemption should be enabled because the HSRP active device is often also the STP Root, and a best practice is to set these functions on the same device.
Because of the HSRP preemptive behavior, it is important to know how long it will take for the device to reboot and get connected to the rest of the network. Knowing this time will allow proper timers configuration. If the HSRP neighbor relationships form and preemption occurs before the primary device has Layer 3 connectivity to the Core Layer, traffic to the Access Layer could be dropped until full connectivity is established. The recommended practice is to take into account the system boot time and all of the other factors that will influence the amount of time a device will not be able to forward traffic and to set the HSRP preemption delay accordingly. The command to achieve this is “standby preempt delay minimum” in the interface configuration mode. This value should be 50% greater than the amount of time it takes for the system to reboot and for convergence to happen.
Designing the Layer 2 to Layer 3 Boundary
Depending on the network topology and the devices used, different design models for the Layer 2 to Layer 3 boundary can be used. If the Access Layer uses Layer 2 devices and the Distribution Layer uses Layer 3 devices, the links between the Distribution Layer switches can be Layer 2 trunks to support Layer 2 at the Access Layer. When using Layer 2 trunks between the Distribution Layer switches, a best practice would be to use the Rapid Spanning Tree Protocol (RSTP) as the STP version and place the HSRP active device and the primary STP Root Bridge on the same Distribution Layer switch. This design is more complex than the one that uses Layer 3 interconnections between Distribution Layer devices because STP convergence must be taken into consideration.
Layer 3 interconnection design at the Distribution Layer is the recommended design and some of its advantages include the following:
- STP convergence is not an issue if there is an uplink failure.
- Layer 3 VLAN numbers can be mapped to Layer 3 subnets, reducing the management overhead.
Layer 3 interconnection design is often used with GLBP at the Distribution Layer to allow full utilization of the Access Layer uplinks. Even though GLBP has the advantage of offering full link utilization, the downside is it is less deterministic compared to HSRP.
The next design involves using Layer 3 at the Access and Distribution Layers in a routed network design model. This allows for faster convergence and easier implementation because only Layer 3 links are used. A routing protocol is used across the topology, such as EIGRP or OSPF. The most important advantages of a full Layer 3 environment are as follows:
- Simplified implementation
- Faster convergence
- Equal cost load balancing on all of the links
- Eliminates the need for STP
- FHRP configurations are not necessary
- The convergence time to reroute around a failed link is around 200 ms, unlike the 700 to 900 ms in previous scenarios (Layer 2 to Layer 3 boundaries)
However, the one downside to this design is that it does not support VLANs spanning the Distribution Layer switches; however, overall, it is a best practice to keep the VLANs in isolated blocks at the Access Layer. Since both EIGRP and OSPF load share across equal cost paths, this scenario offers a convergence benefit similar to using GLBP. Some of the key actions to perform when using a full Layer 3 topology with EIGRP to the edge include the following:
- Use EIGRP to the edge (to the Access Layer submodule, the WAN submodule, the Internet submodule, and to other submodules).
- Summarize at the Distribution Layer to the Core Layer, like in a traditional Layer 2 to Layer 3 design.
- Configure all the edge switches as EIGRP stub nodes to avoid queries sent in that direction.
- Control route propagation to the edge switches using distribution lists.
- Set the appropriate hello and dead timers to protect against failures where the physical links are active but the route processing has terminated.
Some of the key actions to perform when using OSPF to the edge include the following:
- Configure each Distribution Layer block as a separate totally stubby OSPF area. The distribution switches will become Area Border Routers (ABRs) and their core-facing interfaces will belong to Area 0 (the OSPF backbone).
- Avoid extending Area 0 to the edge switches because the Access Layer should not be a transit area in the Enterprise Campus environment. Each Access Layer switch should be configured to be a unique totally stubby area.
- The OSPF timers (including hello and dead intervals) should be tuned to improve convergence.
Other general recommendations include the following:
- Avoid daisy-chaining the Layer 2 switches together to prevent black holes if there is a link or node failure.
- Consider using Cisco StackWise technology to eliminate the danger of black holes in the Access Layer.
- A balance of redundancy should be determined, as too much redundancy (too many links from the Access Layer to the Distribution Layer) will cause design issues and will increase troubleshooting efforts. On the other hand, too little redundancy might lead to traffic being black holed and might severely affect convergence effects.
- Avoid VLANs spanning across Access Layer switches to prevent asymmetric routing.