The data center concept has greatly evolved over the last few years, passing through many phases because of evolving technology, as illustrated in Figure 4.11 below.
Figure 4.11 – Evolution of Data Centers
At the time of their appearance, data centers were centralized and they used mainframes to manage their data. Mainframes were managed using terminals, which are still used in modern data centers because of their resiliency, although they can be considered legacy components of data centers.
Note: Managing the data in a data center refers to both the storing and the processing of data.
The second-generation data centers used a distributing processing model and introduced the client/server architecture. Business applications were installed on servers, which were accessed by clients using their PCs. This brought a cost benefit compared to the mainframe model.
The third-generation data centers are focused on modern technologies such as virtualization, which further reduces costs (i.e., communication equipment performance has increased while costs have decreased). These factors make this approach more efficient than the distributed data center model. Virtualization results in higher utilization of computing and network resources by sharing and allocating them on a temporary basis.
Data Center Components
From an architecture standpoint, modern data centers include virtualization technologies and processing, storage, and networking services. All of these features combined enable the following:
- Flexibility
- Visibility
- Policy enforcement
The three major components of the Cisco data center architecture framework are as follows:
- Virtualization:
- Cisco Nexus 1000V virtual switch for VMware ESX delivers per-virtual-machine visibility and policy control for SAN, LAN, and unified fabric.
- Cisco Unified Computing System (UCS) unifies data center resources into a single system that offers end-to-end optimization for virtualized environments.
- Virtualization of SAN and device contents helps to converge multiple virtual networks.
- All of the features above lead to simplification of the data center architecture and reduce the TCO.
- Unified fabric:
- Unified fabric technologies include Fibre channel over Ethernet (FCoE) and Internet Small Computer Systems (iSCSI) that usually offer 10 Gbps transfer rates.
- Unified fabric is supported on Cisco Catalyst and Nexus series (iSCSI). Cisco MDS storage series is designed and optimized to support iSCSI.
- Converged network adapters are required for FCoE.
- FCoE is supported on VMware ESX.
- Unified computing:
- Cisco introduced UCS as an innovative next-generation data center platform that converges virtualization, processing, network, and storage into a single system.
- Unified computing allows the virtualization of the network interfaces on servers.
- Unified computing increases productivity with temporal provisioning using service profiles.
Figure 4.12 – Topology of Data Centers
In Figure 4.12 above, the top layer of the data center topology includes virtual machines that are hardware-abstracted into software entities running a guest OS on top of a hypervisor (i.e., resource scheduler).
Below this layer are the unified computing resources, which contain the service profiles that map to the identity of the server. The identity of the server contains details such as the following:
- Memory
- CPU
- Network interfaces
- Storage information
- Boot image
The next layer, consolidated connectivity, contains technologies such as 10 GigabitEthernet, FCoE, and Fibre channel, and all of these are supported on the Cisco Nexus series. FCoE allows native Fibre channel to be used on 10 Gbps Ethernet networks.
Below this layer is the virtualization layer, which includes technologies such as VLANs and VSANs that provide connectivity for virtualized LANs and SANs by segmenting multiple LANs and SANs on the same physical equipment. The logical LANs and SANs do not communicate with each other.
The bottom layer is formed by the virtualized hardware, which consists of storage pools and virtualized network devices.
Server Considerations
Some very important aspects to consider when deploying servers in a data center include the following:
- Required power
- Rack space needed
- Server security
- Virtualization support
- Server management
The increasing number of servers used necessitates more power, and this has led to the need for energy efficiency in the data center. Rack servers usually consume a great deal of energy, even though they are low cost and provide high performance.
An alternative to standalone servers are blade servers. They provide similar computing power but require less space, power, and cabling. Blade servers are installed in a chassis that allows them to share the network connections and power, which reduces the number of cables needed.
Server virtualization is supported on both standalone and blade servers and provides scalability and better utilization of hardware resources, which increases efficiency. Server management is a key factor in server deployment, which is accomplished using different products that offer secure remote management capabilities.
Data Center Facility Spacing Considerations
Facility spacing and other considerations help to size the overall data center and decide where to position the equipment in order to provide scalability. The available space defines the number of racks that can be installed for servers and network equipment. An important factor to consider is the floor loading parameters.
Estimating the correct size of the data center has great influence on costs, longevity, and flexibility. Several factors must be considered, including the following:
- The number of servers
- The amount of storage equipment
- The amount of network equipment
- The number of employees served by the data center infrastructure
- The space needed for non-infrastructure areas: storage rooms, office space, and other areas
- The weight of the equipment
- Loading (this determines how many devices should be installed)
- Heat dissipation
- Power consumption and power type (including UPS and PDU)
An oversized data center will induce unnecessary costs; on the other hand, an undersized data center will not satisfy computing, storage, and networking requirements but will impact productivity.
When thinking about the data center facility, the following factors must be considered:
- Available space
- Floor load
- Power capacity
- Cooling capacity (required temperature and humidity levels)
- Cabling needs
Physical security is another important consideration because data centers contain equipment that hosts sensitive company data, which must be secured from outsiders. Access to the data center must be well controlled. In addition to physical security, other factors are of great importance, including fire suppression, which will help protect the equipment from disasters.
Data center facilities must be properly designed for future use because they are limited in capacity. They must also provide an infrastructure that can recover network services, data, and applications, as well as provide high availability.
Data center design must be considered early in the building development process, and this must be accomplished with a team of experts in various fields such as networking, power, heating, ventilation, and air conditioning. The team members must work together to ensure the component systems interoperate effectively.
Data Center Power Considerations
The power in the data center facility is used to power servers, storage, network equipment, cooling devices, sensors, and other additional systems. The most power-consuming systems include servers, storage, and cooling systems. The process of determining the power requirements for data center equipment is difficult because of the many variables that must be taken into consideration. In addition, power usage is greatly impacted by the server load.
Various levels of power redundancy can affect the capital and operational expenses, based on the chosen options. Determining the right amount of power redundancy to meet the data center’s requirements requires careful planning.
Estimating necessary power capacity involves collecting the requirements for all the current and future equipment, such as the following:
- Servers
- Storage
- Network devices
- UPS
- Generators
- HVAC
- Lighting
The power system should be designed so it also includes additional components such as PDUs, electrical conduits, and circuit breaker panels. Implementing a redundant system will provide protection for utility power failures, surges, and other electrical problems.
Some key considerations related to the data center power system include the following:
- Provide physical electrical infrastructure
- Design for redundancy
- Define the overall power capacity
Data Center Cooling Considerations
Based on the type of equipment used, careful heating and cooling calculations must be provided. Blade server deployments allow for more efficient use of space but increase the amount of heat per server.
The increased use of high-density servers must be addressed through careful data center design. Considerations for cooling need to be taken into account for proper sizing of the servers. Some cooling solutions to address increased heat production include the following:
- Increase the space between the racks
- Increase the number of HVAC units
- Increase the airflow between the devices
Data center equipment produces variable amounts of heat depending on the load. Heat has a negative effect of decreasing the reliability of the devices, so cooling must be used to control their temperature and humidity levels. This applies to data center subsystems, racks, and individual devices.
In order to design a proper cooling system, environmental conditions must be measured. Computing power and memory requirements demand more power, which leads to more heat being dissipated. The increase in device density also leads to an increase in the heat level.
The amount of heat can be reduced by designing proper airflow. Sufficient cooling equipment must be available to produce acceptable temperatures within the data center. An efficient technique is arranging the data center racks with an alternating pattern of “hot” and “cold” aisles. The cold aisle should have equipment arranged face to face and the hot aisle should have them arranged back to back. The cold aisle should have perforated floor tiles drawing cold air from the floor into the face of the equipment, while the hot aisle should be isolated in order to prevent hot air mixing with cold air.
Some other cooling techniques that can be used for equipment that does not exhaust heat to the rear include the following:
- Using cabinets with mesh fronts and backs
- Increasing the height of the raised floor
- Spreading out equipment onto unused racks
- Blocking unnecessary air escapes to increase airflow
Data Center Cabling Considerations
A passive infrastructure for the data center is essential for optimal system performance. The physical network cabling between devices determines how these devices communicate with one another and with external systems.
The cabling infrastructure type chosen impacts the physical connectors and the media type of the connectors. This must be compatible with the equipment interfaces. Two options are widely used today: copper and fiber optic cabling.
The advantages of fiber optics are they are less susceptible to external interferences and they operate over greater distances than copper cables do. Cabling must remain well organized in order to maintain the passive infrastructure easily. Cabling infrastructure usability and simplicity is influenced by the following:
- Number of connections
- Media selection
- Type of cabling termination organizers
All of these parameters must be considered in the initial design of the data center. Suitable cabling infrastructure design can prevent against issues such as the following:
- Hard to implement troubleshooting
- Downtimes
- Improper cooling
Using cable management systems is essential in preventing these issues. These systems consist of integrated channels located above the rack for connectivity. Cabling should be located in the front or rear of the rack for easy access. When data center cabling is deployed, space and device location constraints make cabling infrastructure reconfiguration difficult. Scalable cabling is essential for proper data center operation and lifespan. On the other hand, badly designed cabling will lead to downtime because of expansion requirements that were not planned in the design phase.
Enterprise Data Center Infrastructure
As with any enterprise network, the enterprise data center architecture follows the multilayer approach, and it can be structured in the Core, Distribution/Aggregation, and Access Layers. This is the most common model used, and it supports a variety of devices, including standalone servers, blade servers, and mainframes, as illustrated in Figure 4.13 below:
Figure 4.13 – Enterprise Data Center Infrastructure
In the diagram above, the Access Layer provides physical port density and both Layer 2 and Layer 3 services for server connectivity. The Aggregation Layer connects the Access and Core Layers and provides security (i.e., ACLs, firewalls, or IPSs) and other server farm services, such as caching, content switching, and SSL offloading. Redundancy is an essential factor at the Aggregation and Core Layers, as well as between them, in order to avoid single points of failure.
Data Center Core Layer
The data center’s Core Layer (Figure 4.14) is usually a centralized Layer 3 routing layer in which one or more Aggregation Layers connect. The Core Layer can inject a default route into the data center Aggregation Layer after the data center networks are summarized. If multicast applications are used, the data center’s Core Layer must also support IP Multicast features.
Figure 4.14 – Data Center’s Core Layer
Smaller data centers may use a collapsed Core Layer design that combines the Aggregation Layer and the Core Layer into a single entity. Some of the most important data center Core Layer features include the following:
- Low-latency switching
- 10 GigabitEthernet
- Scalable IP Multicast support
Data Center Aggregation Layer
The role of the data center’s Aggregation Layer (Figure 4.15) is to aggregate Layer 2 and/or Layer 3 links from the Access Layer and to connect upstream links to the data center’s Core Layer. Layer 3 connectivity is usually implemented between the Aggregation and Core Layers.
Figure 4.15 – Data Center’s Aggregation Layer
This is a critical point for security (Layer 4) and application services, including the following:
- Server load balancing
- Firewalls
- IPS services
- SSL offloading
These services are commonly deployed in pairs using Cisco Catalyst 6500 chassis clusters, and their role is to maintain the session state for redundancy purposes. By implementing this kind of architecture, the management overhead is reduced due to simplifying the number of devices that must be managed.
The boundary between Layer 2 and Layer 3 can be located at multilayer switches, firewalls, or content switching devices in the Aggregation (Distribution) Layer. If the enterprise needs separate policy and security zones, multiple independent Aggregation Layers can be built in order to support these requirements. This is also the place where FHRPs (e.g., HSRP or GLBP) are implemented. The Aggregation Layer is usually the place where the STP Root Bridges are positioned in order to help control the loop-free topology and support a larger STP processing load.
The most important data center Aggregation Layer benefits include the following:
- Aggregates traffic from the Access Layer and connects to the Core Layer
- Supports advanced security features
- Supports advanced application features
- Supports STP processing load
- Flexibility and scalability
Data Center Access Layer
The main purpose of the data center’s Access Layer (Figure 4.16) is to provide Layer 2 and Layer 3 physical port access for different kinds of servers. This layer consists of high-performance and low-latency switches. Most data centers are built using Layer 2 connectivity, although Layer 3 (routed access) is also available from a design standpoint. The Layer 2-based design uses VLAN trunking upstream, allowing aggregation services to be shared across the same VLAN and across multiple switches. Other benefits of Layer 2 access are support for NIC teaming and server clustering.
Figure 4.16 – Data Center’s Access Layer
Possible physical loops that might be present at Layer 2 are managed by STP. The recommended STP operation modes are Rapid Spanning Tree Protocol (RSTP) and Multiple Spanning Tree Protocol (MSTP), which assures improved scalability and fast convergence.
Using STP can be avoided by implementing routed access (Layer 3 switching). In this case, the hosts must be provided with default gateway information because the Access Layer switch becomes the first-hop router in the network.
The most important benefits of the data center’s Access Layer are as follows:
- Provides port density for server farms
- Provides high-performance, low-latency Layer 2 switching
- Supports single-homed and dual-homed servers
- Supports a mix of oversubscription requirements