Storage Area Networks

Storage is an issue that comes up in Enterprise environments, as networks are becoming more robust and are responsible for achieving more and more functions.

SAN Technology Overview

The main objective of Storage Area Networks (SANs) is concentrating all of the data that end-users and devices in the network need to access and separating it from the server infrastructure. SAN devices are pools of storage that everyone can access based on pre-defined rights. In this way, servers can share the storage space, thus increasing resource utilization efficiency and reducing the TCO.

Another important aspect is ensuring high levels of Input/Output (I/O) for servers and other devices, which allows seamless read and write access to the data in the same way as accessing a local hard drive.

As with every major technological area, there is some degree of SAN technology incompatibility between vendors when new generations of products are released, but the compatibility issues disappear as the technology becomes more mature. The main components of a SAN infrastructure are as follows:

• Host Bus Adapters (HBAs), which are very fast adapters that connect to the disk subsystem. HBAs connect via GigabitEthernet or 10-GigabitEthernet copper or optical links.
• The communication subsystem (this evolved from IDE and ATA connectivity to Fibre Channel communications), which allows a large number of devices to connect to the SAN at higher speeds over long distances.
• The actual storage system, which consists of an array of hard drives that can be considered “just a bunch of disks” (JBOD). Storage arrays are groups of devices that provide mass storage and the physical disk space can be virtualized into logical storage structures.

Disk arrays can be organized in various Redundant Array of Independent Disks (RAID) configurations. The most common levels of RAID are:

• RAID 0:
  • Functions by striping across multiple disks
  • Ensures increased performance
  • Offers no redundancy
  • RAID 1:
    • Functions by mirroring the contents on multiple disks
    • Usually offers the same performance as a single disk
    • Offers increased redundancy
    • RAID 3: Allows for error detection
    • RAID 5: Allows for error correction and performance enhancement

The predecessor of SANs is Direct Attached Storage (DAS), which involves specific disks associated with each server. This solution offers limited scalability and low I/O levels and it is difficult to back up. Network Attached Storage (NAS) involves placing certain disk arrays in an IP network and accessing it via specific protocols, such as Network File System (NFS) or Common Internet File System (CIFS). NAS systems offer slow I/O rates and sometimes offer less performance than DAS systems do.

The SAN concept describes a storage network that is separate from the rest of the network, but at the same time, it must be accessed and virtualized. The technologies that make this possible include the following:

• Small Computer System Interface (SCSI): This is a mature technology that offers reliable communication over short distances (up to 25 meters) and has a maximum bandwidth of 320 Mbps. This is a half-duplex technology that supports a maximum of 16 devices on the SCSI bus.
• Fibre Channel (FC): This is a modern technology that offers higher throughput and distances than traditional SCSI connections do. FC can extend SCSI networks and it provides its own protocol – Fibre Channel Protocol (FCP) – to ensure reliable communications, which is similar to TCP. FC offers connectivity over distances up to 10 km and ensures a bandwidth of 1 to 4 Gbps. In addition, it supports up to 16 million devices (practically an unlimited number), as opposed to SCSI, which is limited to 16 devices.

Cisco invented a storage virtualization concept called virtual SAN (VSAN) that allows the creation of logical storage units from physical storage. This technology is similar to VLAN’s functionality in that it allows the end-user to see only a logical space for data storage that has nothing to do with the physical storage system layout. The VSAN system handles the process of mapping to an actual physical location.

VSAN allows the overlay of multiple logical views of the storage space over the high-speed fabric environment that connects all of the storage devices. Traffic moves from one VSAN to another using enhanced ISL. The Cisco MDS 9000 family was the first generation of products that supported this technology.

Traffic moves between VSANs using Inter-VSAN Routing (IVR) as the fabric routing technology. This connects fabric environments without having to merge them and allows the sharing of centralized storage services across VSANs. IVR enables services and/or devices in different VSANs to communicate, for example, a replication service that is able to work from one VSAN to another VSAN without permitting any local device traversal.

The protocol used to route between VSANs is Fabric Shortest Path First (FSPF). This uses a link-state database, associate costs with each link, and uses a dynamic incremental version of Dijkstra’s algorithm to calculate the shortest path.

SAN environments should be designed with security in mind, beyond the features offered by virtualization. The technology for securing SANs is called zoning and this adds another logical layer of security on top of a SAN or VSAN by grouping fabric devices. Zoning can be implemented in two ways:

• Software zoning: Based on DNS technologies; not preferred.
• Hardware zoning: Performs the operations as part of the SAN equipment; Cisco-preferred option.

IBM invented an old communications protocol called Enterprise System Connection Protocol (ESCP), which had a fiber option for connecting their mainframes to the storage system. IBM and Cisco developed the Fibre Connection (FICON) protocol to replace the legacy ESCP protocol (limited to 200 Mbps). FICON runs over longer distances and offers a higher throughput to connect mainframes to SANs. Cisco devices support FICON and Feature Control Policy (FCP) on the same physical infrastructure.

Another SAN-related protocol is SANTap, which defines a standard for accessing data on a SAN. This is a protocol that allows third-party storage applications to be integrated into the SAN, thus offering enhanced functionality, including continuous backup and long-distance replication. SANTap allows data to be written to a storage device so that it can be duplicated by another appliance in the fabric.

SAN Design Considerations

Some of the most important SAN design considerations involve port density and topology. Port density considerations include:

• The number of currently necessary ports
• The number of ports needed in the future
• The expected SAN lifetime

Topology design considerations include:

• Whether the data center is on a single floor
• Whether the data center is in a single building
• Whether SAN extension is required (multi-building scenarios)

Network designers should also analyze what happens when there is oversubscription for mission-critical resources. Typically, a methodology for failing over to directly attached storage must be designed in the event the SAN is oversubscribed. Another issue which should be a concern is traffic management design issues, including:

• QoS
• Bandwidth reservation
• The way different servers have different performance requirements
• Traffic choice between data centers

Fault isolation is another important SAN design issue and this is usually solved using VSANs. VSANs are critical for logically isolating faults in one VSAN so that it does not affect the entire SAN. Regarding convergence and stability in SAN environments, redundancy can be built in on devices and between them. In addition, the overall number of switches and trunks should be minimized to increase overall network stability and to reduce convergence times.

Some of the most popular SAN designs include:

• Collapsed Core Layer architecture: This design collapses the Core Layer into the Distribution Layer when using high-end Cisco devices. With the first generation of this technology, the collapsed Core Layer architecture was not recommended because of limited high availability. However, newer generations of devices allow for the collapsed Core Layer architecture in many implementations because they offer fully redundant single chassis technologies.
• Dual-fabric collapsed Core Layer architecture: This is the enhanced version of the collapsed Core Layer architecture, providing additional redundancy between devices. The dual-fabric collapsed Core Layer architecture can be implemented on a small, medium, or large scale. For example, in a large-scale design, there can be two Core Layer switches for four Enterprise Edge switches, with the Enterprise Edge switches having 64-Gigabit connections to the Core Layer and the Core Layer devices, supporting 256-Gigabit throughput to the storage devices.

SAN extension is a continuously growing technology area because as the distance between Enterprise Data Center components continues to increase, connectivity between SANs and VSANs that logically make them up must be provided. Traditionally, this was accomplished using CWDM and DWDM technologies but these had distance limitations that led to the adoption of modern SAN extension technologies, including:

• Fibre Channel over IP (FCIP): Provides connectivity over any distance between SAN systems.
• iSCSI: Carries the SCSI protocol over IP.

Some other new developments in SAN extension include the following:

• Tape backup acceleration
• FCIP write acceleration
• Hardware-assisted data compression over FCIP
• Hardware-based IP security encryption