Light at the end of the tunnel

Storage area networks (Sans) were designed to ease the burden of managing exploding storage demands. Placing all storage on its own network creates a single point of management, and the technology allows multiple servers to access the same storage.

This is helpful for fail-over and clustering systems, as building a network for storage previously required using a new network protocol. Existing standards, such as Ethernet, are designed for short transfers of packets and don't scale well to the requirements of storage.

Servers will often have to stream large amounts of data, so a new protocol was developed, Fibre Channel. And, unlike other international standards, Fibre Channel has the 'correct' spelling, as the editor of the standard was British.

What's in Fibre Channel?

Fibre Channel is actually a combination of other standards, including Fibre Distributed Data Interface (FDDI), Small Computer System Interface (SCSI), High Performance Parallel Interface (HIPPI) and Intelligent Parallel Interface (IPI). We'll cover how these fit in as we walk through the Fibre Channel standard.

Topology

Fibre Channel offers the best parts of channels (point-to-point) and networks (an aggregation of hosts) through the Fabric. This is an intelligent interconnection scheme.

Hosts only have to monitor their own point-to-point connection to the Fabric. This reduces overhead placed on the host and allows transfer speeds of up to 2Gbps.

As hosts only have to connect into the Fabric, the exact topology is irrelevant and can be point-to-point, arbitrated loop, or fully switched. Exact topologies will depend on management and cost decisions during the implementation of the San.

Fibre Channel layers

Fibre Channel offers a reliable and fast transmission platform. Using single-mode fibre optic media and a longwave light source allows data to be transmitted 10km at 1Gbps. The protocol is defined in a similar way to the Open System Interconnection (OSI) 7-Layer model, although Fibre Channel only has five layers.

FC-0 Layer

As with the physical layer of the OSI model, the FC-0 layer is concerned with providing the physical link. This includes defining the fibre, connectors, optical and electrical parameters for all the available data rates.

This layer has been designed to use a wide range of different link technologies. A San may consist of many different link technologies to give the most efficient performance and cost.

FC-1 Layer

FC-1 defines the transmission protocol. It's responsible for serial transmission and encoding and decoding data. The system used is similar in operation to FDDI, but Fibre Channel uses a 10-bit transmission code opposed to the 5-bit code used by FDDI.

Data is encoded 8-bits at a time into the transmission character. This system introduces some redundancy into the system and allows some recovery in the event of an error.

FC-2 Layer

This layer defines the frame layout and header formats and serves as the transport mechanism. The layer is made up of a set of building blocks used to transfer data across a link. These are Ordered Set, Frame, Sequence, Exchange and Protocol.

Ordered Set

Ordered Sets are 4-byte transmission words that have a special meaning. For example, the start-of-frame and end-of-frame delimiters. Other commands include 'Idle', for declaring a node ready for transmission and reception, and 'Receiver Ready', indicating that the interface buffer can accept more frames.

Frame

Fibre Channel frames are used to contain the data transmitted and have a maximum length of 2,048 bytes. In addition, frames can also be Link Control frames, which are used to send messages including acknowledgements and rejections.

The Fabric has to accept frames from the source port and route them to the destination port. This layer is responsible for splitting data into frames and reassembling them at the other end.

Sequence

A sequence is a collection of related frames in transmission. Each frame is labelled with a unique sequence number so that they can be reassembled in the correct order at the destination end. Sequence numbers are also used for error control to make sure that all of the data has arrived.

Exchange

An exchange is one or more sequences for a single operation. They can be uni- or bi-directional, but only one sequence can be in operation at a time.

Protocol

The protocol defines the service offered by Fibre Channel. It may also be specific to high-layer services. Fibre Channel services include Fabric logon and data transfer.

Flow Control

Part of the job of this layer is to provide adequate flow control. This prevents buffers from being overrun at the receiver's end. Flow control is provided by sending control frames.

When a receiver's buffer is full, a 'Busy' frame is sent over the network. This causes the transmitter to back off until the receiver is ready.

Service Classes

This layer supports the three service classes that Fibre Channel offers. The first is circuit switching with guaranteed delivery in order, which is used by data channels. The second is packet switching with guaranteed delivery. The third is packet switching without guaranteed delivery.

FC-3 Layer

This layer provides common services required for advanced features. These include:
Striping - this multiplies bandwidth by using multiple ports in a large pipe.
Hunt groups - similar to hunt groups on telephones, this service allows multiple ports to respond to the same alias address.
Multicast - this service lets one transmission be delivered to multiple addresses.

FC-4 Layer

This is the highest layer in the Fibre Channel structure, and defines the application interfaces that can be run. Upper layer protocols are mapped onto the Fibre Channel levels below.

Fibre Channel can allow both network and channel information to be transmitted at the same time. The list of supported protocols is: HIPPI, IPI, SCSI, ATM and IP. See 'Jargon Buster', below, for an explanation of these protocols.

How does it fit into the network?

For hosts to connect into the Fibre Channel network, it is essential that a Fibre Channel host adapter is installed. This is a network card that gives a host a physical address and allows them to hook into the fabric.

Hosts include servers, clients, tape libraries and disk drive arrays. Once a client machine connects into the Fabric, it can access storage on any other machine subject to permissions.

This is useful beyond just having a single point of management. It also makes it easier to deploy clusters, as all of the host machines can see the shared storage. Using standard SCSI soon becomes complex when more than two machines are involved.

Sans have the advantage of range. Backup servers and tape libraries can be placed many kilometres from the main storage, effectively making them offsite. The speed of the network ensures that this isn't a noticeable problem.

Sans don't ignore existing network hardware either. Old SCSI boxes can be added into a San through the use of a SCSI to Fibre Channel bridge.

While not as good as pure Fibre Channel, it at least allows existing equipment to be integrated into a new storage domain.

FUTURE DEVELOPMENTS FOR SANS

Fibre Channel might be the technology for Sans at the moment, but new protocols are arriving to challenge its crown. Here are two of the most interesting ones:iSCSI

The internet SCSI is designed to run SCSI commands over IP. This will enable more reliable data transfers over existing networks and allow storage management over long distances.

It is hoped that iSCSI will aid the rollout of Sans, as well as improve the performance of storage over a network. As IP is a common protocol, iSCSI can run across the internet at relatively cheap prices.

It works by sending SCSI commands inside IP packets. The receiving station takes the command out of the packet and passes it to the SCSI controller, which makes the request to the storage. The result is packaged back up in an IP packet and returned to the originator.

Interestingly there's also an approach to develop Fibre Channel over IP. In a similar way to iSCSI, Fibre Channel commands will be packaged inside IP packets to allow distant Sans to be managed. However, this technology can only work with existing Sans, while iSCSI will run over other networks such as Ethernet.

InfiniBand

InfiniBand is the next generation of I/O architecture and is set to replace PCI. It offers throughput of up to 2.5Gbps and support for 64,000 addressable devices. It's actually the combination of two rival protocols: Future I/O, developed by Compaq, IBM and Hewlett Packard; and Next Generation I/O, developed by Intel, Microsoft and Sun Microsystems.

InfiniBand uses a serial bus in a similar way to a network. In fact InfiniBand IPv6 can transmit data in packages. The standard is used to communicate between the processor and storage devices.

WHAT ABOUT NAS?

Inevitably when talking about Sans, the discussion comes round to network attached storage (Nas). The technologies are complementary to each other and we'll discuss the relative benefits of each.

First, Fibre Channel and Sans are designed for mass storage. Standard network protocols are good for transferring small amounts of data in conversations.

However, when servers are talking to storage they usually stream a lot of data backwards and forwards. This requires a fast connection with guaranteed speed. Fibre Channel can offer this, other networks can't.

Nas is still a useful technology to have, though, particularly when used as quick-easy file servers. The boxes, while admittedly expensive, are generally cheaper than a full-blown server and easier to set up. As most users will only be writing small files to a Nas box, the limitations of the network don't really come into play.

In smaller companies, where the additional management task of a San isn't worth the additional budget and hassle, Nas is a perfect alternative.

JARGON BUSTER

HIPPI: The High Performance Parallel Interface is a standard for connecting two devices together over short distances. The standard version transfers 32-bits at transfer rates of 800Mbps. Wide HIPPI doubles the transfer rate to 1.6Gbps by transferring 64-bits in parallel.

IPI: The Intelligent Peripheral Interface is a high bandwidth interface between a computer and a storage device. The latest version, IPI-3, operates with transfer speeds of 25Mbps and supports Redundant Array of Inexpensive Disk.

SCSI: The small computer system interface is a parallel interface standard used for attaching a variety of peripherals to a host computer. SCSI is typically used for storage, hard disks, tape drives and CDRoms, but can also be used for peripherals such as printers and scanners. It is much faster and more flexible than its IDE cousin.

The current variants of SCSI are: SCSI-1 which has an 8-bit bus and supports transfer rates of 4MBps. It connects to the host via a 25-pin connector. SCSI-2 is identical to SCSI-1, but uses a 50-pin connector instead. It also supports multiple devices.

Wide SCSI: This uses a 68-pin connector and a wider cable to use a 16-bit bus.

Fast SCSI: This has an 8-bit bus, but doubles the clock rate to give transfer rates of 10MBps

Fast Wide SCSI: This has a 16-bit bus operating at the same clock rate as Fast SCSI to give 20MBps transfer rates.

Ultra SCSI: This has an 8-bit bus, but a faster clock rate to support transfer rates of 20MBps.

SCSI-3: This has a 16-bit bus and supports rates of 40MBps. Also called Ultra Wide SCSI.

Ultra2 SCSI: This has an 8-bit bus and supports transfer rates of 40MBps.

Wide Ultra2 SCSI: This has a 16-bit bus and supports transfer rates of 80MBps.

Ultra160 SCSI: This has a 16-bit bus, but transfers data on both the rising and falling edge of the clock signal to give transfer rates of 160MBps.