Now that PCI Express has achieved nearly 100% acceptance in desktop and laptop PCs, designers are looking to leverage the technology into other form factors and applications. Over 60 form factors have now been defined using PCIe as the backplane bus architecture.
PCIe also has unique features that facilitate its usage as a box-to-box interconnect. Earlier this year, the PCI-SIG approved the specification for PCIe over cable implementations. With the standard in place, creative engineers are looking to capitalize on a new capability.
Three Usage Models
Allowing a high-speed backplane bus to go over a cable – with full speed with complete software transparency – opens up a number of applications. These applications generally fall into one of three architecture usage models as shown in figure 1.
The first usage is simple bus expansion. In this model the PCIe bus is conceptually stretched to operate over a cable. Some of the first products to come to market using PCIe over cable are expansion chassis that offer additional add-in board slots for either laptops or servers. Today’s PCs have such a mixture of different kinds of slots that is can be difficult to construct a system where multiple similar I/O boards are used. Using PCIe to cable to an external expansion chassis is a high-performance way to solve this problem. Expansion chassis can provide as many as 20 add-in board slots as shown in figure 2, and come in many varieties with different numbers of PCI, PCI-X and PCIe slots.
The bus expansion model can also be used to physically repartition a system’s CPU and I/O functions. Applications can now use a conventional external PC with a cable connection to a CompactPCIe or PXIe instrumentation chassis as shown in figure 3. This repartitioning has the advantages of a stand-alone PC, including greater functionality and lower cost than a PC in the I/O chassis form factor. Since the performance of the cable is identical to the native PCIe, there is no performance penalty for system re-partitioning.
A second usage model for PCIe over cable is that of connecting a PC to any high-performance I/O device. Some have called this model “USB on steroids”, since the data rate of PCIe over cable is 20 to 80 times faster than USB 2.0. The first products coming to market using PCIe over cable in this model are disk RAID arrays, solid-state disk subsystems and video editing subsystems. With PCIe interface cells now available for gate arrays, designers can design their own custom I/O solution in a custom gate array, and have it connect to PCIe over cable to communicate to the PC. Whereas in the past this would only work if the I/O functionality would physically fit within a add-in card, with PCIe over cable, the I/O functionality can be as large as it needs to be to get the job done. Over time, we can expect a number of high-end I/O devices to become available that have PCIe over cable and their connectivity to standard PCs.
The third usage model allows PCIe to be used for networking between PCs or between blades in a backplane as shown in figure 4. This usage model requires new hardware and software that is just now becoming available. In this model, PCIe over cable can be used for high-speed memory-to-memory transfers and/or simply look like a fast Ethernet. The direct data transfer mode is intended to designers who want to send data from PC-to-PC as rapidly as possible. This is particularly useful for multi-computing applications where one PC inputs data and then passes it to a second PC for data manipulation. This PC then passes the data to another PC for graphical display, and another PC for storage. In this example, the PCs could be physically separate communicating with PCIe over cable, or they could be physically located within the same chassis using CompactPCIe. The TCP/IP mode allows users to utilize any TCP/IP applications to conveniently communicate between PCs, map and share disk drives, printers, etc.
Advantages of PCIe over Cable versus other Cable Standards
Whether acting as a slot expander, a super-fast I/O port or as a high-speed network, there are several advantages associated with PCIe over cable – low costs, high bandwidth and software transparency.
These advantages derive from the fact that the PC’s backplane bus is already PCIe. Thus, the cable adapter boards don’t need to convert the protocol or change the speed or timing of any of the signals. The adapters simply route the signals from the motherboard out to the PCIe cable connector and provide some signal conditioning to guarantee the signal integrity is met at the other end of the cable. Sample adapters for the PC, laptop and CompactPCIe are shown in figure 5. Because these adapters are simple, they are inexpensive. Because they don’t convert the PCIe protocol into anything else, they are the incredibly high performance and don’t require any software drivers (for the I/O expansion models).
A comparison of PCIe over cable to other protocols is shown in figure 3. As an I/O expansion protocol, PCIe can be used in place of custom protocols, Starfabric, External SAS, Fiberchannel and other high-end I/O cabling schemes. Compared to each of these, PCIe is the highest performance since there are no protocol conversions or timing delays. No cable connection can transfer data faster than the PCIe slot is capable of, and PCIe over cable is by definition at 100% performance. Any adapter that has to convert PCIe into a different protocol, send the data over the cable, and then convert it back to PCIe at the other end of the cable so that it can communicate with I/O devices, will necessarily be slower – both in throughput and latency – than a pure PCIe over cable solution.
Four cable sizes
In the same way that there are four different connector sizes for PCIe slots, there are also four different cable sizes. These are known as x1 (pronounced “by one”), x4, x8 and x16. The numbers relate to the number of parallel lanes included within each link. A x1 contains a single link, meaning one transmit pair and one receive pair of wires. A x4 contains four sets of these wires and thus can achieve 4 times the performance. The performance rates of the four cable sizes are as follows:
- x1 = 2.5Gb/s
- x4 = 10Gb/s
- x8 = 20Gb/s
- x16 = 40Gb/s
Each of these performance rates will double as PCIe transitions from the currently available Gen 1 timing to Gen 2 timing. The first Gen 2 PCs and add-in boards have recently begun to ship, and should be commonplace by the end of 2008. A x16 connection running Gen 2 timing will be able to transfer data at 80Gb/s.
With four sizes available, PCIe over cable spans a wide range of price and performance. At the low end, the x1 cable operates at 2.5Gb/s with adapters and cables priced under $100. The x1 solution is particularly attractive for laptop slot expansion, where cost-effective I/O expansion is needed. At the mid-range, the x4 solution operates at 10Gb/s with adapters and cables priced at about $200. This level of performance is an excellent fit for most high-performance I/O and networking applications. The x8 level provides another doubling of performance to 20Gb/s with adapters and cables priced at about $300. This performance level is particularly attractive for RAID arrays and solid-state disks where most of the adapters are x8 products. The x16 size is the highest performance and highest priced variation, with adapters and cables priced at about $400. This size is primarily being used for connections to high-performance graphics sub-systems.
It appears that the Gen 2 products will provide double the performance at each size level with little or no cost premium. It is expected that the adapter boards will need to be upgraded to handle the higher speeds, but the cables will remain unchanged.
Why PCIe can go over cable at full speeds?
Traditional buses – such as the original PCI – cannot operate over cables at full performance. The signaling is single-ended with each signal line being defined as relative to a ground signal. Although this technique works well in circuit boards over short distances, it breaks down over longer distances when signal line transmission noise effects occur. In addition, traditional buses are designed such that the skew and noise on the clock line compared to each of the parallel data lines makes the transfers unreliable after only a few feet of cable length.
PCIe is different in two key ways. First, each signal is defined based on the voltage difference between tightly coupled wire pairs. This differential signaling technique works because any noise introduced into one of the wires is also introduced onto the second wire. Then when the voltages are compared, the noise element is eliminated. The second technique that changes PCIe is that the signal clocking is buried within the signal bit stream of each differential pair. This means that the clock and the signals are always in sync. The combination of these two factors – differential signaling and embedded clocking – enable PCIe operate at full performance over cable distances of up to 7 meters.
PCIe was originally defined to support CPU-to-I/O communications, with the basic PC serving as the controller host. CPU-to-CPU communications are more critical within the bus-board industry as designers commonly utilize multiple CPU boards to solve complex solutions. Typically one CPU board captures incoming data then passes the data onto one or more CPUs to perform data manipulation. Then often another CPU is used to display the resulting graphical representation. This style of multi-computing is being added to PCIe through a combination of non-transparent bridging and CPU-to-CPU communications software. This technology expands the applicability of PCIe to a wide variety of high-end applications, including radar and sonar analysis, medical imaging, test and measurement and communications equipment. For CompactPCI applications that currently utilize 2.16 Ethernet over the backplane, the advent of CPU-to-CPU communication over PCIe represents a compelling reason to upgrade to the new CompactPCIe architecture.
PCIe as a Network – ExpressNet
Special hardware and software drivers are required in order to utilize PCIe over cable as a network connection between multiple PCs. In the networking model, CPU components that normally initialize and control their local PCIe buses are interconnected. If each CPU tried to initialize and control the PCIe bus, the system wouldn’t work. The solution is to use non-transparent bridge technology on the cable adapter boards to isolate each side of the bus. With non-transparent bridges, each CPU can control their side of the bus, but not interfere with the portions of the bus controlled by other CPUs. Another feature required for networking over PCIe is the ability to isolate the spread-spectrum clocking coming from both sides of the non-transparent bridging. This can also be accomplished within the cable adapter board.
The software drivers – known as ExpressNet – include features that allow direct data transfers as well as TCP/IP transfers. A diagram of the ISO 7-layer model for communications is shown in figure 6. The software drivers initialize memory address translation registers in the non-transparent bridge components that allow data to be transferred from one CPU’s memory to another. With TCP/IP added, the interface is identical to a normal Ethernet port, except much faster. In addition to driver support, ExpressNet software handles hot-swap events and other networking functions. Currently ExpressNet software support is available for Linux and Windows, with additional OS support planned for 2008.
Future Performance Upgrades
Even though PCIe is already one of the fastest bus structures in the market, the PCI-SIG is committed to driving several generations of backward-compatible performance upgrades. The first performance upgrade, known as Gen 2 timing, doubles the transfer rate of each lane of PCIe from 2.5Gb/s to 5.0Gb/s. Since most boards utilize x4 or x8 lane connections, the Gen 2 data rates will typically be 20 or 40Gb/s. The first PCs to include Gen 2 timing are expected to be generally available in 2008. The PCI-SIG recently announced the basic timing and performance for Gen 3 PCIe. The individual lane performance for Gen 3 will double again, increasing transfer rates to 10Gb/s per lane. Gen 3 products are expected to become available by 2011. Backward compatibility will be achieved by each bus interface component starting the bus training cycle using the Gen 1 timing. If both sides of the interface are compatible with Gen 2 or Gen 3 timing, they will both shift to the higher performance.
Summary – PCIe over Cable
The ability to run PCIe over cable at full performance with complete software transparency opens up a range of new application possibilities for CPU to I/O system re-partitioning. Low cost host bus adapters extend the PCIe bus structure to expansion chassis or dedicated PCIe I/O hardware. PCIe over cable provides a simple and low cost method for extending applications that need more I/O boards than will fit in a single chassis to a multi-chassis solution. PCIe over cable can also be used as a high performance peripheral connection – a super-fast USB of sorts. Designing compatible end-points is straight forward because the PCIe interface is available as a gate array library.
When CPU-to-CPU communications are added to PCIe, the cable interface can be used as a high performance cabled network. A x8 cabled network with Gen 2 timing will transfer data at 40Gb/s – or 40 times faster than today’s 1Gb/s Ethernet interfaces.
Having backplane performance levels available over a cable expands the PCIe usage model to encompass many high-end multi-chassis applications including I/O expansion, disk array subsystems, high speed video and audio editing equipment, medical imaging systems, and many more.
- Figure 1 – PCIe over cable can be used in three architectural usage models – Bus Expansion, High-speed I/O Connectivity, and Networking
- Figure 2 – PCIe over cable can be used to connect to an expansion chassis with 20 additional PCIe slots.
- Figure 3 – PCIe over cable used for high-speed networking between PCs.
- Figure 4 – PCIe over cable bus adapters are available for PCs, Laptops and CompactPCIe
- Figure 5 – A diagram of the ISO 7-layer model as it relates to ExpressNet communications using PCIe over cable.