In 2007 the PCI-SIG approved the PCI Express External Cabling Specification that defines how PCI Express can be implemented over a standard cable. This new capability allows the full bandwidth of the PCIe bus to be utilized within multiple chassis systems and small local networks.

Over the past three years, hundreds of applications have been implemented using PCIe over cable for high-speed I/O, bus expansion and local networking. In these applications, using PCIe provides the benefits of high-performance, low power and reduced costs.

Applications Utilizing PCIe Over Cable

  • Medical – CAT scanners, PET scanners, dental scanners and other imaging
  • Office documentation centers – High-end copy centers
  • Entertainment – HD cameras, film editing and theater projectors
  • Instrumentation – Oscilloscopes and test equipment
  • Data Storage – Disk and flash arrays
  • HPC – GPU acceleration and inter-CPU communications
  • Military – signal acquisition and analysis
  • Other – PC expansion slots

PCIe as a Cabling Standard

Although PCIe is well known as the PC backplane interface standard, it is less known as a high-speed cabling interface.  Previous parallel bus structures such as PCI-X couldn’t easily be routed over a cable due to signal integrity problems.  The serial technology and embedded clocking used within PCIe allows it to be used at full speeds across a motherboard or over a cable.

Figure 1 – PCIe cables come in four sizes supporting x1, x4, x8 and x16 link widths.

The cable specification defines 4 cable connectors – for x1, x4, x8 and x16 links – providing a wide range of price and performance. At the low end, the x1 cable provides a 2.5Gb/s interface over a low-cost cable. At the high end, a x16 cable provides 128Gb/s over a more expensive cable.

Figure 2 – PCIe over cable adapter modules are available for desktop, laptop and industrial bus structures.

Adapter modules support most of the various cable sizes and are available in a number of form factors supporting desktops, laptops, CompactPCI, CompactPCIe, VMEbus, AMC and XMC.  Both the PCIe cables and adapter cards are now available from multiple sources.

Inside the Cable

The cabled version of PCIe contains the same signals as the backplane version of the bus structure.  These include the high-speed differential pairs that transfer data as well as a number of additional signals – known as sideband signals.

The differential pairs are organized into groupings called lanes.  One PCIe lane consists of four wires – one transmit pair and one receive pair of signals.  Each pair of wires is molded together with the pair held side by side, closely together.  Each pair is then individually shielded.  This helps insure that the wires stay length matched though the cable and that any incoming or radiated EMI is minimized.

PCIe provides a wide range of performance by increasing the number of lanes within a link.  A x1 – pronounced “by one” – link includes only one lane.  A x4 included four lane sets, and x8 includes eight sets and a x16 includes sixteen lane sets of wires.

The sideband signals provide additional functionality, but are not directly involved in the PCIe data transfers.  In fact it is possible to operate PCIe without any of the sideband signals present, although this would violate the specification.  The side band signals include:

  • Reference Clock (CREFCLKp, CREFCLKn): 100Mhz reference clock, used to implement spread-spectrum clocking over the bus.
  • Cable Present (CPERST#): Signals that the cable is connected between two systems.
  • Platform Reset (CPERST#): Allows the upstream host to reset the downstream sub-system.
  • Cable Power On (CPWRON#): Allows the upstream host to turn on the downstream sub-system’s power.
  • Sideband Return (SB_RTN): The electrical return for the sideband signals
  • 3.3V Power (+3.3V POWER, PWR_RTN): Provides power to the PCIe cable connector to power active components within the connector.  Note that these power connections are not bused across the cable.

Thus there are 10 wires contained within a x1 cable, 22 wires within a x4 cable, 38 wires within a x8 cable, and 70 wires within a x16 cable.

One can observe that there are no signals within PCIe for data clocking (clocking is embedded within each signal pair), transceiver direction (each signal is unidirectional), arbitration (each link is point-to-point, so there is no arbitration).  Each of these features helps enhance the performance of PCIe.

Gen 1, Gen 2 and Gen 3 Performance

One of the techniques used to create high performance with PCIe is the use of embedded clocking within each differential signal pair. By embedding the clock within the signal stream, there is no distortion or time delay between the clock signal and the data. The clock rate used within each PCIe signal is 2.5Ghz for Gen 1, 5Ghz for Gen 2 and 8Ghz for Gen 3. PCIe interface components automatically detect the speed capability of the other side of the link, and only utilize Gen 2 or Gen 3 timing if both sides are capable of that performance level.

The performance of various PCIe lane widths is:

 

Gen 1

Gen 2

Gen 3

x1

2.5Gb/s

5Gb/s

8Gb/s

x4

10Gb/s

20Gb/s

32Gb/s

x8

20GB/s

40Gb/s

64Gb/s

x16

40GB/s

80GB/s

128Gb/s

These performance levels are identical whether the bus is routed over a backplane or over the PCIe cable.

Most PCs today have a mixture of Gen 1 and Gen 2 PCIe slots. Similarly, I/O boards are available in a mix of Gen 1 and Gen 2. Low cost boards, such as USB or Ethernet controllers are typically Gen 1, whereas higher performance boards such as RAID controllers and graphics cards are generally Gen 2. Gen 3 systems and add-in boards are expected to become available in 2011. The first systems supporting Gen 3 will be based on Intel’s Sandy Bridge CPU, which brings PCIe Gen 3 directly out of the CPU chip. The first Gen 3 add-in cards are expected to graphics boards from nvidia and AMD/ATI.

Usage Models

With its industry standardization, high performance and low costs, PCIe over cable is well suited for connection to high speed I/O, bus expansion and local networking.

High Speed I/O

Many designers are choosing PCIe as a connection from a PC to their high-performance I/O device. This usage model is sometimes called “USB on steroids”, since the data rate of PCIe over cable is 20 to 80 times faster than USB. Designers are now developing a wide variety of I/O devices that directly connect to PCIe over cable. These applications include disk arrays, flash disk arrays, video editing subsystems, HD video cameras, hand-held scanning devices and many other data acquisition subsystems.

Bus Expansion

PCIe over cable is also being used as a simple, yet high performance means of bus expansion. In this model the PCIe backplane bus is conceptually stretched to operate over a cable to a second backplane physically located within a second chassis. These expansion chassis offer up to 20 additional add-in board slots for either laptops or servers. By using bridges and switches within the expansion chassis backplane, a variety of PCI, PCI-X and PCIe slots can be supported.

Some of the most economical expansion chassis are based on passive backplanes that have direct connections for PCIe cables. These backplanes are designed in the ATX form factor, so they can be used within standard PC-compatible cases.

Figure 3 – This ATX form factor expansion backplane contains direct PCIe cable connections.

A variation of the bus expansion usage model is the use of a standard PC connected to an industrial bus subsystem. This combination provides applications with the best of both worlds – a cost-effective, robust PC for the human interface and general computing and an industrial or instrumentation bus-based sub-system (such as PXIe, CompactPCIe, or VMEbus) for the specialized I/O. Since the connection rate of the PCIe cable is equal to the PCIe backplane rate, the performance of this repartitioned system is equal to the architecture where the CPU board is in the native form factor of the industrial bus.

Local Networking

PCIe can also be used for direct communications between PCs. In this model, PCIe over cable can be used for high-speed memory-to-memory transfers and/or used as a very fast TCP/IP connection, with application compatibility to standard Ethernet. Direct data transfers are used in designs that send data from one PC’s memory into another PC’s memory as rapidly as possible. This is particularly useful for multi-computing applications where one PC inputs data and then passes it to a second PC for data manipulation. This PC then passes the data to another PC for graphical display, and another PC for storage. In this example, the PCs could be physically separate communicating with PCIe over cable. The TCP/IP mode allows users to utilize TCP/IP applications to conveniently communicate between PCs, map and share disk drives, printers, etc.

Applications that are adopting PCIe over cable for local networking include medical imaging, document center internal data transfer, instrumentation data sharing and high-performance computing inter-networking.

Advantages of PCIe over Cable versus other Cable Standards

Whether being used for high speed I/O, bus expansion or as a high-speed network, there are several advantages associated with PCIe over cable – low cost, low power, high bandwidth and software transparency.

These advantages derive from the fact that today’s CPU components and chip sets directly drive PCIe as their expansion bus.

Thus, the PCIe cable adapter cards are quite simple. The adapters simply route the signals from the motherboard out to the PCIe cable connector and provide some signal conditioning to guarantee the signal integrity is met at the other end of the cable. Because these adapters are simple, they are inexpensive. Because they don’t convert the PCIe protocol into anything else, they are the high performance and don’t require any software drivers (for the I/O expansion models).

As an I/O expansion protocol, PCIe can be used in place of other protocols including: Starfabric, External SAS, Fiberchannel and USB. Compared to each of these, PCIe is the highest performance since there are no protocol conversions or timing delays. No cable connection can transfer data faster than the PCIe slot is capable of, and PCIe over cable is by definition at 100% performance. Any adapter that has to convert PCIe into a different protocol, send the data over the cable, and then convert it back to PCIe at the other end of the cable so that it can communicate with I/O devices, will necessarily be slower – both in throughput and latency – than a pure PCIe over cable solution.

For networking, the comparison is PCIe versus 1Gb and 10Gb Ethernet. 1Gb Ethernet is clearly the most well accepted and lowest cost solution. 10Gb Ethernet continues to demand high prices, particularly for switches, and is used sparingly.

PCIe over cable is much higher performance, but also more expensive than 1 Gb Ethernet. PCIe over cable compares favorably, however, to 10Gb Ethernet. In this comparison, PCIe spans a much broader performance range – 2.5Gb/s for a x1 Gen 1 up to 128Gb/s for a x16 Gen 3 – than 10Gb Ethernet. In general, a PCIe network will perform 2-4 times as fast as 10Gb Ethernet, while requiring half the power and costing half as much.

PCIe over Fiber Optic Cables

The PCI-SIG cable specification doesn’t specify fiber optic solutions for PCIe. Several vendors have however introduced active fiber optic cable products for PCIe. These cables plug into the standard PCIe over cable connector and then convert the signals into optical signals and send them over fiber optic cables. The primary advantages of these products is that the fiber cable can span considerably longer distances – typically up to 500 meters, and that the optical signals are completely immune to EMI interference.

Two types of applications are utilizing PCIe over cable with active fiber optic cables. First are outdoor data acquisition and security systems where the distance feature is key. The second is within heavy industry factory floor applications where the fiber optic natural immunity to EMI (from welding and other machines) is key.

PCIe as a Network

PCIe was originally defined to support CPU-to-I/O communications, with a single PC serving as the bus interface host. Multi-computing can be accomplished using PCIe through a combination of non-transparent bridging and CPU-to-CPU communications software. This technology expands the applicability of PCIe to a wide variety of high-end applications, including radar and sonar analysis, medical imaging, test and measurement and high-performance computing (HPC) inter-communications.

Figure 4 – PCIe can be used as a high-performance local networking architecture, connecting multiple PCs with software transparency to Ethernet.

The use of PCIe as a networking architecture is beginning to gain significant traction. This applies both to inter-blade communications (within a blade server) as well as external PCIe over cable connections between servers. Large HPC clusters with hundreds of interconnected nodes are currently being designed based on PCIe networking.

Summary – PCIe over Cable

The ability to run PCIe over cable at full performance with software transparency opens up a range of new application possibilities for CPU to I/O system re-partitioning. Low cost host bus adapters extend the PCIe bus structure to expansion chassis or dedicated PCIe I/O hardware. PCIe over cable provides a simple and low cost method for extending applications that need more I/O boards than will fit in a single chassis to a multi-chassis solution. PCIe over cable can also be used as a high performance peripheral connection – a super-fast USB of sorts. Designing compatible end-points is straight forward because the PCIe interface is available as a gate array library.

When CPU-to-CPU communications are added to PCIe, the cable interface can be used as a high performance cabled network. A x8 cabled network with Gen 2 timing will transfer data at 40Gb/s – or 40 times faster than today’s 1Gb/s Ethernet interfaces.

Having backplane performance levels available over a cable expands the PCIe usage model to encompass many high-end multi-chassis applications including I/O expansion, disk array subsystems, high speed video and audio editing equipment, medical imaging systems, and many more.

Share this post: