HP BL680c - ProLiant - G5 Referenzhandbuch

HP local I/O technology for ProLiant and BladeSystem

servers

technology brief

Abstract.............................................................................................................................................. 2

Introduction......................................................................................................................................... 2

The parallel bus system......................................................................................................................... 2

PCI Express technology ........................................................................................................................ 3

Server topology ............................................................................................................................... 4

Dual-simplex lanes ........................................................................................................................... 4

8b/10b encoding............................................................................................................................ 5

Performance .................................................................................................................................... 5

Backward compatibility .................................................................................................................... 6

Form factors .................................................................................................................................... 6

PCI Express card interoperability........................................................................................................ 8

75 Watt and 150 Watt PCI Express slots............................................................................................ 8

Emerging hardware support for I/O Virtualization................................................................................... 9

Intel VT-d and AMD IOMMU ............................................................................................................. 9

IOV – I/O Virtualization at the endpoint ........................................................................................... 10

HP Virtual Connect............................................................................................................................. 11

Ongoing ProLiant server transition strategy ........................................................................................... 11

Conclusion........................................................................................................................................ 12

For more information.......................................................................................................................... 13

Call to action .................................................................................................................................... 13

Abstract

This paper provides an overview of the continuing transition from parallel bus architecture to PCI

Express as well as details on developments that continue to occur within PCI Express. In addition, it

provides an introduction to the new I/O Virtualization technologies that are beginning to be

incorporated into server designs to better support server virtualization solutions that have emerged in

the last several years. It is the goal of HP to continue to support I/O technologies that protect customer

investments and extend I/O performance, flexibility, and reliability

Introduction

There is constant pressure on the Peripheral Component Interconnect (PCI) bus to meet the I/O

bandwidth demands of data traffic traveling between increasingly powerful CPUs and I/O devices.

The PCI standard has continued to meet the needs of CPUs and I/O devices by increasing

performance while maintaining backward compatibility. Historically, local I/O performance on

servers has doubled almost every two years with a new phase of the PCI standard.

With the advent of faster and more complex I/O devices, PCI-X 1.0, which defined device speeds of

PCI-X 66 and PCI-X 133, was introduced to extend the performance and RAS (reliability, availability,

and serviceability) features of PCI technology in servers. PCI-X 1.0 leveraged the wide acceptance of

the PCI bus and doubled the maximum bandwidth to 1066 MB/s with a 64-bit, 133-MHz bus. In

2002, the PCI-SIG extended the PCI-X 1.0 specification with PCI-X 2.0. The PCI-X 2.0 specification is

a smarter, more robust I/O technology that doubles and quadruples PCI-X bandwidth with two

additional device speeds: PCI-X 266 and PCI-X 533.

In 2002, the PCI-SIG also introduced a new physical implementation of PCI, called PCI Express,

which was originally optimized for desktop applications. With bandwidths in excess of 4 GB/s, PCI

Express delivers the performance required for next-generation 10-Gb Ethernet adapters.

This paper begins with a summary of today’s parallel bus system and then describes PCI Express

(PCIe) technology, including PCIe 2.0, as well as updating the status of the transition to PCI Express in

HP ProLiant servers. It also provides an introduction to the emerging I/O virtualization technologies

that HP is about to incorporate into server designs in order to enable the next generation Virtual

Machine solutions.

The parallel bus system

The parallel bus system based on PCI or PCI-X standards uses bit-parallel, bi-directional, multi-drop

connections running at relatively low frequency (Figure 1). PCI and PCI-X send 32 or 64 data bits plus

control signals over the parallel bus. The number of pins required corresponds to the width of the bus:

a 32-bit bus requires over 55 pins (data plus control) and a 64-bit bus requires approximately

100 pins.

As the speed of the bus increases, the number of devices that can be shared on a common bus

shrinks. For example:

•33-MHz PCI supports five or six slots.

•66-MHz PCI-X supports four slots.

•100-MHz PCI-X supports two slots.

•133-MHz and higher PCI-X supports one slot.

Figure 1. Typical parallel bus architecture

Memory

Controller

I/O

Bridge

I/O

Bridge

Standard

Peripherals

(KB, video,

mouse, etc.)

CPU

DRAM

iLO

RAID

Controller

NIC

PCI-X

100-MHz

Slots

PCI-X 133 Slots

The PCI and PCI-X specifications maintain full forward and backward compatibility with conventional

3.3-V, 33-MHz PCI; existing conventional 3.3-V, 33-MHz PCI, conventional 66-MHz PCI, PCI-X 66,

PCI-X 133, PCI-X 266, and PCI-X 533 add-in cards will operate in any PCI (3.3 V) or PCI-X system.

PCI-X 133 and 66-MHz PCI systems support universal add-in cards and 3.3-V PCI cards; however,

they do not support 5-V-only PCI cards, which operate only at 33 MHz.

The parallel bus system is highly cost effective with its small silicon footprint and low-frequency design

rules. However, economics have changed. Backward compatibility has become increasingly

expensive as speeds increase. Higher bus speeds require physically shorter connections, which yields

fewer slots. And as Moore’s Law dictates, silicon costs have decreased much faster than package and

pin count. As a result, the industry has transitioned to PCI Express technology, which has proven to be

a more cost-effective solution to provide the bandwidth required for future peripheral devices.

PCI Express technology

PCI Express 1.0 has a signaling rate of 2.5 Gigabits/second per direction per lane, for a combined

receive and transmit bandwidth of 500 Megabytes/second per lane. Multiple lanes can be combined

to form higher bandwidth links. For example, four lanes can be combined to form a 2-

Gigabyte/second link, designated “x4” (pronounced “by four”).

Today, the entire ProLiant server line incorporates support for PCI Express (PCIe) devices, as well as

support the PCI-X standard.

In January of 2007, the base specification for PCI Express 2.0 was released by PCI SIG. As

expected, the primary change is a doubling of the signaling rate to 5 Gb/s, resulting in doubling of

the aggregate bandwidth for PCIe 2.0 slots.

Introduction of PCIe 2.0 slots is currently planned to coincide with the release of the next generation

of HP ProLiant server designs.

Server topology

The PCI Express architecture (Figure 2) provides unidirectional point-to-point connections between

devices in two directions (dual-simplex). PCI Express sends the data serially, one bit after another,

over each link rather than sending the data in parallel, one bit beside another, as in PCI-X. Therefore,

PCI Express uses fewer pins.

The PCIe Root Complex shown in Figure 2 is the origin point for all of the native PCIe ports in the

system.

Figure 2. System architecture with PCI Express

CPU

DRAM

I/O Hub

PCIe Root Complex

PCIe/PCI-X

Bridge

RAID Controller

NIC

PCIe Slots PCI-X 133 Slots

Memory Controller

Dual-simplex lanes

A PCI Express serial link consists of one or more dual-simplex lanes. Each lane contains two pairs of

wire conductors (a send differential pair and a receive differential pair) to simultaneously transmit

data at the signaling rate in both directions. The maximum bandwidth of dual-simplex buses is often

specified as the sum of the transmit and receive channels (Figure 3 left). In contrast, PCI-X uses a half-

duplex scheme where the full bus bandwidth is used either to transmit or to receive data (Figure 3

right).

Figure 3. Dual-simplex communication and half-duplex communication

8b/10b encoding

Serial communication requires a device to convert parallel data into a serial bit stream and vice

versa. This device, called a serializer/deserializer (SerDes), contains a parallel digital interface,

First-In-First-Out (FIFO) caches, 8 bit/10 bit (8b/10b) encoder and decoder, a parallel-to-serial

converter, and a serial-to-parallel converter (Figure 4). The 8b/10b encoder converts each 8-bit data

byte to a 10-bit transmission symbol, which enables clocking information to be encoded into the data

stream. Although this adds about 20 percent overhead to the data stream, it is the most common data

signaling method when the signaling rates exceed 1 GHz.

Figure 4. The SerDes core integrates 8b/10b coding and decoding logic.

TX FIFO

RX FIFO

8b/10b

encoder

parallel-to-

serial

converter

8b/10b

decoder

serial-to-

parallel

converter

Transmit (TX)

Receive (RX)

8-bit parallel interface

SerDes (serializer/deserializer) core

10-bit serial data

LVD signaling +

out

Performance

As previously mentioned, each PCI Express link’s bandwidth can be linearly scaled by increasing the

number of lanes. PCI Express Gen1 has a signaling rate of 2.5 Gb/s per lane in each direction,

resulting in a unidirectional bandwidth of 250 MB/s after accounting for 8b/10b encoding

overhead. Combining eight lanes provides a total unidirectional bandwidth of 2 GB/s, which would

be sufficient to support the full wire speed of 10-Gb/s technologies such as 10-Gb Ethernet (about

1.25 GB/s, unidirectionally).

PCI Express 2.0 increases the signaling rate from 2.5 Gb/s to 5 Gb/s, giving it a unidirectional

bandwidth of 500 MB/s per lane (after factoring encoding overhead). With this bandwidth, the PCI

Express 2.0 x4 link will support a 10-Gb Ethernet device, and a x8 link will support the high-speed

server I/O peripherals that are expected to exist through the end of the decade.

PCI-SIG is currently researching and defining specifications for a PCI Express 3.0 architecture. PCI

Express 3.0 will support a bit rate of 8 GT/s (Giga Transactions per second). The 8 GT/s bit rate

represents a net doubling of the delivered bandwidth of PCIe 2.0 and is achieved using increased

signaling speed and by removing the requirement for the 8b/10b encoding scheme used in prior

versions of the PCIe architecture, which imposed a 20 percent overhead on the raw bit rate.

Figure 5 illustrates the bandwidths of PCI, PCI-X, and PCI Express, which will provide sufficient I/O

bandwidth migration for server I/O growth requirements. PCI Express bandwidth is shown using the

sum of the receive and transmit bandwidths.

Figure 5. I/O bandwidth of PCI, PCI-X, and PCI Express (PCIe)

1000

2000

3000

4000

5000

6000

7000

8000

9000

Aggregate Bandwidth (MB/s

)

PCI

64/66

PCIe

PCI-X

100

PCI-X

133

PCIe

PCI-X

266

PCIe

PCIe 2.0

x16

PCIe

PCIe 2.0

1x IB

Dual U320 Quad U320

10GbE

10GbF C

1394b

1GbE

1GbF C

Backward compatibility

PCI Express is compatible with PCI at the software layers (operating system, applications, and

drivers). New software is required to take advantage of new and extended PCI Express architecture

features. The PCI Express configuration uses standard PCI plug and play and PCI Hot Plug

mechanisms. PCI Express is not compatible with PCI at the board or connector level.

Form factors

The PCI Express specification includes a range of form factors and connector sizes. Connector sizes

include the following:

•x1 connector for desktop I/O

•x4 and x8 connectors for server I/O

•x16 connector for desktop graphics and server I/O

The standard card form factor is similar to the PCI card. Figure 6 illustrates the standard card form

factor and connector sizes.

Figure 6. Standard card form factor and connector sizes

Standard card form factor Connector sizes

x16

The PCI Express standard includes the standard height full length card - which is the same size as

today’s PCI cards – as well as some additional options. Some desktop cards with a x1 connector may

be standard height but a maximum of half length. These cards are limited to a maximum power of 10

W. A low-profile card, which is half length, half height, and limited to 10 W, is also defined. Figure 7

illustrates the standard card sizes.

Figure 7. Standard card sizes

Standard height, full

length (max)

Standard height,

half length, x1,

<10W (desktop)

Low profile

The PCI Express specification also defines two other PCI-like form factors. A PCI Express Mini Card,

similar to the Mini PCI card will include a x1 connector for use in portable products. The ExpressCard

is similar to PCMCIA cards and will include a x1 PCI Express connector and a USB 2.0 connector for

use in desktop and portable products (Figure 8).

Figure 8. PCI Express Mini Card (left) and ExpressCard (right) form factors

PCI Express card interoperability

The PCI Express specification provides for significant interoperability features between PCI Express

add-in cards and slots. Up-plugging (for example, a x4 card into a x8 slot) is supported. Additionally,

the specification also supports down-shifting, which allows the creation of slots with larger physical

connectors that are wired for a smaller logical link to the server’s I/O hub. For best performance, of

course, a card should be used in a slot that supports its logical link size; however the availability of

down-shifted slots provides the flexibility to use larger cards (for example, x8) in slots with smaller

logical links if necessary.

Table 1. PCI Express add-in card interoperability

x4 Connector

x4 Link

x8 Connector

x4 Link

x8 Connector

x8 Link

x16 Connector

x8 Link

x16 Connector

x16 Link

x4 Card x4 Operation x4 Operation x4 Operation x4 Operation x4 Operation

x8 Card Not Allowed x4 Operation x8 Operation x8 Operation x8 Operation

x16 Card Not Allowed Not Allowed Not Allowed x8 Operation x16 Operation

It is HP’s design philosophy to provide larger connectors whenever possible. Most slots with x4 links

have x8 connectors and slots with x8 links come with x16 connectors, thus delivering maximum

flexibility. Connector and link width for the slots should be documented on the hood label of each

ProLiant server.

75 Watt and 150 Watt PCI Express slots

In general, the PCI Express specification calls for slots to support up to 25-watt power consumption for

each add-in card. In order to support the emerging high performance add-in cards that consume more

than 25 watts of power, the specification also supports the concept of a 75-watt slot. Slots rated for

75 watts are distinguishable from others by the required increase in spacing around them which is

needed to dissipate the additional heat associated with the higher power consumption.

At this point in time, the PCI-SIG has finalized specifications for a 150-watt PCI Express slot in order

to provide a standardized solution that can be used by designers/manufacturers of add-in cards with

large power requirements, including high-end graphics engines in particular. Since the PCI Express

connector itself is not designed to deliver more than 75 watts of total power, the additional power

must be delivered using a specific external connection between the system board and the card itself.

HP is beginning to implement a limited number of 75-watt slots into some of the ProLiant server line.

As for the 150-watt slot design, which is being driven primarily by the needs of high performance

graphics engines used in workstations, HP will provide this in server designs that require it.

Emerging hardware support for I/O Virtualization

As use of Virtual Machine technology in server environments continues to grow and as Virtual

Machine Manager (VMM) software evolves, the hardware industry continues to look for ways to

enable VMM software, helping to make it more efficient, more reliable and more secure.

One of the core technical challenges that all VMMs have to solve is the virtualization of I/O such that

each individual virtual machine on the server can access the real (physical) I/O devices of the

underlying server hardware. Any one of three different methodologies – emulation, paravirtualization

or direct assignment – can be employed to accomplish this task. Each of these methods also has

benefits as well as drawbacks.

Emulation and paravirtualization are somewhat similar in that they both involve the creation of a

software layer, or device model, that emulates each I/O device within a virtual machine. Guest OSs

in each virtual machine then communicate with this device model and not directly with the physical

I/O device in the server. That task is left to the VMM, which has to both translate the guest OS’s I/O

requests into actual I/O operations on the physical device as well as manage and schedule the

requests coming from the various virtual machines that have been installed.

Both emulation and paravirtualization do a good job of insulating the various virtual machines from

the server hardware as well as from each other. The price for this, of course, is significantly reduced

performance and potential I/O latency issues as a result of the extra layers of software emulation (the

device models), translation and arbitration that have been introduced into the system.

Direct assignment is designed to solve these issues by eliminating the need for a device emulation

layer in software. As its name implies, with this method each physical I/O device is directly assigned

to a given virtual machine, allowing the virtual machine (and typically only that virtual machine) to

perform I/O operations directly to the device. The result is significantly improved I/O performance in

VM environments. Because of its simpler architecture and improved performance characteristics,

direct assignment is becoming the direction in which VMM software is trending for the future.

Still, direct assignment is not without its challenges. I/O devices that use DMA can have problems

since the guest VM sees a virtualized subset of system memory while the devices themselves

communicate directly with the host physical memory addresses when transferring data. Today’s

VMM’s have to deal with this mismatch by employing software mechanisms to perform this translation

between the guest VM and the real physical addresses.

In order to address these types of issues regarding I/O virtualization, both Intel and AMD have

announced enabling technologies to better support I/O virtualization in VMM environments.

Intel VT-d and AMD IOMMU

Intel Virtualization Technology for Directed I/O (VT-d) and AMD I/O Virtualization Technology

(IOMMU) are new technologies that will be incorporated into processor chipsets in the near future.

Although they each have their own unique programmatic interfaces, both VT-d and IOMMU are

designed to accomplish the same task, to provide hardware-based functionality that VMMs and OSs

can utilize to provide better performing and more secure I/O virtualization for Virtual Machine

environments. In order to do so, they both provide mechanisms to solve the two most important

challenges of I/O virtualization under direct assignment – DMA and interrupt virtualization. In the

language of the PCI-SIG, the generic name for these technologies is Address Translation Services.

DMA virtualization enables system software to create multiple DMA protection domains, each of

which then represents a separate environment to which a subset of the host physical memory is

assigned. Access to a given domain can then be restricted to only the I/O device to which it is

assigned, thus providing the DMA isolation needed for the VMM to provide a more secure operating

environment for the virtual machines.

DMA virtualization services also provide for built-in address translation, transforming (or remapping)

the virtualized memory addresses in DMA requests from the I/O devices into the actual Host Physical

Address and providing caching of the most frequently used remapping entries. This allows VMM

software to offload the execution and management of address translations to the processor hardware

itself, resulting in significant performance improvements.

To complete the picture, both technologies also support the virtualization of interrupts, whether legacy

interrupts from I/O interrupt controllers or MSIs, by re-mapping them using an interrupt-remapping

table.

Although hardware support for I/O virtualization will be delivered by Intel and AMD, and

subsequently incorporated into the HP ProLiant architecture, primary responsibility for exploiting this

new technology will fall to the VMM and OS vendors. Their operating software will need to be

significantly revised in order to take full advantage of the performance and reliability improvements

that can be gained.

Intel VT-d and AMD IOMMU are expected to be incorporated into HP products with the release of the

next generation of the corresponding HP ProLiant servers.

IOV – I/O Virtualization at the endpoint

While Intel VT-d technology is designed to address I/O virtualization issues associated with the

processor complex, IOV (as it is commonly referred to) is designed to provide support for I/O

virtualization at the other end of the I/O stack, with the I/O devices themselves, frequently referred to

in architectural circles as I/O endpoints.

As stated earlier, direct assignment, in its purest form, involves the use of a given I/O device (such as

a NIC) by a single virtual machine. While this approach is cleaner than others, it also usually requires

the use of separate devices for each virtual machine.

IOV technology is designed to help solve this issue by specifying an architecture that will allow for the

creation of I/O devices (such as a NIC) that can support multiple virtual functions that share one or

more physical resources of the endpoint (Figure 10). To the system, each of these virtual functions

appears as a separate PCIe function which can then be directly assigned. This capability allows each

virtual machine to see its own separate, directly-assigned device when the actual physical device is

being shared by multiple VMs.

Andere Handbücher für BL680c - ProLiant - G5

Dieses Handbuch passt für folgende Modelle

Inhaltsverzeichnis

Andere HP Software Handbücher

HP HPE XP P9500 Bedienungsanleitung

HP D2D Bedienungsanleitung

HP 345646-001 - StorageWorks NAS 2000s External Storage... Montage- und Bedienungsanleitung

HP BL680c - ProLiant - G5 Bedienungsanleitung

HP C7791C - DesignJet 130 Color Inkjet Printer Konfigurationshandbuch

HP HP-UX 11i v3 Service-Handbuch

HP 3000 - 37XE System Bedienungsanleitung

HP StorageWorks MSA 2/8 - SAN Switch Bedienungsanleitung

HP C7791C - DesignJet 130 Color Inkjet Printer Konfigurationshandbuch

HP dc5700 - Microtower PC Bedienungsanleitung

HP SureStore Optical 2600fx Installationshandbuch

HP StorageWorks 2000fc Bedienungsanleitung

HP Neoview Messages Bedienungsanleitung

HP BladeSystem bc2000 - Blade PC Bedienungsanleitung

HP 4X - DDR InfiniBand Mezzanine HCA Leitfaden

HP StorageWorks MSA 2/8 - SAN Switch Bedienungsanleitung

HP VS160 - StorageWorks DLT Tape Drive Bedienungsanleitung

HP NonStop SSH 544701-014 Bedienungsanleitung

HP SysMan Bedienungsanleitung

HP Pro 3130 MT Bedienungsanleitung

HP StorageWorks 2/16 - SAN Switch Bedienungsanleitung

HP 500 G2 Sicherheitsleitfaden

HP Vectra VL800 Bedienungsanleitung

HP LeftHand P4000 - SAN Solutions Bedienungsanleitung

Beliebte Software Handbücher anderer Marken

COMPRO

COMPRO COMPROFM Bedienungsanleitung

Muratec

Muratec OFFICEBRIDGE ONLINE Bedienungsanleitung

Oracle

Oracle Contact Center Anywhere 8.1 Bedienungsanleitung

Adobe

Adobe 65007312 - Photoshop Lightroom Handbuch

Avaya

Avaya NULL One-X for RIM Blackberry Bedienungsanleitung

PS Audio

PS Audio PowerPlay Betriebsanleitung

Brady

Brady LOCKOUT PRO 3.0 Service-Handbuch

Avaya

Avaya Interaction Center Bedienungsanleitung

Texas Instruments

Texas Instruments TI-83 Plus Silver Edition Handbuch

Novell

Novell GROUPWISE 8 - INTERNET AGENT Bedienungsanleitung

Oracle

Oracle Application 9i Bedienungsanleitung

Acer

Acer RDM Bedienungsanleitung

Canon

Canon Vixia HF21 Bedienungsanleitung

Canon

Canon ZR950 Bedienungsanleitung

Samsung

Samsung Auto Backup Bedienungsanleitung

Polycom

Polycom Vortex EF2201 Montage- und Bedienungsanleitung

Brocade Communications Systems

Brocade Communications Systems Brocade 8/12c Bedienungsanleitung

Novell

Novell SP2 - S Bedienungsanleitung