HP 4X - DDR InfiniBand Mezzanine HCA Leitfaden

Abstract.............................................................................................................................................. 2

Introduction......................................................................................................................................... 2

InfiniBand technology........................................................................................................................... 4

InfiniBand components...................................................................................................................... 5

InfiniBand software architecture ......................................................................................................... 5

MPI............................................................................................................................................. 7

IPoIB ........................................................................................................................................... 7

RDMA-based protocols.................................................................................................................. 7

RDS ............................................................................................................................................ 8

InfiniBand hardware architecture ....................................................................................................... 8

Link operation.................................................................................................................................. 9

Scale-out clusters built on InfiniBand and HP technology......................................................................... 11

Conclusion........................................................................................................................................ 13

Appendix A: Glossary........................................................................................................................ 14

For more information.......................................................................................................................... 15

Call to action .................................................................................................................................... 15

Using InfiniBand for a

scalable compute infrastructure

technology brief, 3rd edition

Abstract

With business models constantly changing to keep pace with today’s Internet-based, global economy,

IT organizations are continually challenged to provide customers with high performance platforms

while controlling their cost. An increasing number of enterprise businesses are implementing scale-out

architectures as a cost-effective approach for scalable platforms, not just for high performance

computing (HPC) but also for financial services and Oracle-based database applications.

InfiniBand (IB) is one of the most important technologies that enable the adoption of cluster computing.

This technology brief describes InfiniBand as an interconnect technology used in cluster computing,

provides basic technical information, and explains the advantages of implementing InfiniBand-based

scale-out architectures.

Introduction

The overall performance of enterprise servers is determined by the synergetic relationship between

three main subsystems: processing, memory, and input/output. The multiprocessor architecture used

in the latest single-server systems (Figure 1) provides a high degree of parallel processing capability.

However, multiprocessor server architecture cannot scale cost effectively to a large number of

processing cores. Scale-out cluster computing that builds an entire system by connecting stand-alone

systems with interconnect technology has become widely implemented for HPC and enterprise data

centers around the world.

Figure 1. Architecture of a dual-processor single server (node)

CPU(s)

Memory

Controller

Processor 1

Memory

PCI-X

Bridge

PCIe

Port

Controller

Ultra-3 SCSI

Fibre Channel

Gigabit Ethernet

320 MBps

400 MBps

> 1 Gbps

Memory

Controller

Processor 2

CPU(s)

Processor

Input/Output

InfiniBand

> 10 Gbps*

* 4x link, single data rate

Figure 2 shows an example of cluster architecture that integrates computing, storage, and

visualization functions into a single system. Applications are usually distributed to compute nodes

through job scheduling tools.

Figure 2. Sample clustering architecture

Scale-out systems allow infrastructure architects to meet performance and cost goals, but interconnect

performance, scalability, and reliability are key areas that must be carefully considered. A cluster

infrastructure works best when built with an interconnect technology that scales easily, reliably, and

economically with system expansion.

Ethernet is a pervasive, mature interconnect technology that can be cost-effective for some application

workloads. The emergence of 10-Gigabit Ethernet (10GbE) offers a cluster interconnect that meets

higher bandwidth requirements than 1GbE can provide. However, 10GbE still lags the latest

InfiniBand technology in latency and bandwidth performance, and lacks native support for the fat-tree

and mesh topologies used in scale-out clusters. InfiniBand remains the interconnect of choice for

highly parallel environments where applications require low latency and high bandwidth across the

entire fabric.

InfiniBand technology

InfiniBand is an industry-standard, channel-based architecture in which congestion-management

capabilities and zero-copy data transfers using remote direct memory access (RDMA) are core

capabilities—resulting in high-speed, low-latency interconnects for scale-out compute infrastructures.

InfiniBand uses a multi-layer architecture to transfer data from one node to another. In the InfiniBand

layer model (Figure 3), separate layers perform different tasks in the message passing process.

The upper layer protocols (ULPs) work closest to the operating system and application; they define the

services and affect how much software overhead the data transfer will require. The InfiniBand

transport layer is responsible for communication between applications. The transport layer splits the

messages into data payloads and encapsulates each data payload and an identifier of the

destination node into one or more packets. Packets can contain data payloads of up to four kilobytes.

The packets are passed to the network layer, which selects a route to the destination node and, if

necessary, attaches the route information to the packets. The data link layer attaches a local identifier

(LID) to the packet for communication at the subnet level. The physical layer transforms the packet into

an electromagnetic signal based on the type of network media—copper or fiber.

Figure 3. Distributed computing using InfiniBand architecture

InfiniBand has these important characteristics:

Very high bandwidth—up to 40Gbps Quad Data Rate (QDR)

Low latency end-to-end communication—MPI ping-pong latency approaching 1 microsecond

Hardware-based protocol handling, resulting in faster throughput and low CPU overhead due to

efficient OS bypass and RDMA

Native support for fat-tree and other common mesh topologies in fabric design

InfiniBand Link

InfiniBand components

InfiniBand architecture involves four key components:

Host channel adapter

Subnet manager

Target channel adapter

InfiniBand switch

A host node or server requires a host channel adapter (HCA) to connect to an InfiniBand

infrastructure. An HCA can be a card installed in an expansion slot or integrated onto the host’s

system board. An HCA can communicate directly with another HCA, with a target channel adapter,

or with an InfiniBand switch.

InfiniBand uses subnet manager (SM) software to manage the InfiniBand fabric and to monitor

interconnect performance and health at the fabric level. A fabric can be as simple as a point-to-point

connection or multiple connections through one or more switches. The SM software resides on a node

or switch within the fabric, and provides switching and configuration information to all of the switches

in the fabric. Additional backup SMs may be located within the fabric for failover should the primary

SM fail. All other nodes in the fabric will contain an SM agent that processes management data.

Managers and agents communicate using management datagrams (MADs).

A target channel adapter (TCA) is used to connect an external storage unit or I/O interface to an

InfiniBand infrastructure. The TCA includes an I/O controller specific to the device’s protocol (SCSI,

Fibre Channel, Ethernet, etc.) and can communicate with an HCA or an InfiniBand switch.

An InfiniBand switch provides scalability by allowing a number of HCAs, TCAs, and other IB switches

to connect to an InfiniBand infrastructure. The switch handles network traffic by checking the local link

header of each data packet received and forwarding the packet to the proper destination.

The most basic InfiniBand infrastructure will consist of host nodes or servers equipped with HCAs, an

InfiniBand switch, and subnet manager software. More expansive networks will include multiple

switches.

InfiniBand software architecture

InfiniBand, like Ethernet, uses a multi-layer processing stack to transfer data between nodes.

InfiniBand architecture, however, provides OS-bypass features such as the communication processing

duties and RDMA operations as core capabilities and offers greater adaptability through a variety of

services and protocols.

While the majority of existing InfiniBand clusters operate on the Linux platform, drivers and HCA

stacks are also available for Microsoft® Windows®, HP-UX, Solaris, and other operating systems

from various InfiniBand hardware and software vendors.

The layered software architecture of the HCA allows writing code without specific hardware in mind.

The functionality of an HCA is defined by its verb set, which is a table of commands used by the

application programming interface (API) of the operating system being run. A number of services and

software protocols are available (Figure 4) and, depending on type, can be implemented from user

space or from the kernel.

Figure 4. InfiniBand software layers

As indicated in Figure 4, InfiniBand supports a variety of upper level protocols (ULPs) and libraries

that have evolved since the introduction of InfiniBand. The key protocols are discussed in the following

pages.

User APIs

uDAPL

SDP Library

MAD API

Open Fabrics Verbs, CMA, and API

IPoIB

Upper Level Protocols

Provider

User

space

Kernel

space

Hardware-Specific Driver

Hardware

MPIs

Open SM

Application Level

Mid-Layer Modules

Connection Manager

SA Client

MAD Services

Diag Tools

RDS

RDMA-based Protocols

(kDAPL, SDP, SRP, iSER, NFS)

InfiniBand HCA

Open Fabrics Verbs and API

SMA

Clustered DB

VNIC

IP-Based Access

Sockets-Based Access

Block Storage Access

File System Access

MPI

The message passing interface (MPI) protocol is a library of calls used by applications in a parallel

computing environment to communicate between nodes. MPI calls are optimized for performance in a

compute cluster that takes advantage of high-bandwidth and low-latency interconnects. In parallel

computing environments, code is executed across multiple nodes simultaneously. MPI facilitates the

communication and synchronization among these jobs across the entire cluster.

To take advantage of the features of MPI, an application must be written and compiled to include the

libraries from the particular MPI implementation used. Several implementations of MPI are on the

market:

HP-MPI

Intel MPI

Publicly available versions such as MVAPICH2 and Open MPI

MPI has become the de-facto IB ULP standard. In particular, HP-MPI has been accepted by more

independent software vendors (ISVs) than any other commercial MPI. By using shared libraries,

applications built on HP-MPI can transparently select interconnects that significantly reduce the effort

for applications to support various popular interconnect technologies. HP-MPI is supported on HP-UX,

Linux, True64 UNIX, and Microsoft Windows Compute Cluster Server 2003.

IPoIB

Internet Protocol over InfiniBand (IPoIB) allows the use of TCP/IP or UDP/IP-based applications

between nodes connected to an InfiniBand fabric. IPoIB supports IPv4 or IPv6 protocols and

addressing schemes. An InfiniBand HCA is configured through the operating system as a traditional

network adapter that can use all of the standard IP-based applications such as PING, FTP, and

TELNET. IPoIB does not support the RDMA features of InfiniBand. Communication between IB nodes

using IPoIB and Ethernet nodes using IP will require a gateway/router interface.

RDMA-based protocols

DAPL - The Direct Access Programming Library (DAPL) allows low-latency RDMA communications

between nodes. The uDAPL provides user-level access to RDMA functionality on InfiniBand while the

kDAPL provides the kernel-level API. To use RDMA for the data transfers between nodes, applications

must be written with a specific DAPL implementation.

SDP –Sockets Direct Protocol (SDP) is an RDMA protocol that that operates from the kernel.

Applications must be written to take advantage of the SDP interface. SDP is based on the WinSock

Direct Protocol used by Microsoft server operating systems and is suited for connecting databases to

application servers.

SRP –SCSI RDMA Protocol (SRP) is a data movement protocol that encapsulates SCSI commands over

InfiniBand for SAN networking. Operating from the kernel level, SRP allows copying SCSI commands

between systems using RDMA for low-latency communications with storage systems.

iSER –iSCSI Enhanced RDMA (iSER) is a storage standard originally specified on the iWARP RDMA

technology and now officially supported on InfiniBand. The iSER protocol provides iSCSI

manageability to RDMA storage operations.

NFS –The Network File System (NFS) is a storage protocol that has evolved since its inception in the

1980s, undergoing several generations of development while remaining network-independent. With

the development of high-performance I/O such as PCIe and the significant advances in memory

subsystems, NFS over RDMA on InfiniBand offers low-latency performance for transparent file sharing

across different platforms.

RDS

The Reliable Datagram Sockets (RDS) is a low-overhead, low-latency, high-bandwidth transport

protocol that yields high performance for cluster interconnect-intensive environments such as Oracle

Database or Real Application Cluster (RAC). The RDS protocol offloads error-checking operations to

the InfiniBand fabric, freeing more CPU time for application processing.

InfiniBand hardware architecture

InfiniBand fabrics use high-speed, bi-directional serial interconnects between devices. Interconnect

bandwidth and distances are characteristics determined by the type of cabling and connections

employed. The bi-directional links contain dedicated send and receive lanes for full duplex operation.

For quad data rate (QDR) operation, each lane has a signaling rate of 10 Gbps. Bandwidth is

increased by adding more lanes per link. InfiniBand interconnect types include 1x, 4x, or 12x wide

full-duplex links (Figure 5)., The 4x is the most popular configuration and provides a theoretical full-

duplex QDR bandwidth of 80 (2 x 40) gigabits per second.

Figure 5. InfiniBand link types

Encoding overhead in the data transmission process limits the maximum data bandwidth per link to

approximately 80 percent of the signal rate. However, the switched fabric design of InfiniBand allows

bandwidth to grow or aggregate as links and nodes are added. Double data rate (DDR) and

especially quad data rate (QDR) operation increase bandwidth significantly (Table 1).

Table 1. InfiniBand interconnect bandwidth

Link

SDR

Signal rate

DDR

Signal rate

QDR

Signal rate

2.5 Gbps

5 Gbps

10 Gbps

20 Gbps

40 Gbps

12x

30 Gbps

60 Gbps

120 Gbps

1x Link

(1 send channel,

1 receive channel,

each using differential voltage or

fiber optic signaling)

4x Link

12x Link

The operating distances of InfiniBand interconnects are contingent upon the type of cabling (copper or

fiber optic), the connector, and the signal rate. The common connectors used in InfiniBand

interconnects today are CX4 and quad small-form-factor pluggable (QSFP) as shown in Figure 6.

Figure 6. InfiniBand connectors

Fiber optic cable with CX4 connectors generally offers the greatest distance capability. The adoption

of 4X DDR products is widespread, and deployment of QDR systems is expected to increase.

Link operation

Each link can be divided (multiplexed) into a set of virtual lanes, similar to highway lanes (Figure 7).

Each virtual lane provides flow control and allows a pair of devices to communicate autonomously.

Typical implementations have each link accommodating eight lanes1; one lane is reserved for fabric

management and the other lanes for packet transport. The virtual lane design allows an InfiniBand

link to share bandwidth between various sources and targets simultaneously. For example, if a

10Gb/s link were divided into five virtual lanes, each lane would have a bandwidth of 2Gb/s. The

InfiniBand architecture defines a virtual lane mapping algorithm to ensure inter-operability between

end nodes that support different numbers of virtual lanes.

Figure 7. InfiniBand virtual lane operation

1The IBTA specification defines a minimum of two and a maximum of 16 virtual lanes per link.

Sources

Targets

QSFP

CX4

When a connection between two channel adapters is established, one of the following transport layer

communication protocols is selected:

Reliable connection (RC) –data transfer between two entities using receive acknowledgment

Unreliable connection (UC) –same as RC but without acknowledgement (rarely used)

Reliable datagram (RD) –data transfer using RD channel between RD domains

Unreliable datagram (UD) –data transfer without acknowledgement

Raw packets (RP) –transfer of datagram messages that are not interpreted

These protocols can be implemented in hardware; some protocols more efficient than others. The UD

and Raw protocols, for instance, are basic datagram movers and may require system processor

support depending on the ULP used.

When the reliable connection protocol is operating (Figure 8), hardware at the source generates

packet sequence numbers for every packet sent, and the hardware at the destination checks the

sequence numbers and generates acknowledgments for every packet sequence number received. The

hardware also detects missing packets, rejects duplicate packets, and provides recovery services for

failures in the fabric.

Figure 8. Link operation using reliable connection protocol

Andere Handbücher für 4X - DDR InfiniBand Mezzanine HCA

Dieses Handbuch passt für folgende Modelle

Inhaltsverzeichnis

Andere HP Software Handbücher

HP Brocade BladeSystem 4/12 Bedienungsanleitung

HP Surestore Disk Array 12h - And FC60 Bedienungsanleitung

HP ProLiant DL320e Bedienungsanleitung

HP BladeSystem bc2000 - Blade PC Service-Handbuch

HP C7791C - DesignJet 130 Color Inkjet Printer Konfigurationshandbuch

HP T5700 - compaq thin client Bedienungsanleitung

HP NonStop Pathway/iTS Betriebsanleitung

HP NonStop SSH 544701-014 Bedienungsanleitung

HP LASERJET PRO 400 Service-Handbuch

HP Vectra VL800 Bedienungsanleitung

HP SN6000B Bedienungsanleitung

HP StoreFabric SN6500B Service-Handbuch

HP HP NonStop Data Transformation Engine FTP... Bedienungsanleitung

HP Integrity rx5670 Bedienungsanleitung

HP ProLiant BL420c Bedienungsanleitung

HP Rx2620-2 - Integrity - 0 MB RAM Bedienungsanleitung

HP StorageWorks 2/64 - Core Switch Service-Handbuch

HP Compaq Presario,Presario 4400 Bedienungsanleitung

HP T1936AA - Digital Sending Software Gebrauchs- und Pflegehandbuch

HP D5970A - NetServer - LCII Bedienungsanleitung

HP BladeSystem bc2000 - Blade PC Bedienungsanleitung

HP ProBook 6445b - Notebook PC Bedienungsanleitung

HP dc5750 - Microtower PC Bedienungsanleitung

HP DDL D40 Bedienungsanleitung

Beliebte Software Handbücher anderer Marken

COMPRO

COMPRO COMPROFM Bedienungsanleitung

Muratec

Muratec OFFICEBRIDGE ONLINE Bedienungsanleitung

Oracle

Oracle Contact Center Anywhere 8.1 Bedienungsanleitung

Adobe

Adobe 65007312 - Photoshop Lightroom Handbuch

Avaya

Avaya NULL One-X for RIM Blackberry Bedienungsanleitung

PS Audio

PS Audio PowerPlay Betriebsanleitung

Brady

Brady LOCKOUT PRO 3.0 Service-Handbuch

Avaya

Avaya Interaction Center Bedienungsanleitung

Texas Instruments

Texas Instruments TI-83 Plus Silver Edition Handbuch

Novell

Novell GROUPWISE 8 - INTERNET AGENT Bedienungsanleitung

Oracle

Oracle Application 9i Bedienungsanleitung

Acer

Acer RDM Bedienungsanleitung

Canon

Canon Vixia HF21 Bedienungsanleitung

Canon

Canon ZR950 Bedienungsanleitung

Samsung

Samsung Auto Backup Bedienungsanleitung

Polycom

Polycom Vortex EF2201 Montage- und Bedienungsanleitung

Brocade Communications Systems

Brocade Communications Systems Brocade 8/12c Bedienungsanleitung

Novell

Novell SP2 - S Bedienungsanleitung