Nvidia Mellanox ConnectX-6 MCX654106A-ECAT Bedienungsanleitung

Exported onOct/22/2020 07:16 PM

https://docs.mellanox.com/x/NCQuAQ

NVIDIA Mellanox ConnectX-6 InfiniBand/VPI

Adapter Cards User Manual

2
Table of Contents
Table of Contents .......................................................................................................2
Introduction................................................................................................................9
Product Overview .................................................................................................................... 9
ConnectX-6 PCIe x8 Card...................................................................................................... 10
ConnectX-6 PCIe x16 Card.................................................................................................... 11
ConnectX-6 Socket Direct™ Cards....................................................................................... 11
ConnectX-6 Dual-slot Socket Direct Cards (2x PCIe x16) .............................................. 12
ConnectX-6 Single-slot Socket Direct Cards (2x PCIe x8 in a row) ............................... 13
Package Contents ................................................................................................................. 14
ConnectX-6 PCIe x8/x16 Adapter Cards.......................................................................... 14
ConnectX-6 Socket Direct Cards (2x PCIe x16)............................................................... 14
Features and Benefits........................................................................................................... 15
Operating Systems/Distributions .................................................................................... 17
Connectivity ...................................................................................................................... 17
Manageability ................................................................................................................... 17
Interfaces .................................................................................................................18
InfiniBand Interface............................................................................................................... 18
Ethernet QSFP56 Interfaces ................................................................................................. 18
PCI Express Interface ........................................................................................................... 18
LED Interface......................................................................................................................... 18
Heat Sink Interface................................................................................................................ 19
SMBus Interface.................................................................................................................... 20
Voltage Regulators................................................................................................................ 20
Hardware Installation ..............................................................................................21
Safety Warnings..................................................................................................................... 21
Installation Procedure Overview........................................................................................... 21
System Requirements........................................................................................................... 22
Hardware Requirements ................................................................................................. 22
Airflow Requirements ........................................................................................................... 22
Software Requirements ................................................................................................... 23
Safety Precautions ................................................................................................................ 23
Pre-Installation Checklist..................................................................................................... 24

3
Bracket Replacement Instructions ...................................................................................... 24
Installation Instructions........................................................................................................ 25
Cables and Modules......................................................................................................... 25
Identifying the Card in Your System ..................................................................................... 26
ConnectX-6 PCIe x8/16 Installation Instructions................................................................. 27
Installing the Card............................................................................................................ 27
Uninstalling the Card ....................................................................................................... 29
ConnectX-6 Socket Direct (2x PCIe x16) Installation Instructions ...................................... 30
Installing the Card............................................................................................................ 31
Uninstalling the Card ....................................................................................................... 37
Driver Installation ....................................................................................................39
Linux Driver Installation........................................................................................................ 39
Prerequisites .................................................................................................................... 39
Downloading Mellanox OFED .......................................................................................... 39
Installing Mellanox OFED ................................................................................................ 41
Installation Script ........................................................................................................ 41
Installation Procedure ................................................................................................ 42
Installation Results .....................................................................................................44
Installation Logs..........................................................................................................44
openibd Script..............................................................................................................44
Driver Load Upon System Boot ..................................................................................45
mlnxofedinstall Return Codes....................................................................................45
Uninstalling MLNX_OFED...........................................................................................46
Installing MLNX_OFED Using YUM ................................................................................. 46
Setting up MLNX_OFED YUM Repository...................................................................46
Installing MLNX_OFED Using the YUM Tool .............................................................. 47
Uninstalling MLNX_OFED Using the YUM Tool .........................................................48
Installing MLNX_OFED Using apt-get Tool..................................................................... 48
Setting up MLNX_OFED apt-get Repository ..............................................................48
Installing MLNX_OFED Using the apt-get Tool..........................................................48
Uninstalling MLNX_OFED Using the apt-get Tool.....................................................49
Updating Firmware After Installation.............................................................................. 49
Updating the Device Online.........................................................................................49
Updating the Device Manually ....................................................................................50

4
Updating the Device Firmware Automatically upon System Boot ............................50
UEFI Secure Boot............................................................................................................. 51
Enrolling Mellanox's x.509 Public Key on Your Systems...........................................51
Removing Signature from kernel Modules ................................................................51
Performance Tuning ........................................................................................................ 52
Windows Driver Installation.................................................................................................. 52
Software Requirements ................................................................................................... 53
Downloading Mellanox WinOF-2 Driver .......................................................................... 53
Attended Installation...................................................................................................54
Unattended Installation...............................................................................................58
Installation Results .....................................................................................................59
Uninstalling Mellanox WinOF-2 Driver............................................................................ 60
Attended Uninstallation ..............................................................................................60
Unattended Uninstallation..........................................................................................60
Extracting Files Without Running Installation ................................................................ 60
Firmware Upgrade ........................................................................................................... 63
VMware Driver Installation ................................................................................................... 63
Hardware and Software Requirements........................................................................... 63
Installing Mellanox NATIVE ESXi Driver for VMware vSphere........................................ 64
Removing Earlier Mellanox Drivers................................................................................. 64
Firmware Programming .................................................................................................. 65
Updating Adapter Firmware ....................................................................................66
Troubleshooting .......................................................................................................67
GeneralTroubleshooting ...................................................................................................... 67
LinuxTroubleshooting .......................................................................................................... 67
WindowsTroubleshooting..................................................................................................... 68
Specifications ...........................................................................................................69
MCX651105A-EDAT Specifications ...................................................................................... 69
MCX653105A-HDAT Specifications....................................................................................... 70
MCX653106A-HDAT Specifications....................................................................................... 71
MCX653105A-ECAT Specifications ....................................................................................... 73
MCX653106A-ECAT Specifications ....................................................................................... 74
MCX654105A-HCAT Specifications....................................................................................... 75
MCX654106A-HCAT Specifications....................................................................................... 77

5
MCX654106A-ECAT Specifications ....................................................................................... 78
MCX653105A-EFAT Specifications........................................................................................ 80
MCX653106A-EFAT Specifications........................................................................................ 81
Adapter Card and Bracket Mechanical Drawings and Dimensions.................................... 82
ConnectX-6 PCIe x16 Adapter Card................................................................................ 83
ConnectX-6 PCIe x8 Adapter Card.................................................................................. 83
Auxiliary PCIe Connection Card...................................................................................... 84
Tall Bracket ...................................................................................................................... 84
Short Bracket ................................................................................................................... 84
PCIe Express Pinouts Description for Single-Slot Socket Direct Card....................85
Finding the GUID/MAC on the Adapter Card ............................................................86
Document Revision History ......................................................................................88

•

About This Manual

This User Manual describes NVIDIA® Mellanox® ConnectX®-6 InfiniBand/VPI adapter cards. It

provides details as to the interfaces of the board, specifications, required software and firmware for

operating the board, and relevant documentation.

Ordering Part Numbers

The table below provides the ordering part numbers (OPN) for the available ConnectX-6 InfiniBand/VPI

adapter cards.

OPN Marketing Description

MCX654106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR

InfiniBand and 100GbE), dual-port QSFP56, Socket Direct 2x

PCIe 3.0/4.0 x16, tall bracket

MCX653105A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and

100GbE), single-port QSFP56, PCIe3.0/4.0 Socket Direct 2x8 in

a row, tall bracket

MCX653106A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR

IBand100GbE), dual-port QSFP56, PCIe3.0/4.0 Socket Direct

2x8 in a row, tall bracket

MCX651105A-EDAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and

100GbE, single-port QSFP56, PCIe4.0 x8, tall bracket

MCX653105A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and

100GbE), single-port QSFP56, PCIe3.0/4.0 x16, tall bracket

MCX653106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s (HDR100, EDR IB and

100GbE), dual-port QSFP56, PCIe3.0/4.0 x16, tall bracket

MCX653105A-HDAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,

single-port QSFP56, PCIe3.0/4.0 x16, tall bracket

MCX653106A-HDAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,

dual-port QSFP56, PCIe3.0/4.0 x16, tall bracket

MCX654105A-HCAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,

single-port QSFP56, Socket Direct 2x PCIe3.0/4.0x16, tall

bracket

MCX654106A-HCAT ConnectX®-6 VPI adapter card, HDR IB (200Gb/s) and 200GbE,

dual-port QSFP56, Socket Direct 2x PCIe3.0/4.0x16, tall

bracket

Intended Audience

This manual is intended for the installer and user of these cards.

The manual assumes basic familiarity with InfiniBand and Ethernet network and architecture

specifications.

Technical Support

Customers who purchased Mellanox products directly from Mellanox are invited to contact usthrough

the following methods:

URL:http://www.mellanox.com> Support

E-mail:[email protected]

•Tel: +1.408.916.0055

Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails

regarding Technical Support.

Customers who purchased Mellanox products through a Mellanox approved reseller should first

seekassistance through their reseller.

Related Documentation

Mellanox OFED

for Linux User

Manual and

Release Notes

User Manual describing OFED features, performance, band diagnostic, tools content and

configuration. SeeMellanox OFED for Linux Documentation.

WinOF-2 for

WindowsUser

Manual and

Release Notes

User Manual describing WinOF-2 features, performance, Ethernet diagnostic, tools

content and configuration. SeeWinOF-2 for Windows Documentation.

Mellanox

VMware for

Ethernet User

Manual

User Manual describing the various components of the Mellanox ConnectX® NATIVE

ESXi stack. See http://www.mellanox.comProducts > Software > Ethernet Drivers >

VMware Driver > User Manual

Mellanox

VMware for

Ethernet Release

Notes

Release notes for Mellanox ConnectX® NATIVE ESXi stack. See http://

www.mellanox.comSoftware > Ethernet Drivers > VMware Driver > Release Notes

Mellanox

Firmware Utility

(mlxup) User

Manual and

Release Notes

Mellanox firmware update and query utility used to update the firmware. Seehttp://

www.mellanox.com> Products > Software > Firmware Tools > mlxup Firmware Utility

Mellanox

Firmware Tools

(MFT) User

Manual

User Manual describing the set of MFT firmware management tools for a single node.

SeeMFT User Manual.

IEEE Std 802.3

Specification

IEEE Ethernet specification at http://standards.ieee.org

PCI Express

3.0/4.0

Specifications

Industry Standard PCI Express Base and Card Electromechanical Specifications at

https://pcisig.com/specifications

Mellanox LinkX

Interconnect

Solutions

Mellanox LinkX InfiniBand cables and transceivers are designed to maximize the

performance of High-Performance Computing networks, requiring high-bandwidth, low-

latency connections between compute nodes and switch nodes. Mellanox offers one of

industry’s broadest portfolio of QDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/HDR100

(100Gb/s) and HDR (200Gb/s) cables, including Direct Attach Copper cables (DACs),

copper splitter cables, Active Optical Cables (AOCs) and transceivers in a wide range of

lengths from 0.5m to 10km. In addition to meeting IBTA standards,Mellanox tests every

product in an end-to-end environment ensuring a Bit Error Rate of less than 1E-15.Read

more at https://www.mellanox.com/products/interconnect/infiniband-overview.php

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in mega

Bytes. The use of Mb or Mbits (small b) indicates size in mega bits.IB is used in this document to mean

InfiniBand. In this document PCIe is used to mean PCI Express.

Revision History

A list of the changes made to this document are provided inDocument Revision History.

Introduction

Product Overview

This is the user guide for Mellanox technologies VPI adapter cards based on the ConnectX®-6

integrated circuit device. ConnectX-6 connectivity provides the highest performing low latency and

most flexible interconnect solution for PCI Express Gen 3.0/4.0 servers used in enterprise datacenters

and high-performance computing environments.

ConnectX-6 Virtual Protocol Interconnect® adapter cards provide up to two ports of 200Gb/s for

InfiniBand and Ethernet connectivity, sub-600ns latency and 200 million messages per second,

enabling the highest performance and most flexible solution for the most demanding High-

Performance Computing (HPC), storage, and datacenter applications.

ConnectX-6 is a groundbreaking addition to the Mellanox ConnectX series of industry-leading adapter

cards. In addition to all the existing innovative features of past ConnectX versions, ConnectX-6 offers a

number of enhancements that further improve the performance and scalability of datacenter

applications.

ConnectX-6 adapter cards are offered in a variety of PCIe configurations, as described in the below

table.

Configuration OPN Marketing Description

ConnectX-6 PCIe x8 Card MCX651105A-EDAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR IB and 100GbE, single-port

QSFP56, PCIe4.0 x8, tall bracket

ConnectX-6 PCIe x16 Card

MCX653105A-HDAT ConnectX®-6 VPI adapter card, HDR IB

(200Gb/s) and 200GbE, single-port QSFP56,

PCIe4.0 x16, tall bracket

MCX653106A-HDAT ConnectX®-6 VPI adapter card, HDR IB

(200Gb/s) and 200GbE, dual-port QSFP56,

PCIe3.0/4.0 x16, tall bracket

MCX653105A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR IB and 100GbE), single-port

QSFP56, PCIe3.0/4.0 x16, tall bracket

MCX653106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR IB and 100GbE), dual-port

QSFP56, PCIe3.0/4.0 x16, tall bracket

ConnectX-6 Dual-slot Socket Direct

Cards (2x PCIe x16)

MCX654105A-HCAT ConnectX®-6 VPI adapter card kit, HDR IB

(200Gb/s) and 200GbE, single-port QSFP56,

Socket Direct 2x PCIe3.0 x16, tall brackets

MCX654106A-HCAT ConnectX®-6 VPI adapter card, HDR IB

(200Gb/s) and 200GbE, dual-port QSFP56,

Socket Direct 2x PCIe3.0/4.0x16, tall bracket

Make sure to use a PCIe slot that is capable of supplying the required power and airflow to the

ConnectX-6 as stated in Specifications.



Configuration OPN Marketing Description

MCX654106A-ECAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR InfiniBand and 100GbE), dual-

port QSFP56, Socket Direct 2x PCIe3.0/4.0

x16, tall bracket

ConnectX-6 Single-slot Socket Direct

Cards (2x PCIe x8 in a row)

MCX653105A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR IB and 100GbE), single-port

QSFP56, PCIe3.0/4.0 Socket Direct 2x8 in a

row, tall bracket

MCX653106A-EFAT ConnectX®-6 VPI adapter card, 100Gb/s

(HDR100, EDR IBand100GbE), dual-port

QSFP56, PCIe3.0/4.0 Socket Direct 2x8 in a

row, tall bracket

ConnectX-6 PCIe x8 Card

ConnectX-6 with a single PCIe x8 slot can support a bandwidth of up to 100Gb/s in a PCIe Gen 4.0 slot.

Part Number MCX651105A-EDAT

Form Factor/Dimensions PCIe Half Height, Half Length / 167.65mm x 68.90mm

Data Transmission Rate Ethernet: 10/25/40/50/100 Gb/s

InfiniBand: SDR, DDR, QDR, FDR, EDR, HDR100

Network Connector Type Single-port QSFP56

PCIe x8 through Edge Connector PCIe Gen 3.0 / 4.0 SERDES @ 8.0GT/s / 16.0GT/s

RoHS RoHS Compliant

Adapter IC Part Number MT28908A0-XCCF-HVM

Dieses Handbuch passt für folgende Modelle

Andere Nvidia Computerhardware Handbücher

Nvidia

Nvidia DGX H100 Bedienungsanleitung

Nvidia

Nvidia Mellanox ConnectX-5 Bedienungsanleitung

Nvidia

Nvidia TESLA K20 Bedienungsanleitung

Nvidia

Nvidia ForceWare Quadro FX 4000 SDI Bedienungsanleitung

Nvidia

Nvidia RTX 6000 Bedienungsanleitung

Nvidia

Nvidia Geforce PCI-Express Series Bedienungsanleitung

Nvidia

Nvidia ConnectX-7 Bedienungsanleitung

Nvidia

Nvidia DGX A100 Bedienungsanleitung

Nvidia

Nvidia JETSON XAVIER NX Leitfaden

Nvidia

Nvidia TESLA M2090 Bedienungsanleitung

Nvidia

Nvidia GeForce GTX 980 Bedienungsanleitung

Nvidia

Nvidia PCI Express Series Bedienungsanleitung

Nvidia

Nvidia TURING GPU Bedienungsanleitung

Nvidia

Nvidia SP5200BLT Bedienungsanleitung

Nvidia

Nvidia TESLA K20 Bedienungsanleitung

Nvidia

Nvidia TESLA C2050 Bedienungsanleitung

Nvidia

Nvidia TESLA K20 Bedienungsanleitung

Nvidia

Nvidia DGX A100 Bedienungsanleitung

Nvidia

Nvidia Computer Bedienungsanleitung

Nvidia

Nvidia Mellanox ConnectX-5 Bedienungsanleitung

Nvidia

Nvidia BlueField-2 BF2500 Bedienungsanleitung

Nvidia

Nvidia Mellanox Bedienungsanleitung

Nvidia

Nvidia A800 40GB Active Bedienungsanleitung

Nvidia

Nvidia JETSON DEVELOPER KIT Montageanleitung

Beliebte Computerhardware Handbücher anderer Marken

EMC2

EMC2 VNX Series Betriebsanleitung

Panasonic

Panasonic DV0PM20105 Bedienungsanleitung

Mitsubishi Electric

Mitsubishi Electric Q81BD-J61BT11 Bedienungsanleitung

Gigabyte

Gigabyte B660M DS3H AX DDR4 Bedienungsanleitung

Raidon

Raidon iT2300 Bedienungsanleitung

National Instruments

National Instruments PXI-8186 Bedienungsanleitung

Hauppauge

Hauppauge WinTV-HVR-2250 Bedienungsanleitung

Kontron

Kontron CP6002 Bedienungsanleitung

Manhattan

Manhattan 702843 Bedienungsanleitung

Aphex

Aphex 230 Bedienungsanleitung

Intel

Intel NUC 12 Pro Bedienungsanleitung

Galaxy

Galaxy GHDX2-2430S-24F4D Bedienungsanleitung

Intel

Intel AXXRMFBU4 Bedienungsanleitung

Kontron

Kontron DIMM-PC/MD Bedienungsanleitung

ST X-NUCLEO-SAFEA1B Betriebsanleitung

STEINWAY LYNGDORF

STEINWAY LYNGDORF SP-1 Bedienungsanleitung

Advantech

Advantech ASMB-935 Series Bedienungsanleitung

Jupiter

Jupiter RAM PACK Bedienungsanleitung