Nvidia BlueField BF1600 Bedienungsanleitung

Exported on Oct/31/2023 01:39 PM

NVIDIA BlueField BF1600 InfiniBand/Ethernet Controller Card User

Manual

2
Table of Contents
Introduction............................................................................................................................................ 9
Product Overview ................................................................................................................................................ 9
BlueField Multicore System-on-chip (SoC) ..................................................................................................................10
Features and Benefits ..........................................................................................................................................11
Operating Systems/Distributions .............................................................................................................................13
Package Contents ...............................................................................................................................................14
BF1600 Controller Card with Auxiliary Connection Card.............................................................................................. 14
BF1600 Controller Card without Auxiliary Connection Card.......................................................................................... 15
Supported Interfaces ................................................................................................................................ 16
Layout and Connector Information...........................................................................................................................16
BlueField BF1600 Controller Card ........................................................................................................................ 16
PCIe Auxiliary Connection Card........................................................................................................................... 20
PCIe Auxiliary Connection Card - Print Side ........................................................................................................ 22
Interfaces ........................................................................................................................................................23
BlueField System On Chip (SoC) .......................................................................................................................... 23
Ethernet QSFP56 Interfaces ............................................................................................................................... 23
QSFP28 Port Over-current Protection................................................................................................................ 23
PCI Express Interfaces...................................................................................................................................... 24
PCIe x16 Gen 3.0/4.0 Edge Connector............................................................................................................... 24

3
PCIe Gen 3.0 through Two CABLINE-CA II PLUS Connectors....................................................................................... 24
NCSI Management Interface ............................................................................................................................... 25
USB Connector............................................................................................................................................... 26
UART0 Interface............................................................................................................................................. 26
I²C0 Interface ............................................................................................................................................... 27
JTAG CoreSight 10 .......................................................................................................................................... 27
External PCIe Power Supply Interfaces .................................................................................................................. 28
External PCIe Power Supply Interface on the BF1600 Controller Card ......................................................................... 28
External PCIe Power Supply Interface on the Auxiliary PCIe Connection Card ................................................................ 29
CPLD Interface .............................................................................................................................................. 30
eMMC Interface.............................................................................................................................................. 30
DDR4 SODIMM Memory...................................................................................................................................... 31
SPI Flash Memory ........................................................................................................................................... 32
Networking LED Interfaces ................................................................................................................................ 32
CPLD Revision LEDs Interface ............................................................................................................................. 32
CABLINE-CA II PLUS Connector LED Interfaces ......................................................................................................... 33
Power-Good Signal LED Interface......................................................................................................................... 34
PCI Link Status LEDs ........................................................................................................................................ 34
Pin Description ....................................................................................................................................... 35
PCI Express Interface...........................................................................................................................................35
PCIe x16 Gen 3.0/4.0 Edge Connector................................................................................................................... 35

4
PCIe x16 Edge Connector Pins Description - Print Side............................................................................................ 35
PCIe x16 Edge Connector Pinouts Description - Component Side................................................................................ 41
PCIe Gen 3.0 through Two CABLINE-CA II PLUS Connectors....................................................................................... 47
CABLINE-CA II PLUS Connector - Component Side ................................................................................................. 48
CABLINE-CA II PLUS Connector - Print Side.......................................................................................................... 52
Power Sequencing........................................................................................................................................... 57
NCSI Management Interface ...................................................................................................................................57
External Power Supply Interfaces ............................................................................................................................59
External +12V Mechanical Pinout Diagram .............................................................................................................. 60
UART0 Interface.................................................................................................................................................60
JTAG CoreSight 10 Interface ..................................................................................................................................61
DDR4 DRAM Memory Interface ................................................................................................................................62
Cables and Cabling Configurations ................................................................................................................ 64
UART Interface Cabling ........................................................................................................................................64
USB Interface Cabling ..........................................................................................................................................65
Configuration Scenarios............................................................................................................................. 67
Configuration 1: Direct Connectivity to 4-8 NVMe SSDs (No External PCIe Switch)...................................................................67
Configuration 2: Direct Connectivity to 8-16 NVMe SSDs with (No External PCIe Switch) ...........................................................69
Configuration 3: Connectivity to 8-16 NVMe SSDs (with External PCIe Switch) .......................................................................70
Configuration 4: Connectivity to 24 NVMe SSDs (with External PCIe Switch)..........................................................................70
Clocks Distribution..............................................................................................................................................71

5
Thermal Sensors...................................................................................................................................... 72
Hardware Installation ............................................................................................................................... 73
Installation Procedure Overview..............................................................................................................................73
System Requirements ..........................................................................................................................................74
Hardware Requirements ................................................................................................................................... 74
Airflow Requirements .................................................................................................................................. 74
Software Requirements .................................................................................................................................... 75
Pre-Installation Checklist ......................................................................................................................................75
Bracket Replacement Instructions ...........................................................................................................................76
Removing the Existing Bracket............................................................................................................................ 76
Installing the New Bracket ................................................................................................................................ 76
Installation Instructions........................................................................................................................................76
BF1600 Controller Card Installation Instructions - PCIe x16.......................................................................................... 77
Connect the BF1600 Controller Card in an available PCI Express slot on the JBOF system. ................................................ 77
Secure the BF1600 Controller Card to the JBOF system. ......................................................................................... 79
BF1600 Controller and Auxiliary Connection Cards Installation Instructions - PCIe x32 ......................................................... 79
Connect the BF1600 Controller Card with the Auxiliary connection card using the supplied CABLINE-CAII PLUS harnesses........... 80
Connect the BF1600 Controller Card in an available PCI Express slot on the JBOF system. ................................................ 81
Connect the PCIe Auxiliary Connection Card in an available PCI Express slot on the JBOF system. ....................................... 82
Secure the BF1600 Controller and PCIe Auxiliary Connection Cards to the JBOF system.................................................... 83
Cables and Modules.............................................................................................................................................84

6
Networking Cables .......................................................................................................................................... 84
Bring-Up and Driver Installation ................................................................................................................... 86
Troubleshooting ...................................................................................................................................... 87
General Troubleshooting .......................................................................................................................................87
Specifications......................................................................................................................................... 88
Dual-slot Controller Cards Specifications ...................................................................................................................88
Single-slot Controller Card Specifications...................................................................................................................89
LED Interfaces ...................................................................................................................................................91
Networking LED Interfaces ................................................................................................................................ 91
CPLD Revision LEDs Interface ............................................................................................................................. 91
CABLINE-CA II PLUS Connector LED Interfaces ......................................................................................................... 92
BF1600 Controller Card Cabline-CA II Plus Connector LED ....................................................................................... 93
PCIe Auxiliary Connection Card Cabline-CA II Plus Connector LED .............................................................................. 93
Power-Good Signal LED Interface......................................................................................................................... 93
Finding the MAC on the Controller Card.......................................................................................................... 94
Document Revision History ......................................................................................................................... 99



•

About This Manual

This User Manual describes NVIDIA® BlueField® BF1600 Controller Card. It provides details as to the interfaces of the board, specifications, required

software and firmware for operating the card, hardware installation, driver installation and bring-up instructions.

Overview of Document Content

Section Section Description

Introduction Provides a general overview of the BlueField BF1600 Controller Cards and discusses its benefits and features.

Supported Interfaces Provides descriptions of all supported interfaces.

Pin Description Provides interfaces’ Pinouts description.

Configuration Scenarios Provides examples of different configuration scenarios for NVMe SSD connectivity.

Cables and Cabling Configurations Describes the required cables and cabling configurations.

Thermal Sensors Describes the available thermal sensors on the BlueField SoC

Hardware Installation Describes the procedures for installing and uninstalling the Controller Card.

Bring-Up and Driver Installation Describes driver installation and bring-up instructions

Troubleshooting Describes potential system problems - you can use this troubleshooting information to identify and resolve

the problem.

Specifications Lists the physical, electrical, operational and regulatory specifications of the BlueField

Intended Audience

This manual is intended for the installer and user of these cards.The manual assumes basic familiarity with Ethernet network and architecture

specifications.

Technical Support

Customers who purchased NVIDIAproducts directly from NVIDIA are invited to contact usthrough the following methods:

URL:https://www.nvidia.com> Support

• E-mail:[email protected]

Customers who purchased Mellanox M-1 Global Support Services, please see your contract fordetails regarding Technical Support.

Customers who purchased Mellanox products through a Mellanox approved reseller should first seekassistance through their reseller.

Related Documentation

IEEE Std 802.3

Specification

IEEE Ethernet specification at http://standards.ieee.org

PCI Express

Specifications

Industry Standard PCI Express Base and Card Electromechanical Specifications at https://pcisig.com/specifications

Mellanox LinkX

Interconnect

Solutions

Mellanox LinkX InfiniBand cables and transceivers are designed to maximize the performance of High-Performance Computing networks, requiring high-bandwidth, low-

latency connections between compute nodes and switch nodes. Mellanox offers one of the industry’s broadest portfolio of QDR/FDR10 (40Gb/s), FDR (56Gb/s), EDR/

HDR100 (100Gb/s), and HDR (200Gb/s) cables, including Direct Attach Copper cables (DACs), copper splitter cables, Active Optical Cables (AOCs) and transceivers in a

wide range of lengths from 0.5m to 10km. In addition to meeting IBTA standards,Mellanox tests every product in an end-to-end environment ensuring a Bit Error Rate of

less than 1E-15.Read more at https://www.nvidia.com/en-us/networking/interconnect/

Document Conventions

When discussing memory sizes, MB and MBytes are used in this document to mean size in MegaBytes. The use of Mb or Mbits (small b) indicates size in

MegaBits. In this document, PCIe is used to mean PCI Express.

Revision History

A list of the changes made to this document are provided inDocument Revision History.

Introduction

This is the User Manual for BlueField® BF1600 Controller Card. This document contains the complete installation instructions and specifications for the

BF1600 Controller Cards listed in the below table.

Ordering Part Numbers

The below table provides the ordering part numbers (OPN) for the available BlueField BF1600 Controller Cards:

OPN Marketing Description

MBF1M606A-CSNAT BlueField® Controller card, Dual Port 100Gb/s QSFP28, PCIe Gen4.0 x16, BlueField® E-Series 16 cores, Crypto disabled, No

Memory DIMM, FH¾L, Single Slot, Tall Bracket

MBF1M606A-ESNAT BlueField® Controller card, Dual Port InfiniBand/Ethernet 100Gb/s QSFP28, PCIe Gen4.0 x16, BlueField® E-Series 16 cores,

Crypto disabled, No Memory DIMM, FH¾L, Single Slot, Tall Bracket

MBF1M616A-CSNAT BlueField® Controller card, Dual Port 100Gb/s QSFP28, PCIe Gen4.0 x16, BlueField® E-Series 16 cores, Crypto disabled, 2x

16GB Memory DIMM, FH¾L, Single Slot, Tall Bracket

MBF1M626A-CSNAT BlueField® Controller card, Dual Port 100Gb/s QSFP28, BlueField E-Series 16 cores, PCIe Gen3.0 x16, Auxiliary Card PCIe

Gen3.0 x16, CABLINE-CA II PLUS 350mm, Crypto disabled, No Memory DIMM, FH¾L, Single Slot width, Tall Bracket

MBF1M636A-CSNAT BlueField® Controller card, Dual Port 100Gb/s QSFP28, BlueField E-Series 16 cores, PCIe Gen3.0 x16, Auxiliary Card PCIe

Gen3.0 x16, CABLINE-CA II PLUS 350mm, QSFP28, 2x 16GB Memory DIMM, FH¾L, Single Slot width, Tall Bracket

MBF1M646A-CSNAT BlueField® Controller card, Dual Port 100Gb/s, 100GbE dual-port QSFP28, PCIe Gen4.0 x16, Crypto disabled, 2x8GB Memory

DIMM, FH¾L, Single Slot, Tall Bracket

MBF1M656A-CSNAT BlueField® Controller card, Dual Port 100Gb/s QSFP28, BlueField® E-Series 16 cores, PCIe Gen3.0 x16, Auxiliary Card PCIe

Gen3.0 x16, 2 x I-PEX 350mm Cable, Crypto disabled, 2x 8GB Memory DIMM, FH¾L, Single Slot, Tall Bracket

Product Overview

The BlueField BF1600 Dual Port 100Gb/s Controller Card is the perfect solution for managing NVMe storage drives in storage and hyper-converged systems.

BlueField BF1600 Dual Port 100Gb/s Controller Card delivers the highest NVMe-oF target performance, reducing TCO and increasing ROI. The BlueField

BF1600 Controller Cards can transform existing JBOF systems into NVMe-oF compliant solutions, simply by plugging the card into an existing PCIe slot. The

card is available in two configurations: PCIe x16 and PCIe x32. This flexibility makes it easier to match different JBOF system configurations. The card’s

small form factor allows customers to install multiple controller cards in a single system to support a larger number of SSDs and high-availability storage

architecture.

BlueField Multicore System-on-chip (SoC)

BlueField SoC is a highly integrated and efficient controller, optimized for NVMe storage systems, Network Functions Virtualization (NFV), Cloud and AI

workloads.

BlueField SoC integrates all the discrete components of a storage system appliance into a single chip (CPU, PCIe switch, and Network controller), making it

the premier SoC solution for building Just-a-Bunch-Of-Flash (JBOF) systems, All-Flash-Array and storage appliances for NVMe over-Fabrics.

With an integrated NVMe-oF offload accelerator, the BlueField SoC gives the BlueField Dual Port 100Gb/s Controller Card a superior performance

advantage over existing JBOF systems, significantly reducing storage transaction latency, while increasing IOPs (I/O operations per second).

BlueField SoC also offers cost-effective and integrative solutions for Machine Learning appliances, enabling efficient data delivery for real-time analytics