Photo By xzllll | All third-party marks are the property of their respective owners.

HPE Eth 4x25Gb 1p 620QSFP28 Adptr | RDMA(RoCE V2) 配置与测试

实践 | Practices 2025年2月16日

1.First | 序

书接上回

HPE Eth 4x25Gb 1p 620QSFP28 Adptr | 上机和初始化记录
1x1.First | 序 HPE Eth 4x25Gb 1p 620QSFP28 Adptr == QL45214HLCU-HP HPE 对其的概述为 * 620-QSFP28 1-port Ethernet adapter - 4 lanes 25GB, quad small form factor (QSFP) * Part Number * 840134-001 是一款采用 Qlogic 45604 芯片拥有单 QSFP28 端口和四个 25GB 通道的 NIC 即支持使用 1p QSFP28 100G 拆分出 4p SFP 25G 2x1.Features | 特征 以下内容摘取自 QuickSpecs
xzllll's blogxzllll

此篇记录

  1. HPE Eth 4x25Gb 1p 620QSFP28 Adptr 网卡配置 RoCE V2
  2. 测试RDMA (Remote Direct Memory Access) 设备带宽
  3. iperf3 Test
  4. iperf3 Test(bond0)

1.2.OS Version | 示例环境

  1. OS | 系统

  2. SW | 软件

    • rdma-core

    • infiniband-diags

    • libibverbs-utils

    • ibdev2netdev

    • perftest

    • iperf

  3. NIC | 网络适配器

    1. Huawei SP333 STL2IEA2A 2-Port NIC

      • 双口25GB
      • 用于对接 HPE Eth 4x25Gb 1p 620QSFP28 Adptr

    2. HPE Eth 4x25Gb 1p 620QSFP28 Adptr

      • QSFP28 to 4x25G SFP28
        • 4x25G分线 其中 C/D 两根分线插入 Huawei SP333 STL2IEA2A 2-Port NIC

  4. Cable | 线缆

    • Lenovo 100GB QSFP28 to 4x25G SFP28 Breakout DAC Cable

    • 部件号 : 01GV913

    Cable Info

        Identifier                                : 0x11 (QSFP28)
    Extended identifier                       : 0x00
    Extended identifier description           : 1.5W max. Power consumption
    Extended identifier description           : No CDR in TX, No CDR in RX
    Extended identifier description           : High Power Class (> 3.5 W) not enabled
    Power set                                 : Off
    Power override                            : Off
    Connector                                 : 0x23 (No separable connector)
    Transceiver codes                         : 0x88 0x00 0x00 0x00 0x00 0x00 0x00 0x00
    Transceiver type                          : 40G Ethernet: 40G Base-CR4
    Transceiver type                          : 100G Ethernet: 100G Base-CR4 or 25G Base-CR CA-L
    Encoding                                  : 0x05 (64B/66B)
    BR, Nominal                               : 25500Mbps
    Rate identifier                           : 0x00
    Length (SMF,km)                           : 0km
    Length (OM3 50um)                         : 0m
    Length (OM2 50um)                         : 0m
    Length (OM1 62.5um)                       : 0m
    Length (Copper or Active cable)           : 3m
    Transmitter technology                    : 0xa0 (Copper cable unequalized)
    Attenuation at 2.5GHz                     : 7db
    Attenuation at 5.0GHz                     : 10db
    Attenuation at 7.0GHz                     : 12db
    Attenuation at 12.9GHz                    : 18db
    Vendor name                               : LENV-Amphenol
    Vendor OUI                                : a4:8c:db
    Vendor PN                                 : 0000001GV913
    Vendor rev                                : 01
    Vendor SN                                 : 
    Date code                                 : 170803
    Revision Compliance                       : SFF-8636 Rev 2.5/2.6/2.7
    Module temperature                        : 0.00 degrees C / 32.00 degrees F
    Module voltage                            : 0.0000 V

2.Setting | 设置

2.1.Enable RDMA | 启用 RDMA

  1. 安装 rdma-core

    zypper install rdma-core

  2. 查看 /etc/rdma/modules/rdma.conf 文件并选择需要启用的模块

    根据需要增删(我保持默认)

    # These modules are loaded by the system if any RDMA devices is installed
# iSCSI over RDMA client support
ib_iser

# iSCSI over RDMA target support
# ib_isert

# User access to RDMA verbs (supports libibverbs)
ib_uverbs

# User access to RDMA connection management (supports librdmacm)
rdma_ucm

# RDS over RDMA support
# rds_rdma

# NFS over RDMA client support
xprtrdma

# NFS over RDMA server support
svcrdma

    若更改此文件需要 systemctl restart <[email protected]>

  3. 加载 RDMA 驱动

    modprobe qedr

  4. 验证

    • rdma link show

      link qedr0/1 state DOWN physical_state DISABLED netdev enp1s0f0
link qedr1/1 state DOWN physical_state DISABLED netdev enp1s0f1
link qedr2/1 state ACTIVE physical_state LINK_UP netdev enp1s0f2
link qedr3/1 state ACTIVE physical_state LINK_UP netdev enp1s0f3

    • dmesg | grep "qe"

      ......
[   20.490298] [   T1128] qede 0000:01:00.1 enp1s0f1: renamed from eth1
[   20.490446] [   T1175] qede 0000:01:00.0 enp1s0f0: renamed from eth0
[   20.490556] [   T1177] qede 0000:01:00.2 enp1s0f2: renamed from eth2
[   20.491534] [   T1110] qede 0000:01:00.3 enp1s0f3: renamed from eth3
[   23.860135] [      C9] [qede_link_update:2629(enp1s0f3)]Link is up
[   24.317166] [      C8] [qede_link_update:2629(enp1s0f2)]Link is up
[11531.366394] [   T4337] qedr: discovered and registered 4 RDMA funcs

    以下命令需要安装infiniband-diags和libibverbs-utils包

    • ibv_devices

          device                 node GUID
    ------              ----------------
    qedr0               1602ecfffeca05d0
    qedr1               1602ecfffeca05d1
    qedr2               1602ecfffeca05d2
    qedr3               1602ecfffeca05d3

    • ibv_devinfo -d qedr2

      hca_id: qedr2
transport:                      InfiniBand (0)
fw_ver:                         8.59.1.0
node_guid:                      1602:ecff:feca:05d2
sys_image_guid:                 1602:ecff:feca:05d2
vendor_id:                      0x1077
vendor_part_id:                 5718
hw_ver:                         0x1
phys_port_cnt:                  1
        port:   1
                state:                  PORT_ACTIVE (4)
                max_mtu:                4096 (5)
                active_mtu:             1024 (3)
                sm_lid:                 0
                port_lid:               0
                port_lmc:               0x00
                link_layer:             Ethernet

      transport: InfiniBand & link_layer: Ethernet
      即确认其为支持 RoCE 类型的 RDMA 设备

2.2.RoCE configuration | 配置 RoCE

  • 配置 OFED 环境

    安装以下软件包

    • libibverbs
    • perftest

    rdma-core 已附带 libibverbs

2.3.RoCE Verify | 验证 RoCE

  1. 验证已加载模块

    lsmod|grep qedr

    qedr                  155648  0
ib_uverbs             204800  2 rdma_ucm,qedr
ib_core               552960  8 rdma_cm,rpcrdma,iw_cm,ib_iser,rdma_ucm,ib_uverbs,qedr,ib_cm
qede                  233472  1 qedr
qed                  1110016  2 qede,qedr

  2. ibdev2netdev

    需安装 ibdev2netdev 软件包

    使用ibdev2netdev命令查看 RDMA 设备对应的 网络接口

    ibdev2netdev

qedr0/1 gid #1 (down) <===> enp1s0f0 (down)
qedr1/1 gid #1 (down) <===> enp1s0f1 (down)
qedr2/1 gid #1 (active) <===> enp1s0f2 (up)
qedr3/1 gid #1 (active) <===> enp1s0f3 (up)
    • qedr2 对接 SP333 p1
    • qedr3 对接 SP333 p2

  3. 为网络接口添加 IP 地址

    示例使用 nmcli

    • qedr2 即 enp1s0f2

      nmcli connection add type ethernet con-name enp1s0f2 ifname enp1s0f2 ipv4.address 10.7.6.125/24 ipv4.method manual

    • qedr3 即 enp1s0f3

      nmcli connection add type ethernet con-name enp1s0f3 ifname enp1s0f3 ipv4.address 10.7.6.150/24 ipv4.method manual

  4. 验证 3# 操作

    • qedr2 即 enp1s0f2

      nmcli connection show enp1s0f2 | grep "ipv4.addresses"

    • qedr3 即 enp1s0f3

      nmcli connection show enp1s0f3 | grep "ipv4.addresses"

    Ex PIC | 示例图片

  5. ibv_devinfo

    使用 ibv_devinfo 查询设备状态

    ibv_devinfo -d qedr2

    hca_id: qedr2
    transport:                      InfiniBand (0)
    fw_ver:                         8.59.1.0
    node_guid:                      1602:ecff:feca:05d2
    sys_image_guid:                 1602:ecff:feca:05d2
    vendor_id:                      0x1077
    vendor_part_id:                 5718
    hw_ver:                         0x1
    phys_port_cnt:                  1
            port:   1
                    state:                  PORT_ACTIVE (4)
                    max_mtu:                4096 (5)
                    active_mtu:             1024 (3)
                    sm_lid:                 0
                    port_lid:               0
                    port_lmc:               0x00
                    link_layer:             Ethernet

  6. 识别 RoCE v2 GID 索引

    使用 ibv_devinfo -d qedr2 -v|grep GID 命令查看 RoCE v2 GID 索引

    ibv_devinfo -d qedr2 -v|grep GID

    Ex PIC | 示例图片

2.4.RoCE Test | 测试 RoCE

实际上应该和上一标题(RoCE Verify | 验证 RoCE)写在一块的

但 perftest 命令有些多就另起炉灶吧

2.4.1.perftest

使用 perftest 验证(测试) RoCE v2 功能

测试背景

  1. Server: HPE Eth 4x25Gb 1p 620QSFP28 Adptr

    • qedr2 10.7.6.125

    • qedr3 10.7.6.150

  2. Client: Huawei SP333 STL2IEA2A 2-Port NIC

    • mlx5_0 10.7.6.100

    • mlx5_1 10.7.6.200

测试分为两个部分

  1. 首先测试 qedr2 & mlx5_0

  2. 然后同时测试 qedr2 & mlx5_0 & qedr3 & mlx5_1

  1. qedr2 & mlx5_0

    1. ib_send_bw

      • server : ib_send_bw -d qedr2 -F -x 5

      • client : ib_send_bw -d mlx5_0 -F -x 3 10.7.6.125

      Ex PIC | 示例图片

      将 BW average 单位换成 Gb/sec

    2. ib_write_bw

      • server : ib_write_bw -d qedr2 -F -x 5 --report_gbits

      • client : ib_write_bw -d mlx5_0 -F -x 3 10.7.6.125 --report_gbits

      Ex PIC | 示例图片

    3. ibv_rc_pingpong

      • server : ibv_rc_pingpong -d qedr2 -g 5
      • client : ibv_rc_pingpong -d mlx5_0 -g 3 10.7.6.125

      Ex PIC | 示例图片

  2. qedr2 mlx5_0 & qedr3 mlx5_1

    1. ib_send_bw

      • server

        • ib_send_bw -d qedr2 -x 5 --run_infinitely --report_gbits -F
        • ib_send_bw -d qedr3 -x 5 -p 17777 --run_infinitely --report_gbits -F

      • client

        • ib_send_bw -d mlx5_0 -x 3 --run_infinitely --report_gbits -F 10.7.6.125
        • ib_send_bw -d mlx5_1 -x 3 -p 17777 --run_infinitely --report_gbits -F 10.7.6.150

      Ex PIC | 示例图片

    2. ib_write_bw

      • server
        • ib_write_bw -d qedr2 -x 5 --run_infinitely --report_gbits -F
        • ib_write_bw -d qedr3 -x 5 -p 17777 --run_infinitely --report_gbits -F
      • client
        • ib_write_bw -d mlx5_0 -x 3 --run_infinitely --report_gbits -F 10.7.6.125
        • ib_write_bw -d mlx5_1 -x 3 -p 17777 --run_infinitely --report_gbits -F 10.7.6.150

      Ex PIC | 示例图片

2.4.2.iperf3

测试背景如下

  1. Server: HPE Eth 4x25Gb 1p 620QSFP28 Adptr

    • enp1s0f2 10.7.6.125/24

    • enp1s0f3 10.7.5.120/24

  2. Client: Huawei SP333 STL2IEA2A 2-Port NIC

    • p2p1 10.7.6.100/24

    • p2p2 10.7.5.200/24

  3. enp1s0f2 & p2p1

    1. iperf3 -c 10.7.6.125 -B 10.7.6.100 -P 2

      Ex PIC | 示例图片

    2. iperf3 -c 10.7.6.125 -B 10.7.6.100 -P 2 -R

      Ex PIC | 示例图片

  4. enp1s0f3 & p2p2

    1. iperf3 -c 10.7.5.125 -B 10.7.5.100 -P 2

      Ex PIC | 示例图片

    2. iperf3 -c 10.7.5.125 -B 10.7.5.100 -P 2 -R

      Ex PIC | 示例图片

  5. enp1s0f2 p2p1 & enp1s0f3 p2p2

    双端四口 同时进行

    1. iperf3 -c 10.7.x.x -B 10.7.x.x -P 2

      Ex PIC | 示例图片

    2. iperf3 -c 10.7.x.x -B 10.7.x.x -P 2 -R

      Ex PIC | 示例图片

2.4.3.iperf3(bond0)

测试背景如下

  1. Server: HPE Eth 4x25Gb 1p 620QSFP28 Adptr

    • bond0 10.7.6.200/24

  2. Client: Huawei SP333 STL2IEA2A 2-Port NIC

    • bond0 10.7.6.100/24

  1. bond0 & bond0

    1. iperf3 -c 10.7.6.200 -B 10.7.6.100 -P 2

      Ex PIC | 示例图片

    2. iperf3 -c 10.7.6.200 -B 10.7.6.100 -P 2 -R

      Ex PIC | 示例图

Thanks & Copy

Dell EMC 用户指南 智能以太网适配器 QL45212
perftest
GitHub - linux-rdma/perftest: Infiniband Verbs Performance Tests
Infiniband Verbs Performance Tests. Contribute to linux-rdma/perftest development by creating an account on GitHub.
GitHublinux-rdma
HPE Support Center (HPE Ethernet 25Gb 1-port 620QSFP28 Adapter)

标签

xzllll

三分钟热度三分钟收获

向你推荐

本blog与 Hewlett Packard Enterprise (HPE) 和 Marvell Technology 无关。 HPE、Marvell 及其 Logo 是各自公司的商标或注册商标，版权归其所有。 本blog仅用于技术分享，不代表 HPE 或 Marvell 的官方观点或认可。