HPE Eth 4x25Gb 1p 620QSFP28 Adptr | 上机和初始化记录
1.First | 序
HPE Eth 4x25Gb 1p 620QSFP28 Adptr == QL45214HLCU-HP
HPE 对其的概述为
620-QSFP28 1-port Ethernet adapter - 4 lanes 25GB, quad small form factor (QSFP)Part Number- 840134-001
是一款采用 Qlogic 45604 芯片拥有单 QSFP28 端口和四个 25GB 通道的 NIC
即支持使用 1p QSFP28 100G 拆分出 4p SFP 25G
1.1.Features | 特征
以下内容摘取自 QuickSpecs HPE Ethernet 4x25Gb 1-port 620QSFP28 Adapter
• Industry-leading throughput and latency performance
• Up to 200Gb/s bi-directional near line rate throughput
• Over 15 million small packets/s per port, ideal for web/mobile applications, mobile messaging, and social
media
• Improved small packet performance: Supports UEFI and legacy boot options and Tunnel offload support
(NVGRE, VxLAN)
• Low profile design shipping with standard height and low-profile brackets
• Optimized for virtual server environments with support for Single-Root I/O Virtualization (SR-IOV), I2C
capable
• Jumbo Frames, Checksum & Segmentation Offload, IPv6 and RSS
• On chip temperature monitor (OCSD)
• Standard server operating system support
• Standard NC series option kit warranty, support, services
• Field replaceable and upgradeable
• Integrated PHY and MAC
• Data Plane Development Kit (DPDK)
• RDMA over Conveerged Ethernet (RoCE v2, RoCE v1)
1.2.OS Version | 示例环境
-
B550
已附有操作系统内核中加载的
软件驱动程序qed qede
-
Dell R730XD
补充
620-QSFP28在Dell R730XD BIOS可设置项
初始化过程依然基于
B550芯片组
2.Initialization | 初始化
openSUSE Tumbleweed
已附有操作系统内核中加载的软件驱动程序
2.1.NIC Info | 网卡信息
2.1.1.dmesg | grep "qe"
内核驱动 qed* 的初始化过程如下所示
[ 19.808541] [ T1174] QLogic FastLinQ 4xxxx Core Module qed
[ 19.845712] [ T1162] qede init: QLogic FastLinQ 4xxxx Ethernet Driver qede
[ 19.845765] [ T1162] qede 0000:01:00.0: enabling device (0000 -> 0002)
[ 20.296466] [ T1162] qede 01:00.00: Storm FW 8.59.1.0, Management FW 8.7.19.0 [MBI 8.7.32] [eth0]
[ 20.296502] [ T1162] qede 0000:01:00.1: enabling device (0000 -> 0002)
[ 20.449633] [ T1162] qede 01:00.01: Storm FW 8.59.1.0, Management FW 8.7.19.0 [MBI 8.7.32] [eth1]
[ 20.449850] [ T1162] qede 0000:01:00.2: enabling device (0000 -> 0002)
[ 20.467588] [ T1162] qede 01:00.02: Storm FW 8.59.1.0, Management FW 8.7.19.0 [MBI 8.7.32] [eth2]
[ 20.467816] [ T1162] qede 0000:01:00.3: enabling device (0000 -> 0002)
[ 20.485727] [ T1162] qede 01:00.03: Storm FW 8.59.1.0, Management FW 8.7.19.0 [MBI 8.7.32] [eth3]
[ 20.490298] [ T1128] qede 0000:01:00.1 enp1s0f1: renamed from eth1
[ 20.490446] [ T1175] qede 0000:01:00.0 enp1s0f0: renamed from eth0
[ 20.490556] [ T1177] qede 0000:01:00.2 enp1s0f2: renamed from eth2
[ 20.491534] [ T1110] qede 0000:01:00.3 enp1s0f3: renamed from eth3
[ 23.860135] [ C9] [qede_link_update:2629(enp1s0f3)]Link is up
[ 24.317166] [ C8] [qede_link_update:2629(enp1s0f2)]Link is up
关于内核驱动
qed*补充如下
| Kernel Name | ① | ② |
|---|---|---|
qed |
即核心驱动程序模块 | |
qede |
末尾 e 指代 以太网 |
即 以太网驱动程序 |
qedr |
末尾 r 指代 RDMA |
即基于聚合以太网的 RDMA (RoCE) 驱动程序(与 qed 核心模块和 qede 以太网驱动程序协同工作、 这也就意味着凡是支持RoCE类型的RDMA NIC必将同时支持以太网模式和RDMA模式 |
qedi |
末尾 i 指代 iSCSI |
即 iSCSI 卸载驱动程序。此驱动程序与 Open iSCSI 库一起使用 |
qedf |
末尾 f 指代 FCoE |
即 FCoE 卸载驱动程序,此驱动程序与 Open FCoE 库一起使用 |
2.1.2.ethtool
-
ethtool -i enp1s0f0`
回显如下:
driver: qede version: [storm 8.59.1.0] firmware-version: mbi 8.7.32 [mfw 8.7.19.0] expansion-rom-version: bus-info: 0000:01:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: yes supports-priv-flags: yes详析
回显 直译 driver: qede qede version: [storm 8.59.1.0] 驱动版本 firmware-version: mbi 8.7.32 [mfw 8.7.19.0] 固件版本 -
ethtool enp1s0f0
Settings for enp1s0f2: Supported ports: [ FIBRE ] Supported link modes: 1000baseKX/Full 40000baseCR4/Full 25000baseCR/Full 50000baseCR2/Full 100000baseCR4/Full 10000baseCR/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: 1000baseKX/Full 40000baseCR4/Full 25000baseCR/Full 50000baseCR2/Full 100000baseCR4/Full 10000baseCR/Full Advertised pause frame use: No Advertised auto-negotiation: No Advertised FEC modes: Not reported Link partner advertised link modes: 1000baseKX/Full 40000baseCR4/Full 25000baseCR/Full 50000baseCR2/Full 100000baseCR4/Full 10000baseCR/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: No Link partner advertised FEC modes: Not reported Speed: 25000Mb/s Duplex: Full Auto-negotiation: off Port: Direct Attach Copper PHYAD: 0 Transceiver: internal Supports Wake-on: d Wake-on: d Current message level: 0x00000000 (0) Link detected: yesethtool显示支持全速率 但实际此卡速率被锁定至25G
2.1.3.lspci
lspci -vv -nn -d 1077:
01:00.0 Ethernet controller [0200]: QLogic Corp. FastLinQ QL45000 Series 25GbE Controller [1077:1656] (rev 10)
Subsystem: Hewlett Packard Enterprise Device [1590:00cf]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 75
IOMMU group: 9
Region 0: Memory at e1580000 (64-bit, prefetchable) [size=512K]
Region 2: Memory at e1000000 (64-bit, prefetchable) [size=4M]
Region 4: Memory at e1630000 (64-bit, prefetchable) [size=64K]
Expansion ROM at fcf60000 [disabled] [size=128K]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Product Name: HPE Eth 4x25Gb 1p 620QSFP28 Adptr
Read-only fields:
[PN] Part number: 817760-001
[EC] Engineering changes: A-5640
[MN] Manufacture ID: 1590
[V0] Vendor specific: 20W PCIeGen3x16
[V1] Vendor specific: 4.0.0.9
[V2] Vendor specific: 5652
[V3] Vendor specific: 8.7.19.0
[V5] Vendor specific: 0A
[V6] Vendor specific: 1.0.14
[V7] Vendor specific: 620QSFP28
[SN] Serial number: MY5652088L
[V4] Vendor specific: 1402ECCA05D0
[RV] Reserved: checksum good, 7 byte(s) reserved
End
Capabilities: [58] MSI: Enable- Count=1/32 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable+ Count=33 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [ac] Express (v2) Endpoint, IntMsgNum 0
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W TEE-IO-
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr+ NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s, Exit Latency L0s <1us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x16
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp- 10BitTagReq- OBFF Via WAKE#, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
AtomicOpsCtl: ReqEn+
IDOReq- IDOCompl- LTR- EmergencyPowerReductionReq-
10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout+ AdvNonFatalErr- CorrIntErr- HeaderOF-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
Capabilities: [1b0 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [1b8 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [220 v1] Physical Resizable BAR
BAR 2: current size: 4MB, supported: 8MB
Capabilities: [230 v1] Transaction Processing Hints
Interrupt vector mode supported
Device specific mode supported
Steering table in MSI-X table
Capabilities: [300 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [200 v1] Precision Time Measurement
PTMCap: Requester+ Responder- Root-
PTMClockGranularity: Unimplemented
PTMControl: Enabled- RootSelected-
PTMEffectiveGranularity: Unknown
Kernel driver in use: qede
Kernel modules: qede
3.All Over | 总结
3.1.---
-
自由度稍低 速率被强制锁定在
25GBQL45611H应该是支持满速率 -
强制必须使用
QSFP28 to 4x25GB SFP线缆分出四个25GB -
不支持聚合(相应没有单端口
100GB可选)
3.2.+++
- 小黄鱼购入价格低廉
(>200CNY) RoCE SR-IOV ......等数据中心特性支持- 单适配器双向
200 Gb/s速率
3.3.---+++
配合 ConnectX-4 Lx 产品比如 Huawei SP333 STL2IEA2A 2-Port NIC
内网 25G 轻轻松松
同时也支持Network Adapter Teaming
bond0 40G+ 没问题
性比价颇高
👇
测试内容请移步
HPE Eth 4x25Gb 1p 620QSFP28 Adptr | RDMA(RoCE V2) 配置与测试
1x1.First | 序 书接上回 HPE Eth 4x25Gb 1p 620QSFP28 Adptr | 上机和初始化记录1x1.First | 序 HPE Eth 4x25Gb 1p 620QSFP28 Adptr == QL45214HLCU-HP HPE 对其的概述为 * 620-QSFP28 1-port Ethernet adapter - 4 lanes 25GB, quad small form factor (QSFP) * Part Number * 840134-001 是一款采用 Qlogic 45604 芯片拥有单 QSFP28 端口和四个 25GB 通道的 NIC 即支持使用 1p QSFP28
xzllll
4.Renew
4.1.y/m/d | 25/03/07
若将此 NIC 插入 Lenovo ThinkCentre M720q Tiny
请在 BIOS 中关闭 ASPM 并强制将 PCIE 速率设置为 Gen3
否则会导致 PCIE 降级影响速率
示例
-
关闭前
## lspci -vvv LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s, Exit Latency L0s <1us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM L0s Enabled; RCB 64 bytes, LnkDisable- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s (downgraded), Width x8 (downgraded) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- ## iperf3 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 228 MBytes 1.91 Gbits/sec [ 7] 0.00-1.00 sec 200 MBytes 1.67 Gbits/sec [SUM] 0.00-1.00 sec 428 MBytes 3.59 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 181 MBytes 1.52 Gbits/sec [ 7] 1.00-2.00 sec 162 MBytes 1.36 Gbits/sec [SUM] 1.00-2.00 sec 343 MBytes 2.88 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 187 MBytes 1.57 Gbits/sec [ 7] 2.00-3.00 sec 157 MBytes 1.32 Gbits/sec [SUM] 2.00-3.00 sec 344 MBytes 2.89 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 192 MBytes 1.61 Gbits/sec [ 7] 3.00-4.00 sec 160 MBytes 1.34 Gbits/sec [SUM] 3.00-4.00 sec 352 MBytes 2.95 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 191 MBytes 1.60 Gbits/sec [ 7] 4.00-5.00 sec 154 MBytes 1.29 Gbits/sec [SUM] 4.00-5.00 sec 345 MBytes 2.89 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 189 MBytes 1.58 Gbits/sec [ 7] 5.00-6.00 sec 156 MBytes 1.31 Gbits/sec [SUM] 5.00-6.00 sec 344 MBytes 2.89 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 185 MBytes 1.55 Gbits/sec [ 7] 6.00-7.00 sec 152 MBytes 1.28 Gbits/sec [SUM] 6.00-7.00 sec 337 MBytes 2.83 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 184 MBytes 1.54 Gbits/sec [ 7] 7.00-8.00 sec 156 MBytes 1.31 Gbits/sec [SUM] 7.00-8.00 sec 339 MBytes 2.85 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 184 MBytes 1.54 Gbits/sec [ 7] 8.00-9.00 sec 161 MBytes 1.35 Gbits/sec [SUM] 8.00-9.00 sec 346 MBytes 2.90 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 179 MBytes 1.50 Gbits/sec [ 7] 9.00-10.00 sec 157 MBytes 1.32 Gbits/sec [SUM] 9.00-10.00 sec 336 MBytes 2.82 Gbits/sec -
关闭后
## lspci -vvv LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s, Exit Latency L0s <1us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x8 (downgraded) TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- ## iperf3 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.31 GBytes 11.3 Gbits/sec [ 7] 0.00-1.00 sec 1.38 GBytes 11.9 Gbits/sec [SUM] 0.00-1.00 sec 2.70 GBytes 23.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 1.36 GBytes 11.7 Gbits/sec [ 7] 1.00-2.00 sec 1.37 GBytes 11.8 Gbits/sec [SUM] 1.00-2.00 sec 2.73 GBytes 23.4 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 1.33 GBytes 11.4 Gbits/sec [ 7] 2.00-3.00 sec 1.40 GBytes 12.0 Gbits/sec [SUM] 2.00-3.00 sec 2.73 GBytes 23.4 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 1.34 GBytes 11.5 Gbits/sec [ 7] 3.00-4.00 sec 1.35 GBytes 11.6 Gbits/sec [SUM] 3.00-4.00 sec 2.69 GBytes 23.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 1.33 GBytes 11.4 Gbits/sec [ 7] 4.00-5.00 sec 1.37 GBytes 11.8 Gbits/sec [SUM] 4.00-5.00 sec 2.70 GBytes 23.2 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 1.34 GBytes 11.5 Gbits/sec [ 7] 5.00-6.00 sec 1.39 GBytes 11.9 Gbits/sec [SUM] 5.00-6.00 sec 2.73 GBytes 23.4 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 1.33 GBytes 11.4 Gbits/sec [ 7] 6.00-7.00 sec 1.40 GBytes 12.0 Gbits/sec [SUM] 6.00-7.00 sec 2.73 GBytes 23.5 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 1.36 GBytes 11.7 Gbits/sec [ 7] 7.00-8.00 sec 1.35 GBytes 11.6 Gbits/sec [SUM] 7.00-8.00 sec 2.72 GBytes 23.3 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 1.32 GBytes 11.3 Gbits/sec [ 7] 8.00-9.00 sec 1.40 GBytes 12.0 Gbits/sec [SUM] 8.00-9.00 sec 2.72 GBytes 23.4 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 1.36 GBytes 11.6 Gbits/sec [ 7] 9.00-10.00 sec 1.37 GBytes 11.8 Gbits/sec [SUM] 9.00-10.00 sec 2.73 GBytes 23.4 Gbits/sec
Thanks & Copy
HPE 620QSFP28 4x 25GbE Single QSFP28 Port Ethernet Adapter Review
In our HPE 620QSFP28 review we see what this unique 4x 25GbE single QSFP28 port low profile Ethernet adapter offers and some of the challenges
Rohit Kumar
