Yet another problem that can be solved by throwing more money at it.
2024-04-24
Note: on 2024-07-14 I added an addendum to the bottom of this blog post about the ConnectX-4 I briefly mentioned in part one. Thanks Logan for emailing me about it!
Just like part one this isn’t supposed to be a tutorial. It’s just a summary of my thoughts and findings for which I felt the need to write them down and preserve them in hopes that others would get some use out of them.
A couple more things of note have happened ever since I wrote that first post. First of all, I received a surprising number of emails and messages about it, so thank you to everyone who reached out :)
Turns out that there’s another downside to forcing L1 on that 9300 series HBA: automatic SMART tests will fail. Once every couple weeks I would get a notification on my TrueNAS dashboard that a particular drive has failed a SMART test. The drive causing issues is a different one every time and any additional manual SMART tests return no errors. I’ve chosen to ignore it but just wanted to point it out in case anyone else also runs into this problem.
Someone on reddit was also only able to replicate my findings when plugging the card into the PCIe slot that’s connected to the chipset.
Remember when I wrote this back in November of last year?
Part of me would like to go out and buy a 9600-24i for upwards of 1300€ after seeing one of the Tri-Mode MegaRAID controllers get to L1 on its own but I really don’t want to spend that sort of cash.
Well turns out that the Chinese have started dumping 9600 series HBAs onto eBay for far less money. Now I’ll let you take a wild guess who went out and bought one of those… I only had to pay 229€ + VAT (please don’t tell me how this is a waste of money, I crossed that line when I bought the 9305-24i).
While I did initially have concerns about the legitimacy of the card, that is a whole different topic deserving of its own blog post. Serials and all that jazz. For the sake of this blog post let’s all just assume the card I have is the real deal.
What initially drew me towards this card, despite the fact that it is a PCIe 4.0 host bus adapter and therefore frankly just overkill for a bunch of spinning rust, was the new driver:
It was rewritten from scratch and aims to combine the functionality of the mpt3sas and megaraid drivers into one.
With a new driver also comes a new tool to manage the card. All previous generations of Broadcom/LSI’s HBAs and RAID cards were managed using StorCLI. That won’t work anymore, for these new ones StorCLI2 is what you need.
I can’t believe how hard that tool is to find. Google will not serve you any pages with links to it as of me writing this blog post. Searching Broadcom’s site for ‘storcli2’ gets you decent results, however. I just downloaded the one labeled “[zip] Latest StorCLI2”. When installing it with dpkg it doesn’t automatically end up on my PATH but it’s not exactly hard to find:
/opt/MegaRAID/storcli2/storcli2
# wget https://docs.broadcom.com/docs-and-downloads/008.0008.0000.0010_StorCLI2.zip # unzip 008.0008.0000.0010_StorCLI2.zip # dpkg -i Avenger_StorCLI2/Ubuntu/storcli2_008.0008.0000.0010_amd64.deb # /opt/MegaRAID/storcli2/storcli2 /call show CLI Version = 008.0008.0000.0010 Jan 08, 2024 Operating system = Linux6.7.7-060707-generic Controller = 0 Status = Success Description = None Product Name = eHBA 9600-24i Tri-Mode Storage Adapter Board Name = eHBA 9600-24i Board Assembly = 03-50111-01001 Board Tracer Number = SPC2608286 Board Revision = 01001 Chip Name = SAS4024 Chip Revision = B0 Package Version = 8.1.1.0-00000-00001 Firmware Version = 8.1.1.0-00000-00001 Firmware Security Version Number = 00.00.00.00 NVDATA Version = 01.7E.00.39 Driver Name = mpi3mr Driver Version = 8.5.0.0.0 SAS Address = 0x500062b20e202bc0 Serial Number = SPC2608286 Controller Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/11 16:36:34 System Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/11 16:36:34 Board Mfg Date(yyyy/mm/dd) = 2022/07/12 Controller Personality = eHBA Personality Max PCIe Link Rate = 0x08 (16GT/s) Max PCIe Port Width = 8 PCI Address = 00:01:00:0 PCIe Link Width = X8 Lane(s) SAS/SATA = SAS/SATA-6G, SAS-12G, SAS-22.5G PCIe = PCIE-2.5GT, PCIE-5GT, PCIE-8GT, PCIE-16GT PCI Vendor ID = 0x1000 PCI Device ID = 0x00A5 PCI Subsystem Vendor ID = 0x1000 PCI Subsystem ID = 0x4660 Security Protocol = SPDM-1.1.0,1.0.0 PCI Slot Number = 14 # /opt/MegaRAID/storcli2/storcli2 /c0 show all CLI Version = 008.0008.0000.0010 Jan 08, 2024 Operating system = Linux6.7.7-060707-generic Controller = 0 Status = Success Description = None Basics : ====== Product Name = eHBA 9600-24i Tri-Mode Storage Adapter Board Name = eHBA 9600-24i Board Assembly = 03-50111-01001 Board Tracer Number = SPC2608286 Board Revision = 01001 Chip Name = SAS4024 Chip Revision = B0 Board Mfg Date(yyyy/mm/dd) = 2022/07/12 Board Rework Date(yyyy/mm/dd) = 2018/01/01 SAS Address = 0x500062b20e202bc0 Serial Number = SPC2608286 Controller Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/11 16:41:14 System Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/11 16:41:14 OEM = Broadcom SubOEMID = 2 PCI Slot Number = 14 Version : ======= Package Version = 8.1.1.0-00000-00001 Firmware Version = 8.1.1.0-00000-00001 Firmware Security Version Number = 00.00.00.00 FMC Version = 8.1.1.0-00000-00001 FMC Security Version Number = 00.00.00.00 BSP Version = 8.1.1.0-00000-00001 BSP Security Version Number = 00.00.00.00 BIOS Version = 0x08010C00 BIOS Security Version Number = 00.00.00.00 HIIM Version = 08.01.13.00 HIIM Security Version Number = 00.00.00.00 HIIA Version = 08.01.13.00 HIIA Security Version Number = 00.00.00.00 PSOC Hardware Version = 32.64 PSOC Firmware Version = 9.00 PSOC Part Number = 05877-090-aaa NVDATA Version = 01.7E.00.39 Driver Name = mpi3mr Driver Version = 8.5.0.0.0 SL8 Library Version = 08.0807.0000 HostInterface : ============= Max PCIe Link Rate = 0x08 (16GT/s) Max PCIe Port Width = 8 PCI Address = 00:01:00:0 PCIe Link Width = X8 Lane(s) DeviceInterface : =============== SAS/SATA = SAS/SATA-6G, SAS-12G, SAS-22.5G PCIe = PCIE-2.5GT, PCIE-5GT, PCIE-8GT, PCIE-16GT PCI Vendor ID = 0x1000 PCI Device ID = 0x00A5 PCI Subsystem Vendor ID = 0x1000 PCI Subsystem ID = 0x4660 Status : ====== Controller Status = Optimal VD Count = 0 VD Degraded Count = 0 VD Offline Count = 0 PD Count = 0 PD Drive Count = 0 Critical Predictive Failure PD Drive Count = 0 Failed PD Drive Count = 0 Memory Correctable Errors = 0 Memory Uncorrectable Errors = 0 ECC Bucket Count = 0 Preserved Cache Present = No System Reboot Required = No System Shutdown Required = No Controller Reset Required = No Security Key Assigned = No Security Type = None Failed to get Security key on bootup = No Bios was not detected during boot = No Boot Time Secret Key Provider = OOB,In-band EKM Key Provider = OOB,In-band Controller must be rebooted to complete security operation = No Controller has booted into safe mode = No Controller has trial features enabled = No Personality Change Pending = No Security Key Rekey Pending = No Reconfigure types = NVMe reconfigure is not supported Controller Personality = eHBA Personality Supported Controller operations : =============================== Support Enclosure Affinity = No Support Foreign Config Import = Yes Support Foreign Config Clear = No Support Self Diagnostic Check = Yes Support Diag Retention Test = No Support Cache Offload Info = No FW and Event Time in GMT = Yes Support Abort CC on Error = No Support Multipath = Yes Support Security = Yes Support Security Suggest = Yes Support Portable Security Key = No Support Security Key Per PD = No Support LKM = Yes, Supported using both in-band and out-of-band command Support EKM = Yes, Supported using both in-band and out-of-band command Support LKM to EKM = Yes Support EKM to LKM = No Support PFK = No Support Electronic PFK = No Support Ldbbm = No Support Shield State = No Support SSD Ld Disk Cache Change = No Support Operation Suspend Resume = No Support Emergency Spares = No Support Device Mgmt PEL = Yes Support CC Schedule = No Support PatrolRead = No Support SSD PatrolRead = No Support Resize Array = No Support Drive Performance Monitoring = No Support Emulated Drives = Yes Support Limited Dedicated HotSpares = No Support Factory Defaults set = Yes Support Personality = Yes Support NVCache Info Get = No Support NVCache Erase = No Support Cache Offload Encryption = No Support CacheBypass Modes = No Support Purge Cache During VD Delete = No Support BoardLogic Update = Yes Support Ibuttonless = Yes Support Dual FW Update = Yes Support Host Info = Yes Support Cache-Vault Health Info = No Support PCIe Devices = Yes Support Extended MPB VPD pages = Yes Support Snapdump = Yes Support Energy Pack = No Support Energy Pack VPD = No Support Multiple Security keys(One Per PD) = No Support Platform Security = Yes Support SESCtrl In Multipath Config = No Support R6 Individual PD Suspend/Resume = No Support SAS Config = Yes Support limiting of SAS Link speed = Yes, Supports only max link speed limit control of a SAS Phy Support limiting of SAS Link speed per Phy = No Support PCIe Config = Yes Support limiting of PCIe Link Speed = Yes Support limiting of PCIe Link Speed per Link = Yes Support PCIe Config Lane Mapping = Yes Support Immediate Auto-configure = Yes Support NVMe Recover = Yes Support NVMe Reconfigure = No Support PCIe Clock mode = Yes Support Enhanced Scheduling = No Support Parallel Drive FW Download = No Support Replicate IO = No Support Drive Makers Authority = No Support Drive FW download management = No External Key Management : ======================= Capability = Supported Boot Agent Status = Not Available Configured = No Supported PD operations : ======================= Support Force Online = No Support Make Offline = No Support Make Failed = No Support SpinDown Unconfigured = No Support SpinDown HotSpare = No Support SpinDown Configured = No Support T10 Power State = Yes Support Temperature Monitoring = Yes Support WCE = Yes Support Degraded Media Detection = No Support PD Secure Erase = Yes Support SSD Wear Gauge = No Support Mark Missing = No Support Replace Missing = No Support Prepare For Removal = No Support Drive Power State Change = No Supported VD operations : ======================= Support Changing Read Policy = No Support Changing Write Policy = No Support Changing Disk Cache Policy = No Support Online Capacity Expansion = No Support LDBBM = No Support Secure Erase = No Support Unmap for single drive RAID 0 VDs = No Support Unmap for RAID 0 VDs = No Support Unmap for RAID 1/10 VDs = No Support Unmap for RAID 5/50/6/60 VDs = No Support Writesame Unmap for single drive RAID 0 VDs = No Support Writesame Unmap for RAID 0 VDs = No Support Writesame Unmap for RAID 1/10 VDs = No Support Writesame Unmap for RAID 5/50/6/60 VDs = No Support T10 Atomicity = No HwCfg : ===== NVRAM = Present NVRAM Size(KiB) = 128 DDR Memory Size(MiB) = 0 Flash = Present NOR Flash Size(MiB) = 32 CacheVault Flash Size(MiB) = 0 OCM Memory = Absent Current Size of FW Cache(MiB) = 0 DDR Width = 40-bit Energy Pack = Absent Upgrade Key = Absent Upgrade Key Slot = Present On Board Expander = Absent Temperature Sensor for Chip = Present Temperature Sensor for Board = Present Upgradable Board Logic = Present PCI Switch = Absent Serial Debugger = Present Chip temperature(C) = 75 Board temperature(C) = 40 Controller Fan = Absent Max. Supported Config : ===================== Max Number of VDs = 0 Max Number of Physical Drives = 144 Max SAS/SATA Drives = 144 Max NVMe Drives = 16 Max Arrays = 0 Max PD Per Array = 0 Max Spans Per VD = 0 Max Dedicated HSPs = 0 Max Global HSPs = 0 Max VDs Per Array = 0 Max Phys = 24 Max Arrays Per DedicatedHSPs = 0 Max JBODs = 144 Max RAID Configurable PDs = 0 Max Complex RAID VDs = 0 Max Data Transfer Size(Bytes) = 1048576 Max Parallel Commands = 8192 Max NS/LU per JBOD = 1 Max NS/LU per RAID PDs = 1 Max Supported LUNs for SAS PDs = 1 Max Namespaces = 1 Max Persistent Id = 1024 Max VD's per Array in configuration = 0 Max Key Id length = 256 Max Security Key length = 32 Max Secret Key length = 32 Max Parallel FW Download Drives = 0 Max Replicate IO Drives = 0 Max Replicate IO Data Transfer Size(KiB) = 0 Properties : ========== Expose Enclosure Devices = Yes Maintain Drive Fail History - JBOD = No All Online Controller Reset = Yes Auto Online Controller Reset = Yes LED management for JBOD = Yes SES VPD Association = TargetPort Base Enclosure Level = Enclosure level enumeration start with Zero SMART/Temperature PollInterval-For External PDs only(second(s)) = 300 SMART/Temperature PollInterval-For Internal PDs only(second(s)) = 300 Spinup Drive Count = 2 Spinup Delay(second(s)) = 2 Spinup Encl Drive Count = 4 Spinup Enclosure Delay(second(s)) = 12 Boot Mode = COE Name = Capabilities : ============ Supported Drive Interfaces = SAS, SATA, NVMe Supported RAID Levels = None Mix of SAS-HDD/SATA-HDD in VD = Not Allowed Mix of SAS-SSD/SATA-SSD in VD = Not Allowed Mix of SSD/HDD in VD = Not Allowed Mix of SED type(Enterprise,OPAL,and RUBY) for Security Enabled Arrays = Not Allowed Mix of different NS/LU count in a VD = Not Allowed Mix of NVMe SGL and PRP in a VD = Not Allowed Secure Boot Details : =================== Secure Boot State = Enabled Secure Boot Mode = Hard Secure Total Number Of Key Slots = 8 Number Of Key Slots Used = 1 Remaining Key Slots = 7 Current Key Encryption Algorithm = Unused Key Hash Size = SHA256 Key Hash Version = SHA3 Key Hash Algorithm = SHA3-256 Security Version Number = 00.00.00.00 Security Protocol Details : ========================= Security Protocol = SPDM-1.1.0,1.0.0
Side note: StorCLI2 can only find my controller when this tool is run as root.
What I’m interested in is the “personality”. Documentation on personalities outside of Megaraid cards is very sparse.
The old StorCLI tool had the following setting (taken from ServeTheHome | Broadcom (LSI/Avago) StorCLI Reference/User Guide):
storcli /cx show personality (MR) Show personality storcli /cx set personality=RAID|HBA|JBOD storcli /cx set personality=RAID|HBA|eHBA (unlisted) Set the personality of the controller to either RAID, HBA or eHBA mode
And then there’s this page on Broadcom’s website:
I went into all of this thinking I could just set the personality to HBA and be done with it.
# /opt/MegaRAID/storcli2/storcli2 /c0 show personality CLI Version = 008.0008.0000.0010 Jan 08, 2024 Operating system = Linux6.7.7-060707-generic Controller = 0 Status = Success Description = None Personality Information : ======================= ---------------------------------------- Prop Description ---------------------------------------- Controller Personality eHBA Personality ---------------------------------------- Available Personality Information : ================================= ----------------------------------------------------------------------- ID Name IsCurrent IsRequested IsMutable IsMutableWithForce ----------------------------------------------------------------------- 0 eHBA Personality Yes No Yes Yes -----------------------------------------------------------------------
So you’re telling me I have only one personality to choose from? Hmpf.
It doesn’t explicitly tell me that I can’t set it.
# /opt/MegaRAID/storcli2/storcli2 /c0 set personality help
StorCli2 SAS Customization Utility Ver 008.0008.0000.0010 Jan 08, 2024
(c)Copyright 2024, Broadcom Inc. , All Rights Reserved.
Controller : 0
NAME: Set personality on controller
SYNTAX: storcli2 /cx set personality id=<value> [force]
storcli2 /sasx set personality id=<value> [force]
DESCRIPTION: Set the personality of the controller based on the personality ID provided.
OPTIONS:
id - Personality ID to which user wants the personality change to happen.
force - Provided when personality change can happen only with 'force' option.
CONVENTION:
/cx - specifies the controller where X is the controller index
/sasx - specifies the controller where X is the SasAddress of the controller
Note: The process can take up to 5 seconds to complete, do not power off or reset the system until complete.
That’s all fine and good, except these IDs are not documented anywhere.
I figured it’d be faster to try and brute force all possible IDs even though I would probably not get anywhere. A code that exists but is forbidden gets you return code 10, everything else doesn’t.
Have a look at these invalid codes and tell me if you can notice a pattern: 2, 258, 514, 770, 1026, 1282, 1538, 1794, 2050
So… 0 exists and is currently set, 1 is invalid and 2 exists but is forbidden. What a waste of time. Moving on.
Thank you Richard for making this post. The flag activationtype=offline was crucial in the case of my card.
# wget https://docs.broadcom.com/docs-and-downloads/host-bus-adapters/host-bus-adapters-common-files/sas_sata_nvme_24g_p8.8/9600_24i_Pkg_8.8.1.0-00000-00001.zip # unzip 9600_24i_Pkg_8.8.1.0-00000-00001.zip # cd 9600_24i_Pkg_8.8.1.0-00000-00001/Firmware/ # /opt/MegaRAID/storcli2/storcli2 /c0 download file= 9600-24i_full_fw_vsn_pldm_pkg_signed.rom Downloading image. Please wait... CLI Version = 008.0008.0000.0010 Jan 08, 2024 Operating system = Linux6.7.7-060707-generic Controller = 0 Status = Failure Description = Component Image download failed. Need offline activation to activate the Component Images.
# /opt/MegaRAID/storcli2/storcli2 /c0 download file= 9600-24i_full_fw_vsn_pldm_pkg_signed.rom activationtype=offline Downloading image. Please wait... CLI Version = 008.0008.0000.0010 Jan 08, 2024 Operating system = Linux6.7.7-060707-generic Controller = 0 Status = Success Description = Component Image download complete. A Complete Reset is required to activate Component Images. Expected Flash Details Post Activation : ====================================== ------------------------------------------------------------------- ComponentName ComponentVersion SecurityVersionNumber Status ------------------------------------------------------------------- Package Manifest 8.8.1.0-00000-00001 N/A Success FMC 8.8.1.0-00000-00001 00.00.00.00 Success BSP 8.8.1.0-00000-00001 00.00.00.00 Success APP 8.8.1.0-00000-00001 00.00.00.00 Success HIIM 08.08.08.00 00.00.00.00 Success HIIA 08.08.08.00 00.00.00.00 Success BIOS 0x08080500 00.00.00.00 Success -------------------------------------------------------------------
One full reset later:
# /opt/MegaRAID/storcli2/storcli2 /c0 show CLI Version = 008.0008.0000.0010 Jan 08, 2024 Operating system = Linux6.7.7-060707-generic Controller = 0 Status = Success Description = None Product Name = eHBA 9600-24i Tri-Mode Storage Adapter Board Name = eHBA 9600-24i Board Assembly = 03-50111-01001 Board Tracer Number = SPC2608286 Board Revision = 01001 Chip Name = SAS4024 Chip Revision = B0 Package Version = 8.8.1.0-00000-00001 Firmware Version = 8.8.1.0-00000-00001 Firmware Security Version Number = 00.00.00.00 NVDATA Version = 08.0E.00.4B Driver Name = mpi3mr Driver Version = 8.5.0.0.0 SAS Address = 0x500062b20e202bc0 Serial Number = SPC2608286 Controller Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/13 19:07:52 System Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/13 19:07:52 Board Mfg Date(yyyy/mm/dd) = 2022/07/12 Controller Personality = eHBA Max PCIe Link Rate = 0x08 (16GT/s) Max PCIe Port Width = 8 PCI Address = 00:c1:00:0 PCIe Link Width = X8 Lane(s) SAS/SATA = SAS/SATA-6G, SAS-12G, SAS-22.5G PCIe = PCIE-2.5GT, PCIE-5GT, PCIE-8GT, PCIE-16GT PCI Vendor ID = 0x1000 PCI Device ID = 0x00A5 PCI Subsystem Vendor ID = 0x1000 PCI Subsystem ID = 0x4660 Security Protocol = SPDM-1.1.0,1.0.0 PCI Slot Number = 27
This HBA can also be configured via the BIOS. Here’s what that looks like:
Really the only setting that I would change is:
Main Menu -> Controller Management -> Advanced Controller Properties -> Write Cache for SATA Devices
Make sure that that is set to Disabled.
Now, before you go out and buy one, let this be a warning: this driver is still very new. When I first started looking into this card the driver was only working in the nightly releases of TrueNAS 24.10. Support outside of nightly releases only became a thing when 24.04-RC1 dropped.
Now for the big question, does this finally solve the ASPM issue?
Yes.
Here’s what it looks like on my EPYC server:
# cat /etc/version
24.04-RC1#
# uname -a
Linux RAVEN 6.6.16-production+truenas #1 SMP PREEMPT_DYNAMIC Wed Mar 20 19:08:03 UTC 2024 x86_64 GNU/Linux
# lspci -s 41:00.0 -vv
41:00.0 RAID bus controller: Broadcom / LSI Fusion-MPT 24GSAS/PCIe SAS40xx (rev 01)
Subsystem: Broadcom / LSI eHBA 9600-24i Tri-Mode Storage Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Region 0: Memory at 3807fc00000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable- Count=1/32 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [68] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <32us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM L1 Enabled ; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 16GT/s, Width x8
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis- NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq- OBFF Via message, ExtFmt+ EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: Upstream Port
Capabilities: [a4] MSI-X: Enable+ Count=128 Masked-
Vector table: BAR=0 offset=00002000
PBA: BAR=0 offset=00003e00
Capabilities: [b0] Vital Product Data
Product Name: Broadcom HBA 9600-24i Tri-Mode Adapter
Read-only fields:
[PN] Part number: 03-50111-01
[EC] Engineering changes: 001
[SN] Serial number: SPC2608286
[V0] Vendor specific: Broadcom Inc.
[V1] Vendor specific: SAS4024
[V2] Vendor specific: 500062B-220E202BC0
[RV] Reserved: checksum good, 0 byte(s) reserved
End
Capabilities: [100 v1] Device Serial Number 00-80-5e-01-a7-1b-8f-18
Capabilities: [fb4 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP- FCP- CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [138 v1] Power Budgeting <?>
Capabilities: [db4 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [af4 v1] Data Link Feature <?>
Capabilities: [d00 v1] Physical Layer 16.0 GT/s <?>
Capabilities: [d40 v1] Lane Margining at the Receiver <?>
Capabilities: [160 v1] Dynamic Power Allocation <?>
Kernel driver in use: mpi3mr
Kernel modules: mpi3mr
And here’s powertop showing it hitting C8 on my Z590 machine:
With that said, it only does that when I put the HBA into the bottom slot which is connected to the chipset. I have yet to figure out why, but it’s not like it really matters considering that this card is only being used for HDDs.
After I had already completely forgotten about my ConnectX-4 (I had sent it back after all), it showed up on my doorstep again a few months later. The seller didn’t respond to any of my messages and I had already received the full refund ages ago, so I just kept it. I figured this was some higher power’s way of telling me to give it another shot.
Turns out that the lack of ASPM was indeed fixable. The card I had bought was a Huawei-branded MCX4121A-ACA. With this being an OEM card, it is no surprise to anyone that it doesn’t ship with a regular stock firmware. The PSID clearly indicates that this is a Huawei product. As a result of that, all of the Mellanox tooling doesn’t want anything to do with it and can’t pull firmware updates for it.
root@z8-epyc:/home/z8# mlxfwmanager Querying Mellanox devices firmware ... Device #1: ---------- Device Type: ConnectX4LX Part Number: Huawei_MCX4121A-ACA_Ax Description: ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6 PSID:HUA0000000001 PCI Device Name: /dev/mst/mt4117_pciconf0 Base MAC: 98039b084774 Versions: Current Available FW 14.22.1002N/A PXE 3.5.0403N/A UEFI 14.15.0019N/A Status:No matching image found
That can be fixed with a bit of force. Forcefully overriding the PSID with one from a known good firmware image does the trick. It is recommended that you save as many of the original firmware bits and pieces as possible, just in case. Plenty of resources out there showing you how to do that though, so I’m not gonna explain it here.
Oh and all of this also enables ASPM, probably because the old firmware is ancient.
root@z8-epyc:/home/z8/Desktop# wget https://www.mellanox.com/downloads/firmware/fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip --2024-03-02 18:36:51-- https://www.mellanox.com/downloads/firmware/fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip Resolving www.mellanox.com (www.mellanox.com)... 2.21.133.59, 2.21.133.51 Connecting to www.mellanox.com (www.mellanox.com)|2.21.133.59|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://content.mellanox.com/firmware/fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip [following] --2024-03-02 18:36:51-- https://content.mellanox.com/firmware/fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip Resolving content.mellanox.com (content.mellanox.com)... 107.178.241.102 Connecting to content.mellanox.com (content.mellanox.com)|107.178.241.102|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2534616 (2.4M) [application/zip] Saving to: ‘fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip’ fw-ConnectX4Lx-rel-14_32_1010- 100%[====================================================>] 2.42M 10.1MB/s in 0.2s 2024-03-02 18:36:51 (10.1 MB/s) - ‘fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip’ saved [2534616/2534616] root@z8-epyc:/home/z8/Desktop# unzip fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip Archive: fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin.zip root@z8-epyc:/home/z8/Desktop# flint -d /dev/mst/mt4117_pciconf0 query full > flint_query.txt root@z8-epyc:/home/z8/Desktop# flint -d /dev/mst/mt4117_pciconf0 hw query > flint_hwinfo.txt root@z8-epyc:/home/z8/Desktop# flint -d /dev/mst/mt4117_pciconf0 ri orig_firmware.bin root@z8-epyc:/home/z8/Desktop# nano flint_hwinfo.txt root@z8-epyc:/home/z8/Desktop# nano flint_query.txt root@z8-epyc:/home/z8/Desktop# flint -d /dev/mst/mt4117_pciconf0 dc orig_firmware.ini root@z8-epyc:/home/z8/Desktop# flint -d /dev/mst/mt4117_pciconf0 rrom orig_rom.bin root@z8-epyc:/home/z8/Desktop# flint -d /dev/mst/mt4117_pciconf0 -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin-allow_psid_change burn Done. Current FW version on flash: 14.22.1002 New FW version: 14.32.1010 You are about to replace current PSID on flash - "HUA0000000001" with a different PSID - "MT_2420110034". Note: It is highly recommended not to change the PSID. Do you want to continue ? (y/n) [n] : y Burning FW image without signatures - OK Restoring signature - OK -I- To load new FW run mlxfwreset or reboot machine. root@z8-epyc:/home/z8/Desktop# mlxfwreset -d /dev/mst/mt4117_pciconf0 reset Minimal reset level for device, /dev/mst/mt4117_pciconf0: 3: Driver restart and PCI reset Continue with reset?[y/N] y -I- Sending Reset Command To Fw -Done -I- Stopping Driver -Done -I- Resetting PCI -Done -I- Starting Driver -Done -I- Restarting MST -Done -I- FW was loaded successfully. root@z8-epyc:/home/z8/Desktop# mlxfwmanager Querying Mellanox devices firmware ... Device #1: ---------- Device Type: ConnectX4LX Part Number: MCX4121A-ACA_Ax Description: ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6 PSID: MT_2420110034 PCI Device Name: /dev/mst/mt4117_pciconf0 Base MAC: 98039b084774 Versions: Current Available FW 14.32.1010 14.32.1010 PXE 3.6.0502 3.6.0502 UEFI 14.25.0017 14.25.0017 Status: Up to date