The Curse of ASPM - Part II: Broadcom 9600-24i

Yet another problem that can be solved by throwing more money at it.

z8

2024-04-24

Just like part one this isn’t supposed to be a tutorial. It’s just a summary of my thoughts and findings for which I felt the need to write them down and preserve them in hopes that others would get some use out of them.

Some additional notes regarding part one

A couple more things of note have happened ever since I wrote that first post. First of all, I received a surprising number of emails and messages about it, so thank you to everyone who reached out :)

Turns out that there’s another downside to forcing L1 on that 9300 series HBA: automatic SMART tests will fail. Once every couple weeks I would get a notification on my TrueNAS dashboard that a particular drive has failed a SMART test. The drive causing issues is a different one every time and any additional manual SMART tests return no errors. I’ve chosen to ignore it but just wanted to point it out in case anyone else also runs into this problem.

Someone on reddit was also only able to replicate my findings when plugging the card into the PCIe slot that’s connected to the chipset.

Moving on…

Remember when I wrote this back in November of last year?

Part of me would like to go out and buy a 9600-24i for upwards of 1300€ after seeing one of the Tri-Mode MegaRAID controllers get to L1 on its own but I really don’t want to spend that sort of cash.

Well turns out that the Chinese have started dumping 9600 series HBAs onto eBay for far less money. Now I’ll let you take a wild guess who went out and bought one of those… I only had to pay 229€ + VAT (please don’t tell me how this is a waste of money, I crossed that line when I bought the 9305-24i).

While I did initially have concerns about the legitimacy of the card, that is a whole different topic deserving of its own blog post. Serials and all that jazz. For the sake of this blog post let’s all just assume the card I have is the real deal.

9600-24i

What initially drew me towards this card, despite the fact that it is a PCIe 4.0 host bus adapter and therefore frankly just overkill for a bunch of spinning rust, was the new driver:

mpi3mr

It was rewritten from scratch and aims to combine the functionality of the mpt3sas and megaraid drivers into one.

StorCLI2

With a new driver also comes a new tool to manage the card. All previous generations of Broadcom/LSI’s HBAs and RAID cards were managed using StorCLI. That won’t work anymore, for these new ones StorCLI2 is what you need.

I can’t believe how hard that tool is to find. Google will not serve you any pages with links to it as of me writing this blog post. Searching Broadcom’s site for ‘storcli2’ gets you decent results, however. I just downloaded the one labeled “[zip] Latest StorCLI2”. When installing it with dpkg it doesn’t automatically end up on my PATH but it’s not exactly hard to find:

/opt/MegaRAID/storcli2/storcli2
# wget https://docs.broadcom.com/docs-and-downloads/008.0008.0000.0010_StorCLI2.zip
# unzip 008.0008.0000.0010_StorCLI2.zip
# dpkg -i Avenger_StorCLI2/Ubuntu/storcli2_008.0008.0000.0010_amd64.deb
# /opt/MegaRAID/storcli2/storcli2 /call show
CLI Version = 008.0008.0000.0010 Jan 08, 2024
Operating system = Linux6.7.7-060707-generic
Controller = 0
Status = Success
Description = None

Product Name = eHBA 9600-24i Tri-Mode Storage Adapter
Board Name = eHBA 9600-24i
Board Assembly = 03-50111-01001
Board Tracer Number = SPC2608286
Board Revision = 01001
Chip Name = SAS4024
Chip Revision = B0
Package Version = 8.1.1.0-00000-00001
Firmware Version = 8.1.1.0-00000-00001
Firmware Security Version Number = 00.00.00.00
NVDATA Version = 01.7E.00.39
Driver Name = mpi3mr
Driver Version = 8.5.0.0.0
SAS Address = 0x500062b20e202bc0
Serial Number = SPC2608286
Controller Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/11 16:36:34
System Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/11 16:36:34
Board Mfg Date(yyyy/mm/dd) = 2022/07/12
Controller Personality = eHBA Personality
Max PCIe Link Rate = 0x08 (16GT/s)
Max PCIe Port Width = 8
PCI Address = 00:01:00:0
PCIe Link Width = X8 Lane(s)
SAS/SATA = SAS/SATA-6G, SAS-12G, SAS-22.5G
PCIe = PCIE-2.5GT, PCIE-5GT, PCIE-8GT, PCIE-16GT
PCI Vendor ID = 0x1000
PCI Device ID = 0x00A5
PCI Subsystem Vendor ID = 0x1000
PCI Subsystem ID = 0x4660
Security Protocol = SPDM-1.1.0,1.0.0
PCI Slot Number = 14

# /opt/MegaRAID/storcli2/storcli2 /c0 show all
CLI Version = 008.0008.0000.0010 Jan 08, 2024
Operating system = Linux6.7.7-060707-generic
Controller = 0
Status = Success
Description = None


Basics :
======
Product Name = eHBA 9600-24i Tri-Mode Storage Adapter
Board Name = eHBA 9600-24i
Board Assembly = 03-50111-01001
Board Tracer Number = SPC2608286
Board Revision = 01001
Chip Name = SAS4024
Chip Revision = B0
Board Mfg Date(yyyy/mm/dd) = 2022/07/12
Board Rework Date(yyyy/mm/dd) = 2018/01/01
SAS Address = 0x500062b20e202bc0
Serial Number = SPC2608286
Controller Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/11 16:41:14
System Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/11 16:41:14
OEM = Broadcom
SubOEMID = 2
PCI Slot Number = 14


Version :
=======
Package Version = 8.1.1.0-00000-00001
Firmware Version = 8.1.1.0-00000-00001
Firmware Security Version Number = 00.00.00.00
FMC Version = 8.1.1.0-00000-00001
FMC Security Version Number = 00.00.00.00
BSP Version = 8.1.1.0-00000-00001
BSP Security Version Number = 00.00.00.00
BIOS Version = 0x08010C00
BIOS Security Version Number = 00.00.00.00
HIIM Version = 08.01.13.00
HIIM Security Version Number = 00.00.00.00
HIIA Version = 08.01.13.00
HIIA Security Version Number = 00.00.00.00
PSOC Hardware Version = 32.64
PSOC Firmware Version = 9.00
PSOC Part Number = 05877-090-aaa
NVDATA Version = 01.7E.00.39
Driver Name = mpi3mr
Driver Version = 8.5.0.0.0
SL8 Library Version = 08.0807.0000


HostInterface :
=============
Max PCIe Link Rate = 0x08 (16GT/s)
Max PCIe Port Width = 8
PCI Address = 00:01:00:0
PCIe Link Width = X8 Lane(s)


DeviceInterface :
===============
SAS/SATA = SAS/SATA-6G, SAS-12G, SAS-22.5G
PCIe = PCIE-2.5GT, PCIE-5GT, PCIE-8GT, PCIE-16GT
PCI Vendor ID = 0x1000
PCI Device ID = 0x00A5
PCI Subsystem Vendor ID = 0x1000
PCI Subsystem ID = 0x4660


Status :
======
Controller Status = Optimal
VD Count = 0
VD Degraded Count = 0
VD Offline Count = 0
PD Count = 0
PD Drive Count = 0
Critical Predictive Failure PD Drive Count = 0
Failed PD Drive Count = 0
Memory Correctable Errors = 0
Memory Uncorrectable Errors = 0
ECC Bucket Count = 0
Preserved Cache Present = No
System Reboot Required = No
System Shutdown Required = No
Controller Reset Required = No
Security Key Assigned = No
Security Type = None
Failed to get Security key on bootup = No
Bios was not detected during boot = No
Boot Time Secret Key Provider = OOB,In-band
EKM Key Provider = OOB,In-band
Controller must be rebooted to complete security operation = No
Controller has booted into safe mode = No
Controller has trial features enabled = No
Personality Change Pending = No
Security Key Rekey Pending = No
Reconfigure types = NVMe reconfigure is not supported
Controller Personality = eHBA Personality


Supported Controller operations :
===============================
Support Enclosure Affinity = No
Support Foreign Config Import = Yes
Support Foreign Config Clear = No
Support Self Diagnostic Check = Yes
Support Diag Retention Test = No
Support Cache Offload Info = No
FW and Event Time in GMT = Yes
Support Abort CC on Error = No
Support Multipath = Yes
Support Security = Yes
Support Security Suggest = Yes
Support Portable Security Key = No
Support Security Key Per PD = No
Support LKM = Yes, Supported using both in-band and out-of-band command
Support EKM = Yes, Supported using both in-band and out-of-band command
Support LKM to EKM = Yes
Support EKM to LKM = No
Support PFK = No
Support Electronic PFK = No
Support Ldbbm = No
Support Shield State = No
Support SSD Ld Disk Cache Change = No
Support Operation Suspend Resume = No
Support Emergency Spares = No
Support Device Mgmt PEL = Yes
Support CC Schedule = No
Support PatrolRead = No
Support SSD PatrolRead = No
Support Resize Array = No
Support Drive Performance Monitoring = No
Support Emulated Drives = Yes
Support Limited Dedicated HotSpares = No
Support Factory Defaults set = Yes
Support Personality = Yes
Support NVCache Info Get = No
Support NVCache Erase = No
Support Cache Offload Encryption = No
Support CacheBypass Modes = No
Support Purge Cache During VD Delete = No
Support BoardLogic Update = Yes
Support Ibuttonless = Yes
Support Dual FW Update = Yes
Support Host Info = Yes
Support Cache-Vault Health Info = No
Support PCIe Devices = Yes
Support Extended MPB VPD pages = Yes
Support Snapdump = Yes
Support Energy Pack = No
Support Energy Pack VPD = No
Support Multiple Security keys(One Per PD) = No
Support Platform Security = Yes
Support SESCtrl In Multipath Config = No
Support R6 Individual PD Suspend/Resume = No
Support SAS Config = Yes
Support limiting of SAS Link speed = Yes, Supports only max link speed limit control of a SAS Phy
Support limiting of SAS Link speed per Phy = No
Support PCIe Config = Yes
Support limiting of PCIe Link Speed = Yes
Support limiting of PCIe Link Speed per Link = Yes
Support PCIe Config Lane Mapping = Yes
Support Immediate Auto-configure = Yes
Support NVMe Recover = Yes
Support NVMe Reconfigure = No
Support PCIe Clock mode = Yes
Support Enhanced Scheduling = No
Support Parallel Drive FW Download = No
Support Replicate IO = No
Support Drive Makers Authority = No
Support Drive FW download management = No


External Key Management :
=======================
Capability = Supported
Boot Agent Status = Not Available
Configured = No


Supported PD operations :
=======================
Support Force Online = No
Support Make Offline = No
Support Make Failed = No
Support SpinDown Unconfigured = No
Support SpinDown HotSpare = No
Support SpinDown Configured = No
Support T10 Power State = Yes
Support Temperature Monitoring = Yes
Support WCE = Yes
Support Degraded Media Detection = No
Support PD Secure Erase = Yes
Support SSD Wear Gauge = No
Support Mark Missing = No
Support Replace Missing = No
Support Prepare For Removal = No
Support Drive Power State Change = No


Supported VD operations :
=======================
Support Changing Read Policy = No
Support Changing Write Policy = No
Support Changing Disk Cache Policy = No
Support Online Capacity Expansion = No
Support LDBBM = No
Support Secure Erase = No
Support Unmap for single drive RAID 0 VDs = No
Support Unmap for RAID 0 VDs = No
Support Unmap for RAID 1/10 VDs = No
Support Unmap for RAID 5/50/6/60 VDs = No
Support Writesame Unmap for single drive RAID 0 VDs = No
Support Writesame Unmap for RAID 0 VDs = No
Support Writesame Unmap for RAID 1/10 VDs = No
Support Writesame Unmap for RAID 5/50/6/60 VDs = No
Support T10 Atomicity = No


HwCfg :
=====
NVRAM = Present
NVRAM Size(KiB) = 128
DDR Memory Size(MiB) = 0
Flash = Present
NOR Flash Size(MiB) = 32
CacheVault Flash Size(MiB) = 0
OCM Memory = Absent
Current Size of FW Cache(MiB) = 0
DDR Width = 40-bit
Energy Pack = Absent
Upgrade Key = Absent
Upgrade Key Slot = Present
On Board Expander = Absent
Temperature Sensor for Chip = Present
Temperature Sensor for Board = Present
Upgradable Board Logic = Present
PCI Switch = Absent
Serial Debugger = Present
Chip temperature(C) = 75
Board temperature(C) = 40
Controller Fan = Absent


Max. Supported Config :
=====================
Max Number of VDs = 0
Max Number of Physical Drives = 144
Max SAS/SATA Drives = 144
Max NVMe Drives = 16
Max Arrays = 0
Max PD Per Array = 0
Max Spans Per VD = 0
Max Dedicated HSPs = 0
Max Global HSPs = 0
Max VDs Per Array = 0
Max Phys = 24
Max Arrays Per DedicatedHSPs = 0
Max JBODs = 144
Max RAID Configurable PDs = 0
Max Complex RAID VDs = 0
Max Data Transfer Size(Bytes) = 1048576
Max Parallel Commands = 8192
Max NS/LU per JBOD = 1
Max NS/LU per RAID PDs = 1
Max Supported LUNs for SAS PDs = 1
Max Namespaces = 1
Max Persistent Id = 1024
Max VD's per Array in configuration = 0
Max Key Id length = 256
Max Security Key length = 32
Max Secret Key length = 32
Max Parallel FW Download Drives = 0
Max Replicate IO Drives = 0
Max Replicate IO Data Transfer Size(KiB) = 0


Properties :
==========
Expose Enclosure Devices = Yes
Maintain Drive Fail History - JBOD = No
All Online Controller Reset = Yes
Auto Online Controller Reset = Yes
LED management for JBOD = Yes
SES VPD Association = TargetPort
Base Enclosure Level = Enclosure level enumeration start with Zero
SMART/Temperature PollInterval-For External PDs only(second(s)) = 300
SMART/Temperature PollInterval-For Internal PDs only(second(s)) = 300
Spinup Drive Count = 2
Spinup Delay(second(s)) = 2
Spinup Encl Drive Count = 4
Spinup Enclosure Delay(second(s)) = 12
Boot Mode = COE
Name =


Capabilities :
============
Supported Drive Interfaces = SAS, SATA, NVMe
Supported RAID Levels = None
Mix of SAS-HDD/SATA-HDD in VD = Not Allowed
Mix of SAS-SSD/SATA-SSD in VD = Not Allowed
Mix of SSD/HDD in VD = Not Allowed
Mix of SED type(Enterprise,OPAL,and RUBY) for Security Enabled Arrays = Not Allowed
Mix of different NS/LU count in a VD = Not Allowed
Mix of NVMe SGL and PRP in a VD = Not Allowed


Secure Boot Details :
===================
Secure Boot State = Enabled
Secure Boot Mode = Hard Secure
Total Number Of Key Slots = 8
Number Of Key Slots Used = 1
Remaining Key Slots = 7
Current Key Encryption Algorithm = Unused
Key Hash Size = SHA256
Key Hash Version = SHA3
Key Hash Algorithm = SHA3-256
Security Version Number = 00.00.00.00


Security Protocol Details :
=========================
Security Protocol = SPDM-1.1.0,1.0.0

Side note: StorCLI2 can only find my controller when this tool is run as root.

What I’m interested in is the “personality”. Documentation on personalities outside of Megaraid cards is very sparse.

The old StorCLI tool had the following setting (taken from ServeTheHome | Broadcom (LSI/Avago) StorCLI Reference/User Guide):

storcli /cx show personality                (MR) Show personality
storcli /cx set personality=RAID|HBA|JBOD
storcli /cx set personality=RAID|HBA|eHBA   (unlisted) Set the personality of the controller to either RAID, HBA or eHBA mode

And then there’s this page on Broadcom’s website:

I went into all of this thinking I could just set the personality to HBA and be done with it.

# /opt/MegaRAID/storcli2/storcli2 /c0 show personality
CLI Version = 008.0008.0000.0010 Jan 08, 2024
Operating system = Linux6.7.7-060707-generic
Controller = 0
Status = Success
Description = None


Personality Information :
=======================

----------------------------------------
Prop                   Description
----------------------------------------
Controller Personality eHBA Personality
----------------------------------------


Available Personality Information :
=================================

-----------------------------------------------------------------------
ID Name             IsCurrent IsRequested IsMutable IsMutableWithForce
-----------------------------------------------------------------------
 0 eHBA Personality Yes       No          Yes       Yes
-----------------------------------------------------------------------

So you’re telling me I have only one personality to choose from? Hmpf.

It doesn’t explicitly tell me that I can’t set it.

# /opt/MegaRAID/storcli2/storcli2 /c0 set personality help

      StorCli2 SAS Customization Utility Ver 008.0008.0000.0010 Jan 08, 2024

    (c)Copyright 2024, Broadcom Inc. , All Rights Reserved.


Controller : 0
NAME: Set personality on controller

SYNTAX: storcli2 /cx set personality id=<value> [force]
        storcli2 /sasx set personality id=<value> [force]

DESCRIPTION: Set the personality of the controller based on the personality ID provided.
OPTIONS:
id - Personality ID to which user wants the personality change to happen.
force - Provided when personality change can happen only with 'force' option.
CONVENTION:
/cx - specifies the controller where X is the controller index
/sasx - specifies the controller where X is the SasAddress of the controller
Note: The process can take up to 5 seconds to complete, do not power off or reset the system until complete.

That’s all fine and good, except these IDs are not documented anywhere.

I figured it’d be faster to try and brute force all possible IDs even though I would probably not get anywhere. A code that exists but is forbidden gets you return code 10, everything else doesn’t.

Have a look at these invalid codes and tell me if you can notice a pattern: 2, 258, 514, 770, 1026, 1282, 1538, 1794, 2050

So… 0 exists and is currently set, 1 is invalid and 2 exists but is forbidden. What a waste of time. Moving on.

How to update

Thank you Richard for making this post. The flag activationtype=offline was crucial in the case of my card.

# wget https://docs.broadcom.com/docs-and-downloads/host-bus-adapters/host-bus-adapters-common-files/sas_sata_nvme_24g_p8.8/9600_24i_Pkg_8.8.1.0-00000-00001.zip
# unzip 9600_24i_Pkg_8.8.1.0-00000-00001.zip
# cd 9600_24i_Pkg_8.8.1.0-00000-00001/Firmware/
# /opt/MegaRAID/storcli2/storcli2 /c0 download file= 9600-24i_full_fw_vsn_pldm_pkg_signed.rom
Downloading image. Please wait...
CLI Version = 008.0008.0000.0010 Jan 08, 2024
Operating system = Linux6.7.7-060707-generic
Controller = 0
Status = Failure
Description = Component Image download failed. Need offline activation to activate the Component Images.
# /opt/MegaRAID/storcli2/storcli2 /c0 download file= 9600-24i_full_fw_vsn_pldm_pkg_signed.rom activationtype=offline
Downloading image. Please wait...
CLI Version = 008.0008.0000.0010 Jan 08, 2024
Operating system = Linux6.7.7-060707-generic
Controller = 0
Status = Success
Description = Component Image download complete. A Complete Reset is required to activate Component Images.


Expected Flash Details Post Activation :
======================================

-------------------------------------------------------------------
ComponentName    ComponentVersion    SecurityVersionNumber Status
-------------------------------------------------------------------
Package Manifest 8.8.1.0-00000-00001 N/A                   Success
FMC              8.8.1.0-00000-00001 00.00.00.00           Success
BSP              8.8.1.0-00000-00001 00.00.00.00           Success
APP              8.8.1.0-00000-00001 00.00.00.00           Success
HIIM             08.08.08.00         00.00.00.00           Success
HIIA             08.08.08.00         00.00.00.00           Success
BIOS             0x08080500          00.00.00.00           Success
-------------------------------------------------------------------

One full reset later:

# /opt/MegaRAID/storcli2/storcli2 /c0 show
CLI Version = 008.0008.0000.0010 Jan 08, 2024
Operating system = Linux6.7.7-060707-generic
Controller = 0
Status = Success
Description = None

Product Name = eHBA 9600-24i Tri-Mode Storage Adapter
Board Name = eHBA 9600-24i
Board Assembly = 03-50111-01001
Board Tracer Number = SPC2608286
Board Revision = 01001
Chip Name = SAS4024
Chip Revision = B0
Package Version = 8.8.1.0-00000-00001
Firmware Version = 8.8.1.0-00000-00001
Firmware Security Version Number = 00.00.00.00
NVDATA Version = 08.0E.00.4B
Driver Name = mpi3mr
Driver Version = 8.5.0.0.0
SAS Address = 0x500062b20e202bc0
Serial Number = SPC2608286
Controller Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/13 19:07:52
System Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/13 19:07:52
Board Mfg Date(yyyy/mm/dd) = 2022/07/12
Controller Personality = eHBA
Max PCIe Link Rate = 0x08 (16GT/s)
Max PCIe Port Width = 8
PCI Address = 00:c1:00:0
PCIe Link Width = X8 Lane(s)
SAS/SATA = SAS/SATA-6G, SAS-12G, SAS-22.5G
PCIe = PCIE-2.5GT, PCIE-5GT, PCIE-8GT, PCIE-16GT
PCI Vendor ID = 0x1000
PCI Device ID = 0x00A5
PCI Subsystem Vendor ID = 0x1000
PCI Subsystem ID = 0x4660
Security Protocol = SPDM-1.1.0,1.0.0
PCI Slot Number = 27

BIOS/UEFI

This HBA can also be configured via the BIOS. Here’s what that looks like:

Really the only setting that I would change is:

Main Menu -> Controller Management -> Advanced Controller Properties -> Write Cache for SATA Devices

Make sure that that is set to Disabled.

TrueNAS

Now, before you go out and buy one, let this be a warning: this driver is still very new. When I first started looking into this card the driver was only working in the nightly releases of TrueNAS 24.10. Support outside of nightly releases only became a thing when 24.04-RC1 dropped.

Now for the big question, does this finally solve the ASPM issue?

Yes.

Here’s what it looks like on my EPYC server:

# cat /etc/version
24.04-RC1#
# uname -a
Linux RAVEN 6.6.16-production+truenas #1 SMP PREEMPT_DYNAMIC Wed Mar 20 19:08:03 UTC 2024 x86_64 GNU/Linux
# lspci -s 41:00.0 -vv
41:00.0 RAID bus controller: Broadcom / LSI Fusion-MPT 24GSAS/PCIe SAS40xx (rev 01)
    Subsystem: Broadcom / LSI eHBA 9600-24i Tri-Mode Storage Adapter
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Region 0: Memory at 3807fc00000 (64-bit, prefetchable) [size=16K]
    Capabilities: [40] Power Management version 3
            Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
            Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] MSI: Enable- Count=1/32 Maskable+ 64bit+
            Address: 0000000000000000  Data: 0000
            Masking: 00000000  Pending: 00000000
    Capabilities: [68] Express (v2) Endpoint, MSI 00
            DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                    ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W
            DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                    RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                    MaxPayload 512 bytes, MaxReadReq 512 bytes
            DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
            LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <32us
                    ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
            LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
                    ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
            LnkSta: Speed 16GT/s, Width x8
                    TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
            DevCap2: Completion Timeout: Range ABCD, TimeoutDis- NROPrPrP- LTR+
                        10BitTagComp+ 10BitTagReq- OBFF Via message, ExtFmt+ EETLPPrefix-
                        EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                        FRS- TPHComp- ExtTPHComp-
                        AtomicOpsCap: 32bit- 64bit- 128bitCAS-
            DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                        AtomicOpsCtl: ReqEn-
            LnkCap2: Supported Link Speeds: 2.5-16GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
            LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis-
                        Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                        Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
            LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                        EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                        Retimer- 2Retimers- CrosslinkRes: Upstream Port
    Capabilities: [a4] MSI-X: Enable+ Count=128 Masked-
            Vector table: BAR=0 offset=00002000
            PBA: BAR=0 offset=00003e00
    Capabilities: [b0] Vital Product Data
            Product Name: Broadcom HBA 9600-24i Tri-Mode Adapter
            Read-only fields:
                    [PN] Part number: 03-50111-01
                    [EC] Engineering changes: 001
                    [SN] Serial number: SPC2608286
                    [V0] Vendor specific: Broadcom Inc.
                    [V1] Vendor specific: SAS4024
                    [V2] Vendor specific: 500062B-220E202BC0
                    [RV] Reserved: checksum good, 0 byte(s) reserved
            End
    Capabilities: [100 v1] Device Serial Number 00-80-5e-01-a7-1b-8f-18
    Capabilities: [fb4 v1] Advanced Error Reporting
            UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
            UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
            UESvrt: DLP+ SDES- TLP- FCP- CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
            CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
            CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
            AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+
                    MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
            HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [138 v1] Power Budgeting <?>
    Capabilities: [db4 v1] Secondary PCI Express
            LnkCtl3: LnkEquIntrruptEn- PerformEqu-
            LaneErrStat: 0
    Capabilities: [af4 v1] Data Link Feature <?>
    Capabilities: [d00 v1] Physical Layer 16.0 GT/s <?>
    Capabilities: [d40 v1] Lane Margining at the Receiver <?>
    Capabilities: [160 v1] Dynamic Power Allocation <?>
    Kernel driver in use: mpi3mr
    Kernel modules: mpi3mr

And here’s powertop showing it hitting C8 on my Z590 machine:

With that said, it only does that when I put the HBA into the bottom slot which is connected to the chipset. I have yet to figure out why, but it’s not like it really matters considering that this card is only being used for HDDs.