View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001382 | channel: elrepo/el9 | kmod-mpt3sas | public | 2023-09-09 12:51 | 2023-09-12 07:00 |
| Reporter | erichweiler | Assigned To | pperry | ||
| Priority | urgent | Severity | major | Reproducibility | always |
| Status | resolved | Resolution | fixed | ||
| Summary | 0001382: mpt3sas elrepo driver doesn't seem to work on RHEL 9.2? | ||||
| Description | Hello! I'm running a RHEL 9.2 server, recently upgraded from 8.6. After the upgrade though, the disks behind my LSI 9201-16e HBA don't show up. # uname -a Linux myserver.mydomain 5.14.0-284.25.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 2 14:53:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux This is my device via lspci: 0e:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02) Subsystem: Broadcom / LSI 9201-16e 6Gb/s SAS/SATA PCIe x8 External HBA # /sbin/lspci -n | grep '0e:00.0' 0e:00.0 0107: 1000:0064 (rev 02) And I see in the "Device IS Pairings" matrix on the elrepo website: pci 1000:0064 kmod-mpt3sas Which jives which what driver I thought would be the one I needed. After installing kmod-mpt3sas.x86_64 (43.100.00.00-1.el9_2.elrepo) and rebooting, I still cannot see the disks. I see this message at boot time: "Warning: Disabled Hardware is detected: mpt3sas:0064:1000 @ 0000: 0e:00.0 is no longer enabled in this release." I don't know if that message is from the elrepo driver or the 9.2 stock kernel. I can confirm the driver is loaded: # lsmod | grep mpt3sas mpt3sas 524288 3 raid_class 16384 1 mpt3sas scsi_transport_sas 49152 1 mpt3sas # find /lib/modules -name mpt3sas.ko /lib/modules/5.14.0-284.25.1.el9_2.x86_64/weak-updates/mpt3sas/mpt3sas.ko /lib/modules/5.14.0-284.11.1.el9_2.x86_64/extra/mpt3sas/mpt3sas.ko # modinfo mpt3sas | grep filename filename: /lib/modules/5.14.0-284.25.1.el9_2.x86_64/weak-updates/mpt3sas/mpt3sas.ko Anyone have any ideas? The mpt3sas elrepo driver has always worked in the past, so this one is strange. Thanks for any help! -erich | ||||
| Steps To Reproduce | Install mpt3sas elrepo driver for RHEL 9.2, disks connected to LSI 9201-16e HBA are not seen after a reboot (or before). | ||||
| Tags | No tags attached. | ||||
|
|
I just tried installing the kernel-ml kernel, and that worked, I can now see the mpt3sas based disks. Just the kmod-mpt3sas driver seems to not work... |
|
|
"Warning: Disabled Hardware is detected" That warning may suggest you are using the driver from the installed kernel. What would happen if you run: sudo modprobe -r mpt3sas sudo modprobe mpt3sas |
|
|
I just tried that but it doesn't like it: # modprobe -r mpt3sas modprobe: FATAL: Module mpt3sas is in use. I even tried that command with the '-f' option and it return with the same error. But it sure doesn't seem like it's in use, as I don't see my disks. :/ # fdisk -l Disk /dev/sda: 272.25 GiB, 292326211584 bytes, 570949632 sectors Disk model: PERC H700 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x000a2838 Device Boot Start End Sectors Size Id Type /dev/sda1 2048 4095 2048 1M 83 Linux /dev/sda2 * 4096 1955839 1951744 953M 83 Linux /dev/sda3 1955840 197267455 195311616 93.1G 83 Linux /dev/sda4 197267456 570949631 373682176 178.2G 5 Extended /dev/sda5 197269504 228517887 31248384 14.9G 82 Linux swap / Solaris /dev/sda6 228519936 570949631 342429696 163.3G 83 Linux That's my boot disk, which is fine, but the other 5 disks aren't showing up. When booting into the mainline kernel it works just fine. |
|
|
I just looked up my boot disk for fun (that Perc H700), and it maps to 0000:0079 which is using kmod-megaraid_sas. So it doesn't use mpt3sas, just my LSI 9201-16e. So no conflict there. Thanks for the help!! |
|
|
The mpt3sas driver in the RHEL kernel is configured to disable certain pci ids. Your device seems to be one of them: /* Meteor ~ 2116 */ { MPI2_MFGPAGE_VENDORID_LSI, MPI2_MFGPAGE_DEVID_SAS2116_1, PCI_ANY_ID, PCI_ANY_ID }, { MPI2_MFGPAGE_VENDORID_LSI, MPI2_MFGPAGE_DEVID_SAS2116_2, PCI_ANY_ID, PCI_ANY_ID }, The message "Warning: Disabled Hardware is detected" indicates that your system is using the in-kernel driver. ELRepo's mpt3sas driver enables all of the disabled pci ids. Question is why it is not used despite the fact the kmod package has been installed on your system. We need to figure this out. |
|
|
Please can you show us the output from the following: lsinitrd -k $(uname -r) | grep -i mpt3sas |
|
|
When the elrepo mainline kernel is loaded (and hence everything is working), I see this: # lsinitrd -k $(uname -r) | grep -i mpt3sas -rw-r--r-- 1 root root 40 Apr 12 22:40 etc/depmod.d/kmod-mpt3sas.conf drwxr-xr-x 2 root root 0 Apr 12 22:40 usr/lib/modules/6.5.2-1.el9.elrepo.x86_64/kernel/drivers/scsi/mpt3sas -rwxr--r-- 1 root root 168208 Apr 12 22:40 usr/lib/modules/6.5.2-1.el9.elrepo.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz When I am running the stock 9.2 kernel with kmod-mpt3sas loaded (and hence I don't see my disks), I see: # lsinitrd -k $(uname -r) | grep -i mpt3sas drwxr-xr-x 2 root root 0 Apr 12 22:40 usr/lib/modules/5.14.0-284.25.1.el9_2.x86_64/kernel/drivers/scsi/mpt3sas -rw-r--r-- 1 root root 178264 Apr 12 22:40 usr/lib/modules/5.14.0-284.25.1.el9_2.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz |
|
|
And just for fun, /etc/depmod.d/kmod-mpt3sas.conf looks like this: override mpt3sas * weak-updates/mpt3sas |
|
|
Quote: When I am running the stock 9.2 kernel with kmod-mpt3sas loaded (and hence I don't see my disks), I see: # lsinitrd -k $(uname -r) | grep -i mpt3sas drwxr-xr-x 2 root root 0 Apr 12 22:40 usr/lib/modules/5.14.0-284.25.1.el9_2.x86_64/kernel/drivers/scsi/mpt3sas -rw-r--r-- 1 root root 178264 Apr 12 22:40 usr/lib/modules/5.14.0-284.25.1.el9_2.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz and that looks to be the issue - the initramfs image contains the distro driver (with your device disabled) rather than the elrepo driver. To fix, please boot to the distro kernel and run the following command: dracut -f /boot/initramfs-$(uname -r).img $(uname -r) and check again with: lsinitrd -k $(uname -r) | grep -i mpt3sas The path should match that you previously reported: # find /lib/modules -name mpt3sas.ko /lib/modules/5.14.0-284.25.1.el9_2.x86_64/weak-updates/mpt3sas/mpt3sas.ko /lib/modules/5.14.0-284.11.1.el9_2.x86_64/extra/mpt3sas/mpt3sas.ko the actual driver is in the '/extra/' path and the '/weak-updates/' is a symlink to compatible kernels. Also note the elrepo driver is not compressed (.ko) whereas the distro driver is compressed (.ko.xz) If the paths in the initramfs image now look correct, reboot and lets see if it now works. |
|
|
OK! I ran the commands you suggested in the stock 9.2 kernel: # dracut -f /boot/initramfs-$(uname -r).img $(uname -r) # lsinitrd -k $(uname -r) | grep -i mpt3sas -rw-r--r-- 1 root root 40 Apr 12 22:40 etc/depmod.d/kmod-mpt3sas.conf drwxr-xr-x 2 root root 0 Apr 12 22:40 usr/lib/modules/5.14.0-284.11.1.el9_2.x86_64/extra/mpt3sas -rw-r--r-- 1 root root 1011689 Apr 12 22:40 usr/lib/modules/5.14.0-284.11.1.el9_2.x86_64/extra/mpt3sas/mpt3sas.ko drwxr-xr-x 2 root root 0 Apr 12 22:40 usr/lib/modules/5.14.0-284.25.1.el9_2.x86_64/kernel/drivers/scsi/mpt3sas -rw-r--r-- 1 root root 178264 Apr 12 22:40 usr/lib/modules/5.14.0-284.25.1.el9_2.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz drwxr-xr-x 2 root root 0 Apr 12 22:40 usr/lib/modules/5.14.0-284.25.1.el9_2.x86_64/weak-updates/mpt3sas lrwxrwxrwx 1 root root 83 Apr 12 22:40 usr/lib/modules/5.14.0-284.25.1.el9_2.x86_64/weak-updates/mpt3sas/mpt3sas.ko -> ../../../../../../lib/modules/5.14.0-284.11.1.el9_2.x86_64/extra/mpt3sas/mpt3sas.ko Then I rebooted, back into the 9.2 stock kernel. And it worked! I can now see my disks with the stock 9.2 kernel. So... I guess there is something up with the way the kmod-mpt3sas package is installed? Maybe it's not running the initrd step correctly during the rpm installation? |
|
|
Great that's fixed it. Yes, we've seen similar issues before in dracut. Unfortunately I do not have a rhel9 system to be able to troubleshoot with, but I'm pretty sure it's not an elrepo bug. You mentioned you had upgraded from rhel8, so it _may_ be related to that - but I'm guessing there. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2023-09-09 12:51 | erichweiler | New Issue | |
| 2023-09-09 12:51 | erichweiler | Status | new => assigned |
| 2023-09-09 12:51 | erichweiler | Assigned To | => pperry |
| 2023-09-09 13:17 | erichweiler | Note Added: 0009333 | |
| 2023-09-09 13:30 | toracat | Status | assigned => acknowledged |
| 2023-09-09 13:35 | toracat | Note Added: 0009334 | |
| 2023-09-09 23:20 | erichweiler | Note Added: 0009335 | |
| 2023-09-09 23:25 | erichweiler | Note Added: 0009336 | |
| 2023-09-10 13:50 | toracat | Note Added: 0009337 | |
| 2023-09-10 18:56 | pperry | Note Added: 0009338 | |
| 2023-09-10 23:23 | erichweiler | Note Added: 0009339 | |
| 2023-09-10 23:40 | erichweiler | Note Added: 0009340 | |
| 2023-09-11 03:18 | pperry | Note Added: 0009341 | |
| 2023-09-11 03:21 | pperry | Note Edited: 0009341 | |
| 2023-09-11 15:46 | erichweiler | Note Added: 0009342 | |
| 2023-09-12 07:00 | pperry | Note Added: 0009343 | |
| 2023-09-12 07:00 | pperry | Status | acknowledged => resolved |
| 2023-09-12 07:00 | pperry | Resolution | open => fixed |
