View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001047 | channel: elrepo/el8 | kmod-isci | public | 2020-10-21 12:55 | 2021-06-02 14:00 |
| Reporter | tgl | Assigned To | toracat | ||
| Priority | normal | Severity | major | Reproducibility | have not tried |
| Status | resolved | Resolution | fixed | ||
| Platform | x86_64 | OS | RHEL | OS Version | 8.2 |
| Summary | 0001047: kmod-isci does not work with latest RHEL8 kernel | ||||
| Description | I've been using kmod-isci-1.2.0-3.el8_2.elrepo successfully with RHEL8 kernels up through 4.18.0-193.19.1.el8_2.x86_64. However, it fails to load into yesterday's release, 4.18.0-193.28.1.el8_2.x86_64 (or at least, that kernel cannot see my SCSI disk drives). Don't know if it just needs to be rebuilt or there's some more-subtle issue. | ||||
| Steps To Reproduce | dnf update reboot | ||||
| Additional Information | With the older kernel, although functionality seems fine I notice disturbing boot messages: [ 3.710116] isci: loading out-of-tree module taints kernel. [ 3.710203] isci: module verification failed: signature and/or required key missing - tainting kernel Maybe the real issue is an out of date signature? | ||||
| Tags | No tags attached. | ||||
|
|
Thanks for the report. We will rebuild the kmod against kernel 4.18.0-193.28.1.el8_2. |
|
|
kmod-isci-1.2.0-4.el8_2.elrepo.x86_64.rpm has been built against kernel-4.18.0-193.28.1.el8_2 and will be syncing to our mirrors shortly. |
|
|
Nope, that's a total fail. I did "yum update", and it downloaded 1.2.0-4.el8_2 fine, and what then happened was that the install scriptlet updated /boot/initramfs* for my back kernel versions but NOT initramfs-4.18.0-193.28.1.el8_2.x86_64.img (as indicated by the file mod times on those files). A bit of testing showed that I now had zero bootable systems. Fortunately I could still boot the rescue kernel, so I was able to restore the old initramfs files from backup, or I'd be dead in the water. So: it will update but not function with 4.18.0-193.14.3 and 4.18.0-193.19.1, but it won't update the 4.18.0-193.28.1 initramfs image. I now speculate that the primary reason it failed yesterday was that the older kmod version didn't install into the initramfs image either. I'm not quite sure how to look into an initramfs image to verify that though. I don't know what methods are used to check which initramfs versions to put a kmod into, but that seems to be the place to look next. |
|
|
Oh, "bootable" might've been a poor choice of words there. The kernel boots alright, but it can't see my SAS disk drive, presumably because the kmod is either not installed or not working. |
|
|
Ah, I found lsinitrd, and I can confirm that there are no isci-related files in the initramfs-4.18.0-193.28.1.el8_2.x86_64.img file that was generated in yesterday's kernel update, while I do see some in the previous (working) initramfs files. Unfortunately, I lacked the presence of mind to save copies of the busted back-branch initramfs files, so I can't say whether kmod-isci 1.2.0-4 actually installed any files into them or whether it did do so but then malfunctioned at load time. |
|
|
Thanks for reporting back with the findings. I've moved the rpms to the testing repo (will happen in the next sync). We will investigate the situation. |
|
|
I am suspecting that this is closely related to issue 0001046. Looking at the contents of my old initramfs files, it seems like there should be "weak-updates" symlinks similar to lrwxrwxrwx 1 root root 49 Feb 28 2020 usr/lib/modules/4.18.0-193.19.1.el8_2.x86_64/weak-updates/isci/isci.ko -> ../../../4.18.0-193.el8.x86_64/extra/isci/isci.ko but no such file is on my disk: $ find /lib/modules -name '*isci*' /lib/modules/4.18.0-193.28.1.el8_2.x86_64/extra/isci /lib/modules/4.18.0-193.28.1.el8_2.x86_64/extra/isci/isci.ko There's also this file in the old initramfs images, which has no equivalent in my filesystem now: -rw-r--r-- 1 root root 232 Dec 14 2019 usr/lib/firmware/isci/isci_firmware.bin |
|
|
Oh, scratch that very last bit, I fat-fingered the search somehow. I *do* have such a file: -rw-r--r-- 1 root root 232 Dec 14 2019 /usr/lib/firmware/isci/isci_firmware.bin (belongs to linux-firmware, not kmod-isci). But it's not getting copied into the new kernel's initramfs. |
|
|
There was a similar situation reported here too: http://lists.elrepo.org/pipermail/elrepo/2020-October/005442.html I would suggest running dracut manually again, and then checking to see if isci.ko has been added to your initramfs image. See also my comments in the above email thread about adding a conf file at /etc/dracut.conf.d/ to specifically add your driver to the initramfs if it's missing (shouldn't need it, but doesn't hurt to manually specify adding it). http://lists.elrepo.org/pipermail/elrepo/2020-October/005445.html |
|
|
OK, this stuff is out of my comfort zone ... but I did sudo dracut -f --add-drivers isci --kver 4.18.0-193.28.1.el8_2.x86_64 and verified that the initramfs now contains some isci files, and rebooted, and it worked! I do not see the "weak-updates" symlink that is in the 19.1 initramfs, but perhaps that's to be expected given this method of building the initramfs. So at this point I can say that kmod-isci-1.2.0-4.el8_2 is functional with this kernel, and the problem is just dracut not wanting to include it by default. In other words, likely the same problem as Gerwin Krist has with a different driver. Much appreciate your help! |
|
|
So after updating to RHEL8.3, I've got the same issue again, except now I can't force inclusion of the module with dracut either: $ sudo dracut -f --add-drivers isci --kver 4.18.0-240.1.1.el8_3.x86_64 dracut-install: Failed to find module 'isci' dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.IsMQvG/initramfs --kerneldir /lib/modules/4.18.0-240.1.1.el8_3.x86_64/ -m isci Indeed, there's no isci module under that /lib/modules tree, which perhaps is not too surprising because the RPM has only installed into one version-specific tree: $ rpm -ql kmod-isci-1.2.0-4.el8_2.elrepo.x86_64 /etc/depmod.d/kmod-isci.conf /lib/modules/4.18.0-193.28.1.el8_2.x86_64 /lib/modules/4.18.0-193.28.1.el8_2.x86_64/extra /lib/modules/4.18.0-193.28.1.el8_2.x86_64/extra/isci /lib/modules/4.18.0-193.28.1.el8_2.x86_64/extra/isci/isci.ko /usr/share/doc/kmod-isci-1.2.0 /usr/share/doc/kmod-isci-1.2.0/GPL-v2.0.txt /usr/share/doc/kmod-isci-1.2.0/greylist.txt I guess I don't understand how any of this worked before last month; previously I'd done several kernel upgrades without incident, but all of a sudden it's become a hair-pulling exercise. |
|
|
For RHEL8.3, you will need to update to the package built for 8.3: kmod-isci-1.2.0-6.el8_3.elrepo.x86_64.rpm It's in the testing repository, so try: yum --enablerepo=elrepo-testing update kmod-isci If the module is compatible with other kernels, it will be weak linked into their /weak-updates/ directory. After updating, please show the output from: find /lib/modules -name isci.ko If it shows an entry for the 4.18.0-240.1.1.el8_3.x86_64 kernel then you should simply be able to reboot to that kernel and things should just work. If not, then you should then be able to re-run your dracut command to force the module into the initramfs if it is not already there, but hopefully that should not be necessary due to the fix we applied in the latest package. |
|
|
Ah, thanks, the testing update version seems to work with this kernel. The process wasn't exactly smooth though; when I did "yum update" I got this: Running transaction Preparing : 1/1 Upgrading : kmod-isci-1.2.0-6.el8_3.elrepo.x86_64 1/2 Running scriptlet: kmod-isci-1.2.0-6.el8_3.elrepo.x86_64 1/2 Running scriptlet: kmod-isci-1.2.0-4.el8_2.elrepo.x86_64 2/2 Cleanup : kmod-isci-1.2.0-4.el8_2.elrepo.x86_64 2/2 Running scriptlet: kmod-isci-1.2.0-4.el8_2.elrepo.x86_64 2/2 dracut-install: Failed to find module 'isci' dracut: FAILED: /usr/lib/dracut/dracut-install -D /var/tmp/dracut.j0dW9u/initramfs --kerneldir /lib/modules/4.18.0-193.28.1.el8_2.x86_64/ -m isci Running scriptlet: kmod-isci-1.2.0-6.el8_3.elrepo.x86_64 2/2 Running scriptlet: kmod-isci-1.2.0-4.el8_2.elrepo.x86_64 2/2 Verifying : kmod-isci-1.2.0-6.el8_3.elrepo.x86_64 1/2 Verifying : kmod-isci-1.2.0-4.el8_2.elrepo.x86_64 2/2 Installed products updated. Upgraded: kmod-isci-1.2.0-6.el8_3.elrepo.x86_64 It appears that after failing to update the 193.28 initramfs file (but nonetheless trashing it; fortunately I had a copy), it abandoned trying to update any other initramfs file, because the one for the 240.1 had not been updated. After manually running dracut, I had a working initramfs file for that version, and was okay. Still, this business of forcibly updating the back-version initramfs files is not nice. The whole point of keeping those is to have a known-bootable system. |
|
|
The issue here is that whenever Red Hat change the kernel ABI of symbols used by the driver, the driver is no longer compatible with that kernel. So when we make the jump from el8.2 to el8.3, a new driver release for el8.3 is required and it is likely not going to be backward compatible with el8.2 (or earlier) kernels. Because the driver is supplied by a kmod package, there is only one version of the driver on your system, and it is sym-linked to each compatible kernel in /weak-updates/. If the driver is not compatible with other kernels installed on the system, the symlink will be missing (no point trying to symlink something we know will not work) and thus dracut will thrown a warning when you try to forcibly include a driver that does not exist (for that kernel). That is the warning/error you saw for kernel-4.18.0-193.28.1.el8_2.x86_64 The moment you updated to kmod-isci-1.2.0-6.el8_3.elrepo.x86_64 (for el8.3) there was no fallback position of a known bootable system using an earlier kernel, because the driver that was bootable on that kernel no longer exists as it just got updated, so those old el8.2 initramfs images are useless at that point (unless you downgrade kmod-isci back to the el8.2 compatible version) There are no easy foolproof solutions here. The easy foolproof solution would be to have the driver in the kernel, but as Red Hat have chosen to discontinue support of the driver in their kernel, we are where we are. Personally I would prefer not be booting RHEL8 on unsupported hardware. I would either seek to run an OS that fully supported my critical hardware or I would put a cheap SSD into those machines as the boot drive and use the SCSI array for storage only. There may well become a point where we are no longer able to provide driver updates for el8 at some future point release if Red Hat make changes to the kernel ABI that we are unable to support through backporting in the driver. If and when that happens, you will have limited options moving forward. Just my opinion. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2020-10-21 12:55 | tgl | New Issue | |
| 2020-10-21 12:55 | tgl | Status | new => assigned |
| 2020-10-21 12:55 | tgl | Assigned To | => toracat |
| 2020-10-21 14:01 | toracat | Note Added: 0007252 | |
| 2020-10-21 14:23 | toracat | Note Added: 0007255 | |
| 2020-10-22 04:23 | toracat | Status | assigned => feedback |
| 2020-10-22 12:12 | tgl | Note Added: 0007260 | |
| 2020-10-22 12:12 | tgl | Status | feedback => assigned |
| 2020-10-22 12:16 | tgl | Note Added: 0007261 | |
| 2020-10-22 12:29 | tgl | Note Added: 0007262 | |
| 2020-10-22 12:44 | toracat | Note Added: 0007263 | |
| 2020-10-22 12:52 | tgl | Note Added: 0007264 | |
| 2020-10-22 12:57 | tgl | Note Added: 0007265 | |
| 2020-10-29 12:26 | pperry | Note Added: 0007272 | |
| 2020-10-29 14:54 | tgl | Note Added: 0007273 | |
| 2020-11-11 11:36 | tgl | Note Added: 0007280 | |
| 2020-11-11 11:50 | pperry | Note Added: 0007281 | |
| 2020-11-11 11:52 | pperry | Note Edited: 0007281 | |
| 2020-11-11 13:30 | tgl | Note Added: 0007282 | |
| 2020-11-11 14:09 | pperry | Note Added: 0007283 | |
| 2021-03-11 13:01 | toracat | Status | assigned => resolved |
| 2021-03-11 13:01 | toracat | Resolution | open => fixed |
| 2021-06-02 14:00 | toracat | Relationship added | related to 0001102 |
