View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000919 | channel: elrepo/el8 | kmod-mpt3sas | public | 2019-06-18 13:55 | 2019-10-29 18:52 |
| Reporter | pperry | Assigned To | pperry | ||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | resolved | Resolution | fixed | ||
| Summary | 0000919: kmod-mpt3sas driver works for kernel-4.18.0-80.el8.x86_64 but fails on newer kernels | ||||
| Description | Users have reported the Driver Update Disk providing an updated kmod-mpt3sas driver works on the initial RHEL8.0 install disk kernel, but that systems fail to reboot to newer kernels after a system update. | ||||
| Additional Information | See this community discussion: https://access.redhat.com/discussions/3722151 | ||||
| Tags | No tags attached. | ||||
|
|
I don't have hardware to test, but working through this to see if we can identify where the issue may be: $ rpm -q kmod-mpt3sas kmod-mpt3sas-0.0-1.el8.elrepo.x86_64 $ uname -a Linux Chroot64R8 4.18.0-80.1.2.el8_0.x86_64 #1 SMP Sun Apr 28 09:21:22 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux $ modinfo mpt3sas | grep filename filename: /lib/modules/4.18.0-80.1.2.el8_0.x86_64/weak-updates/mpt3sas/mpt3sas.ko $ sudo lsinitrd -k $(uname -r) | grep mpt3sas <no output> I would have expected to see the driver in the initramfs image? $ sudo lsinitrd -k 4.18.0-80.el8.x86_64 | grep mpt3sas <no output> But it's not in the base kernel image either, and that boots - must be picking the driver up from /lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas/mpt3sas.ko $ sudo modprobe mpt3sas $ lsmod | grep mpt3sas mpt3sas 286720 0 raid_class 16384 1 mpt3sas scsi_transport_sas 45056 1 mpt3sas Note: Assuming you are running the original kernel (as that's the only kernel that will boot) and you are wanting to update (dracut) the new updated kernel below, you'll want to swap $(uname -r) below with the kernel release, e.g 4.18.0-80.1.2.el8_0.x86_64 With our mpt3sas module loaded, lets try updating the initramfs: $ sudo dracut -f /boot/initramfs-$(uname -r).img $(uname -r) $ sudo lsinitrd -k $(uname -r) | grep mpt3sas drwxr-xr-x 2 root root 0 Jan 15 11:49 usr/lib/modules/4.18.0-80.1.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas -rw-r--r-- 1 root root 109980 Jan 15 11:49 usr/lib/modules/4.18.0-80.1.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz drwxr-xr-x 2 root root 0 Jan 15 11:49 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas -rw-r--r-- 1 root root 597609 Jan 15 11:49 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas/mpt3sas.ko Hmm, so now both the distro driver AND our kmod mpt3sas drivers are in the initramfs. That doesn't look right, but can someone try that and see if that image will boot? If not, lets rename/remove the distro driver, run depmod and try rebuilding the initramfs again $ sudo mv /usr/lib/modules/4.18.0-80.1.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz /usr/lib/modules/4.18.0-80.1.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas-orig.ko.xz $ sudo depmod -a $ sudo dracut -f /boot/initramfs-$(uname -r).img $(uname -r) $ sudo lsinitrd -k $(uname -r) | grep mpt3sas drwxr-xr-x 2 root root 0 Jan 15 11:49 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas -rw-r--r-- 1 root root 597609 Jan 15 11:49 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas/mpt3sas.ko so now we only have to updated kmod driver in the initramfs - does that boot? $ rpm -q kernel kernel-4.18.0-80.el8.x86_64 kernel-4.18.0-80.1.2.el8_0.x86_64 We have 2 kernels installed, so lets try an update to pull in the third kernel as an update and see what drivers end up in the initramfs image for the new kernel. $ sudo yum update <snip> $ rpm -q kernel kernel-4.18.0-80.el8.x86_64 kernel-4.18.0-80.1.2.el8_0.x86_64 kernel-4.18.0-80.4.2.el8_0.x86_64 $ sudo lsinitrd -k 4.18.0-80.4.2.el8_0.x86_64 | grep mpt3sas drwxr-xr-x 2 root root 0 Jan 15 11:49 usr/lib/modules/4.18.0-80.4.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas -rw-r--r-- 1 root root 110088 Jan 15 11:49 usr/lib/modules/4.18.0-80.4.2.el8_0.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz drwxr-xr-x 2 root root 0 Jan 15 11:49 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas -rw-r--r-- 1 root root 597609 Jan 15 11:49 usr/lib/modules/4.18.0-80.el8.x86_64/extra/mpt3sas/mpt3sas.ko So as expected, because the driver is loaded when the kernel was installed, the resultant initramfs contains the mpt3sas driver, but again it contains BOTH versions, so I'll be interested to see which the kernel tries to load. I'm kind of fishing for clues here as to why the updated kernels do not boot. I'd appreciate any clues from users, hopefully the above walk through might be useful on a real system. |
|
|
> This makes me think that the weak links / override mechanisms are broken somewhere. > I suspect it's dracut that is not creating the initramfs correctly. I thought I found the bug in dracut. In SOURCES/dracut-049/install/dracut-install.c I found the following: /* FIXME and add more paths*/ r = asprintf(&path2, "%s/kernel/%s", kerneldir, &argv[i][1]); if (r < 0) { log_error("Out of memory!"); exit(EXIT_FAILURE); } r = asprintf(&path1, "%s/extra/%s", kerneldir, &argv[i][1]); if (r < 0) { log_error("Out of memory!"); exit(EXIT_FAILURE); } r = asprintf(&path3, "%s/updates/%s", kerneldir, &argv[i][1]); if (r < 0) { log_error("Out of memory!"); exit(EXIT_FAILURE); } The paths include kernel/, extra/, and updates/ but is missing weak-updates/. I went ahead and added the missing piece and rebuilt dracut. To my disappointment, the modified dracut did not resolve the issue. By the way, in the EL7 dracut, 'weak-updates' was added by this patch: 0160-dracut-functions.sh-add-lib-modules-extras-weak-upda.patch --- a/dracut-functions.sh +++ b/dracut-functions.sh @@ -1530,7 +1530,7 @@ find_kernel_modules_by_path () { _OLDIFS=$IFS IFS=: while read a rest; do - [[ $a = */$1/* ]] || [[ $a = updates/* ]] || continue + [[ $a = */$1/* ]] || [[ $a = updates/* ]] || [[ $a = extra/* ]] || [[ $a = weak-updates/* ]] ||continue printf "%s\n" "$srcmods/$a" done < "$srcmods/modules.dep" IFS=$_OLDIFS There is no equivalent file in the source code of EL8 dracut. |
|
|
While browsing through the dracut git issues (both open and closed), I saw some error messages in one of them [1] that resemble those posted by the RHEL user who performed the testing [2]: [ TIME ] Timed out waiting for device dev-mapper-rhel\x2dswap.device. [DEPEND] Dependency failed for Resume from hibernation using device /dev/mapper/rhel-swap. ... [130.626999] dracut-initqueue[605]: Warning: dracut-initqueue timeout - starting timeout scripts. The patch is [3]. This patch is in Fedora's dracut [3] but not in RHEL-8. [1] https://github.com/dracutdevs/dracut/issues/480 [2] https://access.redhat.com/discussions/3722151#comment-1554141 [3] https://github.com/dracutdevs/dracut/commit/fca93eac19ee47d8132670dca41c1cb4390e4984 [4] https://bugzilla.redhat.com/show_bug.cgi?id=1641268 We should try building dracut with the patch and see if that resolves the problem. |
|
|
Our interim solution is to patch the in-kernel module and rebuild the kernel for each kernel update. They are found in: http://elrepo.org/people/akemi/kernel/el8/ The current version has, in addition to mpt3sas, patched megaraid_sas ,mptsas, and mptspi modules. [EDIT] The DVD iso images to be used for the installation of RHEL 8.0 are in: https://elrepo.org/linux/dud/el8/x86_64/ |
|
|
This issue was traced to a bug in dracut and has been fixed in dracut: https://github.com/dracutdevs/dracut/pull/614 but has yet to be fixed in RHEL8 (or RHEL8.1 beta). We have backported the patch and built a fixed version of dracut available here: http://elrepo.org/people/akemi/testing/el8/dracut/ We would recommend anyone using kmod packages on el8 use this updated dracut package until Red Hat are able to fix the issue. We would hope the fix will make it into the final release of RHEL 8.1 but will monitor that here. |
|
|
dracut-049-10.git20190115.el8_0.1 released on Oct 29, 2019 has the patch that fixes the issue. From https://access.redhat.com/errata/RHBA-2019:3243 This update fixes the following bug: * dracut is unable to include weak-update modules symbolic link in the initramfs image (BZ#1752888) |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2019-06-18 13:55 | pperry | New Issue | |
| 2019-06-18 13:55 | pperry | Status | new => assigned |
| 2019-06-18 13:55 | pperry | Assigned To | => pperry |
| 2019-06-18 14:46 | pperry | Note Added: 0006356 | |
| 2019-06-18 15:17 | burakkucat | Note Edited: 0006356 | |
| 2019-06-19 15:02 | toracat | Note Edited: 0006356 | |
| 2019-06-19 15:04 | toracat | Note Edited: 0006356 | |
| 2019-06-21 10:21 | toracat | Note Added: 0006359 | |
| 2019-06-21 10:49 | toracat | Note Edited: 0006359 | |
| 2019-06-21 10:49 | toracat | Note View State: 6359: public | |
| 2019-06-23 01:09 | toracat | Note Added: 0006369 | |
| 2019-06-27 10:57 | toracat | Note Added: 0006370 | |
| 2019-06-27 11:00 | toracat | Note Edited: 0006370 | |
| 2019-08-31 02:59 | pperry | Note Added: 0006408 | |
| 2019-10-29 18:52 | toracat | Note Added: 0006625 | |
| 2019-10-29 18:52 | toracat | Status | assigned => resolved |
| 2019-10-29 18:52 | toracat | Resolution | open => fixed |
| 2019-10-29 18:59 | toracat | Note Edited: 0006625 |
