View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001369 | channel: elrepo/el8 | kmod-isci | public | 2023-07-14 17:52 | 2023-07-21 13:28 |
| Reporter | bstillwell | Assigned To | pperry | ||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | resolved | Resolution | no change required | ||
| Summary | 0001369: Rebuild kmod-isci against 4.18.0-477.13.1.el8_8.x86_64 | ||||
| Description | The current kmod-isci package (kmod-isci-1.2.0-12.el8_8.elrepo.x86_64.rpm) is built against kernel 4.18.0-477.10.1.el8_8.x86_64, but the latest kernel is 4.18.0-477.13.1.el8_8.x86_64. | ||||
| Steps To Reproduce | 1. Upgrade to the latest EL8.8 kernel (4.18.0-477.13.1.el8_8.x86_64) 2. Install the latest kmod-isci package (kmod-isci-1.2.0-12.el8_8.elrepo.x86_64.rpm)) 3. Fail to boot the new kernel since the isci kernel module can't be found | ||||
| Tags | No tags attached. | ||||
|
|
Hi, Thank you for reporting the issue. A few questions to help diagnose the potential issue. 1. does the 'old' kernel 4.18.0-477.10.1.el8_8.x86_64 still boot? 2. Please can you show your installed kernels with 'rpm -qa kernel | sort' $ rpm -qa kernel | sort kernel-4.18.0-425.10.1.el8_7.x86_64 kernel-4.18.0-477.10.1.el8_8.x86_64 kernel-4.18.0-477.13.1.el8_8.x86_64 kernel-4.18.0-477.15.1.el8_8.x86_64 3. Now lets look which kernels our isci module is linking against: $ find /lib/modules -name isci /lib/modules/4.18.0-477.10.1.el8_8.x86_64/extra/isci /lib/modules/4.18.0-477.15.1.el8_8.x86_64/weak-updates/isci /lib/modules/4.18.0-477.13.1.el8_8.x86_64/weak-updates/isci As you stated, here we can see our module is built against the 4.18.0-477.10.1.el8_8.x86_64 kernel, but it also weak links against the latest 4.18.0-477.13.1.el8_8.x86_64 and 4.18.0-477.15.1.el8_8.x86_64 kernels. Also note it is not compatible and does not weak link with our earlier el8.7 kernel-4.18.0-425.10.1.el8_7.x86_64 So that all looks as expected. If the system is not booting, then we should next look in the initramfs image to see if our isci driver is present in the image $ sudo lsinitrd -k $(uname -r) | grep -i isci -rw-r--r-- 1 root root 1891 Jun 17 2020 usr/lib/kbd/consolefonts/viscii10-8x16.psfu.gz -rw-r--r-- 1 root root 1418 Jun 17 2020 usr/lib/kbd/consoletrans/viscii1.0_to_tcvn.trans -rw-r--r-- 1 root root 1847 Jun 17 2020 usr/lib/kbd/consoletrans/viscii1.0_to_viscii1.1.trans -rw-r--r-- 1 root root 1811 Jun 17 2020 usr/lib/kbd/unimaps/viscii.uni No, our driver is not there - which in our case is to be expected as _we_ do not have the hardware on this system so the driver is not needed in our initrd image. Here is what the output may look like where the driver is present in the initrd image: $ sudo lsinitrd -k $(uname -r) | grep -i e1000 drwxr-xr-x 2 root root 0 Jan 20 11:30 usr/lib/modules/4.18.0-477.15.1.el8_8.x86_64/kernel/drivers/net/ethernet/intel/e1000 -rw-r--r-- 1 root root 59800 Jan 20 11:30 usr/lib/modules/4.18.0-477.15.1.el8_8.x86_64/kernel/drivers/net/ethernet/intel/e1000/e1000.ko.xz Lets also check the initrd image for the 4.18.0-477.10.1.el8_8.x86_64 base kernel that isci driver was built against: $ sudo lsinitrd -k 4.18.0-477.10.1.el8_8.x86_64 | grep -i isci Is it there for you? Next up, lets try forcing an update of the initramfs image (or substitute 'uname -r' appropriately with the kernel initrd image you wish to update): $ sudo dracut -f /boot/initramfs-$(uname -r).img $(uname -r) and check if the isci driver is now present? Still not present, then lets create a conf file to tell dracut to load the module: Create /etc/dracut.conf.d/dracut-isci.conf containing the line below (noting the spaces around isci inside the double quotes) add_drivers+=" isci " $ cat /etc/dracut.conf.d/dracut-isci.conf add_drivers+=" isci " Now lets update the initrd image again: $ sudo dracut -f /boot/initramfs-$(uname -r).img $(uname -r) and check if our isci driver is now present: $ sudo lsinitrd -k $(uname -r) | grep -i isci drwxr-xr-x 2 root root 0 Jan 20 11:30 usr/lib/firmware/isci -rw-r--r-- 1 root root 232 Jan 20 11:30 usr/lib/firmware/isci/isci_firmware.bin -rw-r--r-- 1 root root 1891 Jun 17 2020 usr/lib/kbd/consolefonts/viscii10-8x16.psfu.gz -rw-r--r-- 1 root root 1418 Jun 17 2020 usr/lib/kbd/consoletrans/viscii1.0_to_tcvn.trans -rw-r--r-- 1 root root 1847 Jun 17 2020 usr/lib/kbd/consoletrans/viscii1.0_to_viscii1.1.trans -rw-r--r-- 1 root root 1811 Jun 17 2020 usr/lib/kbd/unimaps/viscii.uni drwxr-xr-x 2 root root 0 Jan 20 11:30 usr/lib/modules/4.18.0-477.10.1.el8_8.x86_64/extra/isci -rw-r--r-- 1 root root 331721 Jan 20 11:30 usr/lib/modules/4.18.0-477.10.1.el8_8.x86_64/extra/isci/isci.ko drwxr-xr-x 2 root root 0 Jan 20 11:30 usr/lib/modules/4.18.0-477.15.1.el8_8.x86_64/weak-updates/isci lrwxrwxrwx 1 root root 56 Jan 20 11:30 usr/lib/modules/4.18.0-477.15.1.el8_8.x86_64/weak-updates/isci/isci.ko -> ../../../4.18.0-477.10.1.el8_8.x86_64/extra/isci/isci.ko Great - now our isci driver is in the initrd image (and the required firmware blob too!) so our system should now have the necessary drivers to boot. Please can you let me know how you get on with that - thanks :-) |
|
|
Top tip - if you are using kmod drivers (particularly for storage related drivers), keep an old/other kernel on the system that you _know_ will always boot - makes troubleshooting so much easier if you can boot the system and not have to fall back to using rescue media etc. If all else fails, I use our own kernel-lt kernel for such purposes as it is _not_ a distro kernel, and as such a kmod driver will never touch it. |
|
|
First off, thank you for such a detailed response! So what I'm trying to do is upgrade several hundred Cent7 systems which use the C602 to Alma8 using ELevate/Leapp. The problem I'm running into is the phase 1 initrd (initramfs-upgrade.x86_64.img) is getting created without the isci kernel module: # lsinitrd /boot/initramfs-upgrade.x86_64.img | grep -i isci Arguments: -v -v -v -v --force --conf '/dev/null' --confdir '/var/empty' --install 'systemd-nspawn' --add 'sys-upgrade-redhat' --add 'sys-upgrade' --add 'sys-upgrade-redhat' --add 'sys-upgrade' --nomdadmconf --nolvmconf --no-hostonly --kver '4.18.0-477.13.1.el8_8.x86_64' --kernel-image 'vmlinuz-upgrade.x86_64' --add-drivers 'isci' -rw-r--r-- 1 root root 1891 Aug 30 2020 usr/lib/kbd/consolefonts/viscii10-8x16.psfu.gz -rw-r--r-- 1 root root 1418 Aug 30 2020 usr/lib/kbd/consoletrans/viscii1.0_to_tcvn.trans -rw-r--r-- 1 root root 1847 Aug 30 2020 usr/lib/kbd/consoletrans/viscii1.0_to_viscii1.1.trans -rw-r--r-- 1 root root 1811 Aug 30 2020 usr/lib/kbd/unimaps/viscii.uni From what I can tell when kmod-isci is installed the rpm postinstall script isn't doing what it's supposed to do because it was installed into /var/lib/leapp/el8userspace instead of /: # find /var/lib/leapp/el8userspace | grep isci /var/lib/leapp/el8userspace/usr/share/doc/kmod-isci-1.2.0 /var/lib/leapp/el8userspace/usr/share/doc/kmod-isci-1.2.0/GPL-v2.0.txt /var/lib/leapp/el8userspace/usr/share/doc/kmod-isci-1.2.0/greylist.txt /var/lib/leapp/el8userspace/usr/lib/modules/4.18.0-477.10.1.el8_8.x86_64/extra/isci /var/lib/leapp/el8userspace/usr/lib/modules/4.18.0-477.10.1.el8_8.x86_64/extra/isci/isci.ko /var/lib/leapp/el8userspace/usr/lib/kbd/consolefonts/viscii10-8x16.psfu.gz /var/lib/leapp/el8userspace/usr/lib/kbd/unimaps/viscii.uni /var/lib/leapp/el8userspace/usr/lib/kbd/consoletrans/viscii1.0_to_tcvn.trans /var/lib/leapp/el8userspace/usr/lib/kbd/consoletrans/viscii1.0_to_viscii1.1.trans /var/lib/leapp/el8userspace/usr/lib/firmware/isci /var/lib/leapp/el8userspace/usr/lib/firmware/isci/isci_firmware.bin /var/lib/leapp/el8userspace/var/cache/dnf/elrepo-7ae6ad49f622e7e8/packages/kmod-isci-1.2.0-12.el8_8.elrepo.x86_64.rpm /var/lib/leapp/el8userspace/etc/depmod.d/kmod-isci.conf If I manually chroot into /var/lib/leapp/el8userspace and run the postinstall script, it looks better afterwards: # find /var/lib/leapp/el8userspace | grep isci /var/lib/leapp/el8userspace/usr/share/doc/kmod-isci-1.2.0 /var/lib/leapp/el8userspace/usr/share/doc/kmod-isci-1.2.0/GPL-v2.0.txt /var/lib/leapp/el8userspace/usr/share/doc/kmod-isci-1.2.0/greylist.txt /var/lib/leapp/el8userspace/usr/lib/modules/4.18.0-477.10.1.el8_8.x86_64/extra/isci /var/lib/leapp/el8userspace/usr/lib/modules/4.18.0-477.10.1.el8_8.x86_64/extra/isci/isci.ko /var/lib/leapp/el8userspace/usr/lib/modules/4.18.0-477.13.1.el8_8.x86_64/weak-updates/isci /var/lib/leapp/el8userspace/usr/lib/modules/4.18.0-477.13.1.el8_8.x86_64/weak-updates/isci/isci.ko /var/lib/leapp/el8userspace/usr/lib/kbd/consolefonts/viscii10-8x16.psfu.gz /var/lib/leapp/el8userspace/usr/lib/kbd/unimaps/viscii.uni /var/lib/leapp/el8userspace/usr/lib/kbd/consoletrans/viscii1.0_to_tcvn.trans /var/lib/leapp/el8userspace/usr/lib/kbd/consoletrans/viscii1.0_to_viscii1.1.trans /var/lib/leapp/el8userspace/usr/lib/firmware/isci /var/lib/leapp/el8userspace/usr/lib/firmware/isci/isci_firmware.bin /var/lib/leapp/el8userspace/var/cache/dnf/elrepo-7ae6ad49f622e7e8/packages/kmod-isci-1.2.0-12.el8_8.elrepo.x86_64.rpm /var/lib/leapp/el8userspace/etc/depmod.d/kmod-isci.conf Now I just need to figure out how to modify leapp to do this so the initrd gets created properly. |
|
|
I've just posted this to the Alma migration channel. |
|
|
Quoting @bstillwell from Alma chat: I figured out why the weak modules weren't being created. It seems that at the time kmod-isci was being installed the kernel-modules package had not been installed yet, so when it went to create the weak modules it found missing symbols and therefore didn't create the links. |
|
|
More from Alma chat: Andrew Lukoshko "You should either remove 3rd party nodejs before migration or add this 3rd party repo for EL8 to /etc/leapp/files/leapp_upgrade_repositories.repo" Bryan Stillwell "running into a problem now where the final el8 kernel is being built without the isci module. Where can I add the 'kmod-isci' package to the code so it gets installed before the initramfs gets built?" Andrew Lukoshko "For target system you'll need to add kmod-isci to /etc/leapp/transaction/to_install" |
|
|
Closing as 'resolved'. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2023-07-14 17:52 | bstillwell | New Issue | |
| 2023-07-14 17:52 | bstillwell | Status | new => assigned |
| 2023-07-14 17:52 | bstillwell | Assigned To | => toracat |
| 2023-07-14 18:39 | toracat | Status | assigned => acknowledged |
| 2023-07-14 18:40 | toracat | Assigned To | toracat => pperry |
| 2023-07-15 08:48 | pperry | Note Added: 0009282 | |
| 2023-07-15 08:50 | pperry | Relationship added | related to 0001367 |
| 2023-07-15 08:56 | pperry | Note Added: 0009284 | |
| 2023-07-17 16:10 | bstillwell | Note Added: 0009290 | |
| 2023-07-17 16:22 | toracat | Note Added: 0009291 | |
| 2023-07-20 12:15 | toracat | Note Added: 0009293 | |
| 2023-07-21 13:28 | toracat | Note Added: 0009295 | |
| 2023-07-21 13:28 | toracat | Status | acknowledged => resolved |
| 2023-07-21 13:28 | toracat | Resolution | open => no change required |
| 2023-07-21 13:28 | toracat | Note Added: 0009296 |
