|
|
Not sure it is relevant, but the "dracut-initqueue timeout" caught my eye, given that this is likely an el 7.6 system.
There was an EL 7.5 > 7.6 upgrade issue with device-manager\* and lvm2\* thought to be related to mdraid metadata 0.90 that hit me and, on a stock system, it would give lots of "dracut-initqueue timeout" errors and then fail with missing LV errors. I see md127 in your hardware output.
Using kernel-4.14 (don't ask) the same system would boot, but would show really screwed up LVM stats. Basically, it was booting using mdraid members instead of the mdraids themselves as the PVs. Running pvscan gave strange results (missing UUID members, disk partition used as PV, etc.).
The simple diagnostic is to run pvscan on the kernel-ml system and look for strange output. The fix would be to downgrade device-manager\* and lvm2\* to the el 7.5 versions and to exclude= the 7.6 updates in yum.conf (and there were many releases of them).
May be completely unrelated to your issues. |
|
|
|
From your report, I understand it to be a new hardware configuration that has never been used with any version of kernel-lt earlier than the two you have mentioned.
I do not suspect a "missing item" in the kernel-lt configuration and so suspect that if I were to offer you, say, kernel-lt-4.4.150-1.el7.elrepo (from the archive) the end result would still be a failure to boot.
Please try the test documented in paragraph four of note 6065 . . . . for I currently have no other idea what could be causing the problem. |
|
|
|
Yes, this was updated from 7.5 to 7.6. However, the md127 did not have a filesystem or label on it during any of the boot attempts for any of the kernels. The md127 is the NVME testing I'm doing at this time. That grouping of /dev/nvme* and md* have been rebuilt/destroyed many times (wipefs, vgremove etc).
Any idea why the kernel-ml is not identically affected?
Currently running a long duration Phoronix test. It may be a while before I can test according to note paragraph four of note 6065. |
|
|
|
The system will eventually drop me to a dracut:/# prompt after telling me that my "slash" filesystem and swap are not found. Even more interesting is that:
sh: pvscan: command not found
when run from the dracut:/# prompt.
I am remote to that system and will see about getting rdsosreport.txt mentioned. |
|
|
|
Uploaded an initial image of the virtual console with issue.
"lvm_scan" when run from the dracut prompt produces no results.
Also, when at the dracut prompt, there are no /dev/sd* device files. I'd expect to see at least the sdaX for the standard partition for /boot from the PERC H740P RAID card.?? |
|
|
|
When I ran into the 7.5 > 7.6 dm/lvm problem, I discovered on some systems that the dm and lvm dracut modules were not installed in the initramfs image. I think that would trigger your symptoms in posts 0006069 and 0006070.
You can check the image currently in use by:
# lsinitrd -k $(uname -r)|less
(or ...-k <kver>... for other images)
The installed dracut modules are in a list at the start of the output.
You can force the inclusion during manual dracut runs (dracut --add "dm lvm"...) or, as I did, just modify dracut.conf to add the modules.
# grep ^add_dracutmodules /etc/dracut.conf
add_dracutmodules+="dm lvm"
Hope that helps. |
|
|
|
FYI: blkid when run from the dracut prompt, not surprisingly, does not show any of the /dev/sdX. It did show the nvme and lvm partitions, however.
How it can show the lvm part's but not any /dev/sdX when the VGs are built on the sdX devices?
Here is module section from kernel-ml and stock (there are no differences int the modules group - only the "Early CPIO image"):
# lsinitrd -k 4.4.168-1.el7.elrepo.x86_64
Image: /boot/initramfs-4.4.168-1.el7.elrepo.x86_64.img: 21M
========================================================================
Version: dracut-033-554.el7
Arguments: -f
dracut modules:
bash
nss-softokn
i18n
network
ifcfg
drm
plymouth
dm
kernel-modules
lvm
resume
rootfs-block
terminfo
udev-rules
biosdevname
systemd
usrmount
base
fs-lib
microcode_ctl-fw_dir_override
shutdown
# lsinitrd -k 3.10.0-957.1.3.el7.x86_64
Image: /boot/initramfs-3.10.0-957.1.3.el7.x86_64.img: 21M
========================================================================
Early CPIO image
========================================================================
drwxr-xr-x 3 root root 0 Dec 4 07:52 .
-rw-r--r-- 1 root root 2 Dec 4 07:52 early_cpio
drwxr-xr-x 3 root root 0 Dec 4 07:52 kernel
drwxr-xr-x 3 root root 0 Dec 4 07:52 kernel/x86
drwxr-xr-x 2 root root 0 Dec 4 07:52 kernel/x86/microcode
-rw-r--r-- 1 root root 31744 Dec 4 07:52 kernel/x86/microcode/GenuineIntel.bin
========================================================================
Version: dracut-033-554.el7
Arguments: -f
dracut modules:
bash
nss-softokn
i18n
network
ifcfg
drm
plymouth
dm
kernel-modules
lvm
resume
rootfs-block
terminfo
udev-rules
biosdevname
systemd
usrmount
base
fs-lib
microcode_ctl-fw_dir_override
shutdown |
|
|
|
Yes, something strange is going on.
If it is related to the dm/lvm2 updates, then you could revert to 7.5 dm/lvm2, remake initramfs images and retest. I am not clear on the relationship of the dracut dm/lvm2 modules to the installed dm/lvm2, so it might be a waste of time to downgrade.
On the other hand, it is something to try.
On a downgraded system:
# rpm -q device-mapper device-mapper-event device-mapper-event-libs device-mapper-libs lvm2 lvm2-libs
device-mapper-1.02.146-4.el7.x86_64
device-mapper-event-1.02.146-4.el7.x86_64
device-mapper-event-libs-1.02.146-4.el7.x86_64
device-mapper-libs-1.02.146-4.el7.x86_64
lvm2-2.02.177-4.el7.x86_64
lvm2-libs-2.02.177-4.el7.x86_64
and yum.conf includes:
exclude=device-mapper-1.02.149-8.el7
device-mapper-event-1.02.149-8.el7
device-mapper-event-libs-1.02.149-8.el7
device-mapper-libs-1.02.149-8.el7
lvm2-2.02.180-8.el7
lvm2-libs-2.02.180-8.el7
device-mapper-1.02.149-10.el7_6
device-mapper-event-1.02.149-10.el7_6
device-mapper-event-libs-1.02.149-10.el7_6
device-mapper-libs-1.02.149-10.el7_6
lvm2-2.02.180-10.el7_6
lvm2-libs-2.02.180-10.el7_6
device-mapper-1.02.149-10.el7_6.1
device-mapper-event-1.02.149-10.el7_6.1
device-mapper-event-libs-1.02.149-10.el7_6.1
device-mapper-libs-1.02.149-10.el7_6.1
lvm2-2.02.180-10.el7_6.1
lvm2-libs-2.02.180-10.el7_6.1
device-mapper-1.02.149-10.el7_6.2
device-mapper-event-1.02.149-10.el7_6.2
device-mapper-event-libs-1.02.149-10.el7_6.2
device-mapper-libs-1.02.149-10.el7_6.2
lvm2-2.02.180-10.el7_6.2
lvm2-libs-2.02.180-10.el7_6.2 |
|
|
|
Bad news... the dm and lvm2 downgrade is not producing any different results. I removed all kernel-lt:
yum remove kernel-lt\*
I had to have the following in the /etc/yum.con:
exclude=device-mapper-1.02.149-8.el7
device-mapper-event-1.02.149-8.el7
device-mapper-event-libs-1.02.149-8.el7
device-mapper-libs-1.02.149-8.el7
lvm2-2.02.180-8.el7
lvm2-libs-2.02.180-8.el7
device-mapper-1.02.149-10.el7_6
device-mapper-event-1.02.149-10.el7_6
device-mapper-event-libs-1.02.149-10.el7_6
device-mapper-libs-1.02.149-10.el7_6
lvm2-2.02.180-10.el7_6
lvm2-libs-2.02.180-10.el7_6
device-mapper-1.02.149-10.el7_6.1
device-mapper-event-1.02.149-10.el7_6.1
device-mapper-event-libs-1.02.149-10.el7_6.1
device-mapper-libs-1.02.149-10.el7_6.1
lvm2-2.02.180-10.el7_6.1
lvm2-libs-2.02.180-10.el7_6.1
device-mapper-1.02.149-10.el7_6.2
device-mapper-event-1.02.149-10.el7_6.2
device-mapper-event-libs-1.02.149-10.el7_6.2
device-mapper-libs-1.02.149-10.el7_6.2
lvm2-2.02.180-10.el7_6.2
lvm2-libs-2.02.180-10.el7_6.2
lvm2-python-libs-2.02.180-8.el7
lvm2-python-libs-2.02.180-10.el7_6.1
lvm2-python-libs-2.02.180-10.el7_6.2
device-mapper-event-1.02.149-10.el7_6.2
kpartx-0.4.9-123.el7
and then issue the downgrade command:
yum downgrade kpartx\* device-manager\* lvm2\* device-mapper\* --enablerepo=C7.5.1804-updates,C7.5.1804-base
For current list of:
# rpm -qa device-manager\* lvm2\* device-mapper\*
device-mapper-event-1.02.146-4.el7.x86_64
device-mapper-libs-1.02.146-4.el7.x86_64
lvm2-2.02.177-4.el7.x86_64
device-mapper-persistent-data-0.7.3-3.el7.x86_64
device-mapper-1.02.146-4.el7.x86_64
lvm2-python-libs-2.02.177-4.el7.x86_64
device-mapper-event-libs-1.02.146-4.el7.x86_64
device-mapper-multipath-libs-0.4.9-119.el7_5.1.x86_64
device-mapper-multipath-0.4.9-119.el7_5.1.x86_64
lvm2-libs-2.02.177-4.el7.x86_64
Installed the kernel-lt:
yum -y install kernel-lt
elrepo-kernel is enabled by default on this system.
End result: kernel-lt-4.4.168-1.el7.elrepo.x86_64 is not working for some reason (and at least previous)... |
|
|
|
Downgrade produced no positive results for the kernel-lt. Any other ideas. Keeping in mind that neither the kernel-ml and stock C7.x kernels are afflicted.
On a side note, the nvme md performance on the kernel-ml is much better than stock for my first round of testing. |
|
|
|
The attached file, only_in_config-4.4.168.txt, contains 114 lines from the configuration file of kernel-lt that are not present in either the distro or the kernel-ml configuration files.
Perhaps its contents will provide inspiration? |
|
|
|
I know almost nothing about the internals of the kernel. Just scanning the listed differences has me wondering about the CONFIG_NVMEM=m as that is not seen as a module in either stock C7 or 4.19.
It's seems more like something else is borked... why/how could I see the LVM's with the blkid command but not see the underlying /dev/sd* files? |
|
|
|
Sorry I could not be of more help.
One thing to think about is that kernel-lt (kernel-4.4) is getting somewhat (cough) old (cough) in all the stacks. In theory, only security fixes are patched into the stable kernel.org kernels, unlike the backporting that goes on in the stock el7 kernel.
I use kernel-lt on a 10-year-old HP laptop with an early AMD dual-core processor, because that laptop stopped booting the stock kernels around el 7.2. Thank you RHEL for disabling a critical module in addition to all the old NICs. :-)
Also, I needed to build the Broadcom wl module for it and discovered that the Broadcom hybrid driver, which itself is getting old (10/01/2015), built and functioned as-is without modification for kernel-lt.
In contrast, the wireless stack of the standard el7 kernels have been backported so many times that the Broadcom hybrid driver needs patches related to the wireless stacks of kernel-4.7, kernel-4.8, kernel-4.11 and kernel-4.12 before the wl driver will build. Wireless stack changes are also the reasons so many elrepo wireless drivers need a rebuild at some point releases.
It is my assumption that similar things are happening in the other stacks, too.
It could be that your hardware needs drivers not present in kernel-4.4. I see that you are using m.2 nvme ssd drives. Did those even exist in the time frame of kernel-4.4, which was initially released on 2016-01-10? Of course, there should be fallback to some level of PCIe, I would think.
Yeah, like that really fixes your problem. :-) |
|
|
|
In the past, I've tried using the kernel-ml on some systems and found it, frankly, unstable. To a large degree involving the wifi and similar items you've mentioned. Hardware would just stop working for some time, I'd wait for another release or two and it would work again.
Though not used for any OS file system, the nvme drives did show in the /dev/ on the 4.4 kernel.
Thanks for taking the time to reflect on this. I at least have the option to continue to test in both the stock and -ml kernels to maximize IO for a couple of database projects. |
|
