View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0001476 | channel: kernel/el9 | kernel-lt | public | 2024-08-20 13:35 | 2024-08-30 01:46 |
| Reporter | elrepo@mx.zzux.com | Assigned To | toracat | ||
| Priority | normal | Severity | major | Reproducibility | always |
| Status | resolved | Resolution | fixed | ||
| Platform | amd64 | OS | CentOS Stream | OS Version | 9 |
| Summary | 0001476: Update to kernel-lt-6.1.103-1.el9.elrepo.x86_64 causes very slow virtio_net operation | ||||
| Description | After kernel update I have very slow upload speed in all virtual machines with virtio_net NIC. Upload speed in virtual machines with virtio_net interface is 15000-25000 bytes/sec, while download is ok. Workaround: disable Offload Tx.LSO. Or (in my case) downgrade kernel to 6.1.102-1.el9.elrepo.x86_64 The issue is confirmed in the following guest operating systems Windows 7 Windows Server 2008 R2 Windows 10 Freebsd 12 I don't know where is the issue, in virtio or macvlan/macvtap. | ||||
| Steps To Reproduce | >dnf list installed *libvirt* *qemu* ipxe-roms-qemu.noarch 20200823-9.git4bd064de.el9 @AppStream libvirt.x86_64 10.5.0-5.el9 @appstream libvirt-client.x86_64 10.5.0-5.el9 @appstream libvirt-client-qemu.x86_64 10.5.0-5.el9 @appstream libvirt-daemon.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-common.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-config-network.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-config-nwfilter.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-interface.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-network.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-nodedev.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-nwfilter.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-qemu.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-secret.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-storage.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-storage-core.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-storage-disk.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-storage-iscsi.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-storage-logical.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-storage-mpath.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-storage-rbd.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-driver-storage-scsi.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-kvm.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-lock.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-log.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-plugin-lockd.x86_64 10.5.0-5.el9 @appstream libvirt-daemon-proxy.x86_64 10.5.0-5.el9 @appstream libvirt-glib.x86_64 4.0.0-3.el9 @appstream libvirt-libs.x86_64 10.5.0-5.el9 @appstream libvirt-ssh-proxy.x86_64 10.5.0-5.el9 @appstream python3-libvirt.x86_64 10.5.0-1.el9 @appstream qemu-guest-agent.x86_64 17:9.0.0-8.el9 @appstream qemu-img.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-audio-pa.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-block-blkio.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-block-rbd.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-common.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-core.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-device-display-virtio-gpu.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-device-display-virtio-gpu-pci.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-device-display-virtio-vga.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-device-usb-host.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-device-usb-redirect.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-docs.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-tools.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-ui-egl-headless.x86_64 17:9.0.0-8.el9 @appstream qemu-kvm-ui-opengl.x86_64 17:9.0.0-8.el9 @appstream qemu-pr-helper.x86_64 17:9.0.0-8.el9 @appstream >dnf update Nothing to do. Complete! (all updates are installed) >uname -r 6.1.103-1.el9.elrepo.x86_64 | ||||
| Additional Information | All the later kernels are with this issue also. | ||||
| Tags | No tags attached. | ||||
|
|
A quick look at the kernel config, there was no change when going from 6.1.102 to 6.1.103. Therefore, the reported issue most likely comes from the source code. I wonder if you could test-install kernel-ml and see if it has the same problem? |
|
|
If kernel-ml turns out to not have this problem, that would mean that there is (are) patch(es) that have yet to be backported to stable kernels. We'd then need to identify them. |
|
|
I have installed kernel-ml > uname -r 6.10.6-1.el9.elrepo.x86_64 This version seems to be free of the issue above. Please fix lt version also. |
|
|
We (ELRepo) build kernel-lt and kernel-ml using the source code released by kernel.org without any modification. Therefore we cannot "fix" anything that requires changes in the code. I suggest you file a bug report upstream at https://bugzilla.kernel.org just like you did in our bug tracker. Now, the fact that kernel-ml works means that the "fix" was applied to the mainline kernel (currently at 6.10.6) but was NOT backported to stable kernel 6.1.x (thus our kernel-lt). I have found a patch that seems to have fixed the current issue in the source code of the mainline kernel. However kernel developers were unable to backport it to stable kernel 6.1. One possible way to resolve (or workaround) the problem is to remove the patch that led to the issue to begin with. I'm not going into the details here but will be willing to discuss this matter once you have initiated the bug report upstream. I made a guess as to which patch (commit) caused the problem you are experiencing. And I re-built kernel-lt after removing this suspect. It is available here (packages not signed) : https://elrepo.org/people/toracat/devel/bug1476/ Could you give that a try? If my guess is right, this kernel-lt should work fine. |
|
|
> uname -r 6.1.106-1.bug1476.el9.x86_64 Thanks. With this kernel installed there is no upload speed slowdown. |
|
|
Excellent! The commit that was reverted was e269d79c7d35. Now we want to take this upstream (kernel.org). |
|
|
Good news. The fix is coming. Someone came up with patches for stable kernel 6.1. When this is out, kernel-lt will be fixed. https://lore.kernel.org/all/66c5f41884850_da1e7294d2@willemb.c.googlers.com.notmuch/ |
|
|
An update by Salvatore Bonaccorso <carnil@debian.org> to the thread above: Just to give an additional confirmation: Applying 1fd54773c267 ("udp: allow header check for dodgy GSO_UDP_L4 packets.") 9840036786d9 ("gso: fix dodgy bit handling for GSO_UDP_L4") fc8b2a619469 ("net: more strict VIRTIO_NET_HDR_GSO_UDP_L4 validation") 89add40066f9 ("net: drop bad gso csum_start and offset in virtio_net_hdr") addresses the issue from https://bugs.debian.org/1079684 matching https://bugzilla.kernel.org/show_bug.cgi?id=219129 |
|
|
I confirm that the 4 commits in #10061 are now queued for linux 6.1.107. Therefore kernel-lt will be fixed in the next release. |
|
|
kernel-lt-6.1.107-1.el9.elrepo has just been released to our mirror sites. Could you please install it and confirm the fix? |
|
|
> uname -r 6.1.107-1.el9.elrepo.x86_64 I have noticed no any issues after update. The virtio_net performance is ok. Thanks! |
|
|
Thanks for reporting back. Now marking this 'resolved'. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2024-08-20 13:35 | elrepo@mx.zzux.com | New Issue | |
| 2024-08-20 13:35 | elrepo@mx.zzux.com | Status | new => assigned |
| 2024-08-20 13:35 | elrepo@mx.zzux.com | Assigned To | => toracat |
| 2024-08-20 13:57 | toracat | Note Added: 0010040 | |
| 2024-08-20 14:37 | toracat | Note Added: 0010041 | |
| 2024-08-21 02:29 | elrepo@mx.zzux.com | Note Added: 0010044 | |
| 2024-08-21 17:19 | toracat | Note Added: 0010045 | |
| 2024-08-22 00:54 | elrepo@mx.zzux.com | Note Added: 0010051 | |
| 2024-08-22 11:27 | toracat | Note Added: 0010052 | |
| 2024-08-22 12:11 | toracat | Note Added: 0010053 | |
| 2024-08-26 19:54 | toracat | Note Added: 0010061 | |
| 2024-08-27 12:49 | toracat | Note Edited: 0010061 | |
| 2024-08-27 12:49 | toracat | Note View State: 0010061: public | |
| 2024-08-27 12:52 | toracat | Note Added: 0010062 | |
| 2024-08-29 18:34 | toracat | Note Added: 0010069 | |
| 2024-08-29 18:34 | toracat | Status | assigned => feedback |
| 2024-08-30 00:08 | elrepo@mx.zzux.com | Note Added: 0010070 | |
| 2024-08-30 00:08 | elrepo@mx.zzux.com | Status | feedback => assigned |
| 2024-08-30 01:46 | toracat | Status | assigned => resolved |
| 2024-08-30 01:46 | toracat | Resolution | open => fixed |
| 2024-08-30 01:46 | toracat | Note Added: 0010071 |
