View Issue Details

IDProjectCategoryView StatusLast Update
0000938channel: elrepo/el7kmod-ixgbepublic2019-10-01 17:51
Reportergleventhal Assigned Totoracat  
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
Summary0000938: Latency bug in in-tree ixgbe for kernels >= 3.10.0-957 (Also affects the kmod-ixgbe packages)
DescriptionI worked with Alex Duyck at Intel to resolve this already and have done testing on a patched version of the 5.5.5-1.el7 ixgbe-kmod[1] with this patch[2]. I am hoping you will back-port it so that I don't have to maintain the patched kmod-rpm separately from elrepo.org

Thanks!


[1]
(https://elrepo.org/linux/elrepo/el7/SRPMS/ixgbe-kmod-5.5.5-1.el7_6.elrepo.src.rpm)

[2]
https://lkml.org/lkml/2019/9/18/155
Additional Information--- src/ixgbe_main.c
+++ src/ixgbe_main.c
@@ -2622,7 +2622,7 @@ adjust_by_size:
         /* 16K ints/sec to 9.2K ints/sec */
         avg_wire_size *= 15;
         avg_wire_size += 11452;
- } else if (avg_wire_size <= 1980) {
+ } else if (avg_wire_size < 1968) {
         /* 9.2K ints/sec to 8K ints/sec */
         avg_wire_size *= 5;
         avg_wire_size += 22420;
@@ -2655,6 +2655,8 @@ adjust_by_size:
     case IXGBE_LINK_SPEED_2_5GB_FULL:
     case IXGBE_LINK_SPEED_1GB_FULL:
     case IXGBE_LINK_SPEED_10_FULL:
+ if (avg_wire_size > 8064)
+ avg_wire_size = 8064;
         itr += DIV_ROUND_UP(avg_wire_size,
                     IXGBE_ITR_ADAPTIVE_MIN_INC * 64) *
                IXGBE_ITR_ADAPTIVE_MIN_INC;
TagsNo tags attached.
Reported upstream

Activities

gleventhal

2019-09-18 13:38

reporter   ~0006478

Also, for what it's worth: I've already spoken with Phillip Perry about this and he's agreed to back-port this patch already (and asked that I file it here).

burakkucat

2019-09-18 13:45

administrator   ~0006479

This patch can be found upstream in Linus Torvalds' tree as --

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=377228accbbb8b9738f615d791aa803f41c067e0

toracat

2019-09-18 14:30

administrator   ~0006483

The patch apples without an issue. We will soon publish the updated version.

toracat

2019-09-18 15:11

administrator   ~0006484

The updated packages with the patch:

kmod-ixgbe-5.6.3-2.el7_7.elrepo.x86_64.rpm
ixgbe-kmod-5.6.3-2.el7_7.elrepo.src.rpm

will start sync'ing to our mirrors in a moment.

gleventhal

2019-09-19 09:35

reporter   ~0006487

That's great news, thanks so much!

toracat

2019-09-19 09:57

administrator   ~0006488

Thank *you* for the patch. Closing this as 'resolved'.

gleventhal

2019-09-24 14:16

reporter   ~0006503

Sorry to reopen, I just had one question regarding this RPM:

How do I know which kernel this was built against? How do I know that any kernel I run or will run in the future is compatible with this RPM or whether my kernel is compatible with any future releases of this RPM?

toracat

2019-09-24 14:57

administrator   ~0006504

Last edited: 2019-09-24 14:58

The following command will show which kernel the module is built against and which kernels it will work for:

ls -l `find /lib/modules -name ixgbe.ko`
 (note the back ticks)

Loo for the line that has "extra", and check the symbolic links.

gleventhal

2019-09-24 14:59

reporter   ~0006505

Is there a way to get that info from the rpm without installing it?

toracat

2019-09-24 15:07

administrator   ~0006506

The kernel version the module is built against is easy to find. The tag el7_7 means it was built against the EL 7.7 kernel. To identify the actual version, you may need to see the content of the kmod, like so:

rpm2cpio kmod-ixgbe-5.6.3-2.el7_7.elrepo.x86_64.rpm | cpio -idv

and see where the module is installed (extra/)

Finding out if the module works with a certain version of kernel without installing the kmod is not possible.

gleventhal

2019-09-25 07:06

reporter   ~0006511

Since (I believe) this kmod RPM isn't 100% kABI white-list compliant, are you guys going to have to build a new RPM every time a kernel update comes out to guarantee that it's compatible?

How do I know whether this RPM will work with the next kernel update that Red Hat releases under 7.7 ?

gleventhal

2019-09-25 07:34

reporter   ~0006512

# ksc -k /lib/modules/3.10.0-1062.el7.x86_64/extra/ixgbe/ixgbe.ko
Checking against architecture x86_64
Total symbol usage: 258 Total Non white list symbol usage: 127
Score: 50.78%

As you can see, half of the interfaces in this module are not on the kABI white-list, and at https://access.redhat.com/solutions/444773 Red Hat says that there is no guarantee that a non-whitelist interface won't change within a minor release.

Does ElRepo test this RPM with every minor release? If so, what do you do when a needed interface changes? Do you release another RPM that's compatible with that minor release? Do you modify the existing RPM to be backwards and forwards compatible, adding conditionally-applied patches for that minor kernel version?

pperry

2019-09-25 14:10

administrator   ~0006513

Last edited: 2019-09-25 14:13

Most packages are compatible with kernel updates. There are no guarantees as we have no way of knowing what changes RH will make in any future kernels. The one guarantee we have is that symbols on the kABI whitelist will not change, but almost every driver uses some symbols that do not have this whitelisted protection.

So what happens is RH releases a kernel update. You install kernel update. You see if your driver continues to work or not. If it continues to work you are happy/grateful. If it doesn't you file a bug report here and we fix it by rebuilding it against the new kernel RH just released. We will always aim to rebuild and release any driver within 24 hours (often less, but no guarantees as we work for free and some of us have real lives and day jobs and families too). If it's a popular driver that is widely used, someone else may have already reported the issue and the fix may have already been released before you even get around to updating your kernel.

When things break and we fix them, it will always be a new RPM package release that yum will automatically pick up as an update. Generally, new packages that are required to fix kABI breakage are not backwards compatible with older kernels. So if your driver needs rebuilding for el7.7 (and contains .el7_7 in the dist tag), it is unlikely to be backward compatible with 7.6 and earlier kernels.

gleventhal

2019-09-25 14:35

reporter   ~0006514

Thanks for clarifying, I believe that answers all of my questions. I will do my best to be a good citizen and report anything I see from my end. If you have any recommendations on testing future kernel releases, please let me know, otherwise I will just do my best to test however seems appropriate.

Thanks again for everything!

Issue History

Date Modified Username Field Change
2019-09-18 13:31 gleventhal New Issue
2019-09-18 13:31 gleventhal Status new => assigned
2019-09-18 13:31 gleventhal Assigned To => toracat
2019-09-18 13:38 gleventhal Note Added: 0006478
2019-09-18 13:45 burakkucat Note Added: 0006479
2019-09-18 14:30 toracat Note Added: 0006483
2019-09-18 15:11 toracat Note Added: 0006484
2019-09-19 09:35 gleventhal Note Added: 0006487
2019-09-19 09:57 toracat Note Added: 0006488
2019-09-19 09:57 toracat Status assigned => resolved
2019-09-19 09:57 toracat Resolution open => fixed
2019-09-24 14:16 gleventhal Note Added: 0006503
2019-09-24 14:16 gleventhal Status resolved => assigned
2019-09-24 14:16 gleventhal Resolution fixed => reopened
2019-09-24 14:57 toracat Note Added: 0006504
2019-09-24 14:58 toracat Note Edited: 0006504
2019-09-24 14:59 gleventhal Note Added: 0006505
2019-09-24 15:07 toracat Note Added: 0006506
2019-09-25 07:06 gleventhal Note Added: 0006511
2019-09-25 07:34 gleventhal Note Added: 0006512
2019-09-25 14:10 pperry Note Added: 0006513
2019-09-25 14:11 pperry Note Edited: 0006513
2019-09-25 14:13 pperry Note Edited: 0006513
2019-09-25 14:35 gleventhal Note Added: 0006514
2019-10-01 17:51 toracat Status assigned => resolved
2019-10-01 17:51 toracat Resolution reopened => fixed