View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000944 | channel: elrepo/el8 | --elrepo--request-for-enhancement-- | public | 2019-09-24 15:46 | 2020-04-09 00:14 |
| Reporter | tru | Assigned To | pperry | ||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | resolved | Resolution | fixed | ||
| Summary | 0000944: RFE for kmod-nvidia* for rhel8/c8/c8-stream | ||||
| Description | Would it be possible to add the nvidia drivers for el8/c8* ? | ||||
| Additional Information | Nvidia seems to be providing some rpms, but no idea where to find the spec file or srpm. https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=CentOS&target_version=8&target_type=rpmnetwork which points to http://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/ | ||||
| Tags | No tags attached. | ||||
|
|
Acknowledged. I'll see if I can start work on these, but I don't have hardware or an el8 installation atm to test, so good testing/feedback by others will be critical. |
|
|
I really value this package--if it will help expedite delivery, I'm happy to test. I've only ever added the ELRepo repo and the yum plugin, so I'm not sure what it would look like to test, but happy to help to get this underway. Is there some "shadow" repo or something? |
|
|
Following up here. I have the driver installed from the 430.50 .run file, but Wayland must be disabled. GNOME has started to provide some limited support for the use of a Wayland session on top of NVIDIA drivers, but only starting with in GNOME 3.32. Meanwhile, as of this writing, I'm running CentOS Linux release 8.0.1905 (Core) with kernel 4.18.0-80.11.2.el8_0.x86_64, and that only has GNOME Version 3.28.2, so Wayland must be disabled and the legacy Xorg environment must be installed and enabled so that the NVIDIA drivers can work. https://www.nvidia.com/Download/driverResults.aspx/151568/ |
|
|
There's now another driver available: https://www.nvidia.com/Download/driverResults.aspx/153226/en-us |
|
|
I have built some preliminary packages, and made them available in the testing repository. They should show up shortly on mirrors. kmod-nvidia-440.36-1.el8_0.elrepo.src.rpm kmod-nvidia-440.36-1.el8_0.elrepo.x86_64.rpm kmod-nvidia-440.36-1.el8_1.elrepo.src.rpm kmod-nvidia-440.36-1.el8_1.elrepo.x86_64.rpm nvidia-x11-drv-440.36-1.el8.elrepo.nosrc.rpm nvidia-x11-drv-440.36-1.el8.elrepo.x86_64.rpm nvidia-x11-drv-libs-440.36-1.el8.elrepo.i686.rpm nvidia-x11-drv-libs-440.36-1.el8.elrepo.x86_64.rpm kmod packages are built for both el8_0 and el8_1 kernels, so pick whichever is appropriate for you. I should note that the el8_0 kmod package has been built with a newer gcc (that from 8.1) than it's matching el8.0 kernel, which may or may not cause issues. I intend to fix this for release but have been struggling to get the old el8.0 gcc version into my buildroot at present. I've had a very quick look at requires and conflicts for the nvidia-x11-drv(-libs) packages and hopefully resolved most of these for el8. Yum seems to think the package set will install without error but I've done no testing whatsoever. Please can we track any successes / failures / issues here. |
|
|
kmod-nvidia-440.36-1.el8_0.elrepo.x86_64.rpm has been rebuilt using the correct gcc version for the el8.0 kernel. I've not bumped the release, but the new version is now in the testing repository - package dated 7/12/2019. |
|
|
I can also assist with testing. I'm running CentOS 8_0 4.18.0-80.11.2.el8_0.x86_64 I haven't tried yet disabling Wayland on this machine yet (X1 Extreme Gen 1). Do you have any instructions on doing so that worked for you? |
|
|
@tstrange, I don't have hardware to test, but from googling I've found two possible solutions: One method suggests using the 'gear' icon for options on the GDM login screen and there should be the option to login on to a GNOME Xorg session as opposed to GNOME Wayland. The other solution mentioned suggests editing /etc/gdm/custom.conf to uncomment the following line: WaylandEnable=false and presumably rebooting. Let me know if there is something you think the package should be doing, or any additional steps required so we can look to automate things where possible. Thanks. |
|
|
To disable Wayland and enable the legacy X environment, you need to install the dependencies: # dnf groupinstall "Workstation" "base-x" "Legacy X Window System Compatibility" "Development Tools" # dnf install elfutils-libelf-devel "kernel-devel-uname-r == $(uname -r)" This must be done before running the NVIDIA .run file, but I'm not sure about what this means for packaging this up into ELRepo. My notes for installing via the .run file on CentOS 8 have been rock solid: https://sethgoldin.github.io/install-davinci-resolve-centos/ |
|
|
@sethgoldin When packaged, dnf/yum should automatically resolve and pull in any dependencies so we shouldn't need to worry about installing required stuff as would be required when installing from NVIDIA's unpackaged .run file. Thanks for the guide (that will be helpful) - I will take a look through in detail and make sure we are implementing similar steps in our package, where appropriate. |
|
|
@pperry I have been using the packages in elrepo-testing for about a month now without issue in a graphical environment. I think (from my experience) it may be safe to upgrade the driver to the latest version [440.44] and send it to the stable repo. Cheers, Mike |
|
|
Hi Mike, Thank you for the feedback! I will look to get the nvidia driver updated this week to the latest Long Lived release and built for el8.1. I'll probably keep it in the testing repository for now until I've got more feedback but would welcome another date point from yourself once updated. I'll post back here once done. |
|
|
Thanks for your packaging efforts. I'll test your new packages when available and do some more thorough tests (card setting management (i.e. coolbits, etc) and CUDA/OpenCL runs) and report back to you on it. I've got some workloads and benchmarks I can run it through. Cheers, Mike |
|
|
Cool, thanks Mike - much appreciated. |
|
|
The 440.44 driver package has been built for el8.1 and released to the testing repository. It should be available from mirror sites shortly. Any feedback welcomed. Thanks |
|
|
So I gave it a whirl, and I can't install it successfully. I've tried applying it in an active GNOME session, multi-user, and unloading the nvidia_drm module and performing the upgrade. All of them install just fine, the problem is the reboot just hangs on either the "Started custom profiles" or the "...color management..." line. Downgrading back to 440.36 from another tty works just fine. I didn't see anything standing out in dmesg or the journal, any ideas? |
|
|
Are you rebooting to an 8.1 kernel after applying the update? The kmod package is not backward compatible with 8.0 series kernels. |
|
|
Yep, this system never had the 8.0 kernel installed. Been booting with 4.18.0-147.3.1 for some time now. Is the package working on your end? |
|
|
Ah, I forgot I built packages for the previous release for both el8.0 and el8.1. I'd still recommend a full system reboot after updating to be sure though. Anything in the Xorg log? I have no physical hardware available to test so am completely reliant on you to tell me what is broken and what to fix :-) |
|
|
Seems my logging was not configured to be persistent (and Xorg lacks 440.44 references). Just got journald to be persistent and I'm going to do the following: - Reboot to multi-user - Unload nvidia_drm - Upgrade the driver - Reboot - Capture all logs (dmesg, journal, Xorg) to backup files - Perform downgrade procedure I'll get back to you when I've done this. |
|
|
It seems that the system is choosing to still use the 440.36 kmod rather than the 440.44. I don't know how as I expected it to be replaced. Here's a tarball of dmesg, journal, Xorg, and boot.log if it helps. https://www.dropbox.com/s/676fh1yvft157bq/failed-boot.tar?dl=0 |
|
|
Yes, looks like the old version 440.36 module is trying to load. And you have performed a full reboot? as root, please can you paste the output from the following: lsinitrd -k $(uname -r) | egrep 'nvidia.*ko' Lets see which kernel module is in the initramfs image. |
|
|
Yep, always perform a reboot after updating kmods/drivers. From the unstable 440.44 boot: lrwxrwxrwx 1 root root 53 Nov 6 10:07 usr/lib/modules/4.18.0-147.3.1.el8_1.x86_64/weak-updates/nvidia/nvidia.ko -> ../../../4.18.0-147.el8.x86_64/extra/nvidia/nvidia.ko -rw-r--r-- 1 root root 27179801 Nov 6 10:07 usr/lib/modules/4.18.0-147.el8.x86_64/extra/nvidia/nvidia.ko From the stable 440.36 boo: lrwxrwxrwx 1 root root 53 Nov 6 10:07 usr/lib/modules/4.18.0-147.3.1.el8_1.x86_64/weak-updates/nvidia/nvidia.ko -> ../../../4.18.0-147.el8.x86_64/extra/nvidia/nvidia.ko -rw-r--r-- 1 root root 27179801 Nov 6 10:07 usr/lib/modules/4.18.0-147.el8.x86_64/extra/nvidia/nvidia.ko |
|
|
OK, that's not right - the date stamp shows Nov 6 10:07?? That's before _any_ elrepo packages have been released, even the first 440.36 packages (built Dec 1). So that presumably means your initramfs is not being updated? Had you previously done an install of the 440.36 driver using the nvidia installer? I'm assuming the answer to that is yes (on Nov 6 - although that was the release date of 441.31, and your logs show 440.36??). Also, the time stamps for the actual module and the symlink in weak-updates should be different, as the former is the timestamp the file was compiled by me and the latter is the timestamp the package was installed (and the symlink created) by yourself. With the latest packages installed, you could try removing (modprobe -r nvidia) the incorrect 440.36 driver if it's loaded, checking the current version is 440.44 (modinfo nvidia | grep version) and manually loading this driver (modprobe nvidia), check it's loaded (lsmod | grep nvidia) and then try manually rebuilding your initramfs with the correct version kernel module loaded: dracut --add-drivers nvidia -f /boot/initramfs-$(uname -r).img $(uname -r) and Then reboot and keep your fingers crossed! |
|
|
I to was a bit concerned about the timestamps. I set this machine up in mid-December, and never installed the driver from NVIDIA, only elrepo-testing. However, just 'ls -l'ing the /usr/lib/modules/4.18.0-147.el8/extras/nvidia has timestamps of Dec 1 (and if I recall for 440.44, had the Jan 25 timestamp). So lsinitrd is reporting some funky results. If you think it's a better solution, I can trying completely removing the driver and then installing the new one. Something may have gone wrong internally after issuing the dnf downgrade(?). Do you see any obvious issues in the %post %posttrans %preun or %postun parts of the specfile that could cause any problems? I've never had this issue before with ELRepo on EL7. |
|
|
So your solution of rebuilding the initramfs works! I did however try this with a self-built package of 440.59 using the SRPMs available, but they hit the same errors on initial reboot. Performing the dracut flag fixed that problem. An interesting note, lsinitrd is still reporting the timestamp of Nov 6, though the directories (-147/extras/nvidia, -5.1/weak-updates/) list Feb 4 from the build date. And while I was in the faulty state from a TTY console, the modinfo was showing 440.59, not 440.36, so I didn't need to remove it from being loaded. Cheers, Mike |
|
|
I just tested kmod-nvidia-440.44-1.el8_1.elrepo with a fresh CentOS 8.1 install. The RPM did appear to add the nvidia.ko to the initrd, but the nouveau driver was still in there as well, and seemed to take precedence on startup. I created the file /etc/dracut.conf.d/blacklist-nouveau.conf containing: omit_drivers += " nouveau " and ran "dracut -f". After a reboot I can confirm the nvidia driver is active. Perhaps adding this extra blacklist file to the RPM would solve the issue above? |
|
|
Minor correction, the blacklist-nouveau.conf file (with corrected whitespace) should read: omit_drivers+=" nouveau " With this file in place before installing kmod-nvidia, it works great for me. |
|
|
Thanks for the feedback. There is a new upstream release, so I'll try to get our package updated incorporating changes this weekend. Please keep the feedback coming as it's all I have to work on atm. |
|
|
@darrin: Do you mind sharing the output of `sudo grubby --info=ALL`? I'm noticing that after running my build and updating initramfs, I've lost some kernel flags. 440.36 (3.1): args="[normal stuff] rhgb quiet $tuned_params nouveau.modeset=0 rd.driver.blacklist=nouveau plymouth.ignore-udev" 440.59 (5.1): args="[normal stuff] rhgb quiet $tuned_params" |
|
|
@mroche your missing kernel arguments were likely removed when you uninstalled nvidia-x11-drv. The %preun script removes those arguments from any installed kernels. If you then installed your own 440.59 package it would need to reinstate them. |
|
|
I have released an updated 440.59 package set, still in the testing repository for now. @darrin - as suggested, I've added /etc/dracut.conf.d/dracut-nvidia.conf with the following: add_drivers+=" nvidia " omit_drivers+=" nouveau " In addition to specifically omitting nouveau, lets also take the opportunity to specifically add nvidia just for good measure. Please can you test to ensure nouveau is omitted and nvidia added to the initramfs on new installations. Feedback welcome - thanks |
|
|
@mroche, sorry for the delay, I just got back to the office. Here's that grubby output, if still useful. [root@echo ~]# grubby --info=ALL index=0 kernel="/boot/vmlinuz-4.18.0-147.5.1.el8_1.x86_64" args="ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet $tuned_params nouveau.modeset=0 rd.driver.blacklist=nouveau plymouth.ignore-udev" root="/dev/mapper/cl-root" initrd="/boot/initramfs-4.18.0-147.5.1.el8_1.x86_64.img $tuned_initrd" title="CentOS Linux (4.18.0-147.5.1.el8_1.x86_64) 8 (Core)" id="cdcae8de7edb442bb821a7acfff4c9e2-4.18.0-147.5.1.el8_1.x86_64" index=1 kernel="/boot/vmlinuz-4.18.0-147.3.1.el8_1.x86_64" args="ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet $tuned_params nouveau.modeset=0 rd.driver.blacklist=nouveau plymouth.ignore-udev" root="/dev/mapper/cl-root" initrd="/boot/initramfs-4.18.0-147.3.1.el8_1.x86_64.img $tuned_initrd" title="CentOS Linux (4.18.0-147.3.1.el8_1.x86_64) 8 (Core)" id="cdcae8de7edb442bb821a7acfff4c9e2-4.18.0-147.3.1.el8_1.x86_64" index=2 kernel="/boot/vmlinuz-0-rescue-cdcae8de7edb442bb821a7acfff4c9e2" args="ro crashkernel=auto resume=/dev/mapper/cl-swap rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet" root="/dev/mapper/cl-root" initrd="/boot/initramfs-0-rescue-cdcae8de7edb442bb821a7acfff4c9e2.img" title="CentOS Linux (0-rescue-cdcae8de7edb442bb821a7acfff4c9e2) 8 (Core)" id="cdcae8de7edb442bb821a7acfff4c9e2-0-rescue" @pperry thanks for the 440.59 package. I removed my own dracut.conf.d and ran "dnf update" to install it.It looks like the initramfs was not updated - the driver in there its still dated Feb 7 and I get a NVRM error about the kernel/userpsace version mismatch. There could be some confusion here because I'd installed kernel 4.18.0-147.5.1.el8_1, but had not rebooted, so had an older kernel running at the time of the NVIDIA driver update. I tried a "dnf reinstall kmod-nvidia" but this also did not update the initramfs. The posttrans script is looking for /boot/symvers-* files, which don't exist on my systems. D: %posttrans(kmod-nvidia-440.59-1.el8_1.elrepo.x86_64): scriptlet start D: %posttrans(kmod-nvidia-440.59-1.el8_1.elrepo.x86_64): execv(/bin/sh) pid 12750 D: Plugin: calling hook scriptlet_fork_post in selinux plugin + '[' -f /var/lib/rpm-state/kmod-dups/kver/4.18.0-147.el8.x86_64 ']' + kver_base=4.18.0-147.el8 ++ ls -d /lib/modules/4.18.0-147.3.1.el8_1.x86_64 /lib/modules/4.18.0-147.5.1.el8_1.x86_64 /lib/modules/4.18.0-147.6.el8.x86_64 /lib/modules/4.18.0-147.el8.x86_64 + kvers='/lib/modules/4.18.0-147.3.1.el8_1.x86_64 /lib/modules/4.18.0-147.5.1.el8_1.x86_64 /lib/modules/4.18.0-147.6.el8.x86_64 /lib/modules/4.18.0-147.el8.x86_64' + for k_dir in $kvers + k=4.18.0-147.3.1.el8_1.x86_64 + tmp_initramfs=/boot/initramfs-4.18.0-147.3.1.el8_1.x86_64.tmp + dst_initramfs=/boot/initramfs-4.18.0-147.3.1.el8_1.x86_64.img + '[' -e /boot/symvers-4.18.0-147.3.1.el8_1.x86_64.gz ']' + for k_dir in $kvers + k=4.18.0-147.5.1.el8_1.x86_64 + tmp_initramfs=/boot/initramfs-4.18.0-147.5.1.el8_1.x86_64.tmp + dst_initramfs=/boot/initramfs-4.18.0-147.5.1.el8_1.x86_64.img + '[' -e /boot/symvers-4.18.0-147.5.1.el8_1.x86_64.gz ']' + for k_dir in $kvers + k=4.18.0-147.6.el8.x86_64 + tmp_initramfs=/boot/initramfs-4.18.0-147.6.el8.x86_64.tmp + dst_initramfs=/boot/initramfs-4.18.0-147.6.el8.x86_64.img + '[' -e /boot/symvers-4.18.0-147.6.el8.x86_64.gz ']' + for k_dir in $kvers + k=4.18.0-147.el8.x86_64 + tmp_initramfs=/boot/initramfs-4.18.0-147.el8.x86_64.tmp + dst_initramfs=/boot/initramfs-4.18.0-147.el8.x86_64.img + '[' -e /boot/symvers-4.18.0-147.el8.x86_64.gz ']' + rm -f /var/lib/rpm-state/kmod-dups/kver/4.18.0-147.el8.x86_64 + rmdir /var/lib/rpm-state/kmod-dups/kver + rmdir /var/lib/rpm-state/kmod-dups + exit 0 D: %posttrans(kmod-nvidia-440.59-1.el8_1.elrepo.x86_64): waitpid(12750) rc 12750 status 0 [root@echo tmp]# ls /boot config-4.18.0-147.3.1.el8_1.x86_64 initramfs-4.18.0-147.3.1.el8_1.x86_64kdump.img System.map-4.18.0-147.5.1.el8_1.x86_64 config-4.18.0-147.5.1.el8_1.x86_64 initramfs-4.18.0-147.5.1.el8_1.x86_64.img vmlinuz-0-rescue-cdcae8de7edb442bb821a7acfff4c9e2 efi initramfs-4.18.0-147.5.1.el8_1.x86_64kdump.img vmlinuz-4.18.0-147.3.1.el8_1.x86_64 grub2 loader vmlinuz-4.18.0-147.5.1.el8_1.x86_64 initramfs-0-rescue-cdcae8de7edb442bb821a7acfff4c9e2.img lost+found initramfs-4.18.0-147.3.1.el8_1.x86_64.img System.map-4.18.0-147.3.1.el8_1.x86_64 |
|
|
> If you then installed your own 440.59 package it would need to reinstate them. I have since done so, but I figured the package would have handled it considering I used the source rpms for kmod-nvidia and nvidia-x11-drv from elrepo-testing to rebuild, just bumping the version in the specfile. |
|
|
I should add I've never performed an rpm -e or dnf erase, just dnf upgrade <package> and dnf downgrade <package> and the flags were never removed prior. |
|
|
I finally have a rhel8 testing box up and running (huge thanks for the donation!), but only have an older card supported by the 390xx driver. So I have ported nvidia-390xx packages based on the existing packages, and initial testing of the packaging looks sane and they appear to work as intended. Thanks to Seth earlier in this thread for clues how to get Xorg properly installed. However, driver performance is not great. I normally test with glmark2, but it's dependence on python is making it a little tricky to port to el8. For now I've only been able to do some very quick testing with glxgears and performance looks a little strange - I'm seeing abnormally high CPU usage compared to el7. Packages are in the testing repo if anyone would like to have a play - they should still support most cards released before Sept 2019. I've also released nvidia-detect for el8. |
|
|
hi, thanks for your work. Testing a legacy Dell E6420 laptop here with: 01:00.0 VGA compatible controller: NVIDIA Corporation GF119M [NVS 4200M] (rev a1) [tru@localhost ~]$ nvidia-detect kmod-nvidia-390xx [tru@localhost log]$ nvidia-smi Tue Mar 31 15:15:01 2020 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 390.132 Driver Version: 390.132 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 NVS 4200M Off | 00000000:01:00.0 N/A | N/A | | N/A 52C P0 N/A / N/A | 0MiB / 453MiB | N/A Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 Not Supported | +-----------------------------------------------------------------------------+ [tru@localhost log]$ modinfo nvidia filename: /lib/modules/4.18.0-147.5.1.el8_1.x86_64/weak-updates/nvidia-390xx/nvidia.ko alias: char-major-195-* version: 390.132 supported: external license: NVIDIA rhelversion: 8.1 srcversion: C9CBE920F0195665AFA8A5C alias: pci:v000010DEd00000E00sv*sd*bc04sc80i00* alias: pci:v000010DEd*sv*sd*bc03sc02i00* alias: pci:v000010DEd*sv*sd*bc03sc00i00* depends: ipmi_msghandler name: nvidia vermagic: 4.18.0-147.el8.x86_64 SMP mod_unload modversions sig_id: PKCS#7 signer: ELRepo.org Secure Boot Key sig_key: E9:D4:71:CF:B4:FE:13:6C sig_hashalgo: sha256 signature: 8A:D5:E2:52:CA:E6:FC:8A:E6:E5:93:5E:5D:A3:59:64:C6:AB:60:63: ... Great job :D +1 for pushing it to release |
|
|
nvidia packages updated to latest 440.64 release and released to main repo. nvidia-390xx packages moved from testing to main repo. Packages currently syncing to mirrors and should be available in the main repository shortly. Please file new bugs against the respective packages if you have any issues. I'll close this RFE shortly. |
|
|
Thanks so much! As an aside, I can confirm the poor performance you've noticed with 440.64. I think it's an issue with the NVIDA drivers themselves. More info here: https://twitter.com/sethgoldin/status/1245087161600311297 |
|
|
Hi Seth, Actually, the poor performance I noticed was for 390.132 on rhel8. I've only done limited testing with glmark2 on el7, but I see no difference in performance between 440.59 and 440.64 in glmark2 on el7 (I performed a quick sanity test with glmark2 on el7 for every release, and have data going back around 2.5 years) |
|
|
Interesting. I noticed that 440.64 was causing some sort of memory leak. Downgraded back to 440.59 and all is well. Waiting for the next one... |
|
|
Latest 440.82 upstream release packages available from the main elrepo repository shortly. I'm marking this RFE as resolved. Please open new bug reports against the package(s) if you have any issues. Many thanks to everyone for your feedback and support. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2019-09-24 15:46 | tru | New Issue | |
| 2019-09-24 15:46 | tru | Status | new => assigned |
| 2019-09-24 15:46 | tru | Assigned To | => toracat |
| 2019-09-24 16:45 | toracat | Assigned To | toracat => pperry |
| 2019-09-25 00:16 | pperry | Note Added: 0006510 | |
| 2019-10-08 10:07 | sethgoldin | Note Added: 0006587 | |
| 2019-11-01 10:02 | sethgoldin | Note Added: 0006641 | |
| 2019-11-01 10:43 | burakkucat | Summary | RFE for kmod-nvidia* for rhel8/c8/c8-stream ;P => RFE for kmod-nvidia* for rhel8/c8/c8-stream |
| 2019-11-07 13:30 | sethgoldin | Note Added: 0006658 | |
| 2019-12-01 09:36 | pperry | Note Added: 0006719 | |
| 2019-12-08 04:24 | pperry | Note Added: 0006732 | |
| 2019-12-09 10:14 | tstrange | Note Added: 0006737 | |
| 2019-12-09 12:05 | pperry | Note Added: 0006738 | |
| 2019-12-09 17:12 | sethgoldin | Note Added: 0006739 | |
| 2019-12-10 00:32 | pperry | Note Added: 0006740 | |
| 2020-01-19 22:18 | mroche | Note Added: 0006765 | |
| 2020-01-20 15:02 | pperry | Note Added: 0006766 | |
| 2020-01-20 15:10 | mroche | Note Added: 0006767 | |
| 2020-01-20 15:56 | pperry | Note Added: 0006768 | |
| 2020-01-25 15:58 | pperry | Note Added: 0006775 | |
| 2020-01-26 11:05 | mroche | Note Added: 0006779 | |
| 2020-01-26 13:23 | pperry | Note Added: 0006780 | |
| 2020-01-26 14:11 | mroche | Note Added: 0006781 | |
| 2020-01-26 15:52 | pperry | Note Added: 0006783 | |
| 2020-01-26 19:22 | mroche | Note Added: 0006784 | |
| 2020-01-28 10:23 | mroche | Note Added: 0006788 | |
| 2020-01-28 13:27 | pperry | Note Added: 0006790 | |
| 2020-01-28 16:46 | mroche | Note Added: 0006793 | |
| 2020-01-29 00:05 | pperry | Note Added: 0006795 | |
| 2020-01-29 08:43 | mroche | Note Added: 0006797 | |
| 2020-02-04 16:23 | mroche | Note Added: 0006801 | |
| 2020-02-05 11:45 | darrin | Note Added: 0006802 | |
| 2020-02-05 16:14 | darrin | Note Added: 0006803 | |
| 2020-02-05 16:34 | pperry | Note Added: 0006804 | |
| 2020-02-08 12:50 | mroche | Note Added: 0006805 | |
| 2020-02-09 11:26 | pperry | Note Added: 0006806 | |
| 2020-02-09 11:30 | pperry | Note Added: 0006807 | |
| 2020-02-09 11:31 | pperry | Note Edited: 0006806 | |
| 2020-02-11 12:59 | darrin | Note Added: 0006808 | |
| 2020-02-11 14:24 | mroche | Note Added: 0006809 | |
| 2020-02-11 14:27 | mroche | Note Added: 0006810 | |
| 2020-03-30 16:35 | pperry | Note Added: 0006863 | |
| 2020-03-31 07:18 | tru | Note Added: 0006864 | |
| 2020-04-01 13:49 | pperry | Note Added: 0006874 | |
| 2020-04-02 09:56 | sethgoldin | Note Added: 0006875 | |
| 2020-04-02 14:04 | pperry | Note Added: 0006876 | |
| 2020-04-02 17:09 | sethgoldin | Note Added: 0006877 | |
| 2020-04-09 00:14 | pperry | Note Added: 0006882 | |
| 2020-04-09 00:14 | pperry | Status | assigned => resolved |
| 2020-04-09 00:14 | pperry | Resolution | open => fixed |
