View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0000852 | channel: elrepo/el7 | kmod-cciss | public | 2018-05-16 02:47 | 2022-03-18 01:52 |
| Reporter | soria | Assigned To | pperry | ||
| Priority | normal | Severity | block | Reproducibility | always |
| Status | assigned | Resolution | open | ||
| Summary | 0000852: CCISS kmod module not working with Red Hat Enterprise Linux Server 7.5, kernel 3.10.0-862 | ||||
| Description | I have a HP 360 G4, with a Smart Array 6i Running RHEL 7.4 3.10.0-693.17.1.el7.x86_64 #1 SMP Sun Jan 14 10:36:03 EST 2018 x86_64 x86_64 x86_64 GNU/Linux with kmod-cciss-3.6.26-3.el7.elrepo.x86_64 and the RAID is perfectly recognized. But when upgrading to version 7.5, with kernel 3.10.0-862.el7.x86_64) 7.5 (Maipo) the disks are no longer recognized and the system does not start (kernels 3.10.0-862* ) | ||||
| Additional Information | [root@system ~]# uname -a Linux system 3.10.0-693.17.1.el7.x86_64 #1 SMP Sun Jan 14 10:36:03 EST 2018 x86_64 x86_64 x86_64 GNU/Linux [root@system ~]# rpm -q kernel | sort kernel-3.10.0-693.17.1.el7.x86_64 kernel-3.10.0-862.2.3.el7.x86_64 kernel-3.10.0-862.el7.x86_64 [root@system ~]# rpm -q kmod-cciss kmod-cciss-3.6.26-3.el7.elrepo.x86_64 [root@system ~]# ls -l $(find /lib/modules -name cciss.ko) | sort lrwxrwxrwx. 1 root root 55 abr 23 17:01 /lib/modules/3.10.0-862.el7.x86_64/weak-updates/cciss/cciss.ko -> /lib/modules/3.10.0-229.el7.x86_64/extra/cciss/cciss.ko lrwxrwxrwx. 1 root root 55 may 16 07:55 /lib/modules/3.10.0-693.17.1.el7.x86_64/weak-updates/cciss/cciss.ko -> /lib/modules/3.10.0-229.el7.x86_64/extra/cciss/cciss.ko lrwxrwxrwx. 1 root root 55 may 16 08:05 /lib/modules/3.10.0-862.2.3.el7.x86_64/weak-updates/cciss/cciss.ko -> /lib/modules/3.10.0-229.el7.x86_64/extra/cciss/cciss.ko -rw-r--r--. 1 root root 115528 mar 23 2015 /lib/modules/3.10.0-229.el7.x86_64/extra/cciss/cciss.ko -rw-r--r--. 1 root root 115528 mar 23 2015 /lib/modules/3.10.0-693.17.1.el7.x86_64/extra/cciss/cciss.ko -rw-r--r--. 1 root root 115528 mar 23 2015 /lib/modules/3.10.0-862.2.3.el7.x86_64/extra/cciss/cciss.ko -rw-r--r--. 1 root root 115528 mar 23 2015 /lib/modules/3.10.0-862.el7.x86_64/extra/cciss/cciss.ko | ||||
| Tags | No tags attached. | ||||
| Attached Files | |||||
| Reported upstream | |||||
|
|
[root@system ~]# dmidecode System Information Manufacturer: HP Product Name: ProLiant DL360 G4 [root@system log]# yum list installed | grep kmod-cciss kmod-cciss.x86_64 3.6.26-3.el7.elrepo @elrepo |
|
|
Same thing here on a DL580 G7. The error messages are a bit difficult to get out of the dracut console, but the pertinent part seems to be cmd 0x12 is reported invalid unable to obtain firmware version of controller cmd 0xc2 is reported invalid report logical volume command failed This repeats for each controller and is probably the root cause for the final dracut-initqueue: Warning /dev/disk/by-uuid/... does not exist which should have been the system volume. As for the OR, the previous kernel boots no prob. |
|
|
Acknowledged. We will most likely need to rebuild the module for el7.5. I'll post back here as soon as we have something for you to test. |
|
|
Sorry, this one is G5, not G7. (G7 does not need the cciss driver anymore.) |
|
|
I have built an updated package for el7.5 and uploaded it to the testing repository. Files are syncing to the mirror sites and should be available to test shortly: kmod-cciss-3.6.26-4.el7_5.elrepo.x86_64.rpm Please update with: yum --enablerepo=elrepo-testing update kmod-cciss and then reboot to an el7.5 kernel to test the new kernel module Please note that this module is likely NOT backward compatible with earlier kernels (e.g, el7.4), so if for any reason it does not work you will need to downgrade to the previous kmod-cciss release to work with your el7.4 kernel again. Please let me know if it works as expected. Thanks |
|
|
I install the new kernel module but it does not work. Get a kernel panic (I have attached a screenshot). Back to the 7.4 kernel, with this module still works. [root@system ~]# modinfo cciss filename: /lib/modules/3.10.0-693.17.1.el7.x86_64/extra/cciss/cciss.ko license: GPL version: 3.6.26 description: Driver for HP Smart Array Controllers author: Hewlett-Packard Company rhelversion: 7.1 srcversion: 83606FAD13AA19CCBDDBE63 alias: pci:v0000103Cd00003230sv0000103Csd0000323Dbc*sc*i* alias: pci:v0000103Cd00003230sv0000103Csd00003237bc*sc*i* alias: pci:v0000103Cd00003238sv0000103Csd00003215bc*sc*i* alias: pci:v0000103Cd00003238sv0000103Csd00003214bc*sc*i* alias: pci:v0000103Cd00003238sv0000103Csd00003213bc*sc*i* alias: pci:v0000103Cd00003238sv0000103Csd00003212bc*sc*i* alias: pci:v0000103Cd00003238sv0000103Csd00003211bc*sc*i* alias: pci:v0000103Cd00003230sv0000103Csd00003235bc*sc*i* alias: pci:v0000103Cd00003230sv0000103Csd00003234bc*sc*i* alias: pci:v0000103Cd00003230sv0000103Csd00003223bc*sc*i* alias: pci:v0000103Cd00003220sv0000103Csd00003225bc*sc*i* alias: pci:v00000E11d00000046sv00000E11sd0000409Dbc*sc*i* alias: pci:v00000E11d00000046sv00000E11sd0000409Cbc*sc*i* alias: pci:v00000E11d00000046sv00000E11sd0000409Bbc*sc*i* alias: pci:v00000E11d00000046sv00000E11sd0000409Abc*sc*i* alias: pci:v00000E11d00000046sv00000E11sd00004091bc*sc*i* alias: pci:v00000E11d0000B178sv00000E11sd00004083bc*sc*i* alias: pci:v00000E11d0000B178sv00000E11sd00004082bc*sc*i* alias: pci:v00000E11d0000B178sv00000E11sd00004080bc*sc*i* alias: pci:v00000E11d0000B060sv00000E11sd00004070bc*sc*i* depends: vermagic: 3.10.0-229.el7.x86_64 SMP mod_unload modversions signer: The ELRepo Project (http://elrepo.org): ELRepo.org Secure Boot Key sig_key: F3:65:AD:34:81:A7:B2:0E:34:27:B6:1B:2A:26:63:5B:83:FE:42:7B sig_hashalgo: sha256 parm: cciss_tape_cmds:number of commands to allocate for tape devices (default: 6) (int) parm: cciss_simple_mode:Use 'simple mode' rather than 'performant mode' (int) parm: cciss_allow_hpsa:Prevent cciss driver from accessing hardware known to be supported by the hpsa driver (int) |
|
|
If I'm not mistaken kmod-cciss-3.6.26 was backported from kernel-3.16.7. @pperry, do you think it now requires a backport from a newer kernel? |
|
|
The new module does work for me, though, and solves the problem as far as I am concerned. Proliant DL580 G5 is now booting fine again with the kernel from CentOS 7.5. $ uname -r 3.10.0-862.2.3.el7.x86_64 $ yum info kmod-cciss ... Installed Packages Name : kmod-cciss Arch : x86_64 Version : 3.6.26 Release : 4.el7_5.elrepo Size : 144 k Repo : installed From repo : elrepo-testing Summary : cciss kernel module(s) URL : http://www.kernel.org/ License : GPLv2 Description : This package provides the cciss kernel module(s) for HP Smart Array Controllers : built for the Linux kernel using the x86_64 family of processors. |
|
|
I would also point out that the screenshot of the kernel panic does not show anything specific to the cciss module. Could also be a corrupt kernel image or a corrupt initramfs. What was the original error with the old module for Soria ? |
|
|
I have uninstalled the kmod-cciss-3.6.26-4.el7_5.elrepo.x86_64.rpm, and installed the old kmod-cciss-3.6.26-3.el7.elrepo.x86_64.rpm After restarting the machine, and try again to boot with the kernel 3.10.0-862.2.3.el7.x86_64, I get the result that appears in the screenshots error01.jpg, error02.jpg and error03.jpg Boot with kernel-3.10.0-693.17.1.el7.x86_64 faultlessly, and update kmod-cciss to kmod-cciss-3.6.26-4.el7_5.elrepo.x86_64.rpm again. After a new reboot, the panic kernel appears again (screenshot error04.jpg) By the logs generated, it seems that it fails to attempt to mount a volume on ext4 filesystem?? Back to the previous situation, at least the new module kmod-cciss-3.6.26-4.el7_5.elrepo.x86_64 works, with the kernel-3.10.0-693.17.1.el7.x86_64 |
|
|
@robmaz made a good point. @soria, can you try reinstalling the -862 kernel? Looks as if the kernel update did not complete thus causing the panic. |
|
|
Ok @toracat, I will reinstall it next monday. |
|
|
It has not worked. Using kernel-3.10.0-693.17.1.el7.x86_64, I have delete all kernel and kernel-devel -862, and the test kmod-cciss-3.6.26-4.el7_5.elrepo.x86_64 Rebbot the server, and with 'yum update' updated both rpms. Sorry for the spanish logs: --- yum update Complementos cargados:aliases, changelog, langpacks, product-id, search-disabled-repos, subscription-manager, verify, versionlock Resolviendo dependencias --> Ejecutando prueba de transacción ---> Paquete kernel.x86_64 0:3.10.0-862.2.3.el7 debe ser instalado ---> Paquete kernel-devel.x86_64 0:3.10.0-862.2.3.el7 debe ser instalado --> Resolución de dependencias finalizada Dependencias resueltas ====================================================================================================== Package Arquitectura Versión Repositorio Tamaño ====================================================================================================== Instalando: kernel x86_64 3.10.0-862.2.3.el7 rhel-7-server-rpms 46 M kernel-devel x86_64 3.10.0-862.2.3.el7 rhel-7-server-rpms 16 M Resumen de la transacción ====================================================================================================== Instalar 2 Paquetes Tamaño total de la descarga: 62 M Tamaño instalado: 98 M Is this ok [y/d/N]: y Downloading packages: No Presto metadata available for rhel-7-server-rpms (1/2): kernel-devel-3.10.0-862.2.3.el7.x86_64.rpm | 16 MB 00:00:01 (2/2): kernel-3.10.0-862.2.3.el7.x86_64.rpm | 46 MB 00:00:03 ------------------------------------------------------------------------------------------------------ Total 17 MB/s | 62 MB 00:00:03 Running transaction check Running transaction test Transaction test succeeded Running transaction Instalando : kernel-3.10.0-862.2.3.el7.x86_64 1/2 Instalando : kernel-devel-3.10.0-862.2.3.el7.x86_64 2/2 Comprobando : kernel-devel-3.10.0-862.2.3.el7.x86_64 1/2 Comprobando : kernel-3.10.0-862.2.3.el7.x86_64 2/2 Instalado: kernel.x86_64 0:3.10.0-862.2.3.el7 kernel-devel.x86_64 0:3.10.0-862.2.3.el7 ---- Install the test kmod-cciss ---- yum --enablerepo=elrepo-testing update kmod-cciss Complementos cargados:aliases, changelog, langpacks, product-id, search-disabled-repos, subscription-manager, verify, versionlock Resolviendo dependencias --> Ejecutando prueba de transacción ---> Paquete kmod-cciss.x86_64 0:3.6.26-3.el7.elrepo debe ser actualizado ---> Paquete kmod-cciss.x86_64 0:3.6.26-4.el7_5.elrepo debe ser una actualización --> Resolución de dependencias finalizada Dependencias resueltas ====================================================================================================== Package Arquitectura Versión Repositorio Tamaño ====================================================================================================== Actualizando: kmod-cciss x86_64 3.6.26-4.el7_5.elrepo elrepo-testing 51 k Resumen de la transacción ====================================================================================================== Actualizar 1 Paquete Tamaño total de la descarga: 51 k Is this ok [y/d/N]: y Downloading packages: No Presto metadata available for elrepo-testing kmod-cciss-3.6.26-4.el7_5.elrepo.x86_64.rpm | 51 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Actualizando : kmod-cciss-3.6.26-4.el7_5.elrepo.x86_64 1/2 Working. This may take some time ... Done. Limpieza : kmod-cciss-3.6.26-3.el7.elrepo.x86_64 2/2 Working. This may take some time ... Done. Comprobando : kmod-cciss-3.6.26-4.el7_5.elrepo.x86_64 1/2 Comprobando : kmod-cciss-3.6.26-3.el7.elrepo.x86_64 2/2 Actualizado: kmod-cciss.x86_64 0:3.6.26-4.el7_5.elrepo ¡Listo! ----- And when I reboot the server, kernel panic appears again when booting with the kernel 3.10.0-862.2.3.el7.x86_64 Now, the server is running with the kernel-3.10.0-693 and mod-cciss-3.6.26-4.el7_5.elrepo.x86_64 What else can I try? Thanks in advance. |
|
|
Hi Soria, At this point I'm really not sure what the issue is given the new module works on robmaz's DL580 G5. I will try and see if we can update the driver to a newer version for you to try to see if that helps. You seem to have eliminated the kernel image and/or initramfs as the source of the problem. Maybe the RHEL7.5 kernel just doesn't like the HP 360 G4, but I couldn't find any reports on google to corroborate that. I wonder if you could try booting the machine from either the el7.5 bootable install media or a 7.5 liveCD to see if the hardware (minus the raid array) will boot 7.5 at all. |
|
|
The raid controller is diferent in the G5s. I have a 360G5, and its raid controller is a P400i, running faultlessly RHEL 7.5 with 3.10.0-862.el7.x86_64 kernel, and does not neet the kmod-ciiss driver, only add "hpsa.hpsa_allow_any=1 hpsa.hpsa_simple_mode=1" in the vmlinuz grub2.cfg lines to recognize the raid controller. In the G4, On the other hand, the model is a Smart Array 6i Ok, I will download the 7.5 disc and try it. Thanks @pperry |
|
|
A clarification, only have a kernel panic with -862 kernel and the test kmod-ciiss driver. With same kernel -862 and kmod-cciss-3.6.26-3.el7 the system boot without the kernel panic, what happens is that it does not recognize the discs (screenshots error01.jpg, error02.jpg and error03.jpg) |
|
|
@ppery the initramfs was updated today, when reinstall the -862 kernel -rw-------. 1 root root 18M may 21 16:37 initramfs-3.10.0-862.2.3.el7.x86_64.img |
|
|
Definitely RedHat has made some change in their 7.5 kernels. When booting from the 7.5 x86_64 installation DVD, I also get a panic kernel (screenshot: error_install75.jpg) The initial graphic installation screen does not even appear! |
|
|
Hmm, that's not good. That panic looks very similar to the first screenshot you posted. I'm starting to think this is a RHEL issue rather than an issue with kmod-cciss. I've searched Red Hat's bugzilla but don't see anything helpful. |
|
|
I have updated the package to the 3.18.109 sources, simply because that branch is still maintained (even though it says EOL), but no driver updates have been made in tree since 2014 so I don't expect it will help here. But at least we can pick up any future updates. Updated packages have been released to the el7 testing repository and should be available shortly: kmod-cciss-3.6.26-5.el7_5.elrepo.x86_64.rpm |
|
|
Batch of tests done today: 1. Install new driver kmod-cciss-3.6.26-5.el7_5.elrepo.x86_64 * Is generated a kernel panic with kernel-3.10.0-862.2.3.el7.x86_64 * Runs OK with kernel-3.10.0-693.17.1.el7.x86_64 2. Update to a new RHEL kernel, version 3.10.0-862.3.2.el7.x86_64, and gets again the kernel panic. I return to the initial situation, old kernel 3.10.0-693.17.1.el7.x86_64, and the latest version of the cciss driver kmod-cciss-3.6.26-5.el7_5.elrepo.x86_64 running. |
|
|
Acknowledged, and thank you for the feedback and testing you've done. Unfortunately there's nothing more I can think that we can try / do. I would normally suggest you file a bug with Red Hat, but that becomes difficult when they don't support the cciss driver. You would need to show the kernel panic is not related to the RAID controller, maybe by showing showing the rhel7.5 install media panics on boot whereas an older rhel7.4 install media boots as expected. In fact I wouldn't even mention the RAID controller nor kmod-cciss as I don't believe it's related to the issue. |
|
|
As long as booting the install disk kernel panics, there is no point in even trying the cciss driver, which probably would work fine. You should always be able to get into the dracut console with no disk at least. It looks like something else in the kernel does not like your hardware. The problem is that RHEL 7, and thus CentOS 7, does not support the old Proliants, so they probably won't care. Maybe your hardware is somehow faulty? Have you tried the hardware check from the SPP? Can you boot some other modern Linux? |
|
|
> Can you boot some other modern Linux? Maybe it's worth trying booting kernel-ml or kernel-lt? |
|
|
I can also report that the new kmod-cciss-3.6.26-5.el7_5.elrepo.x86_64 works with the new 3.10.0-862.3.2.el7.x86_64 kernel for my DL580 G5. (And yes, this has the P400 and also a P800 controller). |
|
|
Packages have been promoted to the main elrepo repository based on robmaz's feedback. Thanks. |
|
|
One last suggestion - would you like to try our kernel-lt package to see if your machine will boot with that kernel? http://elrepo.org/linux/kernel/el7/x86_64/RPMS/ Our latest kernel-lt package is kernel-lt-4.4.132-1.el7.elrepo.x86_64.rpm These are packaged versions of vanilla upstream kernels, in the case of kernel-lt it is based upon the upstream LTS kernel-4.4.x and has the native cciss kernel driver enabled. Please note kmod packages are not compatible with our kernel-lt package. You can install with: yum --enablerepo=elrepo-kernel install kernel-lt and try rebooting to that kernel. If it doesn't work, just uninstall it afterwards. |
|
|
@toracat, try the kernel-lt and works! ]#rpm -qa | grep cciss kmod-cciss-3.6.26-5.el7_5.elrepo.x86_64 ]# uname -a Linux 4.4.132-1.el7.elrepo.x86_64 #1 SMP Wed May 16 12:14:27 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux I have only detected an error, and that is the kdump service fails at start with this kernel, Starting Crash recovery kernel arming... No kdump initial ramdisk found. Rebuilding /boot/initramfs-4.4.132-1.el7.elrepo.x86_64kdump.img Error: Can not save vmcore because device /dev/cciss/c0d0p2 is unresettable mkdumprd: failed to make kdump initrd Starting kdump: [FAILED] kdump.service: main process exited, code=exited, status=1/FAILURE Failed to start Crash recovery kernel arming. Unit kdump.service entered failed state. kdump.service failed. Is safe disable the kdump service? @robmaz, I think there is no problem with the server hardware, it works correctly until the 3.10.0-693.17.1.el7.x86_64 kernel. It is from the new version kernel 3.10.0-862 in RHEL 7.5 when it fails. |
|
|
@pperry, I don't have problem between kernel-lt and kmod-cciss, the server works correctly right now with both packages. I would prefer to keep the kmod-cciss package, so that the RHEL 7.4 kernel 3.10.0-693 works if necessary. |
|
|
@pperry, Ok, I see using now /lib/modules/4.4.132-1.el7.elrepo.x86_64/kernel/drivers/block/cciss.ko module. This driver works great. |
|
|
Looking at the RHEL 7.0 release notes at: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.0_release_notes/known-issues-kernel kernel component, BZ#975791 The following cciss controllers are no longer supported: Smart Array 6i, Smart Array P400 and Smart Array P800 are all listed there. So, it's no surprise if they stop working under the 7.x kernel, unfortunately. |
|
|
That is old news and the reason why the kmod-cciss package exists in the first place. |
|
|
Just updated my ProLiant DL365 G1 to CentOS 7.6 and it stopped finding the disk. Previously the grub options "hpsa.hpsa_simple_mode=1 hpsa.hpsa_allow_any=1" had worked, but they no longer do. kmod-cciss only supports 7.5, apparently. Any thoughts on how to make it work for 7.6, including the initramfs? [EDIT by admin] Corrected CentOS 7.5 -> CentOS 7.6 on the first line. |
|
|
sorry, that should have been 7.6. The failing kernel is 3.10.0-957. Reverting to 3.10.0-862.11.6 makes it find the p400i disks again. |
|
|
@krisvale I'm afraid there isn't much we can do. Can you try kernel-lt? |
|
|
Not sure what you mean by kernel-it. I'm not a kernel hacker, I just inherited a Proliant machine with Centos7. Anyway, internet search led me to the elrepo, with kmod-cciss apparently being the kernel module to enable P400i controller support (rather than the apparently temporary solutions of hpsa.hpsa_simple_simple mode, etc.) What I find is that this kernel module hasn't been updated for 7.6. Is it expected to work for 7.6? Is there anything I can do to enable it for 7.6, including the initial ram FS? CentOS7 is expected to stay alive for a few more years, so it's useful to know if such modules will keep getting updates. |
|
|
@krisvale Please see http://elrepo.org/tiki/kernel-lt . You can find some screenshots here: https://www.tecmint.com/install-upgrade-kernel-version-in-centos-7/ Just replace "kernel-ml" with "kernel-lt". |
|
|
Did anyone get a RHEL/CentOS 7.5 kernel working with the HP ProLiant DL360 G4 6i SCSI Controller? |
|
|
Here we have 360 G4 with Smart Array 6i on RHEL 7.9 and kernel-lt 4.4.x working correctly. But when updating to the 5.4 branch of kernel-lt, it stopped working, probably due to problems with the cciss driver, https://elrepo.org/bugs/view.php?id=1209 |
|
|
@soria... As RHEL 8.x is no longer supported, I'm more interested in the RHEL 7.x installation. Did you install RHEL 7.9 and kernel-lt 4.4.x using the grub or kmod-cciss method? Any installation notes related to installing RHEL 7.9 and kernel-lt 4.4.x on the DL360 G4 is greatly appreciated. Thank You! |
|
|
Hi @garycnew The initial version of RHEL on that machine was 7.2, and it has been progressively updated, until reaching the current 7.9 The installation of kernel-lt was when upgrading to version 7.5, since the official 3.10 kernel stopped supporting the cciss driver. You can see the tests that were done then, throughout this thread. So what I would do is install RHEL 7.4, install the latest -lt 4.4.x kernel you can find, and then keep upgrading RHEL until you reach 7.9. |
|
|
@soria... Excellent! That is precisely the type of direction and experience I was hoping to find, here. Thank you very much! –Gary |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2018-05-16 02:47 | soria | New Issue | |
| 2018-05-16 02:47 | soria | Status | new => assigned |
| 2018-05-16 02:47 | soria | Assigned To | => stindall |
| 2018-05-16 02:51 | soria | Note Added: 0005853 | |
| 2018-05-16 05:52 | burakkucat | Project | channel: kernel/el7 => channel: elrepo/el7 |
| 2018-05-16 05:55 | burakkucat | Assigned To | stindall => pperry |
| 2018-05-16 05:55 | burakkucat | Category | --kernel--OTHER-- => kmod-cciss |
| 2018-05-16 05:55 | burakkucat | Additional Information Updated | |
| 2018-05-16 10:09 | robmaz | Note Added: 0005854 | |
| 2018-05-16 15:54 | pperry | Note Added: 0005855 | |
| 2018-05-17 02:52 | robmaz | Note Added: 0005857 | |
| 2018-05-17 13:43 | pperry | Note Added: 0005859 | |
| 2018-05-18 00:23 | soria | File Added: kernel_error.jpg | |
| 2018-05-18 00:28 | soria | Note Added: 0005860 | |
| 2018-05-18 01:34 | toracat | Note Added: 0005861 | |
| 2018-05-18 03:27 | robmaz | Note Added: 0005862 | |
| 2018-05-18 03:37 | robmaz | Note Added: 0005863 | |
| 2018-05-18 05:28 | soria | File Added: error01.jpg | |
| 2018-05-18 05:29 | soria | File Added: error02.jpg | |
| 2018-05-18 05:29 | soria | File Added: error03.jpg | |
| 2018-05-18 05:29 | soria | File Added: error04.jpg | |
| 2018-05-18 05:44 | soria | Note Added: 0005864 | |
| 2018-05-18 06:41 | toracat | Note Added: 0005865 | |
| 2018-05-18 06:55 | soria | Note Added: 0005866 | |
| 2018-05-21 08:54 | soria | Note Added: 0005878 | |
| 2018-05-21 09:53 | pperry | Note Added: 0005879 | |
| 2018-05-21 10:45 | soria | Note Added: 0005880 | |
| 2018-05-21 10:53 | soria | Note Added: 0005881 | |
| 2018-05-21 11:03 | soria | Note Added: 0005882 | |
| 2018-05-21 11:32 | soria | File Added: error_install75.jpg | |
| 2018-05-21 11:36 | soria | Note Added: 0005883 | |
| 2018-05-21 13:19 | pperry | Note Added: 0005884 | |
| 2018-05-22 13:55 | pperry | Note Added: 0005885 | |
| 2018-05-23 03:42 | soria | Note Added: 0005886 | |
| 2018-05-23 06:00 | pperry | Note Added: 0005887 | |
| 2018-05-23 06:01 | pperry | Note Edited: 0005887 | |
| 2018-05-23 09:48 | toracat | Note Edited: 0005878 | |
| 2018-05-23 09:49 | robmaz | Note Added: 0005888 | |
| 2018-05-23 10:10 | toracat | Note Added: 0005889 | |
| 2018-05-23 10:43 | robmaz | Note Added: 0005890 | |
| 2018-05-23 14:26 | pperry | Note Added: 0005891 | |
| 2018-05-24 03:15 | pperry | Note Added: 0005895 | |
| 2018-05-24 03:35 | soria | Note Added: 0005896 | |
| 2018-05-24 04:00 | soria | Note Added: 0005897 | |
| 2018-05-24 04:05 | soria | Note Added: 0005898 | |
| 2018-05-24 11:15 | toracat | Note Added: 0005901 | |
| 2018-05-25 02:35 | robmaz | Note Added: 0005902 | |
| 2018-12-27 08:25 | krisvale | Note Added: 0006080 | |
| 2018-12-27 09:13 | krisvale | Note Added: 0006081 | |
| 2018-12-28 18:08 | toracat | Note Edited: 0006080 | |
| 2018-12-28 18:13 | toracat | Note Added: 0006095 | |
| 2019-01-02 04:29 | krisvale | Note Added: 0006096 | |
| 2019-01-02 14:51 | toracat | Note Added: 0006099 | |
| 2022-03-06 20:24 | garycnew | Note Added: 0008244 | |
| 2022-03-11 03:27 | soria | Note Added: 0008255 | |
| 2022-03-17 02:51 | garycnew | Note Added: 0008262 | |
| 2022-03-17 06:04 | soria | Note Added: 0008263 | |
| 2022-03-18 01:52 | garycnew | Note Added: 0008264 |
