View Issue Details

IDProjectCategoryView StatusLast Update
0000765channel: elrepo/el7kmod-nvidiapublic2017-08-14 12:33
Reportervychytraly Assigned Topperry  
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
Summary0000765: OpenCL does not work with new kmod-nvidia 384xx drivers
DescriptionHello friends, today I was trying darktable with new kmod-nvidia drivers and noticed that Use OpenCL flag in settings is greyed out. So I tried running darktable with debug opencl command (darktable -d opencl) and it showed this error:

[opencl_init] opencl related configuration options:
[opencl_init]
[opencl_init] opencl: 1
[opencl_init] opencl_library: ''
[opencl_init] opencl_memory_requirement: 768
[opencl_init] opencl_memory_headroom: 300
[opencl_init] opencl_device_priority: '*/!0,*/*/*'
[opencl_init] opencl_size_roundup: 16
[opencl_init] opencl_async_pixelpipe: 0
[opencl_init] opencl_synch_cache: 0
[opencl_init] opencl_number_event_handles: 25
[opencl_init] opencl_micro_nap: 1000
[opencl_init] opencl_use_pinned_memory: 0
[opencl_init] opencl_use_cpu_devices: 0
[opencl_init] opencl_avoid_atomics: 0
[opencl_init] opencl_omit_whitebalance: 0
[opencl_init]
[opencl_init] found opencl runtime library 'libOpenCL'
[opencl_init] opencl library 'libOpenCL' found on your system and loaded
[opencl_init] found 2 platforms
[opencl_init] found 3 devices
[opencl_init] device 0 `GeForce GTX 960M' has sm_20 support.
[opencl_init] device 0 `GeForce GTX 960M' supports image sizes of 16384 x 16384
[opencl_init] device 0 `GeForce GTX 960M' allows GPU memory allocations of up to 1011MB
[opencl_init] device 0: GeForce GTX 960M
     GLOBAL_MEM_SIZE: 4044MB
     MAX_WORK_GROUP_SIZE: 1024
     MAX_WORK_ITEM_DIMENSIONS: 3
     MAX_WORK_ITEM_SIZES: [ 1024 1024 64 ]
     DRIVER_VERSION: 384.59
     DEVICE_VERSION: OpenCL 1.2 CUDA
[opencl_init] could not create context for device 0: 221
[opencl_init] FINALLY: opencl is NOT AVAILABLE on this system.
[opencl_init] initial status of opencl enabled flag is OFF.

Then I tried running oclDeviceQuery and it shows this error:

[oclDeviceQuery] starting...

./oclDeviceQuery Starting...

OpenCL SW Info:

 CL_PLATFORM_NAME: NVIDIA CUDA
 CL_PLATFORM_VERSION: OpenCL 1.2 CUDA 9.0.130
 OpenCL SDK Revision: 7027912


OpenCL Device Info:

 1 devices found supporting OpenCL:

Error 221 in clCreateContext call !!!

oclDeviceQuery, Platform Name = NVIDIA CUDA, Platform Version = OpenCL 1.2 CUDA 9.0.130, SDK Revision = 7027912, NumDevs = 1
...
Linux version 3.10.0-514.26.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Jul 4 15:04:05 UTC 2017


[oclDeviceQuery] test results...
FAILED

> exiting in 3 seconds: 3...2...1...done!



So as Phil recommended I tried to downgrade kmod-nvidia (to previous one - 375.66) and got these results:
darktable -d opencl:

[opencl_init] found opencl runtime library 'libOpenCL'
[opencl_init] opencl library 'libOpenCL' found on your system and loaded
[opencl_init] found 2 platforms
[opencl_init] found 3 devices
[opencl_init] device 0 `GeForce GTX 960M' has sm_20 support.
[opencl_init] device 0 `GeForce GTX 960M' supports image sizes of 16384 x 16384
[opencl_init] device 0 `GeForce GTX 960M' allows GPU memory allocations of up to 1011MB
[opencl_init] device 0: GeForce GTX 960M
     GLOBAL_MEM_SIZE: 4044MB
     MAX_WORK_GROUP_SIZE: 1024
     MAX_WORK_ITEM_DIMENSIONS: 3
     MAX_WORK_ITEM_SIZES: [ 1024 1024 64 ]
     DRIVER_VERSION: 375.66
     DEVICE_VERSION: OpenCL 1.2 CUDA
...
[opencl_init] kernel loading time: 0.0044
[opencl_init] discarding CPU device 2 `Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz'.
[opencl_init] OpenCL successfully initialized.
[opencl_init] here are the internal numbers and names of OpenCL devices available to darktable:
[opencl_init] 0 'GeForce GTX 960M'
[opencl_init] 1 'Intel(R) HD Graphics'
[opencl_init] these are your device priorities:
[opencl_init] image preview export thumbnail
[opencl_init] 0 1 0 0
[opencl_init] 1 -1 1 1
[opencl_init] FINALLY: opencl is AVAILABLE on this system.
[opencl_init] initial status of opencl enabled flag is ON.

Also oclDeviceQuery runs fine with older drivers:

[oclDeviceQuery] starting...

./oclDeviceQuery Starting...

OpenCL SW Info:

 CL_PLATFORM_NAME: NVIDIA CUDA
 CL_PLATFORM_VERSION: OpenCL 1.2 CUDA 8.0.0
 OpenCL SDK Revision: 7027912


OpenCL Device Info:

 1 devices found supporting OpenCL:

 ---------------------------------
 Device GeForce GTX 960M
 ---------------------------------
  CL_DEVICE_NAME: GeForce GTX 960M
  CL_DEVICE_VENDOR: NVIDIA Corporation
  CL_DRIVER_VERSION: 375.66
  CL_DEVICE_VERSION: OpenCL 1.2 CUDA
  CL_DEVICE_OPENCL_C_VERSION: OpenCL C 1.2
  CL_DEVICE_TYPE: CL_DEVICE_TYPE_GPU
  CL_DEVICE_MAX_COMPUTE_UNITS: 5
  CL_DEVICE_MAX_WORK_ITEM_DIMENSIONS: 3
  CL_DEVICE_MAX_WORK_ITEM_SIZES: 1024 / 1024 / 64
  CL_DEVICE_MAX_WORK_GROUP_SIZE: 1024
  CL_DEVICE_MAX_CLOCK_FREQUENCY: 1176 MHz
  CL_DEVICE_ADDRESS_BITS: 64
  CL_DEVICE_MAX_MEM_ALLOC_SIZE: 1011 MByte
  CL_DEVICE_GLOBAL_MEM_SIZE: 4044 MByte
  CL_DEVICE_ERROR_CORRECTION_SUPPORT: no
  CL_DEVICE_LOCAL_MEM_TYPE: local
  CL_DEVICE_LOCAL_MEM_SIZE: 48 KByte
  CL_DEVICE_MAX_CONSTANT_BUFFER_SIZE: 64 KByte
  CL_DEVICE_QUEUE_PROPERTIES: CL_QUEUE_OUT_OF_ORDER_EXEC_MODE_ENABLE
  CL_DEVICE_QUEUE_PROPERTIES: CL_QUEUE_PROFILING_ENABLE
  CL_DEVICE_IMAGE_SUPPORT: 1
  CL_DEVICE_MAX_READ_IMAGE_ARGS: 256
  CL_DEVICE_MAX_WRITE_IMAGE_ARGS: 16
  CL_DEVICE_SINGLE_FP_CONFIG: denorms INF-quietNaNs round-to-nearest round-to-zero round-to-inf fma

  CL_DEVICE_IMAGE <dim> 2D_MAX_WIDTH 16384
                    2D_MAX_HEIGHT 16384
                    3D_MAX_WIDTH 4096
                    3D_MAX_HEIGHT 4096
                    3D_MAX_DEPTH 4096

  CL_DEVICE_EXTENSIONS: cl_khr_global_int32_base_atomics
                    cl_khr_global_int32_extended_atomics
                    cl_khr_local_int32_base_atomics
                    cl_khr_local_int32_extended_atomics
                    cl_khr_fp64
                    cl_khr_byte_addressable_store
                    cl_khr_icd
                    cl_khr_gl_sharing
                    cl_nv_compiler_options
                    cl_nv_device_attribute_query
                    cl_nv_pragma_unroll
                    cl_nv_copy_opts


  CL_DEVICE_COMPUTE_CAPABILITY_NV: 5.0
  NUMBER OF MULTIPROCESSORS: 5
MapSMtoCores SM 5.0 is undefined (please update to the latest SDK)!
  NUMBER OF CUDA CORES: 4294967291
  CL_DEVICE_REGISTERS_PER_BLOCK_NV: 65536
  CL_DEVICE_WARP_SIZE_NV: 32
  CL_DEVICE_GPU_OVERLAP_NV: CL_TRUE
  CL_DEVICE_KERNEL_EXEC_TIMEOUT_NV: CL_TRUE
  CL_DEVICE_INTEGRATED_MEMORY_NV: CL_FALSE
  CL_DEVICE_PREFERRED_VECTOR_WIDTH_<t> CHAR 1, SHORT 1, INT 1, LONG 1, FLOAT 1, DOUBLE 1


  ---------------------------------
  2D Image Formats Supported (75)
  ---------------------------------
  # Channel Order Channel Type

  1 CL_R CL_FLOAT
  2 CL_R CL_HALF_FLOAT
  3 CL_R CL_UNORM_INT8
  4 CL_R CL_UNORM_INT16
  5 CL_R CL_SNORM_INT16
  6 CL_R CL_SIGNED_INT8
  7 CL_R CL_SIGNED_INT16
  8 CL_R CL_SIGNED_INT32
  9 CL_R CL_UNSIGNED_INT8
  10 CL_R CL_UNSIGNED_INT16
  11 CL_R CL_UNSIGNED_INT32
  12 CL_A CL_FLOAT
  13 CL_A CL_HALF_FLOAT
  14 CL_A CL_UNORM_INT8
  15 CL_A CL_UNORM_INT16
  16 CL_A CL_SNORM_INT16
  17 CL_A CL_SIGNED_INT8
  18 CL_A CL_SIGNED_INT16
  19 CL_A CL_SIGNED_INT32
  20 CL_A CL_UNSIGNED_INT8
  21 CL_A CL_UNSIGNED_INT16
  22 CL_A CL_UNSIGNED_INT32
  23 CL_RG CL_FLOAT
  24 CL_RG CL_HALF_FLOAT
  25 CL_RG CL_UNORM_INT8
  26 CL_RG CL_UNORM_INT16
  27 CL_RG CL_SNORM_INT16
  28 CL_RG CL_SIGNED_INT8
  29 CL_RG CL_SIGNED_INT16
  30 CL_RG CL_SIGNED_INT32
  31 CL_RG CL_UNSIGNED_INT8
  32 CL_RG CL_UNSIGNED_INT16
  33 CL_RG CL_UNSIGNED_INT32
  34 CL_RA CL_FLOAT
  35 CL_RA CL_HALF_FLOAT
  36 CL_RA CL_UNORM_INT8
  37 CL_RA CL_UNORM_INT16
  38 CL_RA CL_SNORM_INT16
  39 CL_RA CL_SIGNED_INT8
  40 CL_RA CL_SIGNED_INT16
  41 CL_RA CL_SIGNED_INT32
  42 CL_RA CL_UNSIGNED_INT8
  43 CL_RA CL_UNSIGNED_INT16
  44 CL_RA CL_UNSIGNED_INT32
  45 CL_RGBA CL_FLOAT
  46 CL_RGBA CL_HALF_FLOAT
  47 CL_RGBA CL_UNORM_INT8
  48 CL_RGBA CL_UNORM_INT16
  49 CL_RGBA CL_SNORM_INT16
  50 CL_RGBA CL_SIGNED_INT8
  51 CL_RGBA CL_SIGNED_INT16
  52 CL_RGBA CL_SIGNED_INT32
  53 CL_RGBA CL_UNSIGNED_INT8
  54 CL_RGBA CL_UNSIGNED_INT16
  55 CL_RGBA CL_UNSIGNED_INT32
  56 CL_BGRA CL_UNORM_INT8
  57 CL_BGRA CL_SIGNED_INT8
  58 CL_BGRA CL_UNSIGNED_INT8
  59 CL_ARGB CL_UNORM_INT8
  60 CL_ARGB CL_SIGNED_INT8
  61 CL_ARGB CL_UNSIGNED_INT8
  62 CL_INTENSITY CL_FLOAT
  63 CL_INTENSITY CL_HALF_FLOAT
  64 CL_INTENSITY CL_UNORM_INT8
  65 CL_INTENSITY CL_UNORM_INT16
  66 CL_INTENSITY CL_SNORM_INT16
  67 CL_LUMINANCE CL_FLOAT
  68 CL_LUMINANCE CL_HALF_FLOAT
  69 CL_LUMINANCE CL_UNORM_INT8
  70 CL_LUMINANCE CL_UNORM_INT16
  71 CL_LUMINANCE CL_SNORM_INT16
  72 CL_BGRA CL_SNORM_INT8
  73 CL_BGRA CL_SNORM_INT16
  74 CL_ARGB CL_SNORM_INT8
  75 CL_ARGB CL_SNORM_INT16

  ---------------------------------
  3D Image Formats Supported (75)
  ---------------------------------
  # Channel Order Channel Type

  1 CL_R CL_FLOAT
  2 CL_R CL_HALF_FLOAT
  3 CL_R CL_UNORM_INT8
  4 CL_R CL_UNORM_INT16
  5 CL_R CL_SNORM_INT16
  6 CL_R CL_SIGNED_INT8
  7 CL_R CL_SIGNED_INT16
  8 CL_R CL_SIGNED_INT32
  9 CL_R CL_UNSIGNED_INT8
  10 CL_R CL_UNSIGNED_INT16
  11 CL_R CL_UNSIGNED_INT32
  12 CL_A CL_FLOAT
  13 CL_A CL_HALF_FLOAT
  14 CL_A CL_UNORM_INT8
  15 CL_A CL_UNORM_INT16
  16 CL_A CL_SNORM_INT16
  17 CL_A CL_SIGNED_INT8
  18 CL_A CL_SIGNED_INT16
  19 CL_A CL_SIGNED_INT32
  20 CL_A CL_UNSIGNED_INT8
  21 CL_A CL_UNSIGNED_INT16
  22 CL_A CL_UNSIGNED_INT32
  23 CL_RG CL_FLOAT
  24 CL_RG CL_HALF_FLOAT
  25 CL_RG CL_UNORM_INT8
  26 CL_RG CL_UNORM_INT16
  27 CL_RG CL_SNORM_INT16
  28 CL_RG CL_SIGNED_INT8
  29 CL_RG CL_SIGNED_INT16
  30 CL_RG CL_SIGNED_INT32
  31 CL_RG CL_UNSIGNED_INT8
  32 CL_RG CL_UNSIGNED_INT16
  33 CL_RG CL_UNSIGNED_INT32
  34 CL_RA CL_FLOAT
  35 CL_RA CL_HALF_FLOAT
  36 CL_RA CL_UNORM_INT8
  37 CL_RA CL_UNORM_INT16
  38 CL_RA CL_SNORM_INT16
  39 CL_RA CL_SIGNED_INT8
  40 CL_RA CL_SIGNED_INT16
  41 CL_RA CL_SIGNED_INT32
  42 CL_RA CL_UNSIGNED_INT8
  43 CL_RA CL_UNSIGNED_INT16
  44 CL_RA CL_UNSIGNED_INT32
  45 CL_RGBA CL_FLOAT
  46 CL_RGBA CL_HALF_FLOAT
  47 CL_RGBA CL_UNORM_INT8
  48 CL_RGBA CL_UNORM_INT16
  49 CL_RGBA CL_SNORM_INT16
  50 CL_RGBA CL_SIGNED_INT8
  51 CL_RGBA CL_SIGNED_INT16
  52 CL_RGBA CL_SIGNED_INT32
  53 CL_RGBA CL_UNSIGNED_INT8
  54 CL_RGBA CL_UNSIGNED_INT16
  55 CL_RGBA CL_UNSIGNED_INT32
  56 CL_BGRA CL_UNORM_INT8
  57 CL_BGRA CL_SIGNED_INT8
  58 CL_BGRA CL_UNSIGNED_INT8
  59 CL_ARGB CL_UNORM_INT8
  60 CL_ARGB CL_SIGNED_INT8
  61 CL_ARGB CL_UNSIGNED_INT8
  62 CL_INTENSITY CL_FLOAT
  63 CL_INTENSITY CL_HALF_FLOAT
  64 CL_INTENSITY CL_UNORM_INT8
  65 CL_INTENSITY CL_UNORM_INT16
  66 CL_INTENSITY CL_SNORM_INT16
  67 CL_LUMINANCE CL_FLOAT
  68 CL_LUMINANCE CL_HALF_FLOAT
  69 CL_LUMINANCE CL_UNORM_INT8
  70 CL_LUMINANCE CL_UNORM_INT16
  71 CL_LUMINANCE CL_SNORM_INT16
  72 CL_BGRA CL_SNORM_INT8
  73 CL_BGRA CL_SNORM_INT16
  74 CL_ARGB CL_SNORM_INT8
  75 CL_ARGB CL_SNORM_INT16

oclDeviceQuery, Platform Name = NVIDIA CUDA, Platform Version = OpenCL 1.2 CUDA 8.0.0, SDK Revision = 7027912, NumDevs = 1, Device = GeForce GTX 960M
...
Linux version 3.10.0-514.26.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Jul 4 15:04:05 UTC 2017


[oclDeviceQuery] test results...
PASSED

> exiting in 3 seconds: 3...2...1...done!



This leads me to conclusion that new nvidia drivers have probably some problem with OpenCL. If you need any more information or testing please let me know and I will be happy to provide it.

P.S.: in both cases I had opencl-filesystem opencl-headers and ocl-icd installed

P.S.2: Here is one thread I found where debian users had similar problems (but they managed to solve it - with a summary of the changes between their fixed version and the previous one attached on the bottom of the page), maybe it could be helpful: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839193

Have a nice day! :)
TagsNo tags attached.
Reported upstream

Activities

pperry

2017-07-30 15:33

administrator   ~0005367

I really have no idea so I'd suggest you look in the NVIDIA / OpenCL forums for clues.

The debian bug you cite only really mentions libnvidia-ptxjitcompiler, which is installed by our package.

One other thing worth trying is to uninstall out drivers and install 384.59 with the NVIDIA installer .run package and see if that fixes it - that would at least determine if it is a packaging issue introduce by elrepo, or if the nvidia drivers are broken.

vychytraly

2017-07-31 14:18

reporter   ~0005368

Ok I will try installing NVidia run drivers, but they tend to break my Intel GPU drivers and that renders my system unusable, kmod-nvidia are much kinder to Intel GPU drivers :) Please can you tell me how do you do it with kmod-nvidia (and how should I install it) without breaking Intel GPU drivers? :)

pperry

2017-07-31 15:45

administrator   ~0005369

I'm only suggesting a quick install to see if they work.

You would need to uninstall the elrepo nvidia drivers ('yum erase kmod-nvidia' and dependencies), reboot, then as root run the nvidia .run package, and reboot.

To uninstall the nvidia package, as root do 'nvidia-installer --uninstall', then reboot and reinstall the elrepo drivers.

Is that feasible for you? If the nvidia drivers remain broken from their install then we know you need to file a bug upstream with nvidia. However, if the nvidia drivers work as expected then we know the issue is with elrepo's packaging of the nvidia drivers.

vychytraly

2017-08-01 05:16

reporter   ~0005370

Yesterday I was thinking a lot how could I test it in the best possible way, but since laptop Im using cant switch off Intel GPU I can access NVidia only with bumblebee. And Im not sure if bumblebee will be working fine with nvidia .run drivers. I cant run software on NVidia card on this laptop any other way. But I have available one older desktop with NVidia card (Even it is older card it can use the new drivers) so Ill try to test the .run drivers and kmod-nvidia there and report how it works. But first I have to get some harddrive and install Centos there so it will probably take me few days to get it all working. Anyway Ill try to do it fast and report my findings as soon as possible. :)

vychytraly

2017-08-07 14:15

reporter   ~0005380

Phil it seems that VirtualBox is not the best option for testing GPU drivers so I ordered a new harddrive for installing Centos on the desktop PC I mentioned and will be testing new drivers this weekend. I will report my findings then. Sorry for being late.

vychytraly

2017-08-13 08:33

reporter   ~0005383

I just installed fresh EL7.3 on desktop with only nvidia GPU, installed kmod-nvidia package and opencl seems running fine. So it seems that it is not problem of new driver or elrepo package. I will investigate further what could be the cause.

pperry

2017-08-13 10:05

administrator   ~0005384

Thanks for reporting back :-)

vychytraly

2017-08-13 14:30

reporter   ~0005385

Phil I fixed it! :) I uninstalled kmod-nvidia, bumblebee and primus, deleted all remaining nvidia folders.

I noticed one thing - in folder /lib64/nvidia there was still remaining one file (I am not sure if it was libnvidia-ptxjitcompiler.so.1 or libnvidia-ptxjitcompiler.so.384.59) - but anyway ptxjitcompiler was the file which was doing problems on debian forums wasnt it? So I also manually removed it together with whole nvidia directory.

Then I did clean install of kmod-nvidia, bumblebee and primus, configured bumblebee as always and now everything is working fine, even OpenCL is working fine as before.

So my conclusion is that current kmod-nvidia package is fine, also nvidia drivers are fine. But probably there was some problem during updating from kmod-nvidia 37* to 38* drivers, maybe the ptxjitcompiler was not updated and it was uncompatible with new drivers then? Is it possible?

pperry

2017-08-13 16:34

administrator   ~0005389

Last edited: 2017-08-13 16:36

Glad you fixed it.

Yes, you've spotted a minor bug, there was a missing symlink for libnvidia-ptxjitcompiler.so.1 which I've corrected for the next release so there won't be orphaned symlinks left behind upon uninstall.

As to what fixed the issue, I've still no real idea, but clearly an uninstall and fresh reinstall has fixed it for you :-)

Thanks for the feedback.

vychytraly

2017-08-14 02:28

reporter   ~0005392

Thanks for assistance :) So I think that we could mark this thread as resolved :)

Issue History

Date Modified Username Field Change
2017-07-30 12:11 vychytraly New Issue
2017-07-30 12:11 vychytraly Status new => assigned
2017-07-30 12:11 vychytraly Assigned To => pperry
2017-07-30 15:33 pperry Note Added: 0005367
2017-07-31 14:18 vychytraly Note Added: 0005368
2017-07-31 15:45 pperry Note Added: 0005369
2017-08-01 05:16 vychytraly Note Added: 0005370
2017-08-07 14:15 vychytraly Note Added: 0005380
2017-08-13 08:33 vychytraly Note Added: 0005383
2017-08-13 10:05 pperry Note Added: 0005384
2017-08-13 14:30 vychytraly Note Added: 0005385
2017-08-13 16:34 pperry Note Added: 0005389
2017-08-13 16:36 pperry Note Edited: 0005389
2017-08-14 02:28 vychytraly Note Added: 0005392
2017-08-14 12:33 pperry Status assigned => resolved
2017-08-14 12:33 pperry Resolution open => fixed