Cuda runtime error cuda driver version is insufficient for cuda runtime version

Hi

I encountered this error: Check failed: error == cudaSuccess (35 vs. 0) CUDA driver version is insufficient for CUDA runtime version during my app processing.

Machine details:
GCE with GPU Tesla K80
OS: Ubuntu 14.04
The Driver installed is: NVIDIA-Linux-x86_64-375.66.run with nvidia-docker nvidia-docker_1.0.1.

Im using the following docker image, based on nvidia/cuda:7.0-cudnn4-devel-ubuntu14.04NVIDIA official Docker image
kaixhin/cuda-caffe:7.0

On top of that I compiled OpenCV 3.0.0

$nvidia-smi / run --rm nvidia/cuda:7.0-runtime nvidia-smi

Thu Jun 15 19:24:53 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 0000:00:04.0     Off |                    0 |
| N/A   55C    P8    31W / 149W |      0MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

$ldconfig -p | grep cuda

	libicudata.so.52 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libicudata.so.52
	libcuda.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcuda.so.1
	libcuda.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcuda.so

$nvidia-docker run --rm device

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Tesla K80"
  CUDA Driver Version / Runtime Version          8.0 / 8.0
  CUDA Capability Major/Minor version number:    3.7
  Total amount of global memory:                 11440 MBytes (11995578368 bytes)
  (13) Multiprocessors, (192) CUDA Cores/MP:     2496 CUDA Cores
  GPU Max Clock rate:                            824 MHz (0.82 GHz)
  Memory Clock rate:                             2505 Mhz
  Memory Bus Width:                              384-bit
  L2 Cache Size:                                 1572864 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Enabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 0 / 4
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = Tesla K80
Result = PASS

Can you pls help me solve this issue?
D.

Status: CUDA driver version is insufficient for CUDA runtime version #21832

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Kernel: 2.6.32-573.12.1.el6.x86_64
Host: RHEL 6.7
Container: Ubuntu 16.04.5 LTS

TensorFlow installed from (source or binary):
Singularity

TensorFlow version (use command below):
Tensorflow:1.10.0-devel-gpu-py3

Python version:
Python 3.5.2

GCC/Compiler version (if compiling from source):
GCC 5.4.0

CUDA/cuDNN version:
9

GPU model and memory:
Singularity tensorflow:1.10.0-devel-gpu-py3:

Exact command to reproduce:
$ # install nvidia driver v352.39
$ sudo singularity build —sandbox /path/to/sandbox docker://tensorflow/tensorflow/1.10.0-devel-gpu-py3
$ singularity shell -nv /path/to/sandbox
Singularity tensorflow:1.10.0-devel-gpu-py3:

> python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type «help», «copyright», «credits» or «license» for more information.

Describe the problem

I built a tensorflow container with singularity. I think there might be a mismatch between the some of the card drivers and cuda libraries between the host and container. I have the container built as a sandbox so I’m able to make modifications quiet easily, I was curious if there’s a way I can install appropriate cuda driver and runtimes to the container, and have the container run off those instead of pulling libraries from the host which are incompatible with the container? Is this the right way to do it? Or should I be updating the cuda drivers / libraries on the host to match the container?

The text was updated successfully, but these errors were encountered:

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
Bazel version
Mobile device

Have I written custom code
N/A
Bazel version
N/A
Mobile device
N/A

Would https://github.com/NIH-HPC/gpu4singularity be viable for Singularity 2.6.0 with —nv flags or would I need to make additional modification to library paths?

This is not a tensorflow issue: according to https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html your nvidia driver is not new enough for cuda 9.0

Sure. But the question is more on how to integrate compatible drivers into a tensorflow container. The adage about containerization is: build once, run anywhere; and not: build once, run anywhere with Nvidia drivers v485 and above plus a kernel supporting experimental filesystem overlays. Even experimental / unofficial documentation on this scenario would be extremely helpful for most HPC environments that are still running epel6. ¯_(ツ)_/¯

The world is not perfect. I’m afraid «build once, run anywhere with nvidia drivers>=384.81» is the way to go. At least that’s what nvidia says: https://github.com/NVIDIA/nvidia-docker/wiki/CUDA#requirements

Running a CUDA container requires a machine with at least one CUDA-capable GPU and a driver compatible with the CUDA toolkit version you are using.

I hit exactly this problem and someone else with the same combination ( tensorflow 1.11 + CUDA runtime 9.0 + cudnn 7.3 + nvidia driver 390 ) hit this problem too, though nvidia driver 390 is new enough for CUDA runtime 9.0 . This person opened an issue in the Nvidia DevTalk:

And I downgraded the tensorflow version from 1.11 (the latest conda version) to 1.7 and this problem got solved. And my question is if the newer tensorflow, say 1.10+ , has a dependency on specific nvidia drivers /cuda versions?

We upgraded to a recent version of drivers 396 and the issue resolved.

@mforde84 Thanks for the confirmation. That’s what I was thinking too, but I had trouble upgrading to 396.54 due to a broken dependency, however, after having read your confirmation, I managed to install 396.54 and now it works with tensorflow 1.11.0 , Yoho! Thanks! Upgraded the ticket in the Nvidia DevTalk.

tensorflow 1.11 + CUDA runtime 9.0 + cudnn 7.3 + nvidia driver 390
the combo should have worked. Note with 396.54 there will be one more upgrade once TF switches to CUDA 10.

@nicolefinnie , thanks, I downgraded the tensorflow version fromt o 1.7 and this problem got solved.

I tested the recommendations in this thread, but I was not able to install any other driver than 390 on Ubuntu 18.04 and downgrading tensorflow to 1.7 resulted in a new error message:

Which is strange, as I had installed version 7.3.1 on my system, but it seems that anaconda installs its own cudnn in the enviroment.

I tested the recommendations in this thread, but I was not able to install any other driver than 390 on Ubuntu 18.04 and downgrading tensorflow to 1.7 resulted in a new error message:

Which is strange, as I had installed version 7.3.1.

@saskra ,I was use deepin15.8, nvidia-driver==390.67, cuda==9.0,cudnn==7.0, and miniconda installed tensorflow-gpu==1.7,and the problem got solved.

Saskra are you running in a container?

No. But I now found the solution: Anaconda creates an environment with its own incompatible cudnn version which has to be overwritten manually. 🙂

No. But I now found the solution: Anaconda creates an environment with its own incompatible cudnn version which has to be overwritten manually. 🙂

I have the same problem. 🙁
Which version of which exact conda module did you have to use to overwrite?

I have Ubuntu 18.04 which needs Nvidia driver 390. Anaconda brings cuDNN 7.2.1, which seems to be too old for this driver version: https://anaconda.org/anaconda/cudnn Now I am using the newest cuDNN version (7.3.1) as suggested by the official download site: https://developer.nvidia.com/rdp/cudnn-download btw: Anaconda’s cuDNN version depends on its TensorFlow version, I have the newest one here as well (1.11).

PS: I suggested to update the version: ContinuumIO/anaconda-issues#10224

@mforde84 Would you mind sharing how you upgraded it?

check whether your nvidia-driver support your cuda version from here https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html

@mforde84 Would you mind sharing how you upgraded it?

As for me, upgrading my driver worked out. I run a Windows 10 PC and use TF 1.13.
( NOTE: _Just an aside, I needed to activate my virtual environment and start Jupyter notebook in that env before I was able to use TF in the notebook.)

Here is how I upgraded my driver:

1. Open Device Manager
2. Expand the display adapters
3. Locate your NVIDIA Graphics adapter
4. Right-click and click Update driver

Alternative

• I found this software ( GeForce Experience ) on the NVIDIA website for my graphics family which can also be downloaded, installed and used to update the driver(s). This should work as well, though I didn’t go that way.

Same issue here and I can’t find an appropriate tensorflow version. I currently have ubuntu version 16.04.6 , driver version 410.78 , cuda version 10 , conda version 4.7.11 and none of the above-mentioned tensorflow versions works for me. I tried 1.13.1 , 1.7 and 1.14 .
Anaconda installs cudnn with version 7.6.0 . Edit: I forced conda to use the version 10.0 for cudatoolkit and not cuda10.1_0 as it was before (according to @saskra’s suggestion), but nothing changed unfortunately.

Updating anaconda also didn’t help. In fact, conda update —all and conda update conda outputs many new errors like:
InvalidArchiveError(‘Error with archive . You probably need to delete and re- download or re-create this file. Message from libarchive was.

Creating a conda environment with my current specs or simply running my python script also produces various InvalidArchiveError messages like above:

I had a similar issue using driver 384.130. Turns out that versions of the cudatoolkit inside anaconda environment and the cuda supported by my driver did not match.

These two links helped me identifying my driver and cuda version and, later, to install the correct version of tensorflow_gpu that matched the cuda in my machine

The cuda versions may have minor-versions (9.0, 9.2), thus you should double check what exactly you are installing with conda.
To check what you have inside your conda enviroment and how to install a different version
https://stackoverflow.com/a/55351774/2971299

So, I identified my cuda version

And installed the correct anaconda environment:

Thank you very much @agostini01 . I actually have all versions aligned correctly. The only thing that actually worked out is the second answer here: https://stackoverflow.com/questions/41402409/tensorflow-doesnt-seem-to-see-my-gpu
I uninstalled tensorflow and reinstalled tensorflow-gpu. Apparently they don’t go well together?
Now Python sees my GPUs and when I do watch-smi I can see my job using them.

@KonstantinaLazaridou no problems. I believe your suggested link is for when you are installing cuda system wide.

This line: conda create -n gpu tensorflow-gpu==1.9.0 jupyter cudatoolkit==XX should work as long as you match the anaconda tensorflow-gpu version with the correct anaconda cudatoolkit (XX) and «system-wide installed» cuda driver. Unfortunately I dont remember what to use for the XX value anymore.
|Apparently they don’t go well together?
indeed! Nice catch. The advantage of using conda is that you can have tensorflow in one environment and tensorflow-gpu in another.

@mforde84 I had a similar issue using driver 384.81,but Nvidia recommended Tesla k80 need install driver 384.183.So upgraded to a recent version of drivers 396 is a good choice.
GPU Tesla k80
tensorflow-gpu 1.10.0
CDUNN 7.0.5
CUDA 9.0

2019-12-17 09:55:46.558571: E tensorflow/stream_executor/cuda/cuda_dnn.cc:455] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2019-12-17 09:55:46.558747: E tensorflow/stream_executor/cuda/cuda_dnn.cc:463] possibly insufficient driver version: 384.81.0
2019-12-17 09:55:46.558864: F tensorflow/core/kernels/conv_ops.cc:713] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)

### nvidia drivers mismatch

my nvidia driver is 384.90.

before: error which is same as the title of the thread.
tensorflow-gpu 1.15.0 with cudatoolkit 10.0.130 + cudnn 7.6.5

after: Worked
tensorflow-gpu 1.12.0 with cudatoolkit 9.0

solution:
conda uninstall cudatoolkit (10.0.130)
conda install tensorflow-gpu 1.12 cudatoolkit=9.0

Источник

[0.15.0.dev11] Error: Insufficient CUDA driver: 9010. Not bug. You must upgrade CUDA to 9.2 #1138

Ethminer from archive ethminer-0.15.0.dev11-Linux.tar.gz

On start have following:

nvidia-smi

nvcc —version

With ethminer version 0.14.0 and 0.15.0.dev10 have no problems.

The text was updated successfully, but these errors were encountered:

try 390.59 from Nvidia? dev11 is compiled with cuda 9.2 and on Win10 works fine with latest nvidia drivers.

Oh, yes. Now I see «Travis CI: Build with g++-7, upgrade CUDA to 9.2» comments in commits.
Issue can be closed.

The explanation here seems inconsistent. I have just built v0.14.0 with CUDA 9.2 and I get the same error .

Not sure about v0.14 . I know 0.15 has the CUDA 9.2 code. You also must have the Linux 390.59 drivers from Nvidia for Linux or the latest drivers for Windows that also includes CUDA 9.2.

The drivers must match the code that ethermine uses.

Well something does not match. I just checked out the 0.15.0dev11, built and got the same error.
I’m on Debian 9, I have the proper CUDA, the proper driver and all that jazz

I tried the debug version but I cannot get more from the log . any suggestions?

EDIT:
I just downloaded the released version of 0.15.0dev11 and I have the same behavior .
but it works with the released versioned of 0.14.0!

The problem is in nvidia driver, not the CUDA SDK,

@celavek How do you installed cuda 9.2? I downloaded the cuda toolkit deb network (https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=debnetwork) but after the installation i was unable to login. I had to run sudo apt-get purge nvidia-* for fix this problem, reinstall nvida-390 driver ( sudo apt-get install nvidia-390 ) and also the cuda toolkit ( sudo apt install nvidia-cuda-toolkit ).
Now if i run nvcc —version i have the 7.5.17 version

But if i run this command i have the 9.2.88 version

Same issue here,

m 03:45:06|ethminer| ethminer 0.15.0.dev11-79+commit.83b75508
m 03:45:06|ethminer| Build: linux/release
Error: Insufficient CUDA driver: 9010

terminate called without an active exception
./miner.sh: line 40: 3437 Aborted

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Wed_Apr_11_23:16:29_CDT_2018
Cuda compilation tools, release 9.2, V9.2.88
(to get this result had to manually add the path
PATH=$PATH:/usr/local/cuda-9.2/bin
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-9.2/bin/../lib:/usr/local/cuda-9.2/bin/../lib64
)

Driver Version: 390.59

Any help welcomed

How do you precisely set manually the path?

@invidtiv you may have to reinstall nvidia drivers.

@AndreaLanfranchi Thanks , actually the issue was no purge from apt-get with the * , had to manually purge every nvidia , don t know why.
Got it working, actually installing with sudo apt-get install nvidia-390 it simply did not work for me .
I downloaded the cuda from nvidia cuda_9.2.88.1_linux.run , plus the NVIDIA-Linux-x86_64-396.24.run.
And installed the driver from the cuda install setup.
Again don t know why, I had these issues , they are not etherminer issues, sudo apt-get purge nvidia* always worked for me previously.

@Scorpion2185
just write
PATH=$PATH:/usr/local/cuda-9.2/bin
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-9.2/bin/../lib:/usr/local/cuda-9.2/bin/../lib64
It will add the path. you are setting PATH=$PATH:(new path to include)

@invidtiv i wrote in the terminal:

I used sudo apt-get install cuda and again i wasn’ t able to login, and i had to do all the things that i said before to fix it.

Источник

I’m trying to use CUDA in a laptopt with Nvidia optimus configuration (MSI cx61 2pc) using Bumblebee. I’ve installed Nvidia-390xx drivers as it seems to be the appropriate version for my card, mesa package and also xf86-video-intel. I’ve tried to generate a xorg.conf file with nvidia-xconfig but, doing that I can’t get xorg working and I have some screen-tilling problem so I tried to remove the xorg.conf file and not using any of them I get xorg working.

After beeing fighting with this problems and have things working I installed cuda package, as well as opencl-390xx… I wrote a program in CUDA, compile and execute it, everything seemed to be ok… but then I tried to compile and execute cuda default samples… more specifically, I tried to execute deviceQuery example… and I got the error:  CUDA driver version is insufficient for CUDA runtime version

I have tried to use nvidia driver (without 390xx and then I don’t get that error, but I can’t load nvidia module as the driver is not the apropiate to my card, dmesg show a warning saying that I should use 390xx version)…. So… I think maybe I should uninstall cuda package and install an older cuda version from AUR but… I’m nor sure which version should I use and I can’t find any correspondence between the driver 390xx and CUDA available versions…

Any idea? Am I wrong supposing that? maybe is there a way to use CUDA 9.0 with nvidia-390xx driver?

Thanks for read

some info:

 $  lspci -k | grep -A 2 -E "(VGA|3D)"                         
00:02.0 VGA compatible controller: Intel Corporation 4th Gen Core Processor Integrated Graphics Controller (rev 06)
	Subsystem: Micro-Star International Co., Ltd. [MSI] 4th Gen Core Processor Integrated Graphics Controller
	Kernel driver in use: i915
--
01:00.0 3D controller: NVIDIA Corporation GF117M [GeForce 610M/710M/810M/820M / GT 620M/625M/630M/720M] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] GeForce 820M
	Kernel driver in use: nvidia

everyone,
My machine is ubuntu16.04
I got 2 Tesla P100 ,
Firstly,I install nvidia-driver 418, the result is good.
Then Then I install cuda9.2(The reason why I didn’t choose cuda10.0 is that I want to use pytorch0.4 but it doesn’t support cuda10.0).
I got this:

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:07:04_CDT_2018
Cuda compilation tools, release 9.2, V9.2.148

nvidia-smi

But when I check ,

cd /usr/local/cuda-9.2/samples/1_Utilities/deviceQuery
sudo make
./deviceQuery

it gave me ->CUDA driver version is insufficient for CUDA runtime version.

I installed nvidia-driver by

sudo apt-get install nvidia-418

And I installed cuda by download the cuda_9.2.148_396.37_linux and use sudo sh cuda_9.2.148_396.37_linux,(without installing the driver and I have set the relevant PATH).

Can anyone help me ?
(After I installed the driver418, I installed the cuda9.0 firstly,but it didn’t work because the same reason.Then I uninstalled it and tried the cuda9.2,)
Beg your help!

When running the CUDA example /usr/local/cuda/samples/1_Utilities/deviceQuery$ with the sudo ./deviceQuery command, the output was :

 ./deviceQuery Starting...
 CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

On using the lspci -v | grep -i command I get :

NVIDIA Corporation GF117M [GeForce 610M/710M/820M / GT 620M/625M/630M/720M] (rev a1)

The lshw -c video command gives :

PCI (sysfs)  


  *-display               
       description: VGA compatible controller
       product: Haswell-ULT Integrated Graphics Controller
       vendor: Intel Corporation
       physical id: 2
       bus info: pci@0000:00:02.0
       version: 0b
       width: 64 bits
       clock: 33MHz
       capabilities: vga_controller bus_master cap_list rom
       configuration: driver=i915 latency=0
       resources: irq:63 memory:b5000000-b53fffff memory:c0000000-cfffffff     ioport:6000(size=64)
  *-display
       description: 3D controller
       product: GF117M [GeForce 610M/710M/820M / GT 620M/625M/630M/720M]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:09:00.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: bus_master cap_list
       configuration: driver=nouveau latency=0
       resources: irq:62 memory:b3000000-b3ffffff memory:a0000000-afffffff memory:b0000000-b1ffffff ioport:3000(size=128)

So might it be that CUDA doesn’t work because the i915 driver is in play instead of the nvidia one ?
If so how do I get this working ?

The last guide I followed to install the nvidia drivers really messed up my system and it needed a reinstall, please suggest a guide that works well for Ubuntu 14.04.