I know that this is a common bug, but I have read a variety of issues + bug reports as well as https://github.com/nvidia/nvidia-container-runtime#docker-engine-setup and still cannot resolve the issue.
1. Issue or feature description
$ docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
docker: Error response from daemon: Unknown runtime specified nvidia.
2. Steps to reproduce the issue
I follow the Ubuntu instructions here: https://github.com/NVIDIA/nvidia-docker
I have verified that nvidia-docker is not installed, and nvidia-docker2 is.
My daemon.json is reasonable and set by the package install:
$ cat /etc/docker/daemon.json
{
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}
I try to restart the dockerd as suggested here https://github.com/nvidia/nvidia-container-runtime#docker-engine-setup:
sudo pkill -SIGHUP dockerd
But still get:
$ docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
docker: Error response from daemon: Unknown runtime specified nvidia.
3. Information to attach (optional if deemed irrelevant)
- Kernel version from
uname -a
Linux Ubuntu-1804-bionic-64-minimal 4.15.0-36-generic Using nvidia-docker from third-party tools #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux - Any relevant kernel output lines from
dmesg
n/a - Driver information from
nvidia-smi -a
==============NVSMI LOG==============
Timestamp : Sat Oct 6 19:32:52 2018
Driver Version : 410.48
Attached GPUs : 1
GPU 00000000:01:00.0
Product Name : GeForce GTX 1080
Product Brand : GeForce
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-761e2aa7-1578-4bdc-12c8-d7cac834813a
Minor Number : 0
VBIOS Version : 86.04.17.00.01
MultiGPU Board : No
Board ID : 0x100
GPU Part Number : N/A
Inforom Version
Image Version : G001.0000.01.03
OEM Object : 1.1
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : None
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0x01
Device : 0x00
Domain : 0x0000
Device Id : 0x1B8010DE
Bus Id : 00000000:01:00.0
Sub System Id : 0x119E10DE
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : 1000 KB/s
Rx Throughput : 1000 KB/s
Fan Speed : 41 %
Performance State : P0
Clocks Throttle Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
FB Memory Usage
Total : 8119 MiB
Used : 0 MiB
Free : 8119 MiB
BAR1 Memory Usage
Total : 256 MiB
Used : 2 MiB
Free : 254 MiB
Compute Mode : Default
Utilization
Gpu : 4 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 43 C
GPU Shutdown Temp : 99 C
GPU Slowdown Temp : 96 C
GPU Max Operating Temp : N/A
Memory Current Temp : N/A
Memory Max Operating Temp : N/A
Power Readings
Power Management : Supported
Power Draw : 35.90 W
Power Limit : 180.00 W
Default Power Limit : 180.00 W
Enforced Power Limit : 180.00 W
Min Power Limit : 90.00 W
Max Power Limit : 180.00 W
Clocks
Graphics : 974 MHz
SM : 974 MHz
Memory : 5005 MHz
Video : 873 MHz
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Max Clocks
Graphics : 1911 MHz
SM : 1911 MHz
Memory : 5005 MHz
Video : 1708 MHz
Max Customer Boost Clocks
Graphics : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : None
- Docker version from
docker version
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:24:51 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:28:38 2018
OS/Arch: linux/amd64
Experimental: false
- NVIDIA packages version from
dpkg -l '*nvidia*'orrpm -qa '*nvidia*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================================================================-=====================================-=====================================-========================================================================================================================================
un libgldispatch0-nvidia <none> <none> (no description available)
ii libnvidia-cfg1-410:amd64 410.48-0ubuntu1 amd64 NVIDIA binary OpenGL/GLX configuration library
un libnvidia-cfg1-any <none> <none> (no description available)
un libnvidia-common <none> <none> (no description available)
ii libnvidia-common-410 410.48-0ubuntu1 all Shared files used by the NVIDIA libraries
ii libnvidia-compute-410:amd64 410.48-0ubuntu1 amd64 NVIDIA libcompute package
ii libnvidia-container-tools 1.0.0-1 amd64 NVIDIA container runtime library (command-line tools)
ii libnvidia-container1:amd64 1.0.0-1 amd64 NVIDIA container runtime library
un libnvidia-decode <none> <none> (no description available)
ii libnvidia-decode-410:amd64 410.48-0ubuntu1 amd64 NVIDIA Video Decoding runtime libraries
un libnvidia-encode <none> <none> (no description available)
ii libnvidia-encode-410:amd64 410.48-0ubuntu1 amd64 NVENC Video Encoding runtime library
un libnvidia-fbc1 <none> <none> (no description available)
ii libnvidia-fbc1-410:amd64 410.48-0ubuntu1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
un libnvidia-gl <none> <none> (no description available)
ii libnvidia-gl-410:amd64 410.48-0ubuntu1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
un libnvidia-ifr1 <none> <none> (no description available)
ii libnvidia-ifr1-410:amd64 410.48-0ubuntu1 amd64 NVIDIA OpenGL-based Inband Frame Readback runtime library
un nvidia-304 <none> <none> (no description available)
un nvidia-340 <none> <none> (no description available)
un nvidia-384 <none> <none> (no description available)
un nvidia-390 <none> <none> (no description available)
ii nvidia-compute-utils-410 410.48-0ubuntu1 amd64 NVIDIA compute utilities
ii nvidia-container-runtime 2.0.0+docker18.06.1-1 amd64 NVIDIA container runtime
ii nvidia-container-runtime-hook 1.4.0-1 amd64 NVIDIA container runtime hook
ii nvidia-cuda-dev 9.1.85-3ubuntu1 amd64 NVIDIA CUDA development files
ii nvidia-cuda-doc 9.1.85-3ubuntu1 all NVIDIA CUDA and OpenCL documentation
ii nvidia-cuda-gdb 9.1.85-3ubuntu1 amd64 NVIDIA CUDA Debugger (GDB)
ii nvidia-cuda-toolkit 9.1.85-3ubuntu1 amd64 NVIDIA CUDA development toolkit
ii nvidia-dkms-410 410.48-0ubuntu1 amd64 NVIDIA DKMS package
un nvidia-dkms-kernel <none> <none> (no description available)
un nvidia-docker <none> <none> (no description available)
ii nvidia-docker2 2.0.3+docker18.06.1-1 all nvidia-docker CLI wrapper
un nvidia-driver <none> <none> (no description available)
ii nvidia-driver-410 410.48-0ubuntu1 amd64 NVIDIA driver metapackage
un nvidia-driver-binary <none> <none> (no description available)
un nvidia-kernel-common <none> <none> (no description available)
ii nvidia-kernel-common-410 410.48-0ubuntu1 amd64 Shared files used with the kernel module
un nvidia-kernel-source <none> <none> (no description available)
ii nvidia-kernel-source-410 410.48-0ubuntu1 amd64 NVIDIA kernel source package
un nvidia-legacy-340xx-vdpau-driver <none> <none> (no description available)
un nvidia-libopencl1 <none> <none> (no description available)
un nvidia-libopencl1-dev <none> <none> (no description available)
ii nvidia-modprobe 410.48-0ubuntu1 amd64 Load the NVIDIA kernel driver and create device files
ii nvidia-opencl-dev:amd64 9.1.85-3ubuntu1 amd64 NVIDIA OpenCL development files
un nvidia-opencl-icd <none> <none> (no description available)
un nvidia-persistenced <none> <none> (no description available)
ii nvidia-prime 0.8.8 all Tools to enable NVIDIA's Prime
ii nvidia-profiler 9.1.85-3ubuntu1 amd64 NVIDIA Profiler for CUDA and OpenCL
ii nvidia-settings 410.48-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver
un nvidia-settings-binary <none> <none> (no description available)
un nvidia-smi <none> <none> (no description available)
un nvidia-utils <none> <none> (no description available)
ii nvidia-utils-410 410.48-0ubuntu1 amd64 NVIDIA driver support binaries
un nvidia-vdpau-driver <none> <none> (no description available)
ii nvidia-visual-profiler 9.1.85-3ubuntu1 amd64 NVIDIA Visual Profiler for CUDA and OpenCL
ii xserver-xorg-video-nvidia-410 410.48-0ubuntu1 amd64 NVIDIA binary Xorg driver
- NVIDIA container library version from
nvidia-container-cli -V
version: 1.0.0
build date: 2018-09-20T20:19+00:00
build revision: 881c88e2e5bb682c9bb14e68bd165cfb64563bb1
build compiler: x86_64-linux-gnu-gcc-7 7.3.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
-
NVIDIA container library logs (see troubleshooting)
n/a -
Docker command, image and tag used
See above
I tried to install the nvidia-docker after installing docker-ce. I followed this : https://github.com/NVIDIA/nvidia-docker to install nvidia-docker. It seems to have installed correctly.
I tried to run:
$ sudo docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
docker: Error response from daemon: Unknown runtime specified nvidia.
See 'docker run --help'.
Although, this works (without —runtime=nvidia):
$ docker container run -ti ubuntu bash
Some additional info on my system: It is an ubuntu server 16.04 with 8 GPUs (Titan Xp) and nvidia driver version 387.26. I can run nvidia-smi -l 1 on the host system and it works as expected.
$ dpkg -l | grep -E '(nvidia|docker)'
ii docker-ce 18.06.1~ce~3-0~ubuntu amd64 Docker: the open-source application container engine
ii libnvidia-container-tools 1.0.0-1 amd64 NVIDIA container runtime library (command-line tools)
ii libnvidia-container1:amd64 1.0.0-1 amd64 NVIDIA container runtime library
ii nvidia-container-runtime 2.0.0+docker18.06.1-1 amd64 NVIDIA container runtime
ii nvidia-container-runtime-hook 1.4.0-1 amd64 NVIDIA container runtime hook
ii nvidia-docker2 2.0.3+docker18.06.1-1 all nvidia-docker CLI wrapper
$ cat /etc/docker/daemon.json
{
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}
I have come across: https://github.com/NVIDIA/nvidia-docker/issues/501, but I am not sure how I should go about it.
asked Oct 18, 2018 at 2:15
3
From nvidia-docker github repo:
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2
sudo pkill -SIGHUP dockerd
answered Aug 11, 2021 at 14:47
scepedascepeda
4427 silver badges14 bronze badges
Actually, you can try to restart docker daemon by following command.
sudo systemctl daemon-reload
sudo systemctl restart docker
Or you can try to reboot your system.
to make nvidia-docker work
answered Nov 23, 2018 at 9:06
This is how I resolve the above problem for CentOS 7; hopefully it can help anyone who has similar problems.
- Add necessary repos to get nvidia-container-runtime:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.repo | sudo tee /etc/yum.repos.d/nvidia-container-runtime.repo
- (Optional) In my case, I disabled the experimental repos:
sudo yum-config-manager --disable libnvidia-container-experimental
sudo yum-config-manager --disable nvidia-container-runtime-experimental
- Install nvidia-container-runtime package:
sudo yum install nvidia-container-runtime
- Update docker daemon:
sudo vim /etc/docker/daemon.json
with the path to nvidia-container-runtime:
{
"runtimes": {
"nvidia": {
"path": "/usr/bin/nvidia-container-runtime",
"runtimeArgs": []
}
}
}
- Finally, you need to make docker update the path:
sudo pkill -SIGHUP dockerd
answered Aug 27, 2020 at 5:43
Minh NguyenMinh Nguyen
7202 silver badges11 bronze badges
0
It seems you may need to purge docker and reinstall it as in the post:
github issues
sudo apt remove docker-ce
sudo apt autoremove
sudo apt-get install docker-ce=5:18.09.0~3-0~ubuntu-bionic
sudo apt install nvidia-docker2
answered Feb 7, 2019 at 10:29
5
From nvidia-docker Frequently Asked Questions:
Why do I get the error
Unknown runtime specified nvidia?
Make sure the runtime was registered todockerd. You also need to reload the configuration of the Docker daemon.
LW001
2,3234 gold badges29 silver badges35 bronze badges
answered Sep 25, 2019 at 3:34
1
Change the —runtime=nvidia tag to —runtine=gpus all hopefully it will run
answered Oct 25, 2021 at 17:41
Содержание
- docker: Error response from daemon: Unknown runtime specified nvidiawhile running nvidia docker #578
- Comments
- 1. Issue or feature description
- 2. Steps to reproduce the issue
- 3. Information to attach (optional if deemed irrelevant)
- «Unknown runtime specified nvidia» when running from docker-machine #1018
- Comments
- 1. Issue or feature description
- 2. Steps to reproduce the issue
- 3. Information to attach (optional if deemed irrelevant)
- Unknown runtime specified nvidia after system reboot. Working after restart docker daemon. #951
- Comments
- 1. Issue or feature description
- 2. Steps to reproduce the issue
- 3. Information
- docker: Error response from daemon: Unknown runtime specified nvidia. #1025
- Comments
- 1. Issue or feature description
- 2. Steps to reproduce the issue
- 3. Information to attach (optional if deemed irrelevant)
- [Error (docker)]: response from daemon: Unknown runtime specified nvidia AND could not select device driver «» with capabilities: [[gpu]]. #324
- Comments
- docker: Error response from daemon: Unknown runtime specified nvidia.
- docker: Error response from daemon: could not select device driver «» with capabilities: [[gpu]].
- This comment has been minimized.
- Using nvidia-docker2
- nvidia-container-cli: container error: cgroup subsystem devices not found: unknown
- This comment has been minimized.
docker: Error response from daemon: Unknown runtime specified nvidiawhile running nvidia docker #578
1. Issue or feature description
docker: Error response from daemon: Unknown runtime specified nvidia.
2. Steps to reproduce the issue
sudo docker run —runtime=nvidia —rm nvidia/cuda nvidia-smi
3. Information to attach (optional if deemed irrelevant)
Linux ds-HP-Pavilion-Notebook 4.10.0-28-generic #32
16.04.2-Ubuntu SMP Thu Jul 20 10:19:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
- Driver information from nvidia-smi -a
Timestamp : Sun Dec 17 14:56:13 2017
Driver Version : 384.98
Attached GPUs : 1
GPU 00000000:01:00.0
Product Name : GeForce 940M
Product Brand : GeForce
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-ba67d3da-b480-f0f3-c215-143fea8f5a03
Minor Number : 0
VBIOS Version : 82.08.3C.00.4C
MultiGPU Board : No
Board ID : 0x100
GPU Part Number : N/A
Inforom Version
Image Version : N/A
OEM Object : N/A
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x01
Device : 0x00
Domain : 0x0000
Device Id : 0x134710DE
Bus Id : 00000000:01:00.0
Sub System Id : 0x80A4103C
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Link Width
Max : 4x
Current : 4x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : 0 KB/s
Rx Throughput : 0 KB/s
Fan Speed : N/A
Performance State : P0
Clocks Throttle Reasons
Idle : Not Active
Applications Clocks Setting : Active
SW Power Cap : Not Active
HW Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
FB Memory Usage
Total : 2002 MiB
Used : 0 MiB
Free : 2002 MiB
BAR1 Memory Usage
Total : 256 MiB
Used : 1 MiB
Free : 255 MiB
Compute Mode : Default
Utilization
Gpu : 0 %
Memory : 0 %
Encoder : N/A
Decoder : N/A
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 45 C
GPU Shutdown Temp : 99 C
GPU Slowdown Temp : 94 C
GPU Max Operating Temp : 90 C
Memory Current Temp : N/A
Memory Max Operating Temp : N/A
Power Readings
Power Management : N/A
Power Draw : N/A
Power Limit : N/A
Default Power Limit : N/A
Enforced Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : 1071 MHz
SM : 1071 MHz
Memory : 900 MHz
Video : 1050 MHz
Applications Clocks
Graphics : 1071 MHz
Memory : 900 MHz
Default Applications Clocks
Graphics : 1071 MHz
Memory : 900 MHz
Max Clocks
Graphics : 1176 MHz
SM : 1176 MHz
Memory : 900 MHz
Video : 1152 MHz
Max Customer Boost Clocks
Graphics : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : None
- Docker version from docker version
Client:
Version: 17.09.1-ce
API version: 1.32
Go version: go1.8.3
Git commit: 19e2cf6
Built: Thu Dec 7 22:24:23 2017
OS/Arch: linux/amd64
Server:
Version: 17.09.1-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: 19e2cf6
Built: Thu Dec 7 22:23:00 2017
OS/Arch: linux/amd64
Experimental: false
- NVIDIA packages version from dpkg -l ‘*nvidia*’ or rpm -qa ‘*nvidia*’
ii libnvidia-container-tools 1.0.0 alpha.2-1 amd64 NVIDIA container runtime library (command-line tools)
ii libnvidia-container1:amd64 1.0.0 alpha.2-1 amd64 NVIDIA container runtime library
un nvidia-common (no description available)
ii nvidia-container-runtime 1.1.0+docker17.09.1-1 amd64 NVIDIA container runtime
un nvidia-docker (no description available)
ii nvidia-docker2 2.0.1+docker17.09.1-1 all nvidia-docker CLI wrapper
ii nvidia-modprobe 361.28-1 amd64 utility to load NVIDIA kernel modules and create device nodes
un nvidia-prime (no description available)
NVIDIA container library version from nvidia-container-cli -V
ersion: 1.0.0
build date: 2017-10-30T23:47+00:00
build revision: ec15c7233bd2de821ad5127cb0de6b52d9d2083c
build compiler: gcc-5 5.4.0 20160609
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,—gc-sections
NVIDIA container library logs (see troubleshooting)
Docker command, image and tag used
The text was updated successfully, but these errors were encountered:
Источник
«Unknown runtime specified nvidia» when running from docker-machine #1018
1. Issue or feature description
After install nvidia-docker2 it’s working properly with default docker. However, the runtime isn’t available when trying to use it from a docker-machine.
2. Steps to reproduce the issue
- Installed and compiled from AUR sources (I’m under Manjaro) with my package manager (yay).
- Restart docker daemon.
- Test nvidia-docker installation.
3.1. List available runtimes:
3.2. Test runtime (it works):
- Test nvidia-docker installation inside docker-machine.
6.1. List available runtimes:
3.2. Test runtime (it doesn’t work):
If I leave the docker-machine environment, nvidia-docker works again and nvidia-runtime is found.
3. Information to attach (optional if deemed irrelevant)
All information has been obtained with the docker-machine environment loaded.
- Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info
- Any relevant kernel output lines from dmesg
Nothing relevant obtained with grep, neither searching for docker nor nvidia. - Driver information from nvidia-smi -a
- Docker version from docker version
- NVIDIA packages version from dpkg -l ‘*nvidia*’ or rpm -qa ‘*nvidia*’
Build Date : Mon 15 Jul 2019 08:56:09 PM CEST Install Date : Thu 18 Jul 2019 11:45:50 PM CEST Install Reason : Explicitly installed Install Script : Yes Validated By : Signature»>
- NVIDIA container library version from nvidia-container-cli -V
- NVIDIA container library logs (see troubleshooting)
Tried but no log was generated. Maybe because the runtime didn’t start because it wasn’t found. - Docker command, image and tag used
See section «2. Steps to reproduce the issue» above.
The text was updated successfully, but these errors were encountered:
Docker runtime will not be available on virtual box because it is not installed there. Besides, you will not have your gpus available there either. They need to be made available passthrough over the hypervisor first.
Does not seem like a plausible use case. Was this ever working? Or is this the first time you have tried.
This is the first time I have tried it. I didn’t realize that I must install the runtime on virtual box first, but it makes sense. Just for documentation: will this be possible? And in a favorable case, how?
This is the first time I have tried it. I didn’t realize that I must install the runtime on virtual box first, but it makes sense. Just for documentation: will this be possible? And in a favorable case, how?
Should be possible — add gpu devices as passthrough, install drivers again in the virtualbox, install the nvidia-container-runtime again in the virtualbox (the last two can be part of an image itself). But we need to test/foolproof it before we add it as a known use-case in documentation. Could I encourage you to drive it? 🙂 Thanks!
Meanwhile, I am closing this issue. Please open a new one suggesting it as a ‘feature’. Feel free to re-open this one if there is a bug/code-fix that we need to track.
Источник
Unknown runtime specified nvidia after system reboot. Working after restart docker daemon. #951
1. Issue or feature description
After system reboot the following command reports an error:
In the same terminal the following works:
2. Steps to reproduce the issue
Reboot system.
Run commands described before.
3. Information
- Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info
- Docker version from docker version
- NVIDIA packages version from dpkg -l ‘*nvidia*’ or rpm -qa ‘*nvidia*’
- [x ] NVIDIA container library version from nvidia-container-cli -V
The text was updated successfully, but these errors were encountered:
Sorry for the delay,
seems like your docker daemon isn’t setup properly, are you able to reproduce this behavior reliably?
What is the content of /etc/docker/daemon.json?
The bug reproduces on every reboot. I am wondering if there is a conflict on any daemons initialization order.
Hmm, do you have the contents of the docker systemd unit file?
Sorry for the long delay.
Setting your unit file with the runtime should solve your problem.
See an example here: https://github.com/NVIDIA/nvidia-container-runtime#systemd-drop-in-file
Same issue here. Answer above from @RenaudWasTaken doesn’t solve it for me.
@LuisAyuso How about you?
after restarting the system, docker still does not find nvidia runtime.
when restarting docker manually, everithing seems to work ok.
therefore the issue continues open.
I reinstalled docker and nvidia-docker (purging everything first, as in: link) and also removed snap version of docker ( sudo snap remove docker ) that was installed on my system. Works fine now, even after reboot.
I did re-install the services as well, I had to madison my way in. but is working now.
I came to the conclusion that the error was produced by a docker snap installation that competes for the service on system boot. I did not have a clue that docker could be installed from snap or how it made into the system. Nevertheless the issue is no longer there and nvidia-containers work correctly.
Источник
docker: Error response from daemon: Unknown runtime specified nvidia. #1025
1. Issue or feature description
I installed nvidia-docker on a fresh CentOS 7.6 machine, and while I can run things like docker run —gpus all nvidia/cuda:10.1-base nvidia-smi , all of the other commands that apps rely on ( nvidia-docker , docker run —runtime=nvidia ) don’t work.
I’d love to install the legacy software as well, but the CentOS commands aren’t listed on the front page, only the Ubuntu instructions
2. Steps to reproduce the issue
Install nvidia-docker with the attached instructions on CentOS 7.6
3. Information to attach (optional if deemed irrelevant)
- Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info
Kernel version from uname -a
Linux localhost.localdomain 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Any relevant kernel output lines from dmesg
Driver information from nvidia-smi -a
- Docker version from docker version
- NVIDIA packages version from dpkg -l ‘*nvidia*’ or rpm -qa ‘*nvidia*’
None - NVIDIA container library version from nvidia-container-cli -V
- NVIDIA container library logs (see troubleshooting)
- Docker command, image and tag used
The text was updated successfully, but these errors were encountered:
Источник
[Error (docker)]: response from daemon: Unknown runtime specified nvidia AND could not select device driver «» with capabilities: [[gpu]]. #324
I am unable to troubleshoot this issue can you let me know what information could be helpful to help me .
docker: Error response from daemon: Unknown runtime specified nvidia.
docker: Error response from daemon: could not select device driver «» with capabilities: [[gpu]].
The text was updated successfully, but these errors were encountered:
Hey @Luxcium do you have nvidia-docker2 installed on your system?
Might be related to that!
If you do, may be this discussion might help
I think Fedora team hates people using NVIDIA or NVIDIA team hates people using Fedora
Hey @Luxcium do you have nvidia-docker2 installed on your system?
Might be related to that!
If you do, may be this discussion might help
Thanks @AjayThorve
do you know if I can get it except from https://rpms.if-not-true-then-false.com/inttf.repo (link to the blog post)
I use Fedora release 34 (Thirty Four) as shown in the hidden post above .
I am doing it then.
Using nvidia-docker2
I have a new error message now
nvidia-container-cli: container error: cgroup subsystem devices not found: unknown
after one hour of googling and trying to find a solution I must admit that I will wait to see if someone could help me here I was looking into the container error: cgroup subsystem devices not found: unknown but maybe I am starting to be blind to solution if you know the solution just let me know or please ask me more details about my system or configuration
@Luxcium not entirely sure what you’ve tried, but generally the 3 things you will need (in addition to the driver) are:
I know it’s possible to use GPUs in docker in RHEL, because we publish RHEL (Centos) images for the core RAPIDS libraries. Let me know if it still doesn’t work after installing the above. I don’t have a box with Centos right now, but I could put it on one of my spare machines to test if I need to.
Please see my comment here about the error of container error: cgroup subsystem devices not found: unknown regarding the lack of cgroupv2 support.
Источник
On this page
- WSL + Windows
- With Tensorflow or PyTorch
- Basic installation
- Check info
- Does Docker work with GPU?
- Check cudnn
- Install nvidia-docker2
- Difference: nvidia-container-toolkit vs nvidia-container-runtime
- Using docker-compose?
- Check usage of GPU
- Kill process
- Reset GPU
- Errors with GPU
- Make NVIDIA work in docker (Linux)
- References
👉 Note: All docker notes.
👉 My Dockerfile setting up on Github.
WSL + Windows
👉 Note: WSL + Windows
With Tensorflow or PyTorch
👉 Official doc for TF + docker
👉 Note: Docker + TF.
👉 An example of docker pytorch with gpu support.
Basic installation
You must (successfully) install the GPU driver on your (Linux) machine before proceeding with the steps in this note. Go to the «Check info» section to check the availability of your drivers.
(Maybe just for me) It works perfectly on Pop!_OS 20.04, I tried it and we have a lot of problems with Pop!_OS 21.10 so stay with 20.04!
sudo apt updatesudo apt install -y nvidia-container-runtime
# You may need to replace above line with
sudo apt install nvidia-docker2
sudo apt install nvidia-container-toolkit
sudo apt install -y nvidia-cuda-toolkit
# restard required
If you have problems installing nvidia-docker2, read this section!
Check info
# Verify that your computer has a graphic card
lspci | grep -i nvidia# First, install drivers and check
nvidia-smi
# output: NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0
# It's the maximum CUDA version that your driver supports# Check current version of cuda
nvcc --version
# If nvcc is not available, it may be in /usr/local/cuda/bin/
# Add this location to PATH
# modify ~/.zshrc or ~/.bashrc
export PATH=/usr/local/cuda/bin:$PATH# You may need to install
sudo apt install -y nvidia-cuda-toolkit
If below command doesn’t work, try to install nvidia-docker2 (read this section).
# Install and check nvidia-docker
dpkg -l | grep nvidia-docker
# or
nvidia-docker version# Verifying –gpus option under docker run
docker run --help | grep -i gpus
# output: --gpus gpu-request GPU devices to add to the container ('all' to pass all GPUs)Does Docker work with GPU?
# List all GPU devices
docker run -it --rm --gpus all ubuntu nvidia-smi -L
# output: GPU 0: GeForce GTX 1650 (...)# ERROR ?
# docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown.# ERROR ?
# Error response from daemon: could not select device driver "" with capabilities: [[gpu]]# Solution: install nvidia-docker2
# Verifying again with nvidia-smi
docker run -it --rm --gpus all ubuntu nvidia-smi# Return something like
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.54 Driver Version: 510.54 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| N/A 55C P0 11W / N/A | 369MiB / 4096MiB | 5% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
# and another box like this
It’s archived, but still useful
# Test a working setup container-toolkit
# Update 14/04/2022: the tag "latest" has deprecated => check your system versions and use
# the corresponding tag
# The following code is for reference only, it no longer works
docker run --rm --gpus all nvidia/cuda nvidia-smi# Test a working setup container-runtime
# Update 14/04/2022: below code isn't working anymore because nvidia/cuda doesn't have
# the "latest" tag!
docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi# Error response from daemon: Unknown runtime specified nvidia.
# Search below for "/etc/docker/daemon.json"
# Maybe it helps.
Check cudnn
whereis cudnn
# cudnn: /usr/include/cudnn.h# Check cudnn version
cat /usr/include/cudnn.h | grep CUDNN_MAJOR -A 2
# or try this (it works for me, cudnn 8)
cat /usr/include/cudnn_version.h | grep CUDNN_MAJOR -A 2
Install nvidia-docker2
More information (ref)
This package is the only docker-specific package of any of them. It takes the script associated with the
nvidia-container-runtimeand installs it into docker’s/etc/docker/daemon.jsonfile for you. This then allows you to run (for example)docker run --runtime=nvidia ...to automatically add GPU support to your containers. It also installs a wrapper script around the native docker CLI callednvidia-dockerwhich lets you invoke docker without needing to specify--runtime=nvidiaevery single time. It also lets you set an environment variable on the host (NV_GPU) to specify which GPUs should be injected into a container.
👉 (Should follow this for the up-to-date) Officicial guide to install.
Note: (Only for me) Use the codes below.
Command lines (for quickly preview)
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)# NOTE FOR POPOS 20.04
# replace above line with
distribution=ubuntu20.04
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2
👇 Read more about below error.
# Error?
# Read more:
# Depends: nvidia-container-toolkit (>= 1.9.0-1) but 1.5.1-1pop1~1627998766~20.04~9847cf2 is to be installed# create a new file
sudo nano /etc/apt/preferences.d/nvidia-docker-pin-1002
# with below content
Package: *
Pin: origin nvidia.github.io
Pin-Priority: 1002
# then save
# try again
sudo apt-get install -y nvidia-docker2
# restart docker
sudo systemctl restart docker# wanna check?
sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
# check version
nvidia-docker version👉 What’s the difference between the lastest nvidia-docker and nvidia container runtime？
In this note, with Docker 19.03+ (
docker --version), he says thatnvidia-container-toolkitis used for--gpus(indocker run ...),nvidia-container-runtimeis used for--runtime=nvidia(can also be used indocker-composefile).
However, if you want to use Kubernetes with Docker 19.03, you actually need to continue using nvidia-docker2 because Kubernetes doesn’t support passing GPU information down to docker through the
--gpusflag yet. It still relies on the nvidia-container-runtime to pass GPU information down the stack via a set of environment variables.
👉 Installation Guide — NVIDIA Cloud Native Technologies documentation
Using docker-compose?
Purpose?
# instead of using
docker run
--gpus all
--name docker_thi_test
--rm
-v abc:abc
-p 8888:8888# we use this with docker-compose.yml
docker-compose up# check version of docker-compose
docker-compose --version# If "version" in docker-compose.yml < 2.3
# Modify: /etc/docker/daemon.json
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}# restart our docker daemon
sudo pkill -SIGHUP dockerd# If "version" in docker-compose.yml >=2.3
# docker-compose.yml => able to use "runtime"
version: '2.3' # MUST BE >=2.3 AND <3
services:
testing:
ports:
- "8000:8000"
runtime: nvidia
volumes:
- ./object_detection:/object_detection👉 Check more in my repo my-dockerfiles on Github.
Run the test,
docker pull tensorflow/tensorflow:latest-gpu-jupyter
mkdir ~/Downloads/test/notebooksWithout using docker-compose.yml (tensorflow) (cf. this note for more)
docker run --name docker_thi_test -it --rm -v $(realpath ~/Downloads/test/notebooks):/tf/notebooks -p 8888:8888 tensorflow/tensorflow:latest-gpu-jupyterWith docker-compose.yml?
# ~/Download/test/Dockerfile
FROM tensorflow/tensorflow:latest-gpu-jupyter# ~/Download/test/docker-compose.yml
version: '2'
services:
jupyter:
container_name: 'docker_thi_test'
build: .
volumes:
- ./notebooks:/tf/notebooks # notebook directory
ports:
- 8888:8888 # exposed port for jupyter
environment:
- NVIDIA_VISIBLE_DEVICES=0 # which gpu do you want to use for this container
- PASSWORD=12345Then run,
docker-compose run --rm jupyterCheck usage of GPU
# Linux only
nvidia-smiReturn something like this
# |===============================+======================+======================|
# | 0 GeForce GTX 1650 Off | 00000000:01:00.0 Off | N/A |
# | N/A 53C P8 2W / N/A | 3861MiB / 3914MiB | 2% Default |
# | | | N/A |
# +-------------------------------+----------------------+----------------------+# => 3861MB / 3914MB is used!
# +-----------------------------------------------------------------------------+
# | Processes: GPU Memory |
# | GPU PID Type Process name Usage |
# |=============================================================================|
# | 0 3019 C ...e/scarter/anaconda3/envs/tf1/bin/python 3812MiB |
# +-----------------------------------------------------------------------------+
# => Process 3019 is using the GPU
# All processes that use GPU
sudo fuser -v /dev/nvidia*Kill process
# Kill a single process
sudo kill -9 3019Reset GPU
# all
sudo nvidia-smi --gpu-reset# single
sudo nvidia-smi --gpu-reset -i 0Errors with GPU
# Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
# Function call stack:
# train_function👉 Check this answer as a reference!
👇 Use a GPU.
# Limit the GPU memory to be used
gpus = tf.config.list_physical_devices('GPU')
if gpus:
# Restrict TensorFlow to only allocate 1GB of memory on the first GPU
try:
tf.config.set_logical_device_configuration(
gpus[0],
[tf.config.LogicalDeviceConfiguration(memory_limit=1024)])
logical_gpus = tf.config.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
# Virtual devices must be set before GPUs have been initialized
print(e)Problems with pytorch versions: check this.
RuntimeError: cuda runtime error (804) : forward compatibility was attempted on non supported HW at /pytorch/aten/src/THC/THCGeneral.cpp:47 (after update system including nvdia-cli, maybe) => The same problem with below, need to restart the computer.
nvidia-smi: Failed to initialize NVML: Driver/library version mismatch.
This thread: just restart the computer.
Make NVIDIA work in docker (Linux)
This section still works (on 26-Oct-2020), but it’s obselete for newer methods.
One idea: Use NVIDIA driver of the base machine, don’t install anything in Docker!
Detail of steps
-
First, maker sure your base machine has an NVIDIA driver.
# list all gpus
lspci -nn | grep '[03'# check nvidia & cuda versions
nvidia-smi -
Install
nvidia-container-runtimecurl -s -L https://nvidia.github.io/nvidia-container-runtime/gpgkey | sudo apt-key add -
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.list | sudo tee /etc/apt/sources.list.d/nvidia-container-runtime.list
sudo apt-get update
sudo apt-get install nvidia-container-runtime
-
Note that, we cannot use
docker-compose.ymlin this case!!! -
Create an image
img_dataswithDockerfileisFROM nvidia/cuda:10.2-baseRUN apt-get update &&
apt-get -y upgrade &&
apt-get install -y python3-pip python3-dev locales git# install dependencies
COPY requirements.txt requirements.txt
RUN python3 -m pip install --upgrade pip &&
python3 -m pip install -r requirements.txt
COPY . .# default command
CMD [ "jupyter", "lab", "--no-browser", "--allow-root", "--ip=0.0.0.0" ] -
Create a container,
docker run --name docker_thi --gpus all -v /home/thi/folder_1/:/srv/folder_1/ -v /home/thi/folder_1/git/:/srv/folder_2 -dp 8888:8888 -w="/srv" -it img_datas# -v: volumes
# -w: working dir
# --gpus all: using all gpus on base machine
This article is also very interesting and helpful in some cases.
References
- Difference between
base,runtimeanddevelinDockerfileof CUDA. - Dockerfile on Github of Tensorflow.
Trying to get plex running in docker (docker amateur, sorry if this is dumb). I keep getting the following error when trying to set parameters:
Error response from daemon: Unknown runtime specified nvidia
Below are the parameters I’m trying to set:
sudo docker create
--name=plex
--net=host
--restart=always
--runtime=nvidia
-e VERSION=latest
-e PUID=1001 -e PGID=1001
-e TZ=America/Indiana/Indianapolis
-e NVIDIA_DRIVER_CAPABILITIES=all
-e NVIDIA_VISIBLE_DEVICES=GPU-727dbe8b-c27a-63d8-fbf3-b8ed347c015a
-v /home/docker/plex/config:/config
-v /home/docker/plex/tvshows:/data/tvshows
-v /home/docker/plex/movies:/data/movies
-v /home/docker/plex/transcode:/transcode
plexinc/pms-docker
I did some googling and found a few things to try to add including this from the Arch wiki and this site on how to use GPU in a container. I’ve run the nvidia test thing from that site and this is the output:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.24.02 Driver Version: 465.24.02 CUDA Version: 11.3 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:03:00.0 On | N/A |
| 38% 28C P8 7W / 120W | 253MiB / 3016MiB | 3% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
The error usually occurs when Nvidia driver is newly installed or docker/nvidia-docker is installed.
August 12, 2018
less than 1 minute read
Photo by Junyong Lee
The error usually occurs when Nvidia driver is newly installed or docker/nvidia-docker is installed.
Simply type:
$ sudo service nividia-docker restart
Leave a comment
You may also enjoy
How to configure SSH without Passwords
May 9, 2022
4 minute read
This article introduces a secure private/public key-based SSH connection method to log into remote servers from a local machine.
Here, we can think of the pu…