Nvidia-docker: SE Linux Permissions

[andyneff]

1. Issue

I just updated my machine today, and when I run an nvidia-docker container, I get the following in /var/log/messages

Aug 30 16:12:35 viola audit[25758]: AVC avc:  denied  { entrypoint } for  pid=25758 comm="nvidia-docker" path="/usr/bin/docker" dev="sdc3" ino=12832289 scontext=unconfined_u:system_r:xserver_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_runtime_exec_t:s0 tclass=file permissive=0

Same happens with docker-compose

I can change the secontext on nvidia-docker and that fixes it. So the main question I have

semanage fcontext -l | grep nvidia reveals the context rule causing the problem

/usr/bin/nvidia.*                                  regular file       system_u:object_r:xserver_exec_t:s0 

Which I suspect come from the nvidia cuda driver rpms I get from the official nvidia Fedora 27 cuda repo (Since there is still no Fedora 28 Repo).

If the context for nvidia-docker is indeed wrong, then maybe the nvidia-docker rpms should add a special rule, like

/usr/bin/nvidia-docker                           regular file       system_u:object_r:container_runtime_exec_t:s0

My understanding of selinux is still pretty weak, so I could be way off base here.

2. Steps to reproduce the issue

nvidia-docker run -it --rm nvidia/cuda nvidia-smi
ls -lZ /usr/bin/docker /usr/bin/nvidia-docker

Failure output

[root@computer bin]# nvidia-docker run -it --rm nvidia/cuda nvidia-smi
/usr/bin/nvidia-docker: line 34: /usr/bin/docker: Permission denied
/usr/bin/nvidia-docker: line 34: /usr/bin/docker: Success
-rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 33069424 Aug 21 13:27 /usr/bin/docker
-rwxr-xr-x. 1 root root system_u:object_r:xserver_exec_t:s0                721 Aug 21 20:25 /usr/bin/nvidia-docker

Workaround

sudo chcon system_u:object_r:container_runtime_exec_t:s0 /usr/bin/nvidia-docker
ls -lZ /usr/bin/docker /usr/bin/nvidia-docker

Update

semanage fcontext -a -f f -t container_runtime_exec_t -s system_u /usr/bin/nvidia-docker
nvidia-docker run -it --rm nvidia/cuda echo hi

Workaround output

-rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0 33069424 Aug 21 13:27 /usr/bin/docker
-rwxr-xr-x. 1 root root system_u:object_r:container_runtime_exec_t:s0      721 Aug 21 20:25 /usr/bin/nvidia-docker
hi

3. Additional Information

• Kernel version: 4.17.19-200.fc28.x86_64 #1 SMP Fri Aug 24 15:47:41 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Docker version: 18.06.1-ce
• NVIDIA docker version: 2.0.3
• NVIDIA container library version: 1.0.0
• NVIDIA Driver version: 396.44-1.fc27
• OS: Fedora 28

https://fedoraproject.org/wiki/PackagingDrafts/SELinux#File_contexts

[scott-vsi]

This also affects nvidia-docker v1

[fyang93]

Had the same issue on CentOS 7

[andyneff]

It's on my TODO list to add a PR for this. I've been learning a lot about SE Linux since opening this :(

The workaround is to use semanage fcontext -a -f f -t container_runtime_exec_t -s system_u /usr/bin/nvidia-docker, and this will fix the context issue permanently (the chcon was only temporary, and would be lost if a relabel ever occurred). [[1](https://fedoraproject.org/wiki/PackagingDrafts/SELinux#File_contexts)]

However having semange as a install/uninstall dependency is a little much. And I know most RPMs don't do this. I also can't find any examples of how to handle this correctly... (like in nvidia driver).

What I've pieced together

Generally, this is what you do in an rpm spec for an selinux package, that you make a separate rpm from the "core" package

1. A type enforcement file: nvidia_docker.te

   policy_module(nvidia_docker 1.0);

2. A file context file: nvidia_docker.fc

   /usr/bin/nvidia-docker -- system_u:object_r:container_runtime_exec_t:s0

3. And then you build the policy package file: nvidai_docker.pp

   make -f %{_datadir}/selinux/devel/Makefile

4. And when the preinstall step would say semodule -i nvidia_docker.pp

5. The postuninstall step would say semodule -r nvidia_docker

Couldn't get fixfiles working, so restorecon -F /usr/bin/nvidia-docker if you are running this after install

https://fedoraproject.org/wiki/SELinux_Policy_Modules_Packaging_Draft

Another way to generated a minimal policy

• nvidia_docker.te

  module my_nvidia_docker 1.0;

  require {
  type container_runtime_exec_t;
  }

• Same nvidia_docker.fc

1. checkmodule -M -m -o nvidia_docker.mod nvidia_docker.te
2. semodule_package -o nvidia_docker.pp -m nvidia_docker.mod -f nvidia_docker.fc
3. semodule -X 300 -i nvidia_docker.pp

[Rovanion]

To work around this issue as a user on a machine with SELinux such as CentOS 7 you can run the following commands:

$ sudo semanage fcontext -a -t container_runtime_exec_t /usr/bin/nvidia-docker
$ sudo restorecon -v /usr/bin/nvidia-docker

I look forward to this issue being fixed in the package itself.

[rdwrt]

I think there is a typo here:

$ sudo semanage fcontext -t container_runtime_exec_t /usr/bin/nvidia-docker

[Rovanion]

Ah, yes, there was an -a missing. Edited the comment for posterity.

[Rovanion]

Has this issue been resolved in master?

[andyneff]

Not really afaik. However, with the new "native" support, you don't need nvidia-docker anymore, so this path is "deprecated"

[Rovanion]

Could someone provide a link on how to set up the new implementation?

[jasonheffner]

I wasn't aware this has been integrated either ..

https://github.com/NVIDIA/nvidia-docker/wiki/Installation-(Native-GPU-Support)

Also in the readme..

Note that with the release of Docker 19.03, usage of nvidia-docker2 packages are deprecated since NVIDIA GPUs are now natively supported as devices in the Docker runtime. If you are an existing user of the nvidia-docker2 packages, review the instructions in the “Upgrading with nvidia-docker2” section.

[Rovanion]

Thank you!

Den fre 20 sep. 2019 kl 15:14 skrev Jason Heffner <[email protected]

:

I wasn't aware this has been integrated either ..

https://github.com/NVIDIA/nvidia-docker/wiki/Installation-(Native-GPU-Support)

Also in the readme..

Note that with the release of Docker 19.03, usage of nvidia-docker2
packages are deprecated since NVIDIA GPUs are now natively supported as
devices in the Docker runtime. If you are an existing user of the
nvidia-docker2 packages, review the instructions in the “Upgrading with
nvidia-docker2” section.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/NVIDIA/nvidia-docker/issues/814?email_source=notifications&email_token=AAE2PR67RFWKTCQU3WMNPV3QKTEELA5CNFSM4FSQZJD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7GVAQI#issuecomment-533549121,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAE2PRZ2PLNPPN7HTAPTMV3QKTEELANCNFSM4FSQZJDQ
.

[m-reuter]

another workaround:

sudo mv nvidia-docker nvidia-docker.copy
sudo cp nvidia-docker.copy nvidia-docker
sudo rm nvidia-docker.copy