Nvidia-docker: docker-compose support

[DarioTurchi]

1. Issue or feature description

not able to run containers with GPU acceleration from docker compose

With nvidia-docker2 I was able to run GPU accelerated containers by defining the nvidia runtime within the file /etc/docker/daemon.json:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

and then use the runtime key within the docker compose file (or set the nvidia runtime as the default one)
With the new version of Nvidia-docker, nvidia container toolkit this doesn't work anymore, since a dedicated runtime should not been run anymore, instead the flag --gpus needs to be provided (or at least this is my understanding).
However, the wiki still suggests the 'runtime' solution, is that right ? Furthermore, the current installation doesn't provide the `/usr/bin/nvidia-container-runtime' file anymore

How does the current version of nvidia-docker should be integrated with docker-compose ?
Sorry if I didn't properly report this as an issue: it is indeed more a lack of documentation and I didn't know where to report it

docker-compose version 1.24.1
Docker version 19.03.2, build 6a30dfc

[eliorc]

Exactly this, how can one use docker-compose with the new nvidia-docker versions?

[DuyHung21]

I'm encountering the same problem. Putting "runtime:nvidia" in docker-compose does not work, even if I follow the instructions from wiki.

[mkolod]

Same problem - runtime: nvidia in docker-compose.yml doesn't work, making changes to /etc/docker/daemon.json the way it's documented used to work, but doesn't now. docker run --gpus all works, but replacing runtime: nvidia with gpus: all in docker-compose.yml is invalid.

[RenaudWasTaken]

Hello!

Docker-compose support still works with the nvidia-docker2 package. You can set the default runtime to nvidia as indicated in the documentation.

What I think you are asking is does docker-compose support the `--gpus option? The answer is not yet!

[eliorc]

@RenaudWasTaken Per the documentation it says to set it this way

{
 "default-runtime": "nvidia",
  "runtimes": {
     "nvidia": {
         "path": "/usr/bin/nvidia-container-runtime",
         "runtimeArgs": []
     }
 },

}

In the newest version of nvidia-docker, there is no /usr/bin/nvidia-container-runtime

[RenaudWasTaken]

You need to install the old nvidia-docker2 packages :)

[eliorc]

So in order to work with docker-compose with the latest nvidia-docker, I need to also install the older nvidia-docker2?

[RenaudWasTaken]

Nope, what I'm saying above is that the support in docker-compose for --gpus is not here yet.

But you can still use the support for --runtime by installing the nvidia-docker2 packages.
The great thing with the old packages is that you also get --gpus for free :)

[eliorc]

What do you mean "by installing nvidia-docker packages"? If I followed all the instructions on the main GitHub page for a fresh installation, haven't I installed the nvidia-docker2 packages?

[RenaudWasTaken]

On the readme:

$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

$ sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
$ sudo systemctl restart docker

Replace the line:

$ sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

By:

$ sudo apt-get update && sudo apt-get install -y nvidia-docker2

[fujimo-t]

More detail discussion and progress are in Compose's issue.
https://github.com/docker/compose/issues/6691

[mpatwary]

I tried nvidia-docker2 as suggested by @RenaudWasTaken with the following /etc/docker/daemon.json file and then docker-compose up works well.

{
  "default-runtime": "nvidia",
  "runtimes": {
     "nvidia": {
       "path": "/usr/bin/nvidia-container-runtime",
       "runtimeArgs": []
       }
     }
}

[leon0707]

this is my daemon.json

{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m"
    },
    "storage-driver": "overlay2",
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

docker run --runtime=nvidia works fine
docker-compose up won't work
Unsupported config option for xxx: 'runtime'

[adam-stanek]

@leon0707 You have to use at least version 2.3 in your docker-compose.yml. You might have to update docker-compose to be able to do that: https://docs.docker.com/compose/install/