Nvidia-docker: Connect nvidia-docker as remote python interpreter in Pycharm

[JGuillaumin]

Hi,

Someone found a way to connect a python interpreter from container launched with nvidia-docker as remote interpreter in Pycharm ?

I use Pycharm pro edition, that provides tools for Docker integration (with docker-machine or docker-compose)
I followed some tutorial about this tool in Pycharm, (the tool based on docker-machine).. but it seems to connect only an image launch with docker not with nvidia-docker !
(tuto with docker-machine: https://www.jetbrains.com/help/pycharm/2016.3/configuring-remote-interpreters-via-docker.html)

I would like to build a very efficient deep learning dev environment that uses nvidia-docker technology.

Thank you very for your help.

Julien

[3XX0]

Did you try nvidia-docker-compose? I'm not familiar with Pycharm but maybe it's just a matter of changing the docker-compose executable

[JGuillaumin]

I didn't. I try this tool tomorrow, thank very much.
I found an alternative, by using a remote SSH interpreter (just need to add some package inside the container to allow SSH connection), but the path-mapping (for data and src) is more complex.

[chrismpalmer]

@JGuillaumin Hello! Julian! Have you had luck with this? I am heading down that avenue and wondering how you went.

[ozen]

As far as I understood, PyCharm uses its own internal Docker client which currently can't be replaced with nvidia-docker.

The solution I too came up with was using remote SSH. But I couldn't succeed. PyCharm connected through SSH, but GPU wasn't accessible and I couldn't figure out why.

Then I tried nvidia-docker-compose. I set docker compose executable setting in PyCharm to nvidia-docker-compose but it didn't get executed, so I don't know how that setting works.

Lastly, I tried running nvidia-docker-compose outside of PyCharm to make it create nvidia-docker-compose.yml file, then I set configuration file setting in PyCharm to nvidia-docker-compose.yml. This one worked without problem. Of course this isn't ideal.

[xjx0524]

@ozen Hi, Could you please introduce detailedly the third method? I have the same issue.

[ozen]

@xjx0524 Here you go: http://www.yigitozen.info/posts/2017/02/28/pycharm-docker-nvidia-gpu-container.html

You need to write a standard docker compose file (docker-compose.yml). Check out docker compose docs if you don't know how to do it.
Then you use nvidia-docker-compose -G command and it creates nvidia-docker-compose.yml file. In PyCharm settings, you select that file.

[xjx0524]

@ozen It works well. Thank you very much!

[WillieMaddox]

I'm not seeing in the docker compose docs where they talk about a standard docker compose file. Could you elaborate on the meaning of "standard". Maybe a short example?

[ozen]

@WillieMaddox A google search would give some results but let me give you what I used in my project PyOptiX as an example.

This is the base docker-compose.yml:

version: '2'
services:
  pyoptix:
    environment:
    - DISPLAY=:1
    image: pyoptix
    user: '1000'
    volumes:
    - /home/yigit:/home/yigit
    - /etc/group:/etc/group:ro
    - /etc/passwd:/etc/passwd:ro
    - /etc/shadow:/etc/shadow:ro
    - /etc/sudoers:/etc/sudoers:ro
    - /etc/sudoers.d:/etc/sudoers.d:ro
    - /tmp/.X11-unix:/tmp/.X11-unix:rw
````

It could be as simple as the following but the extra configuration lets the container access the X11 Server on my system to create some windows.

version: '2'
services:
pyoptix:
image: pyoptix

After I run `nvidia-docker-compose -G` on this file, it will generate the following on my computer:

version: '2'
services:
pyoptix:
devices:
- /dev/nvidia0
- /dev/nvidiactl
- /dev/nvidia-uvm
- /dev/nvidia-uvm-tools
environment:
- DISPLAY=:1
image: pyoptix
user: '1000'
volumes:
- /home/yigit:/home/yigit
- /etc/group:/etc/group:ro
- /etc/passwd:/etc/passwd:ro
- /etc/shadow:/etc/shadow:ro
- /etc/sudoers:/etc/sudoers:ro
- /etc/sudoers.d:/etc/sudoers.d:ro
- /tmp/.X11-unix:/tmp/.X11-unix:rw
- nvidia_driver_375.26:/usr/local/nvidia:ro
volumes:
nvidia_driver_375.26:
external: true
```

You can learn what the keys in this YAML file mean from Docker Compose File Reference

[ksnzh]

@ozen I have succeeded add remote interpreter with nvidia-docker-compose.yml last week.
I changed a new pc and I encountered this error:

The command '/bin/sh -c mv /usr/bin/docker-compose /usr/bin/docker-compose-original' returned a non-zero code: 1

[ozen]

@ksnzh I haven't seen this error before.

Is the Docker server local or remote? Current PyCharm documentation says:

The feature works for Linux with the local installations of Docker and macOS, it won't be shown in PyCharm running on Windows platform. On macOS, Docker Compose configuration file must be situated within the default shared folder /Users/.

I haven't tested this method with remote servers.

[ksnzh]

@ozen It is a local docker installation. I consider it is caused by the
docker-compose executes. When I add Python interpreter by docker-compose,
Pycharm will automatically pull a docker-compose image. Is it the right
way?

More information：

➜  ~ docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
<none>                                 <none>              040506ce642d        About an hour ago   19 MB
pycharm_helpers                        PY-171.3780.115     b74951315654        33 hours ago        25.8 MB
docker/compose                         1.12.0              d34ddea767a5        4 days ago          19 MB

➜  ~ docker ps -a
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS                         PORTS               NAMES
16b97505a016        040506ce642d                      "/bin/sh -c 'mv /u..."   About an hour ago   Exited (1) About an hour ago                       eloquent_davinci

2017年4月11日 00:11，"Yigit Ozen" [e-mail]写道：

@ksnzh https://github.com/ksnzh I haven't seen this error before.

Is the Docker server local or remote? Current PyCharm documentation says:

The feature works for Linux with the local installations of Docker and
macOS, it won't be shown in PyCharm running on Windows platform. On macOS,
Docker Compose configuration file must be situated within the default
shared folder /Users/.

I haven't tested this method with remote servers.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NVIDIA/nvidia-docker/issues/303#issuecomment-292998125,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AOMWC2899VQGrI1LyeVJB1Vr8P0nhYaiks5rulSNgaJpZM4Lzp3Q
.

[ksnzh]

@ozen I used a system bulid-in python version with sudo to install docker-compose early.
And now I try anaconda2 to install docker-compose and nvidia-docker-compose, it worked.
More information:

➜  ~ which docker-compose       
/home/ksnzh/anaconda2/bin/docker-compose
➜  ~ which nvidia-docker-compose      
/home/ksnzh/anaconda2/bin/nvidia-docker-compose
➜  ~ docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
intellij/docker/compose                1.10.0              a67b8c236900        9 minutes ago       59.3 MB
pycharm_helpers                        PY-171.3780.115     b74951315654        41 hours ago        25.8 MB
tensorflow/tensorflow                  latest-gpu          82e9178974c3        2 weeks ago         2.8 GB
docker/compose                         1.10.0              62fbca31c5ee        2 months ago        59.3 MB

[3XX0]

Closing, there are no more wrapper/plugin in 2.0/master, hence the integration should be easy.

[geyang]

nvidia-docker-compose is not working with nvidia-docker v2.0.0. The nvidia-docker-compose -G command doesn't work in my case.

[flx42]

@episodeyang if you use compose file version 2.3, then you can use runtime: nvidia.
You can find an example here:
https://github.com/NVIDIA/gpu-monitoring-tools/blob/master/exporters/prometheus-dcgm/docker-compose.yml#L32

[geyang]

@flx42 Thank you for your reply! Here is the error message I'm running into: uhm what do you think is causing it?

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/playground/bin/nvidia-docker-compose", line 55, in <module>
    resp = request.urlopen('http://{0}/docker/cli/json'.format(args.nvidia_docker_host)).read().decode()
  File "/home/ubuntu/anaconda3/envs/playground/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/home/ubuntu/anaconda3/envs/playground/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/home/ubuntu/anaconda3/envs/playground/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/home/ubuntu/anaconda3/envs/playground/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/home/ubuntu/anaconda3/envs/playground/lib/python3.6/urllib/request.py", line 1346, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/home/ubuntu/anaconda3/envs/playground/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>

and my nvidia-docker version is this:

dia-docker version
NVIDIA Docker: 2.0.3
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:17:20 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:15:30 2018
  OS/Arch:      linux/amd64
  Experimental: false

Thanks! 😄

My compose file is like this

version: '2.3'
services:
  main:
    runtime: nvidia
    build: .
    volumes:
      - .:/opt/project

[geyang]

@flx42 I got it to work!

For people who are struggling, here is some quick notes.

Overview

Mac

use pyCharm, prep docker-compose.yml file like below

version: '2.3'
services:
  main:
    build: .
    volumes:
      - .:/opt/project

remote linux server:

1. download nvidia-docker 1.0.1, do NOT use 2.0 b/c nvidia-docker-compose doesn't support it. Because we only use nvidia-docker-compose to generate a nvidia-docker-compose.yml file from the original docker-compose.yml file you provided, you can skip this and manually add all of the GPU and the driver volume.

   sudo apt-get install nvidia-docker
   

   this should install the 1.0.1 version. double check the version with

   nvidia-docker version
   

2. now under your project folder where there is your docker-compose.yml file, run

   nvidia-docker-compose -G 
   

   now you can collect your new nvidia-docker-compose.yml config file. It should look like something like this:

   version: '2.3'
   volumes:
     nvidia_driver_384.130:
       driver: nvidia-docker
       external: true
   services:
     main:
       build: .
       devices:
       - /dev/nvidia0
       - /dev/nvidia1
       - /dev/nvidia2
       - /dev/nvidia3
       - /dev/nvidiactl
       - /dev/nvidia-uvm
       - /dev/nvidia-uvm-tools
       volumes:
       - .:/opt/project
       - nvidia_driver_384.130:/usr/local/nvidia:ro
   

3. Now you want to setup your default nvidia runtime. My /etc/docker/daemon.json file looks like this:

   {
       "debug": true,
       "hosts": ["tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"],
       "default-runtime": "nvidia",
       "runtimes": {
               "nvidia": {
                       "path": "/usr/bin/nvidia-container-runtime",
                       "runtimeArgs": []
               }
       }
   }
   

   Because on ubuntu (or other linux distros), the systemd config conflicts with the hosts field above, you want to follow the instruction here to over ride the default systemd configration:

   reference: https://docs.docker.com/config/daemon/

   vim /etc/systemd/system/docker.service.d/docker.conf
   

   and put in the following

   [Service]
   ExecStart=
   ExecStart=/usr/bin/dockerd
   

   Now reload the configuration:

   sudo systemctl daemon-reload
   

   and finally you can restart the docker service.

   sudo service docker restart
   

In the end you should get working code with tensorflow running fine.

[3XX0]

@episodeyang v1 is deprecated. Just use docker-compose with v2, see here and there

[DrStoop]

q&d-workaround: only set docker "default-runtime" to "nvidia"

...by adding line "default-runtime": "nvidia" to file /etc/docker/daemon.json

for me, actually only setting the docker "default-runtime" to "nvidia" restarting docker and proceeding with the pycharm (professional edition) docker integration tools seems to work out. Setting the default-runtime is done by adding the line "default-runtine": "nvidia" to the /etc/docker/daemon.json-file so with nvidia-docker installed the file may look like this:

{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

restarting docker by e.g. typing systemctl restart docker in the command line, possibly reload daemon-config before systemctl daemon-reload.

i have gpu acceleration on docker & docker-compose projects in pycharm using e.g. an official NVIDIA-image such as nvcr.io/nvidia/pytorch:18.07-py3 (from NVIDIA GPU CLOUD) and testing nvidia-gpu-acceleration with a test-script for pytorch:

import torch
num_devices = torch.cuda.device_count()
if num_devices > 0:
    device_id = torch.cuda.current_device()
    print(torch.cuda.device(device_id))
    print(torch.cuda.get_device_name(device_id))
else:
    print('no gpu-device attached')

this should return your gpu model name. i'm a newbie regarding nvidia-/docker, still:

• this seems to work for the moment
• normal docker-compose.yml work within my limited use cases
• non-NVIDIA-images with the "nvidia"-default-runtime-setting are not affected by the nvidia-runtime.

a short glance into nvidia-docker-script reveals that it only adds a --runtime="nvidia" to the docker run and docker create command, so this simple workaround may work for at least some use cases with pycharm. anyway i'll be happy to read any better/more generalized solutions...

[meredithheskethfortescue]

@DrStoop your workaround is great! But the code you posted is a redundant comma which took me hours to realize, why docker doesn't accept my config file. So for everyone encountering the same error: The correct /etc/docker/daemon.json file is:

{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

[DrStoop]

@DrStoop your workaround is great! But the code you posted is a redundant comma which took me hours to realize, why docker doesn't accept my config file. So for everyone encountering the same error: The correct /etc/docker/daemon.json file is:

{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

@meredithheskethfortescue ups! you're right, sorry for that! i corrected it to not waste somebody else's time, too :+1: thx for noticing :smiley:

[gofortargets]

q&d-workaround: only set docker "default-runtime" to "nvidia"

...by adding line "default-runtime": "nvidia" to file /etc/docker/daemon.json

for me, actually only setting the docker "default-runtime" to "nvidia" restarting docker and proceeding with the pycharm (professional edition) docker integration tools seems to work out. Setting the default-runtime is done by adding the line "default-runtine": "nvidia" to the /etc/docker/daemon.json-file so with nvidia-docker installed the file may look like this:

{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

restarting docker by e.g. typing systemctl restart docker in the command line, possibly reload daemon-config before systemctl daemon-reload.

i have gpu acceleration on docker & docker-compose projects in pycharm using e.g. an official NVIDIA-image such as nvcr.io/nvidia/pytorch:18.07-py3 (from NVIDIA GPU CLOUD) and testing nvidia-gpu-acceleration with a test-script for pytorch:

import torch
num_devices = torch.cuda.device_count()
if num_devices > 0:
    device_id = torch.cuda.current_device()
    print(torch.cuda.device(device_id))
    print(torch.cuda.get_device_name(device_id))
else:
    print('no gpu-device attached')

this should return your gpu model name. i'm a newbie regarding nvidia-/docker, still:

• this seems to work for the moment
• normal docker-compose.yml work within my limited use cases
• non-NVIDIA-images with the "nvidia"-default-runtime-setting are not affected by the nvidia-runtime.

a short glance into nvidia-docker-script reveals that it only adds a --runtime="nvidia" to the docker run and docker create command, so this simple workaround may work for at least some use cases with pycharm. anyway i'll be happy to read any better/more generalized solutions...

May I ask I will change this file in the server (holding Docker) or in the Docker container ?

[DrStoop]

@gofortargets

May I ask I will change this file in the server (holding Docker) or in the Docker container ?

you change it on the host machine, i.e. your server, on which the docker daemon is running that runs your container. in the docker container itself you won‘t have docker installed unless you installed it on purpose (e.g. in the Dockerfile)... so in the container you may even not find the /etc/docker/daemon.json path/file.

[gofortargets]

@gofortargets

May I ask I will change this file in the server (holding Docker) or in the Docker container ?

you change it on the host machine, i.e. your server, on which the docker daemon is running that runs your container. in the docker container itself you won‘t have docker installed unless you installed it on purpose (e.g. in the Dockerfile)... so in the container you may even not find the /etc/docker/daemon.json path/file.

Thank you for the answer. May I ask this solution is applicable to the case where I launch docker container by _docker_, not _nvidia-docker_ ?