Docker-py: containers.run() hangs without output when image is missing (+/- invalid mount)

Created on 16 Feb 2019  ·  7Comments  ·  Source: docker/docker-py

Hi all,

First of all thanks for the continued work on this SDK -- it's been very useful for making quick work of some local e2e tests. I ran into this problem while a colleague was trying to run some tests that "ran on my machine".

Issue

TL;DR - If you do not have the image you want to run pulled (for example minio/minio), and you have a configuration error (such as an invalid mount specified), the containers.run command will hang (due to one or more silent failures) -- the container will never be pulled, the program will not continue past or error, and the underlying mount problem won't cause an error either -- to the end user it just looks like a hang with no output.

The first progress we were able to make was installing the minio/minio image with the docker CLI directly -- this got rid of the missing image problem (maybe docker-py doesn't fetch missing automatically, I thought this was docker run standard behavior), and exposed an invalid mount configuration issue. The mount configuration issue has to do with how OSX temp directories are created (once the error was visible it was easy to fix):

E       docker.errors.APIError: 502 Server Error: Bad Gateway ("b'Mounts denied: \r\nThe path /var/folders/wj/x6b1l_7s1yg977rxrrjpsc8h0000gn/T/project-9dk0ewdx\r\nis not shared from OS X and is not known to Docker.\r\nYou can configure shared paths from Docker -> Preferences... -> File Sharing.\r\nSee https://docs.docker.com/docker-for-mac/osxfs/#namespaces for more info.\r\n.'")

this issue is due to namespacing on OSX, and the interaction with python's tempfile functionality:

import tempfile

# Create tmp local directory for minio to use                                                                                                                                                                                                                                 
d = tempfile.mkdtemp(prefix="e2e-minio") # /var/folders/wj/x6b1l_7s1yg977rxrrjpsc8h0000gn/T/project-minio9dk0ewdx

The fact that this folder can't be used by the spawned docker container is more of a gotcha more than a bug (maybe both of these issues are), so I'm not sure exactly what should be done (maybe it's just adding more documentation).

The main issue is the missing container appearing as a hang (which might have been exacerbated by the invalid mount issue), with no output (error or otherwise) returned to the user.

Reproduction

Steps:

  1. Install docker (native) 18.02 (latest as of 02/16) on OSX
  2. Create a python project that uses tmpfile.mkdtemp
  3. Create a client with docker.from_env()
  4. Run docker_client.containers.run w/ a mounted directory that is the temp directory
picture t3hmrman

Most helpful comment

Thanks for this, new to Docker and this took way to much time to figure out. Seems a simple update to the docs is in order https://docs.docker.com/engine/api/sdk/examples/

import docker
client = docker.from_env()
print client.containers.run("alpine", ["echo", "hello", "world"])

to 

import docker
client = docker.from_env()
print client.containers.run("alpine:latest", ["echo", "hello", "world"])

Edit:
Further to the above, pulling a remote image hangs even with the above. Logging in first works

r = client.login(username="xxxx", password="yyyy")
picture DanielSCrouch on 26 Jun 2020
👍2

All 7 comments

I can reproduce this w/ docker-py 4.1.0 on linux without any mounts or temp files:

❯ python
Python 2.7.15+ (default, Oct  7 2019, 17:39:04)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
>>> docker.from_env().containers.run('ubuntu', 'ls')

wait for a few minutes... nothing happens... then I hit ^C to get this stack trace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/radix/.virtualenvs/devlandia/local/lib/python2.7/site-packages/docker/models/containers.py", line 805, in run
    self.client.images.pull(image, platform=platform)
  File "/home/radix/.virtualenvs/devlandia/local/lib/python2.7/site-packages/docker/models/images.py", line 447, in pull
    for _ in pull_log:
  File "/home/radix/.virtualenvs/devlandia/local/lib/python2.7/site-packages/docker/api/client.py", line 345, in _stream_helper
    data = reader.read(1)
  File "/home/radix/.virtualenvs/devlandia/local/lib/python2.7/site-packages/urllib3/response.py", line 444, in read       data = self._fp.read(amt)
  File "/usr/lib/python2.7/httplib.py", line 583, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 625, in _read_chunked
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
KeyboardInterrupt

turns out that this also happens when I just try to pull explicitly:

>>> docker.from_env().images.pull('ubuntu')

ctrl-C:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/radix/.virtualenvs/devlandia/local/lib/python2.7/site-packages/docker/models/images.py", line 447, in pull
    for _ in pull_log:
  File "/home/radix/.virtualenvs/devlandia/local/lib/python2.7/site-packages/docker/api/client.py", line 345, in _stream_helper
    data = reader.read(1)
  File "/home/radix/.virtualenvs/devlandia/local/lib/python2.7/site-packages/urllib3/response.py", line 444, in read
    data = self._fp.read(amt)
  File "/usr/lib/python2.7/httplib.py", line 583, in read
    return self._read_chunked(amt)
  File "/usr/lib/python2.7/httplib.py", line 625, in _read_chunked
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
KeyboardInterrupt
radix picture radix on 22 Nov 2019

Ok, it seems that my docker.from_env.images.pull('ubuntu') is trying to download ALL of the ubuntu images. Same with containers.run. This is ONLY happening when there are no matching images in the first place.

with a containers.run('ubuntu', 'ls') running in another terminal...

❯ docker images|grep ubuntu
ubuntu                                                            12.04               5b117edd0b76        2 years ago         104MB
ubuntu                                                            12.04.5             5b117edd0b76        2 years ago         104MB
ubuntu                                                            10.04               e21dbcc7c9de        5 years ago         183MB

❯ docker images|grep ubuntu
ubuntu                                                            12.04               5b117edd0b76        2 years ago         104MB
ubuntu                                                            12.04.5             5b117edd0b76        2 years ago         104MB
ubuntu                                                            12.10               3e314f95dcac        5 years ago         172MB
ubuntu                                                            13.04               a58cd502f927        5 years ago         169MB
ubuntu                                                            10.04               e21dbcc7c9de        5 years ago         183MB

❯ docker images|grep ubuntu
ubuntu                                                            12.04               5b117edd0b76        2 years ago         104MB
ubuntu                                                            12.04.5             5b117edd0b76        2 years ago         104MB
ubuntu                                                            12.10               3e314f95dcac        5 years ago         172MB
ubuntu                                                            13.04               a58cd502f927        5 years ago         169MB
ubuntu                                                            13.10               7f020f7bf345        5 years ago         185MB
ubuntu                                                            10.04               e21dbcc7c9de        5 years ago         183MB

it keeps growing, and growing...

radix picture radix on 22 Nov 2019

If the default is for docker-py to download all tags from images (if it can't find the image locally), then that would certainly explain it

t3hmrman picture t3hmrman on 25 Nov 2019

If the default is for docker-py to download all tags from images (if it can't find the image locally), then that would certainly explain it

That's right.
According to docker engine documentation, if no tag is defined, it will pull all the images.

Tag or digest. If empty when pulling an image, this causes all tags for the given image to be pulled.

https://docs.docker.com/engine/api/v1.40/#operation/ImageCreate

feliperuhland picture feliperuhland on 27 Nov 2019
👍1

Thanks for this, new to Docker and this took way to much time to figure out. Seems a simple update to the docs is in order https://docs.docker.com/engine/api/sdk/examples/

import docker
client = docker.from_env()
print client.containers.run("alpine", ["echo", "hello", "world"])

to 

import docker
client = docker.from_env()
print client.containers.run("alpine:latest", ["echo", "hello", "world"])

Edit:
Further to the above, pulling a remote image hangs even with the above. Logging in first works

r = client.login(username="xxxx", password="yyyy")
DanielSCrouch picture DanielSCrouch on 26 Jun 2020
👍2

Thanks to the docs. I pulled entire library of ubuntu image.

shinebayar-g picture shinebayar-g on 3 Oct 2020
😄1

@shinebayar-g Me too!

svkers picture svkers on 11 Oct 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

uggla picture uggla  ·  3Comments
ko-christ picture ko-christ  ·  3Comments
jamesongithub picture jamesongithub  ·  5Comments
vickydgm picture vickydgm  ·  4Comments
pavdmyt picture pavdmyt  ·  3Comments