Docker-py: Issue with creating a docker container while inside a docker container

[EvanBaum1]

When attempting to create a docker container while in a docker container, permission is denied in accessing the /var/lib/docker.sock socket.

Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.5/http/client.py", line 1106, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
self.endheaders(body)
File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
self.send(msg)
File "/usr/lib/python3.5/http/client.py", line 877, in send
self.connect()
File "/usr/local/lib/python3.5/dist-packages/docker/transport/unixconn.py", line 42, in connect
sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

[shin-]

You probably need this: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

[EvanBaum1]

This was attempted, with no change.

[TomasTomecek]

We are doing this all the time and it works just fine.

Since you haven't posted any details it's really hard to help you. What is your container environment? What user do you invoke the container as? What are versions of the software you are using?

Proof:

$ docker run -ti -v /var/run/docker.sock:/var/run/docker.sock fedora:28 bash

[root@9028cd12db44 /]# pip3 install docker
...
Successfully installed certifi-2018.4.16 chardet-3.0.4 docker-3.4.1 docker-pycreds-0.3.0 idna-2.7 requests-2.19.1 urllib3-1.23 websocket-client-0.48.0

[root@9028cd12db44 /]# python3
Python 3.6.5 (default, Mar 29 2018, 18:20:46)
[GCC 8.0.1 20180317 (Red Hat 8.0.1-0.19)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
>>> d = docker.APIClient()
>>> c = d.create_container("fedora:28", "ls")
>>> c.start()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'dict' object has no attribute 'start'
>>> d.start(c)
>>> d.logs(c)
b'bin\nboot\ndev\netc\nhome\nlib\nlib64\nlost+found\nmedia\nmnt\nopt\nproc\nroot\nrun\nsbin\nsrv\nsys\ntmp\nusr\nvar\n'

[EvanBaum1]

This is being run on a secure MapR 6.0.1 cluster. The run file is as follows:
docker run --privileged -it -v /tmp/ticket:/tmp/mapr_ticket:ro -v /var/run/docker.sock:/var/run/docker.sock:ro --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --device /dev/fuse --rm --env-file env_file.env --network data --name service_service service
The code is running on:
Python 3.6.4 (default, May 11 2018, 14:19:50) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux Type "help", "copyright", "credits" or "license" for more information.
the Dockerfile is as follows:
`
FROM pacc_kafka

RUN mkdir /app || true

WORKDIR /app

COPY requirements.txt /app/requirements.txt
COPY requirements_sog.txt /app/requirements_sog.txt
RUN pip3 install -r requirements.txt && pip3 install -r requirements_sog.txt && rm /app/requirements.txt && rm /app/requirements_sog.txt

COPY main.py /app/main.py
COPY Service.py /app/Service.py
COPY config.yml /app/config.yml
COPY consumerConfig.txt /app/consumerConfig.txt
COPY consumerService/ /app/consumerService/

CMD cd /app && export LD_LIBRARY_PATH=/opt/mapr/lib && ln -s /opt/mapr/lib/libMapRClient.so /opt/mapr/lib/libMapRClient_c.so | true && LD_PRELOAD=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so python3 /app/main.py #command line inputs here if required
`

I am able to run the line that says
self.client = docker.from_env()
However it errors when I try to do anything to the self.client. It will also error out when
docker.from_env().info()
Is called.
The error I posted is the result of this call.

[TomasTomecek]

Since you are getting "permission denied" it's clear that you are not allowed to write to the docker.sock from the container.
What's ls -lha /var/run/docker.sock and id in your container?

Edit: mine is:

[root@bffafe6c802d /]# id
uid=0(root) gid=0(root) groups=0(root)
[root@bffafe6c802d /]# ls -lha /var/run/docker.sock 
srw-rw----. 1 root 1001 0 Jul 30 07:57 /var/run/docker.sock

[EvanBaum1]

This is the permissions, where the 'docker' user contains the user 'mapr' which is where I'm running from.
srw-rw---- 1 root docker 0 Aug 2 15:01 /var/run/docker.sock
I'm not sure which you mean by id in my container.
Edit: I see. Here is the id
uid=5000(mapr) gid=5000(mapr) groups=5000(mapr),0(root),993(docker),5002(shadow)

[EvanBaum1]

Figured out the issue. The MapR PACC wasn't able to access things that weren't owned by the user, regardless of permissions.

[ctivanovich]

Yeah, I am running into this issue as well, attempting to launch containers using a containerized version of Airflow (via the docker hook which relies almost entirely on docker.py). I follow instructions elsewhere when setting up my Airflow container to make sure it has a group called docker with a gid matching that of the one in the host system, which should enable it to write to the system's socket, but....permission denied. The only thing that has worked, which kind of scares me, has been to chmod the socket to be writable by all users.

I'm not quite sure why this is needed. I have a system user called airflow, which is also a member of the docker group. An open question for me is whether the writing entity to the socket is root (because it's a container doing the writing) or airflow (because it is the user airflow within the container that would be accessing the mounted socket within the container). Is some executable called docker writing to the socket, or since it is mounted, is it the user within the container writing, in which case that user also needs to exist in the host environment?

[jelther]

Yeah, I am running into this issue as well, attempting to launch containers using a containerized version of Airflow (via the docker hook which relies almost entirely on docker.py). I follow instructions elsewhere when setting up my Airflow container to make sure it has a group called docker with a gid matching that of the one in the host system, which should enable it to write to the system's socket, but....permission denied. The only thing that has worked, which kind of scares me, has been to chmod the socket to be writable by all users.

I'm not quite sure why this is needed. I have a system user called airflow, which is also a member of the docker group. An open question for me is whether the writing entity to the socket is root (because it's a container doing the writing) or airflow (because it is the user airflow within the container that would be accessing the mounted socket within the container). Is some executable called docker writing to the socket, or since it is mounted, is it the user within the container writing, in which case that user also needs to exist in the host environment?

I was able to fix that using the z flag when mounting the volume inside the docker container.
Also, upgraded docker version to the latest one available to avoid any hassle.