Compose: docker-compose build fails with "couldn't connect to http+docker://localunixsocket" error

[rajeshyogeshwar]

I have a somewhat peculiar scenario. I tend to execute command docker-compose -f dev.yml up --build to get the containers up and work. Here, dev.yml is development version of docker-compose.yml. Till this very morning every thing was working fine, and all of the sudden, I started getting error regarding failure to connect to docker-daemon.

Now, this problem only occurs when I have --build included in the command. If I only execute docker-compose -f dev.yml up it works fine. If I include --build into the command and execute it using sudo it works fine again.

OS: Ubuntu 16.04 LTS
Docker Engine: 17.03 CE
Docker-Compose: 1.11.2

Here is the dev.yml

version: '2'

volumes:
  postgres_data_dev: {}
  postgres_backup_dev: {}

services:
  postgres:
    build: ./compose/postgres
    volumes:
      - postgres_data_dev:/var/lib/postgresql/data
      - postgres_backup_dev:/backups
    environment:
      - POSTGRES_USER=rocky


  django:
    build:
      context: .
      dockerfile: ./compose/django/development/Dockerfile
    depends_on:
      - postgres
    environment:
      - POSTGRES_USER=rhombus
      - USE_DOCKER=yes
    volumes:
      - .:/app
      - /tmp/
    links:
      - postgres
      - redis
    expose:
      - "8000"
    env_file:
      - ./dev.env


  nginx:
    build: 
      context: .
      dockerfile: ./compose/nginx/development/Dockerfile
    depends_on:
      - django
    ports:
      - "0.0.0.0:80:80"
    links:
      - django
    volumes_from:
      - django


  redis:
    image: redis:latest
    hostname: redis


  celeryworker:
    build:
      context: .
      dockerfile: ./compose/django/development/Dockerfile
    env_file: ./dev.env
    depends_on:
      - django
      - redis
      - postgres
    volumes_from:
      - django
    command: celery -A rhombus.taskapp worker


  celerybeat:
    build:
      context: .
      dockerfile: ./compose/django/development/Dockerfile
    env_file: ./dev.env
    depends_on:
      - django
      - redis
      - postgres
      - celeryworker
    volumes_from:
      - django
    command: celery -A rocky.taskapp beat

I am just caught unaware as to what might have caused for such a sudden failure to build containers? I do not wish to use sudo everytime.

[shin-]

Check the permissions on your project files? Did you upgrade Compose or Docker recently?

[rajeshyogeshwar]

Just checked, docker engine is updated to 17.05 and permissions look fine. Also, had it been permissions issue docker-compose up without sudo prefix should have also resulted in error, which is not the case as it runs smoothly.

Just out of curiosity, what happens with --build is appended to docker-compose up, probably the process itself can reveal some details?

[rajeshyogeshwar]

Update: Today my colleague encountered same issue. But his docker and compose versions were the same as mentioned in original issue.

[rajeshyogeshwar]

Another Update: Just for the sake of it, I removed the containers and tried to execute docker-compose -f dev.yml build and no surprise it failed with same error. I guess, the problem has to be somewhere when the containers are being built either via --build flag or build command.

Using sudo with it works fine, hence it does seem to be permissions issue but I have ownership of all the project related folder/files. Command execution output for files, which I found were related to the issue.

ls -lart /var/run/docker.sock
srw-rw---- 1 root docker 0 Jun 30 09:13 /var/run/docker.sock

ls -lart /var/run/docker/
drwx------ 2 root root 40 Jun 30 09:13 plugins drw------- 2 root root 60 Jun 30 09:14 libnetwork drwx------ 2 root root 40 Jun 30 09:14 swarm drwx------ 7 root root 140 Jun 30 09:14 . drwxr-xr-x 32 root root 1020 Jun 30 09:22 .. drwx------ 12 root root 300 Jun 30 10:33 libcontainerd drwxr-xr-x 2 root root 220 Jun 30 10:34 netns

[shin-]

@Rajesh-Yogeshwar What's the output of the id command? Can you make sure your user in the docker group?

[rajeshyogeshwar]

Output for id command.

uid=1000(houdini) gid=1000(houdini) groups=1000(houdini),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),113(lpadmin),128(sambashare),999(docker)

Verified that user does belong to docker group

[rajeshyogeshwar]

Another update.

We had docker 17.03CE on two of our machines, we updated them to docker 17.06CE and this same issue has resurfaced. So in essence, updating from previous version does cause some issue with permissions. Any insight into this will be greatly helpful.

[rajeshyogeshwar]

I have been now hit with same snag on my servers. I did not even change anything or update anything there.
Versions:
Docker version 17.04.0-ce, build 4845c56
docker-compose version 1.12.0, build b31ff33
Ubuntu 16.04LTS

It has virtually stopped me from building/rebuilding my containers. When I do docker-compose up or docker-compose up -d it works. But as soon as I add --build it stops with error.

[shin-]

So it only fails to connect when you attempt to issue a build?

1. Can you share the output of the following command: docker-compose --verbose build?
2. Does find -type p . in your application's root folder return anything?

[rajeshyogeshwar]

This is a stitched snapshot for output of docker-compose --verbose build

For find -type p, there was no output.

[shin-]

Looks like your first build goes through just fine, but the second one breaks for some reason. Can you share the contents of that Dockerfile? (The one in compose/django/development/)

[rajeshyogeshwar]

These are the contents of Dockerfile, on my server only change is in place of runserver. There I run uwsgi.

FROM python:3.5

ENV PYTHONUNBUFFERED 1

COPY ./requirements /requirements
RUN pip install -r /requirements/local.txt

COPY ./dev.env /dev.env

COPY ./compose/django/development/entrypoint.sh /entrypoint.sh
RUN sed -i 's/\r//' /entrypoint.sh
RUN chmod +x /entrypoint.sh

CMD ["python", "manage.py", "collectstatic", "--noinput", "--settings=config.settings.local"]

CMD ["python", "manage.py", "runserver_plus", "0.0.0.0:8000", "--insecure"]

WORKDIR /app

ENTRYPOINT ["/entrypoint.sh"]

These are the contents of entrypoint.sh

#!/bin/bash
set -e
cmd="$@"

export REDIS_URL=redis://redis:6379

if [ -z "$POSTGRES_USER" ]; then
    export POSTGRES_USER=postgres
fi

export DATABASE_URL=postgres://$POSTGRES_USER:$POSTGRES_PASSWORD@postgres:5432/$POSTGRES_USER

export CELERY_BROKER_URL=$REDIS_URL/0

function postgres_ready(){
python << END
import sys
import psycopg2
try:
    conn = psycopg2.connect(dbname="$POSTGRES_USER", user="$POSTGRES_USER", password="$POSTGRES_PASSWORD", host="postgres")
except psycopg2.OperationalError:
    sys.exit(-1)
sys.exit(0)
END
}

until postgres_ready; do
  >&2 echo "Postgres is unavailable - sleeping"
  sleep 1
done

>&2 echo "Postgres is up - continuing..."
exec $cmd

[rajeshyogeshwar]

@shin- have you been able to find our probable cause for this issue?

[shin-]

Sorry, I haven't had time to look at it closely yet. In the meantime, can you check your daemon logs while running the build and see if any error or crash happens in there?

[rajeshyogeshwar]

Jul 19 10:51:11 akshaypc dockerd[1680]: time="2017-07-19T10:51:11.776636322+05:30" level=error msg="Handler for POST /v1.22/build returned error: Error processing tar file(archive/tar: invalid tar header): "

This is the only error log that I found. If the tar file has anything to do with the requirements that I install into containers for my project, they are as follows.

• wheel==0.29.0
• django==1.11.2
• django-environ==0.4.1
• Pillow==4.0.0
• psycopg2==2.6.2
• pytz==2016.10
• django-redis==4.7.0
• redis>=2.10.5
• celery==4.0.2
• django-mptt==0.8.7
• elasticsearch
• pycrypto
• pysftp
• djangorestframework==3.6.3
• djangorestframework-jwt==1.11.0
• django-tenant-schemas==1.9.0
• pandas
• xlsxwriter
• xlrd
• uwsgi
• graypy
• ldap3
• lxml
• django-session-security
• asgi_redis
• channels
• tenant-schemas-celery
• reportlab
• django-celery-beat
• requests

[rajeshyogeshwar]

Heads up, I tried searching issues relating to msg="Handler for POST /v1.22/build returned error: Error processing tar file(archive/tar: invalid tar header): and found this issue #4437 to be helpful.

From there I could deduce that there are volumes from previous builds dangling and not being used. I removed all the dangling ones, and simply executed docker-compose -f dev.yml build --no-cache and the containers were built without any issue and without use of sudo command. But, its difficult for anyone new to docker to come to this conclusion.

I still have a question, volumes have data within it. Is it entirely safe to simple remove them? I am asking this because I have same issue on server and I want to play a safe game there.

Also, is there a possibility that when the containers are rebuilt the previously attached volumes are cleaned up before actual build begins and then the perform installations within the volume attached with new container, in effect trying to avert the whole scenario.

Yeah and I am not sure that this might be the actual cause or my solution is the right one. Hence, it still needs verification.

[rajeshyogeshwar]

Update: I tried same steps on my server for exact same issue (I checked logs for daemond). But the issue still persists.

[shin-]

@Rajesh-Yogeshwar If you're indeed running into the same issue as #4437, you may want to try running sudo chown $USER:$USER -R . at the root of your project and see if that fixes anything. I don't think removing volumes will help, actually.

[rajeshyogeshwar]

I am not really sure if its same issue. But yes the error "invalid tar header" does appear in logs for daemond. If root cause can be found for this error, may be that can help? Or rather when would this typically occur?

[rajeshyogeshwar]

@shin- Just tried the command you suggested and voila it worked. Just a question, why this command? I mean you surely must have pointed out probable issue. If you can be kind enough to divulge the details, it would be great to know, so that moving forward I can tackle this situations.

[theoden-dd]

I confirm this appearance of the bug and that sudo chown $USER:$USER -R . saves the day.

The only side-effect I know could be solved with the command are the *.pyc files with root/root owners. They are spawned when running container with a python/django app.

[rajeshyogeshwar]

@theoden-dd But the logs specifically mention invalid tar/header which led me to believe it must have been something to do with packages that might get installed on build.

[theoden-dd]

Yes, messages are absolutely misleading for this case.

On Sat, Jul 29, 2017 at 6:02 AM, Rajesh-Yogeshwar [e-mail]
wrote:

@theoden-dd https://github.com/theoden-dd But the logs specifically
mention invalid tar/header which led me to believe it must have been
something to do with packages that might get installed on build.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/4970#issuecomment-318802085,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACER8YgmJp02j7mg5LMFoDIiqlY1iYRTks5sSq7fgaJpZM4OH2lY
.

--
Sincerely,
Denis Untevskiy

[rajeshyogeshwar]

Well closing it then. It made me mad thinking what might be the issue :laughing: , but in the end something to learn and remember. Closing this issue.

[shin-]

Context of the build is sent to the engine as a tarfile. The error is produced by the engine when said context can not be read (corrupted archive). It's a big on Compose's end, but tricky to address.

[rajeshyogeshwar]

@shin- Noted. Thanks for your time in guiding to the solution

[maxim-avramenko]

@shin- sudo chown $USER:$USER -R .
:+1:

Тоже помогает:
sudo find . -user root -exec chown $USER {} \;
:+1:

[zipzit]

I had this issue on a vultr deployment. My fix was to add my login id as a superuser
$ su - username Note: root login access was removed per security recommendations.

[ppo]

I had the same issue… because the image name contained an uppercase letter! 😡

The docker-compose build error message was "Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?".
The docker-compose up -d error message was "no such image: … invalid reference format: repository name must be lowercase".

[haggen]

@shin- Shouldn't it be the case for displaying the appropriate error message in docker-compose? I mean, I fixed it by running a chown but the message imply there's a communication issue between compose and docker daemon. It sounds misleading at best.

[shin-]

@haggen It should be fixed in 1.19