Compose: socket.timeout: timed out on compose push

Created on 7 Aug 2019  路  7Comments  路  Source: docker/compose

Description of the issue

python trace weh pushing to registry

Context information (for bug reports)

we use docker-compose to push build images via jenkins CI to aws ecr

time docker-compose -f .jenkins/docker-compose.images-base.1.yml push

Output of docker-compose version

docker-compose version 1.24.1, build 4667896
docker-py version: 3.7.3
CPython version: 3.7.3
OpenSSL version: OpenSSL 1.1.1b  26 Feb 2019```

**Output of `docker version`**

Client: Docker Engine - Community
Version: 19.03.1
API version: 1.40
Go version: go1.12.5
Git commit: 74b1e89
Built: Thu Jul 25 21:21:22 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.1
API version: 1.40 (minimum version 1.12)
Go version: go1.12.5
Git commit: 74b1e89
Built: Thu Jul 25 21:19:53 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.6
GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683```

python version
Python 3.7.3

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

version: "3.7"

services:
  php70:
    image: ${AWS_ECR_ACCOUNT:-REDACTED}.dkr.ecr.eu-west-1.amazonaws.com/php:php70${COMMIT_ID}
    build:
      context: ../infrastructure-as-code/Docker/php
      dockerfile: Dockerfile-7.0

Steps to reproduce the issue

  1. time docker-compose -f .jenkins/docker-compose.images-base.1.yml push

Observed result

python trace

Expected result

Pushing to ECR
Pushing php70 (REDACTED.dkr.ecr.eu-west-1.amazonaws.com/php70-6d7b27e)...
The push refers to repository [REDACTED.dkr.ecr.eu-west-1.amazonaws.com/php]
php70-6d7b27e: digest: sha256:0c15dbf77f8c98acfde6fad306908c8ce9a71be23bb060bb1b0a5738c4856298 size: 5123

Stacktrace / full error message

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 362, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 444, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.7/http/client.py", line 447, in read
    n = self.readinto(b)
  File "/usr/lib/python3.7/http/client.py", line 481, in readinto
    return self._readinto_chunked(b)
  File "/usr/lib/python3.7/http/client.py", line 576, in _readinto_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/lib/python3.7/http/client.py", line 544, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/lib/python3.7/http/client.py", line 504, in _read_next_chunk_size
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/docker-compose", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 71, in main
    command()
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 127, in perform_command
    handler(command, command_options)
  File "/usr/local/lib/python3.7/dist-packages/compose/cli/main.py", line 776, in push
    ignore_push_failures=options.get('--ignore-push-failures')
  File "/usr/local/lib/python3.7/dist-packages/compose/project.py", line 654, in push
    service.push(ignore_push_failures)
  File "/usr/local/lib/python3.7/dist-packages/compose/service.py", line 1235, in push
    stream_output(output, sys.stdout))
  File "/usr/local/lib/python3.7/dist-packages/compose/progress_stream.py", line 112, in get_digest_from_push
    for event in events:
  File "/usr/local/lib/python3.7/dist-packages/compose/progress_stream.py", line 25, in stream_output
    for event in utils.json_stream(output):
  File "/usr/local/lib/python3.7/dist-packages/compose/utils.py", line 61, in split_buffer
    for data in stream_as_text(stream):
  File "/usr/local/lib/python3.7/dist-packages/compose/utils.py", line 37, in stream_as_text
    for data in stream:
  File "/usr/local/lib/python3.7/dist-packages/docker/api/client.py", line 345, in _stream_helper
    data = reader.read(1)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 461, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 367, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
urllib3.exceptions.ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out.

Additional information

OS version / distribution, docker-compose install method, etc.
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.04
Release: 19.04
Codename: disco

pip3 install \
wheel \
docker-compose \
awscli

stale

Most helpful comment

@nkazakov Hello could you explain me better how to set the environment variables? my problem is with docker-compose up

@brunosousadev Hello. You should create file .env (or add lines to existed file) with this parameters:

DOCKER_CLIENT_TIMEOUT=180
COMPOSE_HTTP_TIMEOUT=180

Time in seconds.

Or you can set these parameters to global environment variables of your OS.

All 7 comments

This also happened to me quite a few times but always assumed it was related with the connection, the push is made from a VM running in US on GCE to a ECR in Europe, in general after a retry it worked. I suspect this issue might also be related to a timeout threshold in docker-compose or the docker engine.

I am seeing a similar issue pushing from a EKS container to Docker Hub. Also seeing this in some system logs: "Not continuing with push after error: context canceled"

I had the same problem with docker-compose push
You can set environment variables with your values:

DOCKER_CLIENT_TIMEOUT: 180
COMPOSE_HTTP_TIMEOUT: 180

It helped me.

@nkazakov Hello could you explain me better how to set the environment variables? my problem is with docker-compose up

@nkazakov Hello could you explain me better how to set the environment variables? my problem is with docker-compose up

@brunosousadev Hello. You should create file .env (or add lines to existed file) with this parameters:

DOCKER_CLIENT_TIMEOUT=180
COMPOSE_HTTP_TIMEOUT=180

Time in seconds.

Or you can set these parameters to global environment variables of your OS.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically closed because it had not recent activity during the stale period.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

29e7e280-0d1c-4bba-98fe-f7cd3ca7500a picture 29e7e280-0d1c-4bba-98fe-f7cd3ca7500a  路  3Comments

squeaky-pl picture squeaky-pl  路  3Comments

dimsav picture dimsav  路  3Comments

saulshanabrook picture saulshanabrook  路  3Comments

Hendrik-H picture Hendrik-H  路  3Comments