Distribution: Can't push to registry backed by S3

[juanjoDiaz]

Docker version

Client:
Version: 17.03.1-ce
API version: 1.27
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:05:44 2017
OS/Arch: linux/amd64

Server:
Version: 17.03.1-ce
API version: 1.27 (minimum version 1.12)
Go version: go1.7.5
Git commit: c6d412e
Built: Mon Mar 27 17:05:44 2017
OS/Arch: linux/amd64
Experimental: false

Issue

I've setup a private registry running on kubernetes and backed by AWS S3.
I'm trying to push an image there (https://hub.docker.com/r/consol/ubuntu-xfce-vnc/ just for testing).
The first few layers are uploaded correctly but the next layers get stuck on re-try for a while until docker eventually gives up and return EOF.

The push refers to a repository [myregistry.com/test]
787ee86c9c3c: Layer already exists 
7f7aa74ef32d: Layer already exists 
111b115eeb06: Layer already exists 
4f8f4ef6462d: Layer already exists 
a407aa97bd4b: Pushing [==================================================>]   272 MB/272 MB
a9522ad33290: Pushing [=======>                                           ] 63.68 MB/424.6 MB
0d9832d3492f: Pushing [===============================>                   ] 121.9 MB/193.6 MB
8cba1f63388c: Pushing [==================================================>] 2.141 MB
3ca6048e825e: Pushing [==================================================>] 20.74 MB
93764d3b0372: Waiting 
44246f27dbf8: Waiting 
c1ef174ccf26: Waiting 
31e3d470e01b: Waiting 
6f0af88b834b: Waiting 
56827159aa8b: Waiting 
440e02c3dcde: Waiting 
29660d0e5bb2: Waiting 
85782553e37a: Waiting 
745f5be9952c: Waiting 
EOF

This are the logs from the registry server

2017-05-31T13:57:44.856514236Z time="2017-05-31T13:57:44Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=79644932-6924-432e-8fc6-2de183b5f344 http.request.method=GET http.request.remoteaddr="X.X.X.X:23652" http.request.uri="/v2/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=98.802039ms http.response.status=200 http.response.written=2 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:44.8565502Z X.X.X.X - - [31/May/2017:13:57:44 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:45.07746058Z time="2017-05-31T13:57:45Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:8d0a1a9d5246bb0be8d23650ff3a467d333d302b1563bfebc7b62d2e04fb41c9 err.message="blob unknown to registry" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=f00f820e-01c7-425e-abea-5d0f2ecfc3b6 http.request.method=HEAD http.request.remoteaddr="X.X.X.X:5047" http.request.uri="/v2/test/blobs/sha256:8d0a1a9d5246bb0be8d23650ff3a467d333d302b1563bfebc7b62d2e04fb41c9" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=120.454875ms http.response.status=404 http.response.written=157 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 vars.digest="sha256:8d0a1a9d5246bb0be8d23650ff3a467d333d302b1563bfebc7b62d2e04fb41c9" vars.name=test version=v2.6.1 
2017-05-31T13:57:45.07749334Z X.X.X.X - - [31/May/2017:13:57:44 +0000] "HEAD /v2/test/blobs/sha256:8d0a1a9d5246bb0be8d23650ff3a467d333d302b1563bfebc7b62d2e04fb41c9 HTTP/1.1" 404 157 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:45.353989351Z time="2017-05-31T13:57:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=affd84ca-fe10-4126-86b3-ee7efc88aa8e http.request.method=POST http.request.remoteaddr="100.80.103.0:14265" http.request.uri="/v2/test/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=271.450816ms http.response.status=202 http.response.written=0 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:45.354022284Z 100.80.103.0 - - [31/May/2017:13:57:45 +0000] "POST /v2/test/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:45.559602877Z time="2017-05-31T13:57:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=8f7912a6-a99b-461b-a220-7e62315ef534 http.request.method=HEAD http.request.remoteaddr="X.X.X.X:5043" http.request.uri="/v2/test/blobs/sha256:4f2b7f69da1cd09349773867217c9a2a9d1e2edd0abedd3f7706f97e525e97d0" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.contenttype="application/octet-stream" http.response.duration=694.474654ms http.response.status=307 http.response.written=0 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:45.559633217Z X.X.X.X - - [31/May/2017:13:57:44 +0000] "HEAD /v2/test/blobs/sha256:4f2b7f69da1cd09349773867217c9a2a9d1e2edd0abedd3f7706f97e525e97d0 HTTP/1.1" 307 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:45.560855688Z time="2017-05-31T13:57:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=85a364e8-0e09-44fb-9428-4f3ab6cc985e http.request.method=HEAD http.request.remoteaddr="X.X.X.X:5049" http.request.uri="/v2/test/blobs/sha256:3029d226b5f7ce8ecfda890f04bee95657adbf2349a0e77394b600d36d1fb8e2" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.contenttype="application/octet-stream" http.response.duration=692.267588ms http.response.status=307 http.response.written=0 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:45.560871365Z X.X.X.X - - [31/May/2017:13:57:44 +0000] "HEAD /v2/test/blobs/sha256:3029d226b5f7ce8ecfda890f04bee95657adbf2349a0e77394b600d36d1fb8e2 HTTP/1.1" 307 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:45.655692033Z time="2017-05-31T13:57:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=1169ede2-b1d5-4fb5-82c5-85112f74a44d http.request.method=HEAD http.request.remoteaddr="X.X.X.X:5041" http.request.uri="/v2/test/blobs/sha256:302be24432802ab310eb9c95bfee691dfb6027d076a98ba8b1a160c3b5c6cea3" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.contenttype="application/octet-stream" http.response.duration=788.820204ms http.response.status=307 http.response.written=0 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:45.655730921Z X.X.X.X - - [31/May/2017:13:57:44 +0000] "HEAD /v2/test/blobs/sha256:302be24432802ab310eb9c95bfee691dfb6027d076a98ba8b1a160c3b5c6cea3 HTTP/1.1" 307 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:45.661644606Z time="2017-05-31T13:57:45Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:3c18f54ebd3a80ee75d8756b4a1ac14ac443f68228462f68fc84dc259f9c8793 err.message="blob unknown to registry" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=69b51d30-316c-4e03-84e2-8c5aed94e8d7 http.request.method=HEAD http.request.remoteaddr="X.X.X.X:5055" http.request.uri="/v2/test/blobs/sha256:3c18f54ebd3a80ee75d8756b4a1ac14ac443f68228462f68fc84dc259f9c8793" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=7.819446ms http.response.status=404 http.response.written=157 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 vars.digest="sha256:3c18f54ebd3a80ee75d8756b4a1ac14ac443f68228462f68fc84dc259f9c8793" vars.name=test version=v2.6.1 
2017-05-31T13:57:45.661670675Z X.X.X.X - - [31/May/2017:13:57:45 +0000] "HEAD /v2/test/blobs/sha256:3c18f54ebd3a80ee75d8756b4a1ac14ac443f68228462f68fc84dc259f9c8793 HTTP/1.1" 404 157 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:45.668833447Z time="2017-05-31T13:57:45Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:9988b6c8fd19bd97dbf91a092b18ab592e389f44227d5d828a9466903744020e err.message="blob unknown to registry" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=abe1280c-7cf5-4b6c-b26a-38f7a0e4aeb4 http.request.method=HEAD http.request.remoteaddr="X.X.X.X:23656" http.request.uri="/v2/test/blobs/sha256:9988b6c8fd19bd97dbf91a092b18ab592e389f44227d5d828a9466903744020e" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=12.187335ms http.response.status=404 http.response.written=157 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 vars.digest="sha256:9988b6c8fd19bd97dbf91a092b18ab592e389f44227d5d828a9466903744020e" vars.name=test version=v2.6.1 
2017-05-31T13:57:45.668856356Z X.X.X.X - - [31/May/2017:13:57:45 +0000] "HEAD /v2/test/blobs/sha256:9988b6c8fd19bd97dbf91a092b18ab592e389f44227d5d828a9466903744020e HTTP/1.1" 404 157 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:45.754718907Z time="2017-05-31T13:57:45Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=da506f09-e7c3-482c-b43a-726e164225b6 http.request.method=HEAD http.request.remoteaddr="X.X.X.X:5045" http.request.uri="/v2/test/blobs/sha256:54ac63baef0b86c94740d2e3ac8078086817b7c5c93f86ba07fa449c3a05de02" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.contenttype="application/octet-stream" http.response.duration=801.201654ms http.response.status=307 http.response.written=0 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:45.754760333Z X.X.X.X - - [31/May/2017:13:57:44 +0000] "HEAD /v2/test/blobs/sha256:54ac63baef0b86c94740d2e3ac8078086817b7c5c93f86ba07fa449c3a05de02 HTTP/1.1" 307 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:46.157298297Z time="2017-05-31T13:57:46Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:f556b780f7d99746cd21e638623ea357fa625a1769bcff5a9c0d79da91cf1d08 err.message="blob unknown to registry" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=91fc0976-7c52-4992-b1e9-11826cc499ae http.request.method=HEAD http.request.remoteaddr="X.X.X.X:5059" http.request.uri="/v2/test/blobs/sha256:f556b780f7d99746cd21e638623ea357fa625a1769bcff5a9c0d79da91cf1d08" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=292.985183ms http.response.status=404 http.response.written=157 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 vars.digest="sha256:f556b780f7d99746cd21e638623ea357fa625a1769bcff5a9c0d79da91cf1d08" vars.name=test version=v2.6.1 
2017-05-31T13:57:46.157329202Z X.X.X.X - - [31/May/2017:13:57:45 +0000] "HEAD /v2/test/blobs/sha256:f556b780f7d99746cd21e638623ea357fa625a1769bcff5a9c0d79da91cf1d08 HTTP/1.1" 404 157 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:46.158194257Z time="2017-05-31T13:57:46Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=61d8e6ba-4e3b-4453-b248-24206ffb34ff http.request.method=POST http.request.remoteaddr="100.80.103.0:14269" http.request.uri="/v2/test/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=492.336302ms http.response.status=202 http.response.written=0 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:46.158209037Z 100.80.103.0 - - [31/May/2017:13:57:45 +0000] "POST /v2/test/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:46.358539516Z time="2017-05-31T13:57:46Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=16092c7f-0d20-4174-969b-d14d9c0e4b1a http.request.method=POST http.request.remoteaddr="X.X.X.X:23660" http.request.uri="/v2/test/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=596.467524ms http.response.status=202 http.response.written=0 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:46.358577759Z X.X.X.X - - [31/May/2017:13:57:45 +0000] "POST /v2/test/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:46.658397935Z time="2017-05-31T13:57:46Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=feb7f193-84de-4f17-9a4c-3d8869fdcc83 http.request.method=POST http.request.remoteaddr="X.X.X.X:5057" http.request.uri="/v2/test/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=796.03513ms http.response.status=202 http.response.written=0 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:46.658428393Z X.X.X.X - - [31/May/2017:13:57:45 +0000] "POST /v2/test/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:46.859524586Z time="2017-05-31T13:57:46Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=060c8de7-2f33-4a69-b136-2737bfa468d4 http.request.method=POST http.request.remoteaddr="X.X.X.X:23664" http.request.uri="/v2/test/blobs/uploads/" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=503.317586ms http.response.status=202 http.response.written=0 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:46.859570134Z X.X.X.X - - [31/May/2017:13:57:46 +0000] "POST /v2/test/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"
2017-05-31T13:57:51.755219256Z time="2017-05-31T13:57:51Z" level=info msg="response completed" go.version=go1.7.3 http.request.host=myregistry.com http.request.id=e303199c-cff7-4bab-972e-71a4a4738859 http.request.method=PATCH http.request.remoteaddr="X.X.X.X:5061" http.request.uri="/v2/test/blobs/uploads/7d1d1999-2ebe-4d81-a8b1-3d83caca0c04?_state=97_bi0a17gimVqOaXQrJFvzX149K6rxUOic1I-Bfhnl7Ik5hbWUiOiJ0ZXN0IiwiVVVJRCI6IjdkMWQxOTk5LTJlYmUtNGQ4MS1hOGIxLTNkODNjYWNhMGMwNCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNy0wNS0zMVQxMzo1Nzo0NS43NjMwOTgxODJaIn0%3D" http.request.useragent="docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))" http.response.duration=5.098547023s http.response.status=202 http.response.written=0 instance.id=7a9d7c30-bdc6-4c63-a68d-46e57d86f591 version=v2.6.1 
2017-05-31T13:57:51.755256952Z X.X.X.X - - [31/May/2017:13:57:46 +0000] "PATCH /v2/test/blobs/uploads/7d1d1999-2ebe-4d81-a8b1-3d83caca0c04?_state=97_bi0a17gimVqOaXQrJFvzX149K6rxUOic1I-Bfhnl7Ik5hbWUiOiJ0ZXN0IiwiVVVJRCI6IjdkMWQxOTk5LTJlYmUtNGQ4MS1hOGIxLTNkODNjYWNhMGMwNCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNy0wNS0zMVQxMzo1Nzo0NS43NjMwOTgxODJaIn0%3D HTTP/1.1" 202 0 "" "docker/17.03.1-ce go/go1.7.5 git-commit/c6d412e kernel/3.10.0-514.21.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/17.03.1-ce \\(linux\\))"

and the logs from the docker daemon:

May 31 13:57:44 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:57:44.752097909Z" level=error msg="Attempting next endpoint for push after error: Get https://myregistry.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
May 31 13:57:53 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:57:53.191427488Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:57:53 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:57:53.209347856Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:57:53 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:57:53.210428799Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:57:53 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:57:53.211665609Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:57:53 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:57:53.253356548Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:58:57 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:58:57.593658846Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:58:57 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:58:57.593654786Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:58:58 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:58:58.184941938Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:58:58 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:58:58.185001880Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:58:58 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:58:58.184950659Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:59:18 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:59:18.065916214Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:59:18 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:59:18.065996825Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:59:18 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:59:18.096494149Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:59:18 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:59:18.096600227Z" level=error msg="Upload failed, retrying: EOF"
May 31 13:59:18 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T13:59:18.097040289Z" level=error msg="Upload failed, retrying: EOF"
May 31 14:00:32 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T14:00:32.184057682Z" level=error msg="Upload failed, retrying: EOF"
May 31 14:00:32 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T14:00:32.184093776Z" level=error msg="Upload failed, retrying: EOF"
May 31 14:00:32 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T14:00:32.184166437Z" level=error msg="Upload failed, retrying: EOF"
May 31 14:00:32 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T14:00:32.593863757Z" level=error msg="Upload failed, retrying: EOF"
May 31 14:00:32 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T14:00:32.593861642Z" level=error msg="Upload failed, retrying: EOF"
May 31 14:01:01 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T14:01:01.123045638Z" level=error msg="Upload failed: EOF"
May 31 14:01:01 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T14:01:01.133422066Z" level=error msg="Upload failed: EOF"
May 31 14:01:01 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T14:01:01.133634352Z" level=error msg="Upload failed: EOF"
May 31 14:01:01 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T14:01:01.135551745Z" level=error msg="Upload failed: EOF"
May 31 14:01:01 XXXXXXXXXXl dockerd[3231]: time="2017-05-31T14:01:01.135901861Z" level=error msg="Attempting next endpoint for push after error: EOF"

If I let the push run, it eventually kills the registry.

Any idea of what might wrong???

[dmcgowan]

Not a lot to go on. The docker message indicates it is a connection problem between the docker host and registry.

Get https://myregistry.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I would start there with investigating why that timeout occurred.

[juanjoDiaz]

Hi Dereck,

Thanks for the response!

My understanding was that after docker fails to connect to https it tries http for registries configured as insecure on daemon.json, which is the case.

It is certainly hitting the server, as shown by the server logs and by the fact that it detects that the first layers are already uploaded.

Or am I missing something here?

[juanjoDiaz]

Just to elaborate on this. I just tried pushing and pulling small images (alpine, nginx, etc.) and it worked just fine.
Only layers in the order of hundreds of MBs fail so it seems that it has something to do with the size of the layers.

[juanjoDiaz]

So it tuned out that the issues was to do with CPU/memory limits on the registry.
On the default registry config from kubernetes documentation, the limits are:

          limits:
            cpu: 100m
            memory: 100Mi

raising those to

          limits:
            cpu: 300m
            memory: 300Mi

fixed my issue.

Is there any documentation of the minimum requirements for the registry??
In any case, I still that it should be considered a bug that the whole thing just fail silently and without any valid error.