Harbor: docker push always getting struck at EOF after multiple retries

[rizwan-kh]

docker info
_Containers: 9
Running: 9
Paused: 0
Stopped: 0
Images: 10
Server Version: 18.05.0-ce
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-693.11.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.638GiB
ID: GGT7:JMUS:KF3O:P2GS:5ZFE:CF5G:XLFO:YZR6:M6KV:3BS6:A4ZE:MM6N
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false_

docker-compose --version
docker-compose version 1.21.2, build a133471

Steps to reproduce:

docker login https://harbor.masked.com -u username
Login Succeeded
docker tag "registry.dev.masked.com/automation2/secret-service:v7.39" "harbor.masked.com/masked/secret-service:v7.39"
docker push "harbor.masked.com/masked/secret-service:v7.39"
The push refers to a repository [harbor.masked.com/masked/secret-service]
a379d11d0c1b: Preparing
7c014dbdd3b5: Preparing
00778a7df930: Preparing
7b7f45886fb4: Preparing
eb644e346134: Preparing
ec006cc6a5b3: Preparing
8195b3872690: Preparing
94d9b81d28c0: Preparing
37b97733984a: Preparing
4b9a18e50bbb: Preparing
bcc97fbfc9e1: Preparing
8195b3872690: Waiting
94d9b81d28c0: Waiting
bcc97fbfc9e1: Waiting
37b97733984a: Waiting
4b9a18e50bbb: Waiting
ec006cc6a5b3: Waiting
eb644e346134: Layer already exists
ec006cc6a5b3: Layer already exists
8195b3872690: Layer already exists
94d9b81d28c0: Layer already exists
37b97733984a: Layer already exists
4b9a18e50bbb: Layer already exists
bcc97fbfc9e1: Layer already exists
7b7f45886fb4: Retrying in 5 seconds
7c014dbdd3b5: Retrying in 5 seconds
00778a7df930: Retrying in 5 seconds
a379d11d0c1b: Retrying in 5 seconds
7b7f45886fb4: Retrying in 4 seconds
7c014dbdd3b5: Retrying in 4 seconds
...
00778a7df930: Retrying in 1 second
a379d11d0c1b: Retrying in 1 second
7b7f45886fb4: Retrying in 1 second
EOF

logs.zip

I searched for similar issue, but that one is closed and I couldn't solve using the solution provided there
https://github.com/vmware/harbor/issues/3753

[suppalapati13]

@rizwan-kh What is your storage backend? I have similar EOF issue while pushing a docker image with S3 as the registry backend. Adding the following to the registry config.yml manually fixes my issue.

storage:
  redirect:
    disable: true

I have an issue opened on harbor-boshrelease project that I am using to deploy harbor. May be it belongs here .
https://github.com/vmware/harbor-boshrelease/issues/15

[rizwan-kh]

@suppalapati13 I initially used s3 as the storage backend, but was facing enormous issues, so switched back to filesystem. My current issue is on a filesystem storage backend.

[reasonerjt]

@rizwan-kh Did you setup Harbor with the hostname: harbor.masked.com?
Is there any proxy/firewall between the host where you issued docker push and the Harbor instance?

[rizwan-kh]

@reasonerjt yes, I did set it up with a hostname as harbor.mycompanyname.com and there is no proxy/firewall between the host from where I am issuing the docker push command. This was working since the start and I have been pushing all images using the same CI/CD workflow where I had configured the push to the registry as a job.

[rizwan-kh]

I am closing this issue, as the push is working fine now. Seems an intermittent issue. Shall open a new ticket in case this arises again.

[psoares]

I have the issue with harbor deployed through the helm chart https://github.com/goharbor/harbor/tree/master/contrib/helm/harbor into a minikube instance.

I used the following command:
helm install . --name registry --set externalDomain=my.registry.host.com,insecureRegistry=true --set externalPort=

[toby1991]

@rizwan-kh What is your storage backend? I have similar EOF issue while pushing a docker image with S3 as the registry backend. Adding the following to the registry config.yml manually fixes my issue.

storage:
  redirect:
    disable: true

I have an issue opened on harbor-boshrelease project that I am using to deploy harbor. May be it belongs here .
vmware/harbor-boshrelease#15

You save my life ! It works like a charm! Thanks !!

[Aisuko]

Hi, I got the same error like below:

f85c58969eb0: Pushing [==================================================>]  141.9MB
82d548d175dd: Pushing [==================================================>]  9.216kB
fe4c16cbf7a4: Pushing [==================================================>]    123MB/123MB
EOF

And I just realized that my current docker was using the manual proxy. So, I mean to pay attention the docker proxy.