Azure-docs: docker push always fails partially with EOF

[megadix]

When I try the docker push step, it consistently fails to upload the last layers. It retries a couple of times, then prints EOF and exists.
E.g.:

The push refers to repository [dimitriacstutorial.azurecr.io/azure-vote-front]
f1199079b064: Layer already exists
c93bcc0e66d8: Layer already exists
8104eeb0e43a: Layer already exists
f51952865a1e: Layer already exists
dbe8605b7892: Layer already exists
a492d383f968: Layer already exists
1143c60ad9b2: Layer already exists
e8810ec72c2b: Layer already exists
9dcde2bd7f22: Layer already exists
0d701ea07f21: Layer already exists
ba1caa9ad1b8: Layer already exists
902245526746: Layer already exists
b800a9224768: Layer already exists
efba3aa8a05c: Layer already exists
a17e9881496c: Layer already exists
60e1a8618c67: Layer already exists
6c7b9546171d: Layer already exists
303c11dcd890: Layer already exists
746cca6f1abf: Layer already exists
dc918b50a9e0: Layer already exists
3b9f5a79fc86: Layer already exists
80ab47649ccb: Layer already exists
6691330056f3: Pushing [==================================================>]  67.26MB/67.26MB
bc0d5482d762: Layer already exists
44246ca595f6: Pushing [==================================================>]  556.4MB/556.4MB
20c527f217db: Pushing [==================================================>]  141.7MB/141.7MB
61c06e07759a: Pushed
bcbe43405751: Pushing [==================================================>]  23.84MB/23.84MB
e1df5dc88d2c: Pushing [=>                                                 ]   3.79MB/100.1MB
EOF

Is this because I'm using the free tier?

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 22dc1f78-5ef2-391c-732f-740c0ca12880
• Version Independent ID: ffad8c17-c1e6-fb99-3a39-98be4d6ed091
• Content: Push Docker image to private Azure registry
• Content Source: articles/container-registry/container-registry-get-started-docker-cli.md
• Service: container-registry
• Product: unspecified
• GitHub Login: @SteveLas
• Microsoft Alias: stevelas

[kobulloc-MSFT]

@megadix Thank you for the feedback! We are investigating this and will get back to you shortly.

[megadix]

Update

I tried pushing from another network, which has far more upload capacity than my home DSL, it worked without problems.
Is it possible that a slow upload link can lead to timeouts? (just speculation)

Thanks,
Dimitri

[mimckitt]

@megadix thanks for the update! Glad to hear you were able to get it to work.

Slow internet speed could absolutely lead to timeouts. There are built in timeouts in Azure to avoid forever running operations. I am not sure of the timeout for container registry but for example if you were pushing a custom script to an Azure VM it has a timeout of 90 mins. After that Azure throws and error and stops the operation. I assume the same is going on here.

How long was it taking to upload on your home network vs the faster network?

[megadix]

On DSL connection:

• quick layers (e.g. f1199079b064, c93bcc0e66d8, etc. see first post) take a minute or so
• slow layers (6691330056f3, bcbe43405751, etc.) can take 3 to 15 minutes each, however when they reach 100% they never actually complete (Pushed)
• after some "thinking" the process stops with just EOF message

On a corporate network it takes a couple of minutes for the whole push, without errors.

Thank you for your support!
Dimitri

[mimckitt]

@megadix thanks for the extra information. I will assign this to the author to review and see if we need to add any additional information to the doc on upload speeds.

@SteveLas can you take a look?

[SteveLasker]

Hi @megadix
I confirmed with our engineering team that this is a common problem with slow/latent/unreliable connections. The larger the image, or more specifically, the larger the layers, the more fragile it is.
You might also see problems if you're attempting to pull a linux image from windows, or a windows image from mac. If you're using Docker for Windows, you might confirm you're set to the proper platform on the task bar of moby. But, I suspect it's more a network latency issue.
Steve

[mimckitt]

Thanks for that @SteveLasker!

@megadix I will close this out for now but if you have further issues let us know.

[mmacy]

@MicahMcKittrick-MSFT Ping on the issue close for this one, Micah. Thanks!

[dtschroe]

Having the exact same issue, fails to push every time. I have a pretty good internet connection (180gb down, 30gb up) and I have tried pushing probably 50 times and every time it times out. I reduced the max upload threads to 2 and still have the issue. The timeout is really short. Seems like it only allows 45 seconds or less per layer.

[GuyHarwood]

i am also having this issue regularly, and it seems to be getting worse.

this is when deploying from VSTS/Azure-Dev-Ops....

2018-10-17T16:54:31.9545767Z 8fad67424c4e: Layer already exists
2018-10-17T16:54:56.2559759Z eef285a08230: Retrying in 5 seconds
2018-10-17T16:54:57.2557473Z eef285a08230: Retrying in 4 seconds
2018-10-17T16:54:58.2574646Z eef285a08230: Retrying in 3 seconds
2018-10-17T16:54:59.2562566Z eef285a08230: Retrying in 2 seconds
2018-10-17T16:55:00.2557512Z eef285a08230: Retrying in 1 second
2018-10-17T16:55:35.2877112Z eef285a08230: Retrying in 10 seconds
2018-10-17T16:55:36.2871480Z eef285a08230: Retrying in 9 seconds
2018-10-17T16:55:37.2870964Z eef285a08230: Retrying in 8 seconds
2018-10-17T16:55:38.2870587Z eef285a08230: Retrying in 7 seconds
2018-10-17T16:55:39.2872255Z eef285a08230: Retrying in 6 seconds
2018-10-17T16:55:40.2870440Z eef285a08230: Retrying in 5 seconds
2018-10-17T16:55:41.2870704Z eef285a08230: Retrying in 4 seconds
2018-10-17T16:55:42.2870817Z eef285a08230: Retrying in 3 seconds
2018-10-17T16:55:43.2871471Z eef285a08230: Retrying in 2 seconds
2018-10-17T16:55:44.2879001Z eef285a08230: Retrying in 1 second
2018-10-17T16:56:25.0888419Z eef285a08230: Retrying in 15 seconds
2018-10-17T16:56:26.0892222Z eef285a08230: Retrying in 14 seconds
2018-10-17T16:56:27.0890335Z eef285a08230: Retrying in 13 seconds
2018-10-17T16:56:28.0890247Z eef285a08230: Retrying in 12 seconds
2018-10-17T16:56:29.0890329Z eef285a08230: Retrying in 11 seconds
2018-10-17T16:56:30.0897416Z eef285a08230: Retrying in 10 seconds
2018-10-17T16:56:31.0890886Z eef285a08230: Retrying in 9 seconds
2018-10-17T16:56:32.0891412Z eef285a08230: Retrying in 8 seconds
2018-10-17T16:56:33.0890405Z eef285a08230: Retrying in 7 seconds
2018-10-17T16:56:34.0889773Z eef285a08230: Retrying in 6 seconds
2018-10-17T16:56:35.0890423Z eef285a08230: Retrying in 5 seconds
2018-10-17T16:56:36.0890222Z eef285a08230: Retrying in 4 seconds
2018-10-17T16:56:37.0891231Z eef285a08230: Retrying in 3 seconds
2018-10-17T16:56:38.0896966Z eef285a08230: Retrying in 2 seconds
2018-10-17T16:56:39.0890683Z eef285a08230: Retrying in 1 second
2018-10-17T16:57:15.5901638Z eef285a08230: Retrying in 20 seconds
2018-10-17T16:57:16.5894720Z eef285a08230: Retrying in 19 seconds
2018-10-17T16:57:17.5893916Z eef285a08230: Retrying in 18 seconds
2018-10-17T16:57:18.5894683Z eef285a08230: Retrying in 17 seconds
2018-10-17T16:57:19.5894516Z eef285a08230: Retrying in 16 seconds
2018-10-17T16:57:20.5893965Z eef285a08230: Retrying in 15 seconds
2018-10-17T16:57:21.5898228Z eef285a08230: Retrying in 14 seconds
2018-10-17T16:57:22.5894284Z eef285a08230: Retrying in 13 seconds
2018-10-17T16:57:23.5892978Z eef285a08230: Retrying in 12 seconds
2018-10-17T16:57:24.5895545Z eef285a08230: Retrying in 11 seconds
2018-10-17T16:57:25.5895377Z eef285a08230: Retrying in 10 seconds
2018-10-17T16:57:26.5893769Z eef285a08230: Retrying in 9 seconds
2018-10-17T16:57:27.5893736Z eef285a08230: Retrying in 8 seconds
2018-10-17T16:57:28.5893997Z eef285a08230: Retrying in 7 seconds
2018-10-17T16:57:29.5893389Z eef285a08230: Retrying in 6 seconds
2018-10-17T16:57:30.5893180Z eef285a08230: Retrying in 5 seconds
2018-10-17T16:57:31.5895613Z eef285a08230: Retrying in 4 seconds
2018-10-17T16:57:32.5893525Z eef285a08230: Retrying in 3 seconds
2018-10-17T16:57:33.5894085Z eef285a08230: Retrying in 2 seconds
2018-10-17T16:57:34.5893569Z eef285a08230: Retrying in 1 second
2018-10-17T16:58:09.3752297Z EOF
2018-10-17T16:58:09.3946790Z ##[error]EOF

the timeout in the docker build task is set to zero. So is this timeout controlled by the container registry?

[wavemotionio]

Hi, this still appears to be a problem. Has anyone resolved this issue?

docker push priveateregistry.azurecr.io/products-service:v1

The push refers to repository [hpuidocker.azurecr.io/products-service]
08e2cdb90fe0: Layer already exists
bc1fd392a87d: Layer already exists
11122785fc00: Layer already exists
c11ef9008a54: Layer already exists
85815f69e311: Layer already exists
ade8acb532e7: Pushing [==================================================>] 83.88MB/83.88MB
29fbe5747f9a: Layer already exists
787062cd94fb: Pushing [==================================================>] 607.5MB/607.5MB
a4e797bc3f15: Pushing [==================================================>] 158.8MB/158.8MB
392f356944ff: Layer already exists
15210a41d4ee: Layer already exists
e2a8a00a83b2: Layer already exists
EOF

[gcormier]

Same issue pushing to Docker Hub using Azure Pipelines Docker@2.

[SteveLasker]

For acr scenarios, we’ve provided a trouble shooting tool: https://aka.ms/acr/health-check