Virtual-environments: Increase the timeout or reduce concurrent operations to upload large layers for Docker on Ubuntu-latest

[trajano]

Description

The default timeout for Docker push is too small for some larger layers. The behaviour is noted in https://stackoverflow.com/questions/41070528/docker-push-net-http-tls-handshake-timeout

This can be mitigated by increasing the timeout or as specified in the SO question reduce the number of concurrent operations. However, there's no capability of doing so with Docker in Azure Pipelines

Area for Triage:

Docker

Question, Bug, or Feature?:

Bug maybe Feature

Virtual environments affected

• [x] Ubuntu 16.04
• [x] Ubuntu 18.04
• [x] Ubuntu 20.04
• [ ] macOS 10.15
• [ ] macOS 11.0
• [ ] Windows Server 2016 R2
• [ ] Windows Server 2019

Expected behavior
We shouldn't get timeouts when uploading a large layer.

Actual behavior
We are getting timeouts when uploading a large layer on occassion. Rerunning sometimes helps.

Repro steps
A description with steps to reproduce the issue. If your have a public example or repo to share,
please provide the link.

1. Create a docker image with many "large" layers (try 8 x 0.5GB) layers that are not highly compressible
2. Push to a repository.

[miketimofeev]

Hi @trajano!
It would be great if you can provide such an image, it should help us a lot during investigation

[trajano]

I'll see if I can generate it for you but like I said it is pretty random.

[trajano]

I have an old docker image that we can probably use as it has a large 600MB layer.

docker pull trajano/jee-webserver:base
docker tag trajano/jee-webserver:base privaterepo.com/test
docker push privaterepo.com/test

Now the tricky part to simulate this would be to have the private repo behind a slow network.

[LeonidLapshin]

Hey, @trajano!
I guess you can try to change this setting in runtime:

sudo sed -i 's/ }/, \"max-concurrent-uploads\": 5 }/' /etc/docker/daemon.json
sudo systemctl restart docker

Could you please check it?

[trajano]

Confirmed it works or at least does not crash. However, as I stated it does not always occur I'll open this again if needed. There should be an issue added to modify the Azure DevOps documentation regarding this but I can't find a good page for it, the closest I can think of is https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/tool/docker-installer?view=azure-devops anyway I'll add an answer to Stack Overflow referencing your comment since people may be looking for it there.