Virtual-environments: Did Dockerhub rate limit affect Github Action?

[BrightRan]

Associated community ticket: https://github.community/t/did-dockerhub-rate-limit-affect-github-action/128158

The customer noticed that Docker hub has updated their pricing, Download rate limit and retention policy recently. He is wondering if the download rate limit affects on Github Action.
Will the steps like below are affected?

jobs:
  my_first_job:
    steps:
      - uses: docker://alpine:3.11

```yaml
jobs:
my_first_job:
steps:
- run: docker pull alpine:3.11

According to the introductions from the docs about "**Download rate limit**", the limit seems only applies for the anonymous users with an eventual limit of 100 pulls per six hours. Logged in users will not be affected at this time.

I setup a workflow with the following step that pull an image 101 times in a loop.
```yaml
- name: Test Docker download rate limits
  run: |
    docker image ls
    index=1
    while [ $index -le 101 ]
    do
      echo "---------------------------------------------------------------------------------------"
      echo "Pull image $index times"
      docker pull alpine:3.11
      docker image ls
      index=$(( $index + 1 ))
      docker image rm alpine:3.11
    done
    docker image ls

It can successfully download the image 101 times without any warning or error messages about the download rate limit.
Note: Consider there may be image cache, I have used the "docker image rm" command to remove the image every time before downloading it again.

Question:
When we try to pull some public images in the workflow, if we do not login with any of our accounts, whether GitHub has a default account to login the docker?
If not, in my above example, why it is not hit the download rate limit?

[maxim-lobanov]

Hello @BrightRan , thank you for raising it.
The new policy will be applied starting from November 1st, 2020. It is the reason why we can't reproduce it right now.
Docker will gradually introduce these rate limits, with full effects starting from November 1st, 2020.
GitHub doesn't have default account to login in the docker. I am not sure if it is possible because of security reasons.

I have found the following Docker use-cases in GA (I could miss something):
1.

jobs:
  my_first_job:
    steps:
      - run: docker pull alpine:3.11

This use-case allow customers to do login by theirselves:

steps:
  - run: |
      docker login --username ${{ secrets.docker_username }} --password ${{ secrets.docker_password }}
      docker pull alpine:3.11

2)

jobs:
  container:
    runs-on: ubuntu-latest
    container: node:10.16-jessie
    steps:
      - run: |
          echo This job does specify a container.
          echo It runs in the container instead of the VM.
        name: Run in container

Looks like pull of such container is happening at the first step of pipeline and we can't login before somehow.

3.

jobs:
  my_first_job:
    runs-on: ubuntu-latest
    steps:
      - uses: docker://alpine:3.11

this use case will be affected too
because even if we invoke

steps:
  - run: docker login
  - uses: docker://alpine:3.11

Container pull happens at the first step of the build before docker login

4.

jobs:
  my_first_job:
    runs-on: ubuntu-latest
    steps:
      - uses: my_custom_action_that_uses_docker_inside@v1

In the first use-case, customer can overcome limit by login to docker as a first step of pipeline: docker login --username ${{ secrets.docker_username }} --password ${{ secrets.docker_password }}.
But looks like use-cases 2,3,4 will be broken.

DockerHub will use IP rate limit so our customers could be affected. Docker will gradually impose download rate limits with an eventual limit of 100 pulls per six hours for anonymous users.
As I can see, DockerHub even provides separate documentation section about rate limits in GitHub Actions. But actually, they are asking action owners to build-in docker login as a part of their actions and it will fix use case 4.

I don't have much experience with docker so could be wrong, may be actions/runner team can help us to understand if there is a way to login in use-cases 2,3,4. cc: @hross , @ericsciple , @TingluoHuang

cc: @thejoebourneidentity @alepauly @bryanmacfarlane

[BrightRan]

@maxim-lobanov ,
Thanks for your reply.
For the use case 3 and 4, can we add a step to login docker before the 'uses' step? Similar to the use case 1 you mentioned.

[maxim-lobanov]

@BrightRan , for 4 - yes.
For 2 and 3 it is not possible because looks like pulling of all images happen before all steps.
This yaml produces the following steps:

steps:
  - run: docker login
  - uses: docker://alpine:3.11

[BrightRan]

@maxim-lobanov ,
Yes, you're right.
I just have tried it, and the results confirm your statement.
Thanks for your detailed explanation.

[hross]

The great thing about this thread is...

We are actually actively working on providing more native login capabilities in actions. See this ADR. This should make it easier for customers to set up authentication for scenarios 2 and 3.

@dakale is adjusting the syntax with @chrispat (today, actually). We should have this ready to go by the end of the month at the latest, which should mean better support before the docker limits are imposed.

Hey @dakale can you add each of these examples to your ADR (1-4) and explain how they can be addressed with the new syntax?

[maxim-lobanov]

@hross , Happy to hear that we are working on feature that will fix the issue.
Also I think we will need a good communication with our customers to migrate them to new approach with auth.
We should avoid case when all docker related builds start to fail one day, and customers will have to fix their builds in rush to unblock builds.

[dakale]

I agree with the scenarios as described by @maxim-lobanov , I think it accurately reflects all the ways we support containers. This only affects 2 and 3. Those are where we pre-pull containers at the start of the job, thus giving no place to run commands to log in. 1 and 4 will be fine as users can run docker login in step if needed

For 2 and 3, we dont currently support pulling with authentication. That previously only meant you couldnt use "private" images in those scenarios, but now it means those scenarios may be impacted by this rate limiting issue. A single solution that allows the pull to be authenticated solves both

I should note that (due to my mistake), the link to the ADR @hross posted is out of date. We have iterated on it since, and the change is significant relative to this issue. Check the PR for the most up to date document: https://github.com/github/c2c-actions-runtime/pull/813/files

That ADR, as currently designed, only solves 2, which currently is impossible to work around (except on self-hosted). It does not address 3 at all, nor does it add anything for 1, 4, but those have viable workarounds. Thats because its designed to solve a different problem that would have accidentally helped the rate limiting problem too

I have no way to gauge what the impact of the rate limiting will be, since I dont know the range of IPs used for the hosted runner vms or how much of the available rate they might consume, but 100 per 6 hours is very very low. Its hard to make a judgement on whether its worth revising the ADR again to account for rate limiting, but I do think its worth considering given how low the limit is.

[vtbassmatt]

We're also thinking about this in Azure Pipelines. We already have a technical solution for scenarios 2 and 3 (container resources can use a service connection for authentication). Most of our current usage is anonymous, though, so we'll have to help customers understand the impact and adapt.

[nihaals]

Judging from the ADRs being private, will GitHub users not get to see the syntax until it's out (and can't have breaking changes)? Since the rate limit would almost definetely affect users, would it be worth adding a deprecation notice once a fix is available eventually forcing all users to pull while being logged in? Could help solve problems before they arise if people new to GitHub Actions start using it and expect it to work without knowing about the limit or that public CIs don't get some kind of whitelist

[chrispat]

@nihaals the syntax is very simple as we are simply adding a credentials element to the job.container and job.services key. We are also trying to understand from Docker if the library images are impacted by these changes. Based on our telemetry we believe that will have the biggest impact in how many users are immediately impacted.

[nihaals]

I made what I think is a related feature request that may make it nicer to login with Actions which sounds like will work with the new syntax

What do you think about the deprecation notice on unauthorised Docker Hub images? Probably blocked by finding out if official images are exempt and the new syntax

[rnorth]

As I understand it one scary and unresolved problem with (1) is going to be sharing Hub credentials with forked repos. Any solution that involves a docker login execution will not work, because (a) secrets are not available to forked repos, and (b) we have no other safe way to provide credentials to a docker login invocation.

Does the solution that's being worked on address this aspect?

[chrispat]

At the moment we do not have a solution to that particular problem but we are investigating options.

[maxim-lobanov]

After the latest GitHub update, you can specify credentials for container downloading: https://github.blog/changelog/2020-09-24-github-actions-private-registry-support-for-job-and-service-containers/

[rnorth]

@chrispat sorry to press on this, but is there any news on solutions to the forked repo problem I mentioned above?

[mattwelke]

@maxim-lobanov

After the latest GitHub update, you can specify credentials for container downloading

I read the blog post. The new feature, being able to authenticate with registries, looks like it will have the nice side effect of solving the rate limiting problem for job containers and service containers by allowing people to authenticate, which would stop them from being forced to share the rate limit pool with other GitHub Actions users.

However, do you know whether it will work for Docker container actions? For example, from the docs (https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#example-using-a-docker-hub-action):

jobs:
  my_first_job:
    steps:
      - name: My first step
        uses: docker://alpine:3.8

In this example, the image used for the Docker container action is being pulled by the GitHub Actions runner from Docker Hub. Because credentials aren't specified for the step, I'm assuming this is being done without authenticating with Docker Hub.

The images used for Docker container actions must be public, which means people don't have to authenticate to use them, but if they choose not to authenticate as they use them, they'll be subject to the IP-based rate limiting, affected by other GitHub Actions users. Is this correct?

[rnorth]

@chrispat sorry to ask again, but it's getting worryingly close to November and, as far as I can tell, there's still no safe way for open source projects to use authenticated access to Docker Hub.

Has there been any progress in finding a way for forks of repos to be able to inherit Docker Hub authentication from the original repo?

[chrispat]

For publicly accessible containers we are working with docker hub to make sure you will not be impacted by the new rate limits. If you need private containers we still do not have a solution for that problem for forks of public repos.

[rnorth]

Thank you @chrispat! As a maintainer of an open source project that uses public Docker images, that's really good news.

[mdeggies]

For publicly accessible containers we are working with docker hub to make sure you will not be impacted by the new rate limits.

@chrispat just to confirm, does this mean we will NOT need to add dockerhub auth to all of our jobs that pull public dockerhub images?

[chrispat]

That is correct. It is important to note that this will only apply to the runners hosted by GitHub, if you are using self-hosted runners you will need to configure your own credentials.

[mdeggies]

Thank you for the confirmation! :)

[lizan]

That is correct. It is important to note that this will only apply to the runners hosted by GitHub, if you are using self-hosted runners you will need to configure your own credentials.

@chrispat Will this be also applicable to Azure Pipelines hosted agents?

[chrispat]

@lizan for now this will apply for Azure Pipelines. However, given you have the ability to enter your own credentials for containers in Azure Pipelines I would encourage you to do that. One of the main reasons we feel this is important for GitHub Actions is to better support our open source customers, and a lot of the individual actions in the marketplace are themselves implemented as containers.

Azure Pipelines tasks by contrast have no formal support for being implemented as a container and while we know some open source projects do use it we are encouraging them to move to GitHub Actions.

[lizan]

Thanks for the clarification, that helps.

Azure Pipelines tasks by contrast have no formal support for being implemented as a container and while we know some open source projects do use it we are encouraging them to move to GitHub Actions.

Yeah I'm asking on behalf of Envoy OSS project :) and we don't have good story to move to GHA for various reasons.

[magthe]

Did this issue reach a conclusion?
Where can I find the answer to the original question, "Did Dockerhub rate limit affect Github Action?"

[maxim-lobanov]

@magthe , https://github.com/actions/virtual-environments/issues/1445#issuecomment-713861495

For publicly accessible containers we are working with docker hub to make sure you will not be impacted by the new rate limits. If you need private containers we still do not have a solution for that problem for forks of public repos.
It is important to note that this will only apply to the runners hosted by GitHub, if you are using self-hosted runners you will need to configure your own credentials.

[magthe]

Yes, I read that, but the wording makes it sound like it's in progress.

Should I interpret the closing of this ticket as there's a finalised an agreement with Docker Hub that ensures GitHub Actions aren't impacted by the rate limits?

[chrispat]

@magthe yes we resolved this through an agreement with Docker.

[mukul-c2fo]

I saw the docker pull limit error on few of the github actions today which build an image from a Dockerfile.

[bfjelds]

@chrispat, is there any public announcement/documentation of this Docker agreement? it would be helpful to understand what is included/excluded (does this apply to all github repo actions/workflows except self-hosted runners? does it apply to public and private repos? etc) Thanks!!

[seglo]

Does this just work by nature of dockerhub allowing unlimited pulls from builds running on github actions (by hostname/ip?)