Azure-pipelines-tasks: CacheBeta@1 cannot find directory that definitely exists

[autarch]

Required Information

Type: Bug
Task Name: CacheBeta@1

Environment

Server: Azure Pipelines
Agent: Hosted

Issue Description

I had a cache task set up as following:

- task: CacheBeta@1
  displayName: Define build dependencies cache
  inputs:
    key: '"v1" | "build-deps" | $(CI_WORKSPACE_DIRECTORY)/cache/tools-perl-version | $(CI_WORKSPACE_DIRECTORY)/cache/build-deps-cpanfile'
    path: /usr/local/perl5/perlbrew/perls/tools-perl/lib/site_perl

The path definitely exists on disk.

However, I got an error when the job tried to save the cache contents.

Task logs

Attached

Error logs

tar: /usr/local/perl5/perlbrew/perls/tools-perl/lib/site_perl: Cannot open: No such file or directory

logs_50.zip

[autarch]

It occurs to me that maybe this is because the path I'm giving is a path that exists inside my Docker container, and maybe the caching is running on the VM host?

If that's the case then this should be more clearly documented. But that still leaves me wondering how I would cache paths inside the container.

[johnterickson]

@autarch Yes, caching runs on the host. @fadnavistanmay Does the logging for container mappings you added help here?

[autarch]

@johnterickson Yeah, I think that logging was useful eventually.

But really, I think this needs to be spelled out much more explicitly in the documentation.

Also, in general, the docs could do more to explain the host/container relationship and how that affects various types of steps. In particular, I think it would be good advice to tell people to _only_ use mounted volumes when writing to disk, as that makes it a lot simpler to map them between host & container using the various pre-defined variables.

[johnterickson]

Good feedback on docs, @autarch! Higher level question - if there were good layer caching, would you use that instead?

@vtbassmatt for:

Also, in general, the docs could do more to explain the host/container relationship and how that affects various types of steps. In particular, I think it would be good advice to tell people to only use mounted volumes when writing to disk, as that makes it a lot simpler to map them between host & container using the various pre-defined variables.

[autarch]

if there were good layer caching, would you use that instead?

Do you mean docker layer caching? I don't see how that would help with what I'm doing, plus I want to do the same kind of caching for builds on macOS and Windows.

That said, I would _love_ to see Docker containers spin up more quickly. CircleCI spins up containers in about 10s whereas it takes a minute or more in Azure.

[fadnavistanmay]

@autarch Yes, caching runs on the host. @fadnavistanmay Does the logging for container mappings you added help here?

Hi John, yup, the logging would show the container path.

@autarch - John has an example of caching the entire docker image here, if you want to refer - https://dev.azure.com/codesharing-su0/cachesandbox/_build/results?buildId=16224 .

Are there any pending action items for this issue, or can it be closed?

[autarch]

I don't think managing my own Docker server seems like a good approach. I just want Azure's container handling to be faster.

[fadnavistanmay]

Thanks. Could you please file a new issue for "I just want Azure's container handling to be faster." cc - @vtbassmatt .

[vtbassmatt]

@autarch it would be good to get some specifics about what's not fast enough. And you should file that over at Developer Community, since the solution is unlikely to (solely) be in the task implementation.

[autarch]

Posted at https://developercommunity.visualstudio.com/idea/809838/make-docker-startup-faster.html

Thanks.