Azure-pipelines-agent: Cache persist doesn't work on server 2012 R2 or 2016 (I assume) because it doesn't inclued tar

[jabbera]

Having issue with Tasks?

Since this is the embedded cache task I believe this belongs here.

Agent Version and Platform

Version of your agent? 2.160 on prem

OS of the machine running the agent? Windows 2012 R2

Azure DevOps Type and Version

dev.azure.com

https://dev.azure.com/gmollc

What's not working?

CacheBeta@1 isn't saving the cache because tar isn't included in 2012 R2.

Agent and Worker's Diagnostic Logs

2019-11-12T21:42:35.5033236Z ##[debug]Starting 'tar' with arguments '-h -v -cf "4092ddf38c57400cbae3ca85f9ae5178_archive.tar" -C "D:\SharedBuild2\1\s\.pipeline-cache\nuget" .'...
2019-11-12T21:42:36.0435164Z Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session 8a46dbca-e561-4dd7-ab21-92c8c86e4de1
2019-11-12T21:42:36.0512117Z ##[error]The system cannot find the file specified
2019-11-12T21:42:36.0520742Z ##[debug]Processed: ##vso[task.logissue type=error;]The system cannot find the file specified
2019-11-12T21:42:36.0521458Z ##[debug]Processed: ##vso[task.complete result=Failed;]
2019-11-12T21:42:36.1115284Z ##[debug]   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Agent.Plugins.PipelineCache.TarUtils.RunProcessAsync(AgentTaskPluginExecutionContext context, ProcessStartInfo processStartInfo, Func`3 additionalTaskToExecuteWhilstRunningProcess, Action actionOnFailure, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.TarUtils.RunProcessAsync(AgentTaskPluginExecutionContext context, ProcessStartInfo processStartInfo, Func`3 additionalTaskToExecuteWhilstRunningProcess, Action actionOnFailure, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.TarUtils.ArchiveFilesToTarAsync(AgentTaskPluginExecutionContext context, String inputPath, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.PipelineCacheServer.GetUploadPathAsync(ContentFormat contentFormat, AgentTaskPluginExecutionContext context, String path, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.PipelineCacheServer.UploadAsync(AgentTaskPluginExecutionContext context, Fingerprint fingerprint, String path, CancellationToken cancellationToken, ContentFormat contentFormat)
   at Agent.Plugins.PipelineCache.SavePipelineCacheV0.ProcessCommandInternalAsync(AgentTaskPluginExecutionContext context, Fingerprint fingerprint, Func`1 restoreKeysGenerator, String path, CancellationToken token)
   at Agent.Plugins.PipelineCache.PipelineCacheTaskPluginBase.RunAsync(AgentTaskPluginExecutionContext context, CancellationToken token)
   at Agent.Plugins.PipelineCache.SavePipelineCacheV0.RunAsync(AgentTaskPluginExecutionContext context, CancellationToken token)
   at Agent.PluginHost.Program.Main(String[] args)
2019-11-12T21:42:36.1219290Z ##[section]Finishing: Cache NuGet packages

[jabbera]

You people are going to think I'm crazy, but I'm not... Here a screenshot of the procmon trace looking for tar.exe:

Here is a screenshot of it NEVER looking for 7z.exe from the same trace (The trace includes process start to terminate):

Just to prove I'm extra, extra not crazy:

And I've rebooted since to be sure the path propagated.

The ONLY way I can see procmon not reporting at least a search for 7z.exe is if somehow isWindows is false here:
https://github.com/microsoft/azure-pipelines-agent/blob/7fc04a145e5158368e8e4fea7f2d996e8a053389/src/Agent.Plugins/PipelineCache/TarUtils.cs#L190

but I don't see how that is possible!?!?!?!

[johnterickson]

@fadnavistanmay Can you take a look?

[jabbera]

Just to add more confusion to the mix. I provisioned a new 2019 server and everthing works (same pipeline). I'm going to try a 2012 R2 new. Then I'm going to put a new agent on my non-working 2012 r2. Will update this post as I investigate.

[fadnavistanmay]

Hi @jabbera - Glad you were able to make it work on the 2019 server. Just a quick note, the error you are getting is in the SaveCache task, which always use tar; the code snippet you have shared is during the Restore Cache task, where we look for 7z and use that if present.

[jabbera]

@fadnavistanmay you hit the nail on the head. There is no fallback to 7-zip here. https://github.com/microsoft/azure-pipelines-agent/blob/7fc04a145e5158368e8e4fea7f2d996e8a053389/src/Agent.Plugins/PipelineCache/TarUtils.cs#L43.

Server 2019 includes tar out of the box! Are you going to support windows os versions that don't include TAR for saving cache artifacts? (Since you have it for downloading I would expect you to:-))

[fadnavistanmay]

@jabbera - did you explicitly installed tar on those machines, and it still fails?? You are using self-hosted agents, yes?

For now, you can fall back to old behavior by setting the variable AZP_CACHING_CONTENT_FORMAT to Files.

[jabbera]

@fadnavistanmay where from? Isn't that a 2019 only feature? (Unless you are talking about the mingw/gnuwin32 tar?) Additionally why would you have the 7 zip fallback for downloading caches but not uploading them? It doesn't make much sense to me.

Thanks for the workaround. I'll give it a try.

[jabbera]

@fadnavistanmay For whatever reason the environment variable doesn't work either. I took a surface level look at the code and it's not obvious why it's not working. I'm happy to setup a screen share or something if you want to take a deeper look. Here are snippits from my log:

[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'ALLUSERSPROFILE': 'C:\ProgramData'
[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'APPDATA': 'C:\Users\<SNIP>\AppData\Roaming'
[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'AZP_CACHING_CONTENT_FORMAT': 'Files'
[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'CommonProgramFiles': 'C:\Program Files\Common Files'
[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'CommonProgramFiles(x86)': 'C:\Program Files (x86)\Common Files'
[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'CommonProgramW6432': 'C:\Program Files\Common Files'

2019-11-13T20:38:27.5300788Z ##[debug]Processed: ##vso[telemetry.publish area=AzurePipelinesAgent;feature=PipelineCache]{"FileCount":"0","PlanId":"b8fde65d-718b-4f2c-8a48-b3ff33b7809f","JobId":"0ab14b9f-e499-56d5-97b1-fd98b70ea339","TaskInstanceId":"aa203359-c3b8-5e0f-0fa6-2cb7b8dcce2a","CacheResult":"Miss","ActionDurationMs":"892","ActionName":"PipelineCache.RestoreCache","ActionResult":"Success","AttemptNumber":"1","ItemCount":"0","Level":"ThirdParty","CreatedUtcNow":"2019-11-13T20:38:26.5686059Z","SentUtcNow":"2019-11-13T20:38:27.4717776Z","BaseAddress":"https://vsblobprodcus3.vsblob.visualstudio.com/Ad31b77fc-dfbf-4068-8419-a4cc92bbfac6/","X_TFS_Session":"2ebdd9b2-d83c-46a7-836f-1080713ffccc","DeploymentEnvironment":"PRODUCTION","DeploymentEnvironmentIsProduction":"True","VSOAccount":"vsblobprodcus3","OSName":"Microsoft Windows","OSVersion":"6.3.9600","FrameworkDescription":".NET Core ","ProcessName":"Agent.PluginHost","DotNetReleaseDword":"-1","Version":"18.159.29324.0 built by: master (a2f0ba0f2a)","ExceptionCount":"0"}
2019-11-13T20:38:27.5302449Z ##[debug]Starting 'tar' with arguments '-h -v -cf "15fad05c549e42999f7259172fa315b6_archive.tar" -C "D:\SharedBuild1\2\s\.pipeline-cache\nuget" .'...
2019-11-13T20:38:28.1735655Z Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session 2ebdd9b2-d83c-46a7-836f-1080713ffccc
2019-11-13T20:38:28.1814906Z ##[error]The system cannot find the file specified
2019-11-13T20:38:28.1824931Z ##[debug]Processed: ##vso[task.logissue type=error;]The system cannot find the file specified
2019-11-13T20:38:28.1825655Z ##[debug]Processed: ##vso[task.complete result=Failed;]
2019-11-13T20:38:28.1988197Z ##[debug]   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Agent.Plugins.PipelineCache.TarUtils.RunProcessAsync(AgentTaskPluginExecutionContext context, ProcessStartInfo processStartInfo, Func`3 additionalTaskToExecuteWhilstRunningProcess, Action actionOnFailure, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.TarUtils.RunProcessAsync(AgentTaskPluginExecutionContext context, ProcessStartInfo processStartInfo, Func`3 additionalTaskToExecuteWhilstRunningProcess, Action actionOnFailure, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.TarUtils.ArchiveFilesToTarAsync(AgentTaskPluginExecutionContext context, String inputPath, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.PipelineCacheServer.GetUploadPathAsync(ContentFormat contentFormat, AgentTaskPluginExecutionContext context, String path, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.PipelineCacheServer.UploadAsync(AgentTaskPluginExecutionContext context, Fingerprint fingerprint, String path, CancellationToken cancellationToken, ContentFormat contentFormat)
   at Agent.Plugins.PipelineCache.SavePipelineCacheV0.ProcessCommandInternalAsync(AgentTaskPluginExecutionContext context, Fingerprint fingerprint, Func`1 restoreKeysGenerator, String path, CancellationToken token)
   at Agent.Plugins.PipelineCache.PipelineCacheTaskPluginBase.RunAsync(AgentTaskPluginExecutionContext context, CancellationToken token)
   at Agent.Plugins.PipelineCache.SavePipelineCacheV0.RunAsync(AgentTaskPluginExecutionContext context, CancellationToken token)
   at Agent.PluginHost.Program.Main(String[] args)
2019-11-13T20:38:28.2093865Z ##[section]Finishing: Cache NuGet packages

[fadnavistanmay]

Hi @jabbera - The reason we went for tarring for upload was, the performance was almost the same for creating a "tar" - for both the tar and 7z process, but while downloading (untarring) - 7z gives much better performance.

Could you please tell us how/where are you setting the environment variable. And if you could give us the redacted logs of the entire build, that would be helpful.

Will it be possible for you. to give @johnterickson and me , permissions for https://dev.azure.com/gmollc , to investigate?

Thanks.

[jabbera]

I’m setting the variable as a system variable.

I’ll attach the logs in the AM.

Feel free to access the org.

This behavior is still crazy. It’s basically saying I can’t successfully complete a build that is a cache miss (because this terminates the build) from a 2012 [R2] or 2016 server unless I install a third party tar or set this environment variable(even if it worked). This cache task should be self contained and bullet proof if it can cause the build to fail. IMO considering you took a dependency on 7 zip for untaring a tar (on 2012 and 2016 since tar isn’t a default install) you should use it for creating one instead of failing.

[jabbera]

PS: the reason the tar thing doesn’t really work well is that tar is not a default option on 2016 or 2012. IMO: You should fall back on 7 zip, and if that isn’t there fall back on files all automatically unless you don’t plan on supporting those platforms as build platforms.

The other thing to keep in mind is if you run your build (and cache) artifacts from a machine that supports tar, but you want to use those artifacts on a machine that doesn’t (windows 2016 without 7 zip) the cache will not hydrate. This may be a use case you don’t care about but again, consistency across platforms would solve this issue. Any way you can pack a tar in the task or in the plugin or make the server deal with compatibility?

[fadnavistanmay]

Hi @jabbera - I still can't access the org. Could you please check.

Thanks.

[jabbera]

I thought you meant on the backend. We don’t allow guest accounts in our tenant. I can open a ticket if that gives you more ability to get in. Otherwise I can do a screen share with you.

[fadnavistanmay]

My bad. Let's check the logs first, if nothing significant comes up, we can do a screen share.

[jabbera]

@fadnavistanmay I was able to reproduce this on my public azure devops. It really is as simple as installing the agent on a 2012 r2 server. I just created one in azure: https://dev.azure.com/mike-barry/Demo2012Issue/_build/results?buildId=420&view=results

If you need additional access please let me know

[jabbera]

@fadnavistanmay any chance to look at this in my public repo?

[johnterickson]

Let's make this error clearer that TAR needs to be on the path

[garfbradaz]

I had this issue on Windows Server 2016. I had to install 7zip and gnuwin32 tar manually and add to the PATH. I'm running self hosted agents as a service, so I restarted the service after adding to the PATH - this fixed the problem for me (saving and restoring from the cache).

@johnterickson / @jtpetty Will the documentation be updated to include the hard dependency on these applications because at the moment there isn't anything:

https://docs.microsoft.com/en-us/azure/devops/pipelines/caching/index?view=azure-devops

I agree with @jabbera though, should this task really have these hard dependencies at all or should it install these applications within the tools cache within _$(Agent.ToolsDirectory)_

[fadnavistanmay]

Hi @garfbradaz , sorry for the delay in documentation update. We have a PR out, it looks like it haven't merged yet. I'll follow up to get the docs updated.

[johnterickson]

Docs are live: https://docs.microsoft.com/en-us/azure/devops/pipelines/caching/index?view=azure-devops#required-software-on-self-hosted-agent

Thanks @fadnavistanmay!

@fadnavistanmay Can we add C:\Program Files\Git\usr\bin to the search path for tar on Windows?

[fadnavistanmay]

Hi @johnterickson , @b-barthel already has a PR out for "Can we add C:Program FilesGitusrbin to the search path for tar on Windows?"

PR: https://github.com/b-barthel/azure-pipelines-agent/commit/6e0d9b7a64cf48c9e558b3e81f86850916ec3e2a

[garfbradaz]

Hi @garfbradaz , sorry for the delay in documentation update. We have a PR out, it looks like it haven't merged yet. I'll follow up to get the docs updated.

Smashing thanks guys - top work as always.

[jtpetty]

The referenced PR should be included in 2.146.6 of the agent

[stephenmichaelf]

This has been rolled out, closing.