Azure-pipelines-agent: Azure DevOps - Self-Hosted Agents - Residual Artifacts Causing Subsequent Pipeline Execution Failures

[AErmie]

Here's an interesting and perplexing issue...

I’m using Azure DevOps Server (on-premises), in connection with using Self-Hosted Agents.

When executing a pipeline that creates several directories (ie. mkdir ABC), and also creates/stores files in these directories, on the very first time the pipeline is executed, everything works without issue.

However, on all subsequent pipeline execution attempts, the very first step (ie. Checkout 'repoName') fails. We've attempted to add the following 'cleaning' commands to the execution, but those didn't resolve the issue.

-job: Job
workspace:
   clean: all
steps:
- checkout: self
   submodules: true
   clean: true

Digging deeper into the debugging... we see the following error reoccurring:

##[error]One or more errors occurred. (One or more errors occurred. (Access to the path '/data/TFSBuilds/01/299/s/terraform/super-linter.report/super-linter-TERRAFORM_TERRASCAN.tap' is denied.)) (One or more errors occurred. (Access to the path '/data/TFSBuilds/01/299/s/terraform/super-linter.report/super-linter-JSON.tap' is denied.)) (Access to the path '/data/TFSBuilds/01/299/s/terraform/super-linter.report/super-linter-TERRAFORM_TERRASCAN.tap' is denied.) (Access to the path '/data/TFSBuilds/01/299/s/terraform/super-linter.report/super-linter-JSON.tap' is denied.)

Note: The referenced 'super-linter.report/super-linter-TERRAFORM_TERRASCAN.tap' and 'super-linter.report/super-linter-TERRAFORM_TERRASCAN.tap' are the folders/files that get created within this pipeline.

This affects all subsequent steps/tasks within the pipeline because they now reference/use existing files versus re-generating them on each independent pipeline execution. What we would appreciate the insight into, is how self-hosted agents are managed and configured, in particular around residual artifacts that may be left behind from pipeline execution.

NOTE: This issue does not occur when executing pipelines using Azure DevOps online, as each Agent machine is ephemeral and thrown away at the end of the execution.

Appreciate any thoughts/insights.

[anatolybolshakov]

Hi @AErmie could you please share full agent logs containing this error?

[AErmie]

Hey @anatolybolshakov, sorry for the delay (I missed your message). Here are the logs.
logs_93296.zip

[vladislav-ryzhov]

Hi @AErmie.
Unfortunately, we were unable to reproduce your issue. Could you tell us more clearly about your issue?
There is one command in your YML file "docker exec superLinter chown -R "$(id -u):$(id -g)" /tmp/lint --verbose"
It looks like these two lines from GitHub Super-Linter Code Scan logs(screenshot №1) are the result of this command. Could this be the cause of your problem? Could docker have changed something in the access rules?

[AErmie]

Hi @vladislav-ryzhov
The extra commands (namely: chown -R "$(id -u):$(id -g)") were required so that the temporary files and directories created by running the Super-Linter are run under 'root'. This causes issues with self-hosted agents, as the Agent account is not root, and therefore cannot delete/clean up any residual files/folders leftover from a previous pipeline run.

[vladislav-ryzhov]

Hi @AErmie .
We tried to run superLinter under the usual user by using this command: --user 1000:1000, but we got this issue

It looks like this plugin cannot work without "root" but git checkout fails then.
So, if you want to use agent under "root" you should configure your agent as a service by this command: sudo ./svc.sh install root (you should use root)
More info you can find here.

[vladislav-ryzhov]

@AErmie Let me close this one since this is external issue. Please feel free to ask any further questions.