Azure-pipelines-agent: Agent runs out of disk space because gc.auto is set to 0

[ChristianStadelmann]

Steps to reproduce:

1. have a large git repo
2. have a build definition defined using this large git repo
3. in the build definition ADS frontend under _Get Sources_ have _Clean Options_ set to _Sources and output directory_ but don't have _Shallow fetch_ enabled.

What happens:

After some time, the repository grows to be quite large. The .git folder contains tens of thousands of files in its objects/ subfolder.

What should happen:

The agent should automatically run git gc regularly so that the objects/ subfolder will always contain at most a few thousand items.

Additional details:

Further investigation showed that every build definition in every agent's clone (from _Get sources_) will create a .git/config file with gc.auto=0.

Workarounds:

Workaround 1 editing git config

On every agent, open every git config file and call git config gc.auto=256.
Note that this will sometimes result in warnings that git prune shall be executed too.

Workaround 2 using build steps

Introduce a build step that calls git gc --auto after setting git config gc.auto=256. Calling git prune after git gc --auto is also necessary sometimes.
Example powershell script:

# Debugging current git config
git config --list --show-origin
git config --global gc.auto 256
# Need to overwrite local git repo config
# default is gc.auto=0
git config --local gc.auto 256
git prune
# Execute Git GC here
git gc --auto

Agent Version and Platform

Agent version info: 2.175.2 (same on a few different / more recent versions. Relevant code referenced below indicates that this issue is there by intention.)

OS: Windows 10

Azure DevOps Type and Version

On-premise Azure DevOps Server 2019

What's not working?

Can we please have the agent's default value for git's gc.auto set to the default (6700) or some other reasonable value (e.g. 256)?

Relevant code

It looks like there are two places hard-coded where gc.auto is being set to 0:

• https://github.com/microsoft/azure-pipelines-agent/blob/1ade25312c505bd81e4f01f2f6c258e29e7184ec/src/Agent.Worker/Build/GitCommandManager.cs#L78
• https://github.com/microsoft/azure-pipelines-agent/blob/f41d99d738767c46a2a907e5fce142e23b1c7602/src/Agent.Plugins/GitSourceProvider.cs#L647

[ppejovic]

@ChristianStadelmann Do you have maintenance jobs enabled on your agent pool? These perform a git repack -f and prune on a defined schedule.

[PaulVrugt]

@ppejovic maintenance jobs don't help in a virtual machine scale set agent. We configured the scale set to re-provision agents each night, but we're getting disk space errors during the day. Even though we have about 35Gb free disk space on the agent. So a maintenance job would run too late.

[ChristianStadelmann]

@ppejovic wrote:

@ChristianStadelmann Do you have maintenance jobs enabled on your agent pool? These perform a git repack -f and prune on a defined schedule.

This helps quite a lot. Thank you!

The issues linked above and some experience in my organization show that there is an issue if you need cleanup more than once per 24 hours or in other special setups.

[ChristianStadelmann]

For me, the maintenance jobs serve solve the problems described above. For my use case, this issue can be closed. It may not fit everyone's use case.

[anatolybolshakov]

@PaulVrugt would it work work you if it would be possible to run maintenance job more than one time during the day?

[PaulVrugt]

@anatolybolshakov yes this would work for us I think. Right now we added manual clean tasks to the end of some of our pipelines to avoid the disk to fill up during the day, but being able to run maintenance during the day would be a much more elegant solution.

Of course, a fully automated monitoring that automatically triggers a maintenance job if disk space is low would be even better, but that might be to big of an ask :)

[anatolybolshakov]

@PaulVrugt got it, thanks! Since this sounds like some DevOps side feature - could you please create ticket on https://developercommunity.visualstudio.com/search?space=21 to get right eyes on it?

[anatolybolshakov]

Let me close this one at the moment since there is nothing to do on agent side currently.