sourcegraph/server disk usage could be improved

[saveman71]

I have a small AWS instance on which I installed sourcegraph, it has about 15 gigs of SSD.

df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           799M   81M  718M  11% /run
/dev/xvda1       15G   13G  1.7G  89% /
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           799M     0  799M   0% /run/user/1000

There is a "lot" of available space for sourcegraph to clone / index the repos I want (3 only). However the cleanup algorithm keeps purging them, and the they get recloned, indefinitely.

Here are the logs:

15:43:12         gitserver | t=2020-02-17T15:43:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-2/.git how old=20.801087248s free space in GiB=1.399 actual percent of disk space free=9.661 desired percent of disk space free=10.000 space freed in GiB=0.051 how much space to free in GiB=0.100
15:43:12         gitserver | t=2020-02-17T15:43:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-1/.git how old=20.27404129s free space in GiB=1.431 actual percent of disk space free=9.883 desired percent of disk space free=10.000 space freed in GiB=0.083 how much space to free in GiB=0.100
15:43:12         gitserver | t=2020-02-17T15:43:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-0/.git how old=6.218551242s free space in GiB=1.643 actual percent of disk space free=11.349 desired percent of disk space free=10.000 space freed in GiB=0.295 how much space to free in GiB=0.100
15:44:12         gitserver | t=2020-02-17T15:44:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-1/.git how old=22.95837904s free space in GiB=1.380 actual percent of disk space free=9.532 desired percent of disk space free=10.000 space freed in GiB=0.032 how much space to free in GiB=0.100
15:44:12         gitserver | t=2020-02-17T15:44:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-2/.git how old=20.515388499s free space in GiB=1.431 actual percent of disk space free=9.883 desired percent of disk space free=10.000 space freed in GiB=0.083 how much space to free in GiB=0.100
15:44:12         gitserver | t=2020-02-17T15:44:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-0/.git how old=6.286950014s free space in GiB=1.643 actual percent of disk space free=11.348 desired percent of disk space free=10.000 space freed in GiB=0.295 how much space to free in GiB=0.100
15:45:12         gitserver | t=2020-02-17T15:45:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-1/.git how old=22.146465173s free space in GiB=1.380 actual percent of disk space free=9.531 desired percent of disk space free=10.000 space freed in GiB=0.032 how much space to free in GiB=0.100
15:45:12         gitserver | t=2020-02-17T15:45:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-2/.git how old=19.755308976s free space in GiB=1.431 actual percent of disk space free=9.882 desired percent of disk space free=10.000 space freed in GiB=0.083 how much space to free in GiB=0.100
15:45:12         gitserver | t=2020-02-17T15:45:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-0/.git how old=6.291681476s free space in GiB=1.643 actual percent of disk space free=11.347 desired percent of disk space free=10.000 space freed in GiB=0.295 how much space to free in GiB=0.100
15:46:12         gitserver | t=2020-02-17T15:46:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-1/.git how old=21.893178262s free space in GiB=1.380 actual percent of disk space free=9.530 desired percent of disk space free=10.000 space freed in GiB=0.032 how much space to free in GiB=0.100
15:46:12         gitserver | t=2020-02-17T15:46:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-2/.git how old=19.490366496s free space in GiB=1.430 actual percent of disk space free=9.880 desired percent of disk space free=10.000 space freed in GiB=0.083 how much space to free in GiB=0.100
15:46:12         gitserver | t=2020-02-17T15:46:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-0/.git how old=5.924100952s free space in GiB=1.643 actual percent of disk space free=11.346 desired percent of disk space free=10.000 space freed in GiB=0.295 how much space to free in GiB=0.100
15:47:12         gitserver | t=2020-02-17T15:47:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-1/.git how old=22.890022255s free space in GiB=1.380 actual percent of disk space free=9.529 desired percent of disk space free=10.000 space freed in GiB=0.032 how much space to free in GiB=0.100
15:47:12         gitserver | t=2020-02-17T15:47:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-2/.git how old=13.865710726s free space in GiB=1.430 actual percent of disk space free=9.880 desired percent of disk space free=10.000 space freed in GiB=0.083 how much space to free in GiB=0.100
15:47:12         gitserver | t=2020-02-17T15:47:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-0/.git how old=4.716110614s free space in GiB=1.643 actual percent of disk space free=11.345 desired percent of disk space free=10.000 space freed in GiB=0.295 how much space to free in GiB=0.100
15:48:12         gitserver | t=2020-02-17T15:48:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-1/.git how old=21.191705472s free space in GiB=1.380 actual percent of disk space free=9.528 desired percent of disk space free=10.000 space freed in GiB=0.032 how much space to free in GiB=0.100
15:48:12         gitserver | t=2020-02-17T15:48:12+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-2/.git how old=19.028979426s free space in GiB=1.430 actual percent of disk space free=9.879 desired percent of disk space free=10.000 space freed in GiB=0.083 how much space to free in GiB=0.100
15:48:13         gitserver | t=2020-02-17T15:48:13+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-0/.git how old=5.430985275s free space in GiB=1.642 actual percent of disk space free=11.344 desired percent of disk space free=10.000 space freed in GiB=0.295 how much space to free in GiB=0.100
15:49:13         gitserver | t=2020-02-17T15:49:13+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-1/.git how old=21.097419021s free space in GiB=1.379 actual percent of disk space free=9.528 desired percent of disk space free=10.000 space freed in GiB=0.032 how much space to free in GiB=0.101
15:49:13         gitserver | t=2020-02-17T15:49:13+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-2/.git how old=14.658655568s free space in GiB=1.430 actual percent of disk space free=9.878 desired percent of disk space free=10.000 space freed in GiB=0.083 how much space to free in GiB=0.101
15:49:13         gitserver | t=2020-02-17T15:49:13+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-0/.git how old=5.13686668s free space in GiB=1.642 actual percent of disk space free=11.343 desired percent of disk space free=10.000 space freed in GiB=0.295 how much space to free in GiB=0.101
15:50:13         gitserver | t=2020-02-17T15:50:13+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-1/.git how old=21.176018536s free space in GiB=1.379 actual percent of disk space free=9.527 desired percent of disk space free=10.000 space freed in GiB=0.032 how much space to free in GiB=0.101
15:50:13         gitserver | t=2020-02-17T15:50:13+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-2/.git how old=19.188143942s free space in GiB=1.430 actual percent of disk space free=9.877 desired percent of disk space free=10.000 space freed in GiB=0.083 how much space to free in GiB=0.101
15:50:13         gitserver | t=2020-02-17T15:50:13+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-0/.git how old=4.818455017s free space in GiB=1.642 actual percent of disk space free=11.343 desired percent of disk space free=10.000 space freed in GiB=0.295 how much space to free in GiB=0.101
15:51:13         gitserver | t=2020-02-17T15:51:13+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-2/.git how old=18.731508011s free space in GiB=1.398 actual percent of disk space free=9.654 desired percent of disk space free=10.000 space freed in GiB=0.051 how much space to free in GiB=0.101
15:51:13         gitserver | t=2020-02-17T15:51:13+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-1/.git how old=18.283687781s free space in GiB=1.430 actual percent of disk space free=9.876 desired percent of disk space free=10.000 space freed in GiB=0.083 how much space to free in GiB=0.101
15:51:13         gitserver | t=2020-02-17T15:51:13+0000 lvl=warn msg="cleanup: removed least recently used repo" repo=/var/opt/sourcegraph/repos/github.com/org/repo-0/.git how old=3.97796611s free space in GiB=1.642 actual percent of disk space free=11.342 desired percent of disk space free=10.000 space freed in GiB=0.295 how much space to free in GiB=0.101

Is there a way to tell it to stop doing that? To disable the cleanup functionality altogether?

[unknwon]

As a workaround/solution, you could set env var SRC_REPOS_DESIRED_PERCENT_FREE to a lower number, e.g. 1 or 0, default is 10.

[saveman71]

Thank you! I feel this could be documented somewhere as some people may just run it somewhere space isn't important / can fill-up (e.g. laptops, staging servers).

I'm closing as this solves my issue. Feel free to re-open if this needs more follow-up.

[slimsag]

I am reopening this issue as a placeholder for us to investigate improving the disk utilization of sourcegraph/server.

I have seen a new customer report indicating that the .sourcegraph/data/cache (which stores archives of repositories) can take up around 97.5G out of the 100G Sourcegraph uses in total.

This is likely due to the fact that Sourcegraph is optimized for long-term production deployments. We should probably change the default here so that these archives are kept for a much less long duration of time on server deployments.