Singularity: Should not warn about inaccessible cache dir when cache is not needed.

[olifre]

Version of Singularity:

What version of Singularity are you using? Run:

$ singularity version
3.6.2-1.el7

Expected behavior

No warning being shown with:

cd /
singularity shell -C /cvmfs/container.example.com/singularity/Debian10/default/1600353192/

The container at /cvmfs/container.example.com/singularity/Debian10/default/1600353192/ is a sandbox container on read-only CVMFS, the user does not have a HOME on the machine, in case that is relevant.

Actual behavior

I get:

WARNING: Cache disabled - cache location / is not writable.

and then the container starts up fine. It's unclear what the cache would be useful for when running a sandbox container, and how to turn the warning off system-wide, and users are alerted that something is wrong, but I don't think that's actually anything to be alerted about for sandbox images, so the warning is just irritating.

Steps to reproduce this behavior

1. Use a sandbox container.
2. Use singularity shell -C on a node where the user does not have a home directory, and do not set SINGULARITY_CACHEDIR, then it seems / will be used.

What OS/distro are you running

$ cat /etc/os-release
CentOS 7.8

How did you install Singularity

EPEL 7.

Furhter information

With debug output activated (d), I get:

DEBUG   [U=67803,P=32219]  persistentPreRun()            Singularity version: 3.6.2-1.el7
DEBUG   [U=67803,P=32219]  persistentPreRun()            Parsing configuration file /etc/singularity/singularity.conf
DEBUG   [U=67803,P=32219]  handleConfDir()               Could not create /nonexistent/home/directory/.singularity: mkdir /nonexistent/home/directory/.singularity: no such file or directory
DEBUG   [U=67803,P=32219]  getCacheParentDir()           environment variable SINGULARITY_CACHEDIR not set, using default image cache
WARNING [U=67803,P=32219]  New()                         Cache disabled - cache location / is not writable.
DEBUG   [U=67803,P=32219]  execStarter()                 Checking for encrypted system partition

[dtrudg]

Patch contributions to address this are welcome but this isn't a high priority, given that:

• A non-existent $HOME is expected to be rare.
• The issue is a warning that doesn't affect functionality
• Singularity initializes cache dirs early.. before it knows what it is doing r.e. sandbox etc. It detects the image format (sandbox) later than it looks for $HOME config and cache things, so it needs some rework or duplication of logic to suppress the warning.

[dtrudg]

Made the topic a bit more generic - as it won't be exclusive to this exact situation.

[olifre]

A non-existent $HOME is expected to be rare.

Is it really? I would expect that to be the case on most HPC / HTC sites working with containers and local scratch directories, keeping the home directories separate from the work directories. At least I know several sites doing that, and we have done it for decades to remove interference between user applications in home directories and jobs running on the cluster.

The issue is a warning that doesn't affect functionality

That's true, but it was added (so one may call it a "regression") and alerted users at our site.

Singularity initializes cache dirs early.. before it knows what it is doing r.e. sandbox etc. It detects the image format (sandbox) later than it looks for $HOME config and cache things, so it needs some rework or duplication of logic to suppress the warning.

That's also what I grasped upon looking at the code. Sadly, I'm not a Go expert :-(.

Thanks for clarifying the title. I think we will then go with setting SINGULARITY_DISABLE_CACHE on all our worker nodes to work around this issue.

[dtrudg]

A non-existent $HOME is expected to be rare.

Is it really? I would expect that to be the case on most HPC / HTC sites working with containers and local scratch directories, keeping the home directories separate from the work directories. At least I know several sites doing that, and we have done it for decades to remove interference between user applications in home directories and jobs running on the cluster.

Hmm, maybe I'm just too biased here. Perhaps it's not uncommon in certain areas then? I've used quite a number of academic/research clusters at universities and broader facilities and have genuinely never come across a system that doesn't have $HOME available in jobs. Possibly reinforced from working in biomedical HPC / bioinformatics where a decent amount of tools won't work without a writable $HOME.

The issue is a warning that doesn't affect functionality

That's true, but it was added (so one may call it a "regression") and alerted users at our site.

Okay, let's call it both - the added warning is beneficial in some other scenarios, where people set a cache dir that isn't writable etc. and it is useful to know. It's just unfortunate it comes up when the cache isn't needed at all.

Singularity initializes cache dirs early.. before it knows what it is doing r.e. sandbox etc. It detects the image format (sandbox) later than it looks for $HOME config and cache things, so it needs some rework or duplication of logic to suppress the warning.

That's also what I grasped upon looking at the code. Sadly, I'm not a Go expert :-(.

Thanks for clarifying the title. I think we will then go with setting SINGULARITY_DISABLE_CACHE on all our worker nodes to work around this issue.

On a second look, this may actually be addressable through some changes for other tidying that is planned for 3.7.0. I'll tentatively tag it so I don't forget to look closer when we get there.

[olifre]

Perhaps it's not uncommon in certain areas then? I've used quite a number of academic/research clusters at universities and broader facilities and have genuinely never come across a system that doesn't have $HOME available in jobs. Possibly reinforced from working in biomedical HPC / bioinformatics where a decent amount of tools won't work without a writable $HOME.

To have this software work, we set $HOME to the scratch directory inside the actual compute job, but on the node on which the container is started, no HOME exists for the user (and I heard some sites even use the nobody user on the host system to run all jobs). But you are probably right, maybe I am biased towards a more HTC / Grid-like setup, in which we target maximum job mobility (i.e. the job may even go to a different cluster in a different datacenter and bring its Singularity container along, but not a home directory) ;-).

Okay, let's call it both - the added warning is beneficial in some other scenarios, where people set a cache dir that isn't writable etc. and it is useful to know. It's just unfortunate it comes up when the cache isn't needed at all.

I agree :-).

On a second look, this may actually be addressable through some changes for other tidying that is planned for 3.7.0. I'll tentatively tag it so I don't forget to look closer when we get there.

Thanks, that sounds good! Since we have a workaround, no urgency of course, but if it comes "cheaply" with 3.7.0, that would be nice.

[dtrudg]

@olifre - I overlooked a simple fix here previously. I hope that #5657 should fix your issue here.

[olifre]

I hope that #5657 should fix your issue here.

Indeed it does, I tested it jut now — many thanks for coming back to this!