Singularity: Unable to properly remove Singularity cache

[Amir-Arsalan]

Version of Singularity:

3.1.1

Expected behavior

Create an image from scratch.

Actual behavior

It looks like doing singularity cache clean -a does not properly clean the cache

Steps to reproduce behavior

I pull NVIDIA's docker image when building my container but for some reason the container build process stops after I choose the OS language (I type 29 and hit enter). I noticed that every time that I run the build script the following lines appear quickly. In the past however, it would take a while for the blobs to get downloaded. I thought this might be causing the issue that the build process stops and I should remove these blobs to build the container from scratch. Note that this was happening when I was using Singularity 3.0. So I upgraded my Singularity version to 3.1.1 and did singularity cache clean -a to remove the blobs but I still face the same issue. singularity cache list also does not show anything. I also manually removed everything in /tmp that were created by Singularity. I wonder whether I should do something else to remove these blobs?

Getting image source signatures
Skipping fetch of repeat blob sha256:34667c7e4631207d64c99e798aafe8ecaedcbda89fb9166203525235cc4d72b9
Skipping fetch of repeat blob sha256:d18d76a881a47e51f4210b97ebeda458767aa6a493b244b4b40bfe0b1ddd2c42
Skipping fetch of repeat blob sha256:119c7358fbfc2897ed63529451df83614c694a8abbd9e960045c1b0b2dc8a4a1
Skipping fetch of repeat blob sha256:2aaf13f3eff07aa25f73813096bd588e6408b514288651402aa3d0357509be7a
Skipping fetch of repeat blob sha256:4d96b2dafaa531e30688f7cbfe03ea807ebe2dac9ca96238aa696d3a63fcb8a6
Skipping fetch of repeat blob sha256:f8c41b380cab6c46eb01ab3e42913ef0d7884fa3f88f9dd082b571cedf0c101a
Skipping fetch of repeat blob sha256:d2c1b485844605305214fd7af18c8b568559bcd5a6e06efb80f6ebb693968047
Skipping fetch of repeat blob sha256:ba985066a7db79c080dd7d4604f8f182e98942377beb9d0720b32bd8421cfeb3
Skipping fetch of repeat blob sha256:82635ec7e8dd88ced53cc5e1a0019d8e72524ee067db7a4b27f6812d61df37f6
Skipping fetch of repeat blob sha256:a2b9b06a3798937048c1025c31616934123b61e3ac476d74e11483c0302f1527
Copying config sha256:af666373bf77db86206737bee9809d7cbb015ab76ca0c304e8dc76f0a4bb79de

And here's my container definition file:

BootStrap: docker
From: nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04

I do sudo singularity build --sandbox cudnnContainer.img DefinitionFile.def to build my container.

[WestleyK]

Since you use sudo to build your container, the cache will be stored in /root/.singularity/cache (not /home/user/.singularity/cache). So to clear you cache, you will need to use sudo, try:

$ sudo singularity cache clean --all

[GodloveD]

I think this is simply a misunderstanding.

First, the Singularity cache is stored in ~/.singularity/cache. Images are built in temporary directories under /tmp but those directories are not reused, so deleting them is not expected to affect caching.

Second when you execute the following command, you do so as root. So the cache ends up in /root/.singularity/cache.

$ sudo singularity build --sandbox cudnnContainer.img DefinitionFile.def

So if you need to clear the cache after building as root, you must do so as root.

$ sudo singularity cache clean -a

Does that make sense?

Looks like this info is recorded here. It should probably be made more clear in a note or something that the cache clean -a command will likely need to be executed as root. I'll open an issue on the docs to that effect and close this issue. Feel free to reopen on ask us to if anything else is needed.

[GodloveD]

Oh whoops. @WestleyK and I were responding at the same time. 😸