Singularity: Can't download DIGITS container from NGC

[GodloveD]

Version of Singularity:

3.0.2

Expected behavior

Docker is able to download the DIGITS container, so I expect Singularity to be able to as well.

Actual behavior

One of the layers returns a 401? Weird.

$ sudo singularity build digits.sif docker://nvcr.io/nvidia/digits:18.12-tensorflow
WARNING: Authentication token file not found : Only pulls of public images will succeed
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:7b8b6451c85f072fd0d7961c97be3fe6e2f772657d471254f6d52ad9f158a580
 41.40 MiB / 41.40 MiB [===================================================] 17s
Copying blob sha256:ab4d1096d9ba178819a3f71f17add95285b393e96d08c8a6bfc3446355bcdc49
 848 B / 848 B [============================================================] 0s
Copying blob sha256:e6797d1788acd741d33f4530106586ffee568be513d47e6e20a4c9bc3858822e
 619 B / 619 B [============================================================] 0s
Copying blob sha256:e25c5c290bded5267364aa9f59a18dd22a8b776d7658a41ffabbf691d8104e36
 168 B / 168 B [============================================================] 0s
Copying blob sha256:81577a65e38dd4a851976484e9cf298ab2cdaa68007d344d29a14f5dc8be6386
 7.36 MiB / 7.36 MiB [======================================================] 6s
Copying blob sha256:a0a17db2d4e0f08a16e9c268b7c7765488eeb20825d2aa9bbedda6fdcdce56a6
 12.72 KiB / 12.72 KiB [====================================================] 0s
Copying blob sha256:b8f595222deeb640f385e42506b2dcea7799e92625daa0c94251415a6780db82
 4.96 KiB / 4.96 KiB [======================================================] 0s
Copying blob sha256:94c925d7ecedb2a263aa650f3ff3634a6112b06045603d355eff39fe08c56093
 361.62 MiB / 361.62 MiB [===============================================] 2m11s
Copying blob sha256:1a0429a42ea35fad610aad8417f3b1a7306a483fa3471db9a1de5ca3f5ef969b
 3.47 KiB / 3.47 KiB [======================================================] 0s
Copying blob sha256:fe611d25e65a6c15ccff8c10d36f1283a1a6d66b143162c7efc17fe5b636da15
 601 B / 601 B [============================================================] 0s
Copying blob sha256:a5fe3ebd10260eecf17125c54c78ed15dd4c650d08f399b086029616db46a927
 1.36 KiB / 1.36 KiB [======================================================] 0s
Copying blob sha256:f8f38db78318c5690b68e79767673fbc2b8fdb90bb759140b6d11c0cd95de01c
 370.15 MiB / 370.15 MiB [===============================================] 2m42s
Copying blob sha256:f838f8b44a2578f4f65593e6d20ceee777f956ea3ac543b57a159686c49ab230
 89 B / 89 B [==============================================================] 0s
Copying blob sha256:1b03f309d18aa08d82eb1aa4979d0adbd4b8be4a0f0ea4d00d0820f6bb95299f
 11.27 KiB / 11.27 KiB [====================================================] 0s
Copying blob sha256:b98d30db49c2058a3816a190cadc9102122c7d90563a0679602551a68a0a8340
 179 B / 179 B [============================================================] 0s
Copying blob sha256:e4ca2959244e0c47128cf42ef90d8e341b86072c9dc9f32fa8815da4cf4254f7
 168.66 KiB / 168.66 KiB [==================================================] 0s
Copying blob sha256:fecc1b9b8c8df09b32bfd6a7303f43181fdae8c16725773f9d219ad2f40bec44
 724 B / 724 B [============================================================] 0s
Copying blob sha256:55eca9e8415b26ea63c5a57004ba8532647a4ca55fa004fea2c5b633f52f0c02
 87.94 MiB / 87.94 MiB [===================================================] 39s
Copying blob sha256:0752dd679c24971b3e3af7d4d6defc0117e6af6605b3b54ea68dcf9c0627475a
 4.13 MiB / 4.13 MiB [======================================================] 3s
Copying blob sha256:67d3dd2846d538d1230eee029f17fa4c9f0256daa13340a6c8755ceac714060a
 4.68 MiB / 4.68 MiB [======================================================] 2s
Copying blob sha256:75e59224ce1c314183e1bb2e13bff76893e550b63e67303e9ef59bef13b46008
 4.96 KiB / 4.96 KiB [======================================================] 0s
Copying blob sha256:360b5dfb9a449ac17d1950ca7c8a50260a0b6b3b62596412d129ba2a755cc7de
 507 B / 507 B [============================================================] 0s
Copying blob sha256:533a69b03556000e1e13da1411e377b14f1ed1686df9d68d4a5995ea086ae264
 91.25 MiB / 91.25 MiB [===================================================] 38s
Copying blob sha256:362820a2c0c304ce3b7c8c202e500faaa9a1a721f0e11ac7111111578ee24305
 73.49 MiB / 73.49 MiB [===================================================] 24s
Copying blob sha256:fdc7f0b000d6bdfd5d93f12a79d8faf241832b6b7b56cebe4a2db3c9e2fdc612
 163 B / 163 B [============================================================] 0s
Copying blob sha256:aeede4ac3334c69d293a4728c6a973b82563c65c6f31e0c8b1dc94c9dbe693d9
 206 B / 206 B [============================================================] 0s
Copying blob sha256:dadb2a2bdac4676c73ac595166de21bb9f39188d788a4d3ba83557e9d11a7d08
 4.99 MiB / 4.99 MiB [======================================================] 3s
Copying blob sha256:bed72b8e03b5dc765928c3719d033302bc86acc5b99484f80557762dc4f38bd6
 37.40 MiB / 37.40 MiB [===================================================] 19s
Copying blob sha256:312405ba168f8c4a97a220a2a2c18eaccc17de7d695e26e72f44bc0500d27c9c
 94.57 MiB / 94.57 MiB [===================================================] 38s
Copying blob sha256:190b9bad439fb3dee10ab25048e00e1d9f4c48194c708d40724d2ecdafc9f925
FATAL:   While performing build: conveyor failed to get: Error initializing source oci:/root/.singularity/cache/oci:39edb868eb2918b6fd2482cbc1c100ebf9dd35a1f2db94270001fa9917877081: Error reading blob sha256:190b9bad439fb3dee10ab25048e00e1d9f4c48194c708d40724d2ecdafc9f925: Invalid status code returned when fetching blob 401

Steps to reproduce behavior

Generate a token on the NCG and stick it in $SINGULARITY_DOCKER_PASSWORD, then set $SINGULARITY_DOCKER_USERNAME equal to $oauthtoken (the actual literal string), and run:

$ sudo singularity build digits.sif docker://nvcr.io/nvidia/digits:18.12-tensorflow

Note

If I run the command 2x in a row without clearing the cache it seems to pick up where it left off and will download 190b9bad439. So maybe it's timing out during the pull? Is that us or them?

[sleeepyjack]

I can reproduce this behaviour with a different NGC image (docker://nvcr.io/nvidia/cuda:10.1-devel-ubuntu18.04)

In my case running the command 3x in a row without clearing the cache resolved the problem.

[truatpasteurdotfr]

I have just tried with singularity 3.2.0 and no issue:

export SINGULARITY_DOCKER_PASSWORD=...
export SINGULARITY_DOCKER_USERNAME=$oauthtoken
singularity build cuda.sif docker://nvcr.io/nvidia/cuda:10.1-devel-ubuntu18.04
or
singularity build digits.sif docker://nvcr.io/nvidia/digits:18.12-tensorflow

[sleeepyjack]

Oh, I should have mentioned that I am using v3.1.0-1 from the Arch AUR repo:

Updating to 3.2.0 might do the trick. Thx @truatpasteurdotfr !

[mem]

The issue cannot be reproduced as described with commit 47181b69bee9557a8d5dbe91398fb91e68a55bde.

[husi253]

I can reproduce this behaviour with a different NGC image (docker://nvcr.io/nvidia/cuda:10.1-devel-ubuntu18.04)

In my case running the command 3x in a row without clearing the cache resolved the problem.

Similarly, using v3.5.3 from Ubuntu to pull docker://nvcr.io/nvidia/tensorflow:20.01-tf2-py3.
Run the command 3x in a row without clearing the cache resolved the problem.

[dtrudg]

@husi253 - what's behavior / errors do you see before it completes ok?

[husi253]

@husi253 - what's behavior / errors do you see before it completes ok?

@dctrud

1st error:
FATAL: While making image from oci registry: while building SIF from layers: conveyor failed to get: Error initializing source oci:/home/.singularity/cache/oci:0eff30886deb9597caa789f8266f005a2e5fe9feaf9d98a3d82db8ded88e0b17: Error reading blob sha256:36289354f878faeae182683b314bad424d6d9f65ba833d288efc476ef519e164: Invalid status code returned when fetching blob 401 (Unauthorized)

2nd error:
FATAL: While making image from oci registry: while building SIF from layers: conveyor failed to get: Error initializing source oci:/home/.singularity/cache/oci:0eff30886deb9597caa789f8266f005a2e5fe9feaf9d98a3d82db8ded88e0b17: Error reading blob sha256:933f8815751a343526048260cede60bc994bd5c0b390460d079eea7ba31f4292: Invalid status code returned when fetching blob 401 (Unauthorized)

[dtrudg]

That's pretty odd you get 2 401 unauthorized responses, but the 3rd one works. We are using some pretty standard libraries to do pulls from OCI registries - will have to trace this when I get a chance.

Just to check - can you confirm whether or not there is any proxy server in use in your environment? Thanks.

[husi253]

I didn't use any proxy when pull the image.