Singularity: Cannot pull from nvidia/cuda inside a Docker container on v3.0.0

Created on 9 Oct 2018 · 14Comments · Source: hpcng/singularity

Version of Singularity:

v3.0.0

Expected behavior

Running singularity build test.simg docker://nvidia/cuda:9.0-base inside a docker container would produce an image

Actual behavior

Get an error

...
FATAL:   While performing build: packer failed to pack: While unpacking tmpfs: unpack: error extracting layer: unable to delete whiteout path: lstat /tmp/sbuild-978382858/fs/etc/ca-certificates/.wh..opq: operation not permitted

Steps to reproduce behavior

Execute

docker run -it --privileged deephorizons/singularity:3.0.0 bash
singularity build test.simg docker://nvidia/cuda:9.0-base

Also, there is no /tmp/sbuild-978382858/fs directory after the error, but there is a /tmp/sbuild-978382858 directory.

Bug Hacktoberfest ImageBuilding Release 3.1.0

Source

DeepHorizons

All 14 comments

Heya @DeepHorizons. Thanks for being an early adopter! This is clearly a critical omission.

GodloveD on 9 Oct 2018

Heya @DeepHorizons it looks like a permissions issue inside Docker
I am able to build from the given URI

[eduardo@rhel7 tmp]$ singularity build test.simg docker://nvidia/cuda:9.0-base
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:3b37166ec61459e76e33282dda08f2a9cd698ca7e3d6bc44e6a6e7580cdeff8e
 41.25 MiB / 41.25 MiB [====================================================] 3s
Copying blob sha256:504facff238fde83f1ca8f9f54520b4219c5b8f80be9616ddc52d31448a044bd
 846 B / 846 B [============================================================] 0s
Copying blob sha256:ebbcacd28e101968415b0c812b2d2dc60f969e36b0b08c073bf796e12b1bb449
 615 B / 615 B [============================================================] 0s
Copying blob sha256:c7fb3351ecad291a88b92b600037e2435c84a347683d540042086fe72c902b8a
 850 B / 850 B [============================================================] 0s
Copying blob sha256:2e3debadcbf7e542e2aefbce1b64a358b1931fb403b3e4aeca27cb4d809d56c2
 168 B / 168 B [============================================================] 0s
Copying blob sha256:173fe415719fa7aea6fcc20dfe5385c5be6f6ffe5c5054fca5e8233c3b44888a
 6.65 MiB / 6.65 MiB [======================================================] 0s
Copying blob sha256:837ba756cd3ad3aac72746e63f8736fb3a1f2bf569328b1efef8fe57ab9b04ab
 397.35 KiB / 397.35 KiB [==================================================] 0s
Copying blob sha256:08a737afb63f13c0e8383e224a78559ff8655ac658453ff0e001fabb525c31b0
 182 B / 182 B [============================================================] 0s
Copying config sha256:a5b4176a79ac64d62793c860fa5c0dda96bb89a8108de9d4dd424593094e57c8
 6.27 KiB / 6.27 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
INFO:    Creating SIF file...
INFO:    Build complete: test.simg
````

Could you run

singularity -v -d build test.simg docker://nvidia/cuda:9.0-base


and paste here the VERBOSE Debug output, thanks.

In the mean while you could also try our [Remote-build](https://www.sylabs.io/2018/04/sylabs-remote-build-service/)

export SYLABS_TOKEN=
docker run -it --privileged -e SYLABS_TOKEN deephorizons/singularity:3.0.0 bash
singularity build --remote test.simg docker://nvidia/cuda:9.0-base
```

ArangoGutierrez on 9 Oct 2018

@ArangoGutierrez

root@020ca2c75acb:/# singularity -v -d build test.simg docker://nvidia/cuda:9.0-base
WARNING [U=0,P=16]         sylabsToken()                 Authentication token file not found : Only pulls of public images will succeed
DEBUG   [U=0,P=16]         func1()                       Registering plugin as type BuildPlugin
DEBUG   [U=0,P=16]         NewBundle()                   Created temporary directory for bundle /tmp/sbuild-867613410
INFO    [U=0,P=16]         Full()                        Starting build...
DEBUG   [U=0,P=16]         initCacheDir()                Creating cache directory: /root/.singularity/cache
DEBUG   [U=0,P=16]         initCacheDir()                Creating cache directory: /root/.singularity/cache/oci
DEBUG   [U=0,P=16]         updateCacheSubdir()           Caching directory set to /root/.singularity/cache/oci
Getting image source signatures
Copying blob sha256:3b37166ec61459e76e33282dda08f2a9cd698ca7e3d6bc44e6a6e7580cdeff8e
 41.25 MiB / 41.25 MiB [====================================================] 1s
Copying blob sha256:504facff238fde83f1ca8f9f54520b4219c5b8f80be9616ddc52d31448a044bd
 846 B / 846 B [============================================================] 0s
Copying blob sha256:ebbcacd28e101968415b0c812b2d2dc60f969e36b0b08c073bf796e12b1bb449
 615 B / 615 B [============================================================] 0s
Copying blob sha256:c7fb3351ecad291a88b92b600037e2435c84a347683d540042086fe72c902b8a
 850 B / 850 B [============================================================] 0s
Copying blob sha256:2e3debadcbf7e542e2aefbce1b64a358b1931fb403b3e4aeca27cb4d809d56c2
 168 B / 168 B [============================================================] 0s
Copying blob sha256:173fe415719fa7aea6fcc20dfe5385c5be6f6ffe5c5054fca5e8233c3b44888a
 6.65 MiB / 6.65 MiB [======================================================] 0s
Copying blob sha256:837ba756cd3ad3aac72746e63f8736fb3a1f2bf569328b1efef8fe57ab9b04ab
 397.35 KiB / 397.35 KiB [==================================================] 0s
Copying blob sha256:08a737afb63f13c0e8383e224a78559ff8655ac658453ff0e001fabb525c31b0
 182 B / 182 B [============================================================] 0s
Copying config sha256:a5b4176a79ac64d62793c860fa5c0dda96bb89a8108de9d4dd424593094e57c8
 6.27 KiB / 6.27 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
FATAL   [U=0,P=16]         func6()                       While performing build: packer failed to pack: While unpacking tmpfs: unpack: error extracting layer: unable to delete whiteout path: lstat /tmp/sbuild-867613410/fs/etc/ca-certificates/.wh..opq: operation not permitted

DeepHorizons on 9 Oct 2018

And another bug, I can't signup for Syslabs to get a token.

DeepHorizons on 9 Oct 2018

I just created a new token, can you tell me what issue you had on the web UI?

ArangoGutierrez on 9 Oct 2018

After I authenticated using GitHub I got a 502 Bad Gateway on the redirect

DeepHorizons on 9 Oct 2018

Going to PING someone on the web front. thanks for your time

ArangoGutierrez on 9 Oct 2018

Heya @DeepHorizons , are you getting the same Err today in the web page?

ArangoGutierrez on 11 Oct 2018

Heya @DeepHorizons. Sorry for the long delay. Can you have a look at the 3.0.2 release candidate and make sure this works as expected?

GodloveD on 3 Jan 2019

I think this is fixed in 3.0.2. If not, feel free to reopen!

GodloveD on 10 Jan 2019

I would like to re-open this issue. I've done some testing with 3.1.0 and I have the same error.

When I was testing with 3.0.0, I eventually tested Singularity on the host and had the same problem. I wrote it off as a virtualisation problem (running on OVirt) and left the ticket open. With the 3.1.0 release, Singularity now runs on the host but does not run in Docker. Oddly enough, it seems to break only on certain images.

Images that work:

ubuntu
ubuntu:16.04
ubuntu:18.04
jupyter/base-notebook
jupyterhub/jupyterhub
nvidia/k8s-device-plugin:1.11

Images it does not work for:

nvidia/cuda:9.0-base-ubuntu16.04
nvidia/opencl

With any of the above images, I get the error:

...
Writing manifest to image destination
Storing signatures
DEBUG   [U=0,P=360]        cleanUp()                     Build bundle cleanup: /tmp/sbuild-135612651
FATAL   [U=0,P=360]        run()                         While performing build: packer failed to pack: While unpacking tmpfs: unpack: error extracting layer: unable to delete whiteout path: lstat /tmp/sbuild-135612651/fs/etc/ca-certificates/.wh..opq: operation not permitted

P.S. I no longer need "--privileged" docker containers to build images, good work!

DeepHorizons on 5 Mar 2019

@DeepHorizons - this issue may be related to your docker configuration. If your docker setup on the host is using an fs driver that is itself using aufs style whiteout files, the whiteout handling that Singularity must apply when pulling a docker image may be blocked. It is occuring in the cuda images as these are relatively complex, with an abundance of whiteouts / opaque directories from upper layers.

On your host, what do you get from... docker info | grep "Storage Driver"

If it's aufs you are likely to run into whiteout issues with Singularity nested inside docker. A workaround is to switch to an alternate driver. See e.g. https://markshust.com/2017/03/02/making-docker-mac-faster-overlay2-filesystem/

If the docker info isn't reporting aufs then this issue is something else that I've not seen before.

dtrudg on 5 Mar 2019

👍1

See also e.g. here: https://github.com/GoogleContainerTools/kaniko/issues/566 for an example that this is a broader issue for docker with aufs backend. It's a pretty generic issue for anything that does container building within a docker container, where the host docker is running aufs

dtrudg on 5 Mar 2019

👍1

That seemed to solve it @dctrud, Thanks for the info

DeepHorizons on 6 Mar 2019

Was this page helpful?

0 / 5 - 0 ratings