Singularity: No space left on device

[renganxu]

Version of Singularity:

2.4-dist

Expected behavior

After building the Singularity image, I should be able to install Tensorflow inside the container successfully.

Actual behavior

There is error message "IOError: [Errno 28] No space left on device". The other detailed messages are:
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 784, in install
**kwargs
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 377, in move_wheel_files
clobber(source, dest, False, fixer=fixer, filter=filter)
File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 323, in clobber
shutil.copyfile(srcfile, destfile)
File "/usr/lib/python2.7/shutil.py", line 84, in copyfile
copyfileobj(fsrc, fdst)
File "/usr/lib/python2.7/shutil.py", line 52, in copyfileobj
fdst.write(buf)
IOError: [Errno 28] No space left on device

The space in the container is as follows:
Singularity ubuntu16.04_tensorflow_dev.img:/mnt/tensorflow> df -h
Filesystem Size Used Avail Use% Mounted on
/dev/loop0 2.7G 2.7G 0 100% /
/dev/mapper/cl-root 35G 27G 8.1G 77% /tmp
devtmpfs 126G 0 126G 0% /dev
tmpfs 126G 96K 126G 1% /dev/shm
/dev/mapper/cl-home 17G 11G 6.7G 61% /mnt

Steps to reproduce behavior

$ singularity build --writable tensorflow_dev.img ubuntu.def
$ singularity exec -s /bin/bash -B $PWD:/mnt -w tensorflow_dev.img /mnt/build_tensorflow.sh
$ cat build_tensorflow.sh
cd /mnt
git clone https://github.com/tensorflow/tensorflow
cd tensorflow
./configure
bazel build -c opt //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
**pip install /tmp/tensorflow_pkg/tensorflow-1.4.0-cp27-cp27mu-linux_x86_64.whl**

The error happened in the last command "pip install ...".

Anyone else has the same error and know how to fix it? Thanks.

Regards,
Rengan

[jekriske]

This is similar to https://github.com/singularityware/singularity/issues/1076
the answer is to build using the -s or --sandbox flag instead of building with --writable

[renganxu]

Thanks @jekriske. I just tried --sandbox and it works. So it this a bug in --writable option, or we should always use --sandbox in the future?

[jekriske-lilly]

It is a bug, but I'm not under the impression that it's going to be fixed. Containers are supposed to be immutable anyway. The sandbox is a much better alternative for development because you don't have to worry about size. You can always convert it later to a squashfs image via
sudo singularity build image_name.simg sandbox_dir

[gmkurtzer]

This is technically not a bug. The --writable option will put the container into a file system with a predefined mount of extra buffer space. You are simply consuming that extra buffer space after the build which is why the sandbox works properly.

If you have to use a writable image, I suggest you precreate the writable image with the singularity image.create --size .... command and then build into that. But I want to reiterate what @jekriske-lilly mentioned in that, it is better to use the default immutable compressed image format, and you can do that from a sandbox.

Good luck!

[mmokrejs]

I hit this issue as well. Please get the page http://singularity.lbl.gov/quickstart fixed. What is the point to make a writable image if it does not support modifications (typically requiring some additional space)? So running

singularity build --writable ubuntu.img docker://ubuntu
singularity shell --writable ubuntu.img              
Singularity ubuntu.img:~> apt-get update
Ign:5 http://archive.ubuntu.com/ubuntu xenial/universe Sources                                                                                                                               
Err:8 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages                                                                                                                        
  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device) [IP: 91.189.88.149 80]

was just a useless exercise to me.

I am evidently misunderstanding the text but the page says the sandbox images can be modified but the changes will be ephemeral. So practically I cannot adjust the packages installed in them. So --sandbox is not a way to go for me.

You can use commands like shell, exec, and run with this directory just as you would with a Singularity image. You can also write files to this directory from within a Singularity session (provided you have the permissions to do so). These files will be ephemeral and will disappear when the container is finished executing. However if you use the --writable option the changes will be saved into your directory so that you can use them the next time you use your container.

[GodloveD]

It means if you use the --writable option with the sandbox your changes
will be saved inside the sandbox. Try it!

Dave

On Wed, Jan 24, 2018 at 10:38 AM, Martin Mokrejš [e-mail]
wrote:

I hit this issue as well. Please get the page http://singularity.lbl.gov/
quickstart fixed. What is the point to make a writable image if it does
not support modifications (typically requiring some additional space). So
running

singularity build --writableubuntu.img docker://ubuntu
singularity shell --writable ubuntu.img
Singularity ubuntu.img:~> apt-get update
Ign:5 http://archive.ubuntu.com/ubuntu xenial/universe Sources
Err:8 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device) [IP: 91.189.88.149 80]

was just a useless exercise to me.

I am evidently misunderstanding the text but the page says the sandbox
images can be modified but the changes will be ephemeral. So practically I
cannot adjust the packages installed in them. So --sandbox is not a way
to go for me.

You can use commands like shell, exec, and run with this directory just as
you would with a Singularity image. You can also write files to this
directory from within a Singularity session (provided you have the
permissions to do so). These files will be ephemeral and will disappear
when the container is finished executing. However if you use the --writable
option the changes will be saved into your directory so that you can use
them the next time you use your container.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/singularityware/singularity/issues/1165#issuecomment-360174340,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHUUXDzNiRBX1ZM_RHFcpNtFOqCBdLYzks5tN06EgaJpZM4Qgy5G
.

[GodloveD]

Perhaps this page can shed a bit more light on the topic.

http://singularity.lbl.gov/docs-build-container#creating---writable-images-and---sandbox-directories

On Wed, Jan 24, 2018 at 10:43 AM, David Godlove [e-mail]
wrote:

It means if you use the --writable option with the sandbox your changes
will be saved inside the sandbox. Try it!

Dave

On Wed, Jan 24, 2018 at 10:38 AM, Martin Mokrejš <[email protected]

wrote:

I hit this issue as well. Please get the page
http://singularity.lbl.gov/quickstart fixed. What is the point to make a
writable image if it does not support modifications (typically requiring
some additional space). So running

singularity build --writableubuntu.img docker://ubuntu
singularity shell --writable ubuntu.img
Singularity ubuntu.img:~> apt-get update
Ign:5 http://archive.ubuntu.com/ubuntu xenial/universe Sources
Err:8 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device) [IP: 91.189.88.149 80]

was just a useless exercise to me.

I am evidently misunderstanding the text but the page says the sandbox
images can be modified but the changes will be ephemeral. So practically I
cannot adjust the packages installed in them. So --sandbox is not a way
to go for me.

You can use commands like shell, exec, and run with this directory just
as you would with a Singularity image. You can also write files to this
directory from within a Singularity session (provided you have the
permissions to do so). These files will be ephemeral and will disappear
when the container is finished executing. However if you use the --writable
option the changes will be saved into your directory so that you can use
them the next time you use your container.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/singularityware/singularity/issues/1165#issuecomment-360174340,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHUUXDzNiRBX1ZM_RHFcpNtFOqCBdLYzks5tN06EgaJpZM4Qgy5G
.

[mmokrejs]

Thanks, this works for me.

root# singularity build --sandbox ubuntu_sandbox_dir docker://ubuntu
root# singularity shell --writable ubuntu_sandbox_dir
Singularity ubuntu.img:~> apt-get update
Singularity ubuntu.img:~> apt-get install my_apps
Singularity ubuntu.img:~> exit
root# singularity build ubuntu.img ubuntu_sandbox_dir

[GodloveD]

So since using a sandbox is the key to this issue (along with the --writable runtime flag) I'm just going to go ahead and close. If anyone needs further assistance please feel free to re-open!

[bermanmaxim]

However the --sandbox variant needs root to be built and cannot be easily transferred to another computer. I wonder how to create a --writable image with a specified size. My attempts have been unsuccessful; even after having created a writable image from a docker image with

sudo singularity build --writable image.img docker://me/my-image

doing

sudo singularity image.expand image.img
gives

ERROR: This is not a writable Singularity image format
ABORT: Aborting with RETVAL=255

What am I doing wrong? Best.

[GodloveD]

@bermanmaxim you should be able to build a sandbox without root. For instance:

$ singularity build --sandbox busybox docker://busybox
WARNING: Building sandbox as non-root may result in wrong file permissions
Docker image path: index.docker.io/library/busybox:latest
Cache folder set to /home/ubuntu/.singularity/docker
Importing: base Singularity environment
Exploding layer: sha256:f70adabe43c0cccffbae8785406d490e26855b8748fc982d14bc2b20c778b929.tar.gz
Exploding layer: sha256:29ade4f96a9909940ce0251b8182993c53f22a7e006a8c7fe868f1e0032d52b5.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Singularity container built: busybox
Cleaning up...

Now, you can see that it issues some warnings. But those should not be an issue for normal use of the container.

If you want to move this directory from one place to another, you can always tar it like so:

$ tar czvf busybox.tar.gz busybox/

But since you can build it without root you can always just build it on the system where you want to run it.

As far as expanding ext3, that is really only supported for overlay file systems since 2.4. (Although I see that the documentation is still out of date on this point. Sorry about that.) Here is a thread that discusses the issue.

https://github.com/singularityware/singularity/issues/1076