Singularity: Get singularity running on mac/in docker?

[cjprybol]

My school recently installed singularity on a cluster I have access to, and I would like to try it, but I'm unable to create any images because I don't have sudo privileges.

To get around that, I'm trying to get an image going on my personal computer (mac) on which I do have sudo privileges, because it seems as though once an image is ready to be run, it can be run anywhere that has singularity installed (so I can build an image on my laptop and transfer it to the cluster to run).

When trying to install singularity on my mac, I run

mac code

git clone https://github.com/gmkurtzer/singularity.git
cd singularity
./autogen.sh
./configure --prefix=/usr/local

truncated output

ERROR!!!!!!

This host does not support the CLONE_NEWNS (mount) namespace flag! You
really really really don't want to run Singularity containers without a
Separate mount name namespace!

From that, I take it installing on mac isn't a good idea, so my next idea was to try and run it in a sandboxed linux VM on my laptop. I tried docker first. I'm using the docker for mac beta here. This is the minimum reproducible example I could come up with.

install starting in a mac bash session with docker installed

# pull image from dockerhub
docker pull debian
# run and enter into a debian shell
docker run -it debian /bin/bash
# update package database
apt-get update
# get packages needed to install singularity
apt-get -y install build-essential curl git sudo man vim autoconf libtool
# add user so we can get out of root
useradd -m user
# set password for user
echo 'user:password'|chpasswd
# give sudo privileges to user
usermod -aG sudo user
# change to user
su user
# go to home directory of user
cd ~
# pull master repo
git clone https://github.com/gmkurtzer/singularity.git
# enter, configure, install
cd singularity
./autogen.sh
./configure --prefix=/usr/local
make
echo "password" | sudo --stdin make install
bash test.sh

output from test.sh

Building test container...

 + sudo singularity create -s 15 container.img                  (retval=1) ERROR
Creating a sparse image with a maximum size of 15MiB...
INFO   : Using given image size of 15
ERROR: Failed binding image: container.img
Full output in: /tmp/tmp.01BQhVNvLh

If I ignore the message, and try and bootstrap the container (or any other container, they give the same results) using the centos.def or debain.def files, I get a segmentation fault.

So I'm asking both

1. What's the easiest way to get singularity up and running for someone without sudo access on a cluster?
2. If using a different system with sudo (like I've tried here) is the only way to get a singularity image up and running, what's the easiest way to get that to work for non-debian/centOS users?

Thanks!

[gmkurtzer]

The test script is a bit of a hack in that it depends on sudo to be passwordless, but... If you already authenticated with sudo, it _should_ work... With that said, I use the test.sh mostly for my own sanity that I don't break more then I fix thus I have not considered it much for portability.

Additionally, I should mention that I have not tested Singularity from a Docker on the Mac. But... this has raised my curiosity and will try that shortly and report back!

Now to your questions:

1. While the Singularity work flow does require root access _somewhere_, it generally is not the cluster. I know of people using Singularity on their personal Linux laptops, desktops, servers, and VM's and then uploading the finalized image to the cluster where they can run it without sudo. I would personally recommend a VM, but as I mentioned, I am quite intrigued by the use case you mentioned above... But I am concerned that it won't work. This is because Docker is a container solution in itself, and thus doesn't have full access to non-shared resources that require host level root access to manipulate (e.g. the loop device interface). I will mess with this, but if you want a quick fix, I'd grab VirtualBox and build within there.
2. I am not sure I completely follow this question, but based on the above info I surmise you are asking about non Debian/RHEL hosts to install and build Singularity images. In theory, Singularity should work on any relatively current Linux OS (going back as far as RHEL5 and similar vintages). And it should be able to run any Linux distribution that is binary compatible with the host OS (e.g. ELF x86). Singularity only currently offers helper scripts for bootstrapping (creating a new image) for Debian and RHEL like operating systems, but if you can bootstrap your own, there is just about no limitation.

I hope that helps, and I'll let you know about the Docker workflow on Mac.

Greg

[vsoch]

I can give insight from some of my colleagues that using Docker on a Mac is still a bit bleeding edge - I would recommend if you want a stable environment for using Docker and Singularity to use a VM, ubuntu 14.04 LTS I can confirm works for me. I can't even begin to wrap my head around the potential issues of running Mac --> Docker with Linux --> then more containers... here is a quick solution that I just got working:

  # Make a folder for your Vagrantfile
  mkdir singularity-vm
  cd singularity-vm
  vagrant init ubuntu/trusty64; vagrant up --provider virtualbox

The above command will make the Vagrantfile, and then download the image and then you should be able to do:

  vagrant ssh

to be on your new linux system! Then, proceed with installing Docker using these instructions (let me know if you run into snags, it worked ok for me) - here is the gist:

  sudo apt-get update
  sudo apt-get install apt-transport-https ca-certificates
  sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
  echo 'deb https://apt.dockerproject.org/repo ubuntu-trusty main' | sudo tee --append /etc/apt/sources.list.d/docker.list
  sudo apt-get update
  sudo apt-get purge lxc-docker
  apt-cache policy docker-engine
  sudo apt-get install --yes linux-image-extra-$(uname -r)
  sudo apt-get install --yes docker-engine
  sudo service docker start
  sudo docker run hello-world

  Then this little bit adds your user to docker group
  sudo usermod -aG docker vagrant

Then add singularity:

  git clone https://github.com/gmkurtzer/singularity.git
  cd singularity

  # Install dependencies
  sudo apt-get install --yes build-essential libtool automake
  ./autogen.sh
  ./configure --prefix=/usr/local
  make
  sudo make install
  echo "SINGULARITY INSTALLATION COMPLETE."
  echo "After reboot, run 'docker run hello-world' to test"  

You will need to reboot to use docker without sudo, after adding vagrant to the group:

  sudo reboot now

I also realize it would be much easier to just have an install script do all this (via the vagrantfile) so I made one for you!.

I did the above steps manually (to make) and it seemed ok, and I'm running the Vagrantfile now to see if it produces the same. Either way, if you do the above you can easily get an environment for working on these things. Hope that helps!

[vsoch]

ok just tested - everything looks good!

[cjprybol]

Thanks to you both for the explanations, @vsoch for the vagrant examples. I'd never used vagrant before, but I got that to work in just a few minutes, and it was much easier than getting a VMware virtual machine set up, which is what I already had access to.

Here's how I got it to work, starting from a mac shell with homebrew installed

brew cask install virtualbox
brew cask install vagrant
brew cask install vagrant-manager
mkdir singularity-vm
cd singularity-vm
vagrant init ubuntu/trusty64; vagrant up --provider virtualbox
vagrant ssh
sudo apt-get update
sudo apt-get -y install build-essential curl git sudo man vim autoconf libtool
git clone https://github.com/gmkurtzer/singularity.git
cd singularity
./autogen.sh
./configure --prefix=/usr/local
make
sudo make install
bash test.sh

and that successfully ran the test.sh script

[gmkurtzer]

Wow, this is fantastic stuff! I'd love to see this written up as a recipe or example for the website! (Any volunteers lol)

[vsoch]

+1 that singularity needs great docs! Where is the current site hosted?

[gmkurtzer]

GitHub pages, (gh-pages branch) of Singularity repo. It has a very basic JavaScript wrapper so all of the content are very simple HTML pages that have no theme or css within them.

[cjprybol]

I was going to make an example of how to get started for the other members of my group. When I'm done, I'd be happy to share it for inclusion in the documentation as an example

[vsoch]

ah, great! I can definitely help out by adding some docs - maybe after we finalizez docker --> singularity?

I was going to suggest to make singularity-app a clean organization, and then store the website under singularity-app.github.io (and you can use the same lbl domain). I use github pages for a lot of stuff, and I definitely like this strategy, but given that singularity is going to grow beyond just the base "singularity" repo - it might make sense to store docs separately. It also is nice just pushing to master (the standard for the username.github.*) repos for a user or organization instead of remembering to checkout gh-pages first :)

[gmkurtzer]

I just made some page stubs for content that I want help on. Please see the updated docs page and more specifically http://singularity.lbl.gov/#singularity_mac.

Creating an organization for Singularity is a great idea. I will look into that!

Thank you again!

[vsoch]

Cool! I took out my contacts so I'm useless for working more today, but will jump into some of this tomorrow. Making documentation is a past time I definitely enjoy, and I'm grateful to be able to help!

[gmkurtzer]

You like writing documentation?!! lol

[macsermkiat]

Mine failed during ./configure phase because missing libarchive.
Just to add that I have to install libarchive manually to /usr/local too.

https://github.com/libarchive/libarchive/wiki/BuildInstructions

[kaczmarj]

I made a repository with instructions to build Singularity images (from a Singularity recipe) within Docker. This is targeted towards HPC users who might not have singularity on their local machines.

https://github.com/kaczmarj/singularity-in-docker

[vsoch]

Thanks @kaczmarj ! Just for documentation, I have a repository that does the same, and provides the containers on Docker Hub. If you see any improvements that might be made to the recipes, please open an issue and we can discuss!

[drozzy]

Why can't it just run on mac? Keep getting "only remote builds are allowed on this platform"...

[jmstover]

Keep getting "only remote builds are allowed on this platform"...

The way it's working on Mac... You can't build from a definition file. But if there is a DockerHub, or similar, you can use pull instead of build

singularity pull mystuff_v3.sif docker://meatdocker/mystuff:v3.0

It's the build command that's cut out because a definition build won't work. :/