Singularity: Roadmap to unprivileged Docker Hub image import

[chrisgorgo]

As for many users my typical workflow is as follows:

1. develop and test code inside a docker container on a local machine (in my case that's Windows, but for many it's Mac)
2. convert the image using docker2singularity
3. transfer the image to the cluster I'm using

The main problem is that container images can get big and with this workflow even the smallest change requires recreating and copying the full container image. I would much prefer the following workflow:

1. develop and test code inside a docker container on a local machine (in my case that's Windows, but for many it's Mac)
2. push the image to Docker hub (provides extra backup and can be sped up because of layer caching)
3. on the cluster import the image from docker hub (taking advantage of local cache of previously downloaded layers)

Step 3 has some constraints: it should not require root privileges or special kernel features.

It seems that all of the building blocks are in place, but they have not been put together. In fact @oschulz created a handy docker2singularity.py script that creates singularity compatible images (in "directory" or "squashfs" formats) using nothing else (not even singularity itself) than docker2aci and not requiring root privileges. Since @vsoch implemented support for docker registry API and layer caching the same functionality could be replicated in singularity for the convenience of users. There are a few things that need to be cleaned up:

• [x] permissions: exclude /dev to avoid permission issues when untarring layers as recommended by @oschulz in https://github.com/singularityware/singularity/issues/364#issuecomment-269201488; make all files world readable and all executable files world executable to make the new container more robust to non-root users
• [x] environment variables: extract variables from the manifest I believe there is a PR for this already https://github.com/singularityware/singularity/pull/376/files
• [ ] mount points: by default we should create mount points available on the system the container is created/imported, but it would be nice to give an option to create additional mount points

Ideally, I would like to do something like: singularity create tensorflow.tar.gz --from-docker-hub tensorflow/tensorflow:0.12.1. This would THE killer feature for Singularity 2.3!

Update 1: added mention of the mount points

[vsoch]

One quick additions - Singularity does have docker layer caching (default is to $HOME/.singularity/docker). We are also finishing up Singularity Hub, which has a workflow of pushing a spec file to a github repo, the image is built in the cloud, and then available from the command line with the shub:// uri. I will leave others to comment on the rest, but I think we are in full support and moving in this direction, and maybe it's just not as quickly as the functionality is needed! :)

[vsoch]

and @chrisfilo you gave up on Linux for Windows? :scream:

[chrisgorgo]

Yes and no. Linux had very poor driver support for my laptop so I switched to Windows. I do use Linux every day though - just through Docker. I think Linux is still a rare choice for a laptop OS in academia. At workshops I see mostly Macs. This is why seamless Docker Hub support in Singularity would be in my opinion a really useful feature. It seems that it's almost there - only a few things left to do.

[vsoch]

ah ok. Well if you ever go back, I've used ThinkPad T4x0 and Ubuntu (from 12.02 to 16.04 now) and I haven't hit a bug yet! It was a bit annoying back in the day when there wasn't a good replacement for Microsoft Word suite, but now with Google Sheets / Docs I don't see much need for anything on Windows. Unless I want to play computer games... then I might run into trouble :)

Yes, we will work hard on these things! Thanks for your ideas and feedback for helping to make Singularity great!

[gmkurtzer]

Hi @chrisfilo!

There are a number of reasons why a tarball does not make a satisfactory container image format (e.g. in this usage, file ownership, permissions and some attributes would be wrong and it can't be executed efficiently for HPC/HTC jobs).

Is the resulting tarball critical in this roadmap, or would a Singularity image do? And if so, we have been thinking about this already, and to do it properly will require some level of privilege escalation. For that reason it will be enabled conditionally via the system administrator.

I think most of the rest of what you request is available in 2.2, but correct me if I am wrong (e.g. pulling from docker-hub, and using the import command instead of create).

Thanks, great thinking!

Greg

[chrisgorgo]

Hi Greg!
I have no preference for one format over another. It was just my reading of the 2.2 changelog that unprivileged usage was only available with directory and tarball images (although I am not able to create such with root using 2.2 on our cluster).

To answer your question tarball is not critical. Actually @oschulz suggested using squashfs instead. Did you have a look at docker2singularity.py?

[gmkurtzer]

Hi Chris,

Nope, I haven't looked at it recently. Singularity already supports Squashfs internally (as long as the file has the appropriate suffix, which I know isn't a good way to check, but it works) and Squashfs is a much better option then tarballs.... AND!!! It can work unprivileged..But... To build or even download the template container from DockerHub, we need to be privileged otherwise the temporary template files will have the wrong perms.

The 2.3 release (the goal is March-April) is planned to have some of this functionally but as I mentioned, I anticipate some system admins disabling it.

Another option if you want something with the current 2.2 release is as follows:

$ mkdir /tmp/tensorflow
$ singularity import /tmp/tensorflow docker://tensorflow/tensorflow:0.12.1
$ singularity shell /tmp/tensorflow

I've been focused on the 2.3 development code that I'm not 100% it will work unprivileged like that, but I encourage you to try. The permissions will be wrong inside the container because it is running unprivileged, but that works today (I hope... please confirm).

Thanks!

[chrisgorgo]

To build or even download the template container from DockerHub, we need to be privileged otherwise the temporary template files will have the wrong perms.

I understand by this you mean all files in the container will be owned by the user creating the container? Is this really a problem? I can't think of a case in which this would break anything. Of course, the file owners would not be the same as in the original image, but as long as everything works this is a small price to pay for avoiding adding another elevated process (which as you mentioned will make some admins cringe).

[gmkurtzer]

It generally works fine for short lived, non-distributable containers. But if you want to share the container, it is not a good format for that because of the permission issue. Also some applications will not work properly if they are expecting their configuration files to be owned by root (or even /).

However, if that is all you need.... I think you can do it now. :)

[chrisgorgo]

That would work! To improve portability of containers I would suggest making all file world readable and files/folders with 'x' world 'x' like we did in the old docker2singularity.

[oschulz]

I would suggest making all file world readable and files/folders with 'x' world 'x'

I'm doing this in docker2singularity.py, too. The (SquashFS) files are owned by root via mksquashfs ... -all-root. I hope that this will work fine for almost all applications.

[chrisgorgo]

From a review of our BIDS Apps paper (which also discussed singularity) done by @ctb

• the Docker-to-Singularity conversion approach is very inconvenient looking,
  and we feel it is a major drawback.

(worth reading in full since it focuses on Singularity - http://academickarma.org/review/l14cxkrdsz7j)

I think this emphasizes how important the ability to import Singularity images directly from Docker on an HCP (without root) is.

[vsoch]

Why is root needed?

singularity shell docker://ubuntu:latest

On Tue, Jan 24, 2017 at 10:04 AM, Chris Filo Gorgolewski <
[email protected]> wrote:

From a review of our BIDS Apps paper (which also discussed singularity)
done by @ctb https://github.com/ctb

• the Docker-to-Singularity conversion approach is very inconvenient
  looking,
  and we feel it is a major drawback.
  (worth reading in full since it focuses on Singularity -
  http://academickarma.org/review/l14cxkrdsz7j
  http://academickarma.org/review/l14cxkrdsz7j)

I think this emphasizes how important the ability to import Singularity
images directly from Docker on an HCP (without root) is.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/singularityware/singularity/issues/446#issuecomment-274885403,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAxs8hnxdN8UR1_ULD783xoQAebgXcexks5rVj1GgaJpZM4LcD-Z
.

--
Vanessa Villamia Sochat
Stanford University
[phone]

[chrisgorgo]

Indeed root does not seem to be needed to run a container

[chrisgor@sherlock-ln04 login_node ~]$ singularity run docker://ubuntu:latest
library/ubuntu:latest
Downloading layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Downloading layer: sha256:2508cbcde94b50cd53356e8730bb508ddb43c76664a35dc29e69bb8b56a0f531
Downloading layer: sha256:bc7277e579f03a13476b4d2dc6607124f7e67341dbd58f9d1cd6555bec086c04
Downloading layer: sha256:946d6c48c2a7d60cb2f4d1c4d3a8131086b412d11a9def59d0bcc0892428dde9
Downloading layer: sha256:695f074e24e392178d364af5ea2405dda7ab0035284001b49939afac5106c187
Downloading layer: sha256:8aec416115fdbd74102c9090bcfe03bfe8926876642d8846c8b917959ea9b552
WARNING: Non existant bind point (directory) in container: '/share/PI'
WARNING: Non existant bind point (directory) in container: '/scratch'
WARNING: Non existant bind point (directory) in container: '/local-scratch'
No Singularity runscript found, executing /bin/sh
Singularity.ubuntu:latest>

But it is required to import a container

[chrisgor@sherlock-ln04 login_node ~]$ singularity import /tmp/test_image docker://ubuntu:latest
ERROR  : User namespace not supported, and program not running privileged.
ABORT  : Retval = 255

A couple more things I noticed (this is singularity 2.2 on Sherlock@Stanford):

1. Docker layers do not seem to be cached. If I try to run the same docker URL second time in a row all layers are downloaded again.
2. When running a docker URL the system mount points are not created in the image making it impossible to access most of the important data.

WARNING: Non existant bind point (directory) in container: '/share/PI'
WARNING: Non existant bind point (directory) in container: '/scratch'
WARNING: Non existant bind point (directory) in container: '/local-scratch'

Without layer caching or ability to create mount points running Docker URLs is unfortunately of limited use, but those limitations could be relatively easily overcome (judging by the work by @oschulz).

[vsoch]

Thanks @chrisfilo! I don't have all the answers, but these details might be helpful.

The version on Sherlock is an older version of 2.2, and likely the singularity cache directory PR is not included with that version. The admins (I believe) are mostly used to installing "official" versions of software, and since there haven't been any since 2.2, it hasn't been updated.

For the import, @gmkurtzer is working on an updated (internal) API that I believe can handle import without sudo.

The warnings I noticed as well, and this was either a bug related to mounting them automatically, and as a workaround you can create these points in the container.

So to answer your question, caching is already implemented (try the commands on your local machine) and most of this is an issue with not having Singularity on Sherlock updated with master. Given the caching of docker layers to use for images, this also addresses the desire for "layers" although I don't see Singularity images themselves being served in layers akin to Docker.

[chrisgorgo]

Thank you for clarifying. I'll wait for @gmkurtzer to comment on the importing images without root.

As for the mount points - is this really a bug? Should we open an issue for it?

If it's not a bug wouldn't it be more convenient for users if the import/run/shell commands created all of the necessary mount points by default?

[vsoch]

That's a good question - I'm not sure about Sherlock, because the software is crusty :) I just ran a bunch of jobs with a container that has directories created for all points minus /share/PI and I did see this warning:

WARNING: Non existant bind point (directory) in container: '/share/PI'

but it didn't influence the output, and I also bound the directories that I wanted, something like:

  -B $SCRATCH/data:/scratch/data

The attempt to bind (and fail when there is no path in the container) is likely because we have it configured to automatically try mounting those points:

# the container. The file or directory must exist within the container on
# which to attach to. you can specify a different source and destination
# path (respectively) with a colon; otherwise source and dest are the same.
#bind path = /etc/singularity/default-nsswitch.conf:/etc/nsswitch.conf
#bind path = /opt
#bind path = /scratch
bind path = /etc/hosts
bind path = /dev
bind path = /tmp
bind path = /var/tmp
bind path = /home
bind path = /share/PI
bind path = /scratch
bind path = /local-scratch

but then automatically creating them in the container is disabled on sherlock:

# ENABLE OVERLAY: [BOOL]
# DEFAULT: yes
# Enabling this option will make it possible to specify bind paths to locations
# that do not currently exist within the container. Some limitations still exist
# when running in completely non-privileged mode. (note: this option is only
# supported on hosts that support overlay file systems).
# note: currently disabled because RHEL7 kernel crashes with it... :(
enable overlay = no

which sort of seems like a configuration that should spit out a warning, or minimally doesn't make a lot of sense. I can attest that running Singularity (fresh install from master) on several cloud servers didn't produce any of these error messages, so likely the defaults better deal with the issue. Anyway, those again (and other admin variables) might not be updated in the current install. @gmkurtzer will ping in when he emerges from the linux bat cave. :bat: :robot:

[chrisgorgo]

This feature requires overlay file system which might not be supported by all HPCs. I think creating the directories during import would be more robust and compatible with more systems.

[gmkurtzer]

Version 2.3 (under heavy development right now) supports non-root imports to Singularity image files from Docker (and potentially other sources). While I've done a lot to secure this process, there are some security related caveats outside my reach (e.g. tar may become an attack vector) so expect this feature to be enabled (or disabled) by a site.

[chrisgorgo]

Thanks for the update - looking forward to this.

One thing I don't quite understand is the extent of those security caveats.
It seems that they would apply to all directory and tarball (potentially
even squashfs) based images independent of their origin. Is that correct?

On Tue, Jan 24, 2017 at 12:07 PM, Gregory M. Kurtzer <
[email protected]> wrote:

Version 2.3 (under heavy development right now) supports non-root imports
to Singularity image files from Docker (and potentially other sources).
While I've done a lot to secure this process, there are some security
related caveats outside my reach (e.g. tar may become an attack vector) so
expect this feature to be enabled (or disabled) by a site.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/singularityware/singularity/issues/446#issuecomment-274923258,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkpzH87arwfLv94njwFJ-YNzXN5ykVks5rVloQgaJpZM4LcD-Z
.

[gmkurtzer]

The security implication I am referring to is translating the Docker layers (tarballs) to Singularity containers. Because Singularity container images embed a POSIX file system, to properly explode a tarball within it, the process must be running as root. I have mitigated any obvious security concerns by limiting the tar command drastically such that it can only take a pipe stream with no way to manipulate the arguments.

But, as @vsoch mentioned, if you just want to run a Docker image as yourself, you can do it now, without an import. You don't get the advantages of using a Singularity image, so the Docker layers (tarballs) will need to be cached on every node/process running the containers and because it is a non-persistent run, the files created from the untar'ing are all owned by you (rather then root which would be required for a distributable, relocatable image).

Hope that helps!

[chrisgorgo]

On Tue, Jan 24, 2017 at 5:56 PM, Gregory M. Kurtzer <
[email protected]> wrote:

Because Singularity container images embed a POSIX file system, to
properly explode a tarball within it, the process must be running as root.

But is this only true for .img images? The docker2aci based
docker2singularity.py script (
https://github.com/oschulz/singularity-utils/blob/master/bin/docker2singularity.py)
seems to be able to create "directory" and "squashfs" style singularity
images without the need for root access.

[oschulz]

The docker2aci based docker2singularity.py script [...] seems to be abel to create "directory" and "squashfs" style singularity images without the need for root access.

Yes, my docker2singularity.py can do that. It "cheats" a bit, though: When the original container gets unpacked to a directory, all files belong to the current user. Then I use the mksquashfs option -all-root to make all files in the image root-owned. I also make all files world-readable to prevent permission issues. This has worked well for me so far, but file ownerships and permissons are not quite the same as in the original container - if that's needed, the original container must be unpacked with root privileges before an image (not matter which format) is created. I haven't run into a situation where that was necessary yet, though.

A big advantage (besides it's compactness) of SquashFS images is indeed that they can be created without root privileges. This means that images can also be unpacked, modified, and re-packed be ordinary users at any time (proot is a very useful tool to modify singularity containers without root privileges).

[gmkurtzer]

While mksquashfs can be run as non-root, doesn't it copy the permissions of the host environment, thus you need to be root to explode the tarballs to disk (or can it operate directly on a tarball)? Even if it has the ability to take an option to override the ownership of files, they wouldn't necessarily be correct if we assume that root should own everything either.

The only way I see of guaranteeing you have the appropriate ownership and permissions of every file within a container is if they are created as root.

[oschulz]

While mksquashfs can be run as non-root, doesn't it copy the permissions of the host environment

By default it does, of course. I use the -all-root option, which makes all files in the image root-owned. I also to a chmod before the mksqashfs and make all files world-readable.

So far I haven't run into any issue with this approach - the only problem would be an application that expects (i.e. explicitly checks for) specific files owners and/or permissions for certain files in the image. I can't think of any such case, right now - but as you say, Greg, in that situation, there's currently no way around creating the image (in whatever image format) as root.

Maybe singularity-import could have an option to work in user-mode, making all files user-owned (while discarding special files as dev-files) without warnings/errors? Naturally, user-mode import would only support images types like directory, SquashFS, tarball, etc..

Maybe bootstrap could also offer a secure non-root mode via PRoot or so?

It would be nice if mksquashfs had a way to provide owner/permissions lists for files. I guess it wouldn't be a problem in principle - e.g. have an option to specify a file containing a list of (file-name, owner, group, permission)-tuples - but someone would have to implement it and do a PR for squashfs-tools.

[gmkurtzer]

Heya @oschulz,

The --all-root option would fix a lot of the files and make them more correct, but pedantically speaking, it won't be exact. There will still be files with missing permission bits that you as a user can't replicate and not all files should be owned by root (e.g. some maybe owned by nobody). But as you elude, for the most part it will be "good enough" for most people not to run into any issues (putting aside my own personal issues with OCD not liking anything that isn't perfect LOL).

In the development branch, import is working as non-root:

$ singularity import /tmp/gmk-test.img < /tmp/test.tar
$ singularity import /tmp/gmk-test.img docker://busybox

That will also retain all of the correct permissions within the tar streams (or Docker layers) which I believe at this point satisfies the original target of this issue. @chrisfilo, can you confirm?

As far as bootstrapping as non-root, I don't see how PRoot could get us around it considering in the end, files will still be owned by the calling user.

Thoughts?

[oschulz]

As far as bootstrapping as non-root, I don't see how PRoot could get us around it considering in the end, files will still be owned by the calling user.

Sure - it's only usable in the case where precise file permissions are not that important. I just mentioned it because it would be very secure, as there's absolutely no road to privilege escalation with PRoot. :-)

[gmkurtzer]

Yeah, I see your point, but I'm just not sure it will get us the ability to bootstrap as non-root generally for Singularity images. Unless..... If it were possible to use PRoot to create a tarball, that we can then import?

[oschulz]

Not generally, no. Well, I guess you guys can do the same thing that PRoot does with namespaces, right?

[gmkurtzer]

Technically, the user namespace might be able to do some of it, but it wouldn't be the same because neither can circumvent actual security of file system ACLs, thus the files that get written out to disk will be technically owned by the calling user.

Also, the user namespace isn't exactly a commonly viable solution as it seems no distributions have uniform implementations of it. :(

We may want to close this, and come back to it when/if the user namespace gets legs.

Thanks!

[chrisgorgo]

@gmkurtzer awesome to see that this:

$ singularity import /tmp/gmk-test.img docker://busybox

is working without root!

I have not had a chance to test this personally, but maybe you will be able to answer some questions:

1. In this scenario /tmp/gmk-test.img doesn't exist and will be created? If not can I create /tmp/gmk-test.img without root?
2. Does this require any special kernel features? Like overlay fs or user namespaces?
3. Does this import translate Docker ENV statements?
4. Is there a way to instruct singularity to automatically create mount points for automounts of the system it is imported in? Most Docker images will not have /scratch /share and many systems explicitly limit what you can mount and dod not have overlay fs. Such feature would be crucial.
5. I agree with @oschulz that making all files world readable (and folder world browsable) is a very convenient way to sanitize a Docker container upon import. Even though not perfect it would make a lot of users life much easier. Especially, for importing containers that were not intended to be run with singularity.

[gmkurtzer]

Heya @chrisfilo

To answer your questions in order:

1. It does not automatically create the image, but yes, you can create it also without needing root.
2. Nope, it does not require any special kernel features.
3. No. To get those you will have to do a full bootstrap. This will just import the raw container.
4. That is the point of OverlayFS which (as of 2.3) will be enabled by default
5. I don't think I follow. But if I do, the answer would have to be no on import. This is because, we can get away with not being root, because we are not manipulating the contents of the container. We simply explode a tarball (and in the case of docker, we explode a series of tarballs) into a container. We don't add, change or modify anything with one exception; there is a static environment layer tarball that Singularity adds to all images.

[chrisgorgo]

Thanks for the quick reply! Some followup questions:

1. Do I need root or special kernel features to do a full bootstrap? If so that would mean I cannot import containers from docker hub respecting ENV and ENTRYPOINT statements.
2. Do you know how many systems support OverlayFS? Is this going to be a problem in terms of adoption?
3. If a need to create a .img file before importing a container from Docker Hub into it wouldn't this mean I need to know how big the imported container will be? Isn't this a problem?

Unless I don't understand something it seems that the urge to preserve ownership and permissions of the files in the imported container comes with the cost of not being able to modify it (hence not mount points, ENV or ENTRYPOINT). If you extract the tarballs from Docker Hub as the user who performs the import (instead of root), fix permissions (give all files world readability), add mount points (no need for overlayfs), set up environment and runscripts it would result in much more usable images. Even though they would not preserve ownership and permissions perfectly.

I hope this feedback helps - I'm trying to give you some user perspective on this feature. If done well it will be a true game changer!

[gmkurtzer]

Lets see... :)

1. Yes, you are correct. Import will not honor all of the dockerfile bits at this time. You will still need root still to do a full bootstrap.
2. Most reasonably modern systems support OverlayFS (e.g. RHEL7 supports it while 6 does not).
3. Yes you do need to know approximately. Yes, it could be a problem if you select an image size that is too small. Agreed. ;) ...

Your feedback is very valuable, but I have to balance several things. We can use the resulting output of the tarballs extracted by a non-privileged user as a reproducible image. You can surely use it non-persistently (e.g. temporary usage as you can do with singularity shell docker://ubuntu), but not in an image. The point of the image is to reproduce perfectly the upstream distribution, so everything within it behaves exactly as the upstream vendor expects it to. As soon as we start changing permissions, things break in un-anticipated ways.

As an example, I helped someone recently debug why ssh stopped working for a single user. Turns out a necessary directory/file was world readable. Another thing to think about... If I create a container image with my UID being 32092, then everyone who gets or uses this container will see files owned by someone that is not them. Nor do I have the appropriate permissions to even set all of the correct permissions on things like temp directories. While we maybe able to find some workarounds for some of these, I fear that any permission changing modifications to any of the base operating system will haunt us all and eat our brains.

So as valuable as your comments are, I can't satisfy them with a kludgey hack. With that said, I am happy to consider how to solve them in an appropriate manner. If I (myself) comes up with something, or if anybody has any ideas, I am open to them! They are fantastic ideas that deserve a well considered solution! :)

[chrisgorgo]

Ah the perfectionist in you! I have no right to complaint since you are doing all the hard work.

Just to clarify one thing. When I do:

singularity shell docker://ubuntu

ENV and ENTRYPOINT are converted to their singularity counterparts, but when I do

singularity import /path/some.img docker://ubuntu

ENV and ENTRYPOINT are ignored?

[gmkurtzer]

Haha @chrisfilo... I'm a bit OCD, I know this. I'm working on this!

Yes, you have that exactly right! I don't see a way (at present) of including the ENV and ENTRYPOINTs yet... But.. hrmm... Actually... Maybe you do (and/or @vsoch)....

Is there a python interface to create ad-hoc a virtual tar stream? An interface where you can build a tarball with custom information? I've done this (by hand) in Perl before for CPIO, and it was a nightmare, and TAR is much more complicated a format then CPIO. But... maybe there is a library or method to do that? If so, and if we can utilize that, we build an ad-hoc tarball on demand that has all of the appropriate permissions that we need! Thoughts?

[chrisgorgo]

Yes - you can use the built in tarfile library to create tar files and cotrol ownership and permissions of all the files https://docs.python.org/3/library/tarfile.html
You can definitely do it in memory or into a temporary file - not sure about streaming (keeping the whole tar in memory seems like a bad idea).

[gmkurtzer]

A cursory look seems like that would be perfect! We can write the tarball to disk even as long as the permissions inside reference files being owned by "root" (uid=0)!

If we can do that, we can also include the appropriate ENV and ENTRYPOINT into a container imported. :)

[vsoch]

@gmkurtzer let's review the steps for this - to clarify we want to do a non sudo required import that includes environment, etc. Right now we can't do that because the higher level bootstrap functions needed for the image (including the metadata folder .singularity) require root. So - is the idea for the python to write a tarball that includes the metadata folder and docker environment file, and return that via one of the layers in the content files, and then dump into the image? Given that the docker environment file is separate, this seems reasonable, but it breaks given that there is one already existing. Is there any possible risk you see with regard to doing this?

[oschulz]

I think the idea of using a Python-generated tarball is very elegant - kind of an on-the-fly added Docker image layer.

[vsoch]

just a quick update on some progress:

files
[{'content': 'export PATH="hello"',
  'name': './.singularity/env/10-docker.sh',
  'permission': '0755'}]

create_tar(files,'/tmp/bits.tar.gz')
'/tmp/bits.tar.gz'

and as desired, the permissions are set by default to be sudo, I can't cat without it:

cd /tmp && tar -xzvf bits.tar.gz
...
sudo cat .singularity/env/10-docker.sh 
[sudo] password for vanessa: 
export PATH="hello"

[vsoch]

are we sure this isn't a security hazard? Having a function that takes variables for files and contents, run to write sudo tarballs without needing sudo - seems not sound to do. If there is a way the user could insert a command to extract a (root owned) file into an image on some cluster... there is nothing that could go wrong with this approach?

[gmkurtzer]

Can we set the owner and group of the files within the tar stream to root?

This is not a security risk, as this can only make changes to images in which the user already owns or has writable access to.

[vsoch]

yes the owner, group and all that stuff is set to root, aka:

info.uid = info.gid = 0
info.uname = info.gname = "root"

and I confirmed this in testing because when I tried manually opening (as vanessa user) I got permission denied.

Also note that entire function can be cleaned up a bit ! I want to get a base for us to talk about / work with.

[gmkurtzer]

We actually might be able to use this instead of the static tarball I have. Can we do links in there too?

Sheesh, more and more reliance into Python! I may actually have to learn some Python at some point!

[vsoch]

What a crazy idea! Off the bat I would have said no (because it's a pointer in memory right?) but it looks like tar does have a way to store them... https://www.gnu.org/software/tar/manual/html_node/dereference.html

I'll keep reading the internet and see if there is a way to do this in python. What a cool little code journey this is turning into!

[chrisgorgo]

FYI I just heard from a user struggling because of read access denied issues on a container imported from Docker. I know that "fixing" (making everything world readable) the permissions technically changes the container, but it will make people lives easier! Just saying!

[gmkurtzer]

Is this now related to #571?

[chrisgorgo]

No. This was a different issue and the problem was with read not write
permissions.

On Apr 8, 2017 7:47 AM, "Gregory M. Kurtzer" [e-mail]
wrote:

Is this now related to #571
https://github.com/singularityware/singularity/issues/571?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/singularityware/singularity/issues/446#issuecomment-292722640,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAOkp9rFBcHx_-q1sHrXglj_wx2xLo7Wks5rt54cgaJpZM4LcD-Z
.

[gmkurtzer]

My question about relatedness is about the solution (modifying permissions within a tarstream) rather then the symptom.

[chrisgorgo]

Sorry - I didn't understand at first. Yes modifying permissions withing the tarstream should solve this issue.

[gmkurtzer]

Modifying the tars that @vsoch is doing is proving to be quite useful!

Granted, I'm internally battling with my OCD's on changing the contents of containers, but considering we are working with a model that is flawed to start with (I really don't like the Docker implementation of layers), this is an elegant workaround. :)

[vsoch]

So what do you want me to do with them layers? lol. Right now I do:

    # Check permissions for folders, not symlinks
    for member in tar:  
        if member.isdir() or member.isfile() and not member.issym():
            if not has_permission(member,permission):
                bot.verbose3("Fixing permission for %s" %(member.name))
                member.mode = permission | member.mode
            extracted = tar.extractfile(member)        
            fixed_tar.addfile(member, extracted)
        else:    
            fixed_tar.addfile(member)

you know @gmkurtzer what a great idea that we could allow filters and edits to the list of the tar! I just completely thought of this now, and not related to any conversation we've had previously :wink: Hey what do you think about this @satra ?

[chrisgorgo]

Great! a+r for files and a+rw for folders is what we have been doing in docker2singularity.

[vsoch]

can you translate that to mode numbers so I don't get it wrong? Eg:

In [1]: import stat
In [2]: stat.S_IWUSR
Out[2]: 128

So many :heart: lol

[chrisgorgo]

0444 for files and 0555 for folders. This worked for .img images because the image itself had a set of permissions so even if everything was world readable inside the image you could still make the image file inaccessible to other users.

I am not sure how this works with docker import. If at the end of the day you are producing a single .img file (or single squashfs file) this should not be a problem. However, if you are storing the imported image as a folder you could accidentally give read access to files that were suppose to be secret. This seems to be an edge case though because you in vast majority of the cases you will pull public images from docker hub. One could, however, imagine a case where you import an image from private secret docker registry and save it as directory of files with world readable permissions. This could be a problem. For such cases we would have to limit the permissions to the user performing the import (which unfortunately also makes the resulting container not usable by other users, so not really portable). We can also ignore this edge case for now (I don't really care about it since all of my code is public).

[oschulz]

I'd suggest to use 0666 for files and 0755 for folder - it'll work just as well (yes, the container is read-only, but that doesn't have to be reflected in the permissions, it's a file-system property) and it's closer to the original permissions.

[oschulz]

Actually, I think it's probably best to do the equivalent of a

chmod -R ugo+rX /

instead of setting fixed files permissions. It's least intrusive, but will still make everything accessible.