Singularity: Cannot pull image from local docker registry at paths like docker.internal/image

[ilmagico]

Version of Singularity:

Tested on 2.3 and 2.4 and latest development branch.

Expected behavior

To be able to build an image from a local docker registry where the image is at, say, docker.internal/image.
Note that there's no namespace/image, and this is actually not required, as shown in this page from docker's own docs: https://docs.docker.com/registry/deploying/#copy-an-image-from-docker-hub-to-your-registry
In the example from the docker documentation page above, an image is uploaded to a local registry simply as localhost:5000/my-ubuntu without any namespace.

Actual behavior

Singularity doesn't parse the image name correctly and tries to pull it from docker's main hub, index.docker.io:

$ cat Singularity 
Bootstrap: docker
From: docker.internal/brewst:latest
$ sudo singularity build test.simg Singularity 
Using container recipe deffile: Singularity
Sanitizing environment
Adding base Singularity environment to container
Docker image path: index.docker.io/docker.internal/brewst:latest
ERROR UNAUTHORIZED: authentication required
ERROR Check existence, naming, and permissions
Cleaning up...
$ 

Steps to reproduce behavior

See above. Basically, start a local docker registry and upload an image like in the example at https://docs.docker.com/registry/deploying/#copy-an-image-from-docker-hub-to-your-registry, then try to pull it from singularity.
Here are the full instructions, including spinning up a local registry, uploading a test image, then trying to pull it down with singularity:

$ sudo docker run -d -p 5000:5000 --restart=always --name registry registry:2
Unable to find image 'registry:2' locally
2: Pulling from library/registry
90f4dba627d6: Pull complete 
b3e11d7b4f5e: Pull complete 
1f032f3c8932: Pull complete 
425585e7aedb: Pull complete 
f45f535a83d2: Pull complete 
Digest: sha256:0f8fe61fa337b8ef02217702ba979b47a7d68717d4628f31592ebff85915f3ba
Status: Downloaded newer image for registry:2
fe5bb4406a0086fde42252ce4e52cbb7f853e48442938cd30c14974985c0e46d
$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
fe5bb4406a00        registry:2          "/entrypoint.sh /e..."   5 seconds ago       Up 4 seconds        0.0.0.0:5000->5000/tcp                           registry
$ sudo docker tag ubuntu:16.04 localhost:5000/my-ubuntu
$ sudo singularity build test.simg Singularity 
Using container recipe deffile: Singularity
Sanitizing environment
Adding base Singularity environment to container
Docker image path: index.docker.io/localhost:5000/my-ubuntu:latest
ERROR UNAUTHORIZED: authentication required
ERROR Check existence, naming, and permissions
Cleaning up...
$ singularity --version
2.4-master.g3c0553d

I had a local patch for this but I lost it when upgrading from 2.3 to 2.4 ... it basically boils down to not treating the image path specially, assuming a format like namespace/imagename, and instead allowing a generic path. I can probably craft it back onto 2.4 if needed.

[vsoch]

hey @ilmagico! I wish it were as simple as parsing a tag from a single image name and then a general name space. The issue is that we do need to distinguish the registry from the name space, both of which can have several slashes in them, and so it's a reasonable method to assume that since I name space is required and registry is not, given a pattern like you are showing we assume that the name given is the name space. I think it would be great if you did find a solution that is able to parse all the pieces correctly in all circumstances, but from what I've tried my recommendation is to simply specify the registry. It's trivial to say 'localhost:5000/namespace/container:tag@digest' and I would argue that your recipe carries the important information that the container didn't come from Docker hub, as most would expect. did you try simply specifying this local registry to obtain the image? Note that you would need to export the variable SINGULARITY_NOHTTPS=true so that http will be used.

[ilmagico]

On second thought, my local patch was more properly described as a hack and would've broken something else. Fundamentally, there's no way to tell if something/blah should be interpreted as docker://index.docker.io/something/blah or as docker://something/blah (and the reasonable default is the current behavior).
So, how does docker itself resolve this ambiguity? Based on the doc page linked in the description, it seems that it does so based on whether a port was specified in the url:

Tag the image as localhost:5000/my-ubuntu. This creates an additional tag for the existing image. When the first part of the tag is a hostname and port, Docker interprets this as the location of a registry, when pushing.

That should be easy enough to do, if a port is specified interpret the name before the port as a registry.

[ilmagico]

Hi @vsoch! Didn't see your response as it appeared while I was writing my previous comment. So, yes, you state basically the same thing as me, the parsing is ambiguous. But, as I say, not so if the port is specified.

Unfortunately, I don't control my company's internal docker registry, and the localhost:5000 example was there just to reproduce the issue, so adding a namespace is not an option for me, at least not right now.

You say I should be able to "simply specify this local registry"? how would I do that?

[vsoch]

We definitely do parse the port:

https://github.com/singularityware/singularity/blob/master/libexec/python/shell.py

if you want to give it a second shot at working on the function go for it! I would be glad to test. Thanks for your contribution on this!

[vsoch]

You just need the ip address ( or pretty url ) of the machine where it's being served, and of course the port. You can first get it working on your test registry using localhost and port 5000, and then when you know that's working test on the actual one.

[ilmagico]

Well, as I said, in my recipe file I have

Bootstrap: docker
From: docker://localhost:5000/my-ubuntu:latest

and that doesn't work. I tried several variations, including docker:// or not, including tag or not, but that doesn't work. Is there any other way to specify the registry that I missed? Or you mean, after I add the namespace part (which I can do locally, but not on the "real" one)?

[ilmagico]

Hmmm, I totally missed the Registry: my.registry option, seems like it's new to 2.4! I just updated from 2.3, so I didn't know about it. If that worked, it would solve my use case perfectly!
But that would be too good to be true:

$ cat Singularity 
Bootstrap: docker
From: my-ubuntu:latest
Registry: localhost:5000
$ SINGULARITY_NOHTTPS=true sudo -E singularity build test.simg Singularity 
Using container recipe deffile: Singularity
Sanitizing environment
Adding base Singularity environment to container
/usr/libexec/singularity/functions: line 87: [: DEBUG: integer expression expected
Traceback (most recent call last):
  File "/usr/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/lib/python3.6/http/client.py", line 1400, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python3.6/ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib/python3.6/ssl.py", line 808, in __init__
    self.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:748)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/libexec/singularity/python/import.py", line 109, in <module>
    main()
  File "/usr/libexec/singularity/python/import.py", line 86, in main
    layerfile=LAYERFILE)
  File "/usr/libexec/singularity/python/docker/main.py", line 83, in IMPORT
    client = DockerApiConnection(image=image, auth=auth)
  File "/usr/libexec/singularity/python/docker/api.py", line 89, in __init__
    self.load_image(kwargs['image'])
  File "/usr/libexec/singularity/python/docker/api.py", line 141, in load_image
    self.update_token()
  File "/usr/libexec/singularity/python/docker/api.py", line 152, in update_token
    response = self.get_tags(return_response=True)
  File "/usr/libexec/singularity/python/docker/api.py", line 271, in get_tags
    return_response=return_response)
  File "/usr/libexec/singularity/python/base.py", line 262, in get
    return_response=return_response)
  File "/usr/libexec/singularity/python/base.py", line 278, in submit_request
    response = urlopen(request)
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 1361, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:748)>
Cleaning up...

At this point I have no clue, but I'll try to debug it ...

[vsoch]

The header should read:

Bootstrap: docker
From: localhost:5000/my-ubuntu:latest

but let's try a simpler idea first. Just do pull, which doesn't require sudo:

SINGULARITY_NOHTTPS=true singularity pull docker://localhost:5000/my-ubuntu:latest

[ilmagico]

@vsoch, I spent a lot of time debugging this issue in singularity, and there are at least two problems here:

1. Singularity currently does not handle the case of an empty namespace. No matter how you slice it, passing the URL on the command line like in your last example, using a "Registry: " tag in the recipe, using a full URL, it doesn't matter, it fails to parse it and does a number of different (wrong) things.
2. SINGULARITY_HTTPS doesn't work, at least not for me, and it insists in connecting with https:// (see my last message above).

I also found a few other minor thing, like an error while trying to print some error messages, and impossibility to set SINGULARITY_MESSAGELEVEL from the command line.

The good news is, I have a patch for all of these problems, and with my patches applied, everything works fine, both the test local registry using http, and the "real" one using https.
I'm still cleaning it up and looking at the tests (some are failing), then I'd like to make a pull request if you don't mind, so you can review my changes, and make sure I didn't break anything else by mistake. Does that work?

[vsoch]

yes that works! You are correct that we don't handle an empty namespace ( I didn't think this was even possible! ) I think that the Registry keyword was discussed, but I'm not sure implemented fully. The SINGULARITY_NOHTTPS not working is strange, it might have to do with exporting and then using sudo? I've never tried that, and several users have use it successfully.

Looking forward to your PR! The best thing about bugs is that we can squash them :)

[vsoch]

For the SINGULARITY_NOHTTPS, maybe try first without sudo, and then try one of these ideas? https://stackoverflow.com/questions/8633461/how-to-keep-environment-variables-when-using-sudo (export first and then -E flag)

[ilmagico]

@vsoch For the SINGURARITY_NOHTTPS problem, the issue was that src/builddef.c doesn't reexport it by calling envar_set(), and coincidentally it exports SINGULARITY_MESSAGELEVEL twice... copy paste error?
I'm not 100% sure what that does, but that fixed it for me.

[ilmagico]

And yes, of course I've been calling sudo with the -E flag, that's the first thing I tried when it didn't work. I was like "oh I'm so dumb, I forgot to tell sudo to keep the environment, so here it goes ... nope, that was too easy..."! So, after that, I figured I'll just debug it and get to the bottom of it.

[ilmagico]

@vsoch Do tests pass in the development branch? They currently fail for me. I spun up a vagrant box with ubuntu 14.04 and cloned singularity there, compiles and installs fine, but tests/20-build.sh fails. Originally, it failed because I installed singularity in /opt, and that apparently breaks the tests, but even after changing it to the default, I get:

vagrant@vagrant-ubuntu-trusty-64:~/singularity$ bash test.sh 20-build.sh 
Building/Installing Singularity to temporary directory
Reinvoking in a clean shell

################################################################################
Build tests (script: 20-build.sh, testdir: /tmp/stest.oeFlp0)

 + sudo singularity build /tmp/stest.oeFlp0/container ../examples/busybox/Singularity            (retval=0) OK
 + singularity exec /tmp/stest.oeFlp0/container true                                           (retval=255) ERROR
ERROR  : No valid /bin/sh in container
ABORT  : Retval = 255
Full output in: /tmp/stest.oeFlp0
ERROR: Failed running test: 20-build.sh

vagrant@vagrant-ubuntu-trusty-64:~/singularity$ git status
On branch development
Your branch is up-to-date with 'github/development'.

nothing to commit, working directory clean

Many other tests fail similarly as well. I'd like to make sure my patches pass all the tests, and add/modify relevant tests, but I cannot do that if tests fail with a clean working copy.

[vsoch]

I don't think we've tested for installation in non system locations. It would make sense that build fails because we enforce squashfs-tools to be in specific locations on the path, and opt is not one of them afaik.

You can see all of our testing on travis:

https://travis-ci.org/singularityware/singularity/branches

[ilmagico]

@vsoch What I meant is, singularity was not installed in system location, squashfs-tools and everything else was. Normally for development, I wouldn't install it in a system location, I'd have the last stable release in the default system location instead.
Now, this being just a virtual machine I decided it doesn't matter, so I did install it in the default location, and the test still failed, with the error above ... any idea about what could it be?

[vsoch]

Maybe https://github.com/singularityware/singularity/issues/887 ?

I'm looping in @GodloveD since he wrote the build test - I think it might be helpful to see the full output, and running the command in question with --debug.

[ilmagico]

@vsoch Thanks, maybe not that one but related. As it turned out, even after reconfiguring with ./configure --prefix=/usr/local/ and rebuilding, there were still remnants of singularity in /opt and it somehow was picking them. Once I completely cleaned up my (virtual) environment, and recompiled again, then the tests started passing. Thanks again! I'll go back to my local branch and fix the tests there so I can finish this PR ...

[GodloveD]

Yay!

[vsoch]

+1!

[ilmagico]

@vsoch @GodloveD Of course with my initial "quick and dirty" patch, some tests were failing. After some thinking and some information digging about how docker itself handles this, I decided that if I wanted to both keep backward compatibility for shub:// and parse docker:// correctly, I had to use two different set of rules.
As it turned out, docker checks for either a '.' or a ':' in the registry name to decide it's a registry and not a namespace, while singularity treats every image with more than three "levels" separated by / to be registry/namespace/repo. So, I made it such docker:// uris follow docker rules, and all other uris follow singularity rules.
Please review my PR and let me know what you think. Also, if you want me to make changes, split it, squash it, etc. just let me know.

[dtrudg]

Hi @ilmagico - thanks for your efforts on this. Would you be able to double check this works on the current release-2.4 branch. Are we good to close this issue now?

[ilmagico]

@dctrud Just tested right now and it's working well.

Singularity version: 2.4.3-release-2.4.g8c5df73b