Singularity: Key search ignoring proxy?

[rhattersley]

Version of Singularity:

$ singularity version
3.7.1-1.el7

Expected behavior

Key search should respect proxy environment variables, as for other commands, e.g. "pull" or "remote status".

$ set | grep -i proxy
http_proxy=http://proxy:9999
https_proxy=http://proxy:9999
no_proxy='localhost,127.0.0.0/8,::1'
$ singularity remote status
INFO:    Checking status of default remote.
SERVICE    STATUS  VERSION             URI
Builder    OK      v1.3.8-0-g96579fc   https://build.sylabs.io
Consent    OK      v1.4.6-0-g39637f8   https://auth.sylabs.io/consent
Keyserver  OK      v1.17.5-0-g4922045  https://keys.sylabs.io
Library    OK      v1.2.10-0-g00e24c0  https://library.sylabs.io
Token      OK      v1.4.6-0-g39637f8   https://auth.sylabs.io/token

No authentication token set (logged out).
$ singularity key search E5F780B2C22F59DF748524B435C3844412EE233B
Showing 1 results

KEY ID    BITS  NAME/EMAIL
12EE233B  4096  David Trudgian (demo) <[email protected]>
$ curl "https://keys.sylabs.io/pks/lookup?fingerprint=on&op=index&options=mr&search=0xE5F780B2C22F59DF748524B435C3844412EE233B&x-pagesize=256"
info:1:1
pub:E5F780B2C22F59DF748524B435C3844412EE233B:1:4096:1573833294::
uid:David Trudgian (demo) <[email protected]>:1573833294::

Actual behavior

$ set | grep -i proxy
http_proxy=http://proxy:9999
https_proxy=http://proxy:9999
no_proxy='localhost,127.0.0.0/8,::1'
$ singularity remote status
INFO:    Checking status of default remote.
SERVICE    STATUS  VERSION             URI
Builder    OK      v1.3.8-0-g96579fc   https://build.sylabs.io
Consent    OK      v1.4.6-0-g39637f8   https://auth.sylabs.io/consent
Keyserver  OK      v1.17.5-0-g4922045  https://keys.sylabs.io
Library    OK      v1.2.10-0-g00e24c0  https://library.sylabs.io
Token      OK      v1.4.6-0-g39637f8   https://auth.sylabs.io/token

No authentication token set (logged out).
$ singularity key search E5F780B2C22F59DF748524B435C3844412EE233B
ERROR:   search failed: failed to get key: Get "https://keys.sylabs.io/pks/lookup?fingerprint=on&op=index&options=mr&search=0xE5F780B2C22F59DF748524B435C3844412EE233B&x-pagesize=256": Get "https://keys.sylabs.io/pks/lookup?fingerprint=on&op=index&options=mr&search=0xE5F780B2C22F59DF748524B435C3844412EE233B&x-pagesize=256": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
$ curl "https://keys.sylabs.io/pks/lookup?fingerprint=on&op=index&options=mr&search=0xE5F780B2C22F59DF748524B435C3844412EE233B&x-pagesize=256"
info:1:1
pub:E5F780B2C22F59DF748524B435C3844412EE233B:1:4096:1573833294::
uid:David Trudgian (demo) <[email protected]>:1573833294::

Steps to reproduce this behavior

If you're in an environment that requires proxy env vars then the sequence shown above should be sufficient. If you're in an environment that doesn't use a proxy then setting the proxy to an invalid hostname will cause the "remote status", "pull" and curl to break but the key search will continue working, as shown below:

$ set | grep -i proxy
$ singularity remote status
INFO:    Checking status of default remote.
SERVICE    STATUS  VERSION             URI
Builder    OK      v1.3.8-0-g96579fc   https://build.sylabs.io
Consent    OK      v1.4.6-0-g39637f8   https://auth.sylabs.io/consent
Keyserver  OK      v1.17.5-0-g4922045  https://keys.sylabs.io
Library    OK      v1.2.10-0-g00e24c0  https://library.sylabs.io
Token      OK      v1.4.6-0-g39637f8   https://auth.sylabs.io/token
$ singularity pull library://sylabsed/examples/lolcow
INFO:    Downloading library image
79.9MiB / 79.9MiB [==============================================================================] 100 % 18.0 MiB/s 0s
WARNING: integrity: signature not found for object group 1
WARNING: Skipping container verification
$ singularity key search E5F780B2C22F59DF748524B435C3844412EE233B
Showing 1 results

KEY ID    BITS  NAME/EMAIL
12EE233B  4096  David Trudgian (demo) <[email protected]>
$ curl "https://keys.sylabs.io/pks/lookup?fingerprint=on&op=index&options=mr&search=0xE5F780B2C22F59DF748524B435C3844412EE233B&x-pagesize=256"
info:1:1
pub:E5F780B2C22F59DF748524B435C3844412EE233B:1:4096:1573833294::
uid:David Trudgian (demo) <[email protected]>:1573833294::

$ export https_proxy=https://foo/
$ singularity remote status
INFO:    Checking status of default remote.
SERVICE    STATUS  VERSION  URI
Builder    N/A              https://build.sylabs.io
Consent    N/A              https://auth.sylabs.io/consent
Keyserver  N/A              https://keys.sylabs.io
Library    N/A              https://library.sylabs.io
Token      N/A              https://auth.sylabs.io/token

No authentication token set (logged out).
$ singularity pull library://sylabsed/examples/lolcow
FATAL:   While pulling library image: error fetching image: error making request to server:
        Get "https://library.sylabs.io/v1/images/sylabsed/examples/lolcow:latest?arch=amd64": proxyconnect tcp: dial tcp: lookup foo on 172.31.0.2:53: no such host
$ singularity key search E5F780B2C22F59DF748524B435C3844412EE233B
Showing 1 results

KEY ID    BITS  NAME/EMAIL
12EE233B  4096  David Trudgian (demo) <[email protected]>
$ curl "https://keys.sylabs.io/pks/lookup?fingerprint=on&op=index&options=mr&search=0xE5F780B2C22F59DF748524B435C3844412EE233B&x-pagesize=256"
curl: (5) Could not resolve proxy: foo

What OS/distro are you running

$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

How did you install Singularity

$ sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
$ sudo yum install singularity

[dtrudg]

This behavior definitely needs to be addressed. Thanks for reporting the issue.

[rhattersley]

FYI, the same behaviour exists for the verify command. For example, if a proxy is required (and defined in the env vars) then the command fails:

$ singularity verify signed-image.sif
Verifying image: signed-image.sif
WARNING: failed to get key material: Get "https://keys.sylabs.io/pks/lookup?exact=on&op=get&search=0x<key-id>": Get "https://keys.sylabs.io/pks/lookup?exact=on&op=get&search=0x<key-id>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Error encountered during signature verification: signature object 4 not valid: openpgp: signature made by unknown entity
FATAL:   Failed to verify container: integrity: signature object 4 not valid: openpgp: signature made by unknown entity

[rhattersley]

As a partial workaround for an installation that requires use of a proxy, if you have access to another installation with working key functionality you can manually transfer public key(s) required for verification to the target installation's personal/global key ring. The verification process will check these before trying to use the network.

For example:

working-machine $ singularity key list
Public key listing (/home/user/.singularity/sypgp/pgp-public):

working-machine $ singularity key pull E5F780B2C22F59DF748524B435C3844412EE233B
1 key(s) added to keyring of trust /home/user/.singularity/sypgp/pgp-public

broken-machine $ singularity key list
Public key listing (/home/user/.singularity/sypgp/pgp-public):

broken-machine $ scp working-machine:./singularity/sypgp/pgp-public ./singularity/sypgp/pgp-public

[rhattersley]

Thank you @dtrudg 👍