Singularity: ERROR Improperly formed manifest, layers or fsLayers must be present

Created on 13 Sep 2017 · 10Comments · Source: hpcng/singularity

Version of Singularity:

2.3.1-dist

Expected behavior

Should open a ubuntu shell in a singularity container

Actual behavior

After a fresh install I'm just doing a simple test:

$singularity -vvv shell docker://ubuntu:latest
Increasing verbosity level (4)
Singularity version: 2.3.1-dist
Exec'ing: /usr/lib/x86_64-linux-gnu/singularity/cli/shell.exec
Evaluating args: 'docker://ubuntu:latest'
VERBOSE2 SINGULARITY_COMMAND_ASIS found as False
VERBOSE2 SINGULARITY_ROOTFS found as /tmp/.singularity-runtime.StCchUoz/ubuntu:latest
VERBOSE2 SINGULARITY_METADATA_FOLDER found as /tmp/.singularity-runtime.StCchUoz/ubuntu:latest/.singularity.d
VERBOSE2 SINGULARITY_FIX_PERMS found as False
VERBOSE2 SINGULARITY_COLORIZE not defined (None)
VERBOSE2 SINGULARITY_DISABLE_CACHE found as False
VERBOSE2 SINGULARITY_CACHEDIR found as /home/local/VANDERBILT/blaberj/.singularity
VERBOSE2 SINGULARITY_ENVIRONMENT found as /tmp/.singularity-runtime.StCchUoz/ubuntu:latest/.singularity.d/environment
VERBOSE2 SINGULARITY_RUNSCRIPT found as /tmp/.singularity-runtime.StCchUoz/ubuntu:latest/singularity
VERBOSE2 SINGULARITY_TESTFILE found as /tmp/.singularity-runtime.StCchUoz/ubuntu:latest/.singularity.d/test
VERBOSE2 SINGULARITY_DEFFILE found as /tmp/.singularity-runtime.StCchUoz/ubuntu:latest/.singularity.d/Singularity
VERBOSE2 SINGULARITY_ENVBASE found as /tmp/.singularity-runtime.StCchUoz/ubuntu:latest/.singularity.d/env
VERBOSE2 SINGULARITY_LABELFILE found as /tmp/.singularity-runtime.StCchUoz/ubuntu:latest/.singularity.d/labels.json
VERBOSE2 SINGULARITY_INCLUDECMD found as False
VERBOSE2 SINGULARITY_PULLFOLDER found as /home-nfs/masi-shared-home/home/local/VANDERBILT/blaberj/singularity
VERBOSE2 SHUB_NAMEBYHASH not defined (None)
VERBOSE2 SHUB_NAMEBYCOMMIT not defined (None)
VERBOSE2 SHUB_CONTAINERNAME not defined (None)
VERBOSE2 SINGULARITY_CONTENTS found as /tmp/.singularity-layers.1mHN1GG9
VERBOSE2 SINGULARITY_PYTHREADS found as 9
VERBOSE2 SINGULARITY_CONTAINER found as docker://ubuntu:latest
VERBOSE2 SINGULARITY_DOCKER_USERNAME not defined (None)
VERBOSE2 SINGULARITY_DOCKER_PASSWORD found
VERBOSE Docker image: ubuntu:latest
VERBOSE2 Specified Docker ENTRYPOINT as %runscript.
VERBOSE Registry: index.docker.io
VERBOSE Namespace: library
VERBOSE Repo Name: ubuntu
VERBOSE Repo Tag: latest
VERBOSE Version: None
VERBOSE Obtaining tags: https://index.docker.io/v2/library/ubuntu/tags/list
VERBOSE3 Response on obtaining token is None.
Docker image path: index.docker.io/library/ubuntu:latest
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/ubuntu/manifests/latest
ERROR Improperly formed manifest, layers or fsLayers must be present

Steps to reproduce behavior

I installed with:

git clone https://github.com/singularityware/singularity.git
cd singularity
./autogen.sh
./configure --prefix=/usr/local
make
sudo make install

This also happens when I install with:

sudo apt-get install -y singularity-container

Source

justinblaber

👍1

Most helpful comment

good to hear! I'm going to close this, everyone please ping again if there is another issue.

vsoch on 15 Sep 2017

👍2

All 10 comments

Thanks! We are well aware and will have a fix in soon. Docker changed their manifest format and it's causing quite the ruckus.

https://github.com/singularityware/singularity/pull/923

vsoch on 13 Sep 2017

🎉1

Ah - seems you're all aware, but I can confirm the same issue in OS X using vagrant per the recommended install instructions.

Version of Singularity

2.3.1-dist

Expected Behavior

Should open a ubuntu shell in a singularity container

Actual behavior

vagrant@vagrant:~$ singularity --debug shell docker://ubuntu:latest
Enabling debugging
Ending argument loop
Singularity version: 2.3.1-dist
Exec'ing: /usr/local/libexec/singularity/cli/shell.exec
Evaluating args: 'docker://ubuntu:latest'
VERBOSE2 SINGULARITY_COMMAND_ASIS found as False
VERBOSE2 SINGULARITY_ROOTFS found as /tmp/.singularity-runtime.jD7wF2kP/ubuntu:latest
VERBOSE2 SINGULARITY_METADATA_FOLDER found as /tmp/.singularity-runtime.jD7wF2kP/ubuntu:latest/.singularity.d
VERBOSE2 SINGULARITY_FIX_PERMS found as False
VERBOSE2 SINGULARITY_COLORIZE not defined (None)
VERBOSE2 SINGULARITY_DISABLE_CACHE found as False
VERBOSE2 SINGULARITY_CACHEDIR found as /home/vagrant/.singularity
VERBOSE2 SINGULARITY_ENVIRONMENT found as /tmp/.singularity-runtime.jD7wF2kP/ubuntu:latest/.singularity.d/environment
VERBOSE2 SINGULARITY_RUNSCRIPT found as /tmp/.singularity-runtime.jD7wF2kP/ubuntu:latest/singularity
VERBOSE2 SINGULARITY_TESTFILE found as /tmp/.singularity-runtime.jD7wF2kP/ubuntu:latest/.singularity.d/test
VERBOSE2 SINGULARITY_DEFFILE found as /tmp/.singularity-runtime.jD7wF2kP/ubuntu:latest/.singularity.d/Singularity
VERBOSE2 SINGULARITY_ENVBASE found as /tmp/.singularity-runtime.jD7wF2kP/ubuntu:latest/.singularity.d/env
VERBOSE2 SINGULARITY_LABELFILE found as /tmp/.singularity-runtime.jD7wF2kP/ubuntu:latest/.singularity.d/labels.json
VERBOSE2 SINGULARITY_INCLUDECMD found as False
VERBOSE2 SINGULARITY_PULLFOLDER found as /home/vagrant
VERBOSE2 SHUB_NAMEBYHASH not defined (None)
VERBOSE2 SHUB_NAMEBYCOMMIT not defined (None)
VERBOSE2 SHUB_CONTAINERNAME not defined (None)
VERBOSE2 SINGULARITY_CONTENTS found as /tmp/.singularity-layers.AtJtnBNx
VERBOSE2 SINGULARITY_PYTHREADS found as 9
VERBOSE2 SINGULARITY_CONTAINER found as docker://ubuntu:latest
DEBUG Found uri docker://
DEBUG 
*** STARTING DOCKER IMPORT PYTHON  ****
DEBUG Docker layers and (env,labels,runscript) will be written to: /tmp/.singularity-layers.AtJtnBNx
VERBOSE2 SINGULARITY_DOCKER_USERNAME not defined (None)
VERBOSE2 SINGULARITY_DOCKER_PASSWORD found
DEBUG Starting Docker IMPORT, includes environment, runscript, and metadata.
VERBOSE Docker image: ubuntu:latest
VERBOSE2 Specified Docker ENTRYPOINT as %runscript.
DEBUG Headers found: Content-Type,Accept
VERBOSE Registry: index.docker.io
VERBOSE Namespace: library
VERBOSE Repo Name: ubuntu
VERBOSE Repo Tag: latest
VERBOSE Version: None
VERBOSE Obtaining tags: https://index.docker.io/v2/library/ubuntu/tags/list
DEBUG GET https://index.docker.io/v2/library/ubuntu/tags/list
DEBUG Http Error with code 401
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:library/ubuntu:pull
DEBUG Headers found: Content-Type,Authorization,Accept
VERBOSE3 Response on obtaining token is None.
Docker image path: index.docker.io/library/ubuntu:latest
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/ubuntu/manifests/latest
DEBUG GET https://index.docker.io/v2/library/ubuntu/manifests/latest
ERROR Improperly formed manifest, layers or fsLayers must be present

Steps to reproduce behavior

brew cask install virtualbox
brew cask install vagrant
brew cask install vagrant-manager
mkdir singularity_vm
cd singularity_vm
vagrant init singularityware/singularity-2.3.1
vagrant up
vagrant ssh
sudo singularity shell docker://ubuntu:latest

joshrule on 13 Sep 2017

cool thanks! Yes the bug will be present anywhere that uses the Docker Registry (Hub) Api.

vsoch on 13 Sep 2017

I'm running into the same problem. It seems to happen with the centos and the Ubuntu containers. Any idea when this will be fixed? I'm writing up a proof of concept for my management and cannot complete until this is fixed. Will it be fixed very soon like in the next few days or is this something that could take weeks?

shanrid on 13 Sep 2017

Like, within a few hours at most. use the development branch if you need it now.

vsoch on 13 Sep 2017

excellent. Thank you!

shanrid on 13 Sep 2017

See here for announcement:

https://groups.google.com/a/lbl.gov/forum/#!topic/singularity/ZpBW0oHoDRo

vsoch on 13 Sep 2017

Perfect!

git clone -b development https://github.com/singularityware/singularity.git

EDIT: To be more descriptive, the above clones the development branch, which, when installed, fixes the problem in OP

justinblaber on 13 Sep 2017

👍1

I've been tracking this one down for the last day too. Shame the API changed. What really threw me was that I was able to pull docker://centos:5 but not docker://centos:6. Both were getting a 200 response. Output of https://index.docker.io/v2/library/centos/manifests/5:

{
   "schemaVersion": 1,
   "name": "library/centos",
   "tag": "5",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "sha256:..."
      },
      {
         "blobSum": "sha256:..."
      },
      {
         "blobSum": "sha256:..."
      },
      {
         "blobSum": "sha256:..."
      }
   ],
   "history": [
      {
         "v1Compatibility": "{\"architecture\":\"amd64\",\"author\":\"The CentOS Project \\[email protected]\\u003e\",\"config\":{\"Hostname\":\"a4098d36af2d\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/bash\"],\"Image\":\"sha256:....\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":{\"build-date\":\"2016-03-31\",\"license\":\"GPLv2\",\"name\":\"CentOS Base Image\",\"vendor\":\"CentOS\"}},\"container\":\"1251477385d0a63c6051aea9f01517600960a3a1b0c11238f71b11ccc591236b\",\"container_config\":{\"Hostname\":\"a4098d36af2d\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"CMD [\\\"/bin/bash\\\"]\"],\"Image\":\"sha256:....\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":{\"build-date\":\"2016-03-31\",\"license\":\"GPLv2\",\"name\":\"CentOS Base Image\",\"vendor\":\"CentOS\"}},\"created\":\"2016-08-30T18:19:35.965870239Z\",\"docker_version\":\"1.12.1\",\"id\":\"...\",\"os\":\"linux\",\"parent\":\"...\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"...\",\"parent\":\"...\",\"created\":\"2016-08-30T18:19:33.374089465Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  LABEL name=CentOS Base Image vendor=CentOS license=GPLv2 build-date=2016-03-31\"]},\"author\":\"The CentOS Project \\[email protected]\\u003e\",\"throwaway\":true}"
      },
      {
         "v1Compatibility": "{\"id\":\"...\",\"parent\":\"...\",\"created\":\"2016-08-30T18:19:28.080275476Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:.... in / \"]},\"author\":\"The CentOS Project \\[email protected]\\u003e\"}"
      },
      {
         "v1Compatibility": "{\"id\":\"...\",\"created\":\"2016-08-30T18:19:16.841905967Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  MAINTAINER The CentOS Project \\[email protected]\\u003e\"]},\"author\":\"The CentOS Project \\[email protected]\\u003e\",\"throwaway\":true}"
      }
   ],
   "signatures": [
      {
         "header": {
            "jwk": {
               "crv": "P-256",
               "kid": "..",
               "kty": "EC",
               "x": "...",
               "y": "..."
            },
            "alg": "ES256"
         },
         "signature": "...",
         "protected": "..."
      }
   ]
}

Output of https://index.docker.io/v2/library/centos/manifests/5:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 529,
         "digest": "sha256:01e4c089df051ce67f60db1513f5f5d72d2907976f1f2c95ffad0f8e8ffb71b0",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

Anyway, I'm very happy to report that using the development branch solved the issue. 🎉