Buildah: podman build should support building V2.2 manifests

[nathanbrophy]

Description

podman build and buildah should support building docker V2.2 (Manifest Version 2, Schema 2) manifests. There is --format and BUILDAH_FORMAT flags and environment settings that seem to give user option to specify the format, where docker is the V2.2 manifest. Today, if I follow documented guidance to build a V2.2 manifest by setting --format docker or export BUILDAH_FORMAT=docker, and then push the built image to a registry that supports the V2.2 manifest, the manifest that gets pushed to the repository is a V2.1 manifest.

Steps to reproduce the issue:

1. Export the format environment variable export BUILDAH_FORMAT=docker
2. Optionally specify the --format docker flag
3. Build the image podman build -f <dockerfile> -t <tag> --format=docker .
4. Push the image podman push <tag>
5. Inspect the manifest skopeo inspect <tag> --raw
6. The returned raw manifest is a V2.1 manifest, not a V2.2 manifest as expected.
7. If I replace podman with docker for the container build tool, then get V2.2 as I would expect

Describe the results you received:
The manifest that gets pushed to the image repository after building the image is a V2.1 manifest, not a V2.2 manifest.

Describe the results you expected:
I expect for a V2.2 manifest to be pushed to the image repository, not a V2.1, when specifying the --format docker settings for creating a V2.2 manifest.

Output of podman version if reporting a podman build issue:

# podman version
Version:            1.9.0
RemoteAPI Version:  1
Go Version:         go1.10.1
OS/Arch:            linux/amd64

Output of cat /etc/*release:

# cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Output of uname -a:

Linux cdjohnson-ubuntu1.fyre.ibm.com 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information

# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary read-write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# AdditionalImageStores is used to pass paths to additional read-only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers (currently overlay, zfs, vfs, btrfs)
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

[nathanbrophy]

UPDATE:

I have found that if I run the following

# podman build -f dockerfile-test -t "quay.io/nathanbrophy/test-image-v2s2:docker" --format docker . 
# podman push "quay.io/nathanbrophy/test-image-v2s2:docker"  --remove-signatures

Then the manifest stored in the quay repository will be a V2s1 manifest, not V2s2.

If I run:

# docker build -f dockerfile-test -t "quay.io/nathanbrophy/test-image-v2s2:docker" . 
# docker push "quay.io/nathanbrophy/test-image-v2s2:docker"

Then the manifest stored in the quay repository will be a V2s2 manifest.

I have found that if I run the following

# podman build -f dockerfile-test -t "docker.io/nathanbrophy/test-image-v2s2:docker" --format docker . 
# podman push "docker.io/nathanbrophy/test-image-v2s2:docker"  --remove-signatures

Then the manifest stored in the docker repository will be a V2s2 manifest.

If I run:

# docker build -f dockerfile-test -t "docker.io/nathanbrophy/test-image-v2s2:docker" . 
# docker push "docker.io/nathanbrophy/test-image-v2s2:docker"

Then the manifest stored in the docker repository will be a V2s2 manifest.

This is what I see as results:

| Container Tool | Repository | V2s1 manifest stored | V2s2 manifest stored |
| --------------- | ---------- | --------------------- | ---------------------- |
| docker | dockerhub | NO | YES |
| docker | quay | NO | YES |
| podman / buildah | dockerhub | NO | YES |
| podman / buildah | quay | YES | NO |

So it seems that the V2s1 manifest is being used as default in podman when being pushed to different repositories. I am not sure if this is a podman build or podman push issue. If this is not a podman build issue, I can move the ticket to the appropriate git repository.

[nathanbrophy]

The following comes from the output of # podman push "quay.io/nathanbrophy/test-image-v2s2:docker" --remove-signatures --log-level debug

DEBU[0004] PUT https://quay.io/v2/nathanbrophy/test-image-v2s2/manifests/docker 
DEBU[0004] Writing manifest using preferred type application/vnd.docker.distribution.manifest.v2+json failed: Error writing manifest: Error uploading manifest docker to quay.io/nathanbrophy/test-image-v2s2: manifest invalid: manifest invalid 
DEBU[0004] Trying to use manifest type application/vnd.docker.distribution.manifest.v1+prettyjws… 
DEBU[0004] exporting opaque data as blob "sha256:9173e5f3c5b4f1bb20add46556363caf4ff7f521b70324c8ff94015be04b5f88" 
DEBU[0004] Uploading empty layer during conversion to schema 1 
DEBU[0004] Checking /v2/nathanbrophy/test-image-v2s2/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 
DEBU[0004] HEAD https://quay.io/v2/nathanbrophy/test-image-v2s2/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 
DEBU[0004] ... already exists                           
Writing manifest to image destination
DEBU[0004] PUT https://quay.io/v2/nathanbrophy/test-image-v2s2/manifests/docker

Failure: Writing manifest using preferred type application/vnd.docker.distribution.manifest.v2+json failed: Error writing manifest: Error uploading manifest docker to quay.io/nathanbrophy/test-image-v2s2: manifest invalid: manifest invalid

It seems that the manifest for the V2s2 manifest created is not being properly created as per the specification, as the quay registry is rejecting the manifest json.

[mrunalp]

@nalind @mtrmac @vrothberg ptal

[mtrmac]

Are you using _exactly_ the same quay.io repository for both the Podman and Docker tests? Quay used to be schema1 only until fairly recently, and it might be a per-repository setting that depends on the time of the repository creation. I’m not sure.

---

If it is not a repository property, but truly a behavior difference: If the same Docker schema2 manifest can be successfully uploaded to the Docker Hub, it seems that almost _by definition_ it is a valid Docker manifest, and Quay is incorrect in rejecting it.

---

I’m afraid I don’t know, for Quay,

• How to check / configure whether a repository accepts schema2
• How to debug the Quay server rejecting a supposedly valid manifest.

[nathanbrophy]

If it is not a repository property, but truly a behavior difference: If the same Docker schema2 manifest can be successfully uploaded to the Docker Hub, it seems that almost by definition it is a valid Docker manifest, and Quay is incorrect in rejecting it.

I am able to use docker to push to the same exact quay repository, and only failing when I use Podman. The quay repository I am pushing to supports V2s2.

I have V2s2 schema manifests in my quay repository, and those are built using docker. Quay only rejects V2s2 manifests built with podman/buildah.

In the debug text Quay rejects the manifest because of the content in the manifest, not the mediatype being incorrect

[mtrmac]

The “manifest invalid” text is a constant in the protocol, it’s not really a deeply thoughtful result of an analysis.

Can you attach the schema2 manifest (as it is accepted in Docker), please? I.e. push it and then skopeo inspect --raw docker://…. I don’t really expect we’ll be able to just _guess_ which part Quay does not like (the manifest format is rather simple), but it’s one of the last things we can do without seeing the Quay server’s end.

[nathanbrophy]

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 7399,
    "digest": "sha256:9173e5f3c5b4f1bb20add46556363caf4ff7f521b70324c8ff94015be04b5f88"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 2890315,
      "digest": "sha256:36626be740dfb39ec77d800cb5863ed119bd9f7e17a36b78b5a6db95dda50083"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 308900,
      "digest": "sha256:09fe191fd2e1d19c3ec666f383901e2ebf8e8acc9dc36b40c15e5b52f71ee74d"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 163,
      "digest": "sha256:9ce3ffe16127aa2b6513c69d5deaae9d2c1109bb0f4e5b9c9a7b3ed05300922d"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 132136234,
      "digest": "sha256:9bd99701eabfc58fba284a558eae64c4cd818258a00f70e5259f50946b037483"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 133,
      "digest": "sha256:309d565c77fc963f8b734ed30452b54d3587b41d0481d7879595aab9518933f5"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 96188431,
      "digest": "sha256:d979bd4bb3966038855a832ecfe601d72df1c7ca8c0ebf5895f54769d9066437"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 139,
      "digest": "sha256:11fca5ed88211df9e96297c72a7a9308747a4bea460dffa5e071b4a848525e23"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 10881975,
      "digest": "sha256:fa685e735410be4168217468c09fb67489e1c44b6880d1df63271bc53fb0b636"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 10560,
      "digest": "sha256:4c9d5911a136778559081afa05168c404ffad16e6e22085ee49bd048f864b5d9"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 1204172,
      "digest": "sha256:e4b170a28d6f38d02fe4f4d2c39c33d27133f7dd292e18a76bfd18b6819bdd02"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 721,
      "digest": "sha256:72964c1198555b5a777c039f3a6a62509ca5f40799fc1eda9bbe6370290fd60c"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 1122,
      "digest": "sha256:b45a1f903a2f69f447e5714fc8254b5b36a5357bd742f7635a0dc79dcc17c886"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 113362442,
      "digest": "sha256:5ced66b0b08c97de45ba4833db2da49a8793f37b22f527fc4f44276653917924"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 3541887,
      "digest": "sha256:03be84c851d500f710bc02bd2f74df2731c51d260f196ab2127c9a260533a35d"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 62981585,
      "digest": "sha256:b2fef538c4ed1e5a5152ee465489dd7000b5bfcd3d2a9940b12dcf56e29de8b8"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",
      "size": 4672,
      "digest": "sha256:4c5b10d8794fbd155b3c15291e197a70c1c2430934dfeb93a7b3be91c8ecbcbb"
    }
  ]
}

[mtrmac]

      "mediaType": "application/vnd.docker.image.rootfs.diff.tar",

Hum, that _is_ surprising, it is supposed to be ….tar.gzip. Reproduced, this happens on a _second_ upload of the same layers: https://github.com/containers/image/issues/733 . OTOH this is, to my knowledge, the first report of this actually breaking pushes.

Does this work again (once) after removing /var/lib/containers/cache/blob-info-cache-v1.boltdb (when run as root) / the equivalent in users’ home directory (when run as non-root)?

[nathanbrophy]

@mtrmac are you requesting I remove the /var/lib/containers/cache/blob-info-cache-v1.boltdb file and re-build the image?

[mtrmac]

@nathanbrophy yes, remove the file, and then try podman push again. No need to rebuild.

[mtrmac]

Note that removing the file EDIT will is expected to only help _once_, so please test the Quay schema2 behavior immediately afterwards, not the Docker repository first.

[nathanbrophy]

Right so here are steps I will take:

1. Remove /var/lib/containers/cache/blob-info-cache-v1.boltdb
2. Push the image to quay
3. Re-tag the image
4. Push again
5. Compare the manifests

[nathanbrophy]

Results From Test:

# rm -f /var/lib/containers/cache/blob-info-cache-v1.boltdb

# podman push "quay.io/nathanbrophy/test-image-v2s2:docker" --remove-signatures

# skopeo inspect "docker://quay.io/nathanbrophy/test-image-v2s2:docker"

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 7399,
    "digest": "sha256:9173e5f3c5b4f1bb20add46556363caf4ff7f521b70324c8ff94015be04b5f88"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 2890315,
      "digest": "sha256:36626be740dfb39ec77d800cb5863ed119bd9f7e17a36b78b5a6db95dda50083"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 308900,
      "digest": "sha256:09fe191fd2e1d19c3ec666f383901e2ebf8e8acc9dc36b40c15e5b52f71ee74d"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 163,
      "digest": "sha256:9ce3ffe16127aa2b6513c69d5deaae9d2c1109bb0f4e5b9c9a7b3ed05300922d"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 132136234,
      "digest": "sha256:9bd99701eabfc58fba284a558eae64c4cd818258a00f70e5259f50946b037483"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 133,
      "digest": "sha256:309d565c77fc963f8b734ed30452b54d3587b41d0481d7879595aab9518933f5"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 96188431,
      "digest": "sha256:d979bd4bb3966038855a832ecfe601d72df1c7ca8c0ebf5895f54769d9066437"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 139,
      "digest": "sha256:11fca5ed88211df9e96297c72a7a9308747a4bea460dffa5e071b4a848525e23"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 10881975,
      "digest": "sha256:fa685e735410be4168217468c09fb67489e1c44b6880d1df63271bc53fb0b636"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 10560,
      "digest": "sha256:4c9d5911a136778559081afa05168c404ffad16e6e22085ee49bd048f864b5d9"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1204172,
      "digest": "sha256:e4b170a28d6f38d02fe4f4d2c39c33d27133f7dd292e18a76bfd18b6819bdd02"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 721,
      "digest": "sha256:72964c1198555b5a777c039f3a6a62509ca5f40799fc1eda9bbe6370290fd60c"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1122,
      "digest": "sha256:b45a1f903a2f69f447e5714fc8254b5b36a5357bd742f7635a0dc79dcc17c886"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 113362442,
      "digest": "sha256:5ced66b0b08c97de45ba4833db2da49a8793f37b22f527fc4f44276653917924"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 3541887,
      "digest": "sha256:03be84c851d500f710bc02bd2f74df2731c51d260f196ab2127c9a260533a35d"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 62981585,
      "digest": "sha256:b2fef538c4ed1e5a5152ee465489dd7000b5bfcd3d2a9940b12dcf56e29de8b8"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 4672,
      "digest": "sha256:4c5b10d8794fbd155b3c15291e197a70c1c2430934dfeb93a7b3be91c8ecbcbb"
    }
  ]
}

# podman push "quay.io/nathanbrophy/test-image-v2s2:docker-2" --remove-signatures

# skopeo inspect "docker://quay.io/nathanbrophy/test-image-v2s2:docker"

{
  "name": "nathanbrophy/test-image-v2s2",
  "tag": "docker",
  "architecture": "amd64",
  "fsLayers": [
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:4c5b10d8794fbd155b3c15291e197a70c1c2430934dfeb93a7b3be91c8ecbcbb"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:b2fef538c4ed1e5a5152ee465489dd7000b5bfcd3d2a9940b12dcf56e29de8b8"
    },
    {
      "blobSum": "sha256:03be84c851d500f710bc02bd2f74df2731c51d260f196ab2127c9a260533a35d"
    },
    {
      "blobSum": "sha256:5ced66b0b08c97de45ba4833db2da49a8793f37b22f527fc4f44276653917924"
    },
    {
      "blobSum": "sha256:b45a1f903a2f69f447e5714fc8254b5b36a5357bd742f7635a0dc79dcc17c886"
    },
    {
      "blobSum": "sha256:72964c1198555b5a777c039f3a6a62509ca5f40799fc1eda9bbe6370290fd60c"
    },
    {
      "blobSum": "sha256:e4b170a28d6f38d02fe4f4d2c39c33d27133f7dd292e18a76bfd18b6819bdd02"
    },
    {
      "blobSum": "sha256:4c9d5911a136778559081afa05168c404ffad16e6e22085ee49bd048f864b5d9"
    },
    {
      "blobSum": "sha256:fa685e735410be4168217468c09fb67489e1c44b6880d1df63271bc53fb0b636"
    },
    {
      "blobSum": "sha256:11fca5ed88211df9e96297c72a7a9308747a4bea460dffa5e071b4a848525e23"
    },
    {
      "blobSum": "sha256:d979bd4bb3966038855a832ecfe601d72df1c7ca8c0ebf5895f54769d9066437"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:309d565c77fc963f8b734ed30452b54d3587b41d0481d7879595aab9518933f5"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:9bd99701eabfc58fba284a558eae64c4cd818258a00f70e5259f50946b037483"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:9ce3ffe16127aa2b6513c69d5deaae9d2c1109bb0f4e5b9c9a7b3ed05300922d"
    },
    {
      "blobSum": "sha256:09fe191fd2e1d19c3ec666f383901e2ebf8e8acc9dc36b40c15e5b52f71ee74d"
    },
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:36626be740dfb39ec77d800cb5863ed119bd9f7e17a36b78b5a6db95dda50083"
    }
  ],
  "history": [
    {
      "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"0503e4262cb4\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"50051/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"GOLANG_VERSION=1.13.10\",\"GOPATH=/go\"],\"Cmd\":[\"registry\",\"serve\",\"--database\",\"/database/index.db\"],\"Image\":\"\",\"Volumes\":{},\"WorkingDir\":\"/build\",\"Entrypoint\":[\"/bin/opm\"],\"OnBuild\":[],\"Labels\":{\"operators.operatorframework.io.index.database.v1\":\"/database/index.db\"}},\"container\":\"dbb0407a7ae6f29ad7142a93fab4a675f14a87b445f0f513a8068666a0a39149\",\"container_config\":{\"Hostname\":\"0503e4262cb4\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"ExposedPorts\":{\"50051/tcp\":{}},\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"GOLANG_VERSION=1.13.10\",\"GOPATH=/go\"],\"Cmd\":[\"registry\",\"serve\",\"--database\",\"/database/index.db\"],\"Image\":\"\",\"Volumes\":{},\"WorkingDir\":\"/build\",\"Entrypoint\":[\"/bin/opm\"],\"OnBuild\":[],\"Labels\":{\"operators.operatorframework.io.index.database.v1\":\"/database/index.db\"}},\"created\":\"2020-04-21T16:44:34.033292152Z\",\"id\":\"1fb779ec4781f2947b1ebdc97846aa397e22c6e1b75b32b9fe72cd9d27d19300\",\"os\":\"linux\",\"parent\":\"140c28cd78d911905a979fc79bde2fe49169fb5b45f703a736ba224ef3eda464\",\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"140c28cd78d911905a979fc79bde2fe49169fb5b45f703a736ba224ef3eda464\",\"parent\":\"a90080b372f2627257663de42270dea97717adde4e69beac3f3bf72b3f013810\",\"created\":\"2020-04-21T16:44:33.793278975Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ENTRYPOINT [\\\"/bin/opm\\\"]\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"a90080b372f2627257663de42270dea97717adde4e69beac3f3bf72b3f013810\",\"parent\":\"ce9404e40d952b913d1ef19f90136c82cdea7c04df92c1c6adf08bec292f868d\",\"created\":\"2020-04-21T16:44:33.513051057Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) EXPOSE 50051\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"ce9404e40d952b913d1ef19f90136c82cdea7c04df92c1c6adf08bec292f868d\",\"parent\":\"6be05e598ce0e52304d16cc75a86234f5cd944f76c895d7119c5abfb7475d9f3\",\"created\":\"2020-04-21T16:44:33.077766793Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:723d45cf36f131c5c3be8dcbc1dc8b128400e28028312b2be0bdd967ae08c9f0 in /database/index.db \"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"6be05e598ce0e52304d16cc75a86234f5cd944f76c895d7119c5abfb7475d9f3\",\"parent\":\"afea541c0df2280c06444225b8cc78f66550766476551cbf61a599704d368a9b\",\"created\":\"2020-04-21T16:13:00.453205774Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) LABEL operators.operatorframework.io.index.database.v1=/database/index.db\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"afea541c0df2280c06444225b8cc78f66550766476551cbf61a599704d368a9b\",\"parent\":\"bbcdf697209e995da6910898f2905555e686511c1e883f2e049f5c5f563eed6e\",\"created\":\"2020-04-17T16:18:39.034635774Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c cp /build/bin/opm /bin/opm \\u0026\\u0026     cp /build/bin/initializer /bin/initializer \\u0026\\u0026     cp /build/bin/appregistry-server /bin/appregistry-server \\u0026\\u0026     cp /build/bin/configmap-server /bin/configmap-server \\u0026\\u0026     cp /build/bin/registry-server /bin/registry-server\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"bbcdf697209e995da6910898f2905555e686511c1e883f2e049f5c5f563eed6e\",\"parent\":\"31f458e44d64be48ff3c9e7d89e8151298dc67b54ff3be86bbaed0c1d55395dd\",\"created\":\"2020-04-17T16:18:32.921739529Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c GRPC_HEALTH_PROBE_VERSION=v0.2.1 \\u0026\\u0026     wget -qO/bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-$(go env GOARCH) \\u0026\\u0026     chmod +x /bin/grpc_health_probe\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"31f458e44d64be48ff3c9e7d89e8151298dc67b54ff3be86bbaed0c1d55395dd\",\"parent\":\"4ad83a9fb51157eac17523bca3d76aa8c1e875ddebb0a596cf8587c5051bbe0e\",\"created\":\"2020-04-17T16:18:30.557282386Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c make static\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"4ad83a9fb51157eac17523bca3d76aa8c1e875ddebb0a596cf8587c5051bbe0e\",\"parent\":\"c540a4cc732bfa1c5aa58b32ac5445c79b7678fd9b44f028c0f133d050739728\",\"created\":\"2020-04-17T16:15:43.141181023Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY file:faf98ae5927a5911832ca9b0d500364393f27ddcaace657bd7d3e8885430fab3 in go.mod \"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"c540a4cc732bfa1c5aa58b32ac5445c79b7678fd9b44f028c0f133d050739728\",\"parent\":\"a61361687cbaa6fb73fa041f65bc11429077c869526ea107b82bfa2e8056f7de\",\"created\":\"2020-04-17T16:15:42.915571898Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY file:0bc9fbe4f1539ad587afe57a7e0f644e9551a675f0df164eea202fde23976cb0 in Makefile \"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"a61361687cbaa6fb73fa041f65bc11429077c869526ea107b82bfa2e8056f7de\",\"parent\":\"977f3021e823dd40062f54ce277e70313c5273f52f5566186e68037196561186\",\"created\":\"2020-04-17T16:15:42.348745589Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:627f70a043dfe7fc4b42b319c90bd5e787397e81899abd21c20a0a166cd0cc33 in pkg \"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"977f3021e823dd40062f54ce277e70313c5273f52f5566186e68037196561186\",\"parent\":\"8b1f30448c5bfc25e5c758c701cd66b5a9a76a8cdedd840b42088d7a666ac30e\",\"created\":\"2020-04-16T21:04:04.129361593Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:0f1e808273f5841fc08388937a47c904e53c1b51d0b1be48ccf8fecc89f0c002 in cmd \"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"8b1f30448c5bfc25e5c758c701cd66b5a9a76a8cdedd840b42088d7a666ac30e\",\"parent\":\"0d6b0488a95be7440059243b1a59a37a2e505c03fca0c76170c407372a87565e\",\"created\":\"2020-04-09T20:57:56.00947905Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) COPY dir:d8b060897f3fd8ab44965e186a9f32887292ce59cf4b9d0b2c353a2ee8663b19 in vendor \"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"0d6b0488a95be7440059243b1a59a37a2e505c03fca0c76170c407372a87565e\",\"parent\":\"0d01ed7e14c20019c53ea017aa25afaafd5c0c2edbd511cf04a4c24afb76c809\",\"created\":\"2020-04-09T20:57:52.677860358Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) WORKDIR /build\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"0d01ed7e14c20019c53ea017aa25afaafd5c0c2edbd511cf04a4c24afb76c809\",\"parent\":\"0c0012d0fb2509239810a74c515d0f1489c04840e19b3d8b66dc7525c5e0c75b\",\"created\":\"2020-04-09T20:57:52.38084362Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c apk update \\u0026\\u0026 apk add sqlite build-base git mercurial bash\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"0c0012d0fb2509239810a74c515d0f1489c04840e19b3d8b66dc7525c5e0c75b\",\"parent\":\"41b29841cc1a0dbed445ca935fbeacb4ec0f5a968dc2c047a84a22a0a5ba094c\",\"created\":\"2020-04-08T23:17:27.595210081Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) WORKDIR /go\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"41b29841cc1a0dbed445ca935fbeacb4ec0f5a968dc2c047a84a22a0a5ba094c\",\"parent\":\"bc01049f0914c929f614ff4101ce39f923ae80cb6498e0372b2f32e69b361a17\",\"created\":\"2020-04-08T23:17:27.247269378Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c mkdir -p \\\"$GOPATH/src\\\" \\\"$GOPATH/bin\\\" \\u0026\\u0026 chmod -R 777 \\\"$GOPATH\\\"\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"bc01049f0914c929f614ff4101ce39f923ae80cb6498e0372b2f32e69b361a17\",\"parent\":\"30be853189fd392d8a5a01e5c5e385ed1d764b7276ff4a1eede3b3672cb38086\",\"created\":\"2020-04-08T23:17:25.535050572Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"30be853189fd392d8a5a01e5c5e385ed1d764b7276ff4a1eede3b3672cb38086\",\"parent\":\"975071e04248b3898d95ebf19ab9e020aa71b52d24cfafcd3fe1508de03f5198\",\"created\":\"2020-04-08T23:17:25.211801839Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ENV GOPATH=/go\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"975071e04248b3898d95ebf19ab9e020aa71b52d24cfafcd3fe1508de03f5198\",\"parent\":\"56f675edff3d5d7638a7e77a49ed1efc1f5b6286792337131fd1639a29ce3aea\",\"created\":\"2020-04-08T23:17:24.377517287Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c set -eux; \\tapk add --no-cache --virtual .build-deps \\t\\tbash \\t\\tgcc \\t\\tmusl-dev \\t\\topenssl \\t\\tgo \\t; \\texport \\t\\tGOROOT_BOOTSTRAP=\\\"$(go env GOROOT)\\\" \\t\\tGOOS=\\\"$(go env GOOS)\\\" \\t\\tGOARCH=\\\"$(go env GOARCH)\\\" \\t\\tGOHOSTOS=\\\"$(go env GOHOSTOS)\\\" \\t\\tGOHOSTARCH=\\\"$(go env GOHOSTARCH)\\\" \\t; \\tapkArch=\\\"$(apk --print-arch)\\\"; \\tcase \\\"$apkArch\\\" in \\t\\tarmhf) export GOARM='6' ;; \\t\\tarmv7) export GOARM='7' ;; \\t\\tx86) export GO386='387' ;; \\tesac; \\t\\twget -O go.tgz \\\"https://golang.org/dl/go$GOLANG_VERSION.src.tar.gz\\\"; \\techo 'eb9ccc8bf59ed068e7eff73e154e4f5ee7eec0a47a610fb864e3332a2fdc8b8c *go.tgz' | sha256sum -c -; \\ttar -C /usr/local -xzf go.tgz; \\trm go.tgz; \\t\\tcd /usr/local/go/src; \\t./make.bash; \\t\\trm -rf \\t\\t/usr/local/go/pkg/bootstrap \\t\\t/usr/local/go/pkg/obj \\t; \\tapk del .build-deps; \\t\\texport PATH=\\\"/usr/local/go/bin:$PATH\\\"; \\tgo version\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"56f675edff3d5d7638a7e77a49ed1efc1f5b6286792337131fd1639a29ce3aea\",\"parent\":\"9c99e7695d6a4c5f1fb0b4be8d49c27310cb6a79635c0d9bda272efa09683662\",\"created\":\"2020-04-08T23:12:35.28291307Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  ENV GOLANG_VERSION=1.13.10\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"9c99e7695d6a4c5f1fb0b4be8d49c27310cb6a79635c0d9bda272efa09683662\",\"parent\":\"4f0709d62a5b072a5841a4440a7962203f80a0f2c6aff7198ac627b387ae796d\",\"created\":\"2020-03-23T23:02:04.091296738Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c [ ! -e /etc/nsswitch.conf ] \\u0026\\u0026 echo 'hosts: files dns' \\u003e /etc/nsswitch.conf\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"4f0709d62a5b072a5841a4440a7962203f80a0f2c6aff7198ac627b387ae796d\",\"parent\":\"8b47cd8053b8906fe7683a390075f2a883817bc310c93b93fcf1ab0a1d352fec\",\"created\":\"2020-03-23T23:02:02.546035752Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c apk add --no-cache \\t\\tca-certificates\"]}}"
    },
    {
      "v1Compatibility": "{\"id\":\"8b47cd8053b8906fe7683a390075f2a883817bc310c93b93fcf1ab0a1d352fec\",\"parent\":\"9c343fa0dd9bbe3613b9cb8c3bd1faa61fc2e7b94c551ac7c12bf8f00f0db427\",\"created\":\"2020-03-23T21:19:34.196162891Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop)  CMD [\\\"/bin/sh\\\"]\"]},\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"9c343fa0dd9bbe3613b9cb8c3bd1faa61fc2e7b94c551ac7c12bf8f00f0db427\",\"created\":\"2020-03-23T21:19:34.027725872Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:0c4555f363c2672e350001f1293e689875a3760afe7b3f9146886afe67121cba in / \"]}}"
    }
  ],
  "schemaVersion": 1,
  "signatures": [
    {
      "header": {
        "jwk": {
          "crv": "P-256",
          "kid": "HXAD:MM7N:OQ5J:TUWF:L2PY:LKOO:427H:POOW:L3SI:FLS6:Q3CY:4SUE",
          "kty": "EC",
          "x": "QNnrlEZ8unL9Yk6LjEcna7t_Yk1H4IcEMxb9gZGf6kQ",
          "y": "yzs4xGrQcR3a-HhSJgRWeUhZ8sVBfPUkQp_UU4W7of0"
        },
        "alg": "ES256"
      },
      "signature": "mtRZsDEMdPRndek4SRt7U9vtGHMm15bp56uPWIQdxNqKLtffsG-RkvKQ5KVheKduLZibI4Grjk5-sImOTS90EA",
      "protected": "eyJmb3JtYXRMZW5ndGgiOjEzNDgxLCJmb3JtYXRUYWlsIjoiZlEiLCJ0aW1lIjoiMjAyMC0wNC0yMlQxNTozNToxOVoifQ"
    }
  ]
}

Important: Removing the suggested file allowed for the V2s2 manifest to be successfully pushed to the quay repository. Once the file was recreated after the first push, and the image was pushed again, the error was encountered again, and the V2s1 manifest was pushed to quay. Also is strangely seems like the --remove-signatures flag is not working properly as the signatures are still in the V2s1 manifest pushed.

[mtrmac]

Important: Removing the suggested file allowed for the V2s2 manifest to be successfully pushed to the quay repository. Once the file was recreated after the first push, and the image was pushed again, the error was encountered again, and the V2s1 manifest was pushed to quay.

Thanks, this confirms that https://github.com/containers/image/issues/733 is indeed the case (and that it is more important than previously thought).

Also is strangely seems like the --remove-signatures flag is not working properly as the signatures are still in the V2s1 manifest pushed.

The --remove-signatures flag affects “simple signing” signatures, not the v2s1 in-line manifest signatures (which are a mandatory part of the format, necessary for Docker to accept the manifest). --remove-signatures should not affect anything for you AFAICT, podman build cannot create such a signature, so there is nothing to remove.

[nathanbrophy]

If https://github.com/containers/image/issues/733 is indeed the case, then do we want to link/mention this issue in 733 and close this out or keep open until resolved?

The --remove-signatures flag affects “simple signing” signatures, not the v2s1 in-line manifest signatures (which are a mandatory part of the format, necessary for Docker to accept the manifest). --remove-signatures should not affect anything for you AFAICT, podman build cannot create such a signature, so there is nothing to remove.

ah... thank you for clearing up my confusion around this flag.

[Frederik-Baetens]

Thanks, this confirms that containers/image#733 is indeed the case (and that it is more important than previously thought).

I'd like to add that that issue is indeed serious for me, it basically renders podman/buildah unusable for me.

[rhatdan]

@vrothberg @mtrmac WDYT?

[mtrmac]

@rhatdan WDIT about what?

[rhatdan]

Can we move containers/image#733 forward?

[vrothberg]

Can we move containers/image#733 forward?

My queue is currently full with v2-work followed by systemd stuff. Unless priorities change, I won't find time to tackle 733 soon.

[mtrmac]

Can we move containers/image#733 forward?

It’s the next thing on my list (unless something more important comes up).

[pcellix]

hi @mtrmac
Have you had an opportunity to work on this issue?
If you haven't, can you give me some guidance to work on this

Thank you
Wojtek

[mtrmac]

I’m afraid other things keep coming up. Basically BlobInfoCache needs to record MIME types, and all of that without breaking any existing API (and callers who use the original BlobInfoCache interface) if we don’t want to move to /v6/.

[rhatdan]

@mtrmac Any progress?

[leifmadsen]

Does this issue effectively make it impossible to use podman to build bundle images and create custom registries? I'm running into what looks like this same issue when using operator-sdk (v0.16.0) and opm to create a custom bundle registry. (I'm trying to migrate from application registries to bundle registries.)

[Frederik-Baetens]

https://github.com/containers/image/issues/733 Used to make it impossible for me, but I haven't been able to reproduce after some brief testing.

[Frederik-Baetens]

Nvm, reproduced it again, but --format docker does seem to fix this, but that's with a sample size of 1 container build, so I don't know how reliable that workaround might be.

[qhaas]

I'd like to add that that issue is indeed serious for me, it basically renders podman/buildah unusable for me.

Yeah, same here... and I didn't find out it is an issue until I tried to build a Singularity image from a docker image in a GitLab docker registry that was built/pushed using podman:

$ singularity pull --docker-login docker://INTERNAL_GITLAB_URL_HERE/MY_IMAGE
...
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: read next entry: archive/tar: invalid tar header

In my Makfile build script, I just send it to the ancient docker version in the RHEL7 repo to push, which is slow/ugly, but works:

podman push $$(podman images -q $(MY_IMAGE)) docker-daemon:$(MY_IMAGE)
docker push $(MY_IMAGE)

[arturobrzut]

the same issue
podman version 2.1.1
operator-sdk v0.19.4
and opm 1.12.3
Does this issue effectively make it impossible to use podman to build bundle images and create custom registries? I'm running into what looks like this same issue when using operator-sdk and opm to create a custom bundle registry.

[kt-pham]

I'd like to add that that issue is indeed serious for me, it basically renders podman/buildah unusable for me.

Yeah, same here... and I didn't find out it is an issue until I tried to build a Singularity image from a docker image in a GitLab docker registry that was built/pushed using podman:

$ singularity pull --docker-login docker://INTERNAL_GITLAB_URL_HERE/MY_IMAGE
...
FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: read next entry: archive/tar: invalid tar header

In my Makfile build script, I just send it to the ancient docker version in the RHEL7 repo to push, which is slow/ugly, but works:

podman push $$(podman images -q $(MY_IMAGE)) docker-daemon:$(MY_IMAGE)
docker push $(MY_IMAGE)

+1 my use case is exactly the same. I also get the same invalid tar header error when trying to build/pull as singularity image after using podman build with --format docker.

[arturobrzut]

I can confirm that in my case solution is to move to the Artficatory
on Quay.io with Podman image from opm tool is always convert to schema v1, during push

DEBU[0004] Writing manifest using preferred type application/vnd.docker.distribution.manifest.v2+json failed: Error writing manifest: Error uploading manifest docker to quay.io/arturobrzut/test-image-v2s2: manifest invalid: manifest invalid
DEBU[0004] Trying to use manifest type application/vnd.docker.distribution.manifest.v1+prettyjws…

BUT image generate from bundle.dockerfile are correctly store in schema v2, also with the Podman and quay.io
It means that some images on Quay.io are converted to schema v1 and some are correctly stored in schema v2 when I used the Podman.
I can not confirm that Quay.io with docker works correctly because I cannot install docker on my RHEL 8.2

[tierpod]

I had kinda similar problem with OKD. I built an image with podman build, pushed it to local nexus repository manager with podman push, and OKD raised the error while pulling this image. I added -f v2s2 to podman push command and this solved my problem:

podman build ...
podman push -f v2s2 ...

[arturobrzut]

In my case when I run

podman push -f v2s2
I got an error that manifest with schema 2 can not be store

[arturobrzut]

and one more info
I can push the image to the artifactory without issues with schema v2
and after that, I can run
skopeo copy from artifactory to quay.io
but during this copy schema is also change to v1

❯ skopeo copy docker://hyc-license-team-docker-local.artifactory.swg-devops.com/ibm-licensing-operator-bundle-index:1.3.0 docker://quay.io/arturobrzut/ibm-licensing-operator-bundle-index:9.3.0
Getting image source signatures
Copying blob 30401b6e2589 skipped: already exists  
Copying blob c666c8d3e70d skipped: already exists  
Copying blob 197dca670825 skipped: already exists  
Copying blob cd6c8b37db9e skipped: already exists  
Copying blob b79f5aced4eb skipped: already exists  
Copying blob 364dcac827e6 [--------------------------------------] 0.0b / 0.0b
Copying config 16576f788e [--------------------------------------] 0.0b / 3.6KiB
Writing manifest to image destination
Writing manifest to image destination
Storing signatures
❯ manifest-tool inspect hyc-license-team-docker-local.artifactory.swg-devops.com/ibm-licensing-operator-bundle-index:1.3.0
hyc-license-team-docker-local.artifactory.swg-devops.com/ibm-licensing-operator-bundle-index:1.3.0: manifest type: application/vnd.docker.distribution.manifest.v2+json
      Digest: sha256:79be25299ab11e29095b2f13f2e48cf19d961734fc36a6ee48e94dc0fca3b4a4
Architecture: amd64
          OS: linux
    # Layers: 6
      layer 1: digest = sha256:364dcac827e61e43513fdbb0b651f1f5e94e7759f0232f0be33f8e2739f14e7d
      layer 2: digest = sha256:30401b6e2589e8f259d8798211a7eb29d34c8f054cd841cf300c1cada14712cc
      layer 3: digest = sha256:c666c8d3e70d66a0a471266b8416ed6862e8c30bab810b324f7dd9aaceb06eea
      layer 4: digest = sha256:b79f5aced4eb1acdeb6539fcca167950adc42a1bc98a7da6b9974ccfb35218a4
      layer 5: digest = sha256:cd6c8b37db9ef61357df31fbdacc409a5a34ba9b223ca133fefc8803ea817254
      layer 6: digest = sha256:197dca6708251e002fdc31706f51d55e7373b056c706a67d9243af5eeebc2bf0
❯ manifest-tool inspect quay.io/arturobrzut/ibm-licensing-operator-bundle-index:9.3.0
quay.io/arturobrzut/ibm-licensing-operator-bundle-index:9.3.0: manifest type: application/vnd.docker.distribution.manifest.v1+json
      Digest: sha256:96dea6cb19a79cea57dc9a0ef2f4354ad38b22ecc4bd4d969bf9f64ca56862c8
Architecture: amd64
          OS: linux
    # Layers: 7
      layer 1: digest = sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
      layer 2: digest = sha256:197dca6708251e002fdc31706f51d55e7373b056c706a67d9243af5eeebc2bf0
      layer 3: digest = sha256:cd6c8b37db9ef61357df31fbdacc409a5a34ba9b223ca133fefc8803ea817254
      layer 4: digest = sha256:b79f5aced4eb1acdeb6539fcca167950adc42a1bc98a7da6b9974ccfb35218a4
      layer 5: digest = sha256:c666c8d3e70d66a0a471266b8416ed6862e8c30bab810b324f7dd9aaceb06eea
      layer 6: digest = sha256:30401b6e2589e8f259d8798211a7eb29d34c8f054cd841cf300c1cada14712cc
      layer 7: digest = sha256:364dcac827e61e43513fdbb0b651f1f5e94e7759f0232f0be33f8e2739f14e7d

from debug skopeo the same error like in podman

DEBU[0005] Writing manifest using preferred type application/vnd.docker.distribution.manifest.v2+json failed: Error writing manifest: Error uploading manifest 10.3.0 to quay.io/arturobrzut/ibm-licensing-operator-bundle-index: manifest invalid: manifest invalid 
DEBU[0005] Trying to use manifest type application/vnd.docker.distribution.manifest.v1+prettyjws… 
DEBU[0005] Uploading empty layer during conversion to schema 1 

[kurtismullins]

Quay developer here -- I just stumbled across this Github issue while investigating a related issue:
https://issues.redhat.com/browse/PROJQUAY-1207

Quay performs mimetype validation and sticks to the specifications relatively strictly. I am having trouble finding any references to uncompressed layers in any specification but am far from an expert.

I'd like to understand why this uncompressed mimetype is used as I can't find any hints on it in any documentation or specs. Would anyone be able to point me in the right direction or shed some light on it? I don't have a lot of historical knowledge on the specifications.

Thanks!

[TomSweeneyRedHat]

@nalind or @mtrmac want to take a crack at this?

[mtrmac]

Quay developer here -- I just stumbled across this Github issue while investigating a related issue:
https://issues.redhat.com/browse/PROJQUAY-1207

Quay performs mimetype validation and sticks to the specifications relatively strictly. I am having trouble finding any references to uncompressed layers in any specification but am far from an expert.

IIRC the uncompressed type is not defined in (any) specification; it’s just a string constant that exists internally in the docker/docker and containerd implementations, and was convenient to use for in-memory and local-storage representations.

I'd like to understand why this uncompressed mimetype is used as I can't find any hints on it in any documentation or specs.

It’s an implementation artifact, not an intentional desire to store uncompressed data in the registry; in fact the actual data _is_ compressed. (IIRC the locally-built images (may) use the uncompressed MIME type in the internal representation of image in local storage; the code does know that images recorded in a registry should be compressed, and ordinarily the compression is accompanied with an update to the MIME type — but there’s a bug in the “we know the layer already exists in the registry” case, where the MIME type is kept unmodified = uncompressed).

To link things together, @nalind has been working on this in https://github.com/containers/image/pull/1089 .

[kurtismullins]

@mtrmac

Thank you very much for the explanation. That provided a lot of insight into how these images are stored locally (or in-memory) and where this MIME type came from. Please let me know if I can assist with verifying any changes against Quay.