buildah bud fails at commit stage for layered images from private registry with internal error

Created on 13 May 2018  Â·  51Comments  Â·  Source: containers/buildah

Description

Whenever I'm building a layered image from my private registry (OpenShift Origin 1.5 registry), it breaks at the end with weird messages like this:

STEP 4: COMMIT containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@ac887cd1c3698c3247f983681a24daeb2e35e9497f0a9e77fc120a310c43cf7b
error copying layers and metadata: Error initializing source containers-storage:centos-buildah-working-container: internal error: history lists 5 non-empty layers, but we have 10 layers on disk

Steps to reproduce the issue:

  1. Set up Origin 1.5
  2. Build a container in it
  3. Use buildah bud to build a container layered on top of that one.

Example "base" container:

FROM centos:7
RUN yum upgrade -y && yum -y install buildah && rm -rf /var/cache/yum/*
CMD /bin/bash

Example "layered" container:

FROM registry.localhost.localdomain/test/centos-buildah
RUN yum -y install docker && rm -rf /var/cache/yum/*
CMD /bin/bash

Describe the results you received:
It fails to build due to internal error with history of layers, as noted in the description.

Describe the results you expected:
I expected the commit stage to succeed and the image to build.

Output of rpm -q buildah:

On my CentOS 7 box:

buildah-0.11-3.git49095a8.el7.x86_64

On my Fedora 28 box:

buildah-0.16-1.git82529cd.fc28.x86_64

Output of buildah version:

On my CentOS 7 box:

Version:       0.11
Go Version:    go1.8.3
Image Spec:    1.0.0
Runtime Spec:  1.0.0
Git Commit:    49095a8
Built:         Wed Mar  7 17:03:34 2018
OS/Arch:       linux/amd64

On my Fedora 28 box:

Version:       0.16
Go Version:    go1.10
Image Spec:    1.0.0
Runtime Spec:  1.0.0
Git Commit:    82529cd
Built:         Wed Apr  4 10:22:20 2018
OS/Arch:       linux/amd64

Output of cat /etc/*release:

CentOS 7 box:

CentOS Linux release 7.4.1708 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.4.1708 (Core)
CentOS Linux release 7.4.1708 (Core)

Fedora 28 box:

Fedora release 28 (Twenty Eight)
NAME=Fedora
VERSION="28 (Twenty Eight)"
ID=fedora
VERSION_ID=28
PLATFORM_ID="platform:f28"
PRETTY_NAME="Fedora 28 (Twenty Eight)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:28"
HOME_URL="https://fedoraproject.org/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=28
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=28
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
Fedora release 28 (Twenty Eight)
Fedora release 28 (Twenty Eight)

Output of uname -a:

CentOS 7 box:

Linux localhost 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Fedora 28 box:

Linux localhost 4.16.4-300.fc28.x86_64 #1 SMP Tue Apr 24 22:15:13 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Output of cat /etc/containers/storage.conf:

# storage.conf is the configuration file for all tools
# that share the containers/storage libraries
# See man 5 containers-storage.conf for more information

# The "container storage" table contains all of the server options.
[storage]

# Default Storage Driver
driver = "overlay"

# Temporary storage location
runroot = "/var/run/containers/storage"

# Primary Read/Write location of container storage
graphroot = "/var/lib/containers/storage"

[storage.options]
# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
additionalimagestores = [
]

# Size is used to set a maximum size of the container image.  Only supported by
# certain container storage drivers.
size = ""

# OverrideKernelCheck tells the driver to ignore kernel checks based on kernel version
override_kernel_check = "true"

Configuration is same for both CentOS and Fedora.

kinbug
Source
picture Conan-Kudo

All 51 comments

You should be able to get a newer version of buildah on Centos? @lsm5 Where is the latest packages available.

rhatdan picture rhatdan on 14 May 2018

It fails with both buildah 0.11 and buildah 0.16, and CentOS 7 is only up to buildah 0.15 with CentOS 7.5, as far as I can tell.

Conan-Kudo picture Conan-Kudo on 14 May 2018

@nalind Any ideas?

rhatdan picture rhatdan on 14 May 2018

@Conan-Kudo, there are newer rpm's available here for CentOS (1.0)
https://cbs.centos.org/repos/virt7-container-common-candidate/x86_64/os/Packages/

Could you install that and re-test to confirm the issue still exists?

pixdrift picture pixdrift on 15 May 2018

I don't have CentOS 7 on hand right now, but I did grab buildah 1.0 from Fedora 28 updates-testing:

[neal@localhost test]$ buildah version
Version:       1.0
Go Version:    go1.10.1
Image Spec:    1.0.0
Runtime Spec:  1.0.0
Git Commit:    1ab80bc
Built:         Mon May  7 10:01:02 2018
OS/Arch:       linux/amd64

I still got the same error:

STEP 4: COMMIT containers-storage:[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]@4de04861d6e0f269b62ee9296dfd3df9fe176616994eb38ce146f71f817299a8
error copying layers and metadata: Error initializing source containers-storage:centos-buildah-working-container: internal error: history lists 5 non-empty layers, but we have 10 layers on disk
Conan-Kudo picture Conan-Kudo on 15 May 2018

Thanks @Conan-Kudo, apologies to keep asking for more details...

To confirm, what is building the original image that you are pulling with buildah?

Can you provide the output from buildah inspect on the container after doing just the:
FROM registry.localhost.localdomain/test/centos-buildah

pixdrift picture pixdrift on 15 May 2018

The original "base" image was built by Origin 1.5 and stored in its registry.

Here's buildah inspect output:

[neal@localhost ~]$ sudo buildah inspect "registry.localhost.localdomain/test/centos-buildah"
{
    "Type": "buildah 0.0.1",
    "FromImage": "registry.localhost.localdomain/test/centos-buildah:latest",
    "FromImageID": "5f945be5832c09cc9468c6f340a50835e03da014185188b8194b8ecee39e4876",
    "Config": "",
    "Manifest": "{\n   \"schemaVersion\": 1,\n   \"name\": \"test/centos-buildah\",\n   \"tag\": \"latest\",\n   \"architecture\": \"amd64\",\n   \"fsLayers\": [\n      {\n         \"blobSum\": \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\"\n      },\n      {\n         \"blobSum\": \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\"\n      },\n      {\n         \"blobSum\": \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\"\n      },\n      {\n         \"blobSum\": \"sha256:1350cd11631cfdcc33b53eef8fb5c3f98d1f24583301ab40f6865ecdfae7a6c3\"\n      },\n      {\n         \"blobSum\": \"sha256:7162bc5ef6b3858e630e8a03e5b0c8e5e6e1f14a519d378d07de769f6b1876a8\"\n      },\n      {\n         \"blobSum\": \"sha256:519dcdaf32b3a8b47d8da2a0d946e8a0397e12051dba674021fa549831e70739\"\n      },\n      {\n         \"blobSum\": \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\"\n      },\n      {\n         \"blobSum\": \"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4\"\n      },\n      {\n         \"blobSum\": \"sha256:364f9b7c969aed6ff05b7ced04b1b53b824386288dffb1e6aec41d5a706dd51e\"\n      }\n   ],\n   \"history\": [\n      {\n         \"v1Compatibility\": \"{\\\"architecture\\\":\\\"amd64\\\",\\\"config\\\":{\\\"Hostname\\\":\\\"c7947be2eb9d\\\",\\\"Domainname\\\":\\\"\\\",\\\"User\\\":\\\"\\\",\\\"AttachStdin\\\":false,\\\"AttachStdout\\\":false,\\\"AttachStderr\\\":false,\\\"Tty\\\":false,\\\"OpenStdin\\\":false,\\\"StdinOnce\\\":false,\\\"Env\\\":[\\\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\\\",\\\"OPENSHIFT_BUILD_NAME=centos-buildah-6\\\",\\\"OPENSHIFT_BUILD_NAMESPACE=test\\\",\\\"OPENSHIFT_BUILD_SOURCE=https://git.localdomain/test/centos-buildah.git\\\",\\\"OPENSHIFT_BUILD_COMMIT=338caa77213aa797a92794e600a1f2c746a045ee\\\"],\\\"Cmd\\\":[\\\"/bin/sh\\\",\\\"-c\\\",\\\"/bin/bash\\\"],\\\"ArgsEscaped\\\":true,\\\"Image\\\":\\\"sha256:03b056c907197cc51baca684cfab7928e63a2034f96334c15da428da4f4987c3\\\",\\\"Volumes\\\":null,\\\"WorkingDir\\\":\\\"\\\",\\\"Entrypoint\\\":null,\\\"OnBuild\\\":[],\\\"Labels\\\":{\\\"build-date\\\":\\\"20170801\\\",\\\"io.openshift.build.commit.author\\\":\\\"Foo User \\\\\\\\[email protected]\\\\\\\\u003e\\\",\\\"io.openshift.build.commit.date\\\":\\\"Thu Mar 22 16:36:55 2018 -0400\\\",\\\"io.openshift.build.commit.id\\\":\\\"338caa77213aa797a92794e600a1f2c746a045ee\\\",\\\"io.openshift.build.commit.message\\\":\\\"Merge branch 'openshift-registry' into 'master'\\\",\\\"io.openshift.build.commit.ref\\\":\\\"master\\\",\\\"io.openshift.build.source-location\\\":\\\"https://git.localdomain/test/centos-buildah.git\\\",\\\"license\\\":\\\"GPLv2\\\",\\\"name\\\":\\\"CentOS Base Image\\\",\\\"vendor\\\":\\\"CentOS\\\"}},\\\"container\\\":\\\"a4a4e4bf704c0b4ef9aef256f2f88602228c05f929788b7ab2a54a8e219c3425\\\",\\\"container_config\\\":{\\\"Hostname\\\":\\\"c7947be2eb9d\\\",\\\"Domainname\\\":\\\"\\\",\\\"User\\\":\\\"\\\",\\\"AttachStdin\\\":false,\\\"AttachStdout\\\":false,\\\"AttachStderr\\\":false,\\\"Tty\\\":false,\\\"OpenStdin\\\":false,\\\"StdinOnce\\\":false,\\\"Env\\\":[\\\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\\\",\\\"OPENSHIFT_BUILD_NAME=centos-buildah-6\\\",\\\"OPENSHIFT_BUILD_NAMESPACE=test\\\",\\\"OPENSHIFT_BUILD_SOURCE=https://git.localdomain/test/centos-buildah.git\\\",\\\"OPENSHIFT_BUILD_COMMIT=338caa77213aa797a92794e600a1f2c746a045ee\\\"],\\\"Cmd\\\":[\\\"/bin/sh\\\",\\\"-c\\\",\\\"#(nop) \\\",\\\"LABEL io.openshift.build.commit.ref=master io.openshift.build.commit.message=Merge branch 'openshift-registry' into 'master' io.openshift.build.source-location=https://git.localdomain/test/centos-buildah.git io.openshift.build.commit.author=Foo User \\\\\\\\[email protected]\\\\\\\\u003e io.openshift.build.commit.date=Thu Mar 22 16:36:55 2018 -0400 io.openshift.build.commit.id=338caa77213aa797a92794e600a1f2c746a045ee\\\"],\\\"ArgsEscaped\\\":true,\\\"Image\\\":\\\"sha256:03b056c907197cc51baca684cfab7928e63a2034f96334c15da428da4f4987c3\\\",\\\"Volumes\\\":null,\\\"WorkingDir\\\":\\\"\\\",\\\"Entrypoint\\\":null,\\\"OnBuild\\\":[],\\\"Labels\\\":{\\\"build-date\\\":\\\"20170801\\\",\\\"io.openshift.build.commit.author\\\":\\\"Foo User \\\\\\\\[email protected]\\\\\\\\u003e\\\",\\\"io.openshift.build.commit.date\\\":\\\"Thu Mar 22 16:36:55 2018 -0400\\\",\\\"io.openshift.build.commit.id\\\":\\\"338caa77213aa797a92794e600a1f2c746a045ee\\\",\\\"io.openshift.build.commit.message\\\":\\\"Merge branch 'openshift-registry' into 'master'\\\",\\\"io.openshift.build.commit.ref\\\":\\\"master\\\",\\\"io.openshift.build.source-location\\\":\\\"https://git.localdomain/test/centos-buildah.git\\\",\\\"license\\\":\\\"GPLv2\\\",\\\"name\\\":\\\"CentOS Base Image\\\",\\\"vendor\\\":\\\"CentOS\\\"}},\\\"created\\\":\\\"2018-03-22T20:38:52.056958533Z\\\",\\\"docker_version\\\":\\\"17.06.0-ce\\\",\\\"id\\\":\\\"24b1ceb8bcbf757d35185805bac655ddd0182b5affcf1cd2b3f06badf9602caa\\\",\\\"os\\\":\\\"linux\\\",\\\"parent\\\":\\\"1bf268e54141f52fca759cef6bc5d18cf14d1bec7a2f4d19430cfe8ee63675e0\\\",\\\"throwaway\\\":true}\"\n      },\n      {\n         \"v1Compatibility\": \"{\\\"id\\\":\\\"1bf268e54141f52fca759cef6bc5d18cf14d1bec7a2f4d19430cfe8ee63675e0\\\",\\\"parent\\\":\\\"d9b052a2eefe7376e556a5ec4133bdeb6936bc501f0b5c30efe3a15e5f4cc1d7\\\",\\\"created\\\":\\\"2018-03-22T20:38:51.754573776Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"/bin/sh -c #(nop)  ENV OPENSHIFT_BUILD_NAME=centos-buildah-6 OPENSHIFT_BUILD_NAMESPACE=test OPENSHIFT_BUILD_SOURCE=https://git.localdomain/test/centos-buildah.git OPENSHIFT_BUILD_COMMIT=338caa77213aa797a92794e600a1f2c746a045ee\\\"]},\\\"throwaway\\\":true}\"\n      },\n      {\n         \"v1Compatibility\": \"{\\\"id\\\":\\\"d9b052a2eefe7376e556a5ec4133bdeb6936bc501f0b5c30efe3a15e5f4cc1d7\\\",\\\"parent\\\":\\\"05439a9b2434a35b0363073df73ab5a4314373b55663d56a480e6352e2a1f880\\\",\\\"created\\\":\\\"2018-03-22T20:38:51.411742378Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"/bin/sh -c #(nop)  CMD [\\\\\\\"/bin/sh\\\\\\\" \\\\\\\"-c\\\\\\\" \\\\\\\"/bin/bash\\\\\\\"]\\\"]},\\\"throwaway\\\":true}\"\n      },\n      {\n         \"v1Compatibility\": \"{\\\"id\\\":\\\"05439a9b2434a35b0363073df73ab5a4314373b55663d56a480e6352e2a1f880\\\",\\\"parent\\\":\\\"5fec6d2b58d3db6ef6b7ca6f05ba263908a4743d4bfd5f56f7dbab04712d51d1\\\",\\\"created\\\":\\\"2018-03-22T20:38:51.045032509Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"/bin/sh -c #(nop) COPY file:6f8a74bf55c02ec9d4b1e85b3ee3c59c143324ab72f1e368c3ffa0a71bdd9c9d in /etc/docker/certs.d/registry.localhost.localdomain/ca.crt \\\"]}}\"\n      },\n      {\n         \"v1Compatibility\": \"{\\\"id\\\":\\\"5fec6d2b58d3db6ef6b7ca6f05ba263908a4743d4bfd5f56f7dbab04712d51d1\\\",\\\"parent\\\":\\\"6da211d9b7eadee2a41fd601f41ef0a62756c2f0d20b05b9e20e21744eca8563\\\",\\\"created\\\":\\\"2018-03-22T20:38:50.461760293Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"/bin/sh -c mkdir -p /etc/docker/certs.d/registry.localhost.localdomain\\\"]}}\"\n      },\n      {\n         \"v1Compatibility\": \"{\\\"id\\\":\\\"6da211d9b7eadee2a41fd601f41ef0a62756c2f0d20b05b9e20e21744eca8563\\\",\\\"parent\\\":\\\"3d7b7187fa4403f60db13f58dfcfd6dda87197503dc01774a724c13c625b253e\\\",\\\"created\\\":\\\"2018-03-22T20:38:48.321320088Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"/bin/sh -c yum upgrade -y \\\\u0026\\\\u0026 yum -y install buildah \\\\u0026\\\\u0026 rm -rf /var/cache/yum/*\\\"]}}\"\n      },\n      {\n         \"v1Compatibility\": \"{\\\"id\\\":\\\"3d7b7187fa4403f60db13f58dfcfd6dda87197503dc01774a724c13c625b253e\\\",\\\"parent\\\":\\\"0dbd303bf3506022774df40d223064191392f1cd228448358500ad6e59345b2d\\\",\\\"created\\\":\\\"2017-08-03T21:34:56.600704666Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"/bin/sh -c #(nop)  CMD [\\\\\\\"/bin/bash\\\\\\\"]\\\"]},\\\"throwaway\\\":true}\"\n      },\n      {\n         \"v1Compatibility\": \"{\\\"id\\\":\\\"0dbd303bf3506022774df40d223064191392f1cd228448358500ad6e59345b2d\\\",\\\"parent\\\":\\\"1d03f599ff50fcb3f5c16fd5d528d9ba320fbe8ec4cdaaba50f01d0ef1583151\\\",\\\"created\\\":\\\"2017-08-03T21:34:55.624825166Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"/bin/sh -c #(nop)  LABEL name=CentOS Base Image vendor=CentOS license=GPLv2 build-date=20170801\\\"]},\\\"throwaway\\\":true}\"\n      },\n      {\n         \"v1Compatibility\": \"{\\\"id\\\":\\\"1d03f599ff50fcb3f5c16fd5d528d9ba320fbe8ec4cdaaba50f01d0ef1583151\\\",\\\"created\\\":\\\"2017-08-03T21:34:51.580892197Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"/bin/sh -c #(nop) ADD file:63492ba809361c51e75605d70390b549ff1187076b6d00485a1a0bb175daa40e in / \\\"]}}\"\n      }\n   ],\n   \"signatures\": [\n      {\n         \"header\": {\n            \"jwk\": {\n               \"crv\": \"P-256\",\n               \"kid\": \"NFHB:CNRV:RVB3:V5UH:SLAE:LH3A:XHJ4:4V2Y:TNBU:MC6D:6RP6:KV7A\",\n               \"kty\": \"EC\",\n               \"x\": \"yYRYZUTXwZwbZJuFnMWUyYaBNKRsM4XSYMuJx-iEFs4\",\n               \"y\": \"iHHZ3KyO963Vzsh67HKPTqTKuSdpoI4loVpEjsjwIDk\"\n            },\n            \"alg\": \"ES256\"\n         },\n         \"signature\": \"A9d1x7nI8fICraMesdigJvTPqKg81badXbfyQb7-rcgDjFYRlgwIABbnuA8FFKy51xes9N_M3eNSpnna1-TB7Q\",\n         \"protected\": \"eyJmb3JtYXRMZW5ndGgiOjgwODcsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxOC0wNS0xM1QwNDo1NjoxMFoifQ\"\n      }\n   ]\n}",
    "Container": "",
    "ContainerID": "",
    "MountPoint": "",
    "ProcessLabel": "",
    "MountLabel": "",
    "ImageAnnotations": null,
    "ImageCreatedBy": "",
    "OCIv1": {
        "created": "2018-03-22T20:38:52.056958533Z",
        "architecture": "amd64",
        "os": "linux",
        "config": {
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "OPENSHIFT_BUILD_NAME=centos-buildah-6",
                "OPENSHIFT_BUILD_NAMESPACE=test",
                "OPENSHIFT_BUILD_SOURCE=https://git.localdomain/test/centos-buildah.git",
                "OPENSHIFT_BUILD_COMMIT=338caa77213aa797a92794e600a1f2c746a045ee"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "/bin/bash"
            ],
            "WorkingDir": "/",
            "Labels": {
                "build-date": "20170801",
                "io.openshift.build.commit.author": "Foo User \\[email protected]\\u003e",
                "io.openshift.build.commit.date": "Thu Mar 22 16:36:55 2018 -0400",
                "io.openshift.build.commit.id": "338caa77213aa797a92794e600a1f2c746a045ee",
                "io.openshift.build.commit.message": "Merge branch 'openshift-registry' into 'master'",
                "io.openshift.build.commit.ref": "master",
                "io.openshift.build.source-location": "https://git.localdomain/test/centos-buildah.git",
                "license": "GPLv2",
                "name": "CentOS Base Image",
                "vendor": "CentOS"
            }
        },
        "rootfs": {
            "type": "layers",
            "diff_ids": []
        },
        "history": [
            {
                "created": "2017-08-03T21:34:51.580892197Z",
                "created_by": "/bin/sh -c #(nop) ADD file:63492ba809361c51e75605d70390b549ff1187076b6d00485a1a0bb175daa40e in / "
            },
            {
                "created": "2017-08-03T21:34:55.624825166Z",
                "created_by": "/bin/sh -c #(nop)  LABEL name=CentOS Base Image vendor=CentOS license=GPLv2 build-date=20170801",
                "empty_layer": true
            },
            {
                "created": "2017-08-03T21:34:56.600704666Z",
                "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",
                "empty_layer": true
            },
            {
                "created": "2018-03-22T20:38:48.321320088Z",
                "created_by": "/bin/sh -c yum upgrade -y && yum -y install buildah && rm -rf /var/cache/yum/*"
            },
            {
                "created": "2018-03-22T20:38:50.461760293Z",
                "created_by": "/bin/sh -c mkdir -p /etc/docker/certs.d/registry.localhost.localdomain"
            },
            {
                "created": "2018-03-22T20:38:51.045032509Z",
                "created_by": "/bin/sh -c #(nop) COPY file:6f8a74bf55c02ec9d4b1e85b3ee3c59c143324ab72f1e368c3ffa0a71bdd9c9d in /etc/docker/certs.d/registry.localhost.localdomain/ca.crt "
            },
            {
                "created": "2018-03-22T20:38:51.411742378Z",
                "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/sh\" \"-c\" \"/bin/bash\"]",
                "empty_layer": true
            },
            {
                "created": "2018-03-22T20:38:51.754573776Z",
                "created_by": "/bin/sh -c #(nop)  ENV OPENSHIFT_BUILD_NAME=centos-buildah-6 OPENSHIFT_BUILD_NAMESPACE=test OPENSHIFT_BUILD_SOURCE=https://git.localdomain/test/centos-buildah.git OPENSHIFT_BUILD_COMMIT=338caa77213aa797a92794e600a1f2c746a045ee",
                "empty_layer": true
            },
            {
                "created": "2018-03-22T20:38:52.056958533Z",
                "created_by": "/bin/sh -c #(nop)  LABEL io.openshift.build.commit.ref=master io.openshift.build.commit.message=Merge branch 'openshift-registry' into 'master' io.openshift.build.source-location=https://git.localdomain/test/centos-buildah.git io.openshift.build.commit.author=Foo User \\[email protected]\\u003e io.openshift.build.commit.date=Thu Mar 22 16:36:55 2018 -0400 io.openshift.build.commit.id=338caa77213aa797a92794e600a1f2c746a045ee",
                "empty_layer": true
            }
        ]
    },
    "Docker": {
        "id": "24b1ceb8bcbf757d35185805bac655ddd0182b5affcf1cd2b3f06badf9602caa",
        "created": "2018-03-22T20:38:52.056958533Z",
        "container": "a4a4e4bf704c0b4ef9aef256f2f88602228c05f929788b7ab2a54a8e219c3425",
        "container_config": {
            "Hostname": "c7947be2eb9d",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "OPENSHIFT_BUILD_NAME=centos-buildah-6",
                "OPENSHIFT_BUILD_NAMESPACE=test",
                "OPENSHIFT_BUILD_SOURCE=https://git.localdomain/test/centos-buildah.git",
                "OPENSHIFT_BUILD_COMMIT=338caa77213aa797a92794e600a1f2c746a045ee"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "/bin/bash"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:03b056c907197cc51baca684cfab7928e63a2034f96334c15da428da4f4987c3",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": null,
            "OnBuild": [],
            "Labels": {
                "build-date": "20170801",
                "io.openshift.build.commit.author": "Foo User \\[email protected]\\u003e",
                "io.openshift.build.commit.date": "Thu Mar 22 16:36:55 2018 -0400",
                "io.openshift.build.commit.id": "338caa77213aa797a92794e600a1f2c746a045ee",
                "io.openshift.build.commit.message": "Merge branch 'openshift-registry' into 'master'",
                "io.openshift.build.commit.ref": "master",
                "io.openshift.build.source-location": "https://git.localdomain/test/centos-buildah.git",
                "license": "GPLv2",
                "name": "CentOS Base Image",
                "vendor": "CentOS"
            }
        },
        "config": {
            "Hostname": "c7947be2eb9d",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "OPENSHIFT_BUILD_NAME=centos-buildah-6",
                "OPENSHIFT_BUILD_NAMESPACE=test",
                "OPENSHIFT_BUILD_SOURCE=https://git.localdomain/test/centos-buildah.git",
                "OPENSHIFT_BUILD_COMMIT=338caa77213aa797a92794e600a1f2c746a045ee"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "/bin/bash"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:03b056c907197cc51baca684cfab7928e63a2034f96334c15da428da4f4987c3",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": null,
            "OnBuild": [],
            "Labels": {
                "build-date": "20170801",
                "io.openshift.build.commit.author": "Foo User \\[email protected]\\u003e",
                "io.openshift.build.commit.date": "Thu Mar 22 16:36:55 2018 -0400",
                "io.openshift.build.commit.id": "338caa77213aa797a92794e600a1f2c746a045ee",
                "io.openshift.build.commit.message": "Merge branch 'openshift-registry' into 'master'",
                "io.openshift.build.commit.ref": "master",
                "io.openshift.build.source-location": "https://git.localdomain/test/centos-buildah.git",
                "license": "GPLv2",
                "name": "CentOS Base Image",
                "vendor": "CentOS"
            }
        },
        "architecture": "amd64",
        "os": "linux",
        "parent": "1bf268e54141f52fca759cef6bc5d18cf14d1bec7a2f4d19430cfe8ee63675e0",
        "rootfs": {
            "type": "layers"
        },
        "history": [
            {
                "created": "2017-08-03T21:34:51.580892197Z",
                "created_by": "/bin/sh -c #(nop) ADD file:63492ba809361c51e75605d70390b549ff1187076b6d00485a1a0bb175daa40e in / "
            },
            {
                "created": "2017-08-03T21:34:55.624825166Z",
                "created_by": "/bin/sh -c #(nop)  LABEL name=CentOS Base Image vendor=CentOS license=GPLv2 build-date=20170801",
                "empty_layer": true
            },
            {
                "created": "2017-08-03T21:34:56.600704666Z",
                "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/bash\"]",
                "empty_layer": true
            },
            {
                "created": "2018-03-22T20:38:48.321320088Z",
                "created_by": "/bin/sh -c yum upgrade -y && yum -y install buildah && rm -rf /var/cache/yum/*"
            },
            {
                "created": "2018-03-22T20:38:50.461760293Z",
                "created_by": "/bin/sh -c mkdir -p /etc/docker/certs.d/registry.localhost.localdomain"
            },
            {
                "created": "2018-03-22T20:38:51.045032509Z",
                "created_by": "/bin/sh -c #(nop) COPY file:6f8a74bf55c02ec9d4b1e85b3ee3c59c143324ab72f1e368c3ffa0a71bdd9c9d in /etc/docker/certs.d/registry.localhost.localdomain/ca.crt "
            },
            {
                "created": "2018-03-22T20:38:51.411742378Z",
                "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/sh\" \"-c\" \"/bin/bash\"]",
                "empty_layer": true
            },
            {
                "created": "2018-03-22T20:38:51.754573776Z",
                "created_by": "/bin/sh -c #(nop)  ENV OPENSHIFT_BUILD_NAME=centos-buildah-6 OPENSHIFT_BUILD_NAMESPACE=test OPENSHIFT_BUILD_SOURCE=https://git.localdomain/test/centos-buildah.git OPENSHIFT_BUILD_COMMIT=338caa77213aa797a92794e600a1f2c746a045ee",
                "empty_layer": true
            },
            {
                "created": "2018-03-22T20:38:52.056958533Z",
                "created_by": "/bin/sh -c #(nop)  LABEL io.openshift.build.commit.ref=master io.openshift.build.commit.message=Merge branch 'openshift-registry' into 'master' io.openshift.build.source-location=https://git.localdomain/test/centos-buildah.git io.openshift.build.commit.author=Foo User \\[email protected]\\u003e io.openshift.build.commit.date=Thu Mar 22 16:36:55 2018 -0400 io.openshift.build.commit.id=338caa77213aa797a92794e600a1f2c746a045ee",
                "empty_layer": true
            }
        ]
    },
    "DefaultMountsFilePath": ""
}
Conan-Kudo picture Conan-Kudo on 15 May 2018
👍1

I have been having trouble creating a reproducer for this, and then it just occured for me in a real world scenario. This is using buildah against an Artifactory private repository.

error committing container "el7-buildah" to "el7-buildah:new": error copying layers and metadata: Error initializing source containers-storage:el7-buildah: internal error: history lists 1 non-empty layers, but we have 3 layers on disk

Basic steps were

  1. Pull image from Artifactory repo using from --pull-always
  2. Add content to container
  3. Attempt to commit container

Everything else about this environment works fine and has been building containers reliably for a month or so (so environment seems OK). It's just in this specific pull, change, commit scenario.

The source container shows 2 non-empty layers in Artifactory before the pull.

-edit Troubleshooting-

Updated to latest commit and the problem still exists.
buildah-1.0-201805230022.git40325d3.el7.x86_64

Reduced the reproducer to the following (no changes need to be made to the image locally before pushing)

newcontainer=$(buildah from --pull-always docker.repository.local/buildah-base:latest)
buildah commit ${newcontainer} buildah-broken-layers

error committing container "buildah-base-working-container-10" to "buildah-broken-layers": error copying layers and metadata: Error initializing source containers-storage:buildah-base-working-container-10: internal error: history lists 1 non-empty layers, but we have 3 layers on disk

Wondering if there is an issue here when it skips the fetch of blobs... does it not update a counter that these blobs are used?

Skipping fetch of repeat blob sha256:eb1f021c41fdb1c41ed24c352cfbd705b661f16c3292f8f9837db0bf490fe2d2
Skipping fetch of repeat blob sha256:b222930bc105954856df7a62c177fcefbbf911525a4dad0ec7942dfa68e5cfe6

-edit 2-

Interestingly, if you look in the Manifest.history attribute of the pulled images, there are a large number of backslashes.. it's like fields are being escaped 2 or 3 times.. is this making it unparseable?

\"history\":[{\"v1Compatibility\":\"{\\\"architecture\\\":\\\"amd64\\\",\\\"config\\\":{\\\"WorkingDir\\\":\\\"/\\\",\\\"Labels\\\":{\\\"name\\\":\\\"el7-apc-base\\\"}},\\\"created\\\":\\\"2018-05-23T06:51:06.011209458Z\\\",\\\"id\\\":\\\"43a3840db11c25bd727a88db625d1973dc427ebe84c7b63022d30ff42d422998\\\",\\\"os\\\":\\\"linux\\\",\\\"parent\\\":\\\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\\\"}\"},{\"v1Compatibility\":\"{\\\"id\\\":\\\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\\\",\\\"created\\\":\\\"2018-04-27T07:06:01.013601243Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"\\\"]}}\"}]

pixdrift picture pixdrift on 24 May 2018

Sorry to post a follow up.. but this one seems like a pretty nasty issue. It doesn't appear to be related to local cache as I have completely removed all local data (containers and images) and re-pulled/commited the image with the reproducer above and it produces the exact same error. The concern here is that the image that is sitting in the Artifactory repository was created by buildah but somehow now breaks buildah on a from/pull.

Same image works fine using docker pull.

-edit-

After several hours I think I have narrowed it down to a specific issue. Following tests provide some insight.

  1. Build image locally from scratch, commit to images, metadata OK
"history":[{"v1Compatibility":"{\"architecture\":\"amd64\",\"config\":{\"WorkingDir\":\"/\",\"Labels\":{\"name\":\"el7-apc-base\"}},\"created\":\"2018-05-23T06:51:06.011209458Z\",\"id\":\"43a3840db11c25bd727a88db625d1973dc427ebe84c7b63022d30ff42d422998\",\"os\":\"linux\",\"parent\":\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\"}"},{"v1Compatibility":"{\"id\":\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\",\"created\":\"2018-04-27T07:06:01.013601243Z\",\"container_config\":{\"Cmd\":[\"\"]}}"}],
  1. Build image locally from scratch, commit to private repository, metadata OK (viewed in Artifactory web UI)
"history":[{"v1Compatibility":"{\"architecture\":\"amd64\",\"config\":{\"WorkingDir\":\"/\",\"Labels\":{\"name\":\"el7-apc-base\"}},\"created\":\"2018-05-23T06:51:06.011209458Z\",\"id\":\"43a3840db11c25bd727a88db625d1973dc427ebe84c7b63022d30ff42d422998\",\"os\":\"linux\",\"parent\":\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\"}"},{"v1Compatibility":"{\"id\":\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\",\"created\":\"2018-04-27T07:06:01.013601243Z\",\"container_config\":{\"Cmd\":[\"\"]}}"}],
  1. Pull the same image from Artifactory (private repository) using buildah from or bud FROM and the Manifest.history portion is corrupted with excessive escaping.
\"history\":[{\"v1Compatibility\":\"{\\\"architecture\\\":\\\"amd64\\\",\\\"config\\\":{\\\"WorkingDir\\\":\\\"/\\\",\\\"Labels\\\":{\\\"name\\\":\\\"el7-apc-base\\\"}},\\\"created\\\":\\\"2018-05-23T06:51:06.011209458Z\\\",\\\"id\\\":\\\"43a3840db11c25bd727a88db625d1973dc427ebe84c7b63022d30ff42d422998\\\",\\\"os\\\":\\\"linux\\\",\\\"parent\\\":\\\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\\\"}\"},{\"v1Compatibility\":\"{\\\"id\\\":\\\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\\\",\\\"created\\\":\\\"2018-04-27T07:06:01.013601243Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"\\\"]}}\"}]

It appears there is something in the from code which is escaping the quotes and corrupting the history metadata if the image is coming from a docker repository. buildah from on the same image locally, before it's pushed to a registry works as expected.

@Conan-Kudo's example is corrupted in the same way coming from an Openshift repository, so it doesn't look specific to repository (Artifactory in this case).

@rhatdan @TomSweeneyRedHat can you take a look? If you need more info I have setup the reproducer in my local dev environment.

pixdrift picture pixdrift on 24 May 2018

Thanks for the very thorough issue report @pixdrift, we'll take a look.

TomSweeneyRedHat picture TomSweeneyRedHat on 24 May 2018

@pixdrift can you pull the image using podman pull? Can you pull it to an OCI Image so we could play with it?

rhatdan picture rhatdan on 24 May 2018

I tested this with the current docker official registry image, and it doesn't appear to have the same problem. The returned image doesn't contain v1Compatability at all, which appears to be the affected key.

pixdrift picture pixdrift on 25 May 2018

OK, I now have a full reproducer using the docker shipped registry. The problem is the v2 schema 1 metadata specifically. Anyone should now be able to reproduce this as long as they have docker + buildah installed.

Will test with podman pull now.

# Setup a private registry
docker run -d -p 5000:5000 --name registry registry:latest

# Pull a container image from docker.io and push it to private registry
newcontainer=$(buildah from alpine)
buildah commit ${newcontainer} myalpine
buildah push --format v2s1 --tls-verify=false myalpine docker://localhost:5000/alpinev2s1

# Use buildah to pull from the private registry and inspect the result to see broken metadata
brokencontainer=$(buildah from --tls-verify=false docker://localhost:5000/alpinev2s1)
buildah inspect ${brokencontainer} | grep Manifest

# Generate error by attempting to commit image
buildah commit ${brokencontainer} broken-layers
error committing container "alpinev2s1-working-container" to "broken-layers": error copying layers and metadata: Error initializing source containers-storage:alpinev2s1-working-container: internal error: history lists 1 non-empty layers, but we have 4 layers on disk

-edit-

As requested @rhatdan

podman pull (podman-0.4.1-4.gitb51d327.el7.x86_64) doesn't like the image

# podman pull --tls-verify=false docker://localhost:5000/alpinev2s1
Trying to pull docker://localhost:5000/alpinev2s1...Getting image source signatures
Skipping fetch of repeat blob sha256:ff3a5c916c92643ff77519ffa742d3ec61b7f591b6b7504599d95a4a41134e28
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
Writing manifest to image destination
Storing signatures
error pulling image "docker://localhost:5000/alpinev2s1": error retrieving local image after pulling docker://localhost:5000/alpinev2s1: invalid reference format

docker pull has no issues

docker pull localhost:5000/alpinev2s1
Using default tag: latest
Trying to pull repository localhost:5000/alpinev2s1 ... 
latest: Pulling from localhost:5000/alpinev2s1
ff3a5c916c92: Already exists 
4f4fb700ef54: Pull complete 
Digest: sha256:d09244c882a4cc80376f697cf3f36e34b90b8e5c18741b9f2b5ca9a2882d9dd7
Status: Downloaded newer image for localhost:5000/alpinev2s1:latest
pixdrift picture pixdrift on 25 May 2018

Thanks again @pixdrift. I'd hoped to take a look at this yesterday, but got waylaid by other fun stuff. Hope to dig at this later today.

TomSweeneyRedHat picture TomSweeneyRedHat on 25 May 2018

@pixdrift, didn't get to this until just a bit ago, but am able to duplicate the issue with your reproducer. We'd a similar issue before with too many slashes, I'll look over the weekend to see if I can dig it up.

TomSweeneyRedHat picture TomSweeneyRedHat on 26 May 2018
👍1

@nalind @mtrmac PTAL

rhatdan picture rhatdan on 31 May 2018

FWIW, this only occurs when pushing using the v2s1 format. Other formats look fine. I think it's somewhere down in the containers code, not sure where. @baude suggested that it might be an issue where the manifest digest is getting converted to json too many times.

TomSweeneyRedHat picture TomSweeneyRedHat on 31 May 2018

@TomSweeneyRedHat Could we make it ignore the v2s1 data, if the processing of that data cannot be fixed?

Conan-Kudo picture Conan-Kudo on 31 May 2018

@Conan-Kudo couple of questions back at you. Is the suggestion to disallow specifying the v2s1 format in the command? If so, that would be relatively easy to do, I'm not sure it's the right thing to do though.

Second question. I'm not real clear on the build command that you used for the initial report. Did you also have --format = v2s1? Can you just drop that as a test if so?

TomSweeneyRedHat picture TomSweeneyRedHat on 31 May 2018

I'm not specifying anything. OpenShift makes those descriptions, and if buildah is going to choke on them and it's not getting fixed, then I need a workaround...

Conan-Kudo picture Conan-Kudo on 31 May 2018

Thanks for the 411 @Conan-Kudo. I've taken a look at it, haven't found a root cause yet but will keep digging.

TomSweeneyRedHat picture TomSweeneyRedHat on 31 May 2018

I can probably add some context, v2s1 is only required in the push for the reproducer, that's not being specified when using the private registry. I suspect that the private registries we are using (Openshift in @Conan-Kudo's case, Artifactory in mine) are falling back to v2s1 for the push for whatever reason (likely compatibility). The v2s1 is required in the reproducer because the native docker registry supports s2v2 schema (or whatever the newer schema is, working off memory here).

Then corruption (excessive escaping) appears to occur when buildah pulls the image when the v2s1 metadata exists. There doesn't appear to be anything wrong with the image in the registry (I have proven this by inspecting data as it sits in Arifactory after the push). When it's pulled, it gets corrupted.

As I have also shown above, Docker pulls the exact same image, from the exact same registry and interprets it correctly, so the image itself doesn't appear to be corrupted in anyway.

pixdrift picture pixdrift on 1 Jun 2018

So is this a Containers/image issue or a Buildah Issue?
@mtrmac @nalind ?

rhatdan picture rhatdan on 1 Jun 2018

I believe this is happening somewhere in containers/image, but I don't have a root cause.

TomSweeneyRedHat picture TomSweeneyRedHat on 1 Jun 2018

I think it's interesting that podman pull also has issues pulling the container.

pixdrift picture pixdrift on 1 Jun 2018

It may be here if it's doing a conversion from v2 -> v1
https://github.com/projectatomic/buildah/blob/master/vendor/github.com/containers/image/image/docker_schema2.go#L227

Or may just be in the schema1 file
https://github.com/projectatomic/buildah/blob/master/vendor/github.com/containers/image/image/docker_schema1.go

pixdrift picture pixdrift on 1 Jun 2018

@pixdrift, podman having issues pulling furthers the thought that the issue is somewhere in container/image.

TomSweeneyRedHat picture TomSweeneyRedHat on 1 Jun 2018

@runcom PTAL

rhatdan picture rhatdan on 1 Jun 2018 
error pulling image "docker://localhost:5000/alpinev2s1": error retrieving local image after pulling docker://localhost:5000/alpinev2s1: invalid reference format

I guess this is unrelated, just one more corner case with all the string edits happening on references all over the libpod/buildah codebases.

mtrmac picture mtrmac on 2 Jun 2018

I suspect that the private registries we are using (Openshift in @Conan-Kudo's case, Artifactory in mine) are falling back to v2s1 for the push for whatever reason (likely compatibility)

Yes, OpenShift 1.5 has acceptschema2 set to false by default in the integrated registry.

mtrmac picture mtrmac on 2 Jun 2018

(I will set up a test VM to confirm next week, so this may be wrong.). This seems to be an inconsistency between how projectatomic/buildah.containerImageRef.NewImageSource creates the manifests and how containers/image/storage.storageImageDestination.Commit builds layers.

In v2s2 / OCI, the DiffID arrays should _not_ have any layers corresponding to History entries with EmptyLayer set. OTOH in v2s1, such layers (marked Throwaway) physically exist as blobs.

What seems to happen is that when storing a v2s1 image to a storageImageDestination, PutBlob is called for each of the Throwaway layers (the blobs exist, and must be reproduced for a consistent v2s1 image, after all), and later, storageImageDestination.Commit builds containers/storage layers for all layers returned by a c/image/manifest/Schema1.LayerInfos (which again includes the Throwaway layers, because they physically exist in v2s1).

Meanwhile, projectatomic/buildah.containerImageRef.NewImageSource walks the containers/storage “parent” relationship to build a list of layers, and _adds all of them to the generated DiffID array_, even if they were Throwaway in v2s1. This is contrary to the DiffID value definition, and a few lines later, https://github.com/projectatomic/buildah/blob/8ef28c959209b2b2336e1a87032b1ae6ebf740f9/image.go#L411 correctly marks the image as inconsistent.

I don’t know whether, in general, containers/storage should or should not contain separate layers for the v2s1 Throwaway history entries; maybe not, if configs are conceptually attached to images, not layers. Or maybe it should, when NewImageSource always adds an EmptyLayer: false layer for the result of commit, whether or not any changes have happened (which is surprising). Or maybe containers/storage does not care what callers store in it to this extent at all?

Either way, right now it seems it would be simpler to fix the inconsistency in the NewImageSource code building the manifests; it can do an earlier check that len(oimage.History) == len(dimage.History) == len(layers), and then use the History array to determine which layers should not appear in DiffIDs.

Again, all of this may be wrong, I will test this properly next week.

mtrmac picture mtrmac on 2 Jun 2018

Any luck @mtrmac?

pixdrift picture pixdrift on 6 Jun 2018

A reproducer without a registry (using ./t as a temporary directory):

$ skopeo copy --format v2s1 docker://mysql:5.5 dir:t
$ img=$(buildah from dir:t)
$ buildah inspect $img
$ python -mjson.tool t/manifest.json |less
$ buildah commit $img test-dir
error committing container "t-working-container" to "test-dir": error copying layers and metadata: Error initializing source containers-storage:t-working-container: internal error: history lists 12 non-empty layers, but we have 21 layers on disk
mtrmac picture mtrmac on 7 Jun 2018

Interestingly, if you look in the Manifest.history attribute of the pulled images, there are a large number of backslashes.. it's like fields are being escaped 2 or 3 times.. is this making it unparseable?

No; you are looking at a JSON format (buildah inspect) containing a string which is itself JSON; and that JSON may include "Cmd" strings which are escaped again. Each of these layers turns \ into a \\, and the like.

mtrmac picture mtrmac on 7 Jun 2018

Either way, right now it seems it would be simpler to fix the inconsistency in the NewImageSource code building the manifests; it can do an earlier check that len(oimage.History) == len(dimage.History) == len(layers), and then use the History array to determine which layers should not appear in DiffIDs.

762 shows this, and it seems to work for my reproducer.

_I don’t know whether it is the right fix._ @nalind ?

mtrmac picture mtrmac on 7 Jun 2018

@mtrmac does #762 fix the excessive escaping ie \\\"key\\\":\\\"value\\\" as it only appears to affect the history values?

I thought this was causing the problem with parsing the image layer history which then leads to the layer count mismatch.

pixdrift picture pixdrift on 8 Jun 2018

@mtrmac does #762 fix the excessive escaping ie \\"key\\":\\"value\\" as it only appears to affect the history values?

See https://github.com/projectatomic/buildah/issues/684#issuecomment-395435422 ; do you still think there is excessive escaping?

I thought this was causing the problem with parsing the image layer history which then leads to the layer count mismatch.

Considering #762 does not change anything about parsing data at all, only about using the History array, and it does fix at least the simplified dir:t reproducer, I haven’t so far seen anything to suggest _parsing_ is the root cause for anything discussed here. Can you point to a specific code path, or a reproducer that directly points at data formatting/parsing to be an issue?

mtrmac picture mtrmac on 8 Jun 2018

I can't, but in the details I have posted here, the history section is the only section with inconsistent (excessive) escaping. Regardless of parsing, I would expect this to be of concern.

Your comment from above discusses escaping a slash with an additional slash, does triple slash escaped value parse correctly? because that is the situation.

My issue is that I can't identify what is adding these slashes, or where they are coming from. If you can show me where the JSON is parsed for the layers, I am happy to test the triple slash escaped input in isolation to rule it out, but the fact remains that the additional escaping shouldn't be there.

Whether it's linked I can't say, but I would like to rule it out.

pixdrift picture pixdrift on 8 Jun 2018

There’s _nothing excessive_ about escaping the JSON

{"path":"c:\\Windows", "lines":"1\n2\n"}

when represented as a string in _another_ JSON, and getting

{
  "manifest":"{\"path\":\"c:\\\\Windows\", \"lines\":\"1\\n2\\n\"}"
}

Or maybe you are seeing a specific instance where the escaping is expected, but actually _incorrect_ when comparing the raw manifest and the escaped value in the buildah inspect output; if so, please point the specific instance out.

But just seeing four or more backslashes in a row is not, per se, unexpected.

mtrmac picture mtrmac on 8 Jun 2018

This is the section that I think is excessive:
https://github.com/projectatomic/buildah/issues/684#issuecomment-391602453

The difference between the values before they go into the registry, and when they come out. There is no reason for the quotes in the history to be triple escaped, so to me, that is _excessive_.

Before:
"history":[{"v1Compatibility":"{\"architecture\":\"amd64\",\"config\":{\"WorkingDir\":\"/\",\"Labels\":{\"name\":\"el7-apc-base\"}},\"created\":\"2018-05-23T06:51:06.011209458Z\",\"id\":\"43a3840db11c25bd727a88db625d1973dc427ebe84c7b63022d30ff42d422998\",\"os\":\"linux\",\"parent\":\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\"}"},{"v1Compatibility":"{\"id\":\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\",\"created\":\"2018-04-27T07:06:01.013601243Z\",\"container_config\":{\"Cmd\":[\"\"]}}"}],

After:
\"history\":[{\"v1Compatibility\":\"{\\\"architecture\\\":\\\"amd64\\\",\\\"config\\\":{\\\"WorkingDir\\\":\\\"/\\\",\\\"Labels\\\":{\\\"name\\\":\\\"el7-apc-base\\\"}},\\\"created\\\":\\\"2018-05-23T06:51:06.011209458Z\\\",\\\"id\\\":\\\"43a3840db11c25bd727a88db625d1973dc427ebe84c7b63022d30ff42d422998\\\",\\\"os\\\":\\\"linux\\\",\\\"parent\\\":\\\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\\\"}\"},{\"v1Compatibility\":\"{\\\"id\\\":\\\"21d60df4818639d4d79981a654c5b8af72fc4747fe6489197864677d4299a9c0\\\",\\\"created\\\":\\\"2018-04-27T07:06:01.013601243Z\\\",\\\"container_config\\\":{\\\"Cmd\\\":[\\\"\\\"]}}\"}]

pixdrift picture pixdrift on 8 Jun 2018

How exactly do you get the before/after data? "history":[{"v1Compatiblity"… is not a valid stand-alone data structure, it is a _fragment_ of a v2s1 manifest. Of course when you are viewing that manifest _raw_, you see "history", a JSON encoding.

But buildah inspect is _not showing a raw manifest_: it is a JSON of many data fields, one which is Manifest and contains a JSON encoding of a _string_ of the raw manifest.buildah . In particular the snippet at https://github.com/projectatomic/buildah/issues/684#issuecomment-389134724 is _perfectly correct_ in containing a \"history\" field.

And quotes are not being “triple-escaped“: \ turns into \\, " turns into \", so \" turns into \\\".

mtrmac picture mtrmac on 9 Jun 2018

@mtrmac, the output is from buildah inspect before/after the pull from the private registry. I have provided a full reproducer above that uses the docker registry image to reproduce this behaviour (the multiple slashes).

The image commits fine before it goes into the registry, but doesn't if it is pulled from the private registry. On inspection before and after, the history component (which I have posted the excerpt from) is different, which is why I have focused on it.

Regardless of _why_ it is escaped, the point still stands. Why is the metadata returned from inspect different before and after it is pushed and then pulled from a registry? I suspect something is happening in the pull (buildah from) as I have shown the metadata still looks OK when it is in the registry (Artifactory is what I used for this).

-edit-

For a full example output exhibiting this behaviour, see @Conan-Kudo's buildah inspect output above after pull from Openshift registry under the Manifest line, about 5 lines in.
https://github.com/projectatomic/buildah/issues/684#issuecomment-389134724

pixdrift picture pixdrift on 9 Jun 2018

Why is the metadata returned from inspect different before and after it is pushed and then pulled from a registry?

The escaping of "Manifest" is a complete red herring, after a simple buildah from alpine the "Manifest" field is escaped in exactly the same way (although the exact contents of the manifest differ, because the original alpine is v2s2, and anyway buildah adds an extra layer).

There _is_, actually, a bug in parsing the manifest (_not_ corrupting the input): buildah completely ignores the pre-existing v2s1 history (which afterwards does not match) if the docker_version field is empty — and paradoxically, it is empty because buildah itself clears it. So, #776 .

mtrmac picture mtrmac on 11 Jun 2018

@mtrmac ack. Appreciate you digging into it.

pixdrift picture pixdrift on 11 Jun 2018

What needs to be done to move this along so #775 is merged into the vendor'd package upstream (containers/image?). Would be nice to close this out before another version bump.

pixdrift picture pixdrift on 26 Jun 2018

@mtrmac is still digging. From the last I heard he'd solved the mainline issue, but ran into some issues in other places. He's trying to squish all the legs of this nasty bug.

TomSweeneyRedHat picture TomSweeneyRedHat on 27 Jun 2018
👍1

I just ran this with buildah-1.2-1.gitbe87762.fc28.x86_64 on Fedora 28, and I still get this issue.

My OpenShift system is now Origin 3.9 (upgraded through manual export and import of project with containers manually rebuilt), and it still persists.

Conan-Kudo picture Conan-Kudo on 25 Jul 2018

@Conan-Kudo I tried @pixdrift 's example as root:

# Setup a private registry
docker run -d -p 5000:5000 --name registry registry:latest

# Pull a container image from docker.io and push it to private registry
newcontainer=$(buildah from alpine)
buildah commit ${newcontainer} myalpine
buildah push --format v2s1 --tls-verify=false myalpine docker://localhost:5000/alpinev2s1

# Use buildah to pull from the private registry and inspect the result to see broken metadata
brokencontainer=$(buildah from --tls-verify=false docker://localhost:5000/alpinev2s1)
buildah inspect ${brokencontainer} | grep Manifest

# Generate error by attempting to commit image
buildah commit ${brokencontainer} broken-layers

and it all worked fine for me on my F28 machine using the latest Buildah. Can you tell me if this example works in your environment? If so, can you tell me exactly the Buildah commands that yuou're using and the Dockerfiles in play to see the error? I know a lot of it is in the earlier reports, but I just want to make sure that we're using what you're using.

Thanks.

TomSweeneyRedHat picture TomSweeneyRedHat on 26 Jul 2018

@Conan-Kudo Also, did you start with clean local container storage, or continue to use previously imported images? A part of the fix applies when pulling schema1 images, so any pre-existing schema1 images must be at least untagged and re-pulled.

mtrmac picture mtrmac on 26 Jul 2018

@mtrmac @TomSweeneyRedHat Ah, it worked after I purged /var/lib/containers and tried again with buildah v1.2.

Conan-Kudo picture Conan-Kudo on 26 Jul 2018

Phew! Thanks @mtrmac for the suggestions and thanks for the update @Conan-Kudo

TomSweeneyRedHat picture TomSweeneyRedHat on 26 Jul 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

bparees picture bparees  Â·  12Comments
bkhadars picture bkhadars  Â·  11Comments
sourabhsharma487 picture sourabhsharma487  Â·  11Comments
tensor5 picture tensor5  Â·  5Comments
sub-mod picture sub-mod  Â·  4Comments