Buildah: Help, buildah push unusable image to registry with 2.7.1

Created on 7 Jun 2020  ·  12Comments  ·  Source: containers/buildah

Problem, Env

Push unusable image to registry 2.7.1 with buildah 1.14.9, buildah 1.14.9 works fine with registry 2.6.2 and 2.5.2.

  • buildah 1.14.9 + registry 2.5.2/2.6.2 (push works fine)
  • buildah 1.14.9 + registry 2.7.1 (push message in buildah is fine, but the image is unusable. The image info can be view via some registry-ui, but the image-tag detail can't be viewed by registry-ui)

My commands: (bh is a short link of buildah)

#login
docker_repo_url=registry-pkg.private.com:5100
docker_repo_user=root
docker_repo_pass=xxx
echo "${docker_repo_pass}" |bh login --tls-verify=false --username=${docker_repo_user} --password-stdin $docker_repo_url

target=registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3
bh bud --pull-always --build-arg TAG=v1.0.2 -t $target .
bh push --tls-verify=false $target

Registry without tls, but with httpdigest auth:

version: '3.3'
services:
  registry:
    container_name: ci-registry
    restart: always
    #2.7.1  18 Jan 2019 · 139 commits to master
    #2.6.2  21 Jul 2017 · 561 commits to master
    #2.5.2  21 Jul 2017 · 777 commits to master
    image: registry:2.6.2
    ports:
      - 5100:5000
    volumes:
      - ./htpasswd:/htpasswd
      - ./data/:/var/lib/registry/
    environment:
      - REGISTRY_AUTH=htpasswd
      - REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm
      - REGISTRY_AUTH_HTPASSWD_PATH=/htpasswd

My build Dockerfile:

FROM registry.cn-shenzhen.aliyuncs.com/infrastlabs/alpine-ext 
USER root
RUN echo 666

Tests

Here is my tests:

Registry 2.6.2

#buildah push
[root@9892b3466f35 t1]# bh version
Version:         1.14.9
Go Version:      go1.14.2
Image Spec:      1.0.1-dev
Runtime Spec:    1.0.1-dev
CNI Spec:        0.4.0
libcni Version:  
image Version:   5.4.3
Git Commit:      
Built:           Thu Jan  1 08:00:00 1970
OS/Arch:         linux/amd64
[root@9892b3466f35 t1]# bh push --tls-verify=false $target
Getting image source signatures
Copying blob b6106ac86a51 done  
Copying blob 5f98e93817f7 done  
Copying blob d59370a73180 done  
Copying blob 9e2ffdf5345f done  
Copying blob d9ff549177a9 done  
Copying blob e5ccc9fe38f6 done  
Copying blob de427f0a0d64 done  
Copying config f17f87c9a5 done  
Writing manifest to image destination
Writing manifest to image destination
Storing signatures

#pull in another pc (pull success)
sam @ deb9-2 in ~ |17:21:49  
$ docker pull registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3
v3: Pulling from ns1/err-cent7-kernel5
e99472525086: Already exists 
78824c9443a7: Already exists 
c938f9c4b37b: Already exists 
815810661f62: Already exists 
d8fb55b406ef: Already exists 
d0072e085162: Already exists 
365a6cac06cc: Pull complete 
Digest: sha256:86a2202fed7b33898907528d836b38f33ac35a6e129692045ea0d2c3de834b17
Status: Downloaded newer image for registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3

Registry 2.7.1

#buildah push (registry reset with version 2.7.1)
[root@9892b3466f35 t1]# bh push --tls-verify=false $target
Getting image source signatures
Copying blob d9ff549177a9 done  
Copying blob 9e2ffdf5345f done  
Copying blob 5f98e93817f7 done  
Copying blob d59370a73180 done  
Copying blob b6106ac86a51 done  
Copying blob e5ccc9fe38f6 done  
Copying blob de427f0a0d64 done  
Copying config f17f87c9a5 done  
Writing manifest to image destination
Storing signatures

#pull in another pc (full fail)
sam @ deb9-2 in ~ |17:21:52  
$ docker rmi registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3
Untagged: registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3
Untagged: registry-pkg.private.com:5100/ns1/err-cent7-kernel5@sha256:86a2202fed7b33898907528d836b38f33ac35a6e129692045ea0d2c3de834b17
Deleted: sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377
Deleted: sha256:90270472a7656bc967aabe9bf8e851ccafb3adae8e8812155e5a651f44e4d943
sam @ deb9-2 in ~ |17:37:27  
$ docker pull registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3
Error response from daemon: manifest for registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3 not found
picture huapox

Most helpful comment

buildah bud --format docker
podman build --format docker

picture rhatdan on 8 Jun 2020
❤2

All 12 comments

And I find that then my built Dockerfile just use FROM xxx, then with nothing in the next line.

Like:

FROM registry.cn-shenzhen.aliyuncs.com/infrastlabs/alpine-ext

Instead of:

FROM registry.cn-shenzhen.aliyuncs.com/infrastlabs/alpine-ext 
USER root
RUN echo 666

In this way, the image can be pushed, and be pulled.

huapox picture huapox on 7 Jun 2020

Full debug with Registry 2.7.1:

[root@ab144007def5 t1]# bh --log-level=debug push --tls-verify=false -D docker://$target
DEBU Destination argument not specified, assuming the same as the source: docker://registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3 
DEBU cached value indicated that overlay is supported 
DEBU cached value indicated that metacopy is not being used 
DEBU cached value indicated that native-diff is usable 
DEBU backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
INFO [graphdriver] using prior storage driver: overlay 
DEBU Loading registries configuration "/etc/containers/registries.conf" 
DEBU registry "registry-pkg.private.com:5100" is not listed in registries configuration "/etc/containers/registries.conf", assuming it's not blocked 
DEBU parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3" 
DEBU pushing image to reference "docker://registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3" is allowed by policy 
DEBU Returning credentials from /run/containers/0/auth.json 
DEBU Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU  No signature storage configuration found for registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3 
DEBU Looking for TLS certificates and private keys in /etc/docker/certs.d/registry-pkg.private.com:5100 
DEBU Using blob info cache at /var/lib/containers/storage/cache/blob-info-cache-v1.boltdb 
DEBU IsRunningImageAllowed for image containers-storage:[overlay@/var/lib/containers/storage]registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3@f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377 
DEBU  Using transport "containers-storage" policy section "" 
DEBU  Requirement 0: allowed                      
DEBU Overall: allowed                             
Getting image source signatures
DEBU Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU ... will first try using the original manifest unmodified 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:e5ccc9fe38f6f8531f7d5d352d71616e73f1ecf2f8bc4719adadd4773ce32a88 
DEBU GET https://registry-pkg.private.com:5100/v2/ 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:d9ff549177a94a413c425ffe14ae1cc0aa254bc9c7df781add08e7d2fba25d27 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:9e2ffdf5345f7b594613ae4dc7c1c05f19bd6c30be414ba5c456e58666499430 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:d59370a73180df9084e85420cf549648f64afd01b139f90f60ba60eed5ddf98f 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:b6106ac86a5197dc3a825a3bcb6832d43086509b2dc7e4349d0f1ae18ae66011 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:5f98e93817f72b75c53c6a0d2630f5d236d8f7c3068a6ddfdddd4c6459999f0c 
DEBU Ping https://registry-pkg.private.com:5100/v2/ err Get "https://registry-pkg.private.com:5100/v2/": http: server gave HTTP response to HTTPS client (&url.Error{Op:"Get", URL:"https://registry-pkg.private.com:5100/v2/", Err:(*errors.errorString)(0xc00037e5a0)}) 
DEBU GET http://registry-pkg.private.com:5100/v2/ 
DEBU Ping http://registry-pkg.private.com:5100/v2/ status 401 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:e5ccc9fe38f6f8531f7d5d352d71616e73f1ecf2f8bc4719adadd4773ce32a88 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:9e2ffdf5345f7b594613ae4dc7c1c05f19bd6c30be414ba5c456e58666499430 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:d59370a73180df9084e85420cf549648f64afd01b139f90f60ba60eed5ddf98f 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:d9ff549177a94a413c425ffe14ae1cc0aa254bc9c7df781add08e7d2fba25d27 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:b6106ac86a5197dc3a825a3bcb6832d43086509b2dc7e4349d0f1ae18ae66011 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:5f98e93817f72b75c53c6a0d2630f5d236d8f7c3068a6ddfdddd4c6459999f0c 
DEBU ... not present                              
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU Trying to reuse cached location sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU ... already exists                           
DEBU ... not present                              
DEBU Skipping blob sha256:e5ccc9fe38f6f8531f7d5d352d71616e73f1ecf2f8bc4719adadd4773ce32a88 (already present): 
DEBU Trying to reuse cached location sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:de427f0a0d64da9331889a02878a18868734e6a92288c5a6ed39ee57ea96c0f7 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:de427f0a0d64da9331889a02878a18868734e6a92288c5a6ed39ee57ea96c0f7 
Copying blob e5ccc9fe38f6 skipped: already exists  
DEBU ... already exists                           
DEBU ... already exists                           
DEBU ... not present                              
DEBU ... already exists                           
Copying blob e5ccc9fe38f6 skipped: already exists  
Copying blob d59370a73180 skipped: already exists  
DEBU Skipping blob sha256:9e2ffdf5345f7b594613ae4dc7c1c05f19bd6c30be414ba5c456e58666499430 (already present): 
DEBU Trying to reuse cached location sha256:365a6cac06cc34579832d72ed72414fffc9e67a51c70a5f1ce036c26d54aea01 in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:365a6cac06cc34579832d72ed72414fffc9e67a51c70a5f1ce036c26d54aea01 
Copying blob e5ccc9fe38f6 skipped: already exists  
Copying blob e5ccc9fe38f6 skipped: already exists  
Copying blob d59370a73180 skipped: already exists  
Copying blob e5ccc9fe38f6 skipped: already exists  
Copying blob d59370a73180 skipped: already exists  
Copying blob 9e2ffdf5345f skipped: already exists  
Copying blob b6106ac86a51 skipped: already exists  
Copying blob d9ff549177a9 skipped: already exists  
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
Copying blob e5ccc9fe38f6 skipped: already exists  
Copying blob e5ccc9fe38f6 skipped: already exists  
Copying blob d59370a73180 skipped: already exists  
Copying blob 9e2ffdf5345f skipped: already exists  
Copying blob b6106ac86a51 skipped: already exists  
Copying blob d9ff549177a9 skipped: already exists  
Copying blob de427f0a0d64 skipped: already exists  
Copying blob 5f98e93817f7 [--------------------------------------] 0.0b / 0.0b
DEBU exporting opaque data as blob "sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377" 
DEBU No compression detected                      
DEBU Using original blob without modification     
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377 
DEBU ... already exists                           
Copying config f17f87c9a5 [======================================] 5.3KiB / 5.3KiB
Writing manifest to image destination
DEBU PUT http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/manifests/v3 
Storing signatures
DEBU pushed image "registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3@sha256:83dacf24500a904420677844186629c3f8bd6d071495fbba377c65119ecfdd10" with digest sha256:83dacf24500a904420677844186629c3f8bd6d071495fbba377c65119ecfdd10 
DEBU Successfully pushed docker://registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3 with digest sha256:83dacf24500a904420677844186629c3f8bd6d071495fbba377c65119ecfdd10
huapox picture huapox on 7 Jun 2020

Debug with Registry 2.6.2:

[root@ab144007def5 t1]# bh --log-level=debug push --tls-verify=false -D docker://$target
DEBU Destination argument not specified, assuming the same as the source: docker://registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3 
DEBU cached value indicated that overlay is supported 
DEBU cached value indicated that metacopy is not being used 
DEBU cached value indicated that native-diff is usable 
DEBU backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
INFO [graphdriver] using prior storage driver: overlay 
DEBU Loading registries configuration "/etc/containers/registries.conf" 
DEBU registry "registry-pkg.private.com:5100" is not listed in registries configuration "/etc/containers/registries.conf", assuming it's not blocked 
DEBU parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3" 
DEBU pushing image to reference "docker://registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3" is allowed by policy 
DEBU Returning credentials from /run/containers/0/auth.json 
DEBU Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU  No signature storage configuration found for registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3 
DEBU Looking for TLS certificates and private keys in /etc/docker/certs.d/registry-pkg.private.com:5100 
DEBU Using blob info cache at /var/lib/containers/storage/cache/blob-info-cache-v1.boltdb 
DEBU IsRunningImageAllowed for image containers-storage:[overlay@/var/lib/containers/storage]registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3@f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377 
DEBU  Using transport "containers-storage" policy section "" 
DEBU  Requirement 0: allowed                      
DEBU Overall: allowed                             
Getting image source signatures
DEBU Manifest has MIME type application/vnd.oci.image.manifest.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+json] 
DEBU ... will first try using the original manifest unmodified 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:e5ccc9fe38f6f8531f7d5d352d71616e73f1ecf2f8bc4719adadd4773ce32a88 
DEBU GET https://registry-pkg.private.com:5100/v2/ 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:d9ff549177a94a413c425ffe14ae1cc0aa254bc9c7df781add08e7d2fba25d27 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:d59370a73180df9084e85420cf549648f64afd01b139f90f60ba60eed5ddf98f 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:5f98e93817f72b75c53c6a0d2630f5d236d8f7c3068a6ddfdddd4c6459999f0c 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:b6106ac86a5197dc3a825a3bcb6832d43086509b2dc7e4349d0f1ae18ae66011 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:9e2ffdf5345f7b594613ae4dc7c1c05f19bd6c30be414ba5c456e58666499430 
DEBU Ping https://registry-pkg.private.com:5100/v2/ err Get "https://registry-pkg.private.com:5100/v2/": http: server gave HTTP response to HTTPS client (&url.Error{Op:"Get", URL:"https://registry-pkg.private.com:5100/v2/", Err:(*errors.errorString)(0xc0002e4000)}) 
DEBU GET http://registry-pkg.private.com:5100/v2/ 
DEBU Ping http://registry-pkg.private.com:5100/v2/ status 401 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:d9ff549177a94a413c425ffe14ae1cc0aa254bc9c7df781add08e7d2fba25d27 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:e5ccc9fe38f6f8531f7d5d352d71616e73f1ecf2f8bc4719adadd4773ce32a88 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:d59370a73180df9084e85420cf549648f64afd01b139f90f60ba60eed5ddf98f 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:5f98e93817f72b75c53c6a0d2630f5d236d8f7c3068a6ddfdddd4c6459999f0c 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:b6106ac86a5197dc3a825a3bcb6832d43086509b2dc7e4349d0f1ae18ae66011 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:9e2ffdf5345f7b594613ae4dc7c1c05f19bd6c30be414ba5c456e58666499430 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 in registry-pkg.private.com:5100/ns1/err3 
DEBU Checking /v2/ns1/err3/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err3/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 in registry-pkg.private.com:5100/ns1/err3 
DEBU Checking /v2/ns1/err3/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err3/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 in registry-pkg.private.com:5100/ns1/err2 
DEBU Checking /v2/ns1/err2/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err2/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 in registry-pkg.private.com:5100/ns1/err3 
DEBU Checking /v2/ns1/err3/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err3/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c in registry-pkg.private.com:5100/ns1/err3 
DEBU Checking /v2/ns1/err3/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err3/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 in registry-pkg.private.com:5100/ns1/err2 
DEBU Checking /v2/ns1/err2/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err2/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 in registry-pkg.private.com:5100/ns1/err3 
DEBU Checking /v2/ns1/err3/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err3/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 in registry-pkg.private.com:5100/ns1/err1 
DEBU Checking /v2/ns1/err1/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err1/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 in registry-pkg.private.com:5100/ns1/err2 
DEBU Checking /v2/ns1/err2/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err2/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c in registry-pkg.private.com:5100/ns1/err2 
DEBU Checking /v2/ns1/err2/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err2/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 in registry-pkg.private.com:5100/ns1/err1 
DEBU Checking /v2/ns1/err1/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err1/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 in registry-pkg.private.com:5100/ns1/alpine-ext 
DEBU Checking /v2/ns1/alpine-ext/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/alpine-ext/blobs/sha256:78824c9443a76d4189154691cc03a5cd6b1dd6db969a0e3dd65905dd108ab699 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd in registry-pkg.private.com:5100/ns1/err3 
DEBU Checking /v2/ns1/err3/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err3/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd in registry-pkg.private.com:5100/ns1/err2 
DEBU Checking /v2/ns1/err2/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err2/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 in registry-pkg.private.com:5100/ns1/err2 
DEBU Checking /v2/ns1/err2/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err2/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c in registry-pkg.private.com:5100/ns1/err1 
DEBU Checking /v2/ns1/err1/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err1/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 in registry-pkg.private.com:5100/ns1/err1 
DEBU Checking /v2/ns1/err1/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err1/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU ... not present                              
DEBU exporting filesystem layer "6c4fee2ad3ba1e137831f25173bc857be851ad3e3557af64c93ba3693421914a" without compression for blob "sha256:d59370a73180df9084e85420cf549648f64afd01b139f90f60ba60eed5ddf98f" 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c in registry-pkg.private.com:5100/ns1/alpine-ext 
DEBU Checking /v2/ns1/alpine-ext/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/alpine-ext/blobs/sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/ZHB7ET5TIJI5SMTNHHOSGIGMZK,upperdir=/var/lib/containers/storage/overlay/6c4fee2ad3ba1e137831f25173bc857be851ad3e3557af64c93ba3693421914a/diff,workdir=/var/lib/containers/storage/overlay/6c4fee2ad3ba1e137831f25173bc857be851ad3e3557af64c93ba3693421914a/work 
DEBU No compression detected                      
DEBU Compressing blob on the fly                  
DEBU Uploading /v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU POST http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 in registry-pkg.private.com:5100/ns1/err1 
DEBU Checking /v2/ns1/err1/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err1/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU ... not present                              
DEBU exporting filesystem layer "d7b3bae9e818405425de655c58e45eeb7025b853eafad1e81e14dbe6ab716908" without compression for blob "sha256:9e2ffdf5345f7b594613ae4dc7c1c05f19bd6c30be414ba5c456e58666499430" 
DEBU overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/UAFUW5D4IEENQAKNAETCBZL2F7:/var/lib/containers/storage/overlay/l/BX7CWC37I4PLGWIMESGU2CORS7:/var/lib/containers/storage/overlay/l/KGXFJSXVWGPCHL2UQ6MLVFFD5T:/var/lib/containers/storage/overlay/l/ZHB7ET5TIJI5SMTNHHOSGIGMZK,upperdir=/var/lib/containers/storage/overlay/d7b3bae9e818405425de655c58e45eeb7025b853eafad1e81e14dbe6ab716908/diff,workdir=/var/lib/containers/storage/overlay/d7b3bae9e818405425de655c58e45eeb7025b853eafad1e81e14dbe6ab716908/work 
DEBU No compression detected                      
DEBU Compressing blob on the fly                  
DEBU Uploading /v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU POST http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 in registry-pkg.private.com:5100/ns1/alpine-ext 
DEBU Checking /v2/ns1/alpine-ext/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/alpine-ext/blobs/sha256:e994725250862727fa2c81afbd7b0a59de989d00950f5c3f73e2e881d1e0b1b3 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 in registry-pkg.private.com:5100/ns1/alpine-ext 
DEBU Checking /v2/ns1/alpine-ext/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/alpine-ext/blobs/sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU ... not present                              
DEBU exporting filesystem layer "d9ff549177a94a413c425ffe14ae1cc0aa254bc9c7df781add08e7d2fba25d27" without compression for blob "sha256:d9ff549177a94a413c425ffe14ae1cc0aa254bc9c7df781add08e7d2fba25d27" 
DEBU overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/d9ff549177a94a413c425ffe14ae1cc0aa254bc9c7df781add08e7d2fba25d27/empty,upperdir=/var/lib/containers/storage/overlay/d9ff549177a94a413c425ffe14ae1cc0aa254bc9c7df781add08e7d2fba25d27/diff,workdir=/var/lib/containers/storage/overlay/d9ff549177a94a413c425ffe14ae1cc0aa254bc9c7df781add08e7d2fba25d27/work 
Copying blob d59370a73180 [--------------------------------------] 8.0b / 6.7MiB
Copying blob 9e2ffdf5345f [--------------------------------------] 8.0b / 4.5KiB
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd in registry-pkg.private.com:5100/ns1/err1 
DEBU Checking /v2/ns1/err1/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err1/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU No compression detected                      
DEBU Compressing blob on the fly                  
DEBU Uploading /v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU POST http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU ... not present                              
DEBU exporting filesystem layer "f5cb8613a589c13b19ed0a6ee4151947575ebda1371f785051a3d3a28e2a821c" without compression for blob "sha256:5f98e93817f72b75c53c6a0d2630f5d236d8f7c3068a6ddfdddd4c6459999f0c" 
DEBU overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/BX7CWC37I4PLGWIMESGU2CORS7:/var/lib/containers/storage/overlay/l/KGXFJSXVWGPCHL2UQ6MLVFFD5T:/var/lib/containers/storage/overlay/l/ZHB7ET5TIJI5SMTNHHOSGIGMZK,upperdir=/var/lib/containers/storage/overlay/f5cb8613a589c13b19ed0a6ee4151947575ebda1371f785051a3d3a28e2a821c/diff,workdir=/var/lib/containers/storage/overlay/f5cb8613a589c13b19ed0a6ee4151947575ebda1371f785051a3d3a28e2a821c/work 
DEBU No compression detected                      
DEBU Compressing blob on the fly                  
DEBU Uploading /v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU POST http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 in registry-pkg.private.com:5100/ns1/alpine-ext 
DEBU Checking /v2/ns1/alpine-ext/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/alpine-ext/blobs/sha256:c938f9c4b37b163dd7668723114e3a4af64c72dd4dd523689bb46473754d2f94 
DEBU PATCH http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/b30297bf-7cbf-4071-9a20-d36a27189d63?_state=P8mPXul9-lJl4bSoHU1SjiGSEVAVHdjW85kfy6ImpeJ7Ik5hbWUiOiJuczEvZXJyLWNlbnQ3LWtlcm5lbDUiLCJVVUlEIjoiYjMwMjk3YmYtN2NiZi00MDcxLTlhMjAtZDM2YTI3MTg5ZDYzIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTA2LTA3VDEwOjA4OjQxLjYyNDIzNzQ3M1oifQ%3D%3D 
DEBU PATCH http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/52c875c5-b83b-4ec9-ad70-2dc4fc332678?_state=XzymMvzq9tyYcBVCopying blob d59370a73180 [--------------------------------------] 8.0b / 6.7MiB
Copying blob 9e2ffdf5345f done  
Copying blob d9ff549177a9 [--------------------------------------] 8.0b / 4.5MiB
Copying blob 5f98e93817f7 done  
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd in registry-pkg.private.com:5100/ns1/alpine-ext 
DEBU Checking /v2/ns1/alpine-ext/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/alpine-ext/blobs/sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU ... not present                              
DEBU exporting filesystem layer "f9e06d03736353e627b49dcbccb106263f9d87b5c6fe285e89a7cda7c6743b48" without compression for blob "sha256:b6106ac86a5197dc3a825a3bcb6832d43086509b2dc7e4349d0f1ae18ae66011" 
DEBU overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/KGXFJSXVWGPCHL2UQ6MLVFFD5T:/var/lib/containers/storage/overlay/l/ZHB7ET5TIJI5SMTNHHOSGIGMZK,upperdir=/var/lib/containers/storage/overlay/f9e06d03736353e627b49dcbccb106263f9d87b5c6fe285e89a7cda7c6743b48/diff,workdir=/var/lib/containers/storage/overlay/f9e06d03736353e627b49dcbccb106263f9d87b5c6fe285e89a7cda7c6743b48/work 
DEBU No compression detected                      
DEBU Compressing blob on the fly                  
DEBU Uploading /v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU POST http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU PATCH http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/4bd3d46f-e96e-4be5-a2cb-aeda259a6a56?_state=6kOqAznWlQh1geUh7xwX0uToGG0hjVVSgnUkDTJNvRp7Ik5hbWUiOiJuczEvZXJyLWNlbnQ3LWtlcm5lbDUiLCJVVUlEIjoiNGJkM2Q0NmYtZTk2ZS00YmU1LWEyY2ItYWVkYTI1OWE2YTU2IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTA2LTA3VDEwOjA4OjQxLjY2NDIxOTE0OFoifQ%3D%3D 
DEBU PATCH http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/f278f5e3-d701-4236-822c-d6c54f3e2647?_state=RxqF8j5VoEo8bIab7llp_1dUhb7ZoOeYbOWqRsVQeIR7Ik5hbWUiOiJuczEvZXJyLWNlbnQ3LWtlcm5lbDUiLCJVVUlEIjoiZjI3OGY1ZTMtZDcwMS00MjM2LTgyMmMtZDZjNTRmM2UyNjQ3IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTA2LTA3VDEwOjA4OjQxLjY1NjQ4Njk2NVoifQ%3D%3D 
DEBU ... not present                              
DEBU exporting filesystem layer "6a2ae71d765fabfad4636ac9cc4265f2e514cbfeb06b818011eae5218da9c9de" without compression for blob "sha256:e5ccc9fe38f6f8531f7d5d352d71616e73f1ecf2f8bc4719adadd4773ce32a88" 
DEBU PUT http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/b30297bf-7cbf-4071-9a20-d36a27189d63?_state=kNjbSvCylDxi4VF_VQWXNRKo21ttuHjjQ_tXilME5hx7Ik5hbWUiOiJuczEvZXJyLWNlbnQ3LWtlcm5lbDUiLCJVVUlEIjoiYjMwMjk3YmYtN2NiZi00MDcxLTlhMjAtZDM2YTI3MTg5ZDYzIiwiT2Zmc2V0IjoxNTQ3LCJTdGFydGVkQXQiOiIyMDIwLTA2LTA3VDEwOjA4OjQxWiJ9&digest=sha256%3Ad8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c 
DEBU PATCH http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/0ce91eb5-7d46-455a-94b4-b5b5c8a1f67a?_state=xkfFp9fzFYJKOjhi2t4ayrItcED4D_XpEUuU2rdslUt7Ik5hbWUiOiJuczEvZXJyLWNlbnQ3LWtlcm5lbDUiLCJVVUlEIjoiMGNlOTFlYjUtN2Q0Ni00NTVhLTk0YjQtYjViNWM4YTFmNjdhIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTA2LTA3VDEwOjA4OjQxLjcwMTExNTY0NloifQ%3D%3D 
DEBU PUT http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/52c875c5-b83b-4ec9-ad70-2dc4fc332678?_state=JnzncgHO2Kc7iCgjjPdILJPRxO0JzEMh3aHlyYnRYsh7Ik5hbWUiOiJuczEvZXJyLWNlbnQ3LWtlcm5lbDUiLCJVVUlEIjoiNTJjODc1YzUtYjgzYi00ZWM5LWFkNzAtMmRjNGZjMzMyNjc4IiwiT2Zmc2V0IjoxNDI3LCJTdGFydGVkQXQiOiIyMDIwLTA2LTA3VDEwOjA4OjQxWiJ9&digest=sha256%3A815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 
DEBU overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/YY4YRDYDFJC6O2RTPWELH43V6Z:/var/lib/containers/storage/overlay/l/UAFUW5D4IEENQAKNAETCBZL2F7:/var/lib/containers/storage/overlay/l/BX7CWC37I4PLGWIMESGU2CORS7:/var/lib/containers/storage/overlay/l/KGXFJSXVWGPCHL2UQ6MLVFFD5T:/var/lib/containers/storage/overlay/l/ZHB7ET5TIJI5SMTNHHOSGIGMZK,upperdir=/var/lib/containers/storage/overlay/6a2ae71d765fabfad4636ac9cc4265f2e514cbfeb06b818011eae5218da9c9de/diff,workdir=/var/lib/containers/storage/overlay/6a2ae71d765fabfad4636ac9cc4265f2e514cbfeb06b818011eae5218da9c9de/work 
DEBU Upload of layer sha256:d8fb55b406efa5bf21f59a1e07715157b704be4184dacbf2dcb2cd08388ae15c complete 
DEBU Upload of layer sha256:815810661f62d36dd28aed96b452e217c19c6c9802152b317033e4972d68c1b1 complete 
Copying blob d59370a73180 [=====================>----------------] 3.8MiB / 6.7MiB
Copying blob 9e2ffdf5345f done  
Copying blob d9ff549177a9 [============>-------------------------] 1.5MiB / 4.5MiB
Copying blob d59370a73180 [======================>---------------] 4.1MiB / 6.7MiB
Copying blob 9e2ffdf5345f done  
Copying blob d9ff549177a9 [============>-------------------------] 1.5MiB / 4.5MiB
Copying blob 5f98e93817f7 done  
Copying blob b6106ac86a51 done  
Copying blob e5ccc9fe38f6 [--------------------------------------] 8.0b / 64.0MiB
DEBU PATCH http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/c79c2da3-a412-403c-9a7d-febdf3af72a2?_state=Q55YaPZd_0gTP5s_ZHtTAwYrjjlVDkAAtalSkkgruYV7Ik5hbWUiOiJuczEvZXJyLWNlbnQ3LWtlcm5lbDUiLCJVVUlEIjoiYzc5YzJkYTMtYTQxMi00MDNjLTlhN2QtZmViZGYzYWY3MmEyIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTA2LTA3VDEwOjA4OjQxLjc4MTE3MTEzMloifQ%3D%3D 
DEBU ... not present                              
DEBU Trying to reuse cached location sha256:365a6cac06cc34579832d72ed72414fffc9e67a51c70a5f1ce036c26d54aea01 in registry-pkg.private.com:5100/ns1/err-cent7-kernel5 
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:365a6cac06cc34579832d72ed72414fffc9e67a51c70a5f1ce036c26d54aea01 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:365a6cac06cc34579832d72ed72414fffc9e67a51c70a5f1ce036c26d54aea01 
DEBU ... not present                              
DEBU exporting filesystem layer "c751c68996918bb67cec61f816c5d984567b67ba315f55c6d204c49bff40e577" without compression for blob "sha256:de427f0a0d64da9331889a02878a18868734e6a92288c5a6ed39ee57ea96c0f7" 
DEBU overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/JVRBZDLR3UV3WWCPP7O6VEYDP2:/var/lib/containers/storage/overlay/l/YY4YRDYDFJC6O2RTPWELH43V6Z:/var/lib/containers/storage/overlay/l/UAFUW5D4IEENQAKNAETCBZL2F7:/var/lib/containers/storage/overlay/l/BX7CWC37I4PLGWIMESGU2CORS7:/var/lib/containers/storage/overlay/l/KGXFJSXVWGPCHL2UQ6MLVFFD5T:/var/lib/containers/storage/overlay/l/ZHB7ET5TIJI5SMTNHHOSGIGMZK,upperdir=/var/Copying blob d59370a73180 done  
Copying blob 9e2ffdf5345f done  
Copying blob d9ff549177a9 [============================>---------] 3.3MiB / 4.5MiB
Copying blob 5f98e93817f7 done  
Copying blob b6106ac86a51 done  
Copying blob e5ccc9fe38f6 [>-------------------------------------] 1.8MiB / 64.0MiB
Copying blob de427f0a0d64 [--------------------------------------] 8.0b / 3.0KiB
Copying blob d59370a73180 done  
Copying blob 9e2ffdf5345f done  
Copying blob d9ff549177a9 done  
Copying blob 5f98e93817f7 done  
Copying blob b6106ac86a51 done  
Copying blob e5ccc9fe38f6 [=>------------------------------------] 3.9MiB / 64.0MiB
Copying blob d59370a73180 done  
Copying blob 9e2ffdf5345f done  
Copying blob d9ff549177a9 done  
Copying blob 5f98e93817f7 done  
Copying blob b6106ac86a51 done  
Copying blob e5ccc9fe38f6 [===>----------------------------------] 6.3MiB / 64.0MiB
Copying blob d59370a73180 done  
Copying blob 9e2ffdf5345f done  
Copying blob d9ff549177a9 done  
Copying blob 5f98e93817f7 done  
Copying blob b6106ac86a51 done  
Copying blob e5ccc9fe38f6 done  
Copying blob de427f0a0d64 done  
DEBU PUT http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/c79c2da3-a412-403c-9a7d-febdf3af72a2?_state=r7GCZNkUJmvTeXi0G0uwZDRQ1WA6Jn4ZzHcW0xvTw_Z7Ik5hbWUiOiJuczEvZXJyLWNlbnQ3LWtlcm5lbDUiLCJVVUlEIjoiYzc5YzJkYTMtYTQxMi00MDNjLTlhN2QtZmViZGYzYWY3MmEyIiwiT2Zmc2V0IjoyMzYxNzczNSwiU3RhcnRlZEF0IjoiMjAyMC0wNi0wN1QxMDowODo0MVoifQ%3D%3D&digest=sha256%3Ad0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd 
DEBU Upload of layer sha256:d0072e085162f72f4994ddfbfec88875445641492815d6862e669bcfa0f365cd complete 
DEBU exporting opaque data as blob "sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377" 
DEBU No compression detected                      
DEBU Using original blob without modification     
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377 
DEBU ... not present                              
DEBU Uploading /v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU POST http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/ 
DEBU PATCH http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/12f44544-23b8-4279-b44b-88fa7f037d88?_state=yPu50X0oFF7T6_VBnyez8-WdP2EVJB010wkhgEYvVml7Ik5hbWUiOiJuczEvZXJyLWNlbnQ3LWtlcm5lbDUiLCJVVUlEIjoiMTJmNDQ1NDQtMjNiOC00Mjc5LWI0NGItODhmYTdmMDM3ZDg4IiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTA2LTA3VDEwOjA4OjQyLjcyMjI0MDY5MVoifQ%3D%3D 
Copying config f17f87c9a5 done  
DEBU PUT http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/uploads/12f44544-23b8-4279-b44b-88fa7f037d88?_state=tD8TxOq3kFJ7kJJFD9Zs7uUTPpxfQVt8cnv6sgeDfml7Ik5hbWUiOiJuczEvZXJyLWNlbnQ3LWtlcm5lbDUiLCJVVUlEIjoiMTJmNDQ1NDQtMjNiOC00Mjc5LWI0NGItODhmYTdmMDM3ZDg4IiwiT2Zmc2V0Ijo1NDY4LCJTdGFydGVkQXQiOiIyMDIwLTA2LTA3VDEwOjA4OjQyWiJ9&digest=sha256%3Af17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377 
DEBU Upload of layer sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377 complete 
Writing manifest to image destination
DEBU PUT http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/manifests/v3 
DEBU Writing manifest using preferred type application/vnd.oci.image.manifest.v1+json failed: Error writing manifest: Error uploading manifest v3 to registry-pkg.private.com:5100/ns1/err-cent7-kernel5: manifest invalid: manifest invalid 
DEBU Trying to use manifest type application/vnd.docker.distribution.manifest.v2+json… 
DEBU exporting opaque data as blob "sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377" 
DEBU No compression detected                      
DEBU Using original blob without modification     
DEBU Checking /v2/ns1/err-cent7-kernel5/blobs/sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377 
DEBU HEAD http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/blobs/sha256:f17f87c9a5bc4369e00f56d0273b7a6fe7448c53de2643fb0d1926e4125ed377 
DEBU ... already exists                           
Writing manifest to image destination
DEBU PUT http://registry-pkg.private.com:5100/v2/ns1/err-cent7-kernel5/manifests/v3 
Storing signatures
DEBU pushed image "registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3@sha256:86a2202fed7b33898907528d836b38f33ac35a6e129692045ea0d2c3de834b17" with digest sha256:86a2202fed7b33898907528d836b38f33ac35a6e129692045ea0d2c3de834b17 
DEBU Successfully pushed docker://registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v3 with digest sha256:86a2202fed7b33898907528d836b38f33ac35a6e129692045ea0d2c3de834b17
huapox picture huapox on 7 Jun 2020

Seems that blobs not pushed to Registry? when it's version is 2.7.1

Same situation with Harbor v1.4.0/v1.9.3-c80751b8, which based on Registry 2.7.1 too

huapox picture huapox on 7 Jun 2020

@mtrmac @vrothberg PTAL

rhatdan picture rhatdan on 7 Jun 2020

Can you try removing the local image and pull it again but with Buildah or Podman instead of Docker?

vrothberg picture vrothberg on 8 Jun 2020

docker/distribution 2.7.1, unlike earlier versions, supports the OCI image format, so the image is created in that format instead of Docker schema2. It seems that your Docker version does not support that.

There is a Buildah option to produce Docker-formatted images.

mtrmac picture mtrmac on 8 Jun 2020
👍1

buildah bud --format docker
podman build --format docker

rhatdan picture rhatdan on 8 Jun 2020
❤2

buildah bud --format docker
podman build --format docker

Thx, I'll have a test with buildah bud --format docker

huapox picture huapox on 9 Jun 2020

buildah bud --format docker
podman build --format docker

> Thx, I'll have a test with buildah bud --format docker

Great! buildah bud --format docker works:

  • build, push
[root@c16a64839347 t1]# cat Dockerfile 
FROM registry.cn-shenzhen.aliyuncs.com/infrastlabs/alpine-ext
USER root
RUN echo 666

[root@c16a64839347 t1]# bh bud --format=docker --pull-always --build-arg TAG=v1.0.2 -t $target .
STEP 1: FROM registry.cn-shenzhen.aliyuncs.com/infrastlabs/alpine-ext
Getting image source signatures
Copying blob 593bd703b55e skipped: already exists  
Copying blob d6b32986f55d skipped: already exists  
Copying blob ff6506d76a48 skipped: already exists  
Copying blob c87736221ed0 skipped: already exists  
Copying blob efc028be2329 skipped: already exists  
Copying blob 79b6946c99fe [--------------------------------------] 0.0b / 0.0b
Copying config 24c0cd0a29 done  
Writing manifest to image destination
Storing signatures
STEP 2: USER root
STEP 3: RUN echo 666
666
STEP 4: COMMIT registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v31
Getting image source signatures
Copying blob d9ff549177a9 skipped: already exists  
Copying blob d59370a73180 skipped: already exists  
Copying blob b6106ac86a51 skipped: already exists  
Copying blob 5f98e93817f7 skipped: already exists  
Copying blob 9e2ffdf5345f skipped: already exists  
Copying blob e5ccc9fe38f6 skipped: already exists  
Copying blob 4049dcac2e8d done  
Copying config 0df5e64e46 done  
Writing manifest to image destination
Storing signatures
--> 0df5e64e460
[Warning] one or more build args were not consumed: [TAG]
0df5e64e46018bab2937e2b8c34c20d21ca9927a2b1d8211ab97d62312fc0025

[root@c16a64839347 t1]# bh push --tls-verify=false $target
Getting image source signatures
Copying blob 9e2ffdf5345f done  
Copying blob d59370a73180 done  
Copying blob 5f98e93817f7 done  
Copying blob d9ff549177a9 done  
Copying blob b6106ac86a51 done  
Copying blob e5ccc9fe38f6 done  
Copying blob 4049dcac2e8d done  
Copying config 0df5e64e46 done  
Writing manifest to image destination
Storing signatures
  • pull
sam @ deb9-2 in ~ |18:20:55  
$ docker pull registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v31
v31: Pulling from ns1/err-cent7-kernel5
e99472525086: Already exists 
78824c9443a7: Already exists 
c938f9c4b37b: Already exists 
815810661f62: Already exists 
d8fb55b406ef: Already exists 
d0072e085162: Already exists 
e9c8a7f5d885: Pull complete 
Digest: sha256:cc2081b94ccd94f3947acfc4b56a4e4b55dd7d878cfead3fed842949504961c6
Status: Downloaded newer image for registry-pkg.private.com:5100/ns1/err-cent7-kernel5:v31
huapox picture huapox on 9 Jun 2020

docker/distribution 2.7.1, unlike earlier versions, supports the OCI image format, so the image is created in that format instead of Docker schema2. It seems that your Docker version does not support that.

There is a Buildah option to produce Docker-formatted images.

My docker's version that run the registry:

  • vm192, which runs Registry
[root@master-192 registry-ui]# docker version
Client: Docker Engine - Community
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:33:34 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       481bc77
  Built:            Sat May  4 02:41:08 2019
  OS/Arch:          linux/amd64
  Experimental:     false
  • debian9, which to pull the images that generated and pushed by buildah for test.
sam @ deb9-2 in ~ |18:42:11  
$ docker version
Client:
 Version:   17.12.1-ce
 API version:   1.35
 Go version:    go1.9.4
 Git commit:    7390fc6
 Built: Tue Feb 27 22:17:21 2018
 OS/Arch:   linux/amd64

Server:
 Engine:
  Version:  17.12.1-ce
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   7390fc6
  Built:    Tue Feb 27 22:15:56 2018
  OS/Arch:  linux/amd64
  Experimental: false

Thanks of all. The problem be resovled,

I'm still confused about OCI and the Docker schema2, with my test got these:

buildah ---------------  registry 2.6.2 ------ Docker schema2 Format.   ---- pull OK;

buildah ---------------   registry 2.7.1 -----  OCI  Format.  -------  pull fail; registryUI got info fail;

buildah(--format docker) ---   registry 2.7.1 -----  Docker schema2 Format.  ------ pull OK;
huapox picture huapox on 9 Jun 2020

Might be time to update.

rhatdan picture rhatdan on 9 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

azatsarynnyy picture azatsarynnyy  ·  4Comments
hendrikhalkow picture hendrikhalkow  ·  11Comments
jawnsy picture jawnsy  ·  6Comments
sourabhsharma487 picture sourabhsharma487  ·  11Comments
QincaiLiu picture QincaiLiu  ·  9Comments