Buildx: Custom registry, push error on self-signed cert
'buildx' errors, while 'docker build' succeeds:
cat <<'EOD' > Dockerfile
FROM alpine
RUN touch /test
EOD
docker buildx build \
-t img.service.consul/alpine:test \
--platform=linux/amd64,linux/arm64,linux/arm \
--push \
.
[+] Building 2.1s (12/12) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 65B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [linux/arm/v7 internal] load metadata for docker.io/library/alpine:latest 1.5s
=> [linux/amd64 internal] load metadata for docker.io/library/alpine:latest 1.3s
=> [linux/arm64 internal] load metadata for docker.io/library/alpine:latest 1.3s
=> CACHED [linux/arm64 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> [linux/arm64 2/2] RUN touch /test 0.1s
=> CACHED [linux/amd64 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> [linux/amd64 2/2] RUN touch /test 0.1s
=> CACHED [linux/arm/v7 1/2] FROM docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> => resolve docker.io/library/alpine@sha256:769fddc7cc2f0a1c35abb2f91432e8beecf83916c421420e6a6da9f8975464b6 0.0s
=> [linux/arm/v7 2/2] RUN touch /test 0.2s
=> ERROR exporting to image 0.4s
=> => exporting layers 0.2s
=> => exporting manifest sha256:879ac4adf9493121ff9bb12f8566ed993fa7079c59ae02b516a9287d6de7daea 0.0s
=> => exporting config sha256:42c2e158eb64d21ea6832e4842a3c11c9fd70c89afbf2fd0fbe2f16dd2698453 0.0s
=> => exporting manifest sha256:64b17d691e5c5ab257ad37b622c9ed219e50ea637bf5f7aa25e2f65f0bd0c26d 0.0s
=> => exporting config sha256:14af510e076d60389743c6fc7c99e2777ba56bdad11cbbffacb438e7c68f6321 0.0s
=> => exporting manifest sha256:2e87dbf064ba1c829a9d18525fa38e77baa26b654c04659b0fa3e75d6ea34ea5 0.0s
=> => exporting config sha256:f1b89d61d625bff13e65a679d2bdb1c513289999789ec2e13fe7acefca39adfd 0.0s
=> => exporting manifest list sha256:53f07aa12e20079138de3650629277928313e7bfdc59c3f22c93834fe11ba9f3 0.0s
=> => pushing layers 0.0s
------
> exporting to image:
------
failed to solve: rpc error: code = Unknown desc = failed to do request: Head https://img.service.consul/v2/alpine/blobs/sha256:8dc302c06141b7124ea05ccf2fdde10013ce594c28e5fe980047b0740891e398: x509: certificate signed by unknown authority
x509: certificate signed by unknown authority, but certificate chain is ok.
test:
: |openssl s_client -connect img.service.consul:443 [...] Verify return code: 0 (ok)
docker build + push works also:
docker build \
-t img.service.consul/x86_64/alpine:test \
.
Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM alpine
---> 055936d39205
Step 2/2 : RUN touch /test
---> Using cache
---> 7bd5dcd02d4c
Successfully built 7bd5dcd02d4c
Successfully tagged img.service.consul/x86_64/alpine:test
docker push img.service.consul/x86_64/alpine:test
The push refers to repository [img.service.consul/x86_64/alpine]
b13e8440598c: Pushed
f1b5933fe4b5: Pushed
test: digest: sha256:e9c2e8f188d0bedc6d3c26b39a6a75c36be5b4cbeedb9defc4f3b48953b4ef45 size: 734
buildx imagetools inspect again works:
docker buildx imagetools inspect img.service.consul/x86_64/alpine:test
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1645,
"digest": "sha256:7bd5dcd02d4c340892fe431a40a39badf5695af58b669a33bd21b61159f4ffe5"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2757034,
"digest": "sha256:e7c96db7181be991f19a9fb6975cdbbd73c65f4a2681348e63a141a2192a5f10"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 97,
"digest": "sha256:6e0d312d9ebb1961db61366af2a5a323ad84155db2018457d2c5168c4f86e410"
}
]
}
perhaps related to https://github.com/docker/buildx/issues/57#issuecomment-491244171
tested:
- docker beta-4 + 20190517171028-f4f33ba16d nightly,
- 'buildx' release + current master.
uname -mrov
4.19.44 #1-NixOS SMP Thu May 16 17:41:32 UTC 2019 x86_64 GNU/Linux
minusdelta
All 10 comments
In my case, it wasn't a self-signed cert but a company-wide CA cert. It's very possible that CA cert wasn't present in the environment I was running buildx from though so I could look into that.
chewi
on 26 May 2019
I'm seeing similar with a standalone registry server and locally created CA/certificates. I'm able to run a docker login after adding a /etc/docker/certs.d/<hostname>:5000/ca.crt file, but buildx doesn't appear to use this.
sudo-bmitch
on 17 Jun 2019
I managed to work-around this - by adding my own CA's cert to the generated container, and restarting it.
$ docker ps|grep 'moby/buildkit'
ee110c9e6dfc moby/buildkit:buildx-stable-1 "buildkitd" 7 minutes ago Up 7 minutes buildx_buildkit_mybuilder0
$ docker exec -it ee110c9e6dfc sh
$$ cat >> /etc/ssl/certs/ca-certificates.crt <<'EOF'
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOF
$$ exit
$ docker restart ee110c9e6dfc
It aint pretty, but it works!
klo2k
on 22 Sep 2019
A marginally more robust work-around, but still not pretty (no error checking etc):
BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDER
A possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.
mjan-jansson
on 13 Oct 2019
as I just bumped into this issue, I'd like to add one more option instead of mangling an existing builder container. driver-opt has a image option (for docker-container driver).
I went with the option to have a 2 line Dockerfile that adds my internal CAs to moby/buildkit and use that image when creating the builder.
fopina
on 3 Jun 2020
@fopina can you elaborate on how you did that? I have a 2 line dockerfile where I copy a cert into /usr/local/share/ca-certificates/, and I can run it and exec into it to verify that it's there. I create the image locally, tagging it like testbuilder:latest
When I run docker buildx create --name tester --builder testbuilder:latest, the resulting container that's spun up doesn't have the cert in that folder:
Create and verify the image:
% docker run --rm -ti testbuilder sh
INFO[0000] auto snapshotter: using native
WARN[0000] using host network as the default
INFO[0000] found worker "t3jdqv8o0ov117tdipqjmfnly", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:000f6536acfa org.mobyproject.buildkit.worker.snapshotter:native], platforms=[linux/amd64 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/arm/v7 linux/arm/v6]
WARN[0000] skipping containerd worker, as "/run/containerd/containerd.sock" does not exist
INFO[0000] found 1 workers, default="t3jdqv8o0ov117tdipqjmfnly"
WARN[0000] currently, only the default worker can be used.
INFO[0000] running server on /run/buildkit/buildkitd.sock
# In a different terminal window:
% docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
000f6536acfa testbuilder "buildkitd sh" 18 seconds ago Up 17 seconds jovial_blackwell
% docker exec -it jovial_blackwell sh
/ # cd /usr/local/share/ca-certificates/
/usr/local/share/ca-certificates # ls
certs.pem
using it as a builder?
% docker buildx create --name tester --builder testbuilder:latest
% docker buildx use tester
% docker buildx inspect --bootstrap
[+] Building 1.0s (1/1) FINISHED
=> [internal] booting buildkit 1.0s
=> => pulling image moby/buildkit:buildx-stable-1 0.4s
=> => creating container buildx_buildkit_tester0 0.6s
Name: tester
Driver: docker-container
Nodes:
Name: tester0
Endpoint: unix:///var/run/docker.sock
Status: running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
% docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d9c5e77418fc moby/buildkit:buildx-stable-1 "buildkitd" 9 seconds ago Up 8 seconds buildx_buildkit_tester0
docker exec -ti d9c5e77418fc sh
/ # ls /usr/local/share/ca-certificates/
/ # % < -- nothing here
bdharrington7
on 23 Jan 2021
@bdharrington7 have you tried curl in that image (on the registry endpoint) to validate that they have been properly installed?
I believe you have to run update-ca command unless you already pushed the total file.
fopina
on 23 Jan 2021
I didn't try curl, but I did try using the --push option when I built another image using buildx, and it still failed, because the registry I'm pushing to uses an internal cert. I assumed that the image I specified with the --builder flag wasn't being used, since it didn't have the same information in the one I actually built when I spun up testbuilder:latest manually (the cert I baked into the testbuilder image at /usr/local/share/ca-certificates/ isn't even there)
bdharrington7
on 26 Jan 2021
A marginally more robust work-around, but still not pretty (no error checking etc):
BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDERA possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.
Tried that, but it didn't work.
hemna
on 17 Feb 2021
Considering this issue has been open for over two years, I was just curious if there is an update and/or plans to address?
ravensorb
on 18 Apr 2021
Related issues
securisec
路
9Comments
scyto
路
5Comments
ep4sh
路
5Comments
smurfpandey
路
5Comments
patrickelectric
路
7Comments
Most helpful comment
A marginally more robust work-around, but still not pretty (no error checking etc):
BUILDER=$(sudo docker ps | grep buildkitd | cut -f1 -d' ')
sudo docker cp YOUR-CA.crt $BUILDER:/usr/local/share/ca-certificates/
sudo docker exec $BUILDER update-ca-certificates
sudo docker restart $BUILDER
A possible solution/suggestion would be to allow for something like:
docker secret create a-ca-secret my-ca-file.crt
docker buildx create ---name builder --driver-opt private-ca-cert=/run/secrets/a-ca-secret
...and then have the e.g. the docker-container builder pull in the cert and call update-ca-certificates when it starts.