Skaffold: x509: certificate signed by unknown authority

[jimanvlad]

Expected behavior

build:
  insecureRegistries:
    - artifactory.xx.xx.com

The error in the title shouldn't occur if repo is added to the insecure registries.

Actual behavior

error pushing image: failed to push to destination artifactory.xx.xx.com/data/:: Get https://artifactory.xx.xx.com/v2/: x509: certificate signed by unknown authority

Information

• Skaffold version: 0.27
• Operating system: macOS
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta8
kind: Config
build:
  insecureRegistries:
    - artifactory.xx.xx.com
  tagPolicy:
    gitCommit: {}
  artifacts:
    - image: artifactory.xx.xx.com/data/my-image
      context: src
      kaniko:
        buildContext:
          localDir: {}
  cluster:
    dockerConfig:
      path: /Users/xx/.docker/config_old.json
      secretName: docker-secret
    pullSecretName: kaniko-secret

Steps to reproduce the behavior

1. Run skaffold with the attached yaml

kaniko.flags is also deprecated so there is no way to use kaniko's --skip-tls-verify-registry

[balopat]

Thanks for filing! This should be relatively straightforward to create a field for and passing it to kaniko. PRs are welcome!

[jimanvlad]

Not 100% sure this is a feature request, as https://github.com/GoogleContainerTools/skaffold/pull/1870 should have already allowed insecure registries to be used.

[balopat]

Ahhh, fair enough - I misunderstood, you're right, this is more of a bug / gap for kaniko. We should pass the --insecure-registry list to kaniko. Would you like to submit a PR for this fix?

[jimanvlad]

My go is pretty rusty so won't be able to send a PR.

[tjk]

I took a quick stab at this (https://github.com/GoogleContainerTools/skaffold/pull/1977) but have no idea if this even solves the issue -- running it to see if it works and adding a test would be wonderful! :)

Edit: I've added tests... now to see if this actually solves the issue. 😁

[tjk]

@jimanvlad would you be able to try my patch?

(I'm done changing it just not sure how to set stuff up to verify it fixes the issue.)

[jimanvlad]

Sure, where can I get a macOS binary from?

[tjk]

I just built this but might be issues since I just used go build (gzipped):

skaffold.gz

---

Alternatively, you can try to reproduce what appveyor does maybe? (https://ci.appveyor.com/project/balopat/skaffold/builds/23948844)

Something like:

cd /tmp
git clone -q https://github.com/GoogleContainerTools/skaffold.git
cd skaffold
git fetch -q origin +refs/pull/1977/merge:
git checkout -qf FETCH_HEAD
go build cmd/skaffold/skaffold.go

[jimanvlad]

Hi,

I used your provided binary and added:

  insecureRegistries:
    - artifactory.xx.xx.com
    - artifactory.xx.xx.com:5000

to the build section, but it still failed:
error pushing image: failed to push to destination artifactory.xx.xx.com/data/project:e2e157f-dirty: Get https://artifactory.xx.xx.com/v2/: x509: certificate signed by unknown authority

[tjk]

I need to figure out a way to test this myself but in the meantime, if you could try something like watch -n1 'ps aux | grep kaniko' and copy paste the kaniko command it runs maybe? I imagine for some reason or another the arguments are not being passed to kaniko command... but I'm really out of my element -- I'll try to set up a repro myself if I find time.

[jimanvlad]

Hi,

This is what gets sent:

(base) PC:~ jimanv$ ps aux | grep kaniko
jimanv           96571  10.7  0.1  4419860  21240   ??  S     2:11pm   0:00.07 kubectl exec -i kaniko-8td2r -c kaniko-init-container -n default -- tar -xzf - -C /kaniko/buildcontext
(base) PC:~ jimanv$ ps aux | grep kaniko
jimanv           96584   0.0  0.1  4419020  10604   ??  R     2:12pm   0:00.01 kubectl exec kaniko-8td2r -c kaniko-init-container -n default -- touch /tmp/complete

[tjk]

I will try to set up kaniko this weekend since I've wanted to play with it and will try to get back to you by then to avoid wasting your time. :)

[jimanvlad]

Your help is greatly appreciated and I'm happy to test any further changes :)

[priyawadhwa]

Hey @jimanvlad we are now passing in insecure registries to kaniko as of #2266. Could you try with the latest release and see if it fixes your issue?

[jimanvlad]

Hi, happy to check. Is this in the released binaries or do I have to build from source?

[priyawadhwa]

The latest release should incorporate this change!

[jimanvlad]

Thanks, I am getting a different error not but not sure if it's related or not.

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "artifactory.xx.xx.com:5000/data/peoject:bae7185-dirty": unrecognized HTTP status: 400 Bad Request

This is with the following config:

build:
  artifacts:
    - image: artifactory.xx.xx.com:5000/data/project
      context: src
      kaniko:
        buildContext:
          localDir: {}
  cluster:
    namespace: default
    dockerConfig:
      path: ~/.docker/config.json

And docker login works fine:

>>docker login https://artifactory.xx.xx.com:5000
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /Users/user/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[balopat]

I think this fell through the cracks a little bit - #1977 got closed due to inactivity, and I can't see how https://github.com/GoogleContainerTools/skaffold/pull/2266 pipes insecure registries definition to kaniko - maybe I'm missing something, @priyawadhwa?

Also just a sanity check @jimanvlad - do you have the insecure registries defined in your global config?

[priyawadhwa]

Looks like we need to pass in the following flags to kaniko here if we are pushing to one of the insecure registries listed here:

• --insecure
• --insecure-pull
• --skip-tls-verify
• --skip-tls-verify-pull

Alternatively, we could add these flags to KanikoArtifact and have the user set them manually in the skaffold config. Personally I prefer the first option.

@jimanvlad, would you be interested in opening a PR for this?

[balopat]

I started looking into this stay tuned on #2809

[tejal29]

Thank you @jimanvlad. we are actively working on this issue and share an update soon.

[jimanvlad]

Thanks for your help, I stand ready to test further updates.

[balopat]

Thanks @jimanvlad - @priyawadhwa took over - she is our local kaniko expert.

[priyawadhwa]

Hey @jimanvlad I just merged #2976 -- could you try installing the bleeding edge binary of skaffold and adding the new skipTLS field to your skaffold config:

  artifacts:
    - image: artifactory.xx.xx.com/data/my-image
      context: src
      kaniko:
        buildContext:
          localDir: {}
        skipTLS: true

please let us know if this does or doesn't resolve your issue!

[balchua]

@priyawadhwa im also getting the x509 unknown authority. The strange thing is, the error pops up before the kaniko pod starts. It looks like skaffold is doing some sanity check on the repositories defined in the Dockerfile. Such as cache.
All my base images are from a https self signed certs registry.

[jimanvlad]

Hey @jimanvlad I just merged #2976 -- could you try installing the bleeding edge binary of skaffold and adding the new skipTLS field to your skaffold config:

  artifacts:
    - image: artifactory.xx.xx.com/data/my-image
      context: src
      kaniko:
        buildContext:
          localDir: {}
        skipTLS: true

please let us know if this does or doesn't resolve your issue!

This fix worked for me, thank you!

[priyawadhwa]

Great, thanks @jimanvlad !

[balchua]

@jiminvlad does your base image that you have used in your Dockerfile are in the same registry with self signed certs?
Coz i tried the bleeding edge version and i still get this unknown authority error.

Thanks.

[jimanvlad]

Yep, base image pulled in from a private artifactory registry.

On Sat, 12 Oct 2019 at 00:32, balchua [e-mail] wrote:

@jiminvlad does your base images that you have used in your Dockerfile are
in the same registry with self signed certs?
Thanks.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleContainerTools/skaffold/issues/1961?email_source=notifications&email_token=AARJEZUUOBFUU67KY35CHC3QOEEKPA5CNFSM4HGBXGQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBBOTGQ#issuecomment-541256090,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AARJEZVPATQLY2BOL43CWD3QOEEKPANCNFSM4HGBXGQQ
.

[balchua]

Thanks @jimanvlad. I wonder where could i be wrong.

[chanRoot]

does skaffold-jib provide skipTLS label or other things to deploy app on k8s cluster with private docker registry which use self-signed certificate?.i have successfully built and deployed app on minikube with maven-plugin-jib which config allowInsecureRegistries in pom.xml. And i'm sure that kubectl in k8s can pull my app from harbor

[chanseokoh]

@chanRoot this is unrelated to your question, but I'd like to point out that you probably don't want to set allowInsecureRegistries in Jib if you are using a self-signed certificate to connect to your private Docker registry. allowInsecureRegistries makes Jib proceed even if it fails to verify whether your self-signed registry is secured; Jib will still be able to connect to your registry even if it doesn't use a self-signed certificate.