Argo-cd: Unable to create app from private repo: x509: certificate signed by unknown authority

[papanito]

We are using internal Bitbucket server for the git repos, which we want to add to argo-cd. Bitbucket servers the repo as https with self-signed certificates. I've added the certificates as secrets to the deployment.

I am able to configure the repo under "Settings / Repositories":

However when I get to create an application I get the following error

Unable to load data: Get https://scm.intra/argocd/test/info/refs?service=git-upload-pack: x509: certificate signed by unknown authority

Steps to reproduce

1. Add certificates as secrets

   kubectl apply -f selfsignedcert.yml
   

2. Download install.yaml and mount the certificates something like

   ...
   # add this to all containers
   volumeMounts:
   - mountPath: /etc/ssl/certs/selfsignedcert.pem
    subPath: selfsignedcert.pem
    name: selfsignedcert
   ...
   volumes:
   - name: selfsignedcert
    secret:
      defaultMode: 420
      secretName: selfsignedcert
      items:
      - key: selfsignedcert.pem
        path: selfsignedcert.pem
   

3. Install argo-cd

   kubectl create namespace argocd
   kubectl apply -n argocd -f install.yaml
   

The selfsignedcert.yml looks like

apiVersion: v1
kind: Secret
metadata:
  name: selfsignedcert
  namespace: argocd
data:
  selfsignedcert.pem: |
    xxxxxx...

[jessesuen]

I think for this issue we should do the following:

1. Have an insecure flag for a repository, which will use the InsecureSkipVerify TLS flag when connecting to the repo. This would allow Argo CD to connect without going the extra step of installing certs in the container.

2. When instantiating the go-git client, we should configure it to pull certs from the host. From what I understand, we would call x509.SystemCertPool() to get the host's cert pool. But I'm surprised that this doesn't happen automagically.

Here is the go-git example which customizes the HTTP client:
https://github.com/src-d/go-git/blob/master/_examples/custom_http/main.go#L23

[jessesuen]

But I'm surprised that this doesn't happen automagically.

From the tls.Config object it says:

    // RootCAs defines the set of root certificate authorities
    // that clients use when verifying server certificates.
    // If RootCAs is nil, TLS uses the host's root CA set.
    RootCAs *x509.CertPool

So I expect this to use system certs automatically.

[papanito]

Sorry for my late response. I had a look again and my issue probably occurred cause there was a certificate missing for one of the deployments. Thus, I've used the latest install.yaml and added the certificate to each kind: Deployment. Looks way better now

[papanito]

Well not entirely, have a different error now:

rpc error: 
code = Internal 
desc = Failed to fetch git repo: 'git fetch origin --tags --force' failed: fatal: unable to access 'https://bitbcuket.intra/scm/test/kube/': Problem with the SSL CA cert (path? access rights?)" grpc.code=Internal grpc.method=GetAppDetails grpc.service=repository.RepositoryService grpc.start_time="2019-04-01T06:17:11Z" grpc.time_ms=162.943 span.kind=server system=grpc

I've opened another issue #1344

[gabrielcalderon]

What's the final solution for getting this to work? Modifying the install.yml myself and adding volumes, secrets, etc?

I ran into this issue trying to get argocd to work for my project.

Has there been any thought of doing this the way Rancher custom-ca's for pipelines? They basically added a settings section for pipelines where you can paste your CA-Certificate into a field in the UI and that's it. It just works.

[papanito]

Yes the solution was to modify the install.yaml by mounting the certificates from secrets. Important is to add all certificates required (I was missing one of the subordinates). Also see to map each certificate individually as shown in #1344

[gabrielcalderon]

Does adding my custom-ca to the trusted chain of the cluster running ArgoCD solve this problem? or do i still need to modify the install.yaml?

[waynetjes]

This applies to most people making use of an internal CA.
I've added our root and sub CA certs post installation successfully to solve this. We were having CA trust issues (certificate signed by unknown authority) when attempting to point to our internal Git repos and when trying to "argocd cluster add". We're running OKD.

Steps:

1. Create your root certificate (and sub if applicable) as a secret as described above.
2. Mount certificates to the 'argocd-server' app as described above. I manually edited the deployment config to test.

Take note of subPath - this must be in the volumeMounts directive. Remember to export and source control your config after you've made changes.

Hope this saves someone some time.

[jannfis]

Just for reference, ArgoCD has native support for self-signed TLS certificates and custom CAs on Git and Helm repositories as well as custom SSH host keys: https://argoproj.github.io/argo-cd/user-guide/private-repositories/#self-signed-untrusted-tls-certificates and https://argoproj.github.io/argo-cd/user-guide/private-repositories/#unknown-ssh-hosts

[papanito]

Thanks @jannfis this is good to know.

[Numblesix]

I tried everything as described above in the link. I check the server pod and i cant find the cert added. In the UI i can see that my repositories are in fact again connected but git wont fetch for new stuff