Argo-cd: Unable to use ssh gitlab connection for on prem gitlab

[balchua]

We wanted to use argocd with our on prem gitlab. Our gitlab is using self signed cert, so decided to use ssh instead.

But i keep getting this error ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

If anyone have an idea on how to get past this one.

[jessesuen]

For private git repos over SSH, you need to either:

1. add the repo using --insecure-ignore-host-key
2. customize the argocd image such that the public ssh key of your gitlab is known to Argo CD. This involves running ssh-keyscan against your private git repo and appending it to /etc/ssh/ssh_known_hosts. You should be able to do this using a volume mount or rebuilding the image.

For private git repos over HTTPS, you need to customize the argocd image servers cert bundle under /etc/ssl/certs. We don't currently have a insecure-tls-skip-verify option for HTTPS at the moment. Adding to /etc/ssl/certs should be possible using a volume mount.

[balchua]

@jessesuen thanks. I will try the insecure-ignore-host-key. With regards to self signed https, i was able to mount the cert to /etc/ssl/certs and add the repo. But started failing when i try to create an app.
I get the same error as this.
https://github.com/argoproj/argo-cd/issues/1171

[jessesuen]

Did you update all services (argocd-server, argocd-repo-server, argocd-application-controller) with the cert bundle? I think we may need all of them to have the change.

If that doesn't work, we can dup this bug to #1171 to figure out why self-signed HTTPS does not work. One user did report adding their cert bundle to /etc/ssl/certs resolved it for them.

[balchua]

Aaa nope. Only added it to argocd-server. Curious to know which component is managing the addition of application? argocd-server or argocd-repo-server? Thanks

[jessesuen]

argocd-server tests the repo before storing it to verify it is valid, so it is needed there. argocd-repo-server is responsible for cloning the repo and rendering the manifests. I don't recall if it is truly necessary in argocd-application-controller.

[balchua]

I tried the insecure-ignore-host-key didn't work i still get ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain error. But mounting the certificate on /etc/ssl/certs/ to both the argocd-server and argocd-repo-server worked.

[mjasion]

I have similar issue with private repository on selfhosted Gitlab :(
Error which I am getting on adding project:

application spec is invalid: InvalidSpecError: No credentials available for source repository and repository is not publicly accessible: error creating SSH agent: "SSH agent requested but SSH_AUTH_SOCK not-specified"

---

My config map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
data:
  repositories: |
    - insecureIgnoreHostKey: true
      sshPrivateKeySecret:
        key: sshPrivateKey
        name: repo-argocd-get-started-1347344391
      url: ssh://[email protected]/marcin.jasion/argocd-get-started.git

And Log from argocd-server pod:

time="2019-04-02T15:38:22Z" level=info msg="received unary call /application.ApplicationService/Create" grpc.method=Create grpc.request.claims="{\"iat\":1554217511,\"iss\":\"argocd\",\"nbf\":1554217511,\"sub\":\"admin\"}" grpc.request.content="application:<TypeMeta:<kind:\"\" apiVersion:\"\" > metadata:<name:\"argo-get-started\" generateName:\"\" namespace:\"\" selfLink:\"\" uid:\"\" resourceVersion:\"\" generation:0 creationTimestamp:<0001-01-01T00:00:00Z> clusterName:\"\" > spec:<source:<repoURL:\"[email protected]:marcin.jasion/argocd-get-started.git\" path:\"/\" targetRevision:\"master\" directory:<recurse:false jsonnet:<> > > destination:<server:\"https://kubernetes.default.svc\" namespace:\"default\" > project:\"default\" syncPolicy:<automated:<prune:true > > > status:<sync:<status:\"\" comparedTo:<source:<repoURL:\"\" path:\"\" targetRevision:\"\" > destination:<server:\"\" namespace:\"\" > > revision:\"\" > health:<status:\"\" message:\"\" > reconciledAt:<0001-01-01T00:00:00Z> observedAt:<0001-01-01T00:00:00Z> sourceType:\"\" > > " grpc.service=application.ApplicationService grpc.start_time="2019-04-02T15:38:22Z" span.kind=server system=grpc
time="2019-04-02T15:38:22Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec is invalid: InvalidSpecError: No credentials available for source repository and repository is not publicly accessible: error creating SSH agent: \"SSH agent requested but SSH_AUTH_SOCK not-specified\"" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2019-04-02T15:38:22Z" grpc.time_ms=10.44 span.kind=server system=grpc

I am using minikube with kubernetes 1.14.0

[balchua]

For now im mounting the certs to the pods and use https instead of ssh. Though i still would want to use ssh access in the future.

[andreyvelich]

I got the same problem with private GitHub repository. I specified public key for the private repo and after run this command:

argocd repo add [email protected]:andreyvelich/test-argocd.git --ssh-private-key-path=<Path to my Private Key>

I got error:

FATA[0000] rpc error: code = Unauthenticated desc = invalid session: signature is invalid 

[andreyvelich]

I got the problem. I need to make argocd login in the command line before run this command.
Right now, I have added my private github repo to the ArgoCD.

[balchua]

I will give it a shot again.

[alexmt]

@balchua . Based on logs from https://github.com/argoproj/argo-cd/issues/1322#issuecomment-477432405 it looks like --ssh-private-key-path was not specified. Can you please confirm that you specified both --ssh-private-key-path and --insecure-ignore-host-key flags?

[balchua]

@alexmt will try this one and let you know. Thanks for looking into this.

[gabrielcalderon]

Is there a way of doing this without having to use argocd cli or modying the default deployment yamls? The CA for my git repo is trusted on the host the node is running, yet the container has 0 awareness of this, it's only getting the regular CA's I dont know from where.

I tested the latest 1.0-RC and this still happens. After initial deployment changing the Deployments in my cluster results in the UI always saying I have the wrong password... And theres no way to change the password in the ArgoCD UI.

[Timoses]

Back on the topic of ssh connection:

I also get:

argocd repo add my_user@myrepo:myport/~my_user/kubernetes.git --insecure-ignore-host-key --ssh-private-key-path ./id_rsa
FATA[0005] rpc error: code = Unknown desc = ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

When I do

kubectl cp ./id_rsa argocd-server-XXX-YYY:/home/argocd/.ssh/id_rsa
kubectl cp ./id_rsa.pub argocd-server-XXX-YYY:/home/argocd/.ssh/id_rsa.pub

kubectl exec -it argocd-server-XXX-YYY sh
    # succeeds (uses ssh key)
    git clone ssh://my_user@myrepo:myport/~my_user/kubernetes.git

    # asks for password ('ssh://' removed from front of repo url)
    git clone my_user@myrepo:myport/~my_user/kubernetes.git

Is there anything else I can investigate or supply to get to the cause?

EDIT:
This hinted me to update to v1.0.0 and it worked ; ).

[alexec]

Next step: create a kustomize application that has a private remote base.

[alexec]

@balchua where are you on this issue?

[balchua]

Sorry guys i haven't updated this issue. Its all good for me now. Im closing this one. Thanks!!

[Kampe]

I see this issue myself when using SSH + declarative setup, what was the fix @balchua?