Argo-cd: Unable to login from CLI via ingress route

[rkusmierek]

Describe the bug
When trying to login from CLI via ingress route, I get the following error message:

$argocd login --insecure argocd.mydomain.com
Username: admin
Password: 
FATA[0008] rpc error: code = Internal desc = transport: received the unexpected content-type "text/plain; charset=utf-8"

However I am able to login from CLI using port-forwarding.
Access to UI from web browser (via ingress route and port-forwarding) also works correctly.

To Reproduce
I performed following steps using on-premise (non-cloud) kubernetes cluster:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v1.1.1/manifests/install.yaml

# get admin pass:
kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2

# disable TLS on argocd server
cat <<EOF > argocd-server-insecure-patch.yaml
spec:
  template:
    spec:
      containers:
      - name: argocd-server
        command:
        - argocd-server
        - --staticassets
        - /shared/app
        - --insecure
EOF

kubectl -n argocd patch deployment argocd-server --patch "$(cat ./argocd-server-insecure-patch.yaml)"

# enable ingress:
cat <<EOF > argocd-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: argocd-server-http-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: argocd-server
          servicePort: http
    host: argocd.mydomain.com
  tls:
  - hosts:
    - argocd.mydomain.com
    secretName: argocd-secret

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: argocd-server-grpc-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: argocd-server
          servicePort: https
    host: grpc.argocd.mydomain.com
  tls:
  - hosts:
    - grpc.argocd.mydomain.com
    secretName: argocd-secret
EOF

kubectl apply -f ./argocd-ingress.yaml

# download & install cli:
sudo curl -L -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/download/v1.1.1/argocd-linux-amd64
sudo chmod +x /usr/local/bin/argocd

# login
argocd login --insecure argocd.mydomain.com

Expected behavior
According to official guide, at this point I should be able to login from CLI.

Screenshots
Not applicable

Version

$ argocd version
argocd: v1.1.1+8544bef.dirty
  BuildDate: 2019-07-24T17:30:15Z
  GitCommit: 8544bef56b8bee19e73a71b2d10ccd64156dd72c
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: linux/amd64
argocd-server: v1.1.1+8544bef.dirty
  BuildDate: 2019-07-24T17:31:17Z
  GitCommit: 8544bef56b8bee19e73a71b2d10ccd64156dd72c
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: 0.13.1

Logs

$kubectl logs argocd-server-xxx 
time="2019-07-25T10:57:27Z" level=info msg="Starting configmap/secret informers"
time="2019-07-25T10:57:27Z" level=info msg="Configmap/secret informer synced"
W0725 10:57:27.655924       1 listers.go:77] can not retrieve list of objects using index : Index with name namespace does not exist
time="2019-07-25T10:57:27Z" level=info msg="Starting configmap/secret informers"
time="2019-07-25T10:57:27Z" level=info msg="secrets informer cancelled"
time="2019-07-25T10:57:27Z" level=info msg="configmap informer cancelled"
time="2019-07-25T10:57:27Z" level=info msg="Configmap/secret informer synced"
2019/07/25 10:57:27 [Model:]
2019/07/25 10:57:27 e.e: some(where (p_eft == allow)) && !some(where (p_eft == deny))
2019/07/25 10:57:27 m.m: g(r_sub, p_sub) && keyMatch(r_res, p_res) && keyMatch(r_act, p_act) && keyMatch(r_obj, p_obj)
2019/07/25 10:57:27 g.g: _, _
2019/07/25 10:57:27 r.r: sub, res, act, obj
2019/07/25 10:57:27 p.p: sub, res, act, obj, eft
2019/07/25 10:57:27 [Policy:]
2019/07/25 10:57:27 [p :  sub, res, act, obj, eft :  []]
2019/07/25 10:57:27 [g :  _, _ :  []]
2019/07/25 10:57:27 [Role links for: g]
time="2019-07-25T10:57:27Z" level=info msg="argocd v1.1.1+8544bef.dirty serving on port 8080 (url: , tls: false, namespace: argocd, sso: false)"
time="2019-07-25T10:57:27Z" level=info msg="0xc000498120 subscribed to settings updates"
time="2019-07-25T10:57:27Z" level=info msg="Starting rbac config informer"
time="2019-07-25T10:57:27Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"
time="2019-07-25T10:59:10Z" level=info msg="received unary call /version.VersionService/Version" grpc.method=Version grpc.request.claims=null grpc.request.content= grpc.service=version.VersionService grpc.start_time="2019-07-25T10:59:10Z" span.kind=server system=grpc
time="2019-07-25T10:59:10Z" level=info msg="ks  version" dir= execID=zCD1O
time="2019-07-25T10:59:11Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.service=version.VersionService grpc.start_time="2019-07-25T10:59:10Z" grpc.time_ms=558.858 span.kind=server system=grpc

[jannfis]

I think for the CLI, you should use either the gRPC backend or use --grpc-web switch to argocd

[rkusmierek]

Thanks! Both solutions seem to work as expected:

argocd login --insecure --grpc-web argocd.mydomain.com
argocd login --insecure grpc.argocd.mydomain.com

Which one is better?

[jannfis]

I'd go with the native gRPC option, because a) you can do it (Ingress supports it), b) it's less overhead and c) the API is gRPC native.

Also, I would take care to get rid of having to use the --insecure option :)

But that's just my 2 cents. Use whatever fits best for you.

[rkusmierek]

Thanks for clarification!
(--insecure option was used only to simplify reproduction steps for this issue. I do not intend to use it in production :-) )