Argo-cd: Unable to login against ingress: FATA[0007] rpc error: code = Internal desc = transport: received the unexpected content-type "text/plain; charset=utf-8

[papanito]

Describe the bug
After creating an ingress I am unable to login

To Reproduce
Steps to reproduce the behavior:

1. Apply ingress.yaml
2. Go to 'argocd.svc.dev.intra'
3. Enter username and password
   
   
   
   • Password is the one from
     
     kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2
   
   
4. Click sign in

Expected behavior
I am logged in the same way as it happens when I connect to https://localhost:8080 when doing with port forwarding

Additional context

Login via client:

When try to login via client I get

argocd login argocd.svc.dev.intra
Username: admin
Password:
FATA[0007] rpc error: code = Internal desc = transport: received the unexpected content-type "text/plain; charset=utf-8"

Log of argocd-server:

time="2019-04-09T14:11:26Z" level=info msg="k8s application watch event channel closed"
time="2019-04-09T14:11:26Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2019-04-09T14:10:24Z" grpc.time_ms=62751.35 span.kind=server system=grpc
time="2019-04-09T14:11:27Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.claims=null grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2019-04-09T14:11:27Z" span.kind=server system=grpc
time="2019-04-09T14:11:27Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2019-04-09T14:11:27Z" grpc.time_ms=0.632 span.kind=server system=grpc
time="2019-04-09T14:11:28Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = Invalid username or password" grpc.code=Unauthenticated grpc.method=Create grpc.service=session.SessionService grpc.start_time="2019-04-09T14:11:28Z" grpc.time_ms=74.912 span.kind=server system=grpc
time="2019-04-09T14:11:32Z" level=info msg="received streaming call /application.ApplicationService/Watch" grpc.method=Watch grpc.request.claims="{\"iat\":1553886824,\"iss\":\"argocd\",\"nbf\":1553886824,\"sub\":\"admin\"}" grpc.request.content= grpc.service=application.ApplicationService grpc.start_time="2019-04-09T14:11:32Z" span.kind=server system=grpc
time="2019-04-09T14:11:32Z" level=info msg="client watch grpc context closed"
time="2019-04-09T14:11:32Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2019-04-09T14:11:32Z" grpc.time_ms=16.728 span.kind=server system=grpc
time="2019-04-09T14:11:32Z" level=info msg="k8s application watch event channel closed"
time="2019-04-09T14:11:32Z" level=info msg="received streaming call /application.ApplicationService/Watch" grpc.method=Watch grpc.request.claims="{\"iat\":1553886824,\"iss\":\"argocd\",\"nbf\":1553886824,\"sub\":\"admin\"}" grpc.request.content= grpc.service=application.ApplicationService grpc.start_time="2019-04-09T14:11:32Z" span.kind=server system=grpc
```` 

`ingress.yaml`:

```yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: argocd
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: argocd.svc.dev.intra
    http:
      paths:
      - backend:
          serviceName: argocd-server
          servicePort: 443
  tls:
  - hosts:
    - argocd.svc.dev.intra
    secretName: wildcard-ingress-cert

[jessesuen]

The error: received the unexpected content-type "text/plain may mean you are hitting the default ingress backend, which means ssl-passthrough may not be taking effect. Note that nginx.ingress.kubernetes.io/ssl-passthrough requires an additional --enable-ssl-passthrough option to nginx-ingress.

[papanito]

Ok, that was one part of the problem. So after enabling passthrough to nginx and updating the tls.crt and tls.key with our own certificate received the unexpected content-type "text/plain" does not appear anymore. However login is still not possible:

FATA[0028] rpc error: code = Unauthenticated desc = Invalid username or password

however it is the password returned by

kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2

[papanito]

When the pod is redeployed the container name changes, but I assume the password is as initially set... So I'm probably using the wrong password. Is there a way to recover it?

[papanito]

Ok problem solved: https://github.com/argoproj/argo-cd/blob/master/docs/faq.md#i-forgot-the-admin-password-how-do-i-reset-it and login works now

[asim0984]

Hi All,
Looking for help to resolve below issue in my ArgoCD instance while trying to create an application via ArgoCD GUI deployed on argocd namespace.
Error:
W1004 12:32:07.729508 1 listers.go:77] can not retrieve list of objects using index : Index with name namespace does not exist
time="2019-10-04T12:32:07Z" level=error msg="finished unary call with code Unimplemented" error="rpc error: code = Unimplemented desc = the server does not allow this method on the requested resource (post applications.argoproj.io)" grpc.code=Unimplemented grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2019-10-04T12:32:04Z" grpc.time_ms=3584.089 span.kind=server system=grpc

[user@node ~]$ kubectl describe pod argocd-server-7c5b6f9b6b-smc9d -n argocd
Name: argocd-server-7c5b6f9b6b-smc9d
Namespace: argocd
Priority: 0
Node: hostname/IP
Start Time: Fri, 04 Oct 2019 14:40:59 +0200
Labels: app.kubernetes.io/name=argocd-server
pod-template-hash=7c5b6f9b6b
Annotations:
Status: Running
IP: IP
Controlled By: ReplicaSet/argocd-server-7c5b6f9b6b
Containers:
argocd-server:
Container ID: docker://aa2c3144d377c9b8309997b5ff3a241eba0caac9269ed975ca0cb44f66f74dbc
Image: url:9999/argoproj/argocd:v1.2.0
Image ID: docker-pullable://url:9999/argoproj/argocd@sha256:c846cae9188254867e75a5029a777f3b7c09a986dac959c7cceb293446feed38
Ports: 8080/TCP, 8083/TCP
Host Ports: 0/TCP, 0/TCP
Command:
argocd-server
--insecure
--staticassets
/shared/app
--sentinel
argocd-redis-ha-announce-0:26379
--sentinel
argocd-redis-ha-announce-1:26379
--sentinel
argocd-redis-ha-announce-2:26379
--sentinelmaster
argocd
State: Running
Started: Fri, 04 Oct 2019 14:41:02 +0200
Ready: True
Restart Count: 0
Liveness: http-get http://:8080/healthz delay=3s timeout=1s period=30s #success=1 #failure=3
Readiness: http-get http://:8080/healthz delay=3s timeout=1s period=30s #success=1 #failure=3
Environment:
Mounts:
/app/config/ssh from ssh-known-hosts (rw)
/app/config/tls from tls-certs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from argocd-server-token-6nnk4 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
static-files:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
ssh-known-hosts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: argocd-ssh-known-hosts-cm
Optional: false
tls-certs:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: argocd-tls-certs-cm
Optional: false
argocd-server-token-6nnk4:
Type: Secret (a volume populated by a Secret)
SecretName: argocd-server-token-6nnk4
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:

[jonasdkhansen]

The error: received the unexpected content-type "text/plain may mean you are hitting the default ingress backend, which means ssl-passthrough may not be taking effect. Note that nginx.ingress.kubernetes.io/ssl-passthrough requires an additional --enable-ssl-passthrough option to nginx-ingress.

If you use the helm chart for nginx-ingress, then just add this as an extraArg to the controller in values.yaml, and run helm upgrade nginx-ingress.

extraArgs:
    enable-ssl-passthrough: ""

[Sicaine]

I have the same issue with traefik but it got solved by adding --grpc-web:
"Enables gRPC-web protocol. Useful if Argo CD server is behind proxy which does not support HTTP2"

[kamal94]

I ran into this issue even after setting up two separate ingresses for http and grpc as describe in the docs.

argocd login argocd.xxxxx.com --grpc-web was the only thing that worked for me, per @Sicaine

[james-nduka]

🙏 @kamal94

[Belorus]

--grpc-web makes things work but it feels like argoCD has few issues that make app sync and app wait commands hang sometimes without proper gRPC. One such case - when you make sync and wait, but no sync is needed and app becomes healthy very quickly