Origin: cluster console keeps redirecting to login screen

[markizm]

After the installation of OKD 3.11, everything seems to work fine except for when trying to access the cluster console page in the UI. For a split second, the console page appears with an error that says "Oh no! Something went wrong. There was an error logging you in. Please log out and try again" It then redirects me back to the login page.

Version

[provide output of the openshift version or oc version command]
oc version
oc v3.11.0+8de5c34-71
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://master-priv.okd-ops.mydomain.net:8443
openshift v3.11.0+8de5c34-71
kubernetes v1.11.0+d4cacc0

Steps To Reproduce

1. Run playbooks/deploy_cluster.yml
2. Installation completes
3. Login to UI: https://master.okd-ops.mydomain.com
4. Navigate to cluster console located in drop down at the top left of the page.

Current Result

Everytime I click on cluster console in the drop down I get redirected back to the login page.
Curling the console pod IP I get a 200 OK, but X-Frame-Options say DENY

curl -Ik https://10.175.166.206:8443
HTTP/1.1 200 OK
Referrer-Policy: strict-origin-when-cross-origin
Set-Cookie: csrf-token=8ljOcWeUjDmO/VKt9kt/Hd3XaTLTLAh38lFlTvdBIJPEYj9GsvkK6OaOyGs3ac9anSZeKfAT4mNa/dOXDwYjag==; Path=/; Secure
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block
Date: Mon, 10 Dec 2018 20:56:43 GMT
Content-Type: text/html; charset=utf-8

Expected Result

Additional Information

[try to run $ oc adm diagnostics (or oadm diagnostics) command if possible]
[if you are reporting issue related to builds, provide build logs with BUILD_LOGLEVEL=5]
[consider attaching output of the $ oc get all -o json -n <namespace> command to the issue]
[visit https://docs.openshift.org/latest/welcome/index.html]

[Note] Running diagnostic: DiagnosticPod
       Description: Create a pod to run diagnostics from the application standpoint

ERROR: [DCli2012 from diagnostic DiagnosticPod@openshift/origin/pkg/oc/cli/admin/diagnostics/diagnostics/client/pod/run_diagnostics_pod.go:208]
       See the errors below in the output from the diagnostic pod:
       [Note] Running diagnostic: PodCheckAuth
              Description: Check that service account credentials authenticate as expected

       Info:  Service account token successfully authenticated to master

       ERROR: [DP1016 from diagnostic PodCheckAuth@openshift/origin/pkg/oc/cli/admin/diagnostics/diagnostics/client/pod/in_pod/auth.go:112]
              DNS resolution for registry address docker-registry.default.svc.cluster.local returned an error; container DNS is likely incorrect. The error was: read udp 10.175.168.88:48943->172.17.0.1:53: read: connection refused

       [Note] Running diagnostic: PodCheckDns
              Description: Check that DNS within a pod works as expected

       ERROR: [DP2003 from diagnostic PodCheckDns@openshift/origin/pkg/oc/cli/admin/diagnostics/diagnostics/client/pod/in_pod/dns.go:98]
              The first /etc/resolv.conf nameserver [172.17.0.1]
              could not resolve kubernetes.default.svc.cluster.local.
              Error: read udp 10.175.168.88:43332->172.17.0.1:53: read: connection refused
              This nameserver points to the master's SkyDNS which is critical for
              resolving cluster names, e.g. for Services.

       WARN:  [DP2015 from diagnostic PodCheckDns@openshift/origin/pkg/oc/cli/admin/diagnostics/diagnostics/client/pod/in_pod/dns.go:150]
              Error querying nameserver [172.17.0.1]:
                read udp 10.175.168.88:60994->172.17.0.1:53: read: connection refused
              This may indicate a problem with DNS.

       [Note] Summary of diagnostics execution (version v3.11.0+06cfa24-67):
       [Note] Warnings seen: 1
       [Note] Errors seen: 2

$ oc logs console-9bfbb7d94-h69w7
2018/12/10 20:30:11 cmd/main: cookies are secure!
2018/12/10 20:30:12 cmd/main: Binding to 0.0.0.0:8443...
2018/12/10 20:30:12 cmd/main: using TLS
2018/12/10 20:30:47 server: authentication failed: unauthenticated
2018/12/10 20:30:47 server: authentication failed: unauthenticated
2018/12/10 20:31:01 auth: unable to verify auth code with issuer: Post https://master.okd-ops.mydomain.net:8443/oauth/token: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

[markizm]

This was a misconfig in a custom playbook that set min-port=1024 in a separate file within /etc/dnsmasq.d. Once the extra file was removed and dnsmasq services were restarted, the console became available.

[skaminskitx]

I had this same issue with the AWS QuickStart. I'll investigate the dnsmasq issue above.

[skaminskitx]

In my situation, I haven't been able to make this work. I have two files in /etc/dnsmasq.d
origin-dns.conf and origin-upstream-dns.conf

In origin-dns.conf, the entry for min-port=1024 was removed and then dnsmasq started on all nodes of the cluster. However, i'm still seeing the same error and unable to view the cluster console. Any other ideas?

[markizm]

@skaminskitx I think min-port=1024 still needs to be defined in origin-dns.conf. In my case I had an another conf file setting the same value which I believe caused a conflict with dns resolution.

Also, the behavior I was seeing was it would log me in and I would actually see the console for a split second before it kicked me back to the login page. Is that what you're seeing as well?

[jomeier]

Any news about that? I'm always getting redirected to the catalog screen if I try to get into the cluster console.

[mslovy]

I got the same issue, is that any solution for this?

[jomeier]

In my case it was a misconfigured DNS server.

[mslovy]

do you mean the dns server in the file origin-upstream-dns.conf ?

[kjw3]

I found that if I logged out of the web console, then went to the cluster console directly and logged in it worked. From then on it allowed me to switch between the consoles.

[jomeier]

You must create a dns entry for the master server (e.g. okd.domain.com) to get access to the openshift console) and another a wildcard dns entry for the OpenShift Router (e.g. *.domain.com) in your DNS server (e.g. an Azure DNS Zone).

[javiervivanco]

Try this

oc delete --all pods -n openshift-console

http://blog.andyserver.com/2018/10/enabling-the-openshift-cluster-console-in-minishift/

https://access.redhat.com/solutions/3676071

[Asgoret]

Nope...On GCP the same issue((( Is there any solution?

[Asgoret]

Question closed. Too security rules on L2.

[saveshnshetty]

@skaminskitx I think min-port=1024 still needs to be defined in origin-dns.conf. In my case I had an another conf file setting the same value which I believe caused a conflict with dns resolution.

Also, the behavior I was seeing was it would log me in and I would actually see the console for a split second before it kicked me back to the login page. Is that what you're seeing as well?