Cloud-on-k8s: Issues with users other than default "elastic"

[spencergilbert]

Bug Report

What did you do?
Followed quickstart guide, additionally created a superuser via the ui

What did you expect to see?

Built-in users, newly created users functioning

What did you see instead? Under which circumstances?

Users UI only shows No items found, GET _security/user returns { }

Attempting to log into kibana as superuser returns Invalid username or password. Please try again.

After creating user via UI, GET _security/user returns:

{
  "test" : {
    "username" : "test",
    "roles" : [
      "superuser"
    ],
    "full_name" : "",
    "email" : "",
    "metadata" : { },
    "enabled" : true
  }
}

Also unable to curl with the new user:

curl -k -u test:password https://localhost:9200?pretty
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "unable to authenticate user [test] for REST request [/?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "unable to authenticate user [test] for REST request [/?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}

Environment

• ECK version:

eck-operator:1.0.0-beta1

• Kubernetes information:

On premise - Rancher 2.3.1
❯ kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

[charith-elastic]

Thanks for reporting this. It looks like the native realm has been disabled due to the operator enabling the file realm by default. You can update your cluster to enable the native realm by running the following:

cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.4.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
      xpack.security.authc.realms:
        native:
          native1: 
            order: 1
EOF

I have raised #2037 to investigate this further for a future release.

[spencergilbert]

Awesome, thanks for the work around @charith-elastic

[mleklund]

Yes, Thanks for the workaround, been staring at this for hours.