Cloud-on-k8s: Support different cluster topologies

[sebgl]

Create various topology for Elasticsearch / Kibana

• I can create master-only nodes
• I can create data-only nodes
• I can create ml nodes
• I can create ingest nodes
• I can create coordinator nodes

[sebgl]

A proposal on the CRD format:

Add an array under nodes, one item per "node type".
A node type represents the node topology (master/data/ingest, capacity, disk, etc.) and the number of nodes to create for that topology.

apiVersion: deployments.k8s.elastic.co/v1alpha1
kind: Stack
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: stack-sample
spec:
  version: "6.4.2"
  elasticsearch:
    setVmMaxMapCount: true
    nodes:
      # master only
      - master: true
        data: false
        ingest: false
        ram: 1GB
        disk: 10GB
        count: 1
      # data only
      - master: false
        data: true
        ingest: false
        ram: 8GB
        disk: 1TB
        count: 3  # create 3 nodes like that
      # ingest only
      - master: false
        ingest: true
        data: false
        count: 3
      # coordinator
      - master: false
        ingest: false
        data: false
        count: 1
      # default to master+data+ingest
      - count: 1

Default values:

count: 1
ram: 1GB
master: true
data: true
ingest: true
disk: 10GB

[marclop]

Proposal looks good, the only thing I'd think about is keeping a consistency between the units that we've got in the stack and the ones we've got in our CRD. See https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory.

Maybe in the future it'd be worth declaring other CRDs with the ES _DNT_ configs?

[sebgl]

Oh yes, better use the k8s standard units 👍

Maybe in the future it'd be worth declaring other CRDs with the ES DNT configs?

You mean we have a CRD dedicated to represent a node type, and we reference this node type in the stack CRD?

# dnt.yaml
name: my-node-type
data: true
master: true

# stack.yaml
nodes:
    - dnt: my-node-type
      count: 3

I guess it's nice when you want to manage many clusters. And it's too complex when you want to manage a few clusters only.

[marclop]

Yeah I think it's a nice way to reduce your declarations in your stack and make them easier to read. And it's like what you say, if you've got multiple stacks you want to kinda have this centrally managed definition. But it can be done later on

[sebgl]

I think if we introduce that later on, we can keep backward compatibility.
Ie. if a "DNT" reference is specified, use it, otherwise, use the given settings.
Anyway I tend to agree we don't need it for now.

[nkvoll]

Something like this?

apiVersion: deployments.k8s.elastic.co/v1alpha1
kind: Stack
metadata:
  labels:
    controller-tools.k8s.io: "1.0"
  name: stack-sample
spec:
  version: "6.4.2"
  elasticsearch:
    setVmMaxMapCount: true
    nodes:
      # only one of flat/toplogies allowed
      # flat caters to the simplest of cases
      flat:
        replicas: 1
      # topologies caters to the DNT-style, more advanced setups
      topologies:
        # master only
        - nodeTypes:
            master: true
            data: false
            ingest: false
          resources:
            limits:
              mem: 1Gi
              disk: 1GB
              cpu: 800m
          replicas: 1
        # data only
        - nodeTypes:
            master: false
            data: true
            ingest: false
          resources:
            limits:
              mem: 8Gi
              disk: 1TB
              cpu: 800m
          replicas: 3     

I nested resources.limits slightly (matching the pod api), nested nodeTypes so they are namespaced a little better, used replicas instead of count, as replicas are more commonly used terminology in K8sland

[sebgl]

After discussion with @nkvoll on zoom, we agreed on:

apiVersion: deployments.k8s.elastic.co/v1alpha1
kind: Stack
metadata:
    labels:
        controller-tools.k8s.io: "1.0"
    name: stack-sample
spec:
    version: "6.4.2"
    elasticsearch:
        setVmMaxMapCount: true
        topologies:
        # master only
        - nodeTypes:
            master: true
            data: false
            ingest: false
          resources:
            limits:
                mem: 1Gi
                disk: 1GB
                cpu: 800m
          nodeCount: 1
        # data only
        - nodeTypes:
            master: false
            data: true
            ingest: false
          resources:
            limits:
              mem: 8Gi
              disk: 1TB
              cpu: 800m
          nodeCount: 3  

[nkvoll]

@sebgl I believe this can be closed now. Reopen otherwise :)