Cloud-on-k8s: Support Elastic Maps Service

[pebrc]

We should investigate whether ECK needs to explicitly support Elastic Maps Service for air-gapped environments.

A simple deployment of Elastic Maps Server can be done with a standard k8s Deployment resource.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ems
  labels:
     app.kubernetes.io/name: ems
spec:
  replicas: 1
  selector:
    matchLabels:
       app.kubernetes.io/name: ems
  template:
    metadata:
      labels:
         app.kubernetes.io/name: ems
    spec:
      containers:
        - name: ems
          image: docker.elastic.co/elastic-maps-service/elastic-maps-server-ubi8:7.11.0-SNAPSHOT
          ports:
            - containerPort: 8080
          env:
            - name: HOST
              value: 0.0.0.0
            - name: ELASTICSEARCH_HOST
              value: "https://elasticsearch-es-http.default.svc:9200"
            - name: ELASTICSEARCH_USERNAME
              value: "elastic" # TODO this should be a user with monitor privileges instead
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: elasticsearch-es-elastic-user
                  key: elastic
            - name: ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES
              value: /usr/src/app/server/certificates/ca.crt
          volumeMounts:
            - name: ca-certs
              mountPath: /usr/src/app/server/certificates
              readOnly: true
      volumes:
        - name: ca-certs
          secret:
            secretName: elasticsearch-es-http-certs-public
---
apiVersion: v1
kind: Service
metadata:
  name: ems
  labels:
    app.kubernetes.io/name: ems
spec:
  ports:
    - name: "tcp"
      port: 8080
      targetPort: 8080
  selector:
    app.kubernetes.io/name: ems
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 7.11.0-SNAPSHOT
  nodeSets:
    - name: default
      count: 3
      config:
        node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 7.11.0-SNAPSHOT
  count: 1
  config:
    map.emsUrl: http://ems.default.svc:8080
  elasticsearchRef:
    name: elasticsearch
---
apiVersion: v1
kind: Secret
metadata:
  name: eck-trial-license
  labels:
    license.k8s.elastic.co/type: enterprise-trial
  annotations:
    elastic.co/eula: accepted
...

A couple of things to note or investigate further:

• minimal user privileges. The example above uses the elastic user that ECK provisions in the file realm which is a superuser. Two options come to mind:
  
  
  
  • creating a restricted user ahead of time via the support for the file realm ECK already has. https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-users-and-roles.html
  
  
  • implementing explicit support for EMS in ECK and have the operator create a user for it with monitor privileges
  
  
• Elasticsearch connection. By default Elasticsearch clusters managed by ECK have a self-signed certificate authority which needs to be explicitly configured in EMS as shown in the example above. However after ~1 year the operator rotates the CA certifcate. Does EMS automatically reload configured certificates when they change on the filesystem? If not we would need to actively manage EMS and rotate the Pod when the certificates change (either because of the aforementioned regular rotation or because of user triggered rotation)
• HTTPS: The example does not use TLS but that could be desirable. Depending on the users certificate management in k8s this could be achieved with cert-manager or by manually creating certificates ahead of time
• Tile database: I was not able to test this as the link generated by EMS to download the tile database currently points to a placeholder zip file. But in general this manual step is problematic in a k8s environment. Of course we could run an init container to download the database but it seems the generated link contains some dynamically computed query parameter that we don't have yet when the init container runs. Also we would need to take extra precautions to download the database only once and not on every Pod restart. This could be achieved for example with extra shell scripting and a marker file or similar. But I think it would be preferable if EMS could just download the tile database itself. The k8s admin would just have to make sure they mount a large enough persistent volume into the EMS deployment. There is the additional question of what should happen if we run multiple instances of EMS. In theory we could mount the same volume read-only into each of the Pods but that leaves the question open of which of these instances should be responsible for downloading it in the first place.

cc @nickpeihl

[nickpeihl]

Thanks for reviewing this, I've added some comments below.

Does EMS automatically reload configured certificates when they change on the filesystem? If not we would need to actively manage EMS and rotate the Pod when the certificates change (either because of the aforementioned regular rotation or because of user triggered rotation)

EMS does not reload certificates when they change which would require a pod rotation. How does ECK handle certificate rotation with Elasticsearch & Kibana?

• Tile database: I was not able to test this as the link generated by EMS to download the tile database currently points to a placeholder zip file.

Apologies. The placeholder has been replaced with the full tile database now.

But I think it would be preferable if EMS could just download the tile database itself. The k8s admin would just have to make sure they mount a large enough persistent volume into the EMS deployment.

I think we can work together on a solution for EMS to download the database itself if there is sufficient storage mounted.

[barkbay]

I'm not sure to understand the cardinality of the relationships between all the components . IIUC there is a 1-1 relationship between EMS and Elasticsearch, but is it possible to have a "many to one" between several Kibana instances and a single EMS ?
Does it make sense for Kibana to use an EMS instance which is associated to a different Elasticsearch instance ?

[pebrc]

I'm not sure to understand the cardinality of the relationships between all the components . IIUC there is a 1-1 relationship between EMS and Elasticsearch, but is it possible to have a "many to one" between several Kibana instances and a single EMS ?

EMS needs one connection to an Elasticsearch cluster for license checks. The one instance is then providing the Maps service in the whole environment.

[pebrc]

I think we can work together on a solution for EMS to download the database itself if there is sufficient storage mounted.

The one thing I forgot was that is probably not possible or desired to auto-download the map tiles, if the main motivation for EMS would be to run in air-gapped environments?

[nickpeihl]

The one thing I forgot was that is probably not possible or desired to auto-download the map tiles, if the main motivation for EMS would be to run in air-gapped environments?

That's a good point so I think it would be preferable for map tile download to be a user action rather than an automatic action. The Docker image does include a small subset of tiles for zoom levels 0-4 (out of 14). So, if there is no volume mounted at /usr/src/app/data, the subset of tiles included in the image are used. This might be good enough for some users who can optionally download and mount the complete tile database if they need it.

Maybe we can document the user process to download and mount the complete tileset? How difficult is this user action?

[nickpeihl]

Bumping this thread. Do we need to resolve anything else? We would love to roadmap this item if possible.

[pebrc]

One more thing that came to mind is whether users would want or need to deploy multiple instances of EMS in the same k8s cluster based on different stack versions.

So say they are running a few Kibanas with 7.11 would they need to deploy EMS at 7.11 and if they then had additional workloads on 7.12 or other stack versions would they need to run additional EMS deployments matching those versions?

[nickpeihl]

So say they are running a few Kibanas with 7.11 would they need to deploy EMS at 7.11 and if they then had additional workloads on 7.12 or other stack versions would they need to run additional EMS deployments matching those versions?

Yes, they would. The Kibana instance and the Elastic Maps Server it is connecting to must be the same version.

[pebrc]

I did continue the investigation by trying to download the map tiles. One idea was to just kubectl exec into the container after mounting a large enough volume and follow the download link. But this is not very realistic as we are simulating an air-gapped environment. So you would have to download the file to your workstation then kubectl cp onto the pod and extract it there. The EMS pod does not come with unzip installed. I ended up creating a temporary Pod and downloading from there on the volume also because I don't have enough disk space one my laptop to download the file.

I am sharing the manifests here.

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pv-claim-ems
spec:
  storageClassName: "standard"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 250G
---
kind: Pod
apiVersion: v1
metadata:
  name: dl-ems-pod
spec:
  terminationGracePeriodSeconds: 0
  securityContext:
    fsGroup: 101
  volumes:
    - name: ems-storage
      persistentVolumeClaim:
       claimName: pv-claim-ems
  containers:
    - name: ems-dl-container
      image: ubuntu
      command: [bash, -c]
      env:
      - name: URL
        value: 'https://download.elastic.co/elastic-maps-server/planet-3.x.zip?license=<readacted>&license_type=trial'
      args:
        - |
          while true; do sleep 10; done
      volumeMounts:
        - mountPath: "/usr/share/world"
          name: ems-storage

I then remounted the volume onto the EMS Pod and reconfigured Kibana to use the local EMS:

Again the adjusted manifests here.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ems
  labels:
     app.kubernetes.io/name: ems
spec:
  replicas: 1
  selector:
    matchLabels:
       app.kubernetes.io/name: ems
  template:
    metadata:
      labels:
         app.kubernetes.io/name: ems
    spec:
      containers:
        - name: ems
          image: docker.elastic.co/elastic-maps-service/elastic-maps-server-ubi8:7.12.0
          ports:
            - containerPort: 8080
          env:
            - name: HOST
              value: 0.0.0.0
            - name: ELASTICSEARCH_HOST
              value: "https://elasticsearch-es-http.default.svc:9200"
            - name: ELASTICSEARCH_USERNAME
              value: "elastic" # TODO this should be a user with monitor privileges instead
            - name: ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: elasticsearch-es-elastic-user
                  key: elastic
            - name: ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES
              value: /usr/src/app/server/certificates/ca.crt
          volumeMounts:
            - name: ca-certs
              mountPath: /usr/src/app/server/certificates
              readOnly: true
            - name: map-data
              mountPath: /usr/src/app/data
      volumes:
        - name: ca-certs
          secret:
            secretName: elasticsearch-es-http-certs-public
        - name: map-data
          persistentVolumeClaim:
            claimName: pv-claim-ems
---
apiVersion: v1
kind: Service
metadata:
  name: ems
  labels:
    app.kubernetes.io/name: ems
spec:
  ports:
    - name: "tcp"
      port: 8080
      targetPort: 8080
  selector:
    app.kubernetes.io/name: ems
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 7.12.0
  nodeSets:
    - name: default
      count: 3
      config:
        node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 7.12.0
  count: 1
  config:
    map.emsUrl: http://localhost:8080 # counter intuitive but I was accessing EMS through a port-foward from my browser
  elasticsearchRef:
    name: elasticsearch
---
apiVersion: v1
kind: Secret
metadata:
  name: eck-trial-license
  labels:
    license.k8s.elastic.co/type: enterprise-trial
  annotations:
    elastic.co/eula: accepted
...

[pebrc]

Given that a separate CRD for Elastic Maps Service will offer some benefits over the bare-bones Deployment approach (certificate and credential management and also potentially upgrade order enforcement) it is maybe worth discussing how that could look like:

apiVersion: maps.k8s.elastic.co/v1
kind: MapsServer # should this be just `Maps` or `ElasticMaps` or `ElasticMapsService`? It needs to have plural for k8s e.g. `MapsServers`
metadata:
  name: ems
spec:
  version: 7.12.0
  count: 1
  elasticsearchRef:
    name: elasticsearch

Another question is how the data volume containing the tiles should be specified. In the example above it is either implicit (a volume of a given name is expected to be present, too confusing?) or no data volume is mounted.

To add a data volume we could then follow the same pattern as in the other CRDs and allow users to spec the volume in the context of the podTemplate attribute:

apiVersion: maps.k8s.elastic.co/v1
kind: MapsServer
metadata:
  name: ems
spec:
  version: 7.12.0
  count: 1
  podTemplate:
    spec:
      containers:
        - name: ems
          volumeMounts:
            - name: map-data
              mountPath: /usr/src/app/data
      volumes:
        - name: map-data
          persistentVolumeClaim:
            claimName: pv-claim-ems      
  elasticsearchRef:
    name: elasticsearch

Given that the tile data volume is sort of special in that it will be always required and has a fixed mount path we could also make it an explicit top-level attribute:

apiVersion: maps.k8s.elastic.co/v1
kind: MapsServer
metadata:
  name: ems
spec:
  version: 7.12.0
  count: 1
  tiles:  # type k8s.io/api/core/v1.VolumeSource naming alternatives: basemaps or simply dataVolume
        persistentVolumeClaim:
             claimName: pv-claim-ems
  elasticsearchRef:
    name: elasticsearch

[idanmo]

@pebrc What did you have in mind regarding the necessary changes to the Kibana CRD, for specifying an ems reference. Something like:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 7.12.0
  count: 1
  emsRef:
    name: ems

?
Also, do you have any thoughts about implicit vs. explicit references to Elasticsearch or Kibana? (If Kibana has a reference to Elasticsearch use that instead of specifying)

[pebrc]

What did you have in mind regarding the necessary changes to the Kibana CRD, for specifying an ems reference.

Initially I thought we would not model the relationship between Kibana and EMS in the operator. Mainly because it is an 1:n relationship:

┌────┐
│ ES │ License check
└────┘        ┌────┐      ┌────┐
  ▲   ┌───────┤ KB ├─────►│ ES │
  │   │       └────┘      └────┘
  │   ▼
┌─┴───┐       ┌────┐      ┌────┐
│ EMS │ ◄─────┤ KB ├─────►│ ES │
└─────┘       └────┘      └────┘
      ▲
      │       ┌────┐      ┌────┐
      └───────┤ KB ├─────►│ ES │
              └────┘      └────┘

There is however the problem of TLS connections. If users chose to enable TLS in EMS and we use a self-signed CA (as we do by default in all other cases) we would need to configure that CA in each Kibana that wants to talk to the Maps Server. My worry is that it will lead to an explosion of associations we have to manage in the operator.

[pebrc]

Another argument against modelling Kibana associations explicitly is that clients which need to talk to Elastic Maps Server will typically reside outside of the K8s cluster. Meaning that both Kibana and EMS need to be exposed through LoadBalancers or Ingress. Kibana does not send any CORS headers currently afaik so both Kibana and EMS also need to be hosted on the same domain. All these factors make it hard to actively manage the map.emsUrl attribute actively (which would be one of the theoretical benefits modelling Kibana assocs explicitly)

┌─────────────────────────┐
│  K8s                    │
│   ┌────┐    ┌────┐      │
│   │ ES │    │ ES │      │
│   └────┘    └────┘      │
│     ▲         ▲         │
│     │         │         │
│     │         │         │
│   ┌─┴───┐   ┌─┴──┐      │
│   │ EMS │   │ KB │      │
│   └─────┘   └────┘      │
│       ▲       ▲         │
│       │       │         │
└───────┼───────┼─────────┘
        │       │
        │       │
       ┌┴───────┴┐
       │Client   │
       └─────────┘

This also leaves the TLS problem I mentioned then clearly in the responsibility of the user. Which is OK for an Ingress based solution imo.