Cloud-on-k8s: Changes to services to support Istio

[charith-elastic]

Istio relies on all accessible ports being advertised by a service. We currently don't add the TCP port (9300) to the <CLUSTER_NAME>-es-http service created by the operator and this causes node discovery to fail when creating an Elasticsearch cluster containing more than one node in an Istio-enabled cluster.

Unfortunately, manually adding the TCP port to the ports list through http.service.spec.ports does not work either as the operator overwrites the ports array during reconciliation.

[charith-elastic]

To clarify, this might not be the root cause. I have simply observed that pausing the operator and adding 9300 to the ports list in the service results in Elasticsearch being able to discover its peers. Unpausing the operator causes the service to lose the port definition for 9300 and then Elasticsearch starts reporting discovery exceptions again.

[pebrc]

One workaround until we fix this would be to manually create a service that advertises the transport port. Something like:

apiVersion: v1
kind: Service
metadata:
  labels:
    common.k8s.elastic.co/type: elasticsearch
    elasticsearch.k8s.elastic.co/cluster-name: $CLUSTER_NAME
  name: $CLUSTER_NAME-es-tp
  namespace: default
spec:
  clusterIP: none
  ports:
  - name: tcp-transport
    port: 9300
    protocol: TCP
    targetPort: 9300
  selector:
    common.k8s.elastic.co/type: elasticsearch
    elasticsearch.k8s.elastic.co/cluster-name: $CLUSTER_NAME

[charith-elastic]

Other things to consider:

• Port naming
• Not overwriting service template fields defined by the user
• Preserving user modifications to existing services

[spencergilbert]

Bizarrely I'm still having issues with the above workarounds - I'm either ending up with the nodes not discovering one-another or getting cert errors in the ES logs

[charith-elastic]

@spencergilbert Are you using the STRICT mode?

[spencergilbert]

mTLS on istio you mean? yes

[charith-elastic]

@spencergilbert yes, I was wondering whether you had enabled strict mTLS enforcement either during Istio installation or by creating a policy like:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: default
spec:
  peers:
  - mtls:
      mode: STRICT

Anyway, in order to get ECK 1.0.0-beta1 to work with Istio 1.3.3, I did the following:

• Enable sidecar injection for the elastic-system namespace and install the operator (or follow whatever other method you prefer to have the Istio sidecar injected into the pods created by the elastic-operator StatefulSet inside the elastic-system namespace). Without the Istio sidecar, the operator will not be able to correctly access the Elasticsearch API.
  shell kubectl label namespace elastic-system istio-injection=enabled kubectl apply -f https://download.elastic.co/downloads/eck/1.0.0-beta1/all-in-one.yaml
• For Elasticsearch, exclude the transport port 9300 from being proxied and rename the service port to http
• For Kibana, enable rewriting the health probe to make the health checks pass.

cat <<EOF | kubectl apply -f -
---
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: elasticsearch-istio
spec:
  version: 7.4.0
  http:
    tls:
      selfSignedCertificate:
        disabled: true
    service:
      spec:
        ports:
          - name: http
            port: 9200
            targetPort: 9200
  nodeSets:
  - count: 3
    name: default
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    podTemplate:
      metadata:
        annotations:
          traffic.sidecar.istio.io/excludeOutboundPorts: "9300"
          traffic.sidecar.istio.io/excludeInboundPorts: "9300"
---
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: kibana-istio
spec:
  version: 7.4.0
  count: 1
  elasticsearchRef:
    name: "elasticsearch-istio"
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  podTemplate:
    metadata:
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"
EOF

The port naming issue has been fixed in master and will not require an override in the manifest in future versions.

[spencergilbert]

Thanks! That looks to work perfectly!

[infa-ddeore]

@spencergilbert yes, I was wondering whether you had enabled strict mTLS enforcement either during Istio installation or by creating a policy like:

apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: default
spec:
  peers:
  - mtls:
      mode: STRICT

Anyway, in order to get ECK 1.0.0-beta1 to work with Istio 1.3.3, I did the following:

• Enable sidecar injection for the elastic-system namespace and install the operator (or follow whatever other method you prefer to have the Istio sidecar injected into the pods created by the elastic-operator StatefulSet inside the elastic-system namespace). Without the Istio sidecar, the operator will not be able to correctly access the Elasticsearch API.
  shell kubectl label namespace elastic-system istio-injection=enabled kubectl apply -f https://download.elastic.co/downloads/eck/1.0.0-beta1/all-in-one.yaml
• For Elasticsearch, exclude the transport port 9300 from being proxied and rename the service port to http
• For Kibana, enable rewriting the health probe to make the health checks pass.

cat <<EOF | kubectl apply -f -
---
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: elasticsearch-istio
spec:
  version: 7.4.0
  http:
    tls:
      selfSignedCertificate:
        disabled: true
    service:
      spec:
        ports:
          - name: http
            port: 9200
            targetPort: 9200
  nodeSets:
  - count: 3
    name: default
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
    podTemplate:
      metadata:
        annotations:
          traffic.sidecar.istio.io/excludeOutboundPorts: "9300"
          traffic.sidecar.istio.io/excludeInboundPorts: "9300"
---
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: kibana-istio
spec:
  version: 7.4.0
  count: 1
  elasticsearchRef:
    name: "elasticsearch-istio"
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  podTemplate:
    metadata:
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"
EOF

The port naming issue has been fixed in master and will not require an override in the manifest in future versions.

@charith-elastic this explains how to disable tls for the http but how to use istio for transport (port 9300) communication as well? it seems the x-pack can't be disabled at this point to achieve this

[charith-elastic]

@infa-ddeore Yes, we don't recommend switching off X-pack security when running Stack applications with ECK. This makes it difficult to make the transport port communications go through Istio at the moment.

Transport port is still secured by TLS and is only used for internal cluster communications between Elasticsearch nodes. I do understand that you lose some visibility and centralised control due to those not being controlled by Istio. Are there any particularly pressing concerns or features you are missing out on right now? We can consider that feedback when prioritising further work on support for service meshes.

[infa-ddeore]

one thing is standardise all our inter-services communication via istio and another one is to use readonlyrest plugin which needs x-pack disabled

[xyphr]

We are also facing problems when trying to set up a cluster using readonlyrest plugin in an istio enabled cluster. Is there any workaround that you were able to find?