K8s-config-connector: GCP resources created by KCC does not allow for dynamic fields

[klaraward]

Use case: Use KCC to create GCP resources used as Traffic Director configuration

1. We create GCP resources using KCC
2. healthcheck
3. backend service

4. urlmap
   ....

5. We annotate k8s service object with cloud.google.com/neg and anthos.cft.dev/autoneg which will respectively create NEGs, and add them to the backend service.

6. These NEGs are removed from the backend service by the KCC controller, which is not the desired behavior for us.

Tried to workaround by

• create healtcheck with KCC
• create backend service with gcloud command
• create urlmap and other resources with KCC

This makes us run into issue #66, that we can't create the resources referring to the backend service.
We tried to workaround this by first creating the backend service with gcloud and then creating the KCC resource for it, this allows for creating the urlmaps etc, but then we run into the same problem we started with, KCC deletes the NEGs from the backend service.

Would it be possible to annotate the KCC resources with some GCP resource fields that KCC will not touch?

[spew]

Hi @klaraward, what does NEG stand for?

[klaraward]

@spew
NEG = Network Endpoint Group, one of the types of “Backends” you can add to a Backend Service (other is MIG = Managed Instance Group)

[caieo]

Hi @klaraward , sorry to hear that KCC keeps removing the NEGs from the backend service. Do you think you can paste the YAML for your backend service?

Re: annotate so that KCC will not touch some GCP resource fields: We don't have anything like this currently and I don't believe we have any current plans for a feature like this. It's possible that you've encountered a bug with the way KCC is handling backend services. We will go through your use case and try to fix it as soon as possible!

[klaraward]

backend service yaml:

apiVersion: compute.cnrm.cloud.google.com/v1alpha3
kind: ComputeBackendService
metadata:
  name: concat-backendservice
  namespace: fabric-rnd
spec:
  healthCheckRef:
    name: concat-healthcheck
  location: global
  loadBalancingScheme: INTERNAL_SELF_MANAGED

[caieo]

@klaraward , thanks for the detailed YAMLs (and in #66 ). Can you help me understand what you mean by this step: "2. We annotate k8s service object with cloud.google.com/neg and anthos.cft.dev/autoneg which will respectively create NEGs, and add them to the backend service"? How are you annotating the NEGs into the k8s object? Are you adding it directly to the YAML or editing it on the command line?

[klaraward]

@caieo
This is how we create our k8s service:

apiVersion: v1
kind: Service
metadata:
  name: concat
  namespace: concat-ns
  annotations:
    cloud.google.com/neg: '{"exposed_ports":{"5990":{}, "8080":{}}}'
    anthos.cft.dev/autoneg: '{"name":"concat-backendservice", "max_rate_per_endpoint":1000}'
...

[caieo]

@klaraward , I marked this issue as an enhancement that is important for us to address within the coming months. We will definitely run into similar scenarios like yours based on the breadth of unique resources es that we deal with. Thank you again for giving us enough context to understand/repro your scenario and we will post updates in this thread!

[spew]

We are planning to support Server Side apply which should make it possible for customers of KCC to specify values for fields they want to manage. For fields they do not specify, KCC will allow "GCP" to be the source of truth. With that information, KCC will be able to sync the state of the BackendService into the API server and know to NOT overwrite the NEG list.

https://kubernetes.io/docs/reference/using-api/api-concepts/#server-side-apply

[AlexBulankou]

We are planning to support Server Side apply which should make it possible for customers of KCC to specify values for fields they want to manage.

CC @soellman, this should enable the scenario of using KCC with your autonegcontroller.

[mikelnrd]

Hi. I am hitting this issue too.

I have a different idea for a solution however (let me know what you think):

Currently the name of the NEG (Network Endpoint Group) that is created automatically when you define a Service is based on the service's name and port and ends with a random string - so isn't knowable in advance - which makes configuration as code difficult.

Image instead that you could configure the name of the NEG in the Service yaml. For example: "name": "my-neg".

apiVersion: v1
kind: Service
metadata:
  name: my-service
  namespace: my-project
  annotations:
    cloud.google.com/neg: '{"exposed_ports":{"5990":{}, "8080":{}}, "name": "my-neg"}'

With this approach, you could configure a BackendService backed by NEGs running in one or more GKE clusters without using the anthos autoneg controller.

You would instead define a ComputeBackendService and ComputeNetworkEndpointGroup using KCC as follows:

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeBackendService
metadata:
  name: my-backend-service
  namespace: my-project
spec:
  affinityCookieTtlSec: 0
  backend:
    - balancingMode: RATE
      capacityScaler: 1.0
      maxRatePerEndpoint: 1000.0
      group:
        networkEndpointGroupRef:
          name: my-neg # reference the NEG defined below
    # optionally add other backends here, eg from another zone in the cluster, or even from another GKE cluster

---

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeNetworkEndpointGroup
metadata:
  name: my-neg # this matches the name defined in the Service annotation above
  namespace: my-project
spec:
  networkEndpointType: GCE_VM_IP_PORT
  networkRef:
    ...
  subnetworkRef:
    ...
  location: ...

This would I imagine need to be filed as an feature request against the GCE Ingress repo.

[mikelnrd]

Just a quick extra note:

For Traffic Director users... the docs currently instruct you to run a command to get the NEG name:

NEG_NAME=$(gcloud beta compute network-endpoint-groups list \
| grep service-test | awk '{print $1}')

https://cloud.google.com/traffic-director/docs/set-up-gke-pods#recording_the_neg_name

Being able to define your own name for the NEG upfront in the kubernetes yaml for the Service object (see my previous post) would not only help avoid this extra step, but you could then bring up all the resources needed for a global load balancer and Traffic Director using KCC.

[omaskery]

We are planning to support Server Side apply which should make it possible for customers of KCC to specify values for fields they want to manage. For fields they do not specify, KCC will allow "GCP" to be the source of truth. With that information, KCC will be able to sync the state of the BackendService into the API server and know to NOT overwrite the NEG list.

https://kubernetes.io/docs/reference/using-api/api-concepts/#server-side-apply

Hi, is there a way to follow/track the progress of this planned feature? I think this may solve a related issue we are having (KCC and GCP fighting over ComputeAddress static IP allocations, in our case) - so eager to understand the roadmap there :) Thanks!

[caieo]

@omaskery, the best way to track the progress of this feature is through this issue thread. As of right now, we are actively working on this feature and are aiming to get a solution out within this quarter/in the next few months.

[Jonpez2]

Could I please clarify - are you aiming for the solution proposed by @mikelnrd , or the Server Side Apply solution? Unless I'm misunderstanding, Server Side Apply would break a model where a project's entire infrastructure is defined in a single code repo, which ConfigSync simply applies. It also - relatedly - makes it difficult/impossible to be certain that the state of the infrastructure is precisely as intended, which harms traceability and security.

Thank you!

[kibbles-n-bytes]

Hey @Jonpez2 , I wrote some follow-up info in the following thread: https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/163 . Let me know if that helps clarify things.

Unless I'm misunderstanding, Server Side Apply would break a model where a project's entire infrastructure is defined in a single code repo, which ConfigSync simply applies.

The approach should still be in harmony with a GitOps, infrastructure-as-code model. All fields specified in the repo will be considered managed by you and the state always enforced as that value on the underlying API unless another manager claims ownership. And in that case, ConfigSync would ensure you're always kept the manager of the fields you care about.

Note the same thing can happen today; If you have a config checked into Git, someone could always come to the Kubernetes cluster and directly modify a value with kubectl edit. And the spec in that case would be treated as the desired state for GCP despite being out-of-sync with respect to the Git repo. But then ConfigSync would come in and quickly get that value back in-sync with the Git repo.

The important part is that, if we supported dynamic fields, KCC wouldn't take over management of a field you already manage. We'd only have potential additional behavior for completely unmanaged fields. Which we already do have in a form today, since we pull the defaulted values back from the underlying API into etcd after creating the resource. It'd just be about making our manager have a more continuous opinion, as opposed to a one-time opinion.

It also - relatedly - makes it difficult/impossible to be certain that the state of the infrastructure is precisely as intended, which harms traceability and security.

In etcd, if the spec says a particular value and the ready condition says UpToDate, you can be certain that, when the condition transitioned, the value on GCP exactly matched the value in the spec. We will never leave any ambiguity there (and we intend to add a status.observedGeneration to remove even more ambiguity as to which spec the status is reporting for the race between the watch triggering the controller and the value updating in etcd).

As for ambiguity around what the value in etcd will be, that's the responsibility of the managers to state their opinion on the fields they care about. KCC would leave the values managed by any others alone, so you can be confident that etcd will reflect the state you have in your Git repo for everything you specified unless some third actor comes in to mess with etcd. But ConfigSync would quickly fix that scenario as well.

[Jonpez2]

I see, thank you for the very clear explanation. Going right back up to
the top of this issue however: should I understand that once this work is
done, we will be able to fully specify a Traffic Director-based
infrastructure using KCC? I'm a little unclear as to whether this solution
will get us there, or whether it gets us to 'users will need to go through
a gcloud-based set of api calls, which will result in unmanaged fields that
KCC/ConfigSync will not break'.

On Mon, May 18, 2020 at 6:52 PM Michael K. [e-mail] wrote:

Hey @Jonpez2 https://github.com/Jonpez2 , I wrote some follow-up info
in the following thread: #163
https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/163
. Let me know if that helps clarify things.

Unless I'm misunderstanding, Server Side Apply would break a model where a
project's entire infrastructure is defined in a single code repo, which
ConfigSync simply applies.

The approach should still be in harmony with a GitOps,
infrastructure-as-code model. All fields specified in the repo will be
considered managed by you and the state always enforced as that value on
the underlying API unless another manager claims ownership. And in that
case, ConfigSync would ensure you're always kept the manager of the fields
you care about.

Note the same thing can happen today; If you have a config checked into
Git, someone could always come to the Kubernetes cluster and directly
modify a value with kubectl edit. And the spec in that case would be
treated as the desired state for GCP despite being out-of-sync with respect
to the Git repo. But then ConfigSync would come in and quickly get that
value back in-sync with the Git repo.

The important part is that, if we supported dynamic fields, KCC wouldn't
take over management of a field you already manage. We'd only have
potential additional behavior for completely unmanaged fields. Which we
already do have in a form today, since we pull the defaulted values back
from the underlying API into etcd after creating the resource. It'd just be
about making our manager have a more continuous opinion, as opposed to a
one-time opinion.

It also - relatedly - makes it difficult/impossible to be certain that the
state of the infrastructure is precisely as intended, which harms
traceability and security.

In etcd, if the spec says a particular value and the ready condition says
UpToDate, you can be certain that, when the condition transitioned, the
value on GCP exactly matched the value in the spec. We will never leave any
ambiguity there (and we intend to add a status.observedGeneration to
remove even more ambiguity as to which spec the status is reporting for
the race between the watch triggering the controller and the value updating
in etcd).

As for ambiguity around what the value in etcd will be, that's the
responsibility of the managers to state their opinion on the fields they
care about. KCC would leave the values managed by any others alone, so you
can be confident that etcd will reflect the state you have in your Git repo
for everything you specified unless some third actor comes in to mess
with etcd. But ConfigSync would quickly fix that scenario as well.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/68#issuecomment-630340867,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABN425OV677O5ZXHXUOI6LTRSFYWXANCNFSM4JVK6I7Q
.

[AlexBulankou]

@Jonpez2 , dynamic fields approach described above would allow backends collection on BackendService to be populated outside of Config Connector, for example, by autoneg controller, without KCC clearing the list, as happens today. This is the immediate fix that we are working towards.

[Jonpez2]

OK awesome, thanks. Is the next fix to support full declarative specs for
Traffic Director?

On Tue, May 19, 2020 at 3:44 PM Alex Bulankou [e-mail]
wrote:

@Jonpez2 https://github.com/Jonpez2 , dynamic fields approach described
above would allow backends collection on BackendService to be populated
outside of Config Connector, for example, by autoneg controller, without
KCC clearing the list, as happens today. This is the immediate fix that we
are working towards.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/68#issuecomment-630867297,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABN425K2K3OGNIZN4OQWIXDRSKLNDANCNFSM4JVK6I7Q
.

[AlexBulankou]

OK awesome, thanks. Is the next fix to support full declarative specs for Traffic Director?

@Jonpez2 , what is the list of resources that that you need configured for TrafficDirector?

Original question that we discussed on this thread included:

• k8s services annotated with neg and autoneg that auto-created NEGs
• GCP backend service
• GCP healthcheck
• GCP urlmap

All of these are supported, however backend service did not integrate well auto-created NEGs, this is what we are fixing.

@Jonpez2 , what is the list of resources that that you need configured for TrafficDirector?

[Jonpez2]

Ok - I think I am totally covered! CC will create the basic infra, which
will auto create negs, and then CC won’t break it. I’m now totally clear,
thank you for bearing with my slow progress.
This whole ecosystem is shaping up to be so lovely; thanks for all the hard
work.

On Wed, 20 May 2020 at 16:13, Alex Bulankou [e-mail]
wrote:

OK awesome, thanks. Is the next fix to support full declarative specs for
Traffic Director?

@Jonpez2 https://github.com/Jonpez2 , what is the list of resources
that that you need configured for TrafficDirector?

Original question that we discussed on this thread included:

• k8s services annotated with neg and autoneg that auto-created NEGs
• GCP backend service
• GCP healthcheck
• GCP urlmap

All of these are supported, however backend service did not integrate well
auto-created NEGs, this is what we are fixing.

@Jonpez2 https://github.com/Jonpez2 , what is the list of resources
that that you need configured for TrafficDirector?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/68#issuecomment-631538719,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABN425PZ5PSZOSFIM7EKCMDRSPXTPANCNFSM4JVK6I7Q
.

[AlexBulankou]

NEG-based MCI scenario is now supported with KCC and this week we published a sample that demonstrates it step by step.

[toumorokoshi]

This looks to be resolved. Please re-open if there are still issues.