Linkerd2: Helm chart proxyInit.resources values are ignored when injecting into pods

[evanfuller]

Bug Report

What is the issue?

The linkerd2 helm chart defines variables global.proxyInit.resources.cpu/memory to use when injecting the linkerd-init container to a pod. However, after deploying the linkerd2 chart alongside my own service (with auto-injection configured), the linkerd-init container is using default values.

I noticed that the linkerd pods themselves (e.g. controller, proxy-injector, etc.) do have the CPU / memory values that I configured for the proxyInit container, but the init container that linkerd's injector deploys is just using the defaults.

This has implications for pod QoS -- according to the docs, a pod can only be the Guaranteed class if the sum of all memory and CPU requests and limits in the pod are equal. As of this kubelet PR, init containers are included in this calculation.

In other words, without being able to configure the init container's resources, we are unable to deploy "Guaranteed" QoS pods; we can only get Burstable.

How can it be reproduced?

1. Deploy linkerd2 via helm chart, and set the proxyInit.resources vars in values.yaml.
2. Configure the "linkerd.io: enabled" annotation on a k8s namespace.
3. Create some k8s deployment in that namespace.
4. Describe a pod in the newly-created deployment and look at the resources allocated to the linkerd-init container; they will not match the values specified in values.yaml, they will instead be the defaults [10 Mi, 50 Mi], [10m, 100m].

linkerd check output

evan @ ?130 linkerd % linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

linkerd-addons
--------------
√ 'linkerd-config-addons' config map exists

linkerd-grafana
---------------
√ grafana add-on service account exists
√ grafana add-on config map exists
√ grafana pod is running

Status check results are √

Environment

• Kubernetes Version: 1.16
• Cluster Environment: EKS
• Host OS: Alpine / Linux
• Linkerd version: stable-2.8.1

Possible solution

If we add a proxyInitResourceRequirements() method to the ResourceConfig receiver, mirrored after the existing method, we should be able to dynamically inject these values into the l5dcharts.ProxyInit struct instead of using constants.

Additional context

[Pothulapati]

@evanfuller Thanks for the detailed error report. You are right, Currently, We are not actually storing the proxyInit resources in the linkerd-config which can be retrieved and applied later on. This configuration is only honored for the proxyInit of the Linkerd control-plane as of now.

As this affects the QoS stuff, It makes sense to honor and store this configuration across injections.

[Pothulapati]

Also, We have been working on a new config format (See #4911) where we store all overridden values, and I was wondering if this would be fixed directly :thinking:

Looks like only the storing will be fixed, but retrieval and usage has to be still updated as we set to the defaults currently? WDYT @adleong

[evanfuller]

@Pothulapati Is there any status update on this ticket? I see the label 2.9 on the issue; can you give a rough time window for when the team hopes to release 2.9?

[Pothulapati]

Hey @evanfuller

Once #5005 lands, I will update the injection to also use the proxyInit.resources configuration. If things go well, This should land in this week's edge release and The plan is to release 2.9 by the end of this month. :) Hope that clarifies your question.

[Pothulapati]

@evanfuller I just verified this on main by modifying proxyInit.resources and installing Linkerd and then It also applied the same config on injection. This is fixed part of https://github.com/linkerd/linkerd2/commit/41c1fc65b0ef680e96fe5f5930b81adf5ce92f0f

 kind-kind in linkerd2 on  main [$!?] via 🐹 v1.14.9
❯ k -n linkerd describe cm linkerd-config | grep proxyInit -A 15
  proxyInit:
    closeWaitTimeoutSecs: 0
    ignoreInboundPorts: ""
    ignoreOutboundPorts: ""
    image:
      name: ghcr.io/linkerd/proxy-init
      pullPolicy: IfNotPresent
      version: v1.3.6
    resources:
      cpu:
        limit: 103m
        request: 11m
      memory:
        limit: 53Mi
        request: 13Mi
    xtMountPath:

 kind-kind in linkerd2 on  main [$!?] via 🐹 v1.14.9
❯ k -n emojivoto describe pod emoji-7dc976587b-f2vzb  | grep -A 30 linkerd-init
  linkerd-init:
    Container ID:  containerd://cbd4e055a9ee4f063fce041923fa0820be12d96ac61049e2e6145b9bbec328b9
    Image:         ghcr.io/linkerd/proxy-init:v1.3.6
    Image ID:      ghcr.io/linkerd/proxy-init@sha256:591d4f578a8c4431fc6343c8f2703338888865440f02f4cb44862a87c5557859
    Port:          <none>
    Host Port:     <none>
    Args:
      --incoming-proxy-port
      4143
      --outgoing-proxy-port
      4140
      --proxy-uid
      2102
      --inbound-ports-to-ignore
      4190,4191
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 13 Oct 2020 23:46:30 +0530
      Finished:     Tue, 13 Oct 2020 23:46:30 +0530
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     103m
      memory:  53Mi
    Requests:
      cpu:        11m
      memory:     13Mi
    Environment:  <none>
    Mounts:

I'm closing this now as its fixed. This should land in this week's edge (which goes day after tomorrow). Feel free try it out and re-open the issue if you have any problem! :+1: