Airflow: KubernetesPodOperator handles a default value of k8s V1ResourceRequirements as None causes incorrect value

[roy-ht]

Apache Airflow version: 1.10.11

Kubernetes version (if you are using kubernetes) (use kubectl version): 1.16

Environment:

• Cloud provider or hardware configuration: Amazon EKS
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

If resources argument has partial parameters, KubernetesPodOperator generates default as None, but should not have a key.

It is written here:
https://github.com/apache/airflow/blob/d23fa2f8896c5e31745131e6917f1970b4aa590f/airflow/kubernetes/pod.py#L82

For example, if I made operator like KubernetesPodOperator(resources={'limit_cpu': '500m'}), generated pods have wrong limits like below:

  24   │ Containers:
  25   │   base:
  26   │     Image:      xxxxxxxx
  27   │     Port:       <none>
  28   │     Host Port:  <none>
  29   │     Limits:
  30   │       cpu:                500m
  31   │       ephemeral-storage:  0
  32   │       memory:             0
  33   │       nvidia.com/gpu:     0
  34   │     Requests:
  35   │       cpu:                0
  36   │       ephemeral-storage:  0
  37   │       memory:             0
  38   │       nvidia.com/gpu:     0
...
...
 107   │   Normal   Created    20m   kubelet, xxx  Created container base
 108   │   Normal   Started    20m   kubelet, xxx  Started container base
 109   │   Warning  Evicted    20m   kubelet, xxx  Pod ephemeral local storage usage exceeds the total limit of containers 0.
 110   │   Normal   Killing    20m   kubelet, xxx  Stopping container base

What you expected to happen:

Omit None value of limits and requests.
For example, Resources.to_k8s_client_obj should be modified like that:

    def to_k8s_client_obj(self):
        limits = {}
        if self.limit_cpu:
            limits['cpu'] = self.limit_cpu
        ...
        return k8s.V1ResourceRequirements(limits=limits, requests=reqests)

How to reproduce it:

Anything else we need to know:

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

cc: @dimberman @kaxil -> I guess that one is too for 1.10.12

[odracci]

could be related to https://github.com/apache/airflow/issues/9812

[kaxil]

could be related to #9812

Yup they are related

[aneesh-joseph]

I used this as a work around - https://github.com/apache/airflow/issues/9812#issuecomment-659113921

[dimberman]

Hi @roy-ht, thank you for posting this. This will be fixed in 1.10.12.

[odavid]

Hi @dimberman, I created the PR #10140, which solves this issue, however the PR is stuck on static checks, which seems not relevant to my changes. Will be super glad if you can take a look.

Cheers...

[dimberman]

https://github.com/apache/airflow/pull/10084 fixed

[mgerlach]

With 1.10.11, I also saw KubernetesPodOperator(resources={'limit_cpu': '500m'}) generating this JSON
"resources": { "limits": { "cpu": "500m", "memory": null, "nvidia.com/gpu": null, "ephemeral-storage": null } }
that does not look right, either.