Amazon-eks-ami: kubectl exec not working as expected on EKS AMIs with GPU support

[itssimon]

What happened:
On EKS nodes running AMIs with GPU support, kubectl cannot pipe STDIN to running containers using kubectl exec. Consequently, some other things don't work either, such as kubectl cp or attaching VSCode to a container running on such a node.

How to reproduce it (as minimally and precisely as possible):
The following command yields an empty file in the pod running on a GPU-enabled node while the file is non-empty on other nodes without GPU support:

cat nonempty_file.txt | kubectl exec -i pod -- tee /home/jovyan/empty_file.txt

This has been tested using the exact same image/container on both nodes.

Anything else we need to know?:
There's been a bit of discussion of this issue on Stack Overflow and in the nvidia-docker repository on GitHub.

While it could be an issue with nvidia-docker, Azure doesn't suffer from this problem. So at the very least, there must be a workaround.

Environment:

• AWS Region: ap-southeast-2
• Instance Type(s): p3.2xlarge
• EKS Platform version: eks.8
• Kubernetes version: 1.14
• AMI Version: ami-038df8ee88d869e42

[therc]

@itssimon does AKS use nvidia-docker? I am 99% sure GKE doesn't, since they also have their own GPU plugin daemonset.

[itssimon]

Not sure actually. I don’t have access to an AKS cluster anymore.

There appears to be a fix for EKS that simply involves updating docker-ce and nvidia-docker in the AMI:
https://github.com/emypar/aws/tree/master/eks/gpu-node-stdin-patch

This was pointed out in https://github.com/NVIDIA/nvidia-docker/issues/1091#issuecomment-622926550.

I just tested the patch and it works indeed! So hopefully the AWS team can just perform these updates in the official AMI and the issue should be solved.

Btw. I made the patched AMI for ap-southeast-2 (Sydney) public:
amazon-eks-node-1.14-v20200423-stdin-patch (ami-08d3396f28bf86847)

[therc]

For anyone not following the other issue, as I just mentioned there, I found a simpler fix that requires just a couple of megabytes over avoids the risks of changing Docker versions:

yum swap -- remove docker-runtime-nvidia \
         -- install nvidia-docker2-2.2.2-1 nvidia-container-runtime-3.1.4-1
systemctl try-restart docker

It would be great if anyone could confirm that it works for them, too.

[therc]

And... I think the bug is actually in the obscure oci-add-hooks, which is a custom AWS tool. See https://github.com/awslabs/oci-add-hooks/issues/5

[Jeffwan]

We will release a new AMI with fix pretty soon. It will come with docker 19 with latest nvidia-docker.

[Jeffwan]

More details:

docker-runtime-nvidia adds a nvidia runtime under /etc/docker-runtimes.d/

#!/bin/sh
exec /usr/bin/oci-add-hooks --hook-config-path /usr/share/docker-runtime-nvidia/hook-config.json --runtime-path /usr/bin/docker-runc "$@"

/usr/share/docker-runtime-nvidia/hook-config.json:

{
  "hooks": {
    "prestart": [
      {
        "path": "/usr/bin/nvidia-container-runtime-hook",
        "args": ["/usr/bin/nvidia-container-runtime-hook", "prestart"]
      }
    ]
  }
}

If we remove docker-runtime-nvidia, following dropin.conf will fail to start because it can not find nvidia runtime. anymore. We should remove --default-runtime=nvidia to unblock docker start. Since Nvidia has been added as devices in docker 19.03. We don't need runtime anymore. docker will use right runc to start container if nvidia gpus are requested.

We should keep oci-hook for other devices.

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_ADD_RUNTIMES \
    --default-runtime=nvidia

For nvidia-docker2 users, it will let docker use nvidia-container-runtime for runtime nvidia. This is configured in /etc/docker/daemon.json. Underneath, it will call nvidia-container-runtime-hook to check if container uses GPU or not, then use libnvidia-container to expose the GPU.

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

To mitigate the issue, add back nvidia-docker2 helps and we will release latest version with nvidia-docker and docker 19.03 soon. (nvidia-docker2 has been deprecated)

[itssimon]

@Jeffwan Do you have an estimate on the release date of the fixed AMI?

[ArchiFleKs]

This seems fixed with latest v20200609 release

[mikestef9]

@ArchiFleKs correct, this has been resolved with latest release. Closing the issue.