Amazon-eks-ami: docker daemon.json missing from GPU AMI, causes failure when enable-docker-bridge set

[lilley2412]

What happened:

Enabled --enable-docker-bridge when using a GPU AMI, the node did not join the cluster. The node had an empty /etc/docker/daemon.json file causing the docker service to fail (the docker service still start if the file doesn't exist, but it won't start if it exists and is empty).

What you expected to happen:

I expected the docker service to start with bridge networking mode and for the node to join the cluster.

How to reproduce it (as minimally and precisely as possible):

Create a GPU node and pass --enable-docker-bridge to bootstrap.sh. I tested all 1.14* versions and they all have this issue.

Anything else we need to know?:

The failure happens due to this line https://github.com/awslabs/amazon-eks-ami/blob/c47e0c078c76a563b04c462a21da3492b5f138ac/files/bootstrap.sh#L222

jq outputs an empty string because the file /etc/docker/daemon.json doesn't exist.

On the non-GPU ami, the docker daemon file contains this content by default:

{
  "bridge": "none",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "10"
  },
  "live-restore": true,
  "max-concurrent-downloads": 10
}

attempted workarounds

I tried adding the default file and setting the flag, but docker startup fails with

./messages:Oct 11 12:58:12 ip-10-191-93-109 dockerd: unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: bridge: (from flag: none, from file: docker0)

I haven't debugged further but guessing that the flag is being set from an env var in /usr/lib/systemd/system/docker.service?

ExecStart=/usr/bin/dockerd $OPTIONS $DOCKER_STORAGE_OPTIONS $DOCKER_ADD_RUNTIMES

So ...
1) is the lack of default daemon.json intentional
2) is there some issue with allowing bridge mode to be enabled on gpu instances?

[Jeffwan]

/cc @Jeffwan

[eddiesimeon]

GPU AMI does not use docker daemon.json file.
The values set in the daemon.json file are instead passed in as an environment file /etc/sysconfig/docker as defined in the unit file /usr/lib/systemd/system/docker.service

/usr/lib/systemd/system/docker.service:
EnvironmentFile=-/etc/sysconfig/docker

/etc/sysconfig/docker:
OPTIONS="--bridge=none --default-ulimit nofile=2048:8192 --log-driver=json-file --log-opt max-size=10m --log-opt max-file=10 --live-restore=true --max-concurrent-downloads=10 --default-runtime=nvidia"

[mazayan]

removing the --bridge=none and the empty /etc/docker/daemon.json file works - my nodes are able to join the cluster now.
Why was --bridge=none built into the options? Wouldn't you want a node to join the cluster by default?

[zeyaddeeb]

Is there a fix yet?

[marcheil]

I've run into this same issue as well. We added the below to the user data for the ASGs:

vi /etc/sysconfig/docker
removing the --bridge=none
rm -rf /etc/docker/daemon.json
systemctl restart docker
systemctl restart kubelet

[heybronson]

The latest release v20210302 fixes this issue. GPU AMIs now have the same daemon.json file defined at https://github.com/awslabs/amazon-eks-ami/blob/master/files/docker-daemon.json.

The --enable-docker-bridge flag in bootstrap.sh will now work for GPU AMIs.

[heybronson]

This was reverted in release v20210310 due to a conflict with eksctl. We're working on resolving the issue.

[ajayk]

@heybronson are there any workarounds for this , the latest ami amazon-eks-gpu-node-1.19-v20210504 fails with the same error

[rajakshay]

The latest release v20210302 fixes this issue. GPU AMIs now have the same daemon.json file defined at https://github.com/awslabs/amazon-eks-ami/blob/master/files/docker-daemon.json.

The --enable-docker-bridge flag in bootstrap.sh will now work for GPU AMIs.

This was reverted in release v20210310 due to a conflict with eksctl. We're working on resolving the issue.

@heybronson can you please point out the commits for the above changes. I wasn't able to find it.

Also, I don't understand a few parts of the AMI creation process. If you could answer them, that'd be great too.

1. Is there a separate method for generating the GPU AMIs? I don't see any mention of the name the GPU AMIs take in the Makefile

2. Does this line not apply to GPU AMIs: https://github.com/awslabs/amazon-eks-ami/blob/master/scripts/install-worker.sh#L121

3. For GPU AMI, where does the value of OPTIONS in the file /etc/sysconfig/docker come from?

4. In the AMI release v20210504, I noticed that if you manually create the file /etc/docker/daemon.json, and restart the systemd unit docker.service, the file gets deleted in GPU AMI, but, not in the CPU AMI. It may be true for other recent releases as well (i haven't checked). I did however test it in the past for the GPU release v20200921, and this wasn't the case then. Would you know what causes this difference of behavior b/w CPU and GPU AMIs

Thanks in advance.

[heybronson]

The commits referenced in the above changes impacted our internal GPU AMI release and is not reflected on this repository.

1. There is a separate package for generating EKS Accelerated AMIs. The package runs all packer scripts included in this repository, and a few more for installing required configurations/drivers.
2. No, that line currently does not apply to the Accelerated AMIs.
3. OPTIONS is included within an additional packer script within the separate package for generating EKS Accelerated AMIs.
4. Accelerated AMIs contain an additional systemd unit required for bootstrapping the AMI correctly. Part of that script removes the existing etc/docker/daemon.json file before starting dockerd.