Eksctl: Allow passing `--enable-docker-bridge` to `amazon-eks-ami` bootstrap

[danielchalef]

Why do you want this feature?
A number of common Kubernetes use cases, including CI and machine learning experiment management, require building docker images on Kubernetes (more generally known as "docker-on-docker"). Amazon recently disabled the docker bridge on the amazon-eks-ami family of images, breaking this capability (see Docker in Docker no longer works without docker0 bridge). Even more recently, Amazon added a --enable-docker-bridge switch to the AMI bootstrap script to reenable the bridge.

What feature/behavior/change do you want?
Allow configuring an eksctl nodegroup for "docker-on-docker" use. If a nodegroup is configured in this way, eksctl will pass the --enable-docker-bridge to the AMI bootstrap.

[errordeveloper]

I believe this will be doable by allowing to customise /etc/docker/daemon.json, in the spirit similar to how we handle kubelet.yaml (see #886).

So you would be looking to do something like this:

nodeGroups:
  - name: ng1-with-docker-in-docker
     extraDockerConfig:
       bridge: docker0
       live-restore: false

To be clear, we don't use bootstrap.sh. For the reference, it does this:

# Replace with custom docker config contents.
if [[ -n "$DOCKER_CONFIG_JSON" ]]; then
    echo "$DOCKER_CONFIG_JSON" > /etc/docker/daemon.json
    systemctl restart docker
fi

if [[ "$ENABLE_DOCKER_BRIDGE" = "true" ]]; then
    # Enabling the docker bridge network. We have to disable live-restore as it
    # prevents docker from recreating the default bridge network on restart
    echo "$(jq '.bridge="docker0" | ."live-restore"=false' /etc/docker/daemon.json)" > /etc/docker/daemon.json
    systemctl restart docker
fi

I think we will want to have the config itself unse /etc/eksctl/docker.json, so that we own it in full and can ensure the content is deterministic. That will require a drop-in unit also, which is also something we already equipped to do.

[tobemedia]

Run in the same problem using eksctl to setup a binder / jupyterhub setup. The built in docke-on-docker build process fails due to dns errors. Is there a workaround to pass --enable-docker-bridge to eksctl commands?

[SkySonR]

Looks like it's possible. See https://github.com/weaveworks/eksctl/pull/522

[mhumeSF]

@SkySonR #522 ~looks specific to ubuntu ami and not eks amazon linux default ami's. Is there documentation on how this flag can get passed to bootstrap on eks-amazon-linux?~

Looks to be for both. Investigating on how to use...

[diestrin]

I'm having the same DNS problem. @SkySonR do you know how can we pass the argument for that PR you mention?

[Arau]

Same here. Is there a way to achieve this in the meantime that this is developed properly?
Can I edit the host template by hand or anything alike?

Thank you for the work and great project!

[tobemedia]

Using a workaround described at the Jupyter Forum I was able to fix the dns errors.
Create a new eks cluster using the linked config file as a starting point.

eksctl create cluster --config-file aws_eks_config.yml

Gist: aws_eks_config.yml

[antodoms]

@tobemedia Your solution worked! I was trying to build docker container using buildkite agent inside our Kubernetes cluster and it was throwing this error!

[kalbir]

I'm closing this given the workaround from @tobemedia above. We are unlikely to implement any other features for this in eksctl.

[duanyutong]

Why not implement something clean like --enable-docker-bridge? The above workaround (specifically preBootstrapCommandsin in the yaml config) is preventing my p3 nodes from joining the cluster when I call eksctl create nodegroup --config-file=nodegroups.yaml.