Skaffold: Pass --network=host during docker build

[romilpunetha]

When using skaffold build during jenkins, the docker build is failing because the containers aren't able to connect to the internet. However, I tried ssh-ing into the machines and ran docker build --network=host and it worked. Is there any provisioning to provide the --network flag to skaffold build ?

Expected behavior

skaffold build should succeed.

Actual behavior

The build is failing as no command with internet as a requirement is passing.

Information

• Skaffold version: v0.24.0
• Operating system: aws ami
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta3
kind: Config
build:
  tagPolicy:
    envTemplate:
      template: "{{.DOCKER_REGISTRY}}/romil-punetha/doorman:{{.VERSION}}"
  artifacts:
  - image: doorman
    context: .
    docker: {}
  local: {}
deploy:
  kubectl:
    manifests:
profiles:
- name: dev
  build:
    tagPolicy:
      envTemplate:
        template: "{{.DOCKER_REGISTRY}}/romil-punetha/doorman:{{.DIGEST_HEX}}"
    artifacts:
    - docker: {}
    local: {}
  deploy:
    helm:
      releases:
      - name: doorman
        chartPath: charts/doorman
        setValueTemplates:
          image.repository: "{{.DOCKER_REGISTRY}}/romil-punetha/doorman"
          image.tag: "{{.DIGEST_HEX}}"

Steps to reproduce the behavior

1. ... setup a jenkinsx cluster on EKS with nodes in private networking mode(theres a flag when using eksctl )
2. ... run a jenkins job with a command like apt-get update in the dockerfile

[corneliusweig]

This sounds more like an EKS configuration issue, than a problem in Skaffold. Besides, could you provide some debug log output?

[romilpunetha]

The output log is that the yarn install on my jenkins job fails to fetch files from the repository due to connectivity issue. I tried apt-get update inside the docker container and that failed for the same reason. I used docker build --network and everything worked fine.
I'm using jenkins-x and the build happens in containers. However, the build containers mount the docker.sock from the instance. The build containers have internet, but the containers being built do not (atleast not without the --network flag).

[corneliusweig]

Have you tried adjusting the mtu inside your containers? Using a too large mtu can lead to black-hole connections, if ICMP mtu discovery is blocked.

[moshid]

I'm having the same issue, when skaffold build during jenkins the container have no internet access and fail. I ended up using docker build --network=host instead of skaffold but I was wondering if anyone found a solution for this?

[romilpunetha]

@corneliusweig How to adjust mtu?

[corneliusweig]

@romil-punetha e.g. https://mlohr.com/docker-mtu/

[remzisenel]

Having the same issue.
skaffold.yaml used in build step:
https://gist.github.com/remzisenel/be6bbd52b43941ffba185ead8dc4adde
jenkinsfile used in build:
https://gist.github.com/remzisenel/3de44d740745d10d9675cf94b883f44d

When I use skaffold build -f skaffold.yaml, there is only the loopback interface within the container. However, when I use docker build --network=host . I have the regular interfaces and can access network as expected.

Edit: ip link output from the container when I use skaffold build -f skaffold.yaml

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

[corneliusweig]

@remzisenel Can you check what interfaces are set up when using docker build --network=bridge? This should be the default for docker, but what you are observing looks more like --network=none.

Can you also check if the network setting is configured in /etc/docker/daemon.json, or via arguments for the docker daemon (pgrep -a dockerd).

[moshid]

It is an EKS configuration issue they disabled the bridge network. Take a look at: https://github.com/awslabs/amazon-eks-ami/issues/183

[corneliusweig]

@moshid Thanks for demystifying that :man_facepalming:

So I would recommend to try enabling the bridge network again (see the issue found by @moshid). If that does not work, there is currently no way to pass this flag from skaffold to docker. However, you can try a very dirty hack to inject the flag: put this file as docker somewhere on your path:

#!/bin/bash
if [[ "$1" = 'build' ]]; then
  exec /path/to/real/docker build --network=host "${@:2}"
else
  exec /path/to/real/docker "$@"
fi

Then use the CLI docker in skaffold by

build:
  tagPolicy:
    envTemplate:
      template: "{{.DOCKER_REGISTRY}}/romil-punetha/doorman:{{.VERSION}}"
  artifacts:
  - image: doorman
    context: .
    docker: {}
  local:
    useDockerCLI: true   # <<<<<<<<<<<<<<<

[balopat]

Thank you for filing this, and for the workarounds!
I think this is a good candidate to implement as a pass through arg to the docker command / docker client (using the NetworkMode setting on ImageBuildOptions). Which can be of value bridge, host, none - wouldn't support container:<name|id>.

[namalk]

Docker version 19.03.9, build 9d98839

"--network=host" is the only solution.

[GodaProjects]

Docker version 19.03.9, build 9d98839

"--network=host" is the only solution.

I am having the same issue with the same version of docker. Does not build with bridge network. Only builds with host network. But docker run seems fine (containers are able to access internet). Is this a known issue?

[briandealwis]

@GodaProjects This sounds like an issue with your local configuration rather than a Skaffold issue.