Skaffold: Kaniko builds won't complete with ISTIO sidecar

[jlewi]

Expected behavior

Running skaffold build will build the container using kaniko as configured in skaffold.yaml.

When the kaniko container is complete; skaffold will detect the container as finished and exit.

Actual behavior

• I'm running Kaniko in a namespace with ISTIO side car injection turned on.
• The kaniko container completes but the ISTIO side car remains running
• kubectl get pods shows 1/2 of pods running confirming that kaniko container completed but istio
  side car is still running
• skaffold continues to wait and eventually prints out a fatal error message
  
  
  
  • (I forgot to copy the logs)
  
  
• Changing to a namespace with ISTIO side car injection disabled the build completes successfully

My conjecture is that Skaffold's kaniko code is assuming there is a single container running kaniko and thus fails in the presence of side cars like the ISTIO side car.

• It would be great if skaffold could be updated to support sidecars

  • Since kaniko is making network calls (e.g. to GCR) it seems reasonable to support running that with ISTIO to provide various security controls

• It might also be nice if skaffold.yaml supported adding labels to the kaniko pod so the istio side
  car could be disabled on just the kaniko jobs

Information

• Skaffold version: v1.1.0
• Contents of skaffold.yaml:

 Reference: https://skaffold.dev/docs/references/yaml/
apiVersion: skaffold/v2alpha1
kind: Config
metadata:
  name: label-microservice
build:
  artifacts:
  - image: gcr.io/issue-label-bot-dev/bot-worker
    # Set the context to the root directory. 
    # All paths in the Dockerfile should be relative to this one.
    context: ..
    kaniko:
      dockerfile: Label_Microservice/deployment/Dockerfile.worker
      buildContext:
        gcsBucket: issue-label-bot-dev_skaffold-kaniko
      env: 
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /secret/user-gcp-sa.json
      cache: {}
  cluster:
    # pullSecret can be set to a local file from which the pull secret should be created.
    pullSecretName: user-gcp-sa
    # TODO(jlewi): This should be changed for each developer; or maybe we should create a reusable one?
    namespace: jlewi-dev
deploy:
  kustomize:
    path: deployment/overlays/dev

Steps to reproduce the behavior

1. Setup a cluster with ISTIO
2. Try to run skaffold build using Kaniko in a namespace with ISTIO side car injection turned on

[dgageot]

Hi @jlewi could you share the output of skaffold build -vdebug. It could maybe help understand what's going on?

[dgageot]

The code here will indeed wait for every container in the kaniko pod to succeed.

[jlewi]

@dgageot do you still need me to run skaffold build -vdebug ? My skaffold project is here
https://github.com/jlewi/code-intelligence/blob/multi_model/Label_Microservice/skaffold.yaml

[dgageot]

No thanks. That’s ok!

[nkubala]

@dgageot it looks like the file you linked originally got updated so I can't see the code snippet where we wait for every container in the pod to succeed 😆

I do think this is still an issue, but I'm not seeing a lot of traction on it so I'm going to knock the priority down since I don't think our team will work on it soon. if people are feeling a lot of pain from this, please speak up in this issue so we can hear you!

[jlewi]

@nkubala Would it be possible to support istio side car disablement in the kaniko jobs? The sidecar can be disabled on specific jobs by adding an appropriate annoation
https://istio.io/docs/setup/additional-setup/sidecar-injection/#policy

If skaffold supported adding annotations to the kaniko jobs this would be easy to do.

This would be an improvement over the current work-around of creating a separate namespace with istio side car injection just for the kaniko builds. That work around is annoying because it requires duplicating namespace setup (e.g. secrets) just for builds.

[nkubala]

@jlewi if this can be accomplished through annotations then I don't see why we couldn't support this, in fact there's a PR open right now that might give you what you need: https://github.com/GoogleContainerTools/skaffold/pull/4280.

[erikkrieg]

Is it known when #4280 will be released? Is it likely to be included in the next release?

[nkubala]

@erikkrieg yep! barring any delays it'll go out with v1.11.0 on thursday afternoon here on the west coast.

[tejal29]

This is closed by #4280