Skaffold: `skaffold dev` doesn't clean up deployments on Windows

[mattpodolak]

Expected behavior

Skaffold cleans up all deployments when dev mode is exited.

Actual behavior

Skaffold doesn't clean up any deployments when dev mode is exited.

Information

After pressing Ctrl + c in Git Bash to exit dev mode, I see "Cleaning up..." in the logs, which exits immediately. All my services and pods are still running afterwards. I have installed Skaffold using chocolatey.

• Skaffold version: v1.14.0
• Operating system: Windows 10 Education (1909)
• Contents of skaffold.yaml:

apiVersion: skaffold/v2alpha3
kind: Config
deploy:
  kubectl:
    manifests:
      - ./infra/k8s/*
build:
  local:
    push: false # disable default push to docker hub
  artifacts:
    - image: mattpodolak/client
      context: client
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: 'src/**/*.js'
            dest: .
    - image: mattpodolak/comments
      context: comments
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: '*.js'
            dest: .
    - image: mattpodolak/event-bus
      context: event-bus
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: '*.js'
            dest: .
    - image: mattpodolak/moderation
      context: moderation
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: '*.js'
            dest: .
    - image: mattpodolak/posts
      context: posts
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: '*.js'
            dest: .
    - image: mattpodolak/query
      context: query
      docker:
        dockerfile: Dockerfile
      sync:
        manual:
          - src: '*.js'
            dest: .

Steps to reproduce the behavior

1. clone this repo: https://github.com/mattpodolak/mini-mern-ms
2. skaffold dev in Git Bash on Windows
3. Ctrl + c
4. kubectl get pods or kubectl get services

[nkubala]

thanks for opening @mattpodolak. we just closed an identical issue not long ago 😄 https://github.com/GoogleContainerTools/skaffold/issues/4058

cc @gsquared94 any idea? could you try reproing on your windows machine?

[mattpodolak]

I'm also facing this issue using Command Prompt

[gsquared94]

@nkubala I don't have my windows machine. The VM approach doesn't work anymore

[IsaacPD]

I was able to reproduce it on my windows machine by following the steps @mattpodolak posted. I'm using skaffold v1.15 on Windows 10 Pro.

[IsaacPD]

It is now happening that when I press Ctrl+C the program just terminates with no "Cleaning up..." message. And with -v trace I still do not see any logs after pressing Ctrl+C

[pabardina]

Hello, I am facing the same problem using skaffold v1.16. I got the 'cleaning up...' and nothing happens

[pabardina]

I did more tests... And I discovered something weird. I got the problem using Skaffold v1.16 installed by chocolatey. But I tried Skaffold built on my VM or using the binary from GitHub and it is working like a charm.

I guess there is a problem with the chocolatey build?

[IsaacPD]

Yeah just confirming @pabardina findings, I was also using the chocolatey build and just tested with the binary on github and it all works as expected so far.

+1 to potential issue with the chocolatey build

[IsaacPD]

Interestingly just by taking out the binary from C:/ProgramData/chocolatey/lib/skaffold/tools and copying/executing it locally it worked just fine for me. Somehow the mechanisms of how chocolatey executes commands from C:/ProgramData/chocolatey/bin get in the way and prevent skaffold from cleaning up.

My guess is that the skaffold.exe binary in chocolatey/bin (which isn't the actual skaffold binary) is eating the Ctrl+C input and killing skaffold in a way that it is not prepared to handle.

[IsaacPD]

From looking at https://github.com/chocolatey/shimgen/issues/32 it seems that my suspicions are justified in that this is not directly a result of skaffold behavior. The choco shim that runs skaffold.exe sends sigkill which just nukes skaffold.