Skaffold: Skaffold v0.23.0 + Windows 10 regression: didn't sync any files

[WoLfulus]

Expected behavior

Sync files when it changes

Actual behavior

Did not sync files. Always shows "didn't sync any files"

Information

• Skaffold version: v0.23.0
• Operating system: Windows 10
• Contents of skaffold.yaml:

Any yamls I tried

Steps to reproduce the behavior

1. Update to v0.23.0 from v0.22.0
2. Sync stops working

[priyawadhwa]

Hey @WoLfulus, I tried to repro this with our hot reload example and it worked as expected. Could you provide the skaffold.yaml you tried?

[jonaseck2]

Can confirm, compiled tag v0.22.0 works, master and release 0.23.0 doesn't

v0.23.0:
Syncing 1 files for some_node_app:d7db341f059c1ea8eaf66d3c0c41593e724860471c7fa2bf84486ad16c28775a
Watching for changes every 1s...

Files aren't updated in the image at all

v.0.22.0
Syncing 1 files for some_node_app:03050b781def4228b6b64f6865293434c633d24393f395448df4bee9d1e309d7
Watching for changes every 1s...

updated, and hot reload works

skaffold yaml:

apiVersion: skaffold/v1beta4
kind: Config
build:
  tagPolicy:
    # gitCommit tags the image with the git commit of your current repository.
    #gitCommit: {}
    # sha256 tags the image with the checksum of the built image (image id).
    sha256: {}
  artifacts:
  - image: some_java_api
    context: ./api
#    jibGradle: {}
    # Skaffold can sync local files with remote pods (alpha) instead
    # of rebuilding the whole artifact's image. This is a mapping
    # of local files to sync to remote folders.
    # sync:
    #   '*.py': .
  - image: some_node_app
    context: ./app
    sync:
      '**/*.js': .
    docker:
      buildArgs:
        NODE_ENV: development
      target: build-deps
# The test section has all the information needed to test images.
test:
    # For each image listed here, Skaffold will run a series of structure tests using
    # the [Container Structure Tests](https://github.com/GoogleContainerTools/container-structure-test)
    # project.
    # - image: gcr.io/k8s-skaffold/skaffold-example
    #   structureTests:
    #     - ./test/*

# The deploy section has all the information needed to deploy.
deploy:
  # The type of the deployment method can be `kubectl` (beta), `helm` (beta) or `kustomize` (beta).

  # The kubectl deployer uses  a client side `kubectl apply` to apply the manifests to the cluster.
  # You'll need a kubectl CLI version installed that's compatible with your cluster.
  kubectl:
    # manifests to deploy from files.
    manifests:
    - ./kubernetes/*
    # kubectl can be passed additional option flags either on every command (Global),
    # on creations (Apply) or deletions (Delete).
    # flags:
    #   global: [""]
    #   apply: [""]
    #   delete: [""]

    # manifests to deploy from remote cluster.
    # The path to where these manifests live in remote kubernetes cluster.
    # Example
    # remoteManifests:
    # - deployment/web-app1
    # - namespace:deployment/web-app2

 # helm:
    # helm releases to deploy.
    # releases:
    # - name: skaffold-helm
    #   chartPath: skaffold-helm
    #   valuesFiles:
    #   - first-values-file.yaml
    #   - second-values-file.yaml
    #   values:
    #     image: skaffold-helm
    #   namespace: skaffold
    #   version: ""
    #   recreatePods: false
    #   # set to true if you need to skip "helm dep build". Necessary for use with remote chart.
    #   skipBuildDependencies: false
    #
    #   # setValues get appended to the helm deploy with --set.
    #   setValues:
    #    key: "value"
    #
    #   # overrides builds an override values.yaml file to run with the helm deploy
    #   overrides:
    #     some:
    #       key: someValue
    #
    #   # packaged section allows to package chart setting specific version
    #   # and/or appVersion using "helm package" command.
    #   packaged:
    #     # version is passed to "helm package --version" flag.
    #     # Note that you can specify both static string or dynamic template.
    #     version: {{ .CHART_VERSION }}-dirty
    #     # appVersion is passed to "helm package --app-version" flag.
    #     # Note that you can specify both static string or dynamic template.
    #     appVersion: {{ .CHART_VERSION }}-dirty

# profiles (beta) section has all the profile information which can be used to override any build or deploy configuration
# profiles:
#   - name: gcb
#     build:
#       googleCloudBuild:
#         projectId: k8s-skaffold

[jeffshadap]

Skaffold v0.23.0 doesn't sync files . However v0.22.0 works fine
Tested with nodejs example

[balopat]

@jeffshadap
@jonaseck2
are you both testing on Windows?

[swoldanski]

Maybe this will be helpful:

Linux environment. I see that problem when my context namespace differ from a namespace defined for Helm chart, for example:

- context:
    cluster: minikube
    namespace: test
    user: minikube
  name: minikube
current-context: minikube

Case 1:

• there is no namespace defined in skaffold.yaml,
• there is test namespace defined in skaffold.yaml
  Result:
• sync working as expected

Case 2:

• there is namespace defined in Helm Chart with a different name then in context,
• there is staging namespace defined in skaffold.yaml
  Result:
• sync is not working at all

Where is a bug?
Well, probably skaffold do not use proper namespace after deployment to communicate with the cluster?

Workaround:
Always change context namespace to a namespace defined in skaffold.yaml or Helm chart before using skaffold.

[jeffshadap]

Yes on Windows 10

[jonaseck2]

Nope: MacOs Mojave 10.14.3

everything running in the default namespace

[laander]

Same here, no syncing after the update. Here's the console output (no error messages):

INFO[0048] files modified: [main.js]
Syncing 1 files for laander/myproject:a8db8a26762164d0d916ce0f7f29cd0db6388a5d74def8327003715c66daa368
INFO[0049] Copying files: map[main.js:main.js] to laander/myproject:a8db8a26762164d0d916ce0f7f29cd0db6388a5d74def8327003715c66daa368
DEBU[0049] Running command: [kubectl exec myproject-myproject-754cfb94cb-7cpqm --namespace default -c myproject -i -- tar xmf - -C / --no-same-owner]
Watching for changes every 1s...

Node.js app with nodemon refreshing upon file change

MacOS Mojave v10.14.3
Docker Desktop v2.0.2.0
Kubernetes v1.13.0

[faxg]

Same here .. no sync of changed files (despite CLI message).
docker-for-desktop on MacOS 10.14.2
no contexts defined in skaffold.yaml
no default repo (unset)

[patnaikshekhar]

I'm facing the same issue with skaffold running on windows syncing js files. The files seem to be going to the root folder instead of the WORKDIR.

[sfxworks]

Hi! This issue occurs with our config for a user on MacOS 10.14.6

apiVersion: skaffold/v1beta14
kind: Config
build:
  tagPolicy:
    sha256: {}
  artifacts:
    - image: our-image
      context: php
      sync:
        manual:
        - src: '**/*.php'
          dest: /app
        - src: '**/*.yaml'
          dest: /app
        - src: '**/*.twig'
          dest: /app
      custom:
        buildCommand: "pwsh -file build.ps1"
        dependencies:
          paths:
            - .
          ignore:
            - README*
    - image: another-image
      context: frontend
      sync:
        manual:
        - src: '**/*.png'
          dest: /usr/share/nginx/html
        - src: '**/*.jpg'
          dest: /usr/share/nginx/html
        - src: '**/*.ico'
          dest: /usr/share/nginx/html
        - src: '**/*.gif'
          dest: /usr/share/nginx/html
        - src: '**/*.svg'
          dest: /usr/share/nginx/html
      custom:
        buildCommand: "pwsh -file build.ps1"
        dependencies:
          paths:
            - .
          ignore:
            - README*
deploy:
  kubectl:
    manifests:
      - cicd/k8s/*

[tejal29]

@sfxworks Are you using the latest version of skaffold?

[sfxworks]

@tejal29 Yes, we just tried 0.40.0 and the bleeding edge binary. Same results.

[sfxworks]

https://gist.github.com/sfxworks/a6541a5767889bb44915393f071807df

Above is a skaffold dev -v debug --port-forward for the dev with the dev doing a file modification to one that is suppose to be synced.

To clarify, this is a bit different from the poster in that there is never a Syncing 1 files... notice. This is on MacOs Mojave 10.14.6 though. On other user's Windows 10 machines this works just fine.

[tejal29]

Since this is working on windows and specific to mac Mojave, decreasing its priority to p2.

[sfxworks]

This issue seems to have been fixed with the latest Catalina update.

[dgageot]

Can someone here share a small project to reproduce? That would really help understand what's going on

[lloydchang]

Is the recommended work-around to upgrade from Mojave to Catalina?

[dgageot]

A workaround is to run skaffold dev with --trigger=polling

[pre]

Every once in a while, I have been experiencing WARN[0032] Skipping deploy due to sync error: copying files: didn't sync any files . This comment was a key for me: https://github.com/GoogleContainerTools/skaffold/issues/1668#issuecomment-466633791

I have Minikube 1.7.3, Kustomize 3.5.4 and Skaffold 1.4.0 with multiple applications in different Kubernetes namespaces.

PROBLEM: Namespace in $KUBECONFIG affects Skaffold Sync

Skaffold sync will only work when the Kubernetes namespace in $KUBECONFIG, at the time of starting Skaffold, matches the namespace of the artifact (application) being deployed.

The critical moment is when Skaffold starts. After Skaffold is running, I can change my current namespace in $KUBECONFIG and Skaffold sync works neatly. But, if the namespace in $KUBECONFIG is different from the hot-sync-artifact's namespace when Skaffold starts, then sync will fail with error WARN[0032] Skipping deploy due to sync error: copying files: didn't sync any files .

For Skaffold Sync to work, the namespace in $KUBECONFIG must either be undefined or it must equal to the hot-sync-artifact's namespace.

EXAMPLE: Namespace undefined

Without namespace being defined, Skaffold Sync works for artifacts in any namespace:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/pre/.minikube/ca.crt
    server: https://192.168.64.100:8443
  name: rex
contexts:
- context:
    cluster: rex
    user: rex
  name: rex
current-context: rex
kind: Config
preferences: {}
users:
- name: rex
  user:
    client-certificate: /Users/pre/.minikube/client.crt
    client-key: /Users/pre/.minikube/client.key

EXAMPLE: Namespace defined

Here namespace is defined and Skaffold Sync will fail for all artifacts (applications) which are not deployed in dev namespace:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/pre/.minikube/ca.crt
    server: https://192.168.64.100:8443
  name: rex
contexts:
- context:
    cluster: rex
    namespace: dev
    user: rex
  name: rex
current-context: rex
kind: Config
preferences: {}
users:
- name: rex
  user:
    client-certificate: /Users/pre/.minikube/client.crt
    client-key: /Users/pre/.minikube/client.key

[pre]

Here's a workaround that fixes Skaffold Sync at least for me.

Wrap skaffold dev in a script which

• first resets the current namespace,
• starts a subshell in the background that'll switch back into the namespace you were,
• and runs skaffold dev

_dev() {
  local current_ns; current_ns="$(kubectl config view --minify --output 'jsonpath={..namespace}')"
  kubectl config set-context --current --namespace=
  (sleep 3 && kubectl config set-context --current --namespace="${current_ns}" ) &

  skaffold dev
}

You may need to adjust sleep 3 for your environment.

This problem still exists in Skaffold 1.5.0.

Fyi @dgageot here's the small repro that you asked for.

[tstromberg]

Marking as stale because we haven't heard from the author in over a year and this specific regression has long since passed.

If you run into sync problems, please open a new issue with a clear reproduction case, and please mention the operating system version. Thank you!