Skaffold: Skaffold dev does not rebuild or deploy after file changes (macOS Catalina)

[dahovey]

Expected behavior

Running skaffold dev should rebuild docker images and re-deploy after local file changes.

Actual behavior

After first deployment, and after making any local file changes nothing happens.

Information

• Skaffold version: 0.40.0
• Operating system: macOS Catalina 10.15

Steps to reproduce the behavior

1. Cloned this repository.
2. cd into an examples folder
3. run skaffold dev.

I tried helm-deployment and microservices samples. Making change to change to any local file doesn't trigger new build.

Deploy complete in 990.395637ms
Watching for changes...

[dahovey]

I discovered that if run skaffold dev as root, skaffold properly rebuilds/re-deploys on file changes. Appears to be disk access permission issue with skaffold and Catalina but not sure how to resolve.

I cloned skaffold repository into ~/projects/temp/skaffold.

[balopat]

Thanks for opening this @dahovey - can you provide some logs for us with -v trace if you can reproduce it?

[nkubala]

@dahovey i'm also curious if running skaffold dev --trigger polling fixes your issue

[dahovey]

Ran below from ~/projects/temp/skaffold/examples/nodejs
So it appears skaffold is not working when:

• Watching files that should be synced
• Watching files to trigger re-build/deploy

If I run skaffold dev --trigger polling rebuild and deploy works as expected

Hope the below helps in some way.

DEBU[0000] found open port: 50053
INFO[0000] starting gRPC server on port 50053
DEBU[0000] found open port: 50054
INFO[0000] starting gRPC HTTP server on port 50054
INFO[0000] Skaffold &{Version:v0.40.0 ConfigVersion:skaffold/v1beta16 GitVersion: GitCommit:4058d66b191038f4c3ac27ea2433a2dfccd40c5e GitTreeState:clean BuildDate:2019-10-11T20:39:46Z GoVersion:go1.12 Compiler:gc Platform:darwin/amd64}
DEBU[0000] Defaulting build type to local build
DEBU[0000] Defaulting deploy type to kubectl
DEBU[0000] validating yamltags of struct SkaffoldConfig
DEBU[0000] validating yamltags of struct Metadata
DEBU[0000] validating yamltags of struct Pipeline
DEBU[0000] validating yamltags of struct BuildConfig
DEBU[0000] validating yamltags of struct Artifact
DEBU[0000] validating yamltags of struct Sync
DEBU[0000] validating yamltags of struct SyncRule
DEBU[0000] validating yamltags of struct ArtifactType
DEBU[0000] validating yamltags of struct DockerArtifact
DEBU[0000] validating yamltags of struct TagPolicy
DEBU[0000] validating yamltags of struct GitTagger
DEBU[0000] validating yamltags of struct BuildType
DEBU[0000] validating yamltags of struct LocalBuild
DEBU[0000] validating yamltags of struct DeployConfig
DEBU[0000] validating yamltags of struct DeployType
DEBU[0000] validating yamltags of struct KubectlDeploy
DEBU[0000] validating yamltags of struct KubectlFlags
INFO[0000] Using kubectl context: docker-desktop
DEBU[0000] Using builder: local
DEBU[0000] setting Docker user agent to skaffold-v0.40.0
DEBU[0000] push value not present, defaulting to false because localCluster is true
Listing files to watch...
 - gcr.io/k8s-skaffold/node-example
TRAC[0000] Checking base image node:10.15.3-alpine for ONBUILD triggers.
DEBU[0000] Found dependencies for dockerfile: [{package-lock.json /app true} {package.json /app true} {. /app true}]
List generated in 20.540364ms
Generating tags...
 - gcr.io/k8s-skaffold/node-example -> DEBU[0000] Running command: [git describe --tags --always]
DEBU[0000] Command output: [v0.40.0-114-g357d6cf3
]
DEBU[0000] Running command: [git status . --porcelain]
DEBU[0000] Command output: [ M examples/nodejs/backend/src/index.js
]
gcr.io/k8s-skaffold/node-example:v0.40.0-114-g357d6cf3-dirty
Tags generated in 106.971628ms
Checking cache...
TRAC[0000] Checking base image node:10.15.3-alpine for ONBUILD triggers.
DEBU[0000] Found dependencies for dockerfile: [{package-lock.json /app true} {package.json /app true} {. /app true}]
 - gcr.io/k8s-skaffold/node-example: Found Locally
Cache check complete in 32.103898ms
Tags used in deployment:
 - Since images are not pushed, they can't be referenced by digest
   They are tagged and referenced by a unique ID instead
 - gcr.io/k8s-skaffold/node-example -> gcr.io/k8s-skaffold/node-example:7184f1161455ad9e348e6bf112892d9b4c738aaa28e8fec0b0cbf648e57515a9
Starting deploy...
DEBU[0000] Running command: [kubectl version --client -ojson]
DEBU[0000] Command output: [{
  "clientVersion": {
    "major": "1",
    "minor": "14",
    "gitVersion": "v1.14.7",
    "gitCommit": "8fca2ec50a6133511b771a11559e24191b1aa2b4",
    "gitTreeState": "clean",
    "buildDate": "2019-09-18T14:47:22Z",
    "goVersion": "go1.12.9",
    "compiler": "gc",
    "platform": "darwin/amd64"
  }
}
]
DEBU[0000] Running command: [kubectl --context docker-desktop create --dry-run -oyaml -f /Users/davidhovey/projects/temp/skaffold/examples/nodejs/k8s/deployment.yaml]
DEBU[0000] Command output: [apiVersion: v1
kind: Service
metadata:
  name: node
  namespace: default
spec:
  ports:
  - port: 3000
  selector:
    app: node
  type: LoadBalancer
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node
  namespace: default
spec:
  selector:
    matchLabels:
      app: node
  template:
    metadata:
      labels:
        app: node
    spec:
      containers:
      - image: gcr.io/k8s-skaffold/node-example
        name: node
        ports:
        - containerPort: 3000
]
DEBU[0000] manifests with tagged images: apiVersion: v1
kind: Service
metadata:
  name: node
  namespace: default
spec:
  ports:
  - port: 3000
  selector:
    app: node
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: node
  namespace: default
spec:
  selector:
    matchLabels:
      app: node
  template:
    metadata:
      labels:
        app: node
    spec:
      containers:
      - image: gcr.io/k8s-skaffold/node-example:7184f1161455ad9e348e6bf112892d9b4c738aaa28e8fec0b0cbf648e57515a9
        name: node
        ports:
        - containerPort: 3000
DEBU[0000] manifests with labels apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/managed-by: skaffold-v0.40.0
    skaffold.dev/builder: local
    skaffold.dev/cleanup: "true"
    skaffold.dev/deployer: kubectl
    skaffold.dev/docker-api-version: "1.40"
    skaffold.dev/run-id: 9e00e367-afff-47f7-b7fe-28a29b9a753d
    skaffold.dev/tag-policy: git-commit
    skaffold.dev/tail: "true"
  name: node
  namespace: default
spec:
  ports:
  - port: 3000
  selector:
    app: node
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/managed-by: skaffold-v0.40.0
    skaffold.dev/builder: local
    skaffold.dev/cleanup: "true"
    skaffold.dev/deployer: kubectl
    skaffold.dev/docker-api-version: "1.40"
    skaffold.dev/run-id: 9e00e367-afff-47f7-b7fe-28a29b9a753d
    skaffold.dev/tag-policy: git-commit
    skaffold.dev/tail: "true"
  name: node
  namespace: default
spec:
  selector:
    matchLabels:
      app: node
  template:
    metadata:
      labels:
        app: node
        app.kubernetes.io/managed-by: skaffold-v0.40.0
        skaffold.dev/builder: local
        skaffold.dev/cleanup: "true"
        skaffold.dev/deployer: kubectl
        skaffold.dev/docker-api-version: "1.40"
        skaffold.dev/run-id: 9e00e367-afff-47f7-b7fe-28a29b9a753d
        skaffold.dev/tag-policy: git-commit
        skaffold.dev/tail: "true"
    spec:
      containers:
      - image: gcr.io/k8s-skaffold/node-example:7184f1161455ad9e348e6bf112892d9b4c738aaa28e8fec0b0cbf648e57515a9
        name: node
        ports:
        - containerPort: 3000
DEBU[0000] 2 manifests to deploy. 2 are updated or new
DEBU[0000] Running command: [kubectl --context docker-desktop apply -f - --force]
 - service/node created
 - deployment.apps/node created
Deploy complete in 1.035500329s
DEBU[0001] getting client config for kubeContext: ``
DEBU[0001] Change detected <nil>
TRAC[0001] Checking base image node:10.15.3-alpine for ONBUILD triggers.
DEBU[0001] Found dependencies for dockerfile: [{package-lock.json /app true} {package.json /app true} {. /app true}]
Watching for changes...
TRAC[0002] Checking base image node:10.15.3-alpine for ONBUILD triggers.
DEBU[0002] Found dependencies for dockerfile: [{package-lock.json /app true} {package.json /app true} {. /app true}]
INFO[0003] Streaming logs from pod: node-59566fc5d4-blbbf container: node
DEBU[0003] Running command: [kubectl --context docker-desktop logs --since=4s -f node-59566fc5d4-blbbf -c node --namespace default]
[node-59566fc5d4-blbbf node]
[node-59566fc5d4-blbbf node] > [email protected] dev /app
[node-59566fc5d4-blbbf node] > nodemon src/index.js
[node-59566fc5d4-blbbf node]
[node-59566fc5d4-blbbf node] [nodemon] 1.18.7
[node-59566fc5d4-blbbf node] [nodemon] to restart at any time, enter `rs`
[node-59566fc5d4-blbbf node] [nodemon] watching: *.*
[node-59566fc5d4-blbbf node] [nodemon] starting `node src/index.js`
[node-59566fc5d4-blbbf node] Example app listening on port 3000!

[dahovey]

Also I can consistently reproduce this with all projects I my machine.
For the time being is it possible to set the default trigger?

[dahovey]

I upgraded to v1.0.0 but issue still persists. Any guidance if issue might be something with my setup?

[dahovey]

Figured out the issue! It has to do with the case of the original folder compared to what was typed in the terminal. Originally I created the folder as ~/Projects but when cd'ing into folder in the terminal I would I would use cd ~/projects/....

I was able to use the below simple code to isolate this as NOT a skaffold specific problem. If using the below code and there are differences in case (created folder case vs. current directory), no file watcher events are triggered.

package main

import (
    "fmt"
    "github.com/rjeczalik/notify"
)

func main() {
    c := make(chan notify.EventInfo, 100)

    if err := notify.Watch("./...", c, notify.All); err != nil {
        fmt.Println("Error watching directory")
        return
    }

    for {
        fmt.Println("Wating for changes...")

        select {
        case e := <-c:
            fmt.Println("Change detected", e)
        }
    }
}

Coming from a Windows background, and being that MacOS doesn't complain about the difference in case I didn't even notice the difference.

I would consider this a bug with the rjeczalik/notify package since there isn't any feedback and macOS allows the current directory to be different than the created directory. I will create an issue in that repository.

[nkubala]

@dahovey glad you were able to dig into this issue, and sorry there's nothing we can really do for you here! going to close this now as it's not a skaffold specific issue.