Skaffold: Hot reload crashes the skaffold dev

[dnmahendra]

service "client-cluster-ip-service" created
deployment.apps "client-deployment" created
persistentvolumeclaim "database-persistent-volume-claim" created
service "elasticsearch-cluster-ip-service" created
configmap "elasticsearch-config" created
deployment.apps "elasticsearch-deployment" created
ingress.extensions "ingress-service" created
persistentvolume "elasticsearch-volume" created
service "server-cluster-ip-service" created
deployment.apps "server-deployment" created
Deploy complete in 703.788038ms
Watching for changes every 1s...
Port Forwarding client-deployment-c9c5bf696-qvqfr/client 4000 -> 4000
Port Forwarding server-deployment-8648665747-jc696/server 80 -> 80
[client-deployment-c9c5bf696-qvqfr client] yarn run v1.13.0
[client-deployment-c9c5bf696-qvqfr client] warning package.json: No license field
[client-deployment-c9c5bf696-qvqfr client] $ ./node_modules/.bin/nodemon --config nodemon.json server.js
[server-deployment-8648665747-jc696 server] yarn run v1.13.0
[server-deployment-8648665747-jc696 server] warning package.json: No license field
[server-deployment-8648665747-jc696 server] $ ./node_modules/.bin/nodemon --config nodemon.json server.js
[client-deployment-c9c5bf696-qvqfr client] [nodemon] 1.18.10
[client-deployment-c9c5bf696-qvqfr client] [nodemon] to restart at any time, enter `rs`
[client-deployment-c9c5bf696-qvqfr client] [nodemon] watching: *.*
[client-deployment-c9c5bf696-qvqfr client] [nodemon] starting `./node_modules/.bin/babel-node server.js`
[server-deployment-8648665747-jc696 server] [nodemon] 1.18.10
[server-deployment-8648665747-jc696 server] [nodemon] to restart at any time, enter `rs`
[server-deployment-8648665747-jc696 server] [nodemon] watching: *.*
[server-deployment-8648665747-jc696 server] [nodemon] starting `./node_modules/.bin/babel-node server.js`
[server-deployment-8648665747-jc696 server] Mon, 11 Mar 2019 05:47:06 GMT agentkeepalive deprecated options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead at src/libs/elasticSearch.js:23:12
[client-deployment-c9c5bf696-qvqfr client] Mon, 11 Mar 2019 05:47:06 GMT agentkeepalive deprecated options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead at src/libs/elasticSearch.js:23:12
[server-deployment-8648665747-jc696 server] 🚀 Server ready at http://localhost:80 in development mode
[client-deployment-c9c5bf696-qvqfr client] 🚀 Server ready at http://localhost:80 in development mode
Cleaning up...
service "client-cluster-ip-service" deleted
deployment.apps "client-deployment" deleted
persistentvolumeclaim "database-persistent-volume-claim" deleted
service "elasticsearch-cluster-ip-service" deleted
configmap "elasticsearch-config" deleted
deployment.apps "elasticsearch-deployment" deleted
ingress.extensions "ingress-service" deleted
persistentvolume "elasticsearch-volume" deleted
service "server-cluster-ip-service" deleted
deployment.apps "server-deployment" deleted
Cleanup complete in 3.576387194s
FATA[0151] calling final callback: sync: retrieving working dir for swift-frontend:19a2bdc0cf768010013b66322e2de91e6897ba40788c54fba28138c0c1557bb1: retrieving remote config: UNAUTHORIZED: "authentication required"

Expected behavior

Skaffold should watch and hot reload the source code changes

Actual behavior

Skaffold crashes immediately after changing the code.

Information

• Skaffold version: v0.24.0
• Operating system: MacOS
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta6
kind: Config
build:
  artifacts:
    - image: swift-frontend
      context: ../swift-client
      sync:
        "**/*.js": .
    - image: swift-backend
      context: ../swift-server
      sync:
        "**/*.js": .
deploy:
  kubectl:
    manifests:
      - "./k8s/*"

Steps to reproduce the behavior

1. You can clone the project here, https://github.com/dnmahendra/skaffold-example
2. You can run cd rc-devops && skaffold dev -f skaffold.yaml
3. Change the code either in the client or server folder to see the issue.

Not sure if I missing something here, any help would be much appreciated.

[dgageot]

This should be fixed in master branch with https://github.com/GoogleContainerTools/skaffold/pull/1735
Could you give it a try?

[dnmahendra]

Yes, sync doesn't crash now. But some reason it doesn't serve the changed code. I need to restart skaffold to see the changes.

[stfnsch]

I had the same issue with the sync crashing and the current master branch build fixed it for me.

But the sync process is not working correctly with subdirectories. I can see a file from say "./src/app/test.js" is copied to "./test.js" in the working directory of the pod. However, I expect the file to be placed within the exact same subdirectory.

[priyawadhwa]

@stfnsch sync patterns should match Docker COPY rules, so to copy a file into a specific subdirectory you'll have to specify it like this in the skaffold.yaml:

sync:
  '*.js' : '/src/app'

We recently found a bug in this & fixed it in the latest release, which may be why your files are being synced to unexpected (but, hopefully, now correct) locations.

[stfnsch]

@priyawadhwa Thanks for your answer. So, if I have a javascript application with say 10 subdirectories (src/app/module1/, src/app/module2/, src/app/module3/, ...), do I need to setup the sync pattern for each of these directories to have the files synced to correct locations?

sync:
  'src/app/module1/*.js' : 'src/app/module1'
  'src/app/module2/*.js' : 'src/app/module2'
  'src/app/module3/*.js' : 'src/app/module3'
  # ...

Or is there a simpler way for this? What I don't like is that you have to add a sync pattern every time you create a new subdirectory (module) in your application.

[priyawadhwa]

@stfnsch unfortunately that's correct, you'll need a sync pattern for each module to sync to the correct location.

If you specify a relative path in the skaffold.yaml, it will be preprended by the current working directory of the image to generate an absolute path. You could simplify the skaffold.yaml a bit by changing the working directory in the image to /src/app in the Dockerfile:

WORKINGDIR /src/app

and then in the skaffold.yaml:

sync:
  'src/app/module1/*.js' : 'module1'
  'src/app/module2/*.js' : 'module2'
  'src/app/module3/*.js' : 'module3'

[Torsten85]

@priyawadhwa so there is no way of syncing a complete path with folder structure? We work on very large projects with 50+ directories in ./src and way more subdirectories inside. If something like sync: 'src: 'src' is not possible, we can't use skaffold any longer.

[corneliusweig]

@Torsten85 Subtree sync is already in the current master, also see #1837 for details.

[nkubala]

@corneliusweig is correct, subtree sync works today in v0.26.0. you should be able to do something like

sync:
  'src/app/***': src/app

and files should end up in their subdirectories where you'd expect. @dnmahendra @Torsten85 @stfnsch can you try on your projects and see if things are working for you?

[stfnsch]

@nkubala It is working nicely! I really like this sync option. However, I think there is a small difference in terms of performance/speed between:

sync:
  'src/app/***': src/app

and:

sync:
  'src/app/module1/*': src/app/module1

[nkubala]

interesting, how small are we talking? :smile: i'd love to see some performance benchmarking around both sync and the watcher.

it's worth mentioning that sync is currently going through a potential redesign from @corneliusweig, see #1844 for the full proposal and discussion. for now I think this issue is resolved though, so I'll close it.