Nodemon: no automatic restart when using boot2docker docker volume

[chrishiestand]

nodemon doesn't seem to restart node when using a docker volume, or in this case a boot2docker ( os x -> virtualbox -> docker) volume. nodemon starts okay, but then fails to restart node when a watched file has been modified in the shared OS X volume.

root@760d5d2ff8cf:/src# nodemon --watch index.js 
20 Nov 21:06:58 - [nodemon] v1.2.1
20 Nov 21:06:58 - [nodemon] to restart at any time, enter `rs`
20 Nov 21:06:58 - [nodemon] watching: index.js
20 Nov 21:06:58 - [nodemon] starting `nodejs index.js`
Running on http://localhost:8080
request received
#index.js changed and saved here but output does not change as it should
request received
request received

nodemon -L causes restarts at least once a second even when no files have changed - way too frequently.

forever -w seems to work just fine. That seems like the best workaround for now.

output of dump:

root@b36ab2778133:/src# nodemon --dump
20 Nov 22:08:27 - [nodemon] v1.2.1
20 Nov 22:08:27 - [nodemon] to restart at any time, enter `rs`
20 Nov 22:08:27 - [nodemon] watching: *.*
--------------
node: v0.10.25
nodemon: v1.2.1
command: node /usr/local/bin/nodemon --dump
cwd: /src
OS: linux x64
--------------
{ run: false,
  system: 
   { cwd: '/src',
     useFind: false,
     useWatch: true,
     useWatchFile: false },
  required: false,
  dirs: [ '/src' ],
  timeout: 1000,
  options: 
   { scriptPosition: 0,
     dump: true,
     script: 'index.js',
     args: [],
     ignore: 
      [ '.git',
        'node_modules',
        'bower_components',
        '.sass-cache',
        re: /\.git|node_modules|bower_components|\.sass\-cache/ ],
     watch: [ '*.*', re: /.*\..*/ ],
     restartable: 'rs',
     execMap: { py: 'python', rb: 'ruby' },
     stdin: true,
     verbose: false,
     stdout: true,
     execOptions: 
      { script: 'index.js',
        exec: 'node',
        args: [],
        nodeArgs: undefined,
        ext: 'js',
        env: {},
        execArgs: [] },
     ext: 'js',
     monitor: 
      [ '*.*',
        '!.git',
        '!/src/node_modules/**/*',
        '!bower_components',
        '!.sass-cache' ] },
  load: [Function],
  reset: [Function: reset],
  lastStarted: 0,
  loaded: [],
  command: 
   { raw: { executable: 'node', args: [ 'index.js' ] },
     string: 'node index.js' } }

[chisleu]

When I use boot2docker with my branch off of library/node, when I try to use nodemon /path/to/app --watch, the cpu goes nuts like it is constantly restarting, although I can't verify.

I tried -L, but it also caused the CPU to go nuts.

I changed to pm2 for now, but it has 2-3 seconds of delay before it restarts. Still, better than nothing.

[tanis2000]

+1 this happens to me as well with a mounted volume on boot2docker (OSX + VirtualBox).

[remy]

I'm marking as "help wanted" - I don't have the infrastructure to even begin testing and debugging this.

I'm sure it's linked to clocks being out of whack somehow, but can't replicate myself with simple tests.

[tanis2000]

@remy if I can help you in any way, just let me know what I should do.

[remy]

@tanis2000 if you can pull the project inside of your virtual machine. I'd start by trying to discover why -L sets the CPU insane. The watching logic is all in watch.js - it's a little complicated because there's multiple methods to actually watch.

Really you want to get a handle on what's causing the issue in the first place. I usually console log the crap out of the code when I'm spelunking for errors.

[mattapperson]

still not seeing an automatic restart with a volume mounted nodejs project in b2d

[mrmurphy]

Remy, thanks for nodemon. Did you close this issue because you resolved it? Or did you not have enough helpful feedback to fix?

We've been experiencing the same problems as mentioned above.

[remy]

No, it looks like it was closed by mistake in the associated PR. I'll reopen and ask for help (from you lot!).

[remy]

If anyone wants to push a Docker image that I can test and debug this with – I'd be grateful.

[olalonde]

:+1: same problem here

[remy]

@olalonde then same question to you: "[do you have] a Docker image that I can test and debug this with – I'd be grateful.".

[chisleu]

I will make one if someone else doesn't. I can't get to it until tonight though, which is why I had not yet replied.

Assuming Ubuntu 12.04 is ok unless I hear otherwise.

A faster solution would be to use vagrant. If you have vagrant and virtualbox, you can install a VM with

vagrant up ubuntu/precise64 --provider virtualbox
vagrant ssh

Then install node/npm and nodemon. Place a node file in a shared resource (such as /vagrant) and you should find these problems to be easy to reproduce.

I'm not sure about docker as I no longer use docker for development (because of this issue, and because I use MacOSX and the boot to docker image causes CPU and sleep problems for my little Macbook Air.)

It still affects my work in Vagrant and seems to have the same results.

BTW, PM2 suffers a similar fate now in Vagrant. It simply fails to restart at all.

[olalonde]

Ok, I just made a test but can't push it to Github at the moment.

Install boot2docker. Run $(boot2docker shellinit).

Put all the following files in the same directory (e.g. nodemontest).

./Dockerfile

FROM node:0.10-slim
RUN mkdir /app
WORKDIR /app

CMD ["./node_modules/nodemon/bin/nodemon.js", "index.js"]

./index.js

console.log(new Date());

./package.json

{
  "name": "testnodemon",
  "devDependencies": {
    "nodemon": "^1.2.1"
  }
}

Once, you have those files, run the following commands.

npm install;
docker build -t testnodemon .;
docker run -v $(pwd):/app testnodemon;

Edit index.js, nodemon doesn't restart the script.

You can use the following to get a shell in the container:

docker run -i -t -v $(pwd):/app testnodemon /bin/bash;
root@762c18df76ee:/app# mount | grep app
none on /app type vboxsf (rw,nodev,relatime)

Related? https://www.virtualbox.org/ticket/10660

[remy]

@olalonde thanks for this. I'll go through the steps this week to try to replicate. I'm sure it's linked the latest information in this issue #11 but I need to see it first hand.

[olalonde]

The problem I believe is that vboxsf doesn't support inotify notifications. The alternative would be to poll the directory but that could be end up eating all the CPU on large code bases. Another solution which I believe was adopted by Ruby's guard is to have a client on the host machine which sends restart messages to a another process running on the guest machine.

Anyways, I hope you can come up with something better :)

[chisleu]

It is such a problem to resolve that we started running node locally with a remote Cassandra DB (since C* would be the only service in docker.)

[olalonde]

Another solution would be to watch files normally on the host and whenever a file changes, write to .file-changed. The nodemon running in the container could just tail .file-changed and reload the process whenever it receives new data.

[olalonde]

I just submitted a PR with the solution I posted above. It's not elegant, but it's simple and it works.

Basically, nodemon needs to run on the host with the --local flag:

nodemon --local

and the VM/container can listen for file change events by using the --remote flag:

nodemon --remote index.js

The local nodemon communicates with the remote one by toggling a file called .nodemon-ping

[remy]

Can you try the development version of nodemon that I've pushed? It contains a lot of other tweaks, but the main thing I've changed is it'll use find instead of fs.watchFile (which I _think_ uses the broken IO notify...).

To install:

npm install -g nodemon@dev

Can someone quickly give me an idea of whether that works or not? I tried it myself, but I'm still pretty new to docker so wasn't quite sure of what I was doing...

[remy]

(it should be [email protected])

[remy]

Nope, okay, I'm pretty sure [email protected] farts and makes a mess in docker. Anyone else want to confirm that's cool, otherwise I'll take this as a fail.

[remy]

Shoot, the issue I'm seeing is this: #454.

[remy]

Sod. I've fixed up all the issues, but now the clock is the main issue. The container says it's 9am, and current time is 10pm, so when I change on my local machine, the container doesn't get the file change correctly, and actually thinks everything is changing all the time...

[olalonde]

The container says it's 9am, and current time is 10pm

Yes I recall having a similar problem (files modified in the future from the POV of the container).

[keithamus]

VMs will fall out of clock sync when you sleep the host machine - so if your clock is messing up on your VM, then you need to have it query the NTP servers to adjust its clock.

For boot2docker (which I assume, as we're talking about docker you're using) you can run the following:

boot2docker ssh sudo ntpclient -s -h pool.ntp.org

and it'll fix clock sync issues. For other VMs you can install the ntp service, e.g. for ubuntu:

sudo apt-get install ntp && sudo start ntp

[remy]

@keithamus this is fine, but I suspect there's times when aligning the clock isn't the solution. Say you're creating a Dockerfile for your US based deploy that runs on US time, but you're developing in the UK.

I've got an idea that I think is fairly solid (and @olalonde has proposed a PR #453 that ping/pongs across the VM), but it all still boils down to _somehow_ telling the user that running nodemon "regular" style ain't going to fly. _Something_ extra is required. Either ntp, or an extra flag/step in running nodemon.

[keithamus]

To add, if your VM/docker seems to be an exact number of hours out (i.e. the minute and second values are both the same) then it is likely your VM or Docker container have a different timezone set to your local machine. In this case, you need to change your timezone in the guest.

For docker, docker/docker#3359 seems to suggest you can copy over your host machines /etc/localtime file over to your container:

docker run -v /etc/localtime:/etc/localtime:ro`

For other VMs, you need to override your systems timezone; for example Ubuntu:

echo "Australia/Adelaide" | sudo tee /etc/timezone && sudo dpkg-reconfigure --frontend noninteractive tzdata

[remy]

I'm actually seeing something very strange here.

When I create a file (with node) in the container, and then stat the file, I get this:

{ dev: 25,
  mode: 33188,
  nlink: 1,
  uid: 1000,
  gid: 50,
  rdev: 0,
  blksize: 4096,
  ino: 448,
  size: 11,
  blocks: 8,
  atime: Wed Jan 07 2015 23:54:22 GMT+0000 (UTC),
  mtime: Thu Jan 08 2015 11:35:24 GMT+0000 (UTC),
  ctime: Thu Jan 08 2015 11:35:24 GMT+0000 (UTC) }

Notice the mtime compared to the atime. The access time is from where I read it, but I created in the VM whose date is: Wed Jan 7 23:56:34 UTC 2015 - so the creation time is actually based on the host machine...

Is this normal? If so, this would be enough to confirm the offset.

Anyone able to try this in different environments?

[remy]

Update: I currently have an offline dev version of nodemon that runs inside of a container and picks up changes made on the host.

It's not ready yet, but it's working without any extra flags or hoops to jump...

[enagorny]

Can we test it?

[remy]

@enagorny it's currently in master, but not tagged (yet).

[saulshanabrook]

You can try this package as well: https://github.com/brikis98/docker-osx-dev

[ChrisCinelli]

https://www.virtualbox.org/ticket/14234

[olalonde]

@ChrisCinelli there was unfortunately a previous bug report that was marked as "wontfix": https://www.virtualbox.org/ticket/10660

[ChrisCinelli]

I saw it. It was also 3 years ago. And the new one is limited in scope: asking just support for Mac OS X host and Linux guests (ext3/ext4)

[ChrisCinelli]

It may not be a bad idea to have other people to comment on it to "upvote it"

[pyper]

I think I may have found what issue is, and a simple workaround to correct it. I am using boot2docker and docker-compose for a multi container environment in OS X. When I add the --watch path to my command path in a docker-compose.yml file it works perfectly, and when I remove it I have the same issue as everyone here. Tested many times over.

I looked into it further I found something interesting. When I do a --dump with the --watch path set I get the following:

system: { cwd: '/', useFind: true, useWatch: false, useWatchFile: false },
app_1   |   required: false,
app_1   |   dirs: [ '/src/code' ],

Now watch what happens without the --watch path:

system: { cwd: '/', useFind: true, useWatch: false, useWatchFile: false },
app_1   |   required: false,
app_1   |   dirs: [ '/' ],

I think it is watching over the whole filesystem. From docker it thinks the current working directory is / and the default behaviour is the cwd without --watch set, so it watches over more the whole filesystem instead of a limited part of it.

As long as you give it much less to chew on with --watch=/somedir everything works fine.

[ryanbennettvoid]

@pyper You're a lifesaver. I've tried Supervisor, Nodemon, and Forever in Docker and they all performed terribly, if at all. Nodemon responded to the first few file changes, then stopped. Here's what I did to get it working:

nodemon --watch /app/server /app/server/server.js

[lox]

I don't think it's actually possible to "implement inotify". It's triggered via the Linux VFS, not the underlying filesystem.

[aisipos]

I'm still seeing this on an NFS mounted share in boot2docker from an OS X host. I can verify that the host modified file produces a change in the container's "view" of the file, but no reload is triggered by nodemon. Which is a little odd, since the fallback of polling should in theory allow nodemon to see the changes. Is there a way to force recent versions of nodemon to use polling? The -L legacy flag is no longer here. I also tried @pyper 's method of explicitly setting --watch, but sadly it also isn't working for me.

[remy]

@aisipos I think when this lands it'll fix the issue: https://github.com/remy/nodemon/issues/633 (if anyone wants to get on with a PR, otherwise it'll probably be next week...earliest).

[remy]

@chrishiestand have you tried the latest build with -L? 1.7.x now supports the polling mode. I've generally had positive feedback that this resolves the docker mounted volumes problem.

[aisipos]

I can confirm for myself at least using -L in 1.7.0 in a docker container is working for me. Without it, it doesn't work for me. (I'll note also, with some sadness, that while polling does work it does use a bit of CPU).

[chrishiestand]

@remy Thanks for your care on this issue. I've changed my workflow and haven't used this feature in some time. If other people have reported success with -L that is good enough for me. Thanks!

[pencilcheck]

-L works but the CPU will soar when run in boot2docker vm.

[remy]

@pencilcheck you should also be wary of the number of files its watching.

[d6u]

@remy -L works for me with docker-compose (previously boot2docker) version 1.4.2 on Node 0.10 with Nodemon 1.7.1

[pencilcheck]

In a VM -L is still needed it seems but I have switched to a different van that setup NFS instead of vboxfs, now it runs much much quieter but unfortunately writing speed is not as responsive as native

[oncletom]

-L works perfectly, thanks!

[akselnordal]

Nothing of the above worked for me, so I ended up creating a small bash-script that starts my servers, restarts the pm2 instance inside docker from the host OS and tails docker logs. To avoid zombie processes, I'vce included a trap that kills the docker processes started by docker-compose

#!/bin/bash

function clean_up {
  docker-compose kill
  exit
}

trap clean_up SIGHUP SIGINT SIGTERM

docker-compose build
docker-compose start

# tail all logs 
docker-compose logs&

# fswatch is available for linux and osx.
fswatch -0 src | while read -d "" event
  do
    echo "file changed: ${event}"
    image=$(docker ps | grep <docker app-name> | cut -d " " -f1 )
    docker exec -t $image ./node_modules/.bin/pm2 restart <pm2 app-name>
  done

[Omnomios]

I was chasing this issue for a while. Nodemon was catching the change no problem but node never received the signal to terminate. I was looking through the nodemon source and saw that there was an issue with child processes and as a result it was using psTree to get to the children.

The docker image I was using (fedora) didn't come with PS pre installed and so nodemon was using a fallback to pull the process that wasn't working for me.

Installing the 'procps' package in the dockerbuild got it working for me.

Hope this helps someone.

[thalesfsp]

So, what's the final solution and how to, to get nodemon watching for changes in Mac OS X -> Boot2Docker -> Docker Container? It's possible or not? There's some documentation/README.md/Wiki? @remy

[pyper]

@thalesfsp It looks as though there is a few issues possibly causing this. @Omnomios has done some next level bug chasing and found something. For me, I simply had to explicitly set --watch.

[altieres]

The best way I have found to have filesystem notifications visible within the container was as follows:
1- Create two folders, one to map the project and a "mirror"
2- Map the project in the first folder
3- Keep a background script running in the container, rsyncing the project folder to "mirror"
4- Run the project from "mirror"

It may not be the most efficient or most elegant way, but this way was transparent to users of the container. No additional script needs to be run.
Not tested in a larger project, but in my case I did not realize performance issues.

https://github.com/altieres/docker-jekyll-s3

[nettofarah]

I have similar solution to @Omnomios.
I wrote this bash script

#!/bin/bash

image=$(docker ps | grep <container_name> | cut -d " " -f1 )
docker exec -t $image pm2 restart server

and hooked it up to webpack in my gulpfile

gulp.task('watch-backend', function() {
  webpack(backendConfig).watch(100, function(err, stats) {
    onBuild(err, stats)
    restartServer()
  })
})

var exec = require('child_process').exec

function restartServer() {
  exec('./restart-container.sh', function(err, stdout, stderr) {
    if (stdout) {
      console.log('stdout: ' + stdout)
    }

    if (stderr) {
      console.log('stderr: ' + stderr)
    }

    if (err !== null) {
      console.log('exec error: ' + err)
    }
  })
}

It has been working pretty well for me.
You have to run webpack (or whatever other watcher) in the host machine though.

[darrinholst]

I cannot figure out how to get this working either. Shelling into the container and touching a file triggers nodemon...doing the same thing from osx doesn't do anything.

[darrinholst]

whoops...I missed the need for -L. Working once I added that.

[kirkstrobeck]

-L ftw ! 👍

[yordis]

I am using Docker for Mac and it doesn't work either but -L flag fixed it

[mshick]

Using a barebones security distro and was having this issue. @Omnomios solution installing procps worked for me.

[bradcosine]

Only -L fixed the issue for me

[IonelLupu]

Has anyone found a better solution, more official fix (maybe without any parameter)t o this -L parameter?

[mshick]

@JCWolf have you tried the solution that worked for me? procps? I haven't been using legacy mode for over a year now and get automatic restarts.

Here's a snippet of my Dockerfile

FROM node:8.9.4-wheezy
RUN apt-get update
RUN apt-get install -y procps make gcc g++ python libpq-dev

[remy]

@mshick would you be able to add this to the FAQ? Sounds like you've solved a problem that many people run in to! Thanks in advance

[mshick]

@remy Added a note. Credit goes to @Omnomios. Hope it helps!

[ms20hj]

a simple patch would be to change the defaultConnectionProperties.properties inside the jar file.

oracle.jdbc.timezoneAsRegion=false

defaultConnectionProperties.properties file path: oracle.jdbc，you can find it from ojdbc6.jar.
My oracle6.jar version 11.2.0.3