Vagrant: NPM Install fails with messed up encoding in random libraries

[Tomino2112]

Vagrant version

1.8.1

Host operating system

Windows 10 - 64 bit

Guest operating system

Ubuntu 12.0.4

Vagrantfile

Vagrant.configure(2) do |config|
  config.vm.box = "hashicorp/precise64"

  # Provision shell scripts
  config.vm.provision :shell, path: "provision/master.sh"
  config.vm.provision :shell, path: "provision/webapp.sh"
  config.vm.provision :shell, path: "provision/website.sh"
  config.vm.provision :shell, path: "provision/api1_python.sh"
  config.vm.provision :shell, path: "provision/api2_node.sh"
  config.vm.provision :shell, path: "provision/final.sh"  

  # Create a private network, which allows host-only access to the machine
  # using a specific IP.
  config.vm.network "private_network", ip: "20.0.0.50"
end

Expected behavior

NPM packages should install correctly

Actual behavior

NPM packages install with "messed up" characters in random places

Steps to reproduce

1. Start Vagrant machine
2. Clone NodeJS project
3. Run npm install --no-bin-links
   

   Detail

In our provision we want to install nodejs dependencies by running npm install --no-bin-links . Unfortunately the Nodejs app always fails to run with error:

C:\_VAGRANT\ipp-dev-env\src\ipushpull.com\dev\api2\node_modules\express\lib\router\index.js:135
p


SyntaxError: Unexpected token ILLEGAL
    at exports.runInThisContext (vm.js:53:16)
    at Module._compile (module.js:387:25)
    at Object.Module._extensions..js (module.js:422:10)
    at Module.load (module.js:357:32)
    at Function.Module._load (module.js:314:12)
    at Module.require (module.js:367:17)
    at require (internal/module.js:16:19)
    at Object.<anonymous> (C:\_VAGRANT\ipp-dev-env\src\ipushpull.com\dev\api2\node_modules\express\lib\application.js:17:14)
    at Module._compile (module.js:413:34)
    at Object.Module._extensions..js (module.js:422:10)

This is only example error, almost every npm install fails in different library in different file.

Same happens if you connect via ssh directly to the box and run npm install --no-bin-links manually.

If you follow the error and open the library code, this is what you see:

/**
 * Dispatch a req, res into the router.
 * @private
 */

p                                                                                                                                                                                                                                                                                                0<  �����<  P����<  @����<  p����<  ����=  ���0=  �����=  `����=  p��>  �&��`>  �/���>  �0���>  `8��(?   :��P?  �:���?  �;���?  �<��@  0=��H@  `=��h@  �E���@  @L��A  PL�� A   S��XA  �S���A   T���A  0U���A  0V���A  �V�� B  0f��PB  �f��hB  pg���B  �k���B  �k���B  @l���B  �l���B  0m��(C  �m��@C  0n��pC  �r���C  �s���C  �t���C  �u�� D  �u��8D  `v��pD  �w���D  �x���D  {�� E  �}��pE  �����E  �����E  0����E  ����F  P��� F  ����8F  �����F  ����F   ����F  ����G  ����PG  �����G  ����G  @��� H  ����hH  ���layer
  req.next = next;

Again, this is just error after one npm install, its always different

If you install npm packages from your host machine, it works fine.

[sethvargo]

Hi @Tomino2112

I think you are hitting this bug. NPM creates really long file paths which break some of the limits on windows file path limits over a shared folder :frowning:

[Tomino2112]

@sethvargo Hi, not really, I cannot see it would have anything to do with long names/paths. This extra chars with messed up encoding appear on random places. I just discovered that it also added some characters in my JS script and my webapp is now failing because of that. The path and file name are very short, no problem there, and here is what I get:

You see those red dots at the bottom, they are not showing up in any editor yet when I open the page they are there. I think it has something to do with bad file share between windows and linux.

Btw pat tho this file is: scripts/controllers/createPageCtrl.js so nothing long

UPDATE: btw this comment has really nothing to do with NPM because in webapp I am not using npm at all, only bower and grunt

[sethvargo]

Hi @Tomino2112

Thank you for your reply. I have a very hard time understanding how this would be happening in Vagrant. Vagrant does not modify the transmitted files in any way. I can maybe see if the line endings were modified between OSes, but having random characters appear feels like there's something outside of Vagrant touching these files. What are your files encoded as? Are you using virtualbox or vmware?

[Tomino2112]

@sethvargo Trust me I am having a very hard time understanding this as well, unfortunately it is happening. There is nothing that is touching the files apart from vagrant mounting it into system. It doesn't have to be Vagrant issue, it can as well be VirtualBox issue, I am not sure.

Couple of hours ago I reloaded the box, didn't touch it since then, just now I opened the webapp and all works fine. I went on to my editor, edited the file - all fine. Couple of minutes and edits later and the same problem is back.

Files are UTF-8 encoded (not sure if all the external libraries as well) and I am using VirtualBox

[sethvargo]

What version of virtualbox (vboxmanage -v)?

[Tomino2112]

Version 5.0.16 r105871 64b

[sethvargo]

Hi @Tomino2112

I'm going to assign this as a bug so we can try to reproduce during triage, but it might be worth trying to reproduce this with _just_ virtualbox using the clicky-click GUI and manually sharing the folder.

[Tomino2112]

@sethvargo Sounds reasonable, I will do that over the weekend

[Tomino2112]

Managed to write 10 lines of code today and now one of my files just got cropped in half and I am stuck again... quite frustrating

[Tomino2112]

UPDATE

If I ssh to vagrant and vim the file manually and do :wq right away (no changes) the file is served to browser correctly

[Tomino2112]

UPDATE

I just find out that if I add more empty lines at the end of the files, the errors eventually go away. I know its considered good practice to always end file with an empty line, which I do, but I actually have to end the file with multiple lines until there is no error.

Above is correct only in about 10% of cases, I just updated a file and it fail with rubbish characters on completely different place

[Tomino2112]

Seems like it might be issue with VB shared folders. In some comments over here: https://www.virtualbox.org/ticket/9069 they describe exactly what I am experiencing

[sethvargo]

Sigh. That's (sadly) what I suspected. Unfortunately I don't know if there's anything Vagrant can do here, do you see anything? You could try rsync or smb folders maybe?

[Tomino2112]

@sethvargo I dont want to go ahead of myself, but according to the suggestions in that thread I have changed my nginx conf at /etc/nginx/nginx.conf to make sure that there is sendfile off

Some googling around showed that sendfile option is only good for files in UNIX systems, so shared folder from Windows is a bummer.

So far it works with the option mentioned above changed, so I hope it will stay like that

[jsergiu]

There are some issues with installing the node packages directly in the shared folder. A good workaround is to install the packages somewhere inside the vm then create a symlink.
Ex:
cd /vagrant
rm -rf node_modules
mkdir /home/vagrant/vagrant_node_modules
cp package.json /home/vagrant/vagrant_node_modules
cd /home/vagrant/vagrant_node_modules
npm install
cd /vagrant
ln -s /home/vagrant/vagrant_node_modules/node_modules node_modules
ls -ld node_modules

[Tomino2112]

@jsergiu Actually the problem was with nginx, read above for solution

[gnulnx]

I know this issue is closed, but I can reliably reproduce it. Anytime I run npm/webpack inside vagrant the resulting bundle files are broken in the browser... same little red dots as @Tomino2112.

What is strange is that I'm the only team member with this issue and we all have vagrant provisioned in similar ways... If anyone has any thoughts I'd love to hear them.

[gnulnx]

I finally found the solution to my issue.

I already had sendfile off in nginx... so that wasn't doing it. However gunicorn was running with sendfile enabled by default. So I added this to my gunicorn start script:

--no-sendfile

http://docs.gunicorn.org/en/stable/settings.html#sendfile

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.