Vagrant: Memory leak in `vagrant ssh`

[alcuadrado]

Vagrant version

Vagrant 1.7.4

Host operating system

OS X 10.11.4

Guest operating system

Linux localhost 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

def have_vmware_fusion()
  File.exist?('/Applications/VMware Fusion.app')
end

def have_parallels()
  File.exist?('/Applications/Parallels Desktop.app')
end

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  if have_vmware_fusion then
    config.vm.box = "puppetlabs/ubuntu-14.04-64-nocm"
  elsif have_parallels then
    config.vm.box = "parallels/ubuntu-14.04"
  else
    config.vm.box = "ubuntu/trusty64"
  end

  config.vm.provision :shell, :path => "provision.sh"

  # REST api
  config.vm.network "forwarded_port", guest: 8080, host: 8080
  config.vm.network "forwarded_port", guest: 8443, host: 8443

  # Admin
  config.vm.network "forwarded_port", guest: 8444, host: 8444

  # Remote debugging
  config.vm.network "forwarded_port", guest: 5005, host: 5005

end

Debug output

I haven't executed anything in the first two processes, and I had a tail -f running on a file (which hasn't changed since I run it) on the other one.

Expected behavior

Memory consumption should be low

Actual behavior

Memory consumption is incredibly high

Steps to reproduce

1. vagrant up
2. Multiple vagrant ssh
3. Leave those ssh open for a long time

[sethvargo]

Hi @alcuadrado

Does this issue reproduce if you run vagrant ssh-config and use that information to connect to the guest directly?

[alcuadrado]

I'll try that @sethvargo, but it'll take some time.

I forgot to mention in the report. I'm using vagrant vmware.

[sethvargo]

Hi @alcuadrado

The reason I ask is that Vagrant execs into an SSH process, so I'm not sure how it could be leaking memory.

[alcuadrado]

If I'm correct vagrant creates a child process, it doesn't replace its process, so maybe there is a leak there.

I'll leave a few ssh sessions running for a some days.

[alcuadrado]

As soon as I run vagrant ssh it uses 138gb now, but I have many non-vagrant ssh with normal memory usage.

[alcuadrado]

20+ hours later, all non-vagrant sessions are ok. Any vagrant session uses 138gb as soon as it starts.

[sethvargo]

Okay sorry. I have no idea. Hopefully someone smarter than me can come along and provide some insight.

[rap2hpoutre]

Similar problem here: https://github.com/mitchellh/vagrant/issues/7870

[chrisroberts]

Hi! I was able to reproduce this behavior, but with any vagrant command executed. The vagrant ssh command is the easiest to see this behavior simply because the process is left running for as long as the ssh session is alive.

The tl;dr version of below is simply: Don't worry about it. VIRT isn't allocated memory. If it were, you would either need massive swap space, or nothing would be working.

So, what's going on here? The vagrant installer includes a small go executable (vagrant) whose job is to setup the current environment with the proper locations of everything it needs. The installers bin directory, the lib directory for ruby and all the other friends, all the gems, and the vagrant gem itself. Once it has all this configured, it spawns off a new process, the actual Ruby vagrant process.

Because your example was referencing vagrant ssh, and as was previously pointed out (https://github.com/mitchellh/vagrant/issues/7296#issuecomment-222270159) a Kernel.exec happens meaning the Ruby process does not persist, I figured it must be the wrapper that was the culprit. After a bit of searching (mostly to find stackoverflow items saying "don't worry about VIRT") I stumbled upon:

keybase/keybase-issues#1908

They refer to the golang FAQ that talks about a bunch of VIRT being claimed up front and it not being a big deal, but never any absolutes about how much was actually being claimed. A link to lwn was dropped in there (https://github.com/keybase/keybase-issues/issues/1908#issuecomment-164165217) regarding golang's behavior on startup of claiming a huge chunk of VIRT, but still everything referenced a much lower amount than I was seeing locally. So I decided to go dig into the golang runtime code, and within malloc.go we find the answer:

golang src/runtime/malloc.go

The _why_ it's happening is because of the go wrapper used to start vagrant. Because the VIRT you see is simply a _reservation_ and not actually _allocated_, it's not a problem and not something that should be worried about.

(There are some interesting conversations on the golang ML around the pros and cons of this approach, all pretty great reads).

[alcuadrado]

Nice work @chrisroberts! Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.