Vagrant: Vagrant halt, freeze with SSH Sending keep alive

[KartoffelToby]

System: Mac OS X
Vagrant 1.7.4
VMware-Fusion 7

vagrant hangs on vagrant halt and vagrant reload.

Here is the output from vagrant halt --debug

....
DEBUG ssh: PTY stdout parsed: Debian GNU/Linux stretch/sid n l

INFO guest: Detected: debian!
DEBUG guest: Searching for cap: halt
DEBUG guest: Checking in: debian
DEBUG guest: Checking in: linux
DEBUG guest: Found cap: halt in linux
INFO guest: Execute capability: halt #
DEBUG ssh: Re-using SSH connection.
INFO ssh: Execute: shutdown -h now (sudo=true)
DEBUG ssh: pty obtained for connection
DEBUG ssh: stdout: export TERM=vt100

DEBUG ssh: Re-using SSH connection.
INFO ssh: Execute: echo; printf $SSH_AUTH_SOCK (sudo=false)
DEBUG ssh: pty obtained for connection
DEBUG ssh: stdout: export TERM=vt100
stty raw -echo
export PS1=
export PS2=
export PROMPT_COMMAND=
printf bccbb768c119429488cfd109aacea6b5-pty
echo; printf $SSH_AUTH_SOCK
exitcode=$?
printf bccbb768c119429488cfd109aacea6b5-pty
exit $exitcode

DEBUG ssh: stdout: root@kartoffel-box:/home/vagrant# export TERM=vt100
root@kartoffel-box:/home/vagrant#
DEBUG ssh: stdout: vagrant@kartoffel-box:~$ expor
DEBUG ssh: stdout: t
DEBUG ssh: stdout: TERM=vt10
DEBUG ssh: stdout: 0

DEBUG ssh: stdout: vagrant@kartoffel-box:~$ stty r
DEBUG ssh: stdout: aw -
DEBUG ssh: stdout: e
DEBUG ssh: stdout: cho

DEBUG ssh: stdout: vagrant@kartoffel-box:~$
DEBUG ssh: stdout: bccbb768c119429488cfd109aacea6b5-pty
DEBUG ssh: stdout:

DEBUG ssh: stdout: /tmp/ssh-NytPFZbPrZ/agent.2229
DEBUG ssh: stdout: bccbb768c119429488cfd109aacea6b5-pty
DEBUG ssh: stdout: Abgemeldet

DEBUG ssh: Exit status: 0
DEBUG ssh: PTY stdout: export TERM=vt100
stty raw -echo
export PS1=
export PS2=
export PROMPT_COMMAND=
printf bccbb768c119429488cfd109aacea6b5-pty
echo; printf $SSH_AUTH_SOCK
exitcode=$?
printf bccbb768c119429488cfd109aacea6b5-pty
exit $exitcode
vagrant@kartoffel-box:~$ export TERM=vt100
vagrant@kartoffel-box:~$ stty raw -echo
vagrant@kartoffel-box:~$ bccbb768c119429488cfd109aacea6b5-pty
/tmp/ssh-NytPFZbPrZ/agent.2229bccbb768c119429488cfd109aacea6b5-ptyAbgemeldet

DEBUG ssh: PTY stdout parsed:
/tmp/ssh-NytPFZbPrZ/agent.2229
INFO ssh: Setting SSH_AUTH_SOCK remotely: /tmp/ssh-NytPFZbPrZ/agent.2229
nt.2229sh: stdout: export SSH_AUTH_SOCK=/tmp/ssh-NytPFZbPrZ/age
root@kartoffel-box:/home/vagrant# stty raw -echo

DEBUG ssh: stdout: root@kartoffel-box:/home/vagrant#
DEBUG ssh: stdout: bccbb768c119429488cfd109aacea6b5-pty
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
...

[sethvargo]

Duplicate of https://github.com/mitchellh/vagrant/issues/6051. Please see my solution there. Thanks! :smile:

[legal90]

@sethvargo We've run into the same issue with parallels provider: https://github.com/Parallels/vagrant-parallels/issues/240
However, It is reproducible even with minimal configuration ( vagrant init -m bento/debian-8.1 && vagrant up --provider=parallels), so your suggestion above could not be applied.

Like in the top message, communicator is sending SSH keep-alive, so there is nothing to fix on the provider side. Should this bug be reopened or we have to file the new one?

P.S. I could reproduce it only with Debian guest. Other guests (CentOS, Ubuntu) are halting fine.
P.P.S. I couldn't reproduce it with virtualbox provider (even on the same version of box bento/debian-8.1). I'm not sure, but probably it is caused by the fact that virtualbox operates SSH connections via forwarded ports, but parallels uses direct IP connection.

[dch]

Just reporting I have this too using debian sid (latest 4.3 kernel). I'm happy to provide more logs or help further.

sample vagrant parallels box is https://atlas.hashicorp.com/ideegeo/boxes/debian-sid

 INFO subprocess: Starting process: ["/usr/local/bin/prlctl", "list", "91e45743-0560-45c0-9143-077ecf538404", "--no-header", "-o", "status"]
DEBUG subprocess: Command not in installer, not touching env vars.
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: stdout: running
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 31999
DEBUG subprocess: Exit status: 0
 INFO interface: output: Attempting graceful shutdown of VM...
 INFO interface: output: ==> default: Attempting graceful shutdown of VM...
==> default: Attempting graceful shutdown of VM...
DEBUG ssh: Checking whether SSH is ready...
DEBUG ssh: Re-using SSH connection.
 INFO ssh: SSH is ready!
DEBUG ssh: Re-using SSH connection.
 INFO ssh: Execute:  (sudo=false)
DEBUG ssh: Exit status: 0
DEBUG guest: Searching for cap: halt
DEBUG guest: Checking in: debian
DEBUG guest: Checking in: linux
DEBUG guest: Found cap: halt in linux
 INFO guest: Execute capability: halt [#<Vagrant::Machine: default (VagrantPlugins::Parallels::Provider)>] (debian)
DEBUG ssh: Re-using SSH connection.
 INFO ssh: Execute: shutdown -h now (sudo=true)
DEBUG ssh: stderr: mesg: ttyname failed: Inappropriate ioctl for device

DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...
DEBUG ssh: Sending SSH keep-alive...

[cliffrowley]

Is anyone currently looking at this? It's driving me mad and I'm stuck using Parallels currently because I need its nested virtualisation. If no-one is looking at it I'll likely take a look myself but I'm not familiar with Vagrant's source.

[cliffrowley]

I'll assume that's a no, then.

[Sharpie]

This has also been reported in oscar-stack/vagrant-hosts#65

One thing I noticed is that some of these DEBUG traces show a PTY being allocated after a command is executed on the SSH session. Is this correct behavior? Shouldn't the PTY be allocated first?

[DanHam]

Hi,

This issue is probably specific to Debian/Debian derivatives and limited to boxes running systemd. There is a very long running thread here where the issue is discussed in some detail! Read through for the full gory details.

Long and short, I managed to prevent this behaviour with my Debian box by:

1/ Ensuring the libpam-systemd package is installed in the guest

sudo apt-get install libpam-systemd

2/ Ensuring that /etc/ssh/sshd_config has the required option set to use PAM

UsePAM yes

3/ Restarting the box - you'll have to put up with the issue one last time... CTRL-C to the rescue!

Once this is taken care of issuing a vagrant halt behaves as expected and everything shuts down cleanly. Similarly, when issuing a 'shutdown -h now' from within an ssh session, the ssh session is automatically closed and control returned to the shell.

Note that the libpam-systemd package is 'recommended' for installation when systemd is installed. Debian systems _should_ install 'recommended' packages by default so any box exhibiting this behaviour has probably been built with the APT setting of Install-Recommends set to false (see /etc/apt/apt.conf.d/00InstallRecommends).

Just for completeness:

Host OS: OS X 10.10.5
Vagrant version: 1.8.5
Virtualisation: VMware Fusion Professional Version 8.1.1 (3771013)
Debian VM: 8.5 (Jessie)

Hope that helps.

Dan

[chrisroberts]

@DanHam 's comment sums this up pretty nicely: https://github.com/mitchellh/vagrant/issues/6207#issuecomment-234332985

I'm going to close this up. If the behavior is still seen with properly configured guests, please open a new issue. Thanks!

[vdloo]

If @DanHam's solution above does not work, try creating a oneshot systemd service that kills sshd like explained here

# cat >/etc/systemd/system/kill-ssh-user-sessions-on-shutdown.service <<EOF
[Unit]
Description=Shutdown all ssh sessions before network
DefaultDependencies=no
Before=network.target shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/bin/killall sshd

[Install]
WantedBy=poweroff.target halt.target reboot.target
EOF

# systemctl enable kill-ssh-user-sessions-on-shutdown

[flaugher]

I can confirm that vdloo's solution worked for me. I'm running VirtualBox 5.2.12, Vagrant 2.1.2, and Debian 8.11.

[DanHam]

@vdloo @flaugher Using a systemd unit to kill ssh should not be necessary if the VM you are using has the libpam-systemd package installed. That workaround is actually in the same bug report that the eventual fix was reported in!

Hope that helps.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.