Deployer: Get stuck when using multiplexing

[leo108]

| Q | A
| ----------------- | ---
| Issue Type | Bug
| Deployer Version | master cdac931
| Local Machine OS | Ubuntu 16.04.1 LTS/ homestead
| Remote Machine OS | Ubuntu 16.04.2 LTS

Description

See deploy file and output log

Steps to reproduce

Content of deploy.php

<?php
namespace Deployer;

host('xxx.com')
    ->user('root')
    ->multiplexing(true)  // it works as expected if set this to false
    ->identityFile('/home/vagrant/.ssh/id_rsa');

task('test', function () {
    run('echo 123;');
});

Output log

➤ Executing task test
[xxx.com] > echo 123;
[xxx.com] < ssh multiplexing initialization
[xxx.com] < 123        # output 123 within 1s
• done on [xxx.com] # output this line after 1min
✔ Ok [60s 804ms]

[antonmedv]

So what is the problem?

[leo108]

@antonmedv it takes 60 seconds to finish the test task when enabling the multiplexing, but it only takes 800ms when disabling the multiplexing.

[leo108]

I changed the test task to this:

task('test', function () {
    run('echo 123;');
    run('echo 321;');
});

the output is:

➤ Executing task test
[xxx.com] > echo 123;
[xxx.com] < ssh multiplexing initialization
[xxx.com] < 123
[xxx.com] > echo 321;
[xxx.com] < ssh multiplexing initialization
[xxx.com] < 321
• done on [xxx.com]
✔ Ok [121s 522ms]

It takes 2 min to finish this simple task, don't you think there is something wrong?

[antonmedv]

Looks like multiplexing isn't initialized correctly. Please chech if it's working on your local machine.

[leo108]

I did some tests and find something strange

$cmd = 'ssh -i /home/vagrant/.ssh/id_rsa -o ControlMaster=auto -o ControlPersist=60 -o ControlPath=~/.ssh/[email protected] [email protected] echo 1;';
$p = new Symfony\Component\Process\Process($cmd);
$p->run(function() {var_dump(func_get_args());});

it outputs

array:2 [
  0 => "out"
  1 => "1\n"
]

and then stuck there for 1min, then outputs error message:

Symfony\Component\Process\Exception\ProcessTimedOutException with message 'The process "ssh -i /home/vagrant/.ssh/id_rsa -o ControlMaster=auto -o ControlPersist=60 -o ControlPath=~/.ssh/[email protected] [email protected] echo 1;" exceeded the timeout of 60 seconds.'

I have tried to run $cmd in terminal directly, it outputs 1 and exits immediately. Do you know what's the problem? (although it's not related with deployer). Thanks.

[staabm]

Sounds like the Symfony Prorcess is waiting for a timeout

[antonmedv]

Try realtime output with tty mode. Maybe remote host prompt for some input.

[leo108]

@antonmedv

$cmd = 'ssh -i /home/vagrant/.ssh/id_rsa -o ControlMaster=auto -o ControlPersist=60 -o ControlPath=~/.ssh/[email protected] [email protected] echo 1;';
$p = new Symfony\Component\Process\Process($cmd);
$p->setTty(true);
$p->run(function() {var_dump(func_get_args());});

it responses 1 immediately and return 0, I think there is no remote host prompt

[leo108]

I think it should be a bug belongs to symfony/process, I have submitted an issue https://github.com/symfony/symfony/issues/22346

[antonmedv]

Well, i don't know that else can it be. Maybe some bug in ssh multiplexing?

[mikeymike]

hey @leo108 I might have run into the same issue that you have here... Turns out the reason it was hanging for me was because of a bug in OpenSSH.

I resolved it by moving my build/deploy process into docker and using a newer version of OpenSSH.

I believe MacOS comes with a relatively new version of OpenSSH with the bug resolved while a lot of Linux distros still ship with older versions.

Hope this helps 👍

[leo108]

@mikeymike that makes sense, all my servers are using openssh 7.2, it should be fixed after 7.3p1 according to your link

[staabm]

is there anything we can do to warn the user about this openssh bug in outdated openssh versions?

[mikeymike]

I think you could parse the output of ssh -V and disable multiplexing because from what I can see it simply won't work.