Paramiko: hanging on stdout.readlines()

[ranamalo]

I have a function that logs on to a host executes a command stores stdout and stderr and logs off. It hangs for a long time, but I have noticed it will eventually close if I leave it for a really long time. Maybe you can tell me what I'm doing wrong.

Here is the function:

def ssh_cmd(current_host, command, sm_username=sm_username, sm_key=sm_key, sm_port=sm_port, print_stdout=True):
    # establish our dict for returning
    ssh_cmd_dict = {}
    # fill in the stuff we know
    ssh_cmd_dict['server'] = current_host
    ssh_cmd_dict['command'] = command
    # establish  our variables
    server, username, keyfile, portnumber = ( current_host, sm_username , sm_key , sm_port )
    # make the ssh object
    ssh = paramiko.SSHClient()
    # add new servers to known_hosts automatically
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    # try to establish the connection to server
    try:
        ssh.connect(server, username=username, key_filename=keyfile, port=portnumber )
    except:
        # record connection failure
        ssh_cmd_dict['connection_successful'] = False
        # if it doesn't work tell 'em
        print('unable to connect to: '+server)
    else:
        # record connection success
        ssh_cmd_dict['connection_successful'] = True
        # execute the command  storing stdout, stdin, and stderr
        stdin, stdout, stderr = ssh.exec_command(command)
        # wait for exit status (that means command finished)
        exit_status = stdout.channel.recv_exit_status()
        # store the stdout
        # it hangs here:
        ssh_cmd_dict['stdout_raw'] = stdout.readlines()

        # create an entry that is pretty to read
        if type(ssh_cmd_dict['stdout_raw']) is ListType:
            ssh_cmd_dict['stdout_print'] = "".join(ssh_cmd_dict['stdout_raw'])
        ssh_cmd_dict['stdout'] = ssh_cmd_dict['stdout_raw']

        # store stdin
        ssh_cmd_dict['stdin'] = command

        # store the stderr
        ssh_cmd_dict['stderr_raw'] = stderr.readlines()
        # create an entry for stderr that is pretty to read
        if type(ssh_cmd_dict['stderr_raw']) is ListType:
            ssh_cmd_dict['stderr_print'] = "".join(ssh_cmd_dict['stderr_raw'])
        ssh_cmd_dict['stderr'] = ssh_cmd_dict['stderr_raw']

        # store exit status
        ssh_cmd_dict['exitstatus'] = exit_status

        if exit_status > 0:
            if ssh_cmd_dict['stderr'] is not None:
                print('Unfortunately the command: '+command+' executed on server: '+server+' had a none zero exit status. Below is the stderr:')
                print(ssh_cmd_dict['stderr'])
        else:
            if print_stdout:
                if ssh_cmd_dict['stdout_print'] is not None:
                    print(ssh_cmd_dict['stdout_print'])
        # close the connection
        ssh.close()
        return ssh_cmd_dict

[bitprophet]

You might want to check how Fabric handles this, it deals with this use case fine all the time. (You might also want to use it for what you're doing anyways ;)) specifically check network.py and operations.py.

[leth]

I've been looking at a similar problem with some of our automation code.

One particular command will complete, and the ssh server will send the exit status, but it seems to never send the channel-eof message, which causes the stdout.read() to block.

I've had to put in a loop after I get the exit status checking channel.eof_received with a timeout on it.
After the timeout I close the channel, so that the reads will not block.

I'd appreciate any insights or improvements on this approach.

[ranamalo]

Fabric didn't suit my needs. It was too high level and didn't offer the flexibility I wanted. Unfortunately I don't remember the details as it was a while ago. Anyway, with the help of the community I did eventually figure out something that works for me. Here is my python function, perhaps it will help someone else. It runs the command on the remote host, getting stdout, stderr, and exit_code, also gives you the option of selecting the port and assumes ssh keys:

# function for running a shell command on a host
def ssh_cmd(current_host, command, sm_username=sm_username, sm_key=sm_key, sm_port=sm_port, print_stdout=True):
    # make sure command ends in a new line
    if not command.endswith('\n'):
        command = command+'\n'
    # establish our dict for returning
    ssh_cmd_dict = {}
    # fill in the stuff we know
    ssh_cmd_dict['server'] = current_host
    ssh_cmd_dict['command'] = command
    # establish  our variables
    server, username, keyfile, portnumber = ( current_host, sm_username , sm_key , sm_port )
    # make the ssh object
    ssh = paramiko.SSHClient()
    # add new servers to known_hosts automatically
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

    # try to establish the connection to server
    try:
        ssh.connect(server, username=username, key_filename=keyfile, port=portnumber )
    except:
        # record connection failure
        ssh_cmd_dict['connection_successful'] = False
        # if it doesn't work tell 'em
        print('unable to connect to: '+server)
        return ssh_cmd_dict
    else:
        # record connection success
        ssh_cmd_dict['connection_successful'] = True
        # execute the command storing stdout, stdin, and stderr
        stdin, stdout, stderr = ssh.exec_command(command)
        # wait for exit status (that means command finished)
        exit_status = stdout.channel.recv_exit_status()
        # flush commands and cut off more writes
        stdin.flush()
        stdin.channel.shutdown_write()

        # close the connection
        ssh.close()

        # store the stdout
        ssh_cmd_dict['stdout_raw'] = stdout.readlines()

        # create an entry that is pretty to read
        if type(ssh_cmd_dict['stdout_raw']) is ListType:
            ssh_cmd_dict['stdout_print'] = "".join(ssh_cmd_dict['stdout_raw'])
        ssh_cmd_dict['stdout'] = ssh_cmd_dict['stdout_raw']

        # store stdin
        ssh_cmd_dict['stdin'] = command

        # store the stderr
        ssh_cmd_dict['stderr_raw'] = stderr.readlines()
        # create an entry for stderr that is pretty to read
        if type(ssh_cmd_dict['stderr_raw']) is ListType:
            ssh_cmd_dict['stderr_print'] = "".join(ssh_cmd_dict['stderr_raw'])
        ssh_cmd_dict['stderr'] = ssh_cmd_dict['stderr_raw']

        # store exit status
        ssh_cmd_dict['exitstatus'] = exit_status

        if exit_status > 0:
            if ssh_cmd_dict['stderr'] is not None:
                print('Unfortunately the command: '+command+' executed on server: '+server+' had a non-zero exit status. Below is the stderr:')
                print(ssh_cmd_dict['stderr'])
        else:
            if print_stdout:
                if ssh_cmd_dict['stdout_print'] is not None:
                    print(ssh_cmd_dict['stdout_print'])
        return ssh_cmd_dict

[conrosebraugh]

I know this is an old thread, but I am still having this issue. I tried running the code in the comment above, but it simply hangs when you get to the "exit_status = stdout.channel.recv_exit_status()" line. I stepped into this line with pdb, and it hangs in the recv_exit_status function at the first line ("self.status_event.wait()"). Has anybody found a solution to this?

[leth]

You could add some logging to show the messages paramiko recieves and compare that to the output of running the same command with ssh -vv ....

[akos-csurai]

I had problem with a similar code. I tried the command with:

ssh @ ""

and ssh also hanged! So finally we managed to find the reason here:
http://superuser.com/questions/617252/why-does-ssh-hang-at-the-end-of-these-commands-and-how-can-i-make-it-exit/617670#617670
and here:
http://www.snailbook.com/faq/background-jobs.auto.html

In a nutshell: you can not start a background process on the remote server side, or you need to redirect the background process stdin/stdout/stderr streams.

[dmitrytokarev]

@ranamalo I also experience this issue. And yes @leth it is due to stdout.channel.eof_received == 0.

>>> import paramiko
>>> client = paramiko.SSHClient()
>>> client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
>>> client.connect("1.1.1.1", username="root", password="pass")
>>> stdin, stdout, stderr = client.exec_command("service XXX start")

stdin, stdout and stderr are staying open...

>>> print stdin
<paramiko.ChannelFile from <paramiko.Channel 3 (open) window=2097152 in-buffer=50 -> <paramiko.Transport at 0x17eff90L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>>
>>> print stdout
<paramiko.ChannelFile from <paramiko.Channel 3 (open) window=2097152 in-buffer=50 -> <paramiko.Transport at 0x17eff90L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>>
>>> print stderr
<paramiko.ChannelFile from <paramiko.Channel 3 (open) window=2097152 in-buffer=50 -> <paramiko.Transport at 0x17eff90L (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>>

So EOF was not received...

>>> print stdin.channel.eof_received
0

Usually I receive True and can just stdout.read(), but to be safe i use this workaround (which works!):
Wait for a timeout, force stdout.channel.close() and then stdout.read():

>>> timeout = 30
>>> import time
>>> endtime = time.time() + timeout
>>> while not stdout.channel.eof_received:
...     sleep(1)
...     if time.time() > endtime:
...         stdout.channel.close()
...         break
>>> stdout.read()
'Starting XXX: \n[  OK  ]\rProgram started . . .\n'
>>>

BTW i use:
Python 2.6.6
paramiko (1.15.2)
Hope this helps...

[fz6]

FWIW, I was experiencing this issue with version 1.14.0, using wget. The -q (quiet) option resolved it, but it also doesn't appear to occur in 1.15.2

[linfengtingxue]

thx for fz6

[kundansharmaindia]

Same problem here
Hangover with command -output = stdout.readlines()

[Knitschi]

I experience the problem when I am connected to the ssh server that is included in Windows 10 (OpenSSH_7.6p1 for Windows). The same code works when I am connected to linux machines.

[alinasj]

Seeing similar issue on SSH to a Windows 10 machine. Code is in Python3.7.
for those interested in work around, there are few options listed under: https://stackoverflow.com/questions/35266753/paramiko-python-module-hangs-at-stdout-read?newreg=0bc8f8964087429980c69b5b74bb7c63

[ploxiln]

The general issue here is that you must also read stderr. If you just read stdout, it's possible that the process wrote "too much" to stderr, so it could not all be buffered, and the process is blocked writing more to stderr, waiting for what was already written to be read.

So, you could try using Channel.settimeout() to set a timeout for Channel.recv() and Channel.recv_stderr(), and have a loop trying both in turn. There is also Channel.setblocking(). You could try using Channel.recv_ready() and Channel.recv_stderr_ready(). Finally, you could try combining stderr into stdout so there's only one stream to read (with 2>&1 in the shell command).

You could also just use Fabric.

[ssbarnea]

I am facing this bug with latest paramiko==2.5.0 while using python docker on python2.7 (macos) and only on 2.7, the same works without problems on other python versions.

exitstatus: 2
    client = docker.from_env(timeout=5, version="auto")
../../.pyenv/versions/2.7.16/lib/python2.7/site-packages/docker/client.py:85: in from_env
terminalreporter: <_pytest.terminal.TerminalReporter object at 0x109c54ed0>
    timeout=timeout, version=version, **kwargs_from_env(**kwargs)
../../.pyenv/versions/2.7.16/lib/python2.7/site-packages/docker/client.py:40: in __init__
    self.api = APIClient(*args, **kwargs)
../../.pyenv/versions/2.7.16/lib/python2.7/site-packages/docker/api/client.py:166: in __init__
    base_url, timeout, pool_connections=num_pools
../../.pyenv/versions/2.7.16/lib/python2.7/site-packages/docker/transport/sshconn.py:84: in __init__
    self._connect()
../../.pyenv/versions/2.7.16/lib/python2.7/site-packages/docker/transport/sshconn.py:94: in _connect
    parsed.hostname, parsed.port, parsed.username,
../../.pyenv/versions/2.7.16/lib/python2.7/site-packages/paramiko/client.py:397: in connect
    t.start_client(timeout=timeout)
../../.pyenv/versions/2.7.16/lib/python2.7/site-packages/paramiko/transport.py:597: in start_client
    event.wait(0.1)
../../.pyenv/versions/2.7.16/lib/python2.7/threading.py:614: in wait
    self.__cond.wait(timeout)
../../.pyenv/versions/2.7.16/lib/python2.7/threading.py:359: in wait
    _sleep(delay)
E   KeyboardInterrupt

Does anyone knows a trick I could use to avoid it?...especially as this is triggered inside a 2rd party library which I am unlikely to patch in a timely manner.

[ploxiln]

blocking in SSHClient.connect() / Transport.start_client() would be a different issue / cause

[ssbarnea]

@ploxiln Any idea what I can do? I am not yet sure if the bug is in docker-py or in paramiko but looking at the source code I apparently see that no timeout is specified for connect, which clearly looks like one issue. Still, why is never ending is another aspect that should be sorted.