Paramiko: Paramiko hangs on import

[Kkevsterrr]

I've noticed every so often my application hangs when trying to import paramiko. I've attached gdb to the process and got the following backtrace:

0x00007f55355d2bc4 in __GI___poll (fds=fds@entry=0x7ffcb96df3a0, nfds=nfds@entry=1, timeout=timeout@entry=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
29  ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.
(gdb) py-bt
Traceback (most recent call first):
  <built-in method sodium_init of CompiledLib object at remote 0x7f55308a1318>
  File "/usr/local/lib/python3.6/dist-packages/nacl/bindings/sodium_core.py", line 22, in _sodium_init
    ensure(lib.sodium_init() != -1,
  <built-in method init_once of CompiledFFI object at remote 0x7f55308a3048>
  File "/usr/local/lib/python3.6/dist-packages/nacl/bindings/sodium_core.py", line 32, in sodium_init
    ffi.init_once(_sodium_init, "libsodium")
  File "/usr/local/lib/python3.6/dist-packages/nacl/bindings/__init__.py", line 377, in <module>
    sodium_init()
  <built-in method exec of module object at remote 0x7f553591c638>
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "/usr/local/lib/python3.6/dist-packages/nacl/signing.py", line 17, in <module>
    import nacl.bindings
  <built-in method exec of module object at remote 0x7f553591c638>
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "/usr/local/lib/python3.6/dist-packages/paramiko/ed25519key.py", line 22, in <module>
    import nacl.signing
  <built-in method exec of module object at remote 0x7f553591c638>
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "/usr/local/lib/python3.6/dist-packages/paramiko/transport.py", line 90, in <module>
    from paramiko.ed25519key import Ed25519Key
  <built-in method exec of module object at remote 0x7f553591c638>
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "/usr/local/lib/python3.6/dist-packages/paramiko/__init__.py", line 22, in <module>
    from paramiko.transport import SecurityOptions, Transport
  <built-in method exec of module object at remote 0x7f553591c638>

This is on a newly created AWS instance:

# cat /proc/version
Linux version 4.15.0-1021-aws (buildd@lcy01-amd64-001) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #21-Ubuntu SMP Tue Aug 28 10:23:07 UTC 2018
# 

Using paramiko 2.4.2:

# python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import paramiko
>>> paramiko.__version__
'2.4.2'
>>> ^D
# 

[ploxiln]

This is actually in a dependency https://github.com/pyca/pynacl in a C function sodium_init()

I have a suspicion that it _might_ be this call to randombytes_stir(): https://github.com/pyca/pynacl/blob/1.3.0/src/libsodium/src/libsodium/sodium/core.c#L55 ending up at this line blocking on /dev/random https://github.com/pyca/pynacl/blob/1.3.0/src/libsodium/src/libsodium/randombytes/sysrandom/randombytes_sysrandom.c#L148 - it generally should not be blocking on /dev/random, it should be using /dev/urandom or getrandom(), but maybe some quirk of how it was built for your system made things not work out as expected. Again, this is all just an educated guess. You can check $ cat /proc/sys/kernel/random/entropy_avail and if the number is less than perhaps 200, this could be the issue.

Anyway, if you want to figure this out and resolve it, you should follow up with pynacl and work through isolating the problem there.

[Kkevsterrr]

Hey @ploxiln - thanks for getting back to me so quickly. Looks like you're absolutely right:

cat /proc/sys/kernel/random/entropy_avail
46

Looking at the pynacl source code, it looks like it has the ability to use /dev/urandom instead of /dev/random (it's just #ifndef-ed out) - could paramiko in setup.py build pynacl specifically with the option set correctly? This would ensure paramiko isn't bit by this issue. I'll open an issue with the pynacl team as well - thanks for your help!

[ruuda]

There is an issue over at pyca/pynacl: https://github.com/pyca/pynacl/issues/327.

[ric79]

Thanks guys. My CentOS had a very minimal config and the entropy_avail was very low:
Simple procedure to solve it:

$ cat /proc/sys/kernel/random/entropy_avail
86
$ sudo yum install -y rng-tools
$ systemctl start rngd.service
$ systemctl enable rngd.service
$ cat /proc/sys/kernel/random/entropy_avail
3103

[qloridant]

Thank you guys !
Like @ric79 , I installed a package to "generate entropy" on my ubuntu (running on a virtual machine) : haveged
This solved this issue.