Salt: [FreeBSD] salt-call on 2019.2.0+ does not return shell prompt if using Python3

[cedwards]

Description of Issue

After migrating to py37 on FreeBSD salt-call behaves strangely in that it does not return the shell prompt after the successful (or failed) application of the state. This does not affect py27 installations.

Steps to Reproduce Issue

pkg install py37-salt on FreeBSD on use salt-call state.apply foo.

Versions Report

Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: 1.13.2
       cherrypy: Not Installed
       dateutil: 2.8.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.6 (default, Jan 30 2020, 01:17:40)
   python-gnupg: Not Installed
         PyYAML: 5.2
          PyZMQ: 18.1.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist:
         locale: UTF-8
        machine: amd64
        release: 11.3-RELEASE
         system: FreeBSD
        version: Not Installed

[Ch3LL]

are you also seeing this on 3000?

[cedwards]

Have not looked at 3000.

[krionbsd]

I can reproduce it locally as well with:

Salt Version:                                                                                                                           
           Salt: 2019.2.3                                                                                                               

Dependency Versions:                                                                                                                    
           cffi: 1.14.0                                                                                                                 
       cherrypy: Not Installed                                                                                                          
       dateutil: Not Installed                                                                                                          
      docker-py: Not Installed                                                                                                          
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.6 (default, Jan 13 2020, 14:10:17)
   python-gnupg: Not Installed
         PyYAML: 5.2
          PyZMQ: 18.1.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist:   
         locale: utf-8
        machine: amd64
        release: 13.0-CURRENT
         system: FreeBSD
        version: Not Installed

We were able to trace it a bit, it seems it waits for events and polls on zmq:

root@q1u001:~ # procstat threads 75938
  PID    TID COMM                TDNAME              CPU  PRI STATE   WCHAN
75938 102287 python3.7           -                    -1  152 sleep   kqread
75938 102296 python3.7           -                    -1  152 sleep   kqread
75938 103116 python3.7           -                    -1  152 sleep   select

#0  _poll () at _poll.S:4
#1  0x00000008005e21e6 in __thr_poll (fds=0x7fffffffe0d0, nfds=1, timeout=-1) at /usr/src/lib/libthr/thread/thr_syscalls.c:338
#2  0x00000008028da08e in ?? () from /usr/local/lib/libzmq.so.5
#3  0x00000008028b95a2 in ?? () from /usr/local/lib/libzmq.so.5
#4  0x00000008028a556b in ?? () from /usr/local/lib/libzmq.so.5
#5  0x00000008028fac7f in zmq_ctx_destroy () from /usr/local/lib/libzmq.so.5
#6  0x0000000802c166a4 in ?? () from /usr/local/lib/python3.7/site-packages/zmq/backend/cython/context.so
#7  0x00000008003adccf in ?? () from /usr/local/lib/libpython3.7m.so.1.0
#8  0x00000008003903c4 in ?? () from /usr/local/lib/libpython3.7m.so.1.0
#9  0x00000008003bd476 in ?? () from /usr/local/lib/libpython3.7m.so.1.0
#10 0x0000000800478e49 in ?? () from /usr/local/lib/libpython3.7m.so.1.0
#11 0x0000000800478517 in _PyGC_CollectNoFail () from /usr/local/lib/libpython3.7m.so.1.0
#12 0x000000080044043f in PyImport_Cleanup () from /usr/local/lib/libpython3.7m.so.1.0
#13 0x000000080044d7d5 in Py_FinalizeEx () from /usr/local/lib/libpython3.7m.so.1.0
#14 0x000000080044e2dd in Py_Exit () from /usr/local/lib/libpython3.7m.so.1.0
#15 0x0000000800456c6d in ?? () from /usr/local/lib/libpython3.7m.so.1.0
#16 0x0000000800456898 in PyErr_PrintEx () from /usr/local/lib/libpython3.7m.so.1.0
#17 0x0000000800455efd in PyRun_SimpleFileExFlags () from /usr/local/lib/libpython3.7m.so.1.0
#18 0x0000000800477447 in ?? () from /usr/local/lib/libpython3.7m.so.1.0
#19 0x0000000800477cb5 in _Py_UnixMain () from /usr/local/lib/libpython3.7m.so.1.0
#20 0x000000000020110f in _start ()
(gdb)

[Ch3LL]

and to clarify this only affects salt-call? it does not affect any other salt commands?

[cedwards]

So far I have only seen the issue with salt-call on the minion and not with salt from the master.

[krionbsd]

and I've to add salt-call is affected only while running state.apply state, running salt-call + execution modules works as expected

[Ch3LL]

thanks for the clarifications

[twangboy]

@cedwards What is the state you're running? Some commands cause communicate to fail. See my comments on a possibly related issue: https://github.com/saltstack/salt/issues/56231#issuecomment-614901232

[ari]

This happens for me on every state and also highstate. FreeBSD 12.1. Salt 2019.2.5

[krionbsd]

I can't reproduce this failure on salt3000.3 which I'm going to port to FreeBSD in the next days. 2019.2.5 is still affected and has this issue though, stay tuned we're working on it.

[krionbsd]

As we moved to 3000.3 in sysutils/py-salt this issue is fixed there. @cedwards any plans to migrate to 3000.3?

[sagetherage]

closing this issue as fixed in 3000.3