salt-master doesn't work in fedora 31 + tornado 6

[itamarjp]

https://bugzilla.redhat.com/show_bug.cgi?id=1748170

No module named 'tornado.stack_context'

https://www.tornadoweb.org/en/stable/releases/v6.0.0.html

APIs deprecated in Tornado 5.1 have been removed. This includes the tornado.stack_context

https://koji.fedoraproject.org/koji/buildinfo?buildID=1338741

[DmitryKuzmenko]

Salt doesn't support Tornado 6.0 yet but is looking at addressing the issue but are unable to give a timeframe as to when support will be available.

[kurobeats]

I don't mean to throw fuel on this fire but I've now lost configuration and control of 15 hosts. They all have a fun memory leak too (if I leave salt-minion enabled).

[leifliddy]

I'm running into this exact same issue. Could someone post a workaround regardless of 'hacky' it is?
I'm trying to avoid setting up a full virtualenv for salt.

I'm thinking about installing tornado5 in an alternate location and changing these references.
Has someone already tried this?

/usr/lib/python3.7/site-packages/salt/minion.py:            with tornado.stack_context.ExceptionStackContext(timeout_handler):
/usr/lib/python3.7/site-packages/salt/minion.py:        with tornado.stack_context.StackContext(functools.partial(RequestContext,
/usr/lib/python3.7/site-packages/salt/minion.py:            with tornado.stack_context.StackContext(minion_instance.ctx):
/usr/lib/python3.7/site-packages/salt/minion.py:            with tornado.stack_context.ExceptionStackContext(timeout_handler):
/usr/lib/python3.7/site-packages/salt/minion.py:            with tornado.stack_context.ExceptionStackContext(timeout_handler):
/usr/lib/python3.7/site-packages/salt/minion.py:        with tornado.stack_context.ExceptionStackContext(timeout_handler):
/usr/lib/python3.7/site-packages/salt/master.py:from tornado.stack_context import StackContext
/usr/lib/python3.7/site-packages/salt/client/mixins.py:import tornado.stack_context
/usr/lib/python3.7/site-packages/salt/client/mixins.py:            with tornado.stack_context.StackContext(self.functions.context_dict.clone):

[brejoc]

@leifliddy You'd have to downgrade python3-tornado to the latest 4.x version. But I only see 6.0.2 in the Fedora repos.

[leifliddy]

@brejoc thanks! tornado 5.x works fine with salt. In fact, I used the python2-tornado-5.0.2-5 rpm on Fedora 30 without issue. With Fedora 31, all of the python salt dependency packages were transitioned from python2 --> python3.
I'm such an idiot, for some reason I was thinking there were other python tornado dependencies on my system. But nope, only salt uses it.
Managed to get salt working by installing tornado 5.x

rpm -e --nodeps python3-tornado
pip3 install --install-option="--prefix=/usr" tornado==5.1.1

[iggy]

Alpine Linux upgraded to tornado 6 as well, so this is going to be an issue there.

[llua]

opensuse tumbleweed has had tornado 6 for awhile too: https://software.opensuse.org/package/python3-tornado

[thekabal]

I also experienced this in Fedora 31, and was able to solve it following the instructions in this comment: https://github.com/saltstack/salt/issues/54392#issuecomment-552099937 . Well done!

[brejoc]

opensuse tumbleweed has had tornado 6 for awhile too: https://software.opensuse.org/package/python3-tornado

It's true that there is tornado 6, but we also have several old tornado versions exactly for cases like this. So openSUSE is good for now.

[arizvisa]

This issue isn't just for salt-master, it's for salt in general which includes salt-minions too. So this includes minions that are bootstrapped without providing the parameters to install from pip. Using salt-cloud/salt-bootstrap to install from pip requires 2 things. Installing from git, which is blocked due to an issue in salt-bootstrap, and to use a virtualenv which isn't available in salt-bootstrap for anything but ubuntu.

I imagine most users would rather install python-tornado into their minion's global python packages and risk ruining dependencies rather than not communicate with any of their minions that salt-bootstrap can't install a downgraded tornado in.

Can the OP update the title to something like "salt doesn't work with tornado 6 which is now default for most distributions"?

[arizvisa]

saltstack/salt-bootstrap#1381 seems to have been merged, so the workaround that I had mentioned wrt installing python-tornado into a minion's global packages is now done by salt-bootstrap. A re-deploy w/ it should perform #54392 (comment)

For users that can't re-deploy, just ssh in and apply the directions mentioned in that comment as prior mentioned by thekabal.

[53c70r]

bootstrap using

curl -o bootstrap-salt.sh -L https://bootstrap.saltstack.com
sudo sh bootstrap-salt.sh git develop

will fail on Fedora31;

WARNING: Running pip install with root privileges is generally not a good idea. Try `python -m pip install --user` instead.
Ignoring tornado: markers 'python_version < "3"' don't match your environment
WARNING: Running pip install with root privileges is generally not a good idea. Try `python -m pip install --user` instead.
Collecting tornado<5.0,>=4.2.1
  Downloading https://files.pythonhosted.org/packages/e3/7b/e29ab3d51c8df66922fea216e2bddfcb6430fb29620e5165b16a216e0d3c/tornado-4.5.3.tar.gz (484kB)
Installing collected packages: tornado
  Running setup.py install for tornado: started
    Running setup.py install for tornado: finished with status 'done'
Successfully installed tornado-4.5.3
 *  INFO: Running config_salt()
 *  INFO: Running install_fedora_git()
bootstrap-salt.sh: line 3942: python2: command not found
 * ERROR: Failed to run install_fedora_git()!!!

[arizvisa]

@53c70r, does saltstack/salt-bootstrap#1424 fix things?

[53c70r]

No since the Merge was 9 days ago and i executed the commands 3 days ago.

[53c70r]

vim bootstrap-salt.sh +3942

install_fedora_git() {
    if [ "${_PY_EXE}" != "" ]; then
        _PYEXE=${_PY_EXE}
        echoinfo "Using the following python version: ${_PY_EXE} to install salt"
    else
        _PYEXE='python2'
    fi

    if [ "${_POST_NEON_INSTALL}" -eq $BS_TRUE ]; then
         __install_salt_from_repo_post_neon "${_PY_EXE}" || return 1
        return 0
    fi

    if [ -f "${_SALT_GIT_CHECKOUT_DIR}/salt/syspaths.py" ]; then
        ${_PYEXE} setup.py --salt-config-dir="$_SALT_ETC_DIR" --salt-cache-dir="${_SALT_CACHE_DIR}" ${SETUP_PY_INSTALL_ARGS} install --prefix=/usr || return 1
    else
        ${_PYEXE} setup.py ${SETUP_PY_INSTALL_ARGS} install --prefix=/usr || return 1
    fi
    return 0
}

_PYEXE='python2'
but since Fedora does not have python2 it can't execute the command.

vim bootstrap-salt.sh +271
_PY_EXE=""

[arizvisa]

@53c70r, what about with '-x python3'?

[53c70r]

Works fine.

[brejoc]

If Salt would be upgraded to version 3000, the tornado dependency would be gone, since it's vendored in salt/ext now.

[dmurphy18]

@itamarjp With Salt 2000 release a version of Tornado compatible with Salt is embedded in Salt, it utilizes Tornado v4.5.3. Salt 3000 is now available for current versions of Fedora.

With the Sodium point release it is hope that Salt will be able to utilize bot Tornado v5.x and v6.x and work is on-going in this effort, and is currently under QA.

If this satisfies the issue , please consider closing this issue.

[arizvisa]

super glad that salt is finally vendorizing core dependencies. updating to newer releases should be significantly safer.

[sagetherage]

closing this issue as fixed in 3000