Description
salt-ssh crashes with an exception before doing seemingly anything
Setup
Just upgraded to salt 3001, 3000 was working fine
Steps to Reproduce the behavior
salt-ssh $host state.highstate test=True
AttributeError: 'Process' object has no attribute '_args_for_getstate'
Traceback (most recent call last):
File "/usr/local/bin/salt-ssh", line 33, in <module>
sys.exit(load_entry_point('salt==3001', 'console_scripts', 'salt-ssh')())
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/scripts.py", line 499, in salt_ssh
client.run()
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/cli/ssh.py", line 26, in run
ssh.run()
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/client/ssh/__init__.py", line 787, in run
for ret in self.handle_ssh():
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/client/ssh/__init__.py", line 604, in handle_ssh
routine.start()
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 755, in __getstate__
args = self._args_for_getstate
AttributeError: 'Process' object has no attribute '_args_for_getstate'
Expected behavior
Salt executes on the host
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
Salt: 3001
Dependency Versions:
cffi: 1.12.2
cherrypy: unknown
dateutil: 2.8.0
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 2.1.15
Jinja2: 2.10.1
libgit2: 1.0.1
M2Crypto: 0.35.2
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.19
pycrypto: Not Installed
pycryptodome: 3.9.7
pygit2: 1.2.1
Python: 3.8.3 (default, May 27 2020, 20:54:22)
python-gnupg: 0.4.4
PyYAML: 5.1.2
PyZMQ: 19.0.0
smmap: 3.0.4
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.2
System Versions:
dist: darwin 19.5.0
locale: utf-8
machine: x86_64
release: 19.5.0
system: Darwin
version: 10.15.5 x86_64
@saltstack/team-core someone with MacOS Kung-Fu skills might take a look into this one
Same for 'SignalHandlingProcess
':
[salt.minion :505 ][ERROR ][308] Exception 'SignalHandlingProcess' object has no attribute '_args_for_getstate' occurred in scheduled job
[tornado.application:353 ][ERROR ][308] Future <salt.ext.tornado.concurrent.Future object at 0x108ebf5b0> exception was never retrieved: Traceback (most recent call last):
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
yielded = next(result)
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/minion.py", line 1719, in _handle_decoded_payload
process.start()
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 867, in start
super(SignalHandlingProcess, self).start()
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 755, in __getstate__
args = self._args_for_getstate
AttributeError: 'SignalHandlingProcess' object has no attribute '_args_for_getstate'
Update: _args_for_getstate
is only defined for Windows platforms:
So method __getstate__
, should not been called from different architectures:
Update 2: This part of the code has been untouched for more than 5 years, so the issue could be related with having changed from Python 3.7 to 3.8 on macOS with salt 3001 update https://github.com/Homebrew/homebrew-core/pull/56461
I ran into this just running a test.ping against a mac minion. Getting my dev environment setup and going to take a swing at digging into this as time allows.
This is a temporary solution until the bug is fixed.
Edit the Homebrew salt
formula (brew edit salt
) to apply the following patch:
diff --git a/Formula/salt.rb b/Formula/salt.rb
index d4a87977f..196e27648 100644
--- a/Formula/salt.rb
+++ b/Formula/salt.rb
@@ -17,7 +17,7 @@ class Salt < Formula
depends_on "libgit2"
depends_on "libyaml"
depends_on "[email protected]"
- depends_on "[email protected]"
+ depends_on "python"
depends_on "zeromq"
on_linux do
@@ -61,7 +61,7 @@ class Salt < Formula
# https://github.com/Homebrew/homebrew-core/pull/52835#issuecomment-617502578
File.write(buildpath/"pkg/osx/req_pyobjc.txt", "")
- venv = virtualenv_create(libexec, Formula["[email protected]"].bin/"python3.8")
+ venv = virtualenv_create(libexec, "python3")
venv.pip_install resources
system libexec/"bin/pip", "install", "-v", "--ignore-installed", buildpath
then:
brew uninstall salt
sudo rm -rf /usr/local/Cellar/salt/3001
brew install --build-from-source salt
May be we could create a PR at homebrew-core to apply this patch to the formula until the next Salt released is ready with the bug fixed.
@weswhet
I have tried adding this changes to salt/utils/process.py
:
diff --git a/salt/utils/process.py b/salt/utils/process.py
index 2ac797b5f3..34e22b84cc 100644
--- a/salt/utils/process.py
+++ b/salt/utils/process.py
@@ -744,6 +744,9 @@ class Process(multiprocessing.Process, NewStyleClassMixIn):
# __setstate__ and __getstate__ are only used on Windows.
def __setstate__(self, state):
+ if not salt.utils.platform.is_windows():
+ self.__dict__.update(state)
+ return
args = state["args"]
kwargs = state["kwargs"]
# This will invoke __init__ of the most derived class.
@@ -752,6 +755,8 @@ class Process(multiprocessing.Process, NewStyleClassMixIn):
self._finalize_methods = self._finalize_methods
def __getstate__(self):
+ if not salt.utils.platform.is_windows():
+ return self.__dict__
args = self._args_for_getstate
kwargs = self._kwargs_for_getstate
if "log_queue" not in kwargs:
but then, the following error is reported:
[tornado.application:353 ][ERROR ][1761] Future <salt.ext.tornado.concurrent.Future object at 0x1116c9550> exception was never retrieved: Traceback (most recent call last):
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
yielded = next(result)
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/minion.py", line 1719, in _handle_decoded_payload
process.start()
File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 872, in start
super(SignalHandlingProcess, self).start()
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
return Popen(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Minion.setup_beacons.<locals>.handle_beacons'
[salt.minion :505 ][ERROR ][1761] Exception cannot pickle '_thread._local' object occurred in scheduled job
It looks this is the origin of the issue: https://docs.python.org/3/library/multiprocessing.html
Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.
Update
Similar issue solved here: https://github.com/horovod/horovod/pull/1904
Gonna need to put up the @s0undt3ch bat signal on this one. He’s already working on it here #57028
Turning multiprocessing off in the minion config may patch this as well. If homebrew is going to be py3.8
This is a known issue being worked on in https://github.com/saltstack/salt/pull/57028 for magnesium
I will look at the PR and see if I can help in the test suite at all and look at if we can bring this into the point release.
This is really a blocker somehow, using @cdalvaro 's solution worked fine on one laptop.. but not on the other.. Dunno why :(
It is not working anymore because python formula has been upgraded from 3.7 to 3.8 ...
brew info python
[email protected]: stable 3.8.3 (bottled)
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/[email protected]/3.8.3_2 (4,229 files, 64.6MB)
Poured from bottle on 2020-07-14 at 14:43:45
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/[email protected]
==> Dependencies
Build: pkg-config ✔
Required: gdbm ✔, [email protected] ✔, readline ✔, sqlite ✔, xz ✔
==> Caveats
Python has been installed as
/usr/local/bin/python3
Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
/usr/local/opt/[email protected]/libexec/bin
You can install Python packages with
pip3 install <package>
They will install into the site-package directory
/usr/local/lib/python3.8/site-packages
See: https://docs.brew.sh/Homebrew-and-Python
==> Analytics
install: 463,840 (30 days), 1,253,709 (90 days), 1,905,378 (365 days)
install-on-request: 48,437 (30 days), 71,239 (90 days), 96,550 (365 days)
build-error: 0 (30 days)
We need to fix the code for making it to work with _python 3.8_
Applying the following patch to your salt formula (brew edit salt
) and install it building from source will allow you to run salt-minion
again.
diff --git a/Formula/salt.rb b/Formula/salt.rb
index d4a87977f..196e27648 100644
--- a/Formula/salt.rb
+++ b/Formula/salt.rb
@@ -17,7 +17,7 @@ class Salt < Formula
depends_on "libgit2"
depends_on "libyaml"
depends_on "[email protected]"
- depends_on "[email protected]"
+ depends_on "[email protected]"
depends_on "zeromq"
on_linux do
@@ -61,7 +61,7 @@ class Salt < Formula
# https://github.com/Homebrew/homebrew-core/pull/52835#issuecomment-617502578
File.write(buildpath/"pkg/osx/req_pyobjc.txt", "")
- venv = virtualenv_create(libexec, Formula["[email protected]"].bin/"python3.8")
+ venv = virtualenv_create(libexec, Formula["[email protected]"].bin/"python3.7")
venv.pip_install resources
system libexec/"bin/pip", "install", "-v", "--ignore-installed", buildpath
brew install --build-from-source salt
Undo your formula changes after installing it to allow future updates for the salt formula.
Do you have a workaround again maybe?
Hi @h0jeZvgoxFepBQ2C, I have checked that the new patch is working fine! Try it and let us know if its working for you too.
I have copied the official salt formula from homebrew-core and patched it to use [email protected]
You can install salt with python3.7 using my homebrew-tap:
brew install cdalvaro/tap/salt
⚠️ I will keep the formula updated using python 3.7 when the official one is updated until this bug is resolved.
You can install salt with python3.7 using my homebrew-tap:
Thanks @cdalvaro worked for me on Darwin.
Working for me now, once I remembered to kill the running salt-minion
before starting up the new one.
This bug report is about salt-ssh though, no running salt-minion involved.
De-scoping from Magnesium as per Pedro https://github.com/saltstack/salt/pull/57028#issuecomment-693098419
Ran into this when trying to use salt-ssh from my mac. Doing the 3.7 fix with brew edit worked.
I cant get cdalvaro/tap/salt to work. I always get the same error even though "importlib-metadata" is installed:
==> Installing cdalvaro/tap/salt
==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi
==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi
==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi
==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi
==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target --install-scripts=/private/tmp/salt--homebrew-virtualen
==> /usr/local/opt/[email protected]/bin/python3.7 -s /private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/bin/virtualenv -p /usr/local/opt/[email protected]/bin/python3.7 /usr/local/Cellar/salt/3001.1/libexec
Last 15 lines from /Users/user/Library/Logs/Homebrew/salt/06.python3.7:
File "/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/bin/virtualenv", line 6, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3254, in <module>
@_call_aside
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3238, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3267, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 584, in _build_master
ws.require(__requires__)
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 901, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 787, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'importlib-metadata<3,>=0.12' distribution was not found and is required by virtualenv
@cdalvaro do you have any idea maybe?
I cant get cdalvaro/tap/salt to work. I always get the same error even though "importlib-metadata" is installed:
==> Installing cdalvaro/tap/salt ==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi ==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi ==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi ==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi ==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target --install-scripts=/private/tmp/salt--homebrew-virtualen ==> /usr/local/opt/[email protected]/bin/python3.7 -s /private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/bin/virtualenv -p /usr/local/opt/[email protected]/bin/python3.7 /usr/local/Cellar/salt/3001.1/libexec Last 15 lines from /Users/user/Library/Logs/Homebrew/salt/06.python3.7: File "/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/bin/virtualenv", line 6, in <module> from pkg_resources import load_entry_point File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3254, in <module> @_call_aside File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3238, in _call_aside f(*args, **kwargs) File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3267, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 584, in _build_master ws.require(__requires__) File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 901, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 787, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'importlib-metadata<3,>=0.12' distribution was not found and is required by virtualenv
@cdalvaro do you have any idea maybe?
I'll check it ASAP!
@sebastianreloaded - brew install cdalvaro/tap/salt
should be working again
@cdalvaro Thank you, this pointed me in the right direction to install salt via pip in a virtualenv.
But using brew i still get pkg_resources.DistributionNotFound: The 'importlib-metadata<3,>=0.12' distribution was not found and is required by virtualenv
I mean https://raw.githubusercontent.com/saltstack/salt/master/requirements/static/pkg/py3.7/darwin.txt doesnt contain importlib-metadata
as a requirement, only the ci-version has: https://github.com/saltstack/salt/blob/12f7e0672055fca36dab79c0295316d9e7a34f79/requirements/static/ci/py3.7/darwin.txt
But i'm too unfamiliar with this to know whether it is an error or a problem on my side.
@cdalvaro Thank you, this pointed me in the right direction to install salt via pip in a virtualenv.
But using brew i still get
pkg_resources.DistributionNotFound: The 'importlib-metadata<3,>=0.12' distribution was not found and is required by virtualenv
I mean https://raw.githubusercontent.com/saltstack/salt/master/requirements/static/pkg/py3.7/darwin.txt doesnt contain
importlib-metadata
as a requirement, only the ci-version has: https://github.com/saltstack/salt/blob/12f7e0672055fca36dab79c0295316d9e7a34f79/requirements/static/ci/py3.7/darwin.txtBut i'm too unfamiliar with this to know whether it is an error or a problem on my side.
@sebastianreloaded - Since this seams to be related with the homebrew formula, please, could you open me an issue in cdalvaro/homebrew-tap repository attaching me the full log of the homebrew installation (/Users/user/Library/Logs/Homebrew/salt/06.python3.7
)?
After finding the solution, I'll post it here too in order to avoid spamming people.
@cdalvaro add here: https://github.com/cdalvaro/homebrew-tap/issues/19
Most helpful comment
This is a known issue being worked on in https://github.com/saltstack/salt/pull/57028 for magnesium