Salt: [BUG] AttributeError: 'Process' object has no attribute '_args_for_getstate'

Created on 21 Jun 2020  ·  26Comments  ·  Source: saltstack/salt

Description
salt-ssh crashes with an exception before doing seemingly anything

Setup
Just upgraded to salt 3001, 3000 was working fine

Steps to Reproduce the behavior
salt-ssh $host state.highstate test=True

AttributeError: 'Process' object has no attribute '_args_for_getstate'

Traceback (most recent call last):
  File "/usr/local/bin/salt-ssh", line 33, in <module>
    sys.exit(load_entry_point('salt==3001', 'console_scripts', 'salt-ssh')())
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/scripts.py", line 499, in salt_ssh
    client.run()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/cli/ssh.py", line 26, in run
    ssh.run()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/client/ssh/__init__.py", line 787, in run
    for ret in self.handle_ssh():
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/client/ssh/__init__.py", line 604, in handle_ssh
    routine.start()
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 755, in __getstate__
    args = self._args_for_getstate
AttributeError: 'Process' object has no attribute '_args_for_getstate'

Expected behavior
Salt executes on the host

Versions Report

salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 3001

Dependency Versions:
           cffi: 1.12.2
       cherrypy: unknown
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 2.0.6
      gitpython: 2.1.15
         Jinja2: 2.10.1
        libgit2: 1.0.1
       M2Crypto: 0.35.2
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: Not Installed
   pycryptodome: 3.9.7
         pygit2: 1.2.1
         Python: 3.8.3 (default, May 27 2020, 20:54:22)
   python-gnupg: 0.4.4
         PyYAML: 5.1.2
          PyZMQ: 19.0.0
          smmap: 3.0.4
        timelib: 0.2.4
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: darwin 19.5.0
         locale: utf-8
        machine: x86_64
        release: 19.5.0
         system: Darwin
        version: 10.15.5 x86_64

Aluminium Bug Confirmed Core MacOS Regression has-failing-test phase-execute severity-critical status-in-prog

Most helpful comment

This is a known issue being worked on in https://github.com/saltstack/salt/pull/57028 for magnesium

All 26 comments

@saltstack/team-core someone with MacOS Kung-Fu skills might take a look into this one

Same for 'SignalHandlingProcess':

[salt.minion      :505 ][ERROR   ][308] Exception 'SignalHandlingProcess' object has no attribute '_args_for_getstate' occurred in scheduled job
[tornado.application:353 ][ERROR   ][308] Future <salt.ext.tornado.concurrent.Future object at 0x108ebf5b0> exception was never retrieved: Traceback (most recent call last):
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/minion.py", line 1719, in _handle_decoded_payload
    process.start()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 867, in start
    super(SignalHandlingProcess, self).start()
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 755, in __getstate__
    args = self._args_for_getstate
AttributeError: 'SignalHandlingProcess' object has no attribute '_args_for_getstate'

Update: _args_for_getstate is only defined for Windows platforms:

https://github.com/saltstack/salt/blob/8a23ec902d2bc9f056adc1d039867c5adc6cdcd4/salt/utils/process.py#L705-L714

So method __getstate__, should not been called from different architectures:

https://github.com/saltstack/salt/blob/8a23ec902d2bc9f056adc1d039867c5adc6cdcd4/salt/utils/process.py#L745-L766

Update 2: This part of the code has been untouched for more than 5 years, so the issue could be related with having changed from Python 3.7 to 3.8 on macOS with salt 3001 update https://github.com/Homebrew/homebrew-core/pull/56461

I ran into this just running a test.ping against a mac minion. Getting my dev environment setup and going to take a swing at digging into this as time allows.

This is a temporary solution until the bug is fixed.

Edit the Homebrew salt formula (brew edit salt) to apply the following patch:

diff --git a/Formula/salt.rb b/Formula/salt.rb
index d4a87977f..196e27648 100644
--- a/Formula/salt.rb
+++ b/Formula/salt.rb
@@ -17,7 +17,7 @@ class Salt < Formula
   depends_on "libgit2"
   depends_on "libyaml"
   depends_on "[email protected]"
-  depends_on "[email protected]"
+  depends_on "python"
   depends_on "zeromq"

   on_linux do
@@ -61,7 +61,7 @@ class Salt < Formula
     # https://github.com/Homebrew/homebrew-core/pull/52835#issuecomment-617502578
     File.write(buildpath/"pkg/osx/req_pyobjc.txt", "")

-    venv = virtualenv_create(libexec, Formula["[email protected]"].bin/"python3.8")
+    venv = virtualenv_create(libexec, "python3")
     venv.pip_install resources

     system libexec/"bin/pip", "install", "-v", "--ignore-installed", buildpath

then:

brew uninstall salt
sudo rm -rf /usr/local/Cellar/salt/3001
brew install --build-from-source salt

May be we could create a PR at homebrew-core to apply this patch to the formula until the next Salt released is ready with the bug fixed.

@weswhet

I have tried adding this changes to salt/utils/process.py:

diff --git a/salt/utils/process.py b/salt/utils/process.py
index 2ac797b5f3..34e22b84cc 100644
--- a/salt/utils/process.py
+++ b/salt/utils/process.py
@@ -744,6 +744,9 @@ class Process(multiprocessing.Process, NewStyleClassMixIn):

     # __setstate__ and __getstate__ are only used on Windows.
     def __setstate__(self, state):
+        if not salt.utils.platform.is_windows():
+            self.__dict__.update(state)
+            return
         args = state["args"]
         kwargs = state["kwargs"]
         # This will invoke __init__ of the most derived class.
@@ -752,6 +755,8 @@ class Process(multiprocessing.Process, NewStyleClassMixIn):
         self._finalize_methods = self._finalize_methods

     def __getstate__(self):
+        if not salt.utils.platform.is_windows():
+            return self.__dict__
         args = self._args_for_getstate
         kwargs = self._kwargs_for_getstate
         if "log_queue" not in kwargs:

but then, the following error is reported:

[tornado.application:353 ][ERROR   ][1761] Future <salt.ext.tornado.concurrent.Future object at 0x1116c9550> exception was never retrieved: Traceback (most recent call last):
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper
    yielded = next(result)
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/minion.py", line 1719, in _handle_decoded_payload
    process.start()
  File "/usr/local/Cellar/salt/3001/libexec/lib/python3.8/site-packages/salt/utils/process.py", line 872, in start
    super(SignalHandlingProcess, self).start()
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Minion.setup_beacons.<locals>.handle_beacons'
[salt.minion      :505 ][ERROR   ][1761] Exception cannot pickle '_thread._local' object occurred in scheduled job

It looks this is the origin of the issue: https://docs.python.org/3/library/multiprocessing.html

Changed in version 3.8: On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess. See bpo-33725.

Update

Similar issue solved here: https://github.com/horovod/horovod/pull/1904

Gonna need to put up the @s0undt3ch bat signal on this one. He’s already working on it here #57028

Turning multiprocessing off in the minion config may patch this as well. If homebrew is going to be py3.8

This is a known issue being worked on in https://github.com/saltstack/salt/pull/57028 for magnesium

I will look at the PR and see if I can help in the test suite at all and look at if we can bring this into the point release.

This is really a blocker somehow, using @cdalvaro 's solution worked fine on one laptop.. but not on the other.. Dunno why :(

It is not working anymore because python formula has been upgraded from 3.7 to 3.8 ...

brew info python
[email protected]: stable 3.8.3 (bottled)
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/[email protected]/3.8.3_2 (4,229 files, 64.6MB)
  Poured from bottle on 2020-07-14 at 14:43:45
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/[email protected]
==> Dependencies
Build: pkg-config ✔
Required: gdbm ✔, [email protected] ✔, readline ✔, sqlite ✔, xz ✔
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/[email protected]/libexec/bin

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.8/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Analytics
install: 463,840 (30 days), 1,253,709 (90 days), 1,905,378 (365 days)
install-on-request: 48,437 (30 days), 71,239 (90 days), 96,550 (365 days)
build-error: 0 (30 days)

We need to fix the code for making it to work with _python 3.8_

Applying the following patch to your salt formula (brew edit salt) and install it building from source will allow you to run salt-minion again.

diff --git a/Formula/salt.rb b/Formula/salt.rb
index d4a87977f..196e27648 100644
--- a/Formula/salt.rb
+++ b/Formula/salt.rb
@@ -17,7 +17,7 @@ class Salt < Formula
   depends_on "libgit2"
   depends_on "libyaml"
   depends_on "[email protected]"
-  depends_on "[email protected]"
+  depends_on "[email protected]"
   depends_on "zeromq"

   on_linux do
@@ -61,7 +61,7 @@ class Salt < Formula
     # https://github.com/Homebrew/homebrew-core/pull/52835#issuecomment-617502578
     File.write(buildpath/"pkg/osx/req_pyobjc.txt", "")

-    venv = virtualenv_create(libexec, Formula["[email protected]"].bin/"python3.8")
+    venv = virtualenv_create(libexec, Formula["[email protected]"].bin/"python3.7")
     venv.pip_install resources

     system libexec/"bin/pip", "install", "-v", "--ignore-installed", buildpath
brew install --build-from-source salt

Undo your formula changes after installing it to allow future updates for the salt formula.

Do you have a workaround again maybe?

Hi @h0jeZvgoxFepBQ2C, I have checked that the new patch is working fine! Try it and let us know if its working for you too.

I have copied the official salt formula from homebrew-core and patched it to use [email protected]

You can install salt with python3.7 using my homebrew-tap:

brew install cdalvaro/tap/salt

⚠️ I will keep the formula updated using python 3.7 when the official one is updated until this bug is resolved.

You can install salt with python3.7 using my homebrew-tap:

Thanks @cdalvaro worked for me on Darwin.

Working for me now, once I remembered to kill the running salt-minion before starting up the new one.

This bug report is about salt-ssh though, no running salt-minion involved.

De-scoping from Magnesium as per Pedro https://github.com/saltstack/salt/pull/57028#issuecomment-693098419

Ran into this when trying to use salt-ssh from my mac. Doing the 3.7 fix with brew edit worked.

I cant get cdalvaro/tap/salt to work. I always get the same error even though "importlib-metadata" is installed:

==> Installing cdalvaro/tap/salt
==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi
==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi
==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi
==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi
==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target --install-scripts=/private/tmp/salt--homebrew-virtualen
==> /usr/local/opt/[email protected]/bin/python3.7 -s /private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/bin/virtualenv -p /usr/local/opt/[email protected]/bin/python3.7 /usr/local/Cellar/salt/3001.1/libexec
Last 15 lines from /Users/user/Library/Logs/Homebrew/salt/06.python3.7:
  File "/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/bin/virtualenv", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3254, in <module>
    @_call_aside
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3238, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3267, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'importlib-metadata<3,>=0.12' distribution was not found and is required by virtualenv

@cdalvaro do you have any idea maybe?

I cant get cdalvaro/tap/salt to work. I always get the same error even though "importlib-metadata" is installed:


==> Installing cdalvaro/tap/salt

==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi

==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi

==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi

==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/vendor --install-scripts=/private/tmp/salt--homebrew-vi

==> /usr/local/opt/[email protected]/bin/python3.7 -c import setuptools... --no-user-cfg install --prefix=/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target --install-scripts=/private/tmp/salt--homebrew-virtualen

==> /usr/local/opt/[email protected]/bin/python3.7 -s /private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/bin/virtualenv -p /usr/local/opt/[email protected]/bin/python3.7 /usr/local/Cellar/salt/3001.1/libexec

Last 15 lines from /Users/user/Library/Logs/Homebrew/salt/06.python3.7:

  File "/private/tmp/salt--homebrew-virtualenv-20201013-95065-w8rmdq/target/bin/virtualenv", line 6, in <module>

    from pkg_resources import load_entry_point

  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3254, in <module>

    @_call_aside

  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3238, in _call_aside

    f(*args, **kwargs)

  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3267, in _initialize_master_working_set

    working_set = WorkingSet._build_master()

  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 584, in _build_master

    ws.require(__requires__)

  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 901, in require

    needed = self.resolve(parse_requirements(requirements))

  File "/usr/local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 787, in resolve

    raise DistributionNotFound(req, requirers)

pkg_resources.DistributionNotFound: The 'importlib-metadata<3,>=0.12' distribution was not found and is required by virtualenv

@cdalvaro do you have any idea maybe?

I'll check it ASAP!

@sebastianreloaded - brew install cdalvaro/tap/salt should be working again

@cdalvaro Thank you, this pointed me in the right direction to install salt via pip in a virtualenv.

But using brew i still get pkg_resources.DistributionNotFound: The 'importlib-metadata<3,>=0.12' distribution was not found and is required by virtualenv

I mean https://raw.githubusercontent.com/saltstack/salt/master/requirements/static/pkg/py3.7/darwin.txt doesnt contain importlib-metadata as a requirement, only the ci-version has: https://github.com/saltstack/salt/blob/12f7e0672055fca36dab79c0295316d9e7a34f79/requirements/static/ci/py3.7/darwin.txt

But i'm too unfamiliar with this to know whether it is an error or a problem on my side.

@cdalvaro Thank you, this pointed me in the right direction to install salt via pip in a virtualenv.

But using brew i still get pkg_resources.DistributionNotFound: The 'importlib-metadata<3,>=0.12' distribution was not found and is required by virtualenv

I mean https://raw.githubusercontent.com/saltstack/salt/master/requirements/static/pkg/py3.7/darwin.txt doesnt contain importlib-metadata as a requirement, only the ci-version has: https://github.com/saltstack/salt/blob/12f7e0672055fca36dab79c0295316d9e7a34f79/requirements/static/ci/py3.7/darwin.txt

But i'm too unfamiliar with this to know whether it is an error or a problem on my side.

@sebastianreloaded - Since this seams to be related with the homebrew formula, please, could you open me an issue in cdalvaro/homebrew-tap repository attaching me the full log of the homebrew installation (/Users/user/Library/Logs/Homebrew/salt/06.python3.7)?

After finding the solution, I'll post it here too in order to avoid spamming people.

Was this page helpful?
0 / 5 - 0 ratings