Salt: cherrypy stats issue

[jcourington]

Description of Issue/Question

curl -ksi https://localhost:8000/stats -H "Accept: application/x-yaml" -H "X-Auth-Token: $AUTHTOKEN"

causes the following error in salt-api
[INFO ] 127.0.0.1 - - [18/Sep/2017:13:54:18] "POST /login HTTP/1.1" 200 207 "" "curl/7.29.0"
[DEBUG ] Error while processing request for: /stats
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/app.py", line 737, in hypermedia_handler
ret = cherrypy.serving.request._hypermedia_inner_handler(args, *kwargs)
File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 60, in __call__
return self.callable(self.args, *self.kwargs)
File "/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/app.py", line 2621, in GET
from cherrypy.lib import cpstats
File "/usr/lib/python2.7/site-packages/cherrypy/lib/cpstats.py", line 438, in
class StatsPage(object):
File "/usr/lib/python2.7/site-packages/cherrypy/lib/cpstats.py", line 478, in StatsPage
@cherrypy.exposed
AttributeError: 'module' object has no attribute 'exposed'

Fix

cherrypy/lib/cpstats.py line 478 needs to be changed from @cherrypy.exposed to @cherrypy.expose (without the d)

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

Versions Report

Salt Version:
Salt: 2017.7.1

Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: Not Installed
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.5 (default, Nov 6 2016, 00:28:07)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: centos 7.3.1611 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-514.26.2.el7.x86_64
system: Linux
version: CentOS Linux 7.3.1611 Core

[garethgreenaway]

@jcourington Is this something that has changed recently in the cherrypy library? This looks like an issue in the upstream project.

[jcourington]

I don't know. We are new to saltstack and I do not have anything to compare it against.

[whiteinge]

Odd that the CherryPy version doesn't show up in the versions report. How was it installed?

[jcourington]

It was installed via "yum install salt-api" from the saltstack repository.

---

From: Seth House [e-mail]
Sent: Tuesday, September 19, 2017 7:31 PM
To: saltstack/salt
Cc: jcourington; Mention
Subject: Re: [saltstack/salt] cherrypy stats issue (#43581)

Odd that the CherryPy version doesn't show up in the versions report. How was it installed?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/saltstack/salt/issues/43581#issuecomment-330702471, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIxo6m8CfN4ip3PV3xjEaHYmwsU9jwHpks5skE7bgaJpZM4PbUVZ.

[whiteinge]

Interesting. Will you please run the following too?

python -c 'import cherrypy; print cherrypy.__version__'

salt-api --version

[jcourington]

$ python -c 'import cherrypy; print cherrypy.__version__'
unknown

$ salt-api --version
salt-api 2017.7.1 (Nitrogen)

---

From: Seth House [e-mail]
Sent: Thursday, September 21, 2017 1:21 AM
To: saltstack/salt
Cc: jcourington; Mention
Subject: Re: [saltstack/salt] cherrypy stats issue (#43581)

Interesting. Will you please run the following too?

python -c 'import cherrypy; print cherrypy.__version__'

salt-api --version

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/saltstack/salt/issues/43581#issuecomment-331054038, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIxo6qGCcf6KozZGm70-qpOm3otH5rx1ks5skfJngaJpZM4PbUVZ.

[whiteinge]

Huh. More oddities. 🤔

Any chance there's a second, overlapping CherryPy install on that system? I'd suggest nuking the package and then searching the system for possible remnants before reinstalling and trying again.

[jfoboss]

Hi, folks!

We've got the same error on our system:

salt-api[12247]: self._target(*self._args, **self._kwargs)
salt-api[12247]: File "/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/__init__.py", line 79, in start
salt-api[12247]: from . import app
salt-api[12247]: File "/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/app.py", line 492, in <module>
salt-api[12247]: from cherrypy.lib import cpstats
salt-api[12247]: File "/usr/lib/python2.7/site-packages/cherrypy/lib/cpstats.py", line 438, in <module>
salt-api[12247]: class StatsPage(object):
salt-api[12247]: File "/usr/lib/python2.7/site-packages/cherrypy/lib/cpstats.py", line 478, in StatsPage
salt-api[12247]: @cherrypy.exposed
salt-api[12247]: AttributeError: 'module' object has no attribute 'exposed'

I tried to remove python-cherrypy, looked up for orphaned files via find / -type f -name "cpstats.py", reinstall salt-api + cherrypy with no result.

Changed @cherrypy.exposed to @cherrypy.expose in /usr/lib/python2.7/site-packages/cherrypy/lib/cpstats.py - and everything began to work again.

---

# python -c 'import cherrypy; print cherrypy.__version__'
5.6.0

# salt-api --version
salt-api 2017.7.2 (Nitrogen)

#lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.4.1708 (Core) 
Release:    7.4.1708
Codename:   Core

[whiteinge]

@jfoboss thank you for the sleuthing. I can reproduce the traceback on that CherryPy version. The PR above should resolve the issue by disabling stats if there is an exception.

I'd still like to track down the source of the missing version string also mentioned in this issue but that can be a separate task.

[stdhi]

hi, guys, I have the same issue.

$ salt-api --versions-report
Salt Version:
           Salt: 2017.7.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: 5.6.0
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.3
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.3.1611 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-514.26.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Cor

$salt-master verions-report

Salt Version:
           Salt: 2017.7.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: 5.6.0
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.3
         pygit2: Not Installed
         Python: 2.7.5 (default, Nov  6 2016, 00:28:07)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.3.1611 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-514.26.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.3.1611 Core

and there is the exception in the journal of system.

Oct 25 15:25:06 salt salt-api[24137]: AttributeError: 'module' object has no attribute 'exposed'
Oct 25 15:25:16 salt salt-api[24137]: [INFO    ] Process <function start at 0x2910ed8> (24744) died with exit status 1, restarting...
Oct 25 15:25:16 salt salt-api[24137]: Process salt.loaded.int.netapi.rest_cherrypy.start:
Oct 25 15:25:16 salt salt-api[24137]: Traceback (most recent call last):
Oct 25 15:25:16 salt salt-api[24137]: File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Oct 25 15:25:16 salt salt-api[24137]: self.run()
Oct 25 15:25:16 salt salt-api[24137]: File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
Oct 25 15:25:16 salt salt-api[24137]: self._target(*self._args, **self._kwargs)
Oct 25 15:25:16 salt salt-api[24137]: File "/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/__init__.py", line 79, in start
Oct 25 15:25:16 salt salt-api[24137]: from . import app
Oct 25 15:25:16 salt salt-api[24137]: File "/usr/lib/python2.7/site-packages/salt/netapi/rest_cherrypy/app.py", line 492, in <module>
Oct 25 15:25:16 salt salt-api[24137]: from cherrypy.lib import cpstats
Oct 25 15:25:16 salt salt-api[24137]: File "/usr/lib/python2.7/site-packages/cherrypy/lib/cpstats.py", line 438, in <module>
Oct 25 15:25:16 salt salt-api[24137]: class StatsPage(object):
Oct 25 15:25:16 salt salt-api[24137]: File "/usr/lib/python2.7/site-packages/cherrypy/lib/cpstats.py", line 478, in StatsPage
Oct 25 15:25:16 salt salt-api[24137]: @cherrypy.exposed

I digged the cherry code. and there is no cherrypy.exposed in 5.6.0, only a cherrypy.expose

fix it quickly like this:
vim /usr/lib/python2.7/site-packages/cherrypy/lib/cpstats.py
edit the @cherrypy.exposed to @cherrypy.expose

[whiteinge]

@stduolc the fix above will be in the next point-release of Salt.

We've also patched the copy of CherryPy that we distribute from the SaltStack repository so try using yum to update your install of python-cherrypy.

[icycle77]

I'm now seeing this issue as well. Re: potential conflicts, here is one:

# pip list |grep -i cherry
CherryPy (5.6.0)
# rpm -qa|grep -i cherry
python-cherrypy-5.6.0-1.el7.noarch

[isaachui]

@stduolc Salt version 2016.11.8 is still using the older rpm here

Had to update my own repository with the specified version: python-cherrypy-5.6.0-2.el7.noarch.rpm

[judexzhu]

Hi , I'm having the issue as well, unfortunately, I 'm trying install salt and saltapi on my mbp via pip

from the 2017.7.2 release

KNOWN ISSUES
On 2017.7.2 when using salt-api and cherrypy version 5.6.0, issue #43581 will occur when starting the salt-api service. We have patched the cherry-py packages for python-cherrypy-5.6.0-2 from repo.saltstack.com. If you are using python-cherrypy-5.6.0-1 please ensure to run yum install python-cherrypy to install the new patched version.

Extended changelog courtesy of Todd Stansell (https://github.com/tjstansell/salt-changelogs):

Generated at: 2017-10-02T21:10:14Z

but I cannot install CheeryPy==5.6.0-2 with pip, I check the repo.saltstack.com.seems only have the cherrypy package for el7

anyway I can fix it by install cherrypy 5.6.0-2 in my macbook ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.