supervisorctl perform start or stop but return with XMLRPC Error

[sboisson]

supervisorctl on same machine than daemon seems to process start and stop command successfully, but return an XMLRPC error when the the command was executed successfully

Running supervisor-3.0b1 on Python 2.6 CentOS;
Tried with and without user/password in supervisors.conf

Steps:

# supervisorctl status myservice
myservice                       RUNNING    pid 6831, uptime 0:02:35

# supervisorctl stop myservice
error: <class 'xmlrpclib.ProtocolError'>, <ProtocolError for 127.0.0.1/RPC2: 500 Internal Server Error>: file: /usr/lib/python2.6/site-packages/supervisor-3.0b1-py2.6.egg/supervisor/xmlrpc.py line: 471

# supervisorctl status myservice
myservice                       STOPPED    Dec 03 09:57 AM

# supervisorctl stop myservice
myservice: ERROR (not running)

# supervisorctl start  myservice
error: <class 'xmlrpclib.ProtocolError'>, <ProtocolError for 127.0.0.1/RPC2: 500 Internal Server Error>: file: /usr/lib/python2.6/site-packages/supervisor-3.0b1-py2.6.egg/supervisor/xmlrpc.py line: 471

# supervisorctl start  myservice
myservice: ERROR (already started)

[mnaberez]

Could you please look in the main log file (supervisord.log) for the 500 Internal Server Error? There should be a Python traceback in the log that can give us more information about what happened.

[mnaberez]

@sboisson We weren't able to reproduce this. Closing for now, if you are able to give more info we'll reopen it.

[bachirelkhoury]

I am having that exact issue when executing any supervisorctl commands.

error: , : file: /root/.virtualenvs/nowcam_env/lib/python2.7/site-packages/supervisor/xmlrpc.py line: 471

Looking through supervisord.log I can see:

2013-04-25 23:48:43,560 CRIT Traceback (most recent call last):
File "/root/.virtualenvs/nowcam_env/lib/python2.7/site-packages/supervisor/xmlrpc.py", line 330, in continue_request
params, method = self.loads(data)
File "/root/.virtualenvs/nowcam_env/lib/python2.7/site-packages/supervisor/xmlrpc.py", line 565, in loads
for action, elem in iterparse(StringIO(data)):
File "/opt/local/lib/python2.7/xml/etree/ElementTree.py", line 1195, in iterparse
parser = XMLParser(target=TreeBuilder())
File "/opt/local/lib/python2.7/xml/etree/ElementTree.py", line 1447, in init
"No module named expat; use SimpleXMLTreeBuilder instead"
ImportError: No module named expat; use SimpleXMLTreeBuilder instead

So not sure what's the fix, or what would be causing this. Bad python build on the server?

The only thing i found was this change to the code
http://stackoverflow.com/questions/1068510/using-simplexmltreebuilder-in-elementtree

Please help pointing in the right direction if you can.
Thanks

[scuml]

It looks like the heart of this issue is a missing dynamic library: /usr/lib/python2.7/lib-dynload/pyexpat.so

Python is trying to run
import pyexpat as expat and raising the above error.

A solution is to reinstall python. I ran into this on a Joyent SmartMachine and fortunately they have the python expat module in their package library so this line fixed things.
sudo pkgin in py27-expat-2.7.5

[mnaberez]

I'm going to close this issue as it seems to be caused by broken Python installations. If there's something Supervisor can do better to help with this, please comment and we can reopen.

[robertogyn19]

Just run into this problem. In my case, follow the steps:

• apt-get install python-dev
• service supervisor stop
• service supervisor start

[zhengl7]

Just ran into same issue.

The workaround I eventually figured out is not to start supervisor daemon with custom conf file location. Putting the supervisord.conf under /etc and start directly with supervisord cmd.

[buhuipao]

I have meet the same errror, like this:

 <class 'xmlrpclib.ProtocolError'>, <ProtocolError for 127.0.0.1/RPC2: 500 Internal Server Error>: file: /usr/lib/pymodules/python2.7/supervisor/xmlrpc.py line: 474

when I excute some command , like below:

supervisor> restart fusion_check
FAILED: attempted to kill fusion_check with sig SIGTERM but it wasn't running
fusion_check: ERROR (already started)
supervisor> stop fusion_check
FAILED: attempted to kill fusion_check with sig SIGTERM but it wasn't running

now, My server havn't started, but in supervisor , it look like this:

fusion_check                     RUNNING    pid 0, uptime 1:44:02

Please help me with this error, thank you!

[jesoonluo]

I have the same error info, that is I use the username, but don't unlocked password in unix_http_server, you can set you passwd when you set username unlocked.

[jesoonluo]

@buhuipao 你先直接把supervisor的进程杀了设置好了再重启，那是你*.conf设置里面用了用户名却没用密码。

[marzdgzmn]

Just ran into same issue.

The workaround I eventually figured out is not to start supervisor daemon with custom conf file location. Putting the supervisord.conf under /etc and start directly with supervisord cmd.

Thanks. Tried, this and now it's workingo n CentOS 7.