Elpy: Backend error

[spacecowboy]

When I open a python file and move the cursor in the buffer, a new pane appears with the following output:

Error initializing Elpy

There was an error when trying to start the backend. Elpy can not work
until this problem is solved. The following lines were received from
Python, and might help identifying the problem.

Unhandled exception in thread started by 

What can I do to find the root cause?

[jorgenschaefer]

That's a new one. Is that the full output in the buffer? There's a buffer called *elpy-rpc... (note the space at the beginning), it might contain more information.

What does python -m elpy output?

[spacecowboy]

Perhaps not so surprising..

Did I miss something? Your quick installation instructions do not specify a python elpy module to be installed?

Edit: And the output I gave above is displayed in the *elpy-rpc... buffer

[spacecowboy]

And to clarify, I am not doing anything different atm (git clone etc) than what is described in the quick installation instructions.

[jorgenschaefer]

Please add the directory of the elpy package – for example, ~/.emacs.d/elpa/elpy-1.7.0/ – to PYTHONPATH and run the command again? E.g.:

PYTHONPATH=$HOME/.emacs.d/elpa/elpy-1.7.0/ python -m elpy

(The Elpy elisp code does that for you when it starts an Elpy RPC process from Emacs, that's why it's not mentioned anywhere, sorry :-))

[spacecowboy]

That seems to work.

[jorgenschaefer]

That is very weird behavior.

Your original error means Emacs is calling Elpy the way you just did, but instead of elpy-rpc ready (1.7.0), it gets something about an unhandled exception. So there is some kind of environment difference between how you call this from the command line and how Emacs calls the process.

The full command Emacs uses is:

`which python` -W ignore -m elpy.__main__

You can see the full python path Emacs found in the name of the *elpy-rpc...* buffer.

Does that work? If yes, does the following work in Emacs?

M-: (pop-to-buffer (elpy-rpc--open (elpy-library-root) elpy-rpc-python-command))

It should pop up a buffer with some contents. I would hope it contains more information than your original error.

The best guess I have right now is that the new importmagic code, which starts a thread to build a symbol index, is stumbling over something unusual in your code base, but I have no idea what that would be – the code is surrounded by a try/except clause, so it should not cause an "unhandled exception".

Do you have any unusual modules installed in your virtualenv? Something that might start a thread simple when it is imported?

[spacecowboy]

There is an error printed when I call it in that fashion (this is due to which python), could that affect things?

Example:

Result of running the Emacs command:

Creating a completely new conda environment with only elpy requirements installed gives the same results.

This is what which python prints:

[jorgenschaefer]

There seems to a problem with your python installation – /home/jonas/anaconda3/bin/python apparently emits a line title:8: parse error? That should not happen. Any idea what might be going on there?

[spacecowboy]

Did some experiments. This is not an issue with elpy, but with zsh which is my normal shell.

And it has nothing to do with python at all it seems:

$ $(which echo)
title:8: parse error
zsh: command not found: echo:

[spacecowboy]

It might be this bug (I use oh-my-zsh):
https://github.com/robbyrussell/oh-my-zsh/issues/2182

[jorgenschaefer]

That would explain the title line in the manual example, but I do not see why the Emacs command would create the error it does. Also, Emacs does not actually use the shell.

If you use the output of which directly, without the shell substitution, does that still work? I.e.

/home/jonas/anaconda3/bin/python -W ignore -m elpy.__main__

I assume that works without a problem, which brings us back to the problem in Emacs.

python -m elpy does very little before it emits the elpy-rpc ready line, and you do not see that at all in your RPC buffer, so something _really_ weird is going on there.

Though I doubt this is it, just to rule it out: Do you have the importmagic module installed in that virtualenv? For testing purposes, could you remove that and see if Emacs still emits that error?

What is the outout of pip freeze in that virtualenv?

[spacecowboy]

I fixed the title-error by updating my zsh-files.

However the error still occurs.

Pip-freeze output (before removal of importmagic):

$ pip freeze          
Error when trying to get requirement for VCS system Command /usr/bin/git config remote.origin.url failed with error code 1 in /home/jonas/workspacepython/pysurvival, falling back to uneditable format
Could not determine repository location of /home/jonas/workspacepython/pysurvival
Cython==0.20.1
DataShape==0.2
Flask==0.10.1
Flask-OAuthlib==0.7.0
Flask-RESTful==0.2.12
Flask-SQLAlchemy==2.0
Jinja2==2.7.2
MarkupSafe==0.18
Pillow==2.4.0
PyYAML==3.11
Pygments==1.6
SQLAlchemy==0.9.7
Sphinx==1.2.2
Werkzeug==0.9.6
XlsxWriter==0.5.5
aniso8601==0.83
argcomplete==0.6.7
astroid==1.2.1
astropy==0.3.2
backports.ssl-match-hostname==3.4.0.2
beautifulsoup4==4.3.1
binstar==0.5.3
bitarray==0.8.1
blaze==0.5
blz==0.6.2
bokeh==0.4.4
cdecimal==2.3
colorama==0.2.7
conda==3.8.4
conda-build==1.3.5
conda-env==2.0.2
configobj==5.0.5
crowdin-cli-py==0.94.2
docutils==0.11
flake8==2.3.0
fortran-magic==0.4.3
future==0.12.1
greenlet==0.4.2
h5py==2.3.0
httplib2==0.9
importmagic==0.1.2
ipython==2.3.1
itsdangerous==0.24
jdcal==1.0
jedi==0.8.1-final0
jkutils==1.1
lifelines==0.5.1.0
llvmpy==0.12.6
logilab-common==0.63.0
lxml==3.3.5
matplotlib==1.4.2
mccabe==0.2.1
mock==1.0.1
multipledispatch==0.4.3
networkx==1.8.1
nose==1.3.3
numba==0.13.2
numexpr==2.3.1
numpy==1.9.1
oauthlib==0.6.3
openpyxl==1.8.5
pandas==0.14.0
patsy==0.2.1
pep8==1.5.7
ply==3.4
psutil==2.1.1
py==1.4.26
pycosat==0.6.1
pycparser==2.10
pycrypto==2.6.1
pyflakes==0.8.1
pylint==1.3.1
pyparsing==2.0.1
pyplotthemes==0.1
## !! Could not determine repository location
pysurvival==1.2
pysurvival-ann==0.9
pytest==2.6.4
python-dateutil==2.1
pytz==2014.9
pyzmq==14.3.0
redis==2.9.1
requests==2.5.1
rope-py3k==0.9.4-1
rpy2==2.4.1
runipy==0.1.0
scikit-image==0.10.0
scipy==0.15.1
six==1.9.0
spyder==2.3.0rc
sympy==0.7.5
tables==3.1.1
tornado==3.2.1
ujson==1.33
xlrd==0.9.3

Removing importmagicseems to "fix" it. Moving around the buffer, I no longer get the second buffer pop-up with the error.

[spacecowboy]

However, in another environment, I do not seem to get the error with importmagic installed:

$ pip freeze
flake8==2.3.0
importmagic==0.1.2
mccabe==0.3
numpy==1.9.1
pandas==0.15.2
pep8==1.6.0
pyflakes==0.8.1
python-dateutil==2.1
pytz==2014.9
rope-py3k==0.9.4.post1
scipy==0.15.1
six==1.9.0

I am able to call C-c C-d to view the doc-string for an item.

[jorgenschaefer]

Aha! That makes more sense now.

This most likely means that importmagic is traversing packages in that large virtualenv, and something somewhere is throwing an error when it is imported/analyzed. That's why removing importmagic makes it work (it doesn't analyze the modules anymore), and why it doesn't fail in the other virtualenv (the problematic module isn't there). I'll try to figure out which module causes the problem there. Thank you for the patience finding the bug here :-)

[jorgenschaefer]

And before I forget: You should be able to use elpy quite fine in that virtualenv if you do not install importmagic. Most of Elpy's features should work.

[jorgenschaefer]

Hm. I can't install most of those modules (I do not use conda), and there are some entries in there that indicate that not everything in that virtualenv is installable at all. Makes it difficult to try and reproduce this. :-(

If you have time, you could try and remove modules one by one until the error vanishes … but that'd likely be rather cumbersome.

@birkenfeld, any idea how to more properly debug this?

[birkenfeld]

I just added some more logging calls to importmagic master.
Elpy/someone then needs to set up logging to emit DEBUG messages.

[jorgenschaefer]

Heh. I'm stupid.

@spacecowboy, could you try this?

(echo '{"id": 1, "method": "init", "params": [{"backend": "jedi", "project_root": "/home/forcer/Projects/test/elpy-482"}]}' ; cat) | PYTHONPATH="/home/forcer/Projects/elpy" python -m elpy

Replace the paths with the appropriate directories (the first one the path to your project, the second path to your elpy directory)

I hope that will reproduce the error.

[spacecowboy]

Result is:

elpy-rpc ready (1.7.0)
{"id": 1, "result": {"backend": "jedi"}}

But this means maybe I did something wrong. This is how I opened the file in the first place:

emacs misc.py

Which might not set project root and such correctly. Am I not supposed to just "open files" anymore with elpy?

[jorgenschaefer]

Opening files like that should work perfectly fine (well, if you open individual files like that every time, startup time will get annoying, but you probably know that).

What are the contents of the Python file you are using to reproduce this?
Does the problem happen in an empty Python file, too?

[spacecowboy]

Doing it now does not show the problem.. Which is odd (still using the big environment).

However, as a separate issue, I am running into messages about relative imports:

>>> Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jonas/workspacepython/ANN/ann/__init__.py", line 9, in <module>
    from ._ann import (matrixnetwork as _matrixnetwork,
SystemError: Parent module '' not loaded, cannot perform relative import

The file I am currently testing to view is the following: https://github.com/spacecowboy/pysurvival-ann/blob/master/ann/__init__.py

[spacecowboy]

Which does not appear when I try it a second time.. Now it all seems to work. Very confused about this.

[spacecowboy]

For the moment it seems to work (and I like it!). I'll get back to you in case I manage to reproduce it with any certainty

[jorgenschaefer]

Heh. I am sorry we could not identify the problem more quickly, and thank you for the patience in helping here. Glad it works now, please do not hesitate to open issues about any problem you encounter!

[cpitclaudel]

To find the culprit you can use this snippet:

import sys
import importmagic

index = importmagic.SymbolIndex()
index.build_index(sys.path)
with open('index.json') as fd:
    index.serialize(fd)

It this does raise an error when run, then you can edit the corresponding line; it's probably something like

  File "~/.local/lib/python3.4/site-packages/importmagic/index.py", line 129, in index_file
    success = subtree.index_source(filename, fd.read())

and add a print statement just before it to get the name of the file that's broken.

On my setup I get:

'utf-8' codec can't decode byte 0xdf in position 1208: invalid continuation byte
/usr/lib/python3/dist-packages/dateutil/parser.py
'utf-8' codec can't decode byte 0xe4 in position 1929: invalid continuation byte
/usr/lib/python3/dist-packages/PIL/WalImageFile.py
'utf-8' codec can't decode byte 0xe4 in position 79: invalid continuation byte
/usr/lib/python3/dist-packages/matplotlib/backends/backend_pdf.py

[cpitclaudel]

Here's a quick workaround: install importmagic locally (pip3 install --user importmagic) force it to ignore all non-utf-8 modules: on line 128, replace

            with open(filename) as fd:
                    success = subtree.index_source(filename, fd.read())

with

            with open(filename) as fd:
                try:
                    success = subtree.index_source(filename, fd.read())
                except UnicodeDecodeError:
                    success = False

Of course that's not a proper fix, but at least you get some of importmagic's nice features.

[birkenfeld]

The 'unhandled exception' message appearing is a bug in elpy but should already be fixed in master. It might need backporting to a fixes branch. The importmagic bug with non utf8 modules should be soon.

[birkenfeld]

importmagic 0.1.3 released with a fix.

[spacecowboy]

Installed importmagic from latest git. Seems to work. No error was thrown in emacs during my short test.

[jorgenschaefer]

Great to hear! Thank you for the patience in trying to find this problem, and I hope you enjoy Elpy :-)

Also, please do not hesitate to create more issues if you have any further problems.

[mikefaille]

First, I tried to install python-magic using :

sudo pip2 install importmagic
sudo pip3 install importmagic

But, I have this output :

There was an error when trying to start the backend. Elpy can not work
until this problem is solved. The following lines were received from
Python, and might help identifying the problem.

dm.c: 1693: not running as root returning empty list
{"result": [], "id": 3}

I updated python-magic using :

git clone https://github.com/ahupp/python-magic
cd python-magic
sudo python2 setup.py install
sudo python3 setup.py install

And it didn't solve my problem. :-/ Could you help me ? Thank you.

[jorgenschaefer]

dm.c: 1693: not running as root returning empty list

This means something in some module deep in Python is emitting something to stdout. I'm afraid Elpy currently has a rather hard time dealing with this – we're working on a solution, but it'll take a bit.

Your options right now would be to either not use importmagic (which imports all sorts of modules and can cause some misbehaving modules to emit such things), or to deinstall whichever module emits this. (Some googling says this might be related to anaconda, but I have no idea what it is, I'm afraid.)

[jorgenschaefer]

Also, the current master branch of Elpy should fix this problem. :-)

[mikefaille]

Sounds good ! Thank you !
I wait for Melpa :3

[jorgenschaefer]

MELPA carries the stable release of Elpy (this might change at some point when the new package-archive-priorities feature is released), so that will take a while. :-)

[mikefaille]

@jorgenschaefer this is a crittical bug fix for some of us. It can help to reach new user if you use Melpa for this fix. :+1:
Maybe, for stable release, you could use melpa-stable ?

[jorgenschaefer]

@jorgenschaefer this is a crittical bug fix for some of us. It can help to reach new user if you use Melpa for this fix. :+1:

A bugfix release would be useful, I guess. Can you confirm the code actually fixes the problem? (You can just download elpy.el from the repository and M-x load-file it to test.)

Maybe, for stable release, you could use melpa-stable ?

I would love to do that if there was a way to have both MELPA and MELPA stable as package archives. That is currently not possible, as that means package.el will always install from MELPA unstable. The new package-archive-priorities feature in Emacs 25 will solve this problem.

As is, having the development version of Elpy on MELPA only created a lot of work for me because users got the unstable versions all the time without wanting to.

[mikefaille]

Thank you for your answer. I'll test it tonight

[mikefaille]

I just test 24f3d5bc61a61dec51658d91724d9d6bd91a632c but I have same error.

Output from Backend

There was some unexpected output from the Elpy backend. This is
usually some module that does not use correct logging, but might
indicate a configuration problem.

Output
dm.c: 1693: not running as root returning empty list

[birkenfeld]

As is, having the development version of Elpy on MELPA only created a lot of work for me because users got the unstable versions all the time without wanting to.

But that is the definition of MELPA, isn't it?

[jorgenschaefer]

But that is the definition of MELPA, isn't it?

Yep. Sadly, judging from the bug reports I received after someone else put Elpy on MELPA, most of its users are not aware of that.

I just test 24f3d5b but I have same error.

This is a message, not an error – you should be able to keep using Elpy normally, get correct completions, etc., and if you put that window away, you should not be bothered by it again. Is this true?

(I can not simply hide the output, as it might contain information about a problem that Elpy is unable to work …)

[vbraun]

For what its worth:

$ egrep -r 'not running as root returning empty list' /usr/lib64/python2.7/site-packages/
Binary file /usr/lib64/python2.7/site-packages/block/dmmodule.so matches
$ rpm -qf /usr/lib64/python2.7/site-packages/block/dmmodule.so
python-pyblock-0.53-8.fc22.x86_64
$ dnf erase python-pyblock

Erasing python-pyblock gets rid of the spurious dm.c: 1693: not running as root returning empty list.