Elpy: Emacs RPC communication on windows freezes Emacs

[adamc55]

I'm having Emacs freeze totally periodically after adding elpy -- I'm using the package elpy-20140213.1028 (version 1.3.0, according to elpy.el).

It happens when I'm just typing -- looks like autocomplete gets triggered, but then there is a total freeze, so I can't tell you what _elpy-rpc_ says.

[jorgenschaefer]

Can you try M-x toggle-debug-on-quite and hit C-g (possibly a few times) when this happens?

When Emacs freezes, is it consuming CPU? Or the Python process elpy starts? You can check this by running top in a separate terminal and using P to sort by CPU usage.

Also, which version of Emacs are you using?

[adamc55]

Many thanks for the reply.

I did try C-g several times (I think I read your advice to someone else) --
no go, complete freeze. I'll give M-x toggle-debug-on-quit next time, but I
didn't quit emacs; I had to kill it. (This was on Windows 7, if that helps
at all.)

It wasn't consuming much if any CPU when I looked in Task Manager. (top is
great but not on Windows.) Dunno about the Python process -- will look for
that next time, thanks.

I'm using GNU Emacs 24.3.1 (i386-mingw-nt6.1.7601).

adam

On Thu, Mar 6, 2014 at 3:06 AM, Jorgen Schäfer [email protected]:

Can you try M-x toggle-debug-on-quite and hit C-g (possibly a few times)
when this happens?

When Emacs freezes, is it consuming CPU? Or the Python process elpy
starts? You can check this by running top in a separate terminal and
using P to sort by CPU usage.

Also, which version of Emacs are you using?

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-36836748
.

[jorgenschaefer]

If C-g (possibly multiple times) doesn't make Emacs responsive again, it's not the execution of lisp code that's responsible. It _might_ be blocking in a call to read from the Python process that Elpy uses to get information about the Python program being edited, but that _should_ also be interrupted by C-g. I'm not sure about Windows there, though, sadly.

Other than that, I'm a bit at a loss. On a GNU/Linux system, the next thing I'd try would be to check using strace what the process is actually doing. I do not know how to do that on Windows. :-(

[adamc55]

No problem; the Windows equivalent (more or less) is procmon.exe. It didn't
show any real activity in either emacs or the child python process (nor did
task manager), but if I kill that python process, I get control back (and
another is spawned in its place). So that would seem to be the place to
investigate further, if you have any suggestions.

FWIW, I'm using virtualenvwrapper.el and the Python being spawned is a
particular virtualenv, so conceivably that plays some role.

Again for what it is worth: The freeze occurs when I'm typing and
autocomplete is trying to offer suggestions, so it is somehow related to
that process.

But at least killing the child restores control.

Thanks again,
adam

On Thu, Mar 6, 2014 at 7:31 AM, Jorgen Schäfer [email protected]:

If C-g (possibly multiple times) doesn't make Emacs responsive again,
it's not the execution of lisp code that's responsible. It _might_ be
blocking in a call to read from the Python process that Elpy uses to get
information about the Python program being edited, but that _should_ also
be interrupted by C-g. I'm not sure about Windows there, though, sadly.

Other than that, I'm a bit at a loss. On a GNU/Linux system, the next
thing I'd try would be to check using strace what the process is actually
doing. I do not know how to do that on Windows. :-(

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-36887529
.

[jorgenschaefer]

Are you using rope as the backend to supply completions? If so, it could be that the project root points to a very large directory (Rope tries to search it all). Could you try and see if setting the project root using M-x elpy-set-project-root to some other directory with few files in it results in no hangs like this?

(Elpy should not wait so long, but I'm entirely unclear about the interprocess communication semantics on Windows :-/)

[adamc55]

Nope, I only have jedi installed in the VM. I also have '(setq
elpy-rpc-backend "jedi")' in my emacs initialization.

All told, this project includes "3,834 Files, 975 Folders" per Windows,
although no more than a few dozen are being actively edited. (Most of the
files are in shared resources brought in via SVN externals.) Would that
count as "very large"?

On Thu, Mar 6, 2014 at 9:09 AM, Jorgen Schäfer [email protected]:

Are you using rope as the backend to supply completions? If so, it could
be that the project root points to a very large directory (Rope tries to
search it all). Could you try and see if setting the project root using M-x
elpy-set-project-root to some other directory with few files in it
results in no hangs like this?

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-36896702
.

[jorgenschaefer]

Well, Rope would read all 3,834 files to figure out if any of them contain Python source to analyze, which I think would indeed take a while. But Jedi shouldn't be this … er … "simplistic", so that's likely not the case. You could try anyhow – just set the project root to some small/empty directory and try and see if it's faster that way, just to rule that out.

Do other Elpy features with the backend work? Do you get a response on M-x elpy-version, for example, that includes "using the Python backend version …"? (Run this in a Python buffer)

[adamc55]

M-x elpy-version works fine, I get "Elpy version 1.3.0 using the Python
backend version 1.3.0". I haven't had a chance to put in much time with a
smaller project since I don't have one that is a priority right now. I have
found that turning autocomplete off (i.e., (setq ac-auto-start nil)) has,
so far, prevented the problem from occurring -- so I'm fairly confident it
involves auto-complete.

Thanks for all your help.

adam connor

On Thu, Mar 6, 2014 at 1:22 PM, Jorgen Schäfer [email protected]:

Well, Rope would read all 3,834 files to figure out if any of them contain
Python source to analyze, which I think would indeed take a while. But Jedi
shouldn't be this ... er ... "simplistic", so that's likely not the case. You
could try anyhow - just set the project root to some small/empty directory
and try and see if it's faster that way, just to rule that out.

Do other Elpy features with the backend work? Do you get a response on M-x
elpy-version, for example, that includes "using the Python backend
version ..."? (Run this in a Python buffer)

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-36925826
.

[jorgenschaefer]

You do not need to use a different project, you can just use M-x elpy-set-project-root and change it to a different folder with fewer files. (Though as I said, if you're using jedi, that's likely not the problem anyhow.)

Sorry I can't help you more with locating the problem. If you find out what it is, please do let me know.

[nfusi]

I have the same issue (win 8, emacs 24.3.1). When the autocompletion is triggered, a python process is spawned and emacs hangs. If I kill the python process emacs becomes responsive again and the autocompletion seems to work on the current call (but hangs again if I trigger another autocompletion).

[jorgenschaefer]

What confuses me about this is that C-g does not make Emacs become responsive again. If this was due to some lisp code running or blocking, C-g should interrupt it and with M-x toggle-debug-on-quit, Emacs should also let us know where it was hanging. That C-g does not work at all means that this is hanging somewhere in Emacs' C code.

I could imagine something like the line ending convention or something, but Emacs shouldn't have a problem with that.

Could either (or both) of you try to run the following and see if the behavior continues?

M-: (set-process-coding-system (elpy-rpc--get-rpc-process) 'utf-8-unix 'utf-8-unix)

and

M-: (set-process-coding-system (elpy-rpc--get-rpc-process) 'utf-8-dos 'utf-8-dos)

[adamc55]

After doing that, not too long afterwards I got this message in the

_elpy-rpc_ buffer:

There was an error in the Elpy backend.
The following lines were received from Python, and might help identifying

the problem.

There was no additional text. However, Emacs is not frozen, so I'm unsure
if it is really the same issue.

On Tue, Mar 11, 2014 at 4:00 AM, Jorgen Schäfer [email protected]:

What confuses me about this is that C-g does not make Emacs become
responsive again. If this was due to some lisp code running or blocking,
C-g should interrupt it and with M-x toggle-debug-on-quit, Emacs should
also let us know where it was hanging. That C-g does not work at all
means that this is hanging somewhere in Emacs' C code.

I could imagine something like the line ending convention or something,
but Emacs shouldn't have a problem with that.

Could either (or both) of you try to run the following and see if the
behavior continues?

M-: (set-process-coding-system (elpy-rpc--get-rpc-process) 'utf-8-unix
'utf-8-unix)

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-37274734
.

[adamc55]

I take that back -- effectively, Emacs is frozen, because when I place the
cursor in another buffer, it takes me back to _Elpy Error_ with that
message.

On Tue, Mar 11, 2014 at 8:51 AM, Adam Connor [email protected] wrote:

After doing that, not too long afterwards I got this message in the

_elpy-rpc_ buffer:

There was an error in the Elpy backend.
The following lines were received from Python, and might help identifying

the problem.

There was no additional text. However, Emacs is not frozen, so I'm unsure
if it is really the same issue.

On Tue, Mar 11, 2014 at 4:00 AM, Jorgen Schäfer [email protected]:

What confuses me about this is that C-g does not make Emacs become
responsive again. If this was due to some lisp code running or blocking,
C-g should interrupt it and with M-x toggle-debug-on-quit, Emacs should
also let us know where it was hanging. That C-g does not work at all
means that this is hanging somewhere in Emacs' C code.

I could imagine something like the line ending convention or something,
but Emacs shouldn't have a problem with that.

Could either (or both) of you try to run the following and see if the
behavior continues?

M-: (set-process-coding-system (elpy-rpc--get-rpc-process) 'utf-8-unix
'utf-8-unix)

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-37274734
.

[jorgenschaefer]

Could you try with M-: (set-process-coding-system (elpy-rpc--get-rpc-process) 'utf-8-dos 'utf-8-dos) as well?

[adamc55]

Did, it still crashed, but I managed to grab something from the elpy buffer
(below). Looks to me like Python 2.6 (which is what I'm using)
documentation for the import statement, and it crashed while trying to
autocomplete an import.

{"result": [["port", "\nThe import
statement\n*******_\n\n import_stmt ::= \"import\"
module \"as\" name_\n |
\"from\" relative_module \"import\" identifier [\"as\"
name]\n ( \",\" identifier [\"as\" name]
)_\n | \"from\" relative_module \"import\" \"(\"
identifier [\"as\" name]\n ( \",\" identifier [\"as\"
name] )_ [\",\"] \")\"\n | \"from\" module \"import\"
\"_\"\n module ::= (identifier \".\")_ identifier\n
relative_module ::= \".\"* module | \".\"+\n name ::=
identifier\n\nImport statements are executed in two steps: (1) find a
module, and\ninitialize it if necessary; (2) define a name or names in the
local\nnamespace (of the scope where the import statement occurs).
The\nstatement comes in two forms differing on whether it uses the
from\nkeyword. The first form (without from) repeats these steps
for\neach identifier in the list. The form with from performs step
(1)\nonce, and then performs step (2) repeatedly.\n\nTo understand how step
(1) occurs, one must first understand how\nPython handles hierarchical
naming of modules. To help organize\nmodules and provide a hierarchy in
naming, Python has a concept of\npackages. A package can contain other
packages and modules while\nmodules cannot contain other modules or
packages. From a file system\nperspective, packages are directories and
modules are files. The\noriginal specification for packages is still
available to read,\nalthough minor details have changed since the writing
of that\ndocument.\n\nOnce the name of the module is known (unless
otherwise specified, the\nterm \"module\" will refer to both packages and
modules), searching for\nthe module or package can begin. The first place
checked is\nsys.modules, the cache of all modules that have been
imported\npreviously. If the module is found there then it is used in step
(2)\nof import.\n\nIf the module is not found in the cache, then
sys.meta_path is\nsearched (the specification for sys.meta_path can
be found in\nPEP 302). The object is a list of _finder_ objects which
are\nqueried in order as to whether they know how to load the module
by\ncalling their find_module() method with the name of the module.
If\nthe module happens to be contained within a package (as denoted by
the\nexistence of a dot in the name), then a second argument
to\nfind_module() is given as the value of the __path__
attribute\nfrom the parent package (everything up to the last dot in the
name of\nthe module being imported). If a finder can find the module it
returns\na _loader_ (discussed later) or returns None.\n\nIf none of
the finders on sys.meta_path are able to find the\nmodule then some
implicitly defined finders are queried.\nImplementations of Python vary in
what implicit meta path finders are\ndefined. The one they all do define,
though, is one that handles\nsys.path_hooks,
sys.path_importer_cache, and sys.path.\n\nThe implicit finder
searches for the requested module in the \"paths\"\nspecified in one of two
places (\"paths\" do not have to be file system\npaths). If the module
being imported is supposed to be contained\nwithin a package then the
second argument passed to find_module(),\n__path__ on the parent
package, is used as the source of paths. If\nthe module is not contained in
a package then sys.path is used as\nthe source of paths.\n\nOnce the
source of paths is chosen it is iterated over to find a\nfinder that can
handle that path. The dict at\nsys.path_importer_cache caches finders
for paths and is checked\nfor a finder. If the path does not have a finder
cached then\nsys.path_hooks is searched by calling each object in the
list with\na single argument of the path, returning a finder or
raises\nImportError. If a finder is returned then it is cached
in\nsys.path_importer_cache and then used for that path entry. If
no\nfinder can be found but the path exists then a value of None
is\nstored in sys.path_importer_cache to signify that an
implicit,\nfile-based finder that handles modules stored as individual
files\nshould be used for that path. If the path does not exist then a
finder\nwhich always returns None is placed in the cache for the
path.\n\nIf no finder can find the module then ImportError is
raised.\nOtherwise some finder returned a loader whose load_module()
method\nis called with the name of the module to load (see PEP 302 for
the\noriginal definition of loaders). A loader has several
responsibilities\nto perform on a module it loads. First, if the module
already exists\nin sys.modules (a possibility if the loader is called
outside of\nthe import machinery) then it is to use that module for
initialization\nand not a new module. But if the module does not exist
in\nsys.modules then it is to be added to that dict
before\ninitialization begins. If an error occurs during loading of the
module\nand it was added to sys.modules it is to be removed from the
dict.\nIf an error occurs but the module was already in sys.modules it
is\nleft in the dict.\n\nThe loader must set several attributes on the
module. __name__ is\nto be set to the name of the module. __file__
is to be the \"path\"\nto the file unless the module is built-in (and thus
listed in\nsys.builtin_module_names) in which case the attribute is not
set.\nIf what is being imported is a package then __path__ is to be
set\nto a list of paths to be searched when looking for modules
and\npackages contained within the package being imported.
__package__\nis optional but should be set to the name of package that
contains the\nmodule or package (the empty string is used for module not
contained\nin a package). __loader__ is also optional but should be set
to\nthe loader object that is loading the module.\n\nIf an error occurs
during loading then the loader raises\nImportError if some other
exception is not already being\npropagated. Otherwise the loader returns
the module that was loaded\nand initialized.\n\nWhen step (1) finishes
without raising an exception, step (2) can\nbegin.\n\nThe first form of
import statement binds the module name in the\nlocal namespace to the
module object, and then goes on to import the\nnext identifier, if any. If
the module name is followed by as,\nthe name following as is used
as the local name for the module.\n\nThe from form does not bind the
module name: it goes through the\nlist of identifiers, looks each one of
them up in the module found in\nstep (1), and binds the name in the local
namespace to the object thus\nfound. As with the first form of import,
an alternate local name\ncan be supplied by specifying \"as
localname\". If a name is not\nfound, ImportError is raised. If the
list of identifiers is\nreplaced by a star ('*'), all public names
defined in the module\nare bound in the local namespace of the import
statement..\n\nThe _public names_ defined by a module are determined by
checking the\nmodule's namespace for a variable named __all__; if
defined, it\nmust be a sequence of strings which are names defined or
imported by\nthat module. The names given in __all__ are all
considered public\nand are required to exist. If __all__ is not
defined, the set of\npublic names includes all names found in the module's
namespace which\ndo not begin with an underscore character ('_').
__all__\nshould contain the entire public API. It is intended to
avoid\naccidentally exporting items that are not part of the API (such
as\nlibrary modules which were imported and used within the module).\n\nThe
from form with * may only occur in a module scope. If the\nwild
card form of import --- import * --- is used in a function\nand the
function contains or is a nested block with free variables,\nthe compiler
will raise a SyntaxError.\n\nWhen specifying what module to import you
do not have to specify the\nabsolute name of the module. When a module or
package is conta

On Tue, Mar 11, 2014 at 8:57 AM, Jorgen Schäfer [email protected]:

Could you try with M-: (set-process-coding-system
(elpy-rpc--get-rpc-process) 'utf-8-dos 'utf-8-dos) as well?

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-37297597
.

[jorgenschaefer]

That output is truncated. Is it truncated in the original buffer, too?

[adamc55]

Yes, exactly. I pasted in everything in the buffer.
On Mar 11, 2014 12:09 PM, "Jorgen Schäfer" [email protected] wrote:

That output is truncated. Is it truncated in the original buffer, too?

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-37321724
.

[jorgenschaefer]

8190 bytes. If that isn't some kind of buffering problem.

We had some problems in elpy on Unix because Emacs uses a pty, which can have too much data transmitted ... which is why Elpy now sets process-connection-type to nil before starting the RPC process. Maybe there's something similar for windows? I wouldn't know, I'm afraid.

[adamc55]

How sure are you that this is in Emacs vs. the Python libraries? What does
the Python library use? (I ask because I see reported issues for modwsgi,
for example: https://code.google.com/p/modwsgi/issues/detail?id=130 )

On Wed, Mar 12, 2014 at 2:57 AM, Jorgen Schäfer [email protected]:

8190 bytes. If that isn't some kind of buffering problem.

We had some problems in elpy on Unix because Emacs uses a pty, which can
have too much data transmitted ... which is why Elpy now sets
process-connection-type to nil before starting the RPC process. Maybe
there's something similar for windows? I wouldn't know, I'm afraid.

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-37383149
.

[jorgenschaefer]

Emacs should not freeze if it's waiting for output from the Python process. At least a C-g should interrupt the reading and return Emacs to a responsive state. That it doesn't means there most likely is a bug in Emacs. It is quite possible the bug is _triggered_ by some specific output from the subprocess, but the subprocess shouldn't be able to freeze Emacs like this.

[adamc55]

I suppose I should really just put aside time to read the source, but: any
chance that it is using the Windows command line? That might explain it.

On Wed, Mar 12, 2014 at 8:36 AM, Jorgen Schäfer [email protected]:

Emacs should not freeze if it's waiting for output from the Python
process. At least a C-g should interrupt the reading and return Emacs to
a responsive state. That it doesn't means there most likely is a bug in
Emacs. It is quite possible the bug is _triggered_ by some specific
output from the subprocess, but the subprocess shouldn't be able to freeze
Emacs like this.

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-37408400
.

[jorgenschaefer]

I do not know. I do not have access to a Windows system.

[adamc55]

Understood. I came across this while looking for another bug, and it
sounded so similar I thought I should mention it:
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=16875

As the report points out, IPython is not subject to this because it inserts
newlines. So I went ahead and enabled IPython for Elpy. So far, that is
working -- no freezes.

I've only tested for a few hours, so I'm not absolutely certain this is a
solution, but I haven't frozen yet. (Previously, I would freeze much sooner
than this, so I am optimistic.)

adam connor

On Wed, Mar 12, 2014 at 12:01 PM, Jorgen Schäfer
[email protected]:

I do not know. I do not have access to a Windows system.

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-37434352
.

[jorgenschaefer]

The RPC process does not use the interactive shell. I don't think ipython has any effect there. But if it doesn't freeze anymore, good.

I'm really sorry I can't be of more help here.

[jorgenschaefer]

A user suggested using pythonw instead of python as the RPC process on windows. Apparently, the comment was deleted(?), so I'm not sure if this was a mistake, but could you try if that solves the problem?

[adamc55]

I think I got it ("tiktok7") -- have been trying the last few days, and so
far, it has been successful -- no freezes if I add
(setq elpy-rpc-python-command "pythonw"). I didn't want to report it until
I'd tested for a day or two.

On Fri, Mar 21, 2014 at 4:34 AM, Jorgen Schäfer [email protected]:

A user suggested using pythonw instead of python as the RPC process on
windows. Apparently, the comment was deleted(?), so I'm not sure if this
was a mistake, but could you try if that solves the problem?

Reply to this email directly or view it on GitHubhttps://github.com/jorgenschaefer/elpy/issues/199#issuecomment-38260258
.

[jorgenschaefer]

Starting with the next release, Elpy will use pythonw as default RPC process under Windows. Thank you for the patience to debug this and find a solution. :-)

If this does _not_ resolve the issue, please do reopen the issue.

[m-soltani]

Hi. I am facing the same problem with emacs 25.1 and jedi 0.10.0
Emacs halts and i have to close the terminal and reopen the file again. It happens i think when jedi tries to provide autocomplete for the code

[aagaard]

I face the same issue. My configuration:

OS: Windows 10
RPC Python........: 3.6.1 (c:/Users/me/.envs/yiu/pythonw.exe)
Emacs.............: 25.1.1
Elpy..............: 1.15.1
Jedi..............: 0.10.3

Pythonw uses 100% CPU and emacs hangs. It cannot be "unlooped" with C-g, so it is something between the OS and emacs that does not response. If I kill the pythonw.exe process, the emacs comes back and modify the buffer.