Elpy: Elpy seems partially incompatible with Emacs 25's 'native completion' feature

[cpitclaudel]

In Emacs 25, when starting Python with C-c C-c, I get the following message:

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "ipython3" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list. Native completions have been disabled locally.

On the other hand, if I disable elpy, start the repl, and re-enable it, everything works fine. Is elpy introducing a process filter for the python process, or something similar? That warning comes from this bit of python.el:

(defun python-shell-completion-native-setup ()
  "Try to setup native completion, return non-nil on success."

which deep down relies on explicitly doing an accept-process-output, which sounds rather bad.

In any case, I'm enclined to think that python.el is doing something wrong and elpy doing something right, but I'd love to get a bit of help pinpointing the bad interaction before opening an Emacs ticket, if we determine this to be the right course of action; especially since Emacs 25 is in a pretty deep freeze, so there's little chance that this will get fixed on the Emacs side before the release.

Let me know if I can help :)

[jorgenschaefer]

Hello, and thanks for the report. What's the value of python-shell-interpreter with and without Elpy enabled? Are you using a virtualenv in either case?

[cpitclaudel]

No virtualenv in any case. The bug happens regardless of the value I give to python-shell-interpreter in my .emacs (python3 or ipython3). Enabling or disabling elpy doesn't seem to change that value.

[jorgenschaefer]

Could you copy the definition of the function python-shell-completion-native-setup into your *scratch* buffer and edit it so that the part before pytho.el: native completion setup failed reads like this:

        print ('python.el: native completion setup loaded')                     
    except:                                                                     
        raise
        print ('python.el: native completion setup failed')                     

i.e. add a raise to the except branch so we can see the error, hopefully. Evaluate the resulting function and try C-c C-c in Elpy again.

[cpitclaudel]

Sorry, I don't think I explained this clearly. There is no error; the python repl buffer contains this after the warning is shown:

Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
Type "copyright", "credits" or "license" for more information.

IPython 1.2.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 
In [1]: python.el: native completion setup loaded

The reason the warning is shown seems to be that the accept-process-output call that python.el does times out without returning data; strangely enough, though, it doesn't happen reliably...

[jorgenschaefer]

There's nothing Elpy does here that should interfere with that code. I have no idea what's going on there. :-(

[cpitclaudel]

I'll close this then, and I'll try to investigate it myself next time it pops up :)

[rthoma]

I receive the same warning as cpitclaudel running Emacs 25.1.50 (9.0) built with homebrew on my mac.

[jguenther-va]

I also have been getting this same warning, I ended up just setting python-shell-completion-native to nil, as I just couldn't get it to work.

I do have working tab-completion in python shell buffers with that setting with ipython as my interpreter though.

[cpitclaudel]

Looks like this is happening again on master :/

[cpitclaudel]

Interestingly, even setting elpy-modules to nil doesn't suppress the problem. What does elpy do that's not in a module?

[jorgenschaefer]

Interestingly, even setting elpy-modules to nil doesn't suppress the problem. What does elpy do that's not in a module?

Not much. This is a problem with python-mode in Emacs, not Elpy specifically.

I suspect this is yet another problem with the newer ipython stuff. As a wild shot, could you add (setenv "IPY_TEST_SIMPLE_PROMPT" "1") and/or (setenv "JUPYTER_CONSOLE_TEST" "1") to your config file and see if that fixes this problem?

[cpitclaudel]

Hmm. I'm not using IPython (I use the regular CPython); besides, everything works fine in both emacs -Q and in my regular Emacs, as long as I don't load Elpy.

[jorgenschaefer]

But the output you showed above was from ipython?

[cpitclaudel]

Indeed; that's what I guess I was testing out at the time. This happens with 2.7 and 3.5 as well.

[jorgenschaefer]

Could you copy the definition of the function python-shell-completion-native-setup into your *scratch* buffer and edit it so that the part before pytho.el: native completion setup failed reads like this:

        print ('python.el: native completion setup loaded')                     
    except:                                                                     
        raise
        print ('python.el: native completion setup failed')                     

i.e. add a raise to the except branch so we can see the error, hopefully. Evaluate the resulting function and try C-c C-c in Elpy again.

[cpitclaudel]

@jorgenschaefer I'm confused. Didn't you ask me to do this before? :)

I made the requested change, but there is no error to see:

Python 3.5.2 (default, Jul  5 2016, 12:43:10) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> python.el: native completion setup loaded
>>> 

The except path is never taken.

[jorgenschaefer]

Yes I did, but that was with ipython which might have had different code paths :-) When trying to debug some weird behavior, it's confusing when the environment changes randomly.

What does M-: (python-shell-completion-native-try) return for you?

I'm still confused why anything that Elpy does would affect this. If you add (setq elpy-modules nil) before elpy-enable, does this still happen?

[cpitclaudel]

I'll try this. I'll also try to debug this on my own a bit, because this is getting complex and annoying, and I don't think I'm being very helpful. Let me create a minimal test case, and see if that helps me; otherwise, I'll at least have stuff to report here :)

[ghost]

I too am facing same problem as @cpitclaudel.
When I do M-: (python-shell-completion-native-try), I get error

Debugger entered--Lisp error: (wrong-type-argument processp nil)
  process-buffer(nil)
  python-shell-completion-native-get-completions(nil nil "")
  python-shell-completion-native-try()
  eval((python-shell-completion-native-try) nil)
  eval-expression((python-shell-completion-native-try) nil)
  funcall-interactively(eval-expression (python-shell-completion-native-try) nil)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

Even with (setq elpy-modules nil) I still get the "Native completions have been disabled locally" warning and completions do not work.
Seems to be due to: bug#22897

[ghost]

I have a question. How can I elpy-shell-send-region-or-buffer to an external Python interpreter process (not running in Emacs), but attached to a terminal emulator?

[cpitclaudel]

Closing this: after a while I ended up opening an Emacs bug report, and was pointed to another, existing one. See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24401

Sorry for the confusion, and thanks for the help and the hard work on Elpy!

[jorgenschaefer]

Thank you for finding that upstream bug! That solves one of my headaches here :-D

[chase-dwelle]

I'm having this issue as well. What's the best way to suppress this error until the fix is pushed out through emacs?

[ChillarAnand]

You can try this

(setq python-shell-prompt-detect-failure-warning nil)

[cpitclaudel]

I pushed a fix to emacs-25, so the problem will go away in the next Emacs release :) In the meantime, you can redefine the affected function in your .emacs, as shown in https://github.com/emacs-mirror/emacs/commit/dbb341022870ecad4c9177485a6770a355633cc0

[jorgenschaefer]

Is it possible to reliably detect the problem? I would not mind having the fix in elpy.el's "backwards compatibility" section …

[cpitclaudel]

I'm not sure about reliably detecting it; but since the issue affects only one version I think it would work to just redefine the affected function in that version.

[chaturv3di]

I just stumbled upon this solution to the problem that I'm facing with Emacs25.1.1 (x86_64-w64-mingw32) on Windows7. I added the function definition in my .emacs.d/init.el file, but I continue to get the warning and the accompanying error in the Python interpreter.

It seems to me that python.el is being loaded after my emacs profile, hence the problem persists. I'm attaching a screenshot for reference. Am I doing something wrong?

[cpitclaudel]

You could wrap this redefinition in (with-eval-after-load 'python …)

[chaturv3di]

Thanks a lot. This worked. I'm attaching the screenshot of the code for whoever might be interested in it.

[jmromer]

(defun python-shell-completion-native-try ()
  "Return non-nil if can trigger native completion."
  (with-eval-after-load 'python
    '(let ((python-shell-completion-native-enable t)
           (python-shell-completion-native-output-timeout
            python-shell-completion-native-try-output-timeout))
       (python-shell-completion-native-get-completions
        (get-buffer-process (current-buffer))
        nil "_"))))

[cpitclaudel]

@jkrmr This use of with-eval-after-load doesn't make much sense to me. What did you have in mind?

[jmromer]

I'll refer that to @chaturv3di -- I just transcribed the screenshot for the benefit of future readers.

This workaround didn't actually work for me but I probably have something else going on in my case.

[cpitclaudel]

I should have looked at that screenshot closer; it looks wrong too. With-eval-after-load should be around the function, not inside it. And there's no need for the quote.

[chaturv3di]

I tried:

1. Removing the quote before (let ((...
2. Wrapping the entire defun block inside (with-eval-after-load 'python ...)

I got the same error as shown in the right half of my first screenshot. That error, and the accompanying warning about native completion goes away if my init.el file contains those lines as I have shown in the second screenshot. It may be wrong, as @cpitclaudel suggests, but it works for me.

[cpitclaudel]

@chaturv3di What your code does is essentially disable that function entirely (it says: "override this function, and every time its run, register a hook that does nothing"). AFAICT, it would be the same to use (defun python-shell-completion-native-try ()). Isn't it?

[chaturv3di]

@cpitclaudel, interesting. Providing the empty definition as you suggest here gets rid of the SyntaxError that you see in the first screenshot, but I still get this warning:

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list. Native completions have been disabled locally.

Providing that incorrect function definition somehow gets rid of/suppresses this warning message too.

[cpitclaudel]

Right. That's because the trivial, empty definition returns nil. With (defun python-shell-completion-native-try t) you'd probably get the same as the complicated definition with with-eval and the quote.

[npostavs]

For the record, the correct fix is

(with-eval-after-load 'python
  (defun python-shell-completion-native-try ()
    "Return non-nil if can trigger native completion."
    (let ((python-shell-completion-native-enable t)
          (python-shell-completion-native-output-timeout
           python-shell-completion-native-try-output-timeout))
      (python-shell-completion-native-get-completions
       (get-buffer-process (current-buffer))
       nil "_"))))

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#44

[jojojames]

That doesn't seem to work for me. Calling run-python still gets the same/similar? error.

GNU Emacs 25.1.1 (x86_64-apple-darwin13.4.0, Carbon Version 157 AppKit 1265.21)

james ~ python --version
Python 2.7.10
james ~

Python 2.7.10 (default, Jul 30 2016, 18:31:42) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import codecs, os;__pyfile = codecs.open('''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''');exec(compile(__code, '''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''', 'exec'));
python.el: native completion setup loaded
>>> 

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally. 

(use-package python
  :ensure t
  :mode ("\\.py\\'" . python-mode)
  :interpreter ("python" . python-mode)
  :init
  (eval-after-load 'python
    (lambda ()
      (defun python-shell-completion-native-try ()
        "Return non-nil if can trigger native completion."
        (let ((python-shell-completion-native-enable t)
              (python-shell-completion-native-output-timeout
               python-shell-completion-native-try-output-timeout))
          (python-shell-completion-native-get-completions
           (get-buffer-process (current-buffer))
           nil "_"))))))

I even tried just starting emacs without any python configuration and then evaling this before (also tried after to exhaust all possibilities) loading a python file.

(eval-after-load 'python
    (lambda ()
      (defun python-shell-completion-native-try ()
        "Return non-nil if can trigger native completion."
        (let ((python-shell-completion-native-enable t)
              (python-shell-completion-native-output-timeout
               python-shell-completion-native-try-output-timeout))
          (python-shell-completion-native-get-completions
           (get-buffer-process (current-buffer))
           nil "_")))))

[npostavs]

GNU Emacs 25.1.1 (x86_64-apple-darwin13.4.0, Carbon Version 157 AppKit 1265.21)

Python 2.7.10 (default, Jul 30 2016, 18:31:42) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import codecs, os;__pyfile = codecs.open('''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''');exec(compile(__code, '''/var/folders/k2/y7hdq90906d_5pgbt3_pt9jw0000gn/T/py68666z81''', 'exec'));

Ah, it seems there is some separate bug on macOS.

• https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#50
• http://stackoverflow.com/questions/33726268/emacs-inferior-python-shell-shows-the-send-message-with-each-python-shell-send-r
• http://emacs.stackexchange.com/questions/29552/python-send-string-echo-begins-with-printvalue-sep-end-n-file-sy

[jojojames]

@npostavs
Installing homebrew python fixed it for me.

[jorgenschaefer]

Thank you all for the work here! Is this fixed version in Emacs git already?

[npostavs]

Installing homebrew python fixed it for me.

Hmm, what's the difference with that python? Does it use readline instead of libedit? (use import readline; print readline.__doc__ to see).

Thank you all for the work here! Is this fixed version in Emacs git already?

Yes, also in 25.2-rc1 and up.

[jorgenschaefer]

Yes, also in 25.2-rc1 and up.

Excellent. Elpy should now provide this fixed version if run in an Emacs version before that.

Thank you, everyone!

[jojojames]

/usr/bin/python
>>> import readline; print readline.__doc__
Importing this module enables command line editing using libedit readline.
>>>

homebrew python
>>> import readline; print readline.__doc__
Importing this module enables command line editing using GNU readline.
>>>

[mambolevis]

I built emacs-25.2-rc2 with release flags. When using elpy-shell-send-region-or-buffer the following Message appears:
Shell native completion is disabled, using fallback

I also tested emacs-25.2-rc2 from Index of /gnu/emacs/pretest/windowsand the same message is shown.

[jorgenschaefer]

@npostavs – is this intentional? (I don't know if it's a problem at all, even …)

[mambolevis]

Hi,
I don't know where should I describe/report this issue, do you know where is the right place?
regards,
Levis

[npostavs]

I built emacs-25.2-rc2 with release flags. When using elpy-shell-send-region-or-buffer the following Message appears:
Shell native completion is disabled, using fallback

If you don't get the Your `python-shell-interpreter' doesn't seem to support readline, yet `python-shell-completion-native-enable' was t and "python" is not part of the `python-shell-completion-native-disabled-interpreters' list. Native completions have been disabled locally. I guess this is intentional.

[jorgenschaefer]

I have no idea what they are even doing there, or why, but this should disable the warning:

(setq python-shell-completion-native-enable nil)

[mambolevis]

Yes, I am using (setq python-shell-completion-native-enable nil) to disable the warning but I don't know if it is the desired setting.

This is my summary regarding this issue:
Emacs version emacs-25.2-rc2
Windows 10

1. Calling elpy-shell-send-region-or-buffer the following warning is shown:
   Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list. Native completions have been disabled locally.

>>> python.el: native completion setup failed

1. Calling elpy-shell-send-region-or-buffer and using (setq python-shell-completion-native-enable nil) the warning is not shown

If it is the desired behavior please let me know.

[jorgenschaefer]

@mambolevis I do not know if this is the desired behavior. This is a feature in Emacs, unrelated to Elpy. If you require this feature, this needs fixing in Emacs itself. If you do not require this completion feature, disabling it explicitly via that variable seems to be right approach.

[npostavs]

This is my summary regarding this issue:
Emacs version emacs-25.2-rc2
Windows 10

What's your python version?

[mambolevis]

Python version 3.5.2

This is my elpy-config in case of requiring:

Virtualenv........: None
RPC Python........: 3.5.2 (c:/Data/Anaconda3/envs/py35/pythonw.exe)
Interactive Python: ipython (c:/Data/Anaconda3/envs/py35/Scripts/ipython.exe)
Emacs.............: 25.1.1
Elpy..............: 1.14.1
Jedi..............: 0.10.0
Rope..............: 0.9.4-1
Importmagic.......: 0.1.7
Autopep8..........: 1.3.1
Yapf..............: 0.16.0
Syntax checker....: flake8.exe (c:/Data/Anaconda3/envs/py35/Scripts/flake8.exe)

[npostavs]

Interactive Python: ipython (c:/Data/Anaconda3/envs/py35/Scripts/ipython.exe)

Does this mean elpy is using ipython? For Emacs 25.2 at least, it's expected that native completion will be disabled when using ipython.

[mambolevis]

The warning is shown with both pythonand ipython

[npostavs]

What do you get from running import readline; print (readline.__doc__) in python?

[mambolevis]

>>> import readline; print(readline.__doc__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'readline'

[npostavs]

ImportError: No module named 'readline'

Oh, so Windows python doesn't come with any readline at all. Then yes, disabling native completion explicitly to quiet the warnings is the right thing.

[mambolevis]

Ok, thank you.
Just one question more for learning
I found PyReadline (unstable, deprecated), a ctypes-based readline for Windows. at
http://www.lfd.uci.edu/~gohlke/pythonlibs/

Is it the package you pointed at?

[npostavs]

Hmm, not sure what that's pointing at. I think readline should normally come with python: https://docs.python.org/3/library/readline.html

[mambolevis]

Ok, thank you.

[amitavmohanty01]

import readline; print (readline.__doc__)

Invoking this via C-c C-c gives me the following output:

Python 2.7.10 (default, Feb  7 2017, 00:08:15) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> Importing this module enables command line editing using libedit readline.
>>> import codecs, os;__pyfile = codecs.open('''/var/folders/tp/1yf5z4g92ds06_cd2x_qx3ss_r7srq/T/py3101gBl''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/tp/1yf5z4g92ds06_cd2x_qx3ss_r7srq/T/py3101gBl''');exec(compile(__code, '''/var/folders/tp/1yf5z4g92ds06_cd2x_qx3ss_r7srq/T/py3101gBl''', 'exec'));
python.el: native completion setup loaded
>>> import readline; print (readline.__doc__)
  File "<stdin>", line 1
    import readline; print (readline.__doc__)
      ^
SyntaxError: invalid syntax
>>> import readline; print (readline.__doc__)
Importing this module enables command line editing using libedit readline.
>>> import readline; print (readline.__doc__)
Importing this module enables command line editing using libedit readline.
>>> 

So, the first two invocations cause warning and error respectively but the third invocation onwards it works fine.

[yichao-cai]

Hey folks,

May I know if this issue has been solved in emacs later version like emacs 25.3?

I just compiled my emacs 25.3.1 and I have the same problem:
Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native-enable’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list. Native completions have been disabled locally.

I downloaded and compiled my emacs 25.3.1 from here. (Following this tutorial)

GNU Emacs 25.3.1
Copyright (C) 2017 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.

This is my elpy configures (I am using anaconda to set up my python):

Virtualenv........: None
RPC Python........: 3.6.2 (/home/yichao/anaconda3/bin/python)
Interactive Python: python (/home/yichao/anaconda3/bin/python)
Emacs.............: 25.3.1
Elpy..............: 1.17.0
Jedi..............: 0.10.2 (0.11.1 available)
Rope..............: 0.10.5
Importmagic.......: Not found (0.1.7 available)
Autopep8..........: 1.3.3
Yapf..............: 0.18.0 (0.20.1 available)
Syntax checker....: flake8 (/home/yichao/anaconda3/bin/flake8)

[npostavs]

May I know if this issue has been solved in emacs later version like emacs 25.3?

It should be, yes. What do you get from import readline; print (readline.__doc__) at a python shell?

[yichao-cai]

Hi @npostavs,

Sorry for late reply.
This is what I got from a python shell:

Python 3.6.2 |Anaconda custom (64-bit)| (default, Sep 30 2017, 18:42:57) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>> print(readline.__doc__)
Importing this module enables command line editing using GNU readline.
>>> 

EDIT:
A surprising thing is that my issue has gone, after two days and I don't know why. (No restart of my computer. I've been using emacs24 instead of emacs25 these days. When I tried emacs25 today it just gone.)

Right now I can execute python code in emacs25 by C-c C-c without the warnings.

Also, I notice that at the beginning of my python shell in emacs25, there is a line saying that native completion setup is used:

Python 3.6.2 |Anaconda custom (64-bit)| (default, Sep 30 2017, 18:42:57) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 3.6.2 |Anaconda custom (64-bit)| (default, Sep 30 2017, 18:42:57) 
[GCC 7.2.0]
Hello world
>>> python.el: native completion setup loaded
>>> 

[npostavs]

Hmm, seems like it should work then. Do you get any messages about in the *Python* buffer about setup failing or succeeding to load?

What do you get from trying the tests in https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#29
and https://debbugs.gnu.org/cgi/bugreport.cgi?bug=25753#35

[yichao-cai]

Hi @npostavs,

There is only a line in the *Python* which says about the setup (which I posted in my last comments):

>>> python.el: native completion setup loaded

BTW, where can I find the python code for testing that you mentioned?

Thank you so much!

[npostavs]

EDIT:
A surprising thing is that my issue has gone, after two days and I don't know why. (No restart of my computer. I've been using emacs24 instead of emacs25 these days. When I tried emacs25 today it just gone.)

Oh, huh. I guess it's all good then.

BTW, where can I find the python code for testing that you mentioned?

It's in python.el, <f1> f python-shell-completion-native-setup RET and then hit the python.el link in the *Help* buffer will bring you to the right place.

[fulminemizzega]

I'm sorry to keep this going, I'm running Fedora 27, emacs 25.3.1, python 3.6.4, elpy 20180318.856 but I still see this issue. I have set python-shell-interpreter to "python3".
I tried this: run emacs -Q, then M-x set-variable python-shell-interpreter "python3", then open a python script, C-p C-c and C-c C-c does not give any warning.
If I run just emacs, then C-c C-c gives me the warning message.

[npostavs]

If I run just emacs, then C-c C-c gives me the warning message.

What's the value of python-shell-interpreter in this case? Could you bisect your .emacs to figure out which part of your configuration causes this?

[cpitclaudel]

I have the same problem, but only with C-c C-c (that is, C-c C-p followed by C-c C-c works, but C-c C-c alone gives the error message). IOW, the problem happens when Python is started by the same function that sends the contents of the buffer (elpy-shell-send-region-or-buffer).

[npostavs]

If I try to run C-c C-c first from emacs -Q, then I get

user-error: Start a Python process first with ‘M-x run-python’ or ‘C-c C-p’.

Does elpy suppress/circumvent this?

[cpitclaudel]

Does elpy suppress/circumvent this?

It does, yes. I think that's the source of the issue.
All if this is confusing, of course, because originally this was a legit Emacs issue, and now it seems to be mostly elpy doing something wrong.

(Btw, (progn (run-python (python-shell-calculate-command) nil t) (python-shell-send-buffer nil t)) seems to work just fine. Question: why doesn't C-c C-c in plain Emacs just call run-python if needed?)

[npostavs]

Hmm, the waiting loop in elpy-shell--ensure-shell-running looks pretty dubious, not sure if Emacs should adopt something like that.

https://github.com/jorgenschaefer/elpy/blob/1.18.0/elpy-shell.el#L257

[cpitclaudel]

not sure if Emacs should adopt something like that.

I don't think so, no. And it does look like the issue is related to that loop, at least on my machine.

[craigmac]

Still getting the 'readline' warning on Windows 7, using Emacs 25.3_1 release, which from what I read people were saying would fix this issue. I also tried starting with no init file (emacs -Q) to make sure it wasn't my .emacs.

[npostavs]

@saltycraig there is no fix on Windows, just (setq python-shell-completion-native-disabled-interpreters '("python")) to suppress the warning (for Emacs 26, the warning is already suppressed on Windows).

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=28580
https://emacs.stackexchange.com/questions/35306/python-shell-interpreter-doesn-t-seem-to-support-readline

[craigmac]

Thank you @npostavs , this is what I ended up doing and just guarded it with a 'windows-nt' system-type detect, hopefully it really is suppressed on Emacs 26, as I did read several answers on threads that this was supposed to have been fixed at or around 25.1.

[bgoodr]

I'm encountered this problem when running a conda-based python 2.7.14
when using the C-c C-c binding that is normally bound to
elpy-shell-send-region-or-buffer. I did some extensive debugging and
came up with a hackaround as below:

(defun bg-python-mode-hook ()
  ...
  (define-key elpy-mode-map (kbd "C-c C-c") 'bg-elpy-shell-send-region-or-buffer)
  ...)

(add-hook 'python-mode-hook 'bg-python-mode-hook)

(defun bg-elpy-shell-send-region-or-buffer (&optional arg)
  (interactive "P")
  ;; Force the process to start completely by sitting a bit to avoid this warning:
  ;;
  ;;   Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native-enable’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally.
  ;;
  ;; Refer to https://github.com/jorgenschaefer/elpy/issues/887
  ;;
  (elpy-shell-get-or-create-process 0.001)
  (elpy-shell-send-region-or-buffer arg))

What I have not been able to determine is why (run-python "python -i")
works but the elpy-shell-send-region-or-buffer function does not. I
speculate that something that elpy is doing behind the scenes is
slowing down python startup to the point that the underlying
python-shell-completion-native-setup function can't see it.

         (save-excursion
           (re-search-backward
            (regexp-quote "python.el: native completion setup loaded") nil t 1))

returns nil, but if you look in the Python process buffer, that
"setup loaded" line shows up. So something is not waiting long enough.

The call to (elpy-shell-get-or-create-process 0.001) above waits just
long enough for the python to invoke. The value of 0.001 may not be
large enough depending upon the latency of the filesystem serving
python (e.g., slow or overloaded NFS), but at least this hack allows
me to increase it slightly.

Side note: Not even sure this is relevant: I discovered I had package
called "color-moccur.el" imported and it defined a global variable
called simply "regexp" that also fouled up the logic inside
python-shell-accept-process-output where by this expression:

(or regexp comint-prompt-regexp)

evaluated to nil even though comint-prompt-regexp is non-nil. This
occurs because the global variable regexp is used even though the
default value is not specified in the particular call to
python-shell-accept-process-output. I've since disabled importing
color-moccur.el since no package should be defining variables simply
named as "regexp".

M-x emacs-version returns:

GNU Emacs 25.2.2 (x86_64-pc-linux-gnu, GTK+ Version 3.22.21) of 2017-09-22, modified by Debian

lsb_release -r -i returns:

Distributor ID: Ubuntu
Release:    17.10

M-x elpy-config returns:

Elpy Configuration

Virtualenv........: None
RPC Python........: 2.7.14 (/home/brentg/conda/Ubuntu.17.10.miniconda2/envs/env1/bin/python)
Interactive Python: python (/home/brentg/conda/Ubuntu.17.10.miniconda2/envs/env1/bin/python)
Emacs.............: 25.2.2
Elpy..............: 1.19.0
Jedi..............: 0.12.0
Rope..............: Not found
Autopep8..........: 1.3.5
Yapf..............: Not found
Syntax checker....: flake8 (/home/brentg/conda/Ubuntu.17.10.miniconda2/envs/env1/bin/flake8)

[raoulhatterer]

I solved the problem with brew doctor indications. A lot of brew link where missing.

[sungwonida]

@jojojames Thanks! You saved my days. Python 3.7 installed by brew works perfectly.

[emanjavacas]

I was having a similar error with no autocompletion and it was solved by downgrading ipython from 7.2.0 to 6.2.1. Just in case it helps anybody I am writing this here.

[avatar-lavventura]

I also have the same error. Given log, when I apply run-python:

-UU-:---F2  doo.py  All (11,0)  :~/hack/Q1/  (User-MacBook.local) -------------|-UUU:**-F2  *Python*  All (7,4)  :~/hack/Q1/  (Alimoglu-MacBook.local) ------------
Python 3.7.3 (default, Apr 24 2020, 18:51:23)
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import codecs, os;__pyfile = codecs.open('''/var/folders/mx/m3c2fd193h169gg6nd5x3p480000gn/T/pyyNbUtw''', encoding='''utf-8''');__code =
__pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/mx/m3c2fd193h169gg6nd5x3p480000gn/T/pyyNbUtw''');exec(compile(__code, '''/var/folders/mx/m3c2fd193h169gg6nd5x3p480000gn/T/pyyNbUtw''', 'exec'));
python.el: native completion setup failed, <class 'Exception'>: libedit based readline is known not to work,
      see etc/PROBLEMS under "In Inferior Python mode, input is echoed".
>>>

[maksadbek]

This error everytime gets longer and longer:

```Python
Python 3.9.1 (default, Jan 6 2021, 06:04:49)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import codecs, os;__pyfile = codecs.open('''/var/folders/51/77zp4bqd4_53zn95hng9pt3h0000gn/T/pyoyHfVB''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/51/77zp4bqd4_53zn95hng9pt3h0000gn/T/pyoyHfVB''');exec(compile(__code, '''/Users/max/code/binsearch.py''', 'exec'));;

-1 15 7
7 15 11
11 15 13
13 15 14
15

In [2]: iimmppoorrtt ccooddeeccss,, ooss;;____ppyyffiillee == ccooddeeccss..ooppeenn((''''''//vvaarr//ffoollddeerrss//5511//7777zzpp44bbqqdd44__5533zznn9955hhnngg99pptt33hh00000000ggnn//TT//ppyyccBBiibbhhCC'''''',, eennccooddiinngg==''''''uuttff--88''''''));;____ccooddee == ____ppyyffiillee..rreeaadd(())..eennccooddee((''''''uuttff--88''''''));;____ppyyffiillee..cclloossee(());;ooss..rreemmoovvee((''''''//vvaarr//ffoollddeerrss//5511//7777zzpp44bbqqdd44__5533zznn9955hhnngg99pptt33hh00000000ggnn//TT//ppyyccBBiibbhhCC''''''));;eexxeecc((ccoommppiillee((____ccooddee,, ''''''//UUsseerrss//mmaaxx//ccooddee//bbiinnsseeaarrcchh..ppyy'''''',, ''eexxeecc''))));;;;

-1 15 7
7 15 11
11 15 13
13 15 14
15

In [3]: iiimmmpppooorrrttt cccooodddeeecccsss,,, ooosss;;;______pppyyyfffiiillleee === cccooodddeeecccsss...ooopppeeennn((('''''''''///vvvaaarrr///fffooollldddeeerrrsss///555111///777777zzzppp444bbbqqqddd444___555333zzznnn999555hhhnnnggg999pppttt333hhh000000000000gggnnn///TTT///pppyyyPPPFFFPPPsssYYYuuu''''''''',,, eeennncccooodddiiinnnggg==='''''''''uuutttfff---888''''''''')));;;______cccooodddeee === ______pppyyyfffiiillleee...rrreeeaaaddd((()))...eeennncccooodddeee((('''''''''uuutttfff---888''''''''')));;;______pppyyyfffiiillleee...ccclllooossseee((()));;;ooosss...rrreeemmmooovvveee((('''''''''///vvvaaarrr///fffooollldddeeerrrsss///555111///777777zzzppp444bbbqqqddd444___555333zzznnn999555hhhnnnggg999pppttt333hhh000000000000gggnnn///TTT///pppyyyPPPFFFPPPsssYYYuuu''''''''')));;;eeexxxeeeccc(((cccooommmpppiiillleee(((______cccooodddeee,,, '''''''''///UUUssseeerrrsss///mmmaaaxxx///cccooodddeee///bbbiiinnnssseeeaaarrrccchhh...pppyyy''''''''',,, '''eeexxxeeeccc'''))))));;;;;;

-1 15 7
7 15 11
11 15 13
13 15 14
15

In [4]: iiiimmmmppppoooorrrrtttt ccccooooddddeeeeccccssss,,,, oooossss;;;;________ppppyyyyffffiiiilllleeee ==== ccccooooddddeeeeccccssss....ooooppppeeeennnn((((''''''''''''////vvvvaaaarrrr////ffffoooollllddddeeeerrrrssss////55551111////77777777zzzzpppp4444bbbbqqqqdddd4444____55553333zzzznnnn99995555hhhhnnnngggg9999pppptttt3333hhhh0000000000000000ggggnnnn////TTTT////ppppyyyynnnnJJJJppppmmmmUUUUbbbb'''''''''''',,,, eeeennnnccccooooddddiiiinnnngggg====''''''''''''uuuuttttffff----8888''''''''''''))));;;;________ccccooooddddeeee ==== ________ppppyyyyffffiiiilllleeee....rrrreeeeaaaadddd(((())))....eeeennnnccccooooddddeeee((((''''''''''''uuuuttttffff----8888''''''''''''))));;;;________ppppyyyyffffiiiilllleeee....cccclllloooosssseeee(((())));;;;oooossss....rrrreeeemmmmoooovvvveeee((((''''''''''''////vvvvaaaarrrr////ffffoooollllddddeeeerrrrssss////55551111////77777777zzzzpppp4444bbbbqqqqdddd4444____55553333zzzznnnn99995555hhhhnnnngggg9999pppptttt3333hhhh0000000000000000ggggnnnn////TTTT////ppppyyyynnnnJJJJppppmmmmUUUUbbbb''''''''''''))));;;;eeeexxxxeeeecccc((((ccccoooommmmppppiiiilllleeee((((________ccccooooddddeeee,,,, ''''''''''''////UUUUsssseeeerrrrssss////mmmmaaaaxxxx////ccccooooddddeeee////bbbbiiiinnnnsssseeeeaaaarrrrcccchhhh....ppppyyyy'''''''''''',,,, ''''eeeexxxxeeeecccc''''))))))));;;;;;;;

-1 15 7
7 15 11
11 15 13
13 15 14
15

In [5]: ```