elpy-shell-send-region-or-buffer show strange letters ^G

[urugang]

Summary

I cannot use C-c C-c to send region to shell.

Steps to reproduce

test.py

alist = [i for i in range(3)]
numbers = [-1, -2, -3]

1) elpy-shell-switch-to-shell
2) python-shell-send-region
it works as expected.
3) elpy-shell-send-region-or-buffer
shell pended as beblow:

In [14]: alist = [i for i in range(3)]
    ...: numbers = [-1, -2, -3]
^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G

My configuration

OS

macos 10.14.2

Result of (elpy-config)

Virtualenv........: None
RPC Python........: 3.7.2 (/usr/local/bin/python3)
Interactive Python: jupyter (/usr/local/bin/jupyter)
Emacs.............: 26.1
Elpy..............: 1.28.0
Jedi..............: 0.13.3
Rope..............: 0.12.0
Autopep8..........: 1.4.3
Yapf..............: 0.26.0
Black.............: Not found (18.9b0 available)
Syntax checker....: flake8 (/Users/urugang/Library/Python/3.7/bin/flake8)

Elpy configuration in my init.el

(use-package elpy
  :ensure t
  :config (setq python-shell-interpreter "python3")

  (put 'set-goal-column 'disabled nil)
  (elpy-enable)
  (setq elpy-rpc-ignored-buffer-size (* 1024 1024 1024))


  ;; interactive python
  (setq python-shell-interpreter-args "-i"
        elpy-rpc-python-command "python3")

  (setq python-shell-interpreter "jupyter"
        python-shell-interpreter-args "console --simple-prompt"
        python-shell-prompt-detect-failure-warning nil)
  (setq python-indent-guess-indent-offset t)
  (setq python-indent-guess-indent-offset-verbose nil)
  (add-to-list 'python-shell-completion-native-disabled-interpreters
             "jupyter"))

[rpgoldman]

I have just developed this same problem, after using elpy successfully for many months now. I haven't been able to track this down yet, but immediately before it happened, I reconfigured my use of flycheck with elpy, and upgraded some elpy packages. I will see if I can figure out what's going on. Are you using flycheck? If you are not, then I may be on the wrong track.

[rpgoldman]

So far I have only negative results. Turning off flycheck doesn't solve this problem. Using flycheck without flycheck-pycheckers doesn't fix it. Switching from ipython to python as the interpreter doesn't fix it. Turning off magit (which I just upgraded) didn't fix it.

[urugang]

I watched that elpy-shell-send-region-or-buffer called python-shell-send-string twice.
the first is my buffer content, the second is

import sys, codecs, os, ast;__pyfile = codecs.open('''/var/folders/hk/nfth06151_j9jw5mn46_ngvc0000gp/T/pyQhqRtI''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/hk/nfth06151_j9jw5mn46_ngvc0000gp/T/pyQhqRtI''');__block = ast.parse(__code, '''/tmp/a.py''', mode='exec'); __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);_ = __block.body.pop() if __isexpr else None;exec(compile(__block, '''/tmp/a.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''/tmp/a.py''', mode='eval')) if __isexpr else None

when I input second in python shell, it displayed "^G^G".
especially it is ok when inputed with

import sys, codecs, os, ast;__pyfile = codecs.open('''/var/folders/hk/nfth06151_j9jw5mn46_ngvc0000gp/T/pyQhqRtI''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/hk/nfth06151_j9jw5mn46_ngvc0000gp/T/pyQhqRtI''');__block = ast.parse(__code, '''/tmp/a.py''', mode='exec'); __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);_ = __block.body.pop() if __isexpr else None;exec(compile(__block, '''/tmp/a.py''', mode='exec'));

and

eval(compile(ast.Expression(__last.value), '''/tmp/a.py''', mode='eval')) if __isexpr else None

I am not familar with python, still do not understand why combind together is not ok.

[urugang]

ignore my previous comment.
Maybe it is that I compiled all el file to elc.
I reinstall all elpy package, it is now ok.

[LuhengStat]

I have encountered the same problem. And I found it only works if we changed the interpreter to "python" by

(setq python-shell-interpreter "python" python-shell-interpreter-args "-i")

@jorgenschaefer .Please check the problem.

Added just now: However, even if I use "python" interpreter, the issue came up randomly.

[rpgoldman]

I still have this problem after uninstalling and reinstalling elpy, and changing the interpreter from ipython to python makes no difference.
I get the following warning, but I have no idea if it is related:

Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to support readline, yet ‘python-shell-completion-native-enable’ was t and "python" is not part of the ‘python-shell-completion-native-disabled-interpreters’ list.  Native completions have been disabled locally. 

[rpgoldman]

@urugang I don't understand -- I don't even see elpy-shell-send-region-or-buffer -- all I see in elpy is elpy-shell--send-region-or-buffer-internal and elpy-shell-send-buffer-and-step.

[urugang]

hi, @rpgoldman.
I encounter problem again for jupyter.
so I reinstall and use config, it is ok.

  ;; interactive python
  (setq python-shell-interpreter "python3"
        elpy-rpc-python-command "python3"
        python-shell-interpreter-args "-i")

[urugang]

It my output for command "decribe-key C-c C-c".

C-c C-c runs the command elpy-shell-send-region-or-buffer (found in
elpy-mode-map), which is an interactive compiled Lisp function in
‘elpy-shell.el’.

It is bound to C-c C-c, C-c C-y r, <menu-bar> <Elpy> <Interactive
Python> <Send Region or Buffer>.

(elpy-shell-send-region-or-buffer &optional ARG)

Run ‘elpy-shell-send-region-or-buffer-and-step’ but retain point position.

[rpgoldman]

@urugang Sorry. I did something wrong when I was searching. I have found it now -- it's defined by the macro expression:

(elpy-shell--defun-step-go elpy-shell-send-region-or-buffer-and-step)

It's a little odd that the macro creates the function by removing and-step, instead of adding it, but what do I know?

I still have this problem, though. Not only do I get the ctrl-G strings when I send the buffer to the inferior python, but I also get it whenever I press the return key in the *Python* buffer.

Return is bound to comint-send-input, and my best guess is that this is what is broken somehow. But I can't figure out how.

[urugang]

it's my config which maybe help you.

(use-package elpy
  :ensure t
  :config
  (put 'set-goal-column 'disabled nil)
  (elpy-enable)
  (setq elpy-rpc-ignored-buffer-size (* 1024 1024 1024))
  (setq python-indent-offset 4)

  ;; interactive python
  (setq python-shell-interpreter "python3"
        elpy-rpc-python-command "python3"
        python-shell-interpreter-args "-i"))

I suspect file encoding problem,but cannot confirm it.

however, I add # -*- coding: utf-8 -*- to my python files.

[liebkne]

I'm have the same issue with iPython with the latest elpy, thought the naive python shell can work.

[liebkne]

@rpgoldman For ipython, you may need the --simple-prompt argument.

[rpgoldman]

@urugang I do have the UTF-8 spec at the top of my file.

I did some tracing and this is what I find gets sent to python (this is reformatted with autopep8 for readability) when I send my query.py buffer:

import sys
import codecs
import os
import ast
__pyfile = codecs.open(
    '''/var/folders/ys/9xhjsh8x2pq8j66_jy4r940c0000gn/T/py35241qvr''', encoding='''utf-8''')
__code = __pyfile.read().encode('''utf-8''')
__pyfile.close()
os.remove('''/var/folders/ys/9xhjsh8x2pq8j66_jy4r940c0000gn/T/py35241qvr''')
__block = ast.parse(
    __code, '''/Users/rpg/projects/xplan/xplan-experiment-analysis/query.py''', mode='exec')
__block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (
    __block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body)
__last = __block.body[-1]
__isexpr = isinstance(__last, ast.Expr)
_ = __block.body.pop() if __isexpr else None
exec(compile(__block, '''/Users/rpg/projects/xplan/xplan-experiment-analysis/query.py''', mode='exec'))
eval(compile(ast.Expression(__last.value),
             '''/Users/rpg/projects/xplan/xplan-experiment-analysis/query.py''', mode='eval')) if __isexpr else None

It does look like something is done in the first few lines (with __pyfile) to do something about coding, but I have no idea what.

There doesn't obviously look like anything here would cause the string of ctrl-G

[galaunay]

Bug in python-mode ?

I still have this problem, though. Not only do I get the ctrl-G strings when I send the buffer to the inferior python, but I also get it whenever I press the return key in the *Python* buffer.

This tends to indicate an issue in python-mode.
Do you still have the ^G problem with Elpy uninstalled ?

elpy-shell-send-file

@rpgoldman: The code you are mentioning is executed by the function elpy-shell-send-file, and basically extracts the last statement of the code to execute it separately. It allows to gather the last statement output to echo it when the elpy-shell-echo-output option is used.

Possible Lead

This bit:

__block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (
    __block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body)

has been added recently (PR #1534 and commit 464e9f8c26f9a4b9332a4bcae3ff3ba06b250166).
I suspect the ^G are coming from there, but I have no clue why...
I failed to reproduce the issue on Arch with Emacs 26.1.

Temporary workaround

As a temporary workaround, you can try overwriting the elpy-shell-send-file function with the old version:

(defun elpy-shell-send-file (file-name &optional process temp-file-name
                                         delete msg)
  "Like `python-shell-send-file' but evaluates last expression separately.

See `python-shell-send-file' for a description of the
arguments. This function differs in that it breaks up the
Python code in FILE-NAME into statements. If the last statement
is a Python expression, it is evaluated separately in 'eval'
mode. This way, the interactive python shell can capture (and
print) the output of the last expression."
  (interactive
   (list
    (read-file-name "File to send: ")   ; file-name
    nil                                 ; process
    nil                                 ; temp-file-name
    nil                                 ; delete
    t))                                 ; msg
  (let* ((process (or process (python-shell-get-process-or-error msg)))
         (encoding (with-temp-buffer
                     (insert-file-contents
                      (or temp-file-name file-name))
                     (python-info-encoding)))
         (file-name (expand-file-name
                     (or (file-remote-p file-name 'localname)
                         file-name)))
         (temp-file-name (when temp-file-name
                           (expand-file-name
                            (or (file-remote-p temp-file-name 'localname)
                                temp-file-name)))))
    (python-shell-send-string
     (format
      (concat
       "import sys, codecs, os, ast;"
       "__pyfile = codecs.open('''%s''', encoding='''%s''');"
       "__code = __pyfile.read().encode('''%s''');"
       "__pyfile.close();"
       (when (and delete temp-file-name)
         (format "os.remove('''%s''');" temp-file-name))
       "__block = ast.parse(__code, '''%s''', mode='exec');"
       "__last = __block.body[-1];" ;; the last statement
       "__isexpr = isinstance(__last,ast.Expr);" ;; is it an expression?
       "_ = __block.body.pop() if __isexpr else None;" ;; if so, remove it
       "exec(compile(__block, '''%s''', mode='exec'));" ;; execute everything else
       "eval(compile(ast.Expression(__last.value), '''%s''', mode='eval')) if __isexpr else None" ;; if it was an expression, it has been removed; now evaluate it
       )
      (or temp-file-name file-name) encoding encoding file-name file-name file-name)
     process)))

But you will then lose output echoing (if you are using it).

[rpgoldman]

@galaunay Thank you very much! Your work-around seems to do the trick for me.

If it helps, I was having this problem on MacOS, using emacs 25.3.50.1, so maybe that's why I am having problems you are not?

[galaunay]

I tend to think this is OS related indeed.
It is difficult for me to investigate without being to reproduce.
Do you know what is the default encoding used in macos ?

What you could do is to play with this function:

(defun elpy-test-shell-send-file ()
  ""
  (interactive)
  (elpy-shell--ensure-shell-running)
  (let* ((process  (python-shell-get-process-or-error t))
         (file-name (expand-file-name (buffer-file-name)))
         (encoding (with-temp-buffer
                     (insert-file-contents file-name)
                     (python-info-encoding))))
    (python-shell-send-string
     (format
      (concat
       "import sys, codecs, os, ast;"
       "__pyfile = codecs.open('''%s''', encoding='''%s''');"
       "__code = __pyfile.read().encode('''%s''');"
       "__pyfile.close();"
       "__block = ast.parse(__code, '''%s''', mode='exec');"
       ;; Has to ba a oneliner, which make conditionnal statements a bit complicated...
       " __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);"
       "__last = __block.body[-1];" ;; the last statement
       "__isexpr = isinstance(__last,ast.Expr);" ;; is it an expression?
       "_ = __block.body.pop() if __isexpr else None;" ;; if so, remove it
       "exec(compile(__block, '''%s''', mode='exec'));" ;; execute everything else
       "eval(compile(ast.Expression(__last.value), '''%s''', mode='eval')) if __isexpr else None" ;; if it was an expression, it has been removed; now evaluate it
       )
      file-name encoding encoding file-name file-name file-name)
     process)))

If you run it in a python buffer with M-x elpy-test-shell-send-file, it should display the ^G.
You could then try to tweak the line:

__block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body)

to try to understand where the bug is coming from...

[rpgoldman]

@galaunay You are correct -- this test restores the string of control-Gs.

What is the scope of the variables in this expression? I was wondering if there's some way to either get into the debugger during its execution, or set a global variable to the contents of __block. My attempts to do so have not been successful, probably because I don't understand the context in which an expression sent by python-shell-send-string is evaluated. I tried adding another string:

"global foo; foo = __block;"

but that does not seem to give me access to the resulting value in the python shell. Is it possible for me to shove strings from this python code into the *Python* buffer, and if so, how?

When you say "what is the default encoding used in macos?" do you mean encoding used by Python, or used by Emacs?

[rpgoldman]

My best guess is now that this happens because MacOS will only accept 1024 characters (and the test example that I entered was 1227). I wasn't able to push the test example successfully into a python interpreter in a terminal window, either. Perhaps this is related: https://unix.stackexchange.com/questions/204815/terminal-does-not-accept-pasted-or-typed-lines-of-more-than-1024-characters

Would it be possible to take this very long text block and define it as an executable function and load it, then invoke said function with just the arguments (file-name, and encoding) instead of thrusting the full python code string at the *Python* buffer in this way?

Here is the text block I captured from python-shell-send-string (with some additional bits to try to get some useful output logged).

with open("/tmp/output", "w") as file: print("starting",flush=True, file=file); import sys, codecs, os, ast;__pyfile = codecs.open('''/Users/rpg/projects/xplan/xplan-experiment-analysis/query.py''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();__block = ast.parse(__code, '''/Users/rpg/projects/xplan/xplan-experiment-analysis/query.py''', mode='exec'); print("Block body before = %r"%__block.body, flush=True, file=file); __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);_ = __block.body.pop() if __isexpr else None;print("Block body after = %r"%__block.body, flush=True, file=file);exec(compile(__block, '''/Users/rpg/projects/xplan/xplan-experiment-analysis/query.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''/Users/rpg/projects/xplan/xplan-experiment-analysis/query.py''', mode='eval')) if __isexpr else None"

[galaunay]

I was wondering if there's some way to either get into the debugger during its execution, or set a global variable to the contents of __block

The whole statement is executed in the shell, so you should be able to to access __block (and everything else) from the shell:

You could also add the following string to pop up the debugger during execution:

"import pdb; pdb.set_trace();"

My best guess is now that this happens because MacOS will only accept 1024 characters.

That 's a valuable information, nice catch. I don't like how we are sending long strings to python either.
This is the way python.el was doing it, and we just complied. But I think you are right, it will be better to have a function loaded at the creation of the interpreter.

[rpgoldman]

@galaunay It turns out to be a little more complicated than I thought. This is a bug in the Python 3.6 for the Mac that is distributed with Homebrew. I upgraded to 3.7.x, and now elpy-shell-send-file works fine. So I guess it's up to the elpy maintainers whether it's worth changing this or just treating 3.6 as unsupported (at least on MacOS).

[galaunay]

We could add a warning for people using python3.6 on MacOs.
Would you have any link regarding this bug I can forward people to ?

[rpgoldman]

@galaunay I'm afraid not. I just know that the new version of elpy did not work on 3.6.5, but did work on 3.7.2, and that it was definitely a problem with input buffering, because I could not cut and paste a string longer than 1024 into the python shell running in a terminal, but I was able to paste the same string into 3.7.2.

I suppose one could try groveling over the Python changelog, but that would be a lot of work for little payoff.

[rthoma]

@galaunay Unfortunately I am experiencing this problem on macOS Mojave with Python 3.7.2. It just started happening today after I did some upgrades in package-list-packages. So the problem does not appear to be limited to Python 3.6. The behavior appears to go away when I switch python-shell-interpreter from ipython to python3.

[rpgoldman]

@rthoma I can replicate this. With 3.7.2, this problem goes away if I use python (python3), but persists in ipython.

[urugang]

my python is also macos python 3.72, still have this problem.
sometimes works, and sometimes does not work. It is because length of filename in codec.open is variable.

so should we seperate into multiple python-shell-send-string call, instead of single call?

    (python-shell-send-string
     (format
      (concat
       "import sys, codecs, os, ast;"
       "__pyfile = codecs.open('''%s''', encoding='''%s''');"
       "__code = __pyfile.read().encode('''%s''');"
       "__pyfile.close();"
       (when (and delete temp-file-name)
         (format "os.remove('''%s''');" temp-file-name))
       "__block = ast.parse(__code, '''%s''', mode='exec');"
       ;; Has to ba a oneliner, which make conditionnal statements a bit complicated...
       " __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);"
       "__last = __block.body[-1];" ;; the last statement
       "__isexpr = isinstance(__last,ast.Expr);" ;; is it an expression?
       "_ = __block.body.pop() if __isexpr else None;" ;; if so, remove it
       "exec(compile(__block, '''%s''', mode='exec'));" ;; execute everything else
       "eval(compile(ast.Expression(__last.value), '''%s''', mode='eval')) if __isexpr else None" ;; if it was an expression, it has been removed; now evaluate it
       )
      (or temp-file-name file-name) encoding encoding file-name file-name file-name)
     process)

[rthoma]

Thanks for confirming @rpgoldman. I have no idea if this is related, but I do think there were some file encoding changes either between macOS High Sierra and Mojave or Python 2.7 and Python 3. I have a Python script that parses files via fileinput. On my old machine it used to work without specifying an openhook, but with Mojave and Python 3, it breaks unless I specify openhook=fileinput.hook_encoded("mac_roman"). Again, no idea if this is related, but interesting that I've bumped into a couple of file encoding issues related to Python in just the past couple of days.

[galaunay]

I made a PR (#1553) to fix this (and avoid sending long and obscure strings to the shell, which is anyway a bad idea).

I cannot really test it on MacOS.
Could someone have a try to see if it is behaving properly ?

[urugang]

@galaunay I tested it on macos, it seems "Python" buffer did not response any input, especially when I input on *Python manually.

;; This buffer is for text that is not saved, and for Lisp evaluation.
;; To create a file, visit it with C-x C-f and enter text in its buffer.


(package-initialize)
(add-to-list 'load-path "~/codes/elpy")
(require 'elpy)

(elpy-enable)
(setq python-shell-interpreter "jupyter"
      python-shell-interpreter-args "console --simple-prompt"
      python-shell-prompt-detect-failure-warning nil)

(add-to-list 'python-shell-completion-native-disabled-interpreters
             "jupyter")

[urugang]

the procedure are

• append code to buffer Python
• compile and eval

To check step 2 is right, make a test file "~/a.py", directly call python-shell-send-string, output is ok. It also indicate the problem is not in terminal input buffer limit.

so the problem is in elpy-shell--append-to-shell-output, or the overall procedure.

a = "aaaa"
b = "bbbb"
print(a)
print(b)

"""
(setq encoding "utf-8")
(setq file-name "/Users/xx/a.py")
(setq temp-file-name nil)
(setq delete nil)
(setq process (python-shell-get-process-or-error ""))
(python-shell-send-string
 (format
  (concat
   "import sys, codecs, os, ast;"
   "__pyfile = codecs.open('''%s''', encoding='''%s''');"
   "__code = __pyfile.read().encode('''%s''');"
   "__pyfile.close();"
   (when (and delete temp-file-name)
    (format "os.remove('''%s''');" temp-file-name))
   "__block = ast.parse(__code, '''%s''', mode='exec');"
   ;; Has to ba a oneliner, which make conditionnal statements a bit complicated...
   " __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);"
   "__last = __block.body[-1];" ;; the last statement
   "__isexpr = isinstance(__last,ast.Expr);" ;; is it an expression?
   "_ = __block.body.pop() if __isexpr else None;" ;; if so, remove it
   "exec(compile(__block, '''%s''', mode='exec'));" ;; execute everything else
   "eval(compile(ast.Expression(__last.value), '''%s''', mode='eval')) if __isexpr else None" ;; if it was an expression, it has been removed; now evaluate it
  )
  (or temp-file-name file-name) encoding encoding file-name file-name file-name)
 process)
"""

[LuhengStat]

I made a PR (#1553) to fix this (and avoid sending long and obscure strings to the shell, which is anyway a bad idea).

I cannot really test it on MacOS.
Could someone have a try to see if it is behaving properly ?

I have tested on macos. However, the problem still exists. Especially when I try to run code line by line.

[LarryLegend33]

Hi galaunay

I think a lot of these issues come from your most recent commit where you changed the echoing behavior and made the elpy-shell-capture-last-multiline-output variable obsolete. When python-shell-send-string is called, It now sends the entire python string (starting with "import sys") to the python buffer, which if you have the --simple-prompt flag for ipython turned on, produces the ^G^G^G behavior. Turning the --simple-prompt flag off and simply setting the python-shell-interpreter variable to "ipython" eliminates the ^G^G^G behavior, but now prints the entire python string on the first line of the interpreter. I assume this used to be eliminated by setting elpy-shell-capture-last-multiline-output to nil, but this is no longer possible. Turning all elpy echo variables to nil also does not solve the problem.

What is now happening to me (on OSX mojave, ipython 5.8, python 2.7.15) is that python-shell-send-string is sending the entire python string to my python buffer every time I run C-c C-c. This comes out as ^G^G^G^G^G and halts if I uncomment the -args line in my .emacs file.

(setq python-shell-interpreter "ipython")
;; python-shell-interpreter-args "--simple-prompt")

In [1]: In [1]: import sys;sys.path.append('~/Escape-Analysis/')

In [2]:
r
e
a
t

d

s
f
t
_
b
o
/
i
'
...: ', mode='eval')) if __isexpr else None
r
e
a
t

d

s
f
t
_
b
o
/
i
'
...: ', mode='eval')) if __isexpr else None

In [3]:

[mddavila]

I got the same error in macOSMojave (v. 10.14.3) running Python 3.7.1 with Emacs 26.1. However the temporary workaround indicated above solves it even when I am using the IPython default configuration indicated in the guide.

[galaunay]

As stated in the documentation, Ipython should be used with --simple-prompt.
It interesting that the ^G disappear without this flag.

Apparently, avoiding to send long lines of code (PR #1553) doesn't fix the issue.
So those ^G have to come from elsewhere...

I made a PR (#1554) that allows not to use the faulty function when elpy-shell-echo-output is set to nil.
It will allow you to continue using Elpy until we can find a proper solution.

Could somebody experiencing the ^G issue tell me if this PR (and setting elpy-shell-echo-output to nil) fixes the problem ?

[mddavila]

Sorry, my mistake, when my configuration is set to this:

(setq python-shell-interpreter "ipython")

my emacs just crust whenever I eval any .py file.

[vancleve]

I'm seeing this problem too on mac OS 10.14.3 with Python 3.7.2 / IPython 7.2.0 / GNU Emacs 26.1 / Elpy 1.28.0

[zhuzhongshu]

+1 on mac OS too

[lljr]

Hmm, I'm on macos mojave and cannot reproduce the issue. Although, I have this

(when (memq window-system '(mac ns x))
      (setenv "LC_CTYPE" "UTF-8")
      (setenv "LC_ALL" "en_US.UTF-8")
      (setenv "LANG" "en_US.UTF-8"))

in my init.el which I grabbed from EmacsWiki:Unicode on Mac OS X

[ConnorNelson]

I have been having this ^G issue as well on macOS. From my tests, the issues seem to be related to exceeding 1024 bytes being input (and maybe newline characters). This issue also exists running ipython --simple-prompt in a terminal (outside of emacs), so it is not strictly an elpy issue (though elpy expecting --simple-prompt does make this really only noticeable with elpy).

Importing readline actually solves all of the issues, and hasn't broken elpy in my basic testing.
Currently, I have my emacs python configured as follows:

(setq python-shell-interpreter "ipython"
      python-shell-interpreter-args "--simple-prompt -c exec('__import__(\\'readline\\')') -i")

This is a great temporary fix for now. I imagine the patch would also involve importing readline (I am not aware of any negative consequences of doing this).

[galaunay]

readline is not even long to import, I guess it is not a big problem to import it by default.

It should be fixed by #1562.

[mddavila]

sadly I found that the new workaround does not work for me:
```
Python 3.7.3 | packaged by conda-forge | (default, Mar 27 2019, 15:43:19)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from mpl_toolkits import mplot3d

In [2]: import numpy as np
import sys, codecs, os, ast, readline;__pyfile = codecs.open('''/var/folders/1l/xcx88qj10w96r83qn2_4xjfc0000gn/T/pyaNrk3f''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/1l/xcx88qj10w96r83qn2_4xjfc0000gn/T/pyaNrk3f''');__block = ast.parse(__code, '''/Users/mddavila/3d plot.py''', mode='exec'); __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);_ = __block.body.pop() if __isexpr else None;exec(compile(__block, '''/Users/mddavila/3d plot.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''/Users/mddavila/3d plot.py''', mode='eval')) if __isexpr else None
```

[ConnorNelson]

Yes, unfortunately this patch does not seem to work. I don't currently have time to investigate this further, but I suspect that the readline needs to be imported even earlier.
On a side note (if you don't just revert it), this patch seems to also break elpy-shell--python-shell-send-string-echo-advice, since readline was not also added here.

For now, I would just revert the patch, and macOS users can just use:

(setq python-shell-interpreter "ipython"
      python-shell-interpreter-args "--simple-prompt -c exec('__import__(\\'readline\\')') -i")

Another solution is something like:

(advice-add 'run-python :after (lambda (_ _ _) (python-shell-send-string "import readline")))

I will try to find some time to investigate for a more permanent solution.

[bbo2adwuff]

When using elpy with the commit https://github.com/jorgenschaefer/elpy/commit/4e5c47310dce5187452970e4a5ce1170f60ba077 then I see following under linux:

>>> 1 + 1
import sys, codecs, os, ast, readline;__pyfile = codecs.open('''/tmp/pyZf7jTP''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/tmp/pyZf7jTP''');__block = ast.parse(__code, '''/home/abc/Documents/csv.py''', mode='exec'); __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);_ = __block.body.pop() if __isexpr else None;exec(compile(__block, '''/home/abc/Documents/csv.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''/home/abc/Documents/csv.py''', mode='eval')) if __isexpr else None
2
>>>

I tried it also with a Windows machine at work...

>>> 1 + 1
import sys, codecs, os, ast, readline;__pyfile = codecs.open('''c:/Users/abc/AppData/Local/Temp/py7gX0Id''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''c:/Users/abc/AppData/Local/Temp/py7gX0Id''');__block = ast.parse(__code, '''m:/user/abc/csv/csv.py''', mode='exec'); __block.body = (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body) if sys.version_info[0] < 3 else (__block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.NameConstant) else __block.body if not __block.body[0].test.value is True else __block.body[0].body);__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);_ = __block.body.pop() if __isexpr else None;exec(compile(__block, '''m:/user/abc/csv/csv.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''m:/user/abc/csv/csv.py''', mode='eval')) if __isexpr else None
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'readline'
>>> 

After reverting commit https://github.com/jorgenschaefer/elpy/commit/4e5c47310dce5187452970e4a5ce1170f60ba077:

Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> 1 + 1
2
>>>

[galaunay]

I reverted 4e5c47310dce5187452970e4a5ce1170f60ba077. There is not point if it is not fixing the issue anyway...

We could also import readline at the shell creation (in elpy-shell-get-or-create-process).

[bbo2adwuff]

Any fix that includes readline would break elpy on Windows anyways, right? (See for example https://github.com/jorgenschaefer/elpy/issues/887#issuecomment-287807317)

[galaunay]

Nice catch.
It would not be difficult to only do so for MacOS (using system-type).

[xelibrion]

Just encountered the issue after upgrading elpy package. Everything works fine with elpy-20190217.2015 and breaks when updating to elpy-20190416.1701 on the same laptop. No issues in Linux.

[galaunay]

@xelibrion The PR leading to this issue was merged the 20th of February, so you indeed have it now.

You can use one of the workarounds from ConnorNelson (related to readline) or just deactivating output echoing by setting elpy-shell-echo-output to nil.

[LuhengStat]

The issue now seems still exists. I had choose the method of overwriting the elpy-shell-send-file to solve this issue temporarily.

[eihli]

I think this is a problem with the input buffer size of the python process.

I can recreate this on-demand by changing the length of the string sent to process-send-string.

DEFUN ("process-send-string", Fprocess_send_string, Sprocess_send_string,
       2, 2, 0,
       doc: /* Send PROCESS the contents of STRING as input.
PROCESS may be a process, a buffer, the name of a process or buffer, or
nil, indicating the current buffer's process.
If STRING is larger than the input buffer of the process (the length
of which depends on the process connection type and the operating
system), it is sent in several bunches.  This may happen even for
shorter strings.  Output from processes can arrive in between bunches.

If PROCESS is a non-blocking network process that hasn't been fully
set up yet, this function will block until socket setup has completed.  */)

On my mac, Mojave 10.14.6 and Python 2.7.11, if I checkout 740b0a0~1 and make the following change, then I get the error.

diff --git a/elpy-shell.el b/elpy-shell.el
index d16baaf..1ce3e5d 100644
--- a/elpy-shell.el
+++ b/elpy-shell.el
@@ -628,6 +628,14 @@ print) the output of the last expression."
       (concat
        "import sys, codecs, os, ast;"
        "__pyfile = codecs.open('''%s''', encoding='''%s''');"
+      "('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)"
+      "('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)"
+      "('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)"
+      "('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)"
+      "('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)"
+      "('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)"
+      "('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)"
+      "('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)"
        "__code = __pyfile.read().encode('''%s''');"
        "__pyfile.close();"
        (when (and delete temp-file-name)

With that change, this is the error I see when I try to evaulate a buffer.

>>> print("HI")
...: print("BYE")
import codecs, os, ast;__pyfile = codecs.open('''/var/folders/jy/nwdc6ngd1012k8mqbb_3tz_c0000gn/T/pyoNfgqv''', encoding='''utf-8''');('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg)__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/jy/nwdc6ngd1012k8mqbb_3tz_c0000gn/T/pyoNfgqv''');__block = ast.parse(__code, '''/tmp/foo.py''', mode='exec');__block.body = __block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __
^G
^G
^G

Scroll to the right of that code block and you'll notice the string is truncated.

If I remove those lines that do nothing but increase the length of the string, I get the following output.

>>> print("HI")
...: print("BYE")
import codecs, os, ast;__pyfile = codecs.open('''/var/folders/jy/nwdc6ngd1012k8mqbb_3tz_c0000gn/T/pypUfhiS''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/jy/nwdc6ngd1012k8mqbb_3tz_c0000gn/T/pypUfhiS''');__block = ast.parse(__code, '''/tmp/foo.py''', mode='exec');__block.body = __block.body if not isinstance(__block.body[0], ast.If) else __block.body if not isinstance(__block.body[0].test, ast.Name) else __block.body if not __block.body[0].test.id == 'True' else __block.body[0].body;__last = __block.body[-1];__isexpr = isinstance(__last,ast.Expr);_ = __block.body.pop() if __isexpr else None;exec(compile(__block, '''/tmp/foo.py''', mode='exec'));eval(compile(ast.Expression(__last.value), '''/tmp/foo.py''', mode='eval')) if __isexpr else None
HI
BYE
>>> 

That string is not truncated and my comint buffer doesn't break. I'm still curious why the import codecs line is being output. But at least the comint buffer doesn't break and start showing those ^G symbols.

[galaunay]

Nice catch.
So if I understand properly, the ^G are a bug that only appears when sending long strings to process-send-string.

Do we then need to report it upstream ?

[eihli]

I don't understand it enough yet to know where to report it. I just discovered another bit of useful info today. I get different behavior by changing my python-shell-interpreter. IPython 2.0.0 works fine. IPython 5.8.0 seems to print one character per line of part of the string sent to process-send-string. The default shell for Python 2.7.10 prints a single line of part of the string sent to process-send-string and gives me the ^G

```
;; Python 2.7.10
;; >>> >>> print("HI")
;; ...: print("BYE")
;; import codecs, os, ast;__pyfile = codecs.open('''/var/folders/jy/nwdc6ngd1012k8mqbb_3tz_c0000gn/T/pyOPOvGb''', encoding='''utf-8''');__code = __pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/jy/nwdc6ngd1012k8mqbb_3tz_c0000gn/T/pyOPOvGb''');__block = ast.parse(__code, '''/tmp/foo.py''', mode='exec');('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg');
;; etc...
('abcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefgabcdefg');('abcdefgabcde
;; 
;; 

;; IPython 5.8.0
;; In [1]: print("HI")
;; ...: print("BYE")
;; In [1]: import sys;sys.path.append('/Users/eihli/code/cur_path')
;; # ... etc. Seems to print 1 char per line for some of the string sent to process-send-string.
;; x
;; '
;; e
;; ...: ), '''/tmp/foo.py''', mode='eval')) if __isexpr else None
;; HI
;; BYE

;; IPython 2.0.0
;; Works fine.
;; In [2]: print("HI")
;; ...: print("BYE")
;; HI
;; BYE
;; In [3]:

````

[PifPof73]

Hi,
I get the same Ctrl-G problem on a quite new setting:

• macOS Catalina 10.15
• Python 3.7.4 (default, Oct 14 2019, 16:45:38) (installed with macports)
• Emacs 27.0.50 (9.0) (installed with macports)
• Elpy 1.31.0

It's a german OS, encoding set to utf-8.

I tested python3 and ipython, but no of the above workarounds worked for me.
If I open ipython or python3 with a python file on the iTerm2 shell, it is opend and displayed without Ctrl-G.
Greetings,
PifPof73

[galaunay]

@PifPof73 So you have tried with:

(setq elpy-shell-echo-output nil
      python-shell-interpreter "ipython"
      python-shell-interpreter-args "--simple-prompt -c exec('__import__(\\'readline\\')') -i")

and it is still producing ^Gs after restarting the python interpreter ?

[gmcclure]

@galaunay This worked for me. I'd like to use Jupyter, so later today (this weekend, later next week, whenever I can breathe) I'll try to adapt it. Great fix -- thank you.

[aiguofer]

Trying to use my same Linux setup in MacOS I ran into this issue. I can confirm that simply having (setq elpy-shell-echo-output nil) works for me.

Btw, I use:

  (setq python-shell-interpreter "jupyter-console"
        python-shell-interpreter-args "--simple-prompt"
        python-shell-prompt-detect-failure-warning nil)

[urugang]

@aiguofer This worked for me.

[jayps95]

@aiguofer As of 4th June 2020, this works for me as well. The config i'm using is:

Emacs.............: 26.3
Elpy..............: 1.34.0
Virtualenv........: None
Interactive Python: python3 3.7.2 (/Library/Frameworks/Python.framework/Versions/3.7/bin/python3)
RPC virtualenv....: rpc-venv (/Users/james/.emacs.d/elpy/rpc-venv)
Python...........: python3 3.7.3 (/Users/james/.emacs.d/elpy/rpc-venv/bin/python3)
Jedi.............: 0.17.0
Rope.............: 0.17.0
Autopep8.........: 1.5.3
Yapf.............: 0.30.0
Black............: 19.10b0
Syntax checker....: Not found (flake8)

[avatar-lavventura]

@aiguofer Adding (setq elpy-shell-echo-output nil) prevented the error but produces the main output along with some additional output as:

import codecs, os;__pyfile = codecs.open('''/var/folders/mx/m3c2fd193h169gg6nd5x3p480000gn/T/pykPnxPt''', encoding='''utf-8''');__code =
__pyfile.read().encode('''utf-8''');__pyfile.close();os.remove('''/var/folders/mx/m3c2fd193h169gg6nd5x3p480000gn/T/pykPnxPt''');exec(compile(__code,
'''/Users/alper/code/string.py''', 'exec'));

Followed by my output.

[nkicg6]

It seems like the proposed solutions (from @aiguofer https://github.com/jorgenschaefer/elpy/issues/1550#issuecomment-574512892 and @galaunay https://github.com/jorgenschaefer/elpy/issues/1550#issuecomment-547515504) use jupyter-console or ipython as the python-shell-interpreter. I just updated and reconfigured elpy and I am still having this issue if I use the default python3 or python shell, but it does work if I install ipython and set that as the interpreter.

Is the consensus that we have to use ipython or jupyter as the interpreter to use elpy on macos?? that's kind of a big dependency, I'd be happy to help work on this as best I can if this is the case...