Spyder: IPython magics no longer work in Spyder 4

[hmaarrfk]

Issue Report Checklist

• [x] Searched the issues page for similar reports
• [ ] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• [x] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• [x] Could not reproduce inside jupyter qtconsole (if console-related)
• [x] Tried basic troubleshooting (if a bug/error)
  
  
  
  • [x] Restarted Spyder
  
  
  • [x] Reset preferences with spyder --reset
  
  
  • [x] Reinstalled the latest version of Anaconda
  
  
  • [x] Tried the other applicable steps from the Troubleshooting Guide
  
  
• [x] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

If we have ipython magic commands in the editor, running them with shift+enter no longer works with Spyder 4

%who print
# %%

runcell(0, 'C:/users/mark/git/vispy/vispy/app/untitled1.py')
  File "C:\users\mark\git\vispy\vispy\app\untitled1.py", line 1
    %who print
    ^
SyntaxError: invalid syntax

What is the expected output? What do you see instead?

It used to work in Spyder 3, has the decision been made to revert things?

Versions

• Spyder version: 4.0.0
• Python version: 3.7.3 64-bit
• Qt version: 5.12.5
• PyQt version: 5.12.3
• Operating System name/version: Windows / Linux

[goanpeca]

@impact27 @bcolsen ?

[ccordoba12]

IPython magics are not valid Python code, so we decided to not support them anymore in Python files. That will avoid the common problem of files that work in Spyder but not outside of it.

If you want to keep using them, you need to run them like

from IPython import get_ipython
get_ipython().magic('who print')

I think @impact27 added support for .ipy files, i.e. IPython script files. In those files you can use magics freely.

[hmaarrfk]

hmm ok, got it

[hmaarrfk]

Thanks for the thorough explication

[bcolsen]

Yes you can currently use line magics in .ipy files.

There is no support for output of the last command to be printed or cellmagics

If you're interested in those features I don't think we have a bug tracking those anymore but you can submit one here:
https://github.com/spyder-ide/spyder-kernels/issues

See more debate about it here: https://github.com/spyder-ide/spyder-kernels/issues/139

[hmaarrfk]

Is it possible to go through a deprecation cycle for this? Whereby users are told exactly what you told me about magics not being python but rather ipython specific and that they should save as ipy and rerun their file?

[impact27]

Alternatively we could create ipy files instead of py files by default? The problem is also that magic is detected as an error in the editor.

On 14 Dec 2019, at 16:06, Mark Harfouche [e-mail] wrote:


Is it possible to go through a deprecation cycle for this? Whereby users are told exactly what you told me about magics not being python but rather ipython specific and that they should save as ipy and rerun their file?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[goanpeca]

The problem is also that magic is detected as an error in the editor.

By the PyLS ? I could add a fix for that (at least on the editor side for a quick fix)?

[hmaarrfk]

@ccordoba12 I would like to request this issue be reopened. While I understand the desire to change the behavior, I feel like this was a key feature of Spyder that has since been removed with no guidance to users that don't fully appreciate the difference between python/ipython.

[ccordoba12]

Alternatively we could create ipy files instead of py files by default?

I don't agree with this. Magics are more of a convenience for interactive computing than something people should be using in production code.

I could add a fix for that (at least on the editor side for a quick fix)?

I don't think that's a good idea because magics are invalid Python syntax.

[impact27]

Does PyLS work with ipy files? We could try to detect ipython magic and have an error message like: ipython magic are not valid python code. Please save your file with an ipython extension (.ipy) for them to work.

On 14 Dec 2019, at 16:31, Gonzalo Peña-Castellanos [e-mail] wrote:


The problem is also that magic is detected as an error in the editor.

By the PyLS ? I could add a fix for that?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[ccordoba12]

I would like to request this issue be reopened

Allowing magics will break linting, folding and indent guidelines in the editor because they generate syntax errors. So I'm afraid you'll have to use the code I gave you above to run them.

We can ask @juanis2112 to document that while she's improving our documentation though.

[ccordoba12]

Does PyLS work with ipy files?

We'd need to extend it to work with them, but I don't think they'll work well with Pylint and Pyflakes.

We could try to detect ipython magic and have an error message like: ipython magic are not valid python code

Another option would be to parse the Python file sent to the PyLS before passing its contents to the linters, and transform magics to valid Python code.

I think IPython has some machinery to do that already.

[hmaarrfk]

Maybe the difference is that pressing "shift + enter" used to copy the code into the console. Which was really good to see exactly what was being typed, this made magics work.

So now we have a bunch of Spyder 3 scripts that used to work, and we love the work you have done with Spyder 4. Unfortunately the error message:

  File "/home/mark/git/delay_test.py", line 11
    %time
    ^
SyntaxError: invalid syntax

isn't very helpful because this wasn't a syntax error before. Something like:

SyntaxError: invalid syntax. In Spyder 4, we no longer allow iPython magics in .py files. If you want to use iPython magics, please save your file as .ipy the iPython file extension.

Would go a long way.

[hmaarrfk]

Even something like:

Detecting that the user is trying to use ipython magics upon save/linting in an .py file, and issuing them a warning, or option to resave their file as a .ipy to enable the magics to work.

[ccordoba12]

Maybe the difference is that pressing "shift + enter" used to copy the code into the console

Ok, you should have started there. If you want the previous behavior back, you need to go to

Preferences > Editor > Run code

and activate the option called Copy full cell contents to the console when running code cells.

[hmaarrfk]

and activate the option called Copy full cell contents to the console when running code cells.

I guess I'm trying to say that I've seen 4/4 users of Spyder 3 migrate to Spyder 4 and been tripped up by this behavior.

People posting on this thread are to say the least, pretty advanced users of python/spyder. This isn't the norm.

I agree with your desire to help people write better code, but I would consider this a breaking change in the user experience, and some kind of guidance would be helpful.

It seems that you would prefer users not to use Copy full cell contents to the console when running code cells, but rather to save their files as ipy to keep them portable.

A little guidance will go a long way.

Changing how code is excuted (copied or now) can be ignored. The fact that scripts developed in spyder no longer work with spyder is not really fun.

[hmaarrfk]

Thanks, I really appreciate understanding our usecase. I also appreciate the pointer to chaging the editor behavior to copy the contents of the cell.

[ccordoba12]

It seems that you would prefer users not to use Copy full cell contents to the console when running code cells, but rather to save their files as ipy to keep them portable.

The thing is not copying cell contents to the console allows pretty advanced things like debugging cells, which was added by @impact27.

The easiest thing for us to do would be to parse the file and replace magics by valid Python code before evaluating them in the kernel. @impact27, what do you think?

A big problem is allowing magics usage is detrimental because it'll break a lot of stuff in the editor. However, we could make some adjustments in the PyLS to deal with that (again, parsing the file to transform magics). @goanpeca, what do you think?

[impact27]

The easiest thing for us to do would be to parse the file and replace magics by valid Python code before evaluating them in the kernel. @impact27, what do you think?

You mean for python files as well? It would be very easy but then we need to do the same before sending to PyLS. But then we have the problem that a .py file might not work outside of spyder.

[goanpeca]

A big problem is allowing magics usage is detrimental because it'll break a lot of stuff in the editor. However, we could make some adjustments in the PyLS to deal with that (again, parsing the file to transform magics). @goanpeca, what do you think?

I am a bit torn, the previous behavior was broken cause % time is not valid python, so even if it used to work, it does not mean it was right for it to work I mean if it was something like

# %time

Then we would not need to really fix anything cause that is a normal comment, but we could add some support for "known" magics and strip the # when sending it to the console, (we could have an option that was ON by default to "parse known ipython magics"

This feels like the best since we do not need to customize the PYLS, we enforce old users to write VALID python code on the editor (with minimal effort), and provide a workaround on top of this to make life easy for users.

By doing this we could also add a better error message for users to indicate them that %time is not valid code, that they should use # %time instead.

Thoughts @impact27 , @ccordoba12 @hmaarrfk

[bcolsen]

I think for python 4 we should go back to using the run cell mechanism in already in ipython.

This fixes line magics, cell magics (like %%timeit), outputs of the last cell line and indentation errors we are having. This is the way the runcell function was originally implemented.

I think the only disadvantage (@impact27 can correct me here) was that the execution name was a hash instead of the real file name. Maybe debug cell would work with magic but I don't know what one should do in that case.

I think that the current plan we have here for magic in only ipy file is a great way forward for spyder 5 when we have proper support for .ipy in the interface. I would like to see an option in for a new ipy file and an explanation. I would also like to cell by cell execution for .ipy files when I use runfile(F5).

[ccordoba12]

I think for python 4 we should go back to using the run cell mechanism in already in ipython.

I guess you mean Spyder 4 above. If we decide to do that, we should change the default for 4.0.1 (to be released next week), instead of leaving more time with the current one.

What do other people think about it? I'd prefer to leave the current default (i.e. using runcell).

[hmaarrfk]

Is there a way to have the new mechanism be transparent?

I.e. make the console "appear" as though things were type in, but in the background use the new "runcell" command?

[ccordoba12]

No, I don't think so. The whole point of runcell is to not copy the contents of cells to the console.

[bcolsen]

Sorry yes Spyder 4. We would use Spyder kernel runcell() but change out the exec() call to use ipython.run_cell() instead. We will loose filenames in the tracebacks but the links and the line numbers will work. It shouldn't be to hard to do, I had it working before.

I want to reiterate that having a .ipy file in Spyder 5 that treats a text file with code cells like a set of ipython cells would be incredible, but we would have to educate about the change or maybe even make our own .spy Spyder script files since .ipy doesn't actually support cells.

[impact27]

I think we should:

• reenable line magic in .py files
• show a warning and suggest either
  
  
  
  • transforming to python code (we could have a button to do that in the source menu)
  
  
  • saving as .ipy
  
  
• in spyder5 have the warning become an error

On 14 Dec 2019, at 19:03, bcolsen [e-mail] wrote:


Sorry yes Spyder 4. We would use Spyder kernel runcell() but change out the exec() call to use ipython.run_cell() instead. We will loose filenames in the tracebacks but the links and the line numbers will work. It shouldn't be to hard to do, I had it working before.

I want to reiterate that having a .ipy file in Spyder 5 that treats a text file with code cells like a set of ipython cells would be incredible, but we would have to educate about the change or maybe even make our own .spy Spyder script files since .ipy doesn't actually support cells.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

[impact27]

So I had a look and I think the work to be done is, in spyder_kernels/customize/spydercustomize.py::exec_code:

• call the transform manager even if is_ipython is false
• Print a warning if the transform manager made something (Easiest would be to try to compile without the manager and checking if a syntaxerror is raised.
• fixing the cell magic by not calling tm.cleanup_transforms = []. Instead, count the number of blank lines removed and add them back after calling code = tm.transform_cell(code) (Otherwise the callback will be messed up.)

Then there is a few outstanding problems that needs to be addressed:

• runfile won't work with cells magic (line magic are fine) as tm.transform_cell only transforms a single cell. Maybe we should iterate on cells?
• tm.transform_cell places the entire cell on one line if a cell magic is used, so any traceback and debugging will be messed up. I am not sure how to fix this one but I don't expect that problem to be super commun.
• We need to call tm.transform_cell before sending the python code to PyLS

Any thoughts?

[impact27]

Implemented in https://github.com/spyder-ide/spyder-kernels/pull/190. I also opened https://github.com/spyder-ide/spyder/issues/11090

[ccordoba12]

Thanks for your help @impact27! Since this too much for a simple bugfix release, I'm going to leave this for 4.1.0

[impact27]

11101 would improve integration of ipython magic in spyder.

[capitanevs]

Hello Everyone.

I know this is a closed issue, but I would like to add to the voice of those who thinks that totally removing the IPython Magics from spyder could be a great mistake, and induce many people to stop upgrading, unless they will be really forced to.

The reason of that, in my view is simple: the ecosystem of programming is wide and broad. There are engineers, professionals, hobbysts, scientists, etc. And many different kind of codes can be produced: some of them are fully operative programs, some are just scripts.

I think that one of the greates beauties of python and of spyder combined in a unique nutshell, is the way it can help - with very little effort - in continuously improving the quality and the fateures of the code, with very little effort. And, believe it or not, the possibility of running sections of a code, is of great help in debugging.

Yes, more evolute instruments are available. But not all the users are professional programmers. And those who are, might be interested in even more sophisticated IDEs, such as pycharm.

I work in a large research facility (a particle accelerator). Coding is daily bread for a lot of persons, from undergraduated stutents to IT engineers. We use it for the dev of softwares, the dev of analysis scripts, and even (a very specific but interesting use) as a "textual console" where you have a list of short-hand macros, that the scientist can change, run, create to prontly solve problems that the hazard of the research activity brings to light. Some of the prototypal solutions, are then implemented as more definite solutions.

However, I fear that such a relentless work might be strongly impaired if such a small feature, like the #%% "magick" (something I was not even aware off) could be removed.

By summing up: spyder is great for creating prototypes. Don't you think that your pushing it too much towards "production", sacrifying the "R&D"?

Sorry for such a long trailer advocay. I might also not have understood all the possible implications (that's 100% sure...I am not a professional developer). But I'm fearing the day I will have to roll back my spyder version :-)

I hope this contribution could be of help and inspiration.

In any case, I want to thank you for the excellent work Carlos and all his collaborators are doing, since spyder is really a great piece of art, which greatly improved with time. I strongly promoted it in the fields of competence of my institution, and I will always be.

Yours Sincerely,

Michele

[impact27]

Hi, thanks for your feedback. We definitely don't want to remove IPython magics from spyder. The problem is with executing a python file with something that is not python. In my opinion, asking the user to save the file as an .ipy rather than as a .py is an acceptable compromise, so it is clear that the code contains ipython-only syntax. The syntax highlighter will be fixed when #11101 is merged.

But maybe you are right and we should just leave the warning indefinitely

[capitanevs]

Hi Quentin,

thank you for the reply.

Sorry for having used the coarse expression "removing IPython Magics", which is clearly what you do not want to do:-)

However my opinion still stays there: Spyder definition is "The Scientific Python Development Environment ", so that science shall be pivotal. And in science you may know with what you start, but not how you finish, and it is perfectly good in science (and since you are physicist, you will know well).

You might not know whether your python should be a .py or .ipy just. And mantaining a distinction between .py or .ipy might turn into an extra overhead.

However, I do not want to be presumptuous :)
Coming to practice:

I am developing a library (so .py files). When I have to do something new, a complicated analysis, a start writing a script rather than a function, because that way it is maintanable. I can do test and debug without using the debugging features. I can split the script into cells with #%%, which makes everything nice. The script grows huge. And finally I code it into a function/class etc.

Such a progressive coding (which probably is bad-practice in programming, but is ordinarily practice in science) is possible thanks to #%%. Using *.ipy would sound weird: my project has all .py files. So, creating a single *.ipy file (which may be called by another portion of the code) sounds cnfusing to me.
Not to talk of the dilemma of the *.py files already existing on my laptop: will I have to turn them into *.py or to remove #%%?

For sure I did not get all the implications of the magick word, and the possible bothering coming from the present "spurious" situation. I trust you guys for that.
But maybe, in this, you are just pushing spyder too "hard" if compared to part of spyder's audience.

But then... I don't get the point. You say "he problem is with executing a python file with something that is not python". But when you use a cell marked by #%%, you are actually using a comment. So... python does not care (even "emotionally" I mean). Or not?

Tnx
M

[impact27]

#%% is not ipython magics, it's just a comment that happens to be interpreted by the editor in a certain way, so you can use it wherever you want. If you use a real ipython magic, say %matplotlib qt5, in a library, you might have other problems:
Let's say someone wants to use your library. It's a python library so they try to use python. This would result in an error because %matplotlib qt5 is not valid python. So if we support this that means we are changing the language: you can run code with spyder that you can not run with python. Therefore what you are writing is not python code, because python can not run it.
Of course we are happy to let users write IPython code rather than python code. This code can only be run by IPython and will crash if you try to run it with python. As the code is not python code, the user should not use the .py extension.

[capitanevs]

Great news to me, so...sorry for the inappropriate advocacy ;-)

However, when I try to run a cell, I get this error (which lead me to this topic):

So I guess the problem is not "magics" but something else... flexible indentation perhaps? :-)

This is how my code looks like

[impact27]

Ha yes this is the flexible indentation. See https://github.com/spyder-ide/spyder/issues/12651

[capitanevs]

Thanks, I'll migrate my comments there!