Spyder: Indentation handling when running code cells (IndentationError: unexpected indent)

[adriesse]

Issue Report Checklist

• [x] Searched the issues page for similar reports
• [x] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• [ ] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• [ ] Could not reproduce inside jupyter qtconsole (if console-related)
• [ ] Tried basic troubleshooting (if a bug/error)
  
  
  
  • [x] Restarted Spyder
  
  
  • [ ] Reset preferences with spyder --reset
  
  
  • [ ] Reinstalled the latest version of Anaconda
  
  
  • [ ] Tried the other applicable steps from the Troubleshooting Guide
  
  
• [x] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

A cell with indented code cannot be run as it could before. In this example the second cell cannot be run:

print ('hello')

if 1:
    print('hello again')

#%%
    print ('hello')

Versions

Spyder version: 4.1.2
Python version: 3.7.2 64b
Qt version: 5.9.6
PyQt version: 5.9.2
Operating System name/version: Win7 64b

Dependencies

# Mandatory:
atomicwrites >=1.2.0           :  1.2.1 (OK)
chardet >=2.0.0                :  3.0.4 (OK)
cloudpickle >=0.5.0            :  0.6.1 (OK)
diff_match_patch >=20181111    :  20181111 (OK)
intervaltree                   :  None (OK)
IPython >=4.0                  :  7.2.0 (OK)
jedi =0.15.2                   :  0.15.2 (OK)
nbconvert >=4.0                :  5.3.1 (OK)
numpydoc >=0.6.0               :  0.8.0 (OK)
paramiko >=2.4.0               :  2.4.2 (OK)
parso =0.5.2                   :  0.5.2 (OK)
pexpect >=4.4.0                :  4.7.0 (OK)
pickleshare >=0.4              :  0.7.5 (OK)
psutil >=5.3                   :  5.4.8 (OK)
pygments >=2.0                 :  2.3.1 (OK)
pylint >=0.25                  :  2.2.2 (OK)
pyls >=0.31.9;<0.32.0          :  0.31.9 (OK)
qdarkstyle >=2.8               :  2.8 (OK)
qtawesome >=0.5.7              :  0.5.7 (OK)
qtconsole >=4.6.0              :  4.6.0 (OK)
qtpy >=1.5.0                   :  1.5.2 (OK)
rtree >=0.8.3                  :  0.8.3 (OK)
sphinx >=0.6.6                 :  1.8.2 (OK)
spyder_kernels >=1.9.0;<1.10.0 :  1.9.0 (OK)
watchdog                       :  None (OK)
zmq >=17                       :  17.1.2 (OK)

# Optional:
cython >=0.21                  :  0.29.2 (OK)
matplotlib >=2.0.0             :  3.0.2 (OK)
numpy >=1.7                    :  1.15.4 (OK)
pandas >=0.13.1                :  0.24.2 (OK)
scipy >=0.17.0                 :  1.2.1 (OK)
sympy >=0.7.3                  :  1.3 (OK)

[andfoy]

Hi @adriesse, it seems that the snippet posted on the issue has wrong identation. That should be fixed by correcting the identation of the offending lines.

[adriesse]

To run a cell, normally only the indentation within that cell must be correct--at least that's the way it used to work. Anyway, I've updated the snippet so that it is correct for the whole file as well. The second cell still does not run.

[bcolsen]

This is a duplicate of #11023, which should be fixed in spyder-kernels.

I can't reproduce the error here using the same spyder-kernels version as @adriesse

[adriesse]

I apologize for not finding that other issue earlier, but the way I read now it is that it was fixed for .ipy files but not for .py files and nevertheless closed. I'm afraid I don't quite understand this.

[bcolsen]

It should work on py and ipy files. I think we are only going to allow ipython magic commands in ipy eventually but code cells will still work in py files.

@impact27 Can you reproduce this one?

[impact27]

With the ipython PR I get:

In [1]: runcell(1, '/Users/quentinpeter/Desktop/untitled2.py')

WARNING: This is not valid Python code. If you want to use IPython magics, flexible indentation, and prompt removal, please save this file with the .ipy extension. This will be an error in a future version of Spyder.

hello

In [2]: 

[ccordoba12]

I can't reproduce it either. @dalthviz, can you reproduce this on Windows?

[bcolsen]

@impact27 I get that too. Off topic but I think that warning is coming up in error. We should check if the % is in a comment.

[impact27]

I think that warning is coming up in error

Why? Random indentation is not correct python code. Executing:

#%%
    print ('hello')

Shouldn't work, unless this is a ipython file.

[adriesse]

I observe the same behaviour, whether the filename ends in '.py' or '.ipy'.

[bcolsen]

Why? Random indentation is not correct python code

It also not correct in ipy files either. This behavior is actually most important in python files particularly for the code cells in the if __name__ = "__main__" section.

[impact27]

Why? Random indentation is not correct python code

It also not correct in ipy files either. This behavior is actually most important in python files particularly for the code cells in the if __name__ = "__main__" section.

Fair, then we can use leading_indent from IPython.core.inputtransformer2 (PY3 only) when running cells and the full TransformerManager for ipython files.

[adriesse]

I thought this was an editor/ide convenience feature, not part of the Python language.

[adriesse]

Have you all decided which behavior Spyder will provide going forward?

[dalthviz]

Hi @adriesse I think some ideas were indeed proposed by @impact27 what do you think @ccordoba12 ?

Also, checking this I wasn't able to reproduce the error (at least using Spyder 4.1.3) using Ctrl + Enter to run the cells.

This is what I'm getting with the Copy full cell contents to the console when running code cell option enabled (this option is available in the Preferences dialog > Editor > Run code):

And this is what I'm getting with the option disabled:

If you keep experiencing the issue (the raise of an IndentationError) could you try to:

• Update to Spyder 4.1.3
• Record a GIF for a better understanding of the way you are running the cells and how an IndentationError is being raised

Any new input (info about the error you experienced and/or perspective on how things should be working) is greatly appreciated, thanks :)

[ccordoba12]

I think some ideas were indeed proposed by @impact27 what do you think @ccordoba12 ?

Let's wait to see what @adriesse says because I can't reproduce this bug either on Linux.

[adriesse]

I'm still unclear on what your intended behavior is, and/or whether there was a conscious decision recently to change the intended behavior, so I don't whether it is a bug. Anyway, here are some screen shots:

With "copy full cell contents":

Without "copy full cell contents":

[dalthviz]

Thanks for the screenshots @adriesse I was able to reproduce this!:

Please update ipython (that should enable you to just see a warning instead of the IndentationError):

[adriesse]

Thanks! The upgrade changes the error to a warning, and now if I rename the file to .ipy the warning goes away.

How will it be in the future?

As @bcolsen observed above:

It also not correct in ipy files either. This behavior is actually most important in python files particularly for the code cells in the if __name__ = "__main__" section.

What is the motivation for disallowing the execution of indented code blocks? Has it caused problems?

[dalthviz]

I think that indentation is a fundamental part of the syntax of Python and taking that into account basically mixed indentation levels are not the correct way to go (the code will not be accepted for the Python interpreter). As an example the following isn't correctly formatted Python code:

Even though currently you can run this inside Spyder (and a warning mentions what I'm saying):

Using .ipy and IPython in general provides a layer that can handle, for example, the mixed indentation since IPython has a transformation logic that not only provides the way to handle the semantic of magics but also can handle things like flexible indentation. However, at the end, the transformation logic that IPython uses translates your code into properly formatted Python (for example removing the leading indentation that isn't supported for the interpreter).

Maybe I'm missing some stuff, and please guys correct me if I'm messing up some where in the explanation, but that's why we not encourage using indented blocks if running from .py files :)

[bcolsen]

@dalthviz The only indentation flexibility that ipython has is if code has an indentation in the first line that indentation is removed from all the other lines. This also isn't corrected in .ipy files because they don't support executing codecells we are tagging that on ourselves.

As @impact27 mentions this, is the solution:

Why? Random indentation is not correct python code

It also not correct in ipy files either. This behavior is actually most important in python files particularly for the code cells in the if __name__ = "__main__" section.

Fair, then we can use leading_indent from IPython.core.inputtransformer2 (PY3 only) when running cells and the full TransformerManager for ipython files.

We could even implement it ourselves:
https://github.com/ipython/ipython/blob/0bc26f65bc775082dc36d89edc1657be031b9e79/IPython/core/inputtransformer2.py#L34

I think that allowing codecells with arbitrary initial indent is fine. I don't think we need to be that strict about it. It will still give plenty of indentation errors:

In [4]:     print("hello")
   ...: print("hello")
hello
hello

In [5]:     print("hello")
   ...: print("hello")
   ...:         print("hello")
  File "<ipython-input-5-ed6671ba29f1>", line 3
    print("hello")
    ^
IndentationError: unexpected indent


In [6]:     print("hello")
   ...:         print("hello")
   ...:         print("hello")
  File "<ipython-input-6-e529d318b1ea>", line 2
    print("hello")
    ^
IndentationError: unexpected indent


In [7]: print("hello")
   ...:     print("hello")
   ...:     print("hello")
  File "<ipython-input-7-e529d318b1ea>", line 2
    print("hello")
    ^
IndentationError: unexpected indent


In [8]:     print("hello")
   ...:     print("hello")
   ...:     print("hello")
hello
hello
hello

In [9]:         print("hello")
   ...:     print("hello")
   ...:     print("hello")
  File "<ipython-input-9-e529d318b1ea>", line 2
    print("hello")
    ^
IndentationError: unexpected indent

[ccordoba12]

I think that allowing codecells with arbitrary initial indent is fine.

I agree.

We could even implement it ourselves

@bcolsen, could you create a PR for this?

[adriesse]

On a related note, I have the impression that Spyder is now saving my file (all changed files?) whenever I run a cell, whereas previously it only did so when running the whole file. As with the previously discussed behavior, it is unclear whether this is by design intent or a side-effect of some other change. I can see the logic in saving other changed files in case there are dependencies, but saving the current file doesn't seem necessary or desirable--at least to me. I have now unchecked the editor/run_code/save_files option and I'll how that goes.

[impact27]

On a related note, I have the impression that Spyder is now saving my file (all changed files?) whenever I run a cell, whereas previously it only did so when running the whole file. As with the previously discussed behavior, it is unclear whether this is by design intent or a side-effect of some other change. I can see the logic in saving other changed files in case there are dependencies, but saving the current file doesn't seem necessary or desirable--at least to me. I have now unchecked the editor/run_code/save_files option and I'll how that goes.

I don't think it should do that. Could you open a new issue for this behaviour?

[impact27]

Fixed by https://github.com/spyder-ide/spyder-kernels/pull/243