Spyder: White space U+000C FORM FEED incorrectly removed by Spyder text editor

[bignose-debian]

The U+000C (FORM FEED) character is valid white space in text, including Python source code.

What steps will reproduce the problem?

• Create a text file foo.txt containing several lines of text, including one line containing only a U+000C character.

• Open the file foo.txt in Spyder.

  Spyder will raise a modal dialogue “foo.txt contains mixed end-of-line characters. Spyder will fix this automatically.”

• The U+000C characters have been removed.

What is the expected output? What do you see instead?

Actual: The text is altered, by removing the valid characters.

Expected: the U+000C character should be treated as valid vertical white space, no message, and left unchanged.

The non-printable U+000C is typically shown in text editors by a visible glyph (e.g. “^L” or “↡”) in a different colour.

Please provide any additional information below

“The form feed character is sometimes used in plain text files of source code as a delimiter for a page break, or as marker for sections of code.” — https://en.wikipedia.org/wiki/Page_break#Form_feed

“A logical line that contains only spaces, tabs, formfeeds and possibly a comment, is ignored” — https://docs.python.org/3/reference/lexical_analysis.html#blank-lines

Version and main components

• Spyder Version: 3.2.4
• Python Version: 3.6.4
• Qt Versions: 5.9.2, PyQt5 5.9.2 on Linux

Dependencies

pyflakes >=0.6.0 :  1.6.0 (OK)
pycodestyle >=2.3:  2.3.1 (OK)
pygments >=2.0   :  2.2.0 (OK)
pandas >=0.13.1  :  0.20.3 (OK)
numpy >=1.7      :  1.13.3 (OK)
sphinx >=0.6.6   :  1.6.5 (OK)
rope >=0.9.4     :  0.10.5 (OK)
jedi >=0.9.0     :  0.11.0 (OK)
psutil >=0.3     :  5.4.2 (OK)
nbconvert >=4.0  :  5.3.1 (OK)
sympy >=0.7.3    :  1.1.1 (OK)
cython >=0.21    :  None (NOK)
qtconsole >=4.2.0:  4.3.1 (OK)
IPython >=4.0    :  6.2.1 (OK)
pylint >=0.25    :  1.7.4 (OK)

[CAM-Gerlach]

@ccordoba12 Is this the intended behavior (i.e. a wontfix), or should it be considered a bug?

[ccordoba12]

Won't fix is fine. @bignose-debian, sorry, we don't have time to deal with this. I'm sure other editors are able to do it.

[bignose-debian]

On 16-Jan-2018, Carlos Cordoba wrote:

Won't fix is fine. @bignose-debian, sorry, we don't have time to
deal with this. I'm sure other editors are able to do it.

Could you re-open this issue, allowing for someone to find and fix the
bug?

--
\ “Ours is a world where people don't know what they want and are |
`\ willing to go through hell to get it.” —Donald Robert Perry |
_o__) Marquis |
Ben Finney [e-mail]

[CAM-Gerlach]

A wontfix generally means what the reporter considers a bug, the development team considers the intentional behavior for the majority of users; ergo, from that perspective there is nothing to be fixed. In this case, I can't comment on @ccordoba12 's exact reasoning, but it seems part of it may be due to that this may be the desired behavior, and some of it just being the fact that we currently have very limited resources due to being unfunded at the moment (though you can help with that). With that in mind, even if it were considered a bug, it would be unlikely to get fixed soon, whether it was open or not.

That being said, if you have clear reasoning behind considering it a bug and would like to see it fixed, you are welcome to submit a pull request (against 3.x) with the changes that would implement this and the rationale behind doing so, and we can hopefully at least consider it. Spyder is developed using Spyder and in the same Python language the code you run in it is, so it should be too difficult if you search for the text in the modal dialog and trace the logic from there.

[jitseniesen]

In my opinion this is a bug. A formfeed character is not an end-of-line character. However, it is a very low priority issue for me because I don't recall seeing formfeeds in Python source files.

I thought that closing as wontfix means we (as developers) don't want to fix it. If we would consider pull requests, I think the bug should be kept open and perhaps given an appropriate label as wishlist (which does not exist) or help wanted (which does exist but is perhaps not so clear).

[CAM-Gerlach]

I thought that closing as wontfix means we (as developers) don't want to fix it.

I've seen different definitions, but yeah I'd trust yours as probably a lot closer to the consensus than what I all too authoritatively mentioned previously. What I was thinking was probably something closer to Invalid, i.e. the behavior reported is, for a variety of reasons, not a bug in Spyder or anything else not under the users control (if it was, that would be NotSpyder, or (as I too frequently have to use it), for issues that contain so little useful information as to be impossible to classify as anything else.

I think the bug should be kept open

Agreed here; there seems no reason why not if there is any chance someone (e.,g. @bignose-debian ) wants to fix it. My naive guess is that it wouldn't be too difficult to do either. I was just unsure with my (mis)understanding of wontfix and @ccordoba12 's response.

label as wishlist (which does not exist)

It does, just as a milestone.

[CAM-Gerlach]

I could even create a Severity 5: Trivial tag for issues like this, but I haven't been regularly assigning those tags and leaving that to more knowledgeable people usually, so I'm not sure if its worth it.

[ccordoba12]

Could you re-open this issue, allowing for someone to find and fix the bug?

I don't see the need to keep issues open for the sake of doing it, because someone is usually one of us, and we don't have time for this.

[CAM-Gerlach]

However, @bignose-debian , if that _someone_ is you, it shouldn't be too hard to fix since you already know Python. Take a look at the relevant functions (near the top) in spyder/utils/sourcecode.py as well as the actual call in spyder/widgets/editor.py.

Also, if it bothers you, you can always disable the behavior under Preferences > Editor > Advanced Settings >Fix Automatically..., or just use a more modern way of breaking up your source code, like with cells (# %%).

[bignose-debian]

I have implemented a fix for this issue, in the branch at https://salsa.debian.org/bignose/spyder/commits/wip/issue/6215/correct-eol-detection which should merge cleanly to master as of commit ab216333d8a0fb4a.

That includes test cases, though I am not overly familiar with Pytest so you could probably improve them :-)

[CAM-Gerlach]

@bignose-debian Thanks, great to hear! If you submit a PR (against 3.x, since its a small bug fix) we'll try to review it as soon as we can.

[bignose-debian]

Thanks, great to hear!

Thank you :-)

If you submit a PR

I don't maintain code at GitHub (because Git is a fully distributed version control system), and I am pretty sure GitHub does not work for pull requests from repositories elsewhere. Sorry! Please use Git's distributed powers to pull from that branch.

[ccordoba12]

I don't maintain code at GitHub (because Git is a fully distributed version control system), and I am pretty sure GitHub does not work for pull requests from repositories elsewhere.

Sorry, we're pretty swamped and don't have time to do your job. If you want your work integrated in Spyder, please open a pull request like everybody else.

[CAM-Gerlach]

Closed due to lack of response. Should you decide that actually getting your change implemented by cooperating with the straightforward procedure used on our project (and many, many others) is more important to you than being an uncompromising git purist, then please feel free to reopen :-)