Mu: Check for UTF-8 encoding cookie

[tjguk]

This issue is a spin-off from issue #360 / PR #364 where the question arose of what to do if Mu tried to read a file in a different encoding from expected. Currently Mu uses open() without a specific encoding when reading and writing files. Python will call locale.getpreferredencoding() to determining the encoding for the file contents. On Windows this calls the GetLocaleInfo API; presumably it does something equivalent on other platforms.

This will raise a UnicodeEncodeError and lose the text if a file is saved or autosaved which contains a codepoint not valid with that encoding. For example, on my (UK-based) Windows, the default encoding is cp1252 which is essentially a variant of iso-8859-1 / latin-1. Putting μ ("Greek Small Letter Mu" U+03BF) into a document will cause it to crash and burn as soon as the autosave kicks in since that codepoint won't encode in cp1252. [And, yes, the μ character was chosen for maximum εἰρωνεία]

The proposal in #364 is to load and save everything as UTF-8 regardless, but this raises the question of what to do when text is encountered which either does not decode via UTF-8 or which produces mojibake.

I propose here that Mu do the following, similar to Python's own file-opening semantics.

When reading a file:

• When reading a file, check for a valid UTF8/UTF16 BOM. If it is found, decode using the appropriate codec
• If no BOM, see if the first line has a PEP 263 encoding cookie. If it does, decode using the appropriate codec
• If there's no BOM and no encoding cookie, use an unadorned open() which use the underlying locale encoding as it always has done

When saving/autosaving a file:

• Add a UTF-8 encoding cookie if it's not already present
• Encode the file contents as UTF-8

This should cover us for both uses: where a new-style Mu file is found (UTF-8 encoded); and where an old-style Mu file is found (locale-encoded). If it's neither of those, Mu will behave no worse than it did previously: attempting to read with the locale encoding and either failing or producing mojibake if the encodings don't match.

If we go with this I would leave PEP 263 cookie present in the top line. If other proposals go ahead to remove comment lines, eg, for micro:bit, these lines will also be removed.

[tjguk]

Unfortunately, Python's in-built mechanism for reading the encoding cookie and switching encodings is closely enmeshed with the low-level details of the parser (cf https://github.com/python/cpython/blob/master/Parser/tokenizer.c). However, PEP 263 defined the encoding cookie in terms of a fairly simple regex (cf https://www.python.org/dev/peps/pep-0263/#defining-the-encoding) so I don't believe we need to reinvent too many wheels.

[ntoll]

@tjguk thank you so much for this suggestion. As far as I can tell, you've managed three extraordinary things:

• Solved the encoding problem as per #364.
• Done so in a way that is obviously "Pythonic"
• Made it, as much as possible, "backwards safe".

Storming work! @carlosperate can you see any problems with this approach (I don't, but concede I speak from the perspective of a native English speaker who never encounters encoding problems)..?

[tjguk]

I'm not quite clear on what we want to happen if we can't decode in. (We should never have a unicode-related issue saving as UTF-8 which can encode every codepoint). We do have options inbound, eg re-decode as utf-8 with a charref replacement and disable autosave; or re-decode as iso-8859-1 which will decode anything albeit resulting in mojibake; simply give an error dialog and give up.

But I don't know which is the least worst option

[ntoll]

@tjguk I'm not sure either. Here's my assessment of the three options:

1. Re-decode with charref but disable autosave - this tacitly changes Mu's (autosave) behaviour, a potential for an unexpected surprise. If possible we should err for expected fails (if you see what I mean) for specific situations.
2. Re-decode as iso-8859-1 - but the problem is, ������ �� ����� (��� ��) ��� � ���, if you see what I mean. :-P
3. Give up with an error message - this feels the simplest and safest solution to my eyes because:
   
   
   
   • We can explain ourselves ("Mu is unable to read that file.")
   
   
   • We can offer some help ("Please re-save this file in the application in which it was created, but re-save it using UTF-8 encoding"... or something)
   
   
   • We defer the choice of what to do to the user (rather than trying to second guess -- always a bad idea).
   
   

Does this make sense..? I'm all ears for any alternatives if people can think of something better to use.

[carlosperate]

I also think it might be better to go for option 3).

One thing to keep in mind is being able copy/paste into the editor. I think there is an option to force UTF in QScintilla, but I'm not 100% sure how it works.

[tjguk]

Ok; I'm fine with option 3. Do we have "prior art", so to speak, within Mu? Is there anywhere else that we throw up an error which stops the process dead? Just so I can copy the approach.

re the copy/paste: good point; I hadn't thought of it. But I think we can ignore for the purpose of this issue which I'd like to keep to reading/saving a file.

I've got an ad hoc test bed to try the approach out with all its variations, then I'll put a PR together

[ntoll]

Hi @tjguk,

If you look here: https://github.com/mu-editor/mu/blob/master/mu/interface/main.py#L629, you'll find the show_message method of the main window class (the object via which all UI related activity happens. Some of the other methods of this class call show_message as do some of the modes (see, for example, mu/modes/microbit.py) for use cases. Hope this helps. Please don't hesitate to ask for help or tell me I'm an idiot for implementing it in such-and-such a way... ;-)

[yonghuming]

seems i need more cs knowledge to help solve probelems

[tjguk]

@yonghuming your contribution is just as valuable when you try things out, report errors, and help us re-test afterwards.

[ntoll]

What @tjguk said multiplied by 1000000 :-)

[ntoll]

Superseded by work done in #399