Openshot-qt: locale issue & flatpak

[eszlari]

In the flatpak version that I maintain (https://flathub.org/apps/details/org.openshot.OpenShot), I have to set LC_ALL=en_US.UTF-8 as a work around to make file importing work (see build manifest: https://github.com/flathub/org.openshot.OpenShot/blob/master/org.openshot.OpenShot.yaml).

Obviously, this should not be needed - there is a bug somewhere in OpenShot related to Unicode.

[DylanC]

@ferdnyc - Would this be related to the locale changes I merged in? What do you think?

[ferdnyc]

Entirely possible, but honestly I don't understand the workaround, or the issue, to begin with. What do you mean by "file import", @eszlari ? Media file loading? Strange that that would be affected, but not impossible.

And what do you mean by "make work"? How does it not "work" _without_ this workaround? Provide a real bug report, please, not details of how you worked around it externally. We can't fix a problem we don't have any information on.

[eszlari]

Without this workaround, when I try to import any file, I get the error message:

file.mp4 is not a valid video, audio, or image file.

[ferdnyc]

Huh, thanks @eszlari. Very strange. Some questions:

1. Import how? Drag-and-drop, File > Import Files... menu, Import Files... from the Project Files context menu? Or, doesn't matter?
2. Are there any non-ASCII characters anywhere in the file path? Does import work if there _aren't_?
3. Was there a flatpak build of OpenShot 2.4.2? Did it have the same import issues, or are they new in 2.4.3?
4. Have you tried the 2.4.3 AppImage on the same system? Any issues with that?

[ferdnyc]

1. Import how? Drag-and-drop, File > Import Files... menu, Import Files... from the Project Files context menu? Or, doesn't matter?

(Note that the latter two of those are the same code path, so there's very little possibility they would be different. It's really drag-and-drop vs. file browser, since we've had issues with differences in those two operations in the past. Though, previously it was only an issue on Windows, due to the UTF-16 filesystem encoding.)

[ferdnyc]

1. Are there any non-ASCII characters anywhere in the file path? Does import work if there _aren't_?

(Copying a file to, say, /var/tmp/file.mp4 and then trying to import it would be a good way to test that, if there are non-ASCII characters in your normal home path.)

[ferdnyc]

Hmm. When I go to install OpenShot from flathub, I get this:

$ flatpak install flathub org.openshot.OpenShot
Required runtime for org.openshot.OpenShot/x86_64/stable (runtime/org.kde.Platform/x86_64/5.11) found in remote flathub
Do you want to install it? [y/n]: y
Installing in system:
org.kde.Platform/x86_64/5.11                         flathub cedcfa68b840
org.freedesktop.Platform.GL.nvidia-396-54/x86_64/1.4 flathub 8c138b8c1a4d
org.freedesktop.Platform.ffmpeg/x86_64/1.6           flathub d757f762489e
org.kde.Platform.Locale/x86_64/5.11                  flathub fcd52429b2ac
org.openshot.OpenShot/x86_64/stable                  flathub d1a34bf22618
  permissions: ipc, network, pulseaudio, x11, dri
  file access: host

I assume the org.kde.Platform and org.kde.Platform.Locale are due to OpenShot using Qt, but I have to wonder if that's somehow involved?

Well, I'll do the install (if I have enough disk space for all that) and see if I can't get a better handle on this. So far I haven't been able to reproduce from either the Fedora package or the AppImage, using different combinations of locale env var settings.

[ferdnyc]

So far I still haven't been able to reproduce, whether I edit /var/lib/flatpak/app/org.openshot.OpenShot/x86_64/stable/d1a34bf22618f80956d74b8986d5c09b34acc318b3c4c50b3aa804333a165694/metadata to include LC_ALL=C in the [Environment] section, or if I remove the line entirely. But, I'm in an en_US.UTF-8 locale anyway so that's hardly surprising.

Here's one other test that might be helpful, @eszlari : When you have OpenShot running from Flatpak so that import _is_ failing to work, open a terminal and get the PID of the openshot-qt process. You're looking for the bubblewrap wrapper it's running in, so something like:

$ ps -ef |grep bwrap
ferd     27473  7721  0 01:40 pts/11   00:00:00 bwrap --args 20 openshot-qt
ferd     27479  1747  0 01:40 pts/11   00:00:00 bwrap --args 20 /usr/libexec/flatpak-dbus-proxy --args=25
ferd     27482 27473  0 01:40 pts/11   00:00:00 bwrap --args 20 openshot-qt
ferd     27742 31553  0 01:41 pts/3    00:00:00 grep bwrap

You'll want the child PID, of the two of them — the one that has the other as its parent. So, in my example that's 27482.

Then, take that PID, and as root, use flatpak enter to grab the environment of the running process:

$ sudo flatpak enter 27482 cat /proc/self/environ | xargs -0 -n1 echo | sort |egrep '(LANG|LC_)'
GDM_LANG=en_US.utf8
LANG=en_US.utf8
LC_ALL=C
LC_MEASUREMENT=en_US.UTF-8
LC_MONETARY=en_US.UTF-8
LC_NUMERIC=en_US.UTF-8
LC_PAPER=en_US.UTF-8
LC_TIME=en_US.UTF-8

To @DylanC 's question, looking at this I would be very surprised if it has anything to do with the translation code changes. It _seems_ like it's just a Flatpak environment issue that we haven't encountered before.

[eszlari]

You don't need to edit any file, just start openshot with:

$ flatpak run --env=LC_ALL=C org.openshot.OpenShot

You can also run a shell inside the sandbox:

$ flatpak run --command=sh org.openshot.OpenShot

[ferdnyc]

You don't need to edit any file, just start openshot with:

$ flatpak run --env=LC_ALL=C org.openshot.OpenShot

Same thing, file import works fine for me.

Please answer as many of the questions in this comment as you can, and collect the LANG/LC_ environment variables from an OpenShot process that's experiencing file-import failures, because so far I am not able to reproduce.

[eszlari]

1. Import how? Drag-and-drop, File > Import Files... menu, Import Files... from the Project Files context menu? Or, doesn't matter?

Doesn't matter.

1. Are there any non-ASCII characters anywhere in the file path? Does import work if there aren't?

Without setting LC_ALL no files can be imported.

1. Was there a flatpak build of OpenShot 2.4.2? Did it have the same import issues, or are they new in 2.4.3?

The first flatpak version was 2.4.1 and it didn't work from the beginning.

1. Have you tried the 2.4.3 AppImage on the same system? Any issues with that?

No issues.

Same thing, file import works fine for me.

Have you tried another loacle setting on your host system? The locale of my host system is de_DE.UTF-8 and the one from the original bug report is probably es_ES.UTF-8:

https://github.com/flathub/org.openshot.OpenShot/issues/2

[ferdnyc]

Have you tried another loacle setting on your host system?

I've tried LC_ALL=C, and had no problems there. I haven't tried reconfiguring my system into a language I don't speak, no.

We know that OpenShot functions in non-en_US locales. The 2.4.3 AppImage worked fine on your system, right? So, this is a Flatpak thing. And not necessarily a universal Flatpak thing, even, but possibly a Flatpak host environment (distro) thing.

It's possible the issue here is how Flatpak's environment is being set up in Fedora vs. whatever distro you're running (I assume not Fedora?). Because on my (Fedora 28) system, Flatpak is passing all of the necessary locale variables through into OpenShot's sandbox:

$ sudo flatpak enter 27482 cat /proc/self/environ | xargs -0 -n1 echo | sort |egrep '(LANG|LC_)'
GDM_LANG=en_US.utf8
LANG=en_US.utf8
LC_ALL=C
LC_MEASUREMENT=en_US.UTF-8
LC_MONETARY=en_US.UTF-8
LC_NUMERIC=en_US.UTF-8
LC_PAPER=en_US.UTF-8
LC_TIME=en_US.UTF-8

(That was when I'd configured the app with LC_ALL=C, to match the issue you'd originally reported. If I leave LC_ALL out entirely, though, everything's still fine because all of the other ones are set and passed through correctly.)

The locale of my host system is de_DE.UTF-8 and the one from the original bug report is probably es_ES.UTF-8:

Which may be a factor, but it's only a factor if Flatpak is even passing that information through to the sandbox. If it were and doing it correctly, though, I'd expect OpenShot to have no problems — because that's exactly the situation when running from the AppImage. Or an installed build (PPA, RPM, etc.) OpenShot runs fine in a de_DE.UTF-8 or es_ES.UTF-8 locale, under any of _those_ conditions.

So, the question is still the same: What's _different_ inside of your Flatpak environment? I'd still really like to see the results of this test, if at all possible:

Here's one other test that might be helpful, @eszlari : When you have OpenShot running from Flatpak so that import _is_ failing to work, open a terminal and get the PID of the openshot-qt process. You're looking for the bubblewrap wrapper it's running in, so something like:

$ ps -ef |grep bwrap
ferd     27473  7721  0 01:40 pts/11   00:00:00 bwrap --args 20 openshot-qt
ferd     27479  1747  0 01:40 pts/11   00:00:00 bwrap --args 20 /usr/libexec/flatpak-dbus-proxy --args=25
ferd     27482 27473  0 01:40 pts/11   00:00:00 bwrap --args 20 openshot-qt
ferd     27742 31553  0 01:41 pts/3    00:00:00 grep bwrap

You'll want the child PID, of the two of them — the one that has the other as its parent. So, in my example that's 27482.

Then, take that PID, and as root, use flatpak enter to grab the environment of the running process:

$ sudo flatpak enter 27482 cat /proc/self/environ |xargs -0 -n1 echo |egrep '(LANG|LC_)'

...Then I'll know the locale details of your Flatpak sandbox environment, and I can try to reproduce that here.

[eszlari]

The output of $ env | egrep '(LANG|LC_)' is the same inside and outside the flatpak sandbox:

LANG=de_DE.UTF-8
LANGUAGE=de:en_US

So, this is a Flatpak thing.

OpenShot is the only app I encountered with this behavior - and I maintain and test quite a lot flatpak apps.

[ferdnyc]

The output of $ env | egrep '(LANG|LC_)' is the same inside and outside the flatpak sandbox:

LANG=de_DE.UTF-8
LANGUAGE=de:en_US

...How is that possible? I thought you were setting LC_ALL in the Flatpak environment? Isn't that what this whole bug report was _about_? Whatever, doesn't matter.

So, this is a Flatpak thing.

OpenShot is the only app I encountered with this behavior - and I maintain and test quite a lot flatpak apps.

Cool. You've been lucky.

I have successfully reproduced this bug. It does not _technically_ depend on being inside the Flatpak environment, simply on having an invalid locale setup (a problem rarely encountered under normal circumstances, but one that Flatpak's environment seems almost _engineered_ to trigger, based on @eszlari 's feedback so far). It does not have _anything_ to do with interface language, and in fact the bug can easily be triggered even if the language is set to US English.

What we're seeing here is an instance of this Python bug, where os.path.exists() can raise an exception (instead of simply returning false) when asked to verify paths with non-ASCII characters in them (even if those paths are _valid_), if the locale is improperly configured. LC_ALL=C represents one such invalid configuration, from the standpoint of Unicode file paths.

Steps to reproduce

1. Create a directory with a non-ascii character in the name, and place a media file inside. I called mine $HOME/Tést/media.mp4.
2. In a shell, run LC_ALL=C openshot-qt
3. Right-click on the Project Files pane, select "Import Files...", navigate to $HOME/Tést/media.mp4 and open it.
4. Notice that _that_ import was successful. However, the seeds of your discontent have already been sown. They lie in this message, logged during the import:
   project_data:INFO _set key: ['import_path'] values: /home/$USER/T\xe9st add: False partial: False remove: False
5. Attempt to import another file using the same right-click, "Import Files..." method.
6. Instead of showing a file selector, OpenShot will log the following exception:
   exceptions:ERROR Unhandled Exception Traceback (most recent call last): File "/home/$USER/.local/lib/python3.6/site-packages/openshot_qt-2.4.3-py3.6.egg/openshot_qt/windows/main_window.py", line 590, in actionImportFiles_trigger if not recommended_path or not os.path.exists(recommended_path): File "/usr/lib64/python3.6/genericpath.py", line 19, in exists os.stat(path) UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 12: ordinal not in range(128)

As soon as the import_path tracking variable is filled with a string (path) containing non-ASCII characters, in an OpenShot instance where Python is treating file paths as exclusively ASCII-based, importing breaks because os.path.exists() starts doing nonsense things like throwing exceptions if we pass that (perfectly valid) path back _into_ it. (This despite the fact that the string was generated by calling os.path.dirname(file_path) on the imported file path, so it's not like os.path hadn't already _seen_ the non-ASCII character, and in fact returned it decoded to "\xe9" before somehow forgetting how to encode it again.)

Still, like I said, you almost have to _try_ to trigger this bug. Or, at least, I had to try pretty hard to make it happen. Forcing LC_ALL=C in a Flatpak environment will do it. But, in _my_ Flatpak environment, simply not setting LC_ALL at all will NOT trigger it, because in the absence of LC_ALL overrides the locale inside my Flatpak runtime is correctly set:

$ flatpak run --command=locale org.openshot.OpenShot
LANG=en_US.utf8
LC_CTYPE="en_US.utf8"
LC_NUMERIC=en_US.UTF-8
LC_TIME=en_US.UTF-8
LC_COLLATE="en_US.utf8"
LC_MONETARY=en_US.UTF-8
LC_MESSAGES="en_US.utf8"
LC_PAPER=en_US.UTF-8
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT=en_US.UTF-8
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=

$ flatpak run org.openshot.OpenShot  # This will be fine, and not subject to this bug

It's only if I set LC_ALL that I get myself into trouble:

$ flatpak run --env=LC_ALL=C --command=locale org.openshot.OpenShot
LANG=en_US.utf8
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C

$ flatpak run --env=LC_ALL=C org.openshot.OpenShot  # NOW I'll encounter the bug

So, I'll ask, @eszlari : You said that switching the environment for org.openshot.OpenShot from LC_ALL=C to LC_ALL=en_US.UTF-8 cleared up the bug. But, why were you setting LC_ALL in the first place? And, what happens if you don't set it in the Flatpak configuration for org.openshot.OpenShot, AT ALL? Not to C, not to en_US.UTF-8, not to anything? Because, that _should_ work, and overriding LC_ALL would seem to be the initial cause of the problem.

If removing the LC_ALL setting doesn't clear up the problem, then I would expect that, unlike my own environment (as you can see above), you'd get something like this output even in the _absence_ of LC_ALL (the output I only get _with_ it set to LC_ALL=C) — and that means the locale was never really set up right in the first place:

$ flatpak run --command=locale org.openshot.OpenShot
LANG=de_DE.UTF-8
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

$ flatpak run org.openshot.OpenShot  # We don't have a valid locale. I don't know WHAT will happen.

If that's the case, we'll have to figure out what to do about this, since I assume sorting out the Flatpak environment to not break things isn't an option. I suppose we could wrap that os.path.exists() call in a try block, inside main_window.py, and just ignore the "bad" import_path project setting if it's triggering this bug. But that's an awfully broken "fix" to be making.

[ferdnyc]

Oh, by the by, the reason we can't simply do what the OP of that stack question indicated as their workaround, and force-encode the argument — os.path.exists(recommended_path.encode('UTF-8')) — is that, as someone else pointed out, that's not portable. Windows paths are UTF-16.

Actually that _might_ work, because Python's auto-conversion might save us from ourselves. But even if it worked, it seems like a bad idea.

[ferdnyc]

Cool. You've been lucky.

Hah! I _swear_ I was just cruising around, "checking out the competition", and by pure coincidence I happened to spot this post highlighted in the "Recent Forum Posts" widget on the kdenlive site:

Kdenlive flatpak bad locale

[eszlari]

How is that possible? I thought you were setting LC_ALL in the Flatpak environment? Isn't that what this whole bug report was about?

I gave you the values without the workaround!

And, what happens if you don't set it in the Flatpak configuration for org.openshot.OpenShot, AT ALL? Not to C, not to en_US.UTF-8, not to anything?

As I already said: In that case I can't import any file!

• LC_ALL='' -> no files can be imported
• LC_ALL=C -> ASCII files can be imported
• LC_ALL=en_US.UTF-8 -> all files can be imported

Hah! I swear I was just cruising around, "checking out the competition", and by pure coincidence I happened to spot this post highlighted in the "Recent Forum Posts" widget on the kdenlive site:

I'm also the maintainer of the kdenlive flatpak and I never encountered such a problem with it. I'm not even sure, what problem this user is experiencing.

[ferdnyc]

How is that possible? I thought you were setting LC_ALL in the Flatpak environment? Isn't that what this whole bug report was about?

I gave you the values without the workaround!

Gotcha, sorry. We came into this halfway through, remember, and I've had to work my way backwards to the initial symptoms. The change you pointed us at was the one that changed LC_ALL=C to LC_ALL=en_US.UTF-8 in the flatpak manifest.

So IIUC, you'd _previously_ added LC_ALL=C as a first workaround, because...

And, what happens if you don't set it in the Flatpak configuration for org.openshot.OpenShot, AT ALL? Not to C, not to en_US.UTF-8, not to anything?

As I already said: In that case I can't import any file!

OK, hopefully I'm fully up to speed now.

So based on all that, I really need to see logs from OpenShot in the _original_ failure scenario. (LC_ALL not set, import fails to function.) The only OpenShot logs I've seen related to this bug (besides my own) were the ones from your user's bug report, after the addition of LC_ALL=C but before the change to LC_ALL=en_US.UTF-8. Those were consistent with the behavior I've reproduced when LC_ALL is set to C, which I thought meant I was getting somewhere. Unfortunately I guess the only thing I've confirmed/reproduced is the ineffectiveness of LC_ALL=C as a workaround.

Whatever problem you're seeing where no imports work, and LC_ALL isn't set, I cannot reproduce. On my system under normal circumstances, OpenShot is fine in that scenario. But on my system, inside the Flatpak environment, the LC_ variables are set to en_US.UTF-8 because they inherit from my user environment. Whereas you've reported that none of them are set inside the Flatpak environment, on your system.

So, I guess please provide logs of an OpenShot session under that scenario, including a failed import attempt. I'm really at a loss here.

Oh, and also include the output of these commands, thanks:

$ flatpak run --command=locale org.openshot.OpenShot

$ flatpak run --command=python3 org.openshot.OpenShot # This will start an interactive session
>>> import locale
>>> locale.getdefaultlocale()
>>> locale.sys.getfilesystemencoding()
>>> exit()

[ferdnyc]

OK, so turns out this is a Python issue. A _very_ well-known Python issue, with container/sandboxed environments. The entirety of PEP538 is devoted to the topic of this exact issue, and how Python is going to address it.

OpenShot is doing nothing unusual or wrong, nor does it do anything different inside the Flatpak runtime than outside it. But it's being thrown into an environment that is not fully configured for locale support, and Python's encoding conversion is breaking. PEP538 goes into great detail about all the ways this can happen.

Hopefully the further logging/information I've asked @eszlari for will shed some light on what, if anything, we can do to unfuck ourselves from the mess Ubuntu's flatpak environment is making. Because, I am so far down #WORKSFORME creek at this point... :angry:

[ferdnyc]

OK, having done a _lot_ more reading on this...

It is definitely a general python issue.

There are no less than _three_ PEPs dealing with the issue of locale handling in Python, especially as it relates to filesystem encoding (where it can cause the most trouble, and the aspect that's biting us). PEP383: Non-decodable Bytes in System Character Interfaces is where things stand with the flatpak runtime. Then there's PEP538: Coercing the legacy C locale to a UTF-8 based locale and PEP540: Add a new UTF-8 Mode, two separate proposals for addressing two different aspects of this issue. Both of those are adopted and implemented as of Python 3.7.

The reason things JFW for me on Fedora under the default Python 3.6 is that Fedora backported the proposed PEP538 (and possibly PEP540?) fixes into Python 3.6. Unfortunately, the Flatpak environment is only using Python 3.5, which is why I can easily reproduce this bug in Flatpak — but only in Flatpak. If the Flatpak environment were at Python 3.7, I believe this issue would no longer exist.

It is nearly impossible to break encoding in Python 3.7. It is hard in Fedora's Python 3.6, thanks to the backported fix. It is very, very easy in Python 3.5.

Since these commands are too wide for the window, they all execute the same python code:

import locale
print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding())
import os
print(os.listdir('/home/ferd/testing'))
````

With all of my locale-related variables forcibly cleared, here's Python 3.5's handling...
```console
$ env|egrep '(LC_|LANG)' 
$ LANG=en_US.UTF-8 ls /home/ferd/testing/ # There's a lone non-ASCII file in this dir
Tést
$ LANG=en_US.UTF-8 python3.5 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']
$ # The pt_BR locale isn't installed on my system
$ LANG=pt_BR.UTF-8 python3.5 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('pt_BR', 'UTF-8') (None, None) ascii
['T\udcc3\udca9st']
$ # The de_DE locale IS installed — before it was, this also came out ascii
$ LANG=de_DE.UTF-8 python3.5 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('de_DE', 'UTF-8') ('de_DE', 'UTF-8') utf-8
['Tést']

But now watch what happens when I start modifying other variables:

$ LANG=de_DE.UTF-8 LC_CTYPE=C python3.5 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
(None, None) (None, None) ascii
['T\udcc3\udca9st']
$ LANG=en_US.UTF-8 LC_ALL=C python3.5 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
(None, None) (None, None) ascii
['T\udcc3\udca9st']
$ LANG=pt_BR.UTF-8 LC_ALL=C.UTF-8 python3.5 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']
$ LANG=C.UTF-8 python3.5 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']
$ LANG=C.UTF-8 LC_ALL=C python3.5 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
(None, None) (None, None) ascii
['T\udcc3\udca9st']
$ LANG=C python3.5 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
(None, None) (None, None) ascii
['T\udcc3\udca9st']
$ LANG=C.UTF-8 LC_ALL=C.UTF-8 python3.5 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']

In Python 3.6, though, because of Fedora's fixes, it's much more solid...

$ LANG=de_DE.UTF-8  python3.6 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('de_DE', 'UTF-8') ('de_DE', 'UTF-8') utf-8
['Tést']
$ LANG=pt_BR.UTF-8 python3.6 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']
$ LANG=de_DE.UTF-8 LC_CTYPE=C python3.6 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']
$ LANG=de_DE.UTF-8 LC_ALL=C python3.6 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
(None, None) (None, None) ascii
['T\udcc3\udca9st']
$ LANG=pt_BR.UTF-8 LC_ALL=C.UTF-8 python3.6 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']
$ LANG=C python3.6 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']
$ LANG=C LC_ALL=C python3.6 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
(None, None) (None, None) ascii
['T\udcc3\udca9st']

And in Python 3.7...

$ LANG=pt_BR.UTF-8 python3.7 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']
$ LANG=pt_BR.UTF-8 LC_CTYPE=C python3.7 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']
$ LANG=pt_BR.UTF-8 LC_ALL=C python3.7 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
(None, None) (None, None) utf-8
['Tést']
$ LANG=C.UTF-8 LC_ALL=C python3.7 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
(None, None) (None, None) utf-8
['Tést']
$ LANG=C LC_ALL=C python3.7 -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
(None, None) (None, None) utf-8
['Tést']

No matter _what_ you set, Python 3.7 refuses to break the filesystem encoding.

But we're not dealing with Python 3.7. Inside the Flatpak environment, it's only Python 3.5. And for reasons I don't understand, things are even worse there than with my system Python 3.5.

$ flatpak run --env=LANG=de_DE.UTF-8 --command=python3 org.openshot.OpenShot -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('de_DE', 'UTF-8') (None, None) ascii
['T\udcc3\udca9st']
$ flatpak run --env=LANG=de_DE.UTF-8 --env=LC_ALL=de_DE.UTF-8 --command=python3 org.openshot.OpenShot -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('de_DE', 'UTF-8') (None, None) ascii
['T\udcc3\udca9st']
$ flatpak run --env=LANG=en_US.UTF-8 --env=LC_ALL=C.UTF-8 --command=python3 org.openshot.OpenShot -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') (None, None) ascii
['T\udcc3\udca9st']
$ flatpak run --env=LANG=en_US.UTF-8 --command=python3 org.openshot.OpenShot -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']
$ # This next one's SUPER weird — note that the default locale is reset to en_US!
$ flatpak run --env=LANG=de_DE.UTF-8 --env=LC_CTYPE=C.UTF-8 --command=python3 org.openshot.OpenShot -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') (None, None) ascii
['T\udcc3\udca9st']
$ flatpak run --env=LANG=de_DE.UTF-8 --env=LC_ALL=en_US.UTF-8 --command=python3 org.openshot.OpenShot -c "import locale; print(locale.getdefaultlocale(), locale.getlocale(), locale.sys.getfilesystemencoding()); import os; print(os.listdir('/home/ferd/testing'));"
('en_US', 'UTF-8') ('en_US', 'UTF-8') utf-8
['Tést']

When LANG=de_DE.UTF-8, even though that's an installed language on my system (now), the _only_ thing that will fix the Python filesystem encoding inside the Flatpak runtime is LC_ALL=en_US.UTF-8. Even things that _should_ work in Python 3.5, like LC_CTYPE=C.UTF-8 or LC_ALL=C.UTF-8, don't work inside Flatpak — in fact, they _break_ the encoding even when LANG=en_US.UTF-8!

And note that _all_ of these commands are direct to the Python3 interpreter, they don't involve the OpenShot application at all.

[eszlari]

The next version of the flatpak KDE runtime will include python 3.7 and therefore solve this problem.

[ferdnyc]

Now _that_ is the best news I've heard all week!

[DylanC]

@ferdnyc @eszlari - I've been following the developing conversations in this for quite some time. I feel its a pretty good time to ask... can we close it now? xD

[ferdnyc]

I mean... honestly, I'd say no? That's my take, anyway, though others may disagree.

Corrupted filesystem encoding under Flatpak is still very much an issue for users on some systems, and we haven't come up with any sort of fix, kludge, or even workaround. (The current LC_ALL=en_US.UTF-8 is _not_ a solution, in fact it's a horrible idea. But it's also the ONLY thing that seems to work, right now. So, at best, it's the slightly lesser of two very great evils.)

I don't know that there _will_ be any sort of genuine solution to this issue, at least not on the OpenShot side. Other than just waiting on Python3.7 in the hopes that it'll sort all this out for us. But if that's the case, it'll still remain an issue for OpenShot (and OpenShot's Flatpak users) until the Python update happens. (BTW, is there any sort of timeframe on that, @eszlari ?)

I mean, my take is: The discussion has probably run its course. I know that I don't really have anything more to add. Whether the issue stays open pending the emergence of Python3.7 into Flatpak I don't have any strong opinions about, though my personal inclination would be to leave it.)

[DylanC]

@ferdnyc - Alright, I'll wait some more until Python is updated for Flatpak. 👍

[eszlari]

The next KDE runtime will be released, when Qt 5.12 is released:

https://wiki.qt.io/Qt_5.12_Release

[eszlari]

The OpenShot flatpak has been updated to the KDE runtime 5.12 (with python 3.7) and the workaround has been removed:

https://github.com/flathub/org.openshot.OpenShot/commit/453ccdb945aef199b56422f7210d94fbe0482ba1