Asbru-cm: UTF8 characters not showing properly

[paliasp]

Describe the bug
UTF8 characters, either letters from non-latin language, or special characters are not displayed properly.
From other clients (SecureCRT, Putty, or openssh client) it works fine.
All configuration options are set to UTF8 where applicable. Terminal emulation is xterm.
Tried to rename the ~/.config/pac directory to force a clean start, but no joy.
If I start a local shell in Asbru and then connect to the remote host, all characters are displayed properly, as you can see in the screenshot.

To Reproduce
Steps to reproduce the behavior:

1. Start a new connection
2. Use defaults
3. You cannot see non latin characters properly.

Expected behavior
To see the correct output as with other clients.

Screenshots
Side to side with the same connection started from local shell, which works fine.
https://flic.kr/p/2iima5x

Desktop (please complete the following information):

• OS: Ubuntu
• Version 18.04.4

Additional context

[egmontkob]

I'm puzzled.

The faulty screenshot looks exactly as if the terminal interpreted the UTF-8 data stream as ISO-8859-1 (Latin-1). Even in the upper row, the last byte of the greek "Τετ" is 0x84 which is IND a.k.a. Index (kind of a cursor down).

VTE has defaulted to UTF-8 for many years, and even the escape sequence that switches between encodings was removed many years ago, waaay before 18.04.

The only way to alter its encoding is the set_encoding() API call, which Ásbrú would then have to call with ISO-8859-1 as parameter, and I see no sign of this in Ásbrú, it also defaults to UTF-8.

Putting all these together, I can't see how VTE in Ásbrú could work in ISO-8859-1, unless you explicitly set this charset (which you don't), or I really missed something.

Now, I have a weird theory.

ssh forwards some selected environment variables, including locale-related ones (see SendEnv LANG LC_* or so in /etc/ssh/ssh_config).

When starting up ssh as the immediate child, without the user's shell, the shell startup filles are skipped. If they set some locale-related env variables, these will be different in the two cases.

As such, the remote side also sees a difference.

Could it be that the terminal operates in UTF-8, and the remote side, due to some misconfiguration, maybe with luit involved, combined with the weird environment variables (or lack thereof) that it receives, sends "double UTF-8"???

[paliasp]

I am not sure if I understood properly your theory, however the /etc/ssh/ssh_config had SendEnv LANG LC_* for every host. I changed it to SendEnv LANG en_GB.utf8 but didn't make a difference. :(

[egmontkob]

however the /etc/ssh/ssh_config had SendEnv LANG LC_* for every host

This is the client config, the file only matters on the host where you ssh _from_.

I changed it to SendEnv LANG en_GB.utf8

Don't! It's not "SendEnv name value", but "SendEnv name1 name2 name3 ...".

My point isn't to change this, but to examine which values actually get sent and actually arrive at the remote host. Although removing all the entries from the list and checking the behavior (both when ssh'ing immediately, which does not work, and when executing ssh from your shell, which works correctly now) might help in debugging.

I am not sure if I understood properly your theory

My theory is that _maybe_ the remote host behaves incorrectly (potentially related to the LANG and/or LC_whatever environment variables being set differently), and sends exactly the garbage-looking data to be displayed, which then Ásbrú displays perfectly.

[paliasp]

I tried to comment out this line, but the result is the same. Directly the characters are malformed and when connecting from the shell they look fine.

I noticed something else that might matter. Other unicode characters are showing correctly, for example if I run the command ls -la:

drwxr-xr-x 7 dietpi dietpi 4,0K Φεβ  27 08:17 .
drwxr-xr-x 5 root   root   4,0K Νοε  23 22:04 ..

The months are printed correctly. So this makes me believe that something might be wrong with the startup script of dietpi.

[gfrenoy]

What is the encoding of your motd file on the target machine ? Would you mind sharing it ? I'm not able to reproduce this :-/

[gfrenoy]

Ok I was able to reproduce this ! Simply adding some weird UTF8 characters in my motd did it.

Not yet sure how to fix this though. It looks like it has something to do with the Expect module that is not managing UTF8 properly.

[egmontkob]

It looks like it has something to do with the Expect module that is not managing UTF8 properly.

Great point, I totally forgot that Ásbrú doesn't run ssh directly, but via a pac_conn wrapper. This must be the culprit.

Perl + UTF-8 problem asks for @hanspr's help :)

[gfrenoy]

The last commit fixes the issue (thanks @hanspr ;p). It will be available in the latest snapshot in a few minutes.

Thanks for reporting it !

[paliasp]

Thank you all for the help and the quick fix! :)

[shvets-alex]

Thanks! My motd is working fine after the fix!

[paliasp]

Same here :)