Mudlet: ">" is replaced to ";" on Mudlet 3.8.1 under Ubuntu using UTF-8

[hluaces]

Brief summary of issue

Since 3.8.1 any text sent to Mudlet's main window which contains the symbol ">" (greater than) is replaced by ";" (semicolon). This seems to happen only on UTF8-enabled muds and terminals.

Steps to reproduce the issue

1. Connect to any mud which supports UTF8 (for example: reinosdeleyenda.es:23).
2. Set the Mud's output to UTF8 (in reinos de leyenda: charset UTF-8).
3. Do the same within Mudlet's configuration.
4. Send any command which contains a ">" symbol (for example: "decir this is a test >>>>").

Error output

See attached image.

Extra information, such as Mudlet version, operating system and ideas for how to solve / implement:

• Mudlet 3.8.1 (not happening on previous versions).
• This seems to be happening only under Llnux systems (tested on Ubuntu 16.0.4 and an unknown version of Lubuntu).

[SlySven]

I am not sure I can reproduce this but I would note that the Reinos de Leyenda (Spanish) MUD does use MXP which I feel sure is a factor in this!

I could not get the effect by using something like lua feedTriggers("\ndecir this is a test >>>>\n") with the encoding set to UTF-8 but then I am not sure the profile was in a state where MXP was active...

[hluaces]

Doesn't seem to be MXP.

Our mud offers the possibility to disable it entirely and when doing so the error is still present.

Doing lua feedTriggers("\ndecir this is a test >>>> test test test\n") returns: decir this is a test asdasd (note the absence of ">" symbols).

[SlySven]

Ah, I wasn't impugning the MUD Server, but our handling of MXP. I am aware it is not precisely as Zugg intended (but what that is is not entirely clear either 😮 ). I suspect something is wrong with the escaping/conversion of > to/from > i.e. the code following:

    if (mMXP) {

in void TBuffer::translateToPlainText(std::string& incoming, const bool isFromServer) ...

[hluaces]

Ah, I wasn't impugning the MUD Server, but our handling of MXP

I supposed it, just adding more info :grinning:

[SlySven]

Yes! If I set TBuffer::mMXP to true within a run in the debugger (whilst NOT connected to a Server) I do get the '>' being suppressed like that when using feedTriggers(...).

However, I am not sure that is in itself a valid situation. This MUSHClient forum topic MXP Spec clarification from twelve years ago seems to touch on the problem with handling < and > character. It seems to be that if MXP is enabled < and > are supposed to be escaped (to < and >) by the Server, but that does not always make sense. 😖

[vadi2]

See also https://github.com/evennia/evennia/issues/1058

[Griatch]

As for the "clarification" from Zugg, I'm with Nick Gammon here (from the discussion that followed) in that thread:

The spec says "you must use < to refer to a less-than symbol directly"
(my emphasis). No wriggle-room there. You must do it. If the server does
not do it, it is not following the spec and is wrong.

Once MXP is turned on, Evennia will thus always convert < and >to < and > when not part of a MXP tag. It seems to be the sane way to handle it. Before Server/Client has agreed that MXP should be on, the server will send < and > as-is and the Client should display them verbatim (and not remove any text between them).

It's all pretty confusing (also from reading that Mushclient thread) but the fact is that we regularly get people wondering about the Evennia/Mudlet behavior to this day (showing the &lt and &gt replacements verbatim in MXP mode, that is). Not sure what to do about it from our end, do you guys have any further suggestions?

[vadi2]

@Griatch Let's track the MXP issue in https://github.com/evennia/evennia/issues/1058 for convenience - this bug is specifically about the UTF-8 code by the looks of it, it occurs in TBuffer::processUtf8Sequence()

[vadi2]

This is related to MXP and UTF-8 because the UTF-8 code does the following:

    } else {
        // Single byte character i.e. Unicode points: U+00000000 to U+0000007F
        mMudLine.append(bufferData.at(pos));
    }

This is incorrect as it needs to append the char ch like the ASCII code does:

mMudLine.append(ch);

Because it's the char ch that has been unescaped and translated the > from the game into >. However that char is never even passed to processUtf8Sequence() :) as a result it picks up what is at the given position, namely, ;. This is why all >>>> correctly sent by the game just get translated into ;;;;

I'm not all that familiar with the utf-8 decoding, @SlySven could you help out with a PR for this?

[SlySven]

I suspect this is because I don't grok MXP (to be honest I thought I did say that with handling all the OOB protocols when dealing with all the different MUD Server encodings I did not know whether that particular one (MXP) would work).

[SlySven]

All three of the multi--byte decoders (UTF-8, GBK/GB18030 and Big5) are involved in converting a string of bytes into the UTF-16 that we use internally.

If it is necessary to convert a < into something else > or do the opposite - it would safest IMHO to be done at the point following the COMMIT_LINE: label where we actually have a complete line of text (I hope) to append to the TBuffer data store. Note that if you are adding or deleting characters you will also need to alter mMudBuffer at the same indexes (as mMudLine).

I am not sure whether I can come up with a PR for this on my own. As I said I really do not get MXP or rather the (very limited as far as I can see) portion that Mudlet does try and do.

[SlySven]

The above mentioned PR is a cosmetic step needed after a past PR - nothing to do with solving this issue but it is in the same area...!

[druuimai]

by the way, one of my closed PR (currently on the development line) will supress MXP from processed through feedTrigger. just FYI.

[vadi2]

That'll help but the issue will still remain afaik.

[druuimai]

I'm unable to connect to reinosdeleyenda.es to test this utf-8 string from a master build. anyone got any good mud that i can test this problem?

[SlySven]

Humm, ch is a reference into localBuffer so assigning to it will insert a single character into the later.

Where the replacement is a different length though this is breaking things by the sound of it.

We adjust the pointer position in those three processXXSequence(...) methods that digest multi-byte encoding sequences (which is why pos is a not a const reference in their parameter list) and convert them to one or two QChars to be inserted into mMudLine.

I guess a rethink about how the localBuffer contents are handled/revised by the MXP code is in order.

[Rys3n]

This bug is also present in Windows on Mudlet 4.4 while playing an Evennia game called Arx using UTF-8.

[Griatch]

@Rys3n Arx is running a very old version of Evennia; not sure if this is the reason, but it's something to keep in mind. A first start for starting to fix this would be for Arx to update so we are all comparing the same code, so worth to keep reminding them to look into that ;).

[mpconley]

There is a report on the LDMud driver email list of this same issue using Mudlet with an experimental mud library I maintain that uses MXP. I'll be on the lookout to reproduce.

[netforce10]

Arx recently updated to evennia 0.9.0 and the problem still persists.

[ellej]

I have the same problem on Discworld MUD with version 4.8.0.

[mpconley]

@gcms Do you have a cure for this with the refactor you are doing for MXP? This is present on Age of Elements on the prompt when one is in UTF-8 server encoding.

[gcms]

@gcms Do you have a cure for this with the refactor you are doing for MXP? This is present on Age of Elements on the prompt when one is in UTF-8 server encoding.

This is probably something related to entity resolution, when it replaces > by >. Is it an old problem or something with the new release?

It seems to be working correctly on the new MXP implementation. If you want to try it, it's already committed in the development branch.

[gcms]

I don't know if it worths to add it to this release, but this can be solved with a small change, setting localBuffer[localBufferPosition] = ch just after the entity resolution part.

https://github.com/gcms/Mudlet/tree/issue%231623

[vadi2]

It's an old problem from two years ago. Looks like it'll be fixed for the next release. You can try out these builds meanwhile:

macOS: https://make.mudlet.org/snapshots/d1fdd5/Mudlet-4.6.2-testing-c63b597.dmg
Linux: https://make.mudlet.org/snapshots/392077/Mudlet-4.6.2-testing-c63b597-linux-x64.AppImage.tar
Windows: https://make.mudlet.org/snapshots/441964/Mudlet-4.6.2-ptb-2020-05-06-c63b5-windows.exe

[ellej]

I've tested with the development branch and the problem is indeed solved there.

[vadi2]

Thanks for confirming and thanks @gcms for the fix :)

[mpconley]

I get a Mudlet crash when opening my ageofelements.org 7680 profile when
testing macOS:
https://make.mudlet.org/snapshots/d1fdd5/Mudlet-4.6.2-testing-c63b597.dmg

That does not happen when opening the StickMUD profile.

On Wed, May 6, 2020 at 4:02 AM Vadim Peretokin [e-mail]
wrote:

Thanks for confirming and thanks @gcms https://github.com/gcms for the
fix :)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/Mudlet/Mudlet/issues/1623#issuecomment-624501089, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXKUASMW62QET3GCR3REKDRQEKPXANCNFSM4E335GQQ
.

[Rys3n]

Appreciate the fix, @gcms .