Winforms: RichTextBox doesn't work right when hidden text is used

[Amy-Li03]

• .NET Core Version:
  .NET SDK 5.0.100-preview.7.20304.9

• Have you experienced this same bug with .NET Framework?:

  1. This issue can reproduce from .NET 4.7 to .NET 4.8 latest build.
  2. This issue cannot reproduce in versions prior to .NET 4.7.

Problem description:
When insert hidden text (e.g. "{rtf1ansi{Sample for {v HIDDEN }text}}") to the RTF fragment, the RichTextBox's Text property doesn't return this hidden text anymore.

Expected behavior:
The hidden text is hidden in RTF view but can be accessed using the Text property and found using Find() method.

Minimal repro:

1. Create a .NET core project with a RichTextBox control. TestApp.zip
2. Add these 3 lines after the call to InitializeComponent():
   _richTextBox1.Rtf = @"{rtf1ansi{Sample for {v HIDDEN }text}}";
   System.Diagnostics.Trace.Assert(richTextBox1.Text == "Sample for HIDDEN text");
   System.Diagnostics.Trace.Assert(richTextBox1.Find("HIDDEN") >= 0);_
3. Build and run the project.

[weltkante]

May want to check on Desktop Framework if toggling the switch for the RTF control version changes behavior. Could have been an intentional change in the RTF control itself?

[mavnorthwind]

Thanks for pointing out this switch. While it can be used to mitigate the problem in that RichTextBox will still use Rich Edit 3.0 (which I had to do with my own workaround as well), it doesn't really solve the problem stemming from Rich Edit 4.1.
Rich Edit 4.1 supports TSF and has vastly better table handling, so of course you'd want to utilize the current version if it could behave like previous versions otherwise.

[weltkante]

Yes I was pointing out to check the switch because if the issue is in the native control then WinForms can't do much about it besides notifying the owners of the control, the source is not in the WinForms repo (its a native control and WinForms is just wrapping it into a managed API)

[mavnorthwind]

After browsing through the reference source a bit and playing around with various ways of accessing the RichEdit's text I think I found something:
When you use EM_GETTEXTEX to get the text contents, you can specify whether or not you want hidden text (there's a flag GT_NOHIDDENTEXT).
This flag is ignored by RichEdit 2.0/3.0 but honored by RichEdit 4.1.

So my guess is that this bug was fixed with 4.0 or 4.1 but not communicated (at least I couldn't find anything about it) and also influenced the result of EM_STREAMOUT (what is used in the RichTextBox's Text getter).

Perhaps I'll overwrite this getter myself...

[RussKie]

You're welcome to send us a PR with a fix too

[hughbe]

Okay can repro in #3466
I'll work on a fix

[hughbe]

Having a slight issue here. Basically, it seems like EM_STREAMOUT and EM_FINDTEXT don't have any support for including hidden values. So I don't think we can actually get StreamOut or FindText to respect hidden strings at themoment.

I created a subclass and called EM_GETTEXTEX as an experiment:

[WinFormsFact]
public void RichTextBox_Text_GetRtfText_ReturnsExpected()
{
    using var control = new SubRichTextBox
    {
        Rtf = "{\\rtf1\\ansi{Sample text}}"
    };
    Assert.Equal("Sample text", control.Text);
    Assert.True(control.IsHandleCreated);
}

[WinFormsFact]
public void RichTextBox_Text_GetHiddenRtfText_ReturnsExpected()
{
    using var control = new SubRichTextBox
    {
        Rtf = "{\\rtf1\\ansi{Sample for {\\v HIDDEN }text}}"
    };
    Assert.Equal("Sample for HIDDEN text", control.Text);
    Assert.True(control.IsHandleCreated);
}

[WinFormsFact]
public void RichTextBox_TextLength_GetHiddenText_ReturnsExpected()
{
    using var control = new SubRichTextBox
    {
        Rtf = "{\\rtf1\\ansi{Sample for {\\v HIDDEN }text}}"
    };
    Assert.Equal(22, control.TextLength);
}

private class SubRichTextBox : RichTextBox
{
    public unsafe override string Text
    {
        get
        {
            int textLength = TextLength;
            char[] textBuffer = ArrayPool<char>.Shared.Rent(textLength + 1);
            string result;
            fixed (char* pTextBuffer = textBuffer)
            {
                var gte = new GETTEXTEX
                {
                    cb = (uint)sizeof(GETTEXTEX),
                    flags = GT.DEFAULT,
                    codepage = 1200u /* CP_UNICODE */
                };
                int count = unchecked((int)(long)User32.SendMessageW(Handle, (User32.WM)RichEditMessages.EM_GETTEXTEX, (IntPtr)(&gte), (IntPtr)(pTextBuffer)));
                result = new string(pTextBuffer, 0, count);
            }
            ArrayPool<char>.Shared.Return(textBuffer);
            return result;
        }
        set => base.Text = value;
    }
}

However, this doesn't seem to respect hidden text either...

  X System.Windows.Forms.Tests.RichTextBoxTests.RichTextBox_Text_GetHiddenRtfText_ReturnsExpected [56ms]
  Error Message:
   Assert.Equal() Failure
                          (pos 15)
Expected: Sample for HIDDEN text
Actual:   Sample for HIDD
                          (pos 15)
  Stack Trace:
     at System.Windows.Forms.Tests.RichTextBoxTests.RichTextBox_Text_GetHiddenRtfText_ReturnsExpected() in C:\Users\hughbe\source\repos\ConsoleApp\RichTextBoxTests.cs:line 45

I ran the above test with net48 TFM

[hughbe]

Bit of an awkward one too - currently TextLength in .NET Core does include hidden text but Text does not (see the PR), so we end up in odd situation where Text.Length != TextLength

[weltkante]

so we end up in odd situation where Text.Length != TextLength

That can be bad, I don't know if WinForms itself assumes these match, but I certainly have written code that does, and I'm sure other people have as well. Considering .NET Core doesn't allow downgrading the RTF control anymore at least this mismatch needs to be fixed. @RussKie does that need a separate issue?

[hughbe]

Potential fix would be to accept the new behaviour of not including hidden text and updating TextLength to have NOHIDDENTEXT flag

[JeremyKuhne]

We should clearly fix this for 5.0. We can review what to do for the other versions once we have a fix.

@amy-li Just to be clear- on the same machine setting the different TFM's causes the behavior you describe (4.6.2 no problem, 4.7.x problem, etc.). If the state changes on the same machine this means the problem is behind an AppContext switch.

[RussKie]

We had a further chat with Jeremy, we want to preserve the original behavior, if at all possible.

[RussKie]

does that need a separate issue?

Let's keep it a part of this work, unless there is no way we can preserve the original behaviour.

[hughbe]

I don't know about the fix for this. Closed https://github.com/dotnet/winforms/pull/3466

[Amy-Li03]

@JeremyKuhne sorry for my delay.
Yes, on the same machine setting the different TFM will cause the behavior: 4.6.2 no problem, 4.7.x & 4.8 problem.

[JeremyKuhne]

@Amy-Li03 thanks for the details- that clearly shows that this problem is behind an AppContext switch. We can narrow down by looking at the switches that were introduced in 4.7.

[weltkante]

@JeremyKuhne wasn't it already confirmed that the RichEdit versioning switch is what triggers this? Older versions probably not supporting hidden text?

Given that you need a pretty old version of RichEdit for this to not be broken I don't think you can derive a workaround from the AppContext switch.

[mavnorthwind]

From what I saw, previous versions used the RichEdit 2.0/3.0 control from RichEd20.dll and from .NET 4.7 onward RichEdit 4.1 from Msftedit.dll was used and this version reacts differently to EM_STREAMOUT.
I've implemented my own workaround by overwriting the getter of Text and SelectedText to use EM_GETTEXTEX instead.
The only additional fix I had to implement was to replace all r line breaks with n to get the same line breaks EM_STREAMOUT returned.

Hope you can implement a fix with that information. If not, feel free to ask.

[RussKie]

@mavnorthwind if you have a working fix, do you mind raising a PR? :)

[RussKie]

I wouldn't be surprised if this issue is also causing #3632.

[mavnorthwind]

@RussKie I'll try, but since it's my first contribution to such a project it could take a few cycles to get it right... :)

[RussKie]

I'll try, but since it's my first contribution to such a project it could take a few cycles to get it right... :)

We all start somewhere :) Feel free to reach out, if you get stuck.

[mavnorthwind]

I think I've fixed the problem and created a PR.
In case I did something wrong please let me know.

[Amy-Li03]

Verified this issue on .NET 5.0.100-rc.1.20426.13 build, issue was fixed, the hidden text is hidden in RTF view and it can be accessed using the Text property and found using Find() method. But a regression issue #3778 was found.