LSP+Rust-Analyzer autocomplete problem

[Jcskz]

Describe the bug
When input "String" or "include" word, the "autocomple" function does not execute correctly,
only the first character "S" or "i" will be left.

Screenshots

Environment (please complete the following information):

• OS: macOS: 11.0.1
• Sublime Text version: 4093
• LSP version: 1.1.0
• Language servers used: rust-analyzer

[rchl]

Please provide the server output from when you trigger a completion. Enable logging with those settings:

  "log_server": ["panel"],
  "log_stderr": true,
  "log_max_size": 0,

[Jcskz]

ServerLog.log

[predragnikolic]

if we have a file like.

fn main() {
    let string = Str| // cursor
    println!("Hello, world!");
}

1. trigger the AC.
2. select the 'String' completion item,
3. press enter to commit the String completion item.

Actual behavior:
Nothing happens.

Expected behavior:
Commit the completion item.

[
...
{
    'label': 'String', 
    'deprecated': False, 
    'insertTextFormat': 1, 
    'textEdit': {
        'newText': 'String', 
        'range': {
            'end': {'line': 1, 'character': 20}, 
            'start': {'line': 1, 'character': 17}
        }
    }, 
    'filterText': 'String', 
    'kind': 22, 
    'additionalTextEdits': [], 
    'documentation': {'kind': 'markdown', 'value': 'A UTF-8 encoded, growable string.\n\nThe `String` type is the most common string type that has ownership over the\ncontents of the string. It has a close relationship with its borrowed\ncounterpart, the primitive [`str`].\n\n[`str`]: ../../std/primitive.str.html\n\n# Examples\n\nYou can create a `String` from a literal string with [`String::from`]:\n\n```rust\nlet hello = String::from("Hello, world!");\n```\n\nYou can append a [`char`] to a `String` with the [`push`] method, and\nappend a [`&str`] with the [`push_str`] method:\n\n```rust\nlet mut hello = String::from("Hello, ");\n\nhello.push(\'w\');\nhello.push_str("orld!");\n```\n\n[`String::from`]: #method.from\n[`char`]: ../../std/primitive.char.html\n[`push`]: #method.push\n[`push_str`]: #method.push_str\n\nIf you have a vector of UTF-8 bytes, you can create a `String` from it with\nthe [`from_utf8`] method:\n\n```rust\n// some bytes, in a vector\nlet sparkle_heart = vec![240, 159, 146, 150];\n\n// We know these bytes are valid, so we\'ll use `unwrap()`.\nlet sparkle_heart = String::from_utf8(sparkle_heart).unwrap();\n\nassert_eq!("💖", sparkle_heart);\n```\n\n[`from_utf8`]: #method.from_utf8\n\n# UTF-8\n\n`String`s are always valid UTF-8. This has a few implications, the first of\nwhich is that if you need a non-UTF-8 string, consider [`OsString`]. It is\nsimilar, but without the UTF-8 constraint. The second implication is that\nyou cannot index into a `String`:\n\n```compile_fail,E0277\nlet s = "hello";\n\nprintln!("The first letter of s is {}", s[0]); // ERROR!!!\n```\n\n[`OsString`]: ../../std/ffi/struct.OsString.html\n\nIndexing is intended to be a constant-time operation, but UTF-8 encoding\ndoes not allow us to do this. Furthermore, it\'s not clear what sort of\nthing the index should return: a byte, a codepoint, or a grapheme cluster.\nThe [`bytes`] and [`chars`] methods return iterators over the first\ntwo, respectively.\n\n[`bytes`]: #method.bytes\n[`chars`]: #method.chars\n\n# Deref\n\n`String`s implement [`Deref`]`<Target=str>`, and so inherit all of [`str`]\'s\nmethods. In addition, this means that you can pass a `String` to a\nfunction which takes a [`&str`] by using an ampersand (`&`):\n\n```rust\nfn takes_str(s: &str) { }\n\nlet s = String::from("Hello");\n\ntakes_str(&s);\n```\n\nThis will create a [`&str`] from the `String` and pass it in. This\nconversion is very inexpensive, and so generally, functions will accept\n[`&str`]s as arguments unless they need a `String` for some specific\nreason.\n\nIn certain cases Rust doesn\'t have enough information to make this\nconversion, known as [`Deref`] coercion. In the following example a string\nslice [`&\'a str`][`&str`] implements the trait `TraitExample`, and the function\n`example_func` takes anything that implements the trait. In this case Rust\nwould need to make two implicit conversions, which Rust doesn\'t have the\nmeans to do. For that reason, the following example will not compile.\n\n```compile_fail,E0277\ntrait TraitExample {}\n\nimpl<\'a> TraitExample for &\'a str {}\n\nfn example_func<A: TraitExample>(example_arg: A) {}\n\nlet example_string = String::from("example_string");\nexample_func(&example_string);\n```\n\nThere are two options that would work instead. The first would be to\nchange the line `example_func(&example_string);` to\n`example_func(example_string.as_str());`, using the method [`as_str()`]\nto explicitly extract the string slice containing the string. The second\nway changes `example_func(&example_string);` to\n`example_func(&*example_string);`. In this case we are dereferencing a\n`String` to a [`str`][`&str`], then referencing the [`str`][`&str`] back to\n[`&str`]. The second way is more idiomatic, however both work to do the\nconversion explicitly rather than relying on the implicit conversion.\n\n# Representation\n\nA `String` is made up of three components: a pointer to some bytes, a\nlength, and a capacity. The pointer points to an internal buffer `String`\nuses to store its data. The length is the number of bytes currently stored\nin the buffer, and the capacity is the size of the buffer in bytes. As such,\nthe length will always be less than or equal to the capacity.\n\nThis buffer is always stored on the heap.\n\nYou can look at these with the [`as_ptr`], [`len`], and [`capacity`]\nmethods:\n\n```rust\nuse std::mem;\n\nlet story = String::from("Once upon a time...");\n\n// Prevent automatically dropping the String\'s data\nlet mut story = mem::ManuallyDrop::new(story);\n\nlet ptr = story.as_mut_ptr();\nlet len = story.len();\nlet capacity = story.capacity();\n\n// story has nineteen bytes\nassert_eq!(19, len);\n\n// We can re-build a String out of ptr, len, and capacity. This is all\n// unsafe because we are responsible for making sure the components are\n// valid:\nlet s = unsafe { String::from_raw_parts(ptr, len, capacity) } ;\n\nassert_eq!(String::from("Once upon a time..."), s);\n```\n\n[`as_ptr`]: #method.as_ptr\n[`len`]: #method.len\n[`capacity`]: #method.capacity\n\nIf a `String` has enough capacity, adding elements to it will not\nre-allocate. For example, consider this program:\n\n```rust\nlet mut s = String::new();\n\nprintln!("{}", s.capacity());\n\nfor _ in 0..5 {\n    s.push_str("hello");\n    println!("{}", s.capacity());\n}\n```\n\nThis will output the following:\n\n```text\n0\n5\n10\n20\n20\n40\n```\n\nAt first, we have no memory allocated at all, but as we append to the\nstring, it increases its capacity appropriately. If we instead use the\n[`with_capacity`] method to allocate the correct capacity initially:\n\n```rust\nlet mut s = String::with_capacity(25);\n\nprintln!("{}", s.capacity());\n\nfor _ in 0..5 {\n    s.push_str("hello");\n    println!("{}", s.capacity());\n}\n```\n\n[`with_capacity`]: #method.with_capacity\n\nWe end up with a different output:\n\n```text\n25\n25\n25\n25\n25\n25\n```\n\nHere, there\'s no need to allocate more memory inside the loop.\n\n[`&str`]: ../../std/primitive.str.html\n[`Deref`]: ../../std/ops/trait.Deref.html\n[`as_str()`]: struct.String.html#method.as_str'}
}, 
...
]

The Completion item has a textEdit,
When we commit the completion item,
I expect the lsp_complete_text_edit command to be run,
but the command is not run.

I am looking into this more.

[predragnikolic]

The bug is due to the completionItem.documentation property.
Rust analyzer returns documentation for some completion items.

And when we construct the sublimeText.CommandCompletionItem,
the text in the completionItem.documentation is not escaped and when we commit the completion
ST doesn't correctly parses the command and args passed when we press enter.

To fix this, we need to escape the documentation property(and probably the details property to??).

[rchl]

sublime completion item doesn't have documentation property, afaik, only details where we put the documentation from LSP completion. I don't get why we would need to escape that since it shouldn't affect inserting of completion...

[predragnikolic]

Correct, ST completion item dont have documentation
but the completion item that I wanted to refer to was export interface CompletionItem from the LSP spec. (LSP CompletionItem may have the documentation field).

/**
     * A human-readable string with additional information
     * about this item, like type or symbol information.
     */
    detail?: string;

    /**
     * A human-readable string that represents a doc-comment.
     */
    documentation?: string | MarkupContent;

Apparently this bug has nothing to do with escaping quotes as I thought initially,
but rather the "💖" character in the LSP completion item documentation is responsible for the bug.

Just to showcase. This will fix the problem with completing the String completion item.

--- a/plugin/core/views.py
+++ b/plugin/core/views.py
@@ -661,6 +661,14 @@ def format_completion(
     # NOTE: Some servers return "textEdit": null. We have to check if it's truthy.
     if item.get("textEdit") or item.get("additionalTextEdits") or item.get("command"):
         command = "lsp_complete_text_edit" if item.get("textEdit") else "lsp_complete_insert_text"
+
+        documentation = item.get("documentation")
+        if documentation:
+            if isinstance(documentation, dict):
+                item['documentation']['value'] = documentation['value'].replace("💖", "")
+            else:
+                item['documentation'] = documentation.replace("💖", "")
+
         args = {"item": item}  # type: Dict[str, Any]
         if item.get("command"):
             # If there is a "command", we'll have to run the command on the same session so store the name.

[rchl]

So then an ST bug?

[predragnikolic]

I would say so.
I will try to make a minimal repro case and open an issue at ST.

[predragnikolic]

The same is true for the includes completion item.

{
    'textEdit': {
        'newText': 'include',
            'range': {'end': {'line': 3, 'character': 11 },
            'start': {'line': 3, 'character': 4 }
        }
    },
    'label': 'include!',
    'deprecated': False,
    'insertTextFormat': 1,
    'kind': 2,
    'additionalTextEdits': [],
    'detail': '#[macro_export]\nmacro_rules! include',
    'documentation': {
        'kind': 'markdown',
        'value': 'Parses a file as an expression or an item according to the context.\n\nThe file is located relative to the current file (similarly to how\nmodules are found).\n\nUsing this macro is often a bad idea, because if the file is\nparsed as an expression, it is going to be placed in the\nsurrounding code unhygienically.This could result in variables\nor functions being different from what the file expected if\nthere are variables or functions that have the same name in\nthe current file.\n\n# Examples\n\nAssume there are two files in the same directory with the following\ncontents: \n\nFile \'monkeys.in\':\n\n```ignore (only-for-syntax-highlight)\n[\'🙈\', \'🙊\', \'🙉\']\n    .iter()\n    .cycle()\n    .take(6)\n    .collect::<String>()\n```\n\nFile \'main.rs\':\n\n```ignore (cannot-doctest-external-file-dependency)\nfn main() {\n    let my_string = include!("monkeys.in");\n    assert_eq!("🙈🙊🙉🙈🙊🙉", my_string); \n    println!("{}", my_string); \n } \n```\n\nCompiling \'main.rs\' and running the resulting binary will print\n"🙈🙊🙉🙈🙊🙉".'
    },
    'filterText': 'include!'
}

it contains "🙈🙊🙉🙈🙊🙉"

[predragnikolic]

Opened the issue https://github.com/sublimehq/sublime_text/issues/3777 :)

[rwols]

If you really need this to work @Jcskz, you can hack sublime.py as follows:

Open /Applications/Sublime Text.app/Contents/MacOS/Lib/python33/sublime.py (with sudo access).
In the function format_command, change json.dumps(args) to json.dumps(args, ensure_ascii=False).
Restart Sublime Text.

[Jcskz]

thanks @predragnikolic @rwols

[Jcskz]

hi, @predragnikolic @rwols
Does the mdpopups component not being able to display the complete document also related to this bug? When the mouse is placed on the String or include!, the popup document will be break by the mdpopups component.

[rchl]

@Jcskz That deserves its own issue.

[rchl]

Previous logs provided by @Jcskz include a completion with such documentation. So it's not the hover contents but a completion but the content might be the same:

  {
    "label": "String",
    "filterText": "String",
    "kind": 22,
    "additionalTextEdits": [],
    "textEdit": {
      "newText": "String",
      "range": {
        "start": {
          "character": 4,
          "line": 1
        },
        "end": {
          "character": 5,
          "line": 1
        }
      }
    },
    "insertTextFormat": 1,
    "deprecated": false,
    "documentation": {
      "value": "A UTF-8 encoded, growable string.\n\nThe `String` type is the most common string type that has ownership over the\ncontents of the string. It has a close relationship with its borrowed\ncounterpart, the primitive [`str`].\n\n# Examples\n\nYou can create a `String` from [a literal string][`str`] with [`String::from`]:\n\n[`String::from`]: From::from\n\n```rust\nlet hello = String::from(\"Hello, world!\");\n```\n\nYou can append a [`char`] to a `String` with the [`push`] method, and\nappend a [`&str`] with the [`push_str`] method:\n\n```rust\nlet mut hello = String::from(\"Hello, \");\n\nhello.push('w');\nhello.push_str(\"orld!\");\n```\n\n[`push`]: String::push\n[`push_str`]: String::push_str\n\nIf you have a vector of UTF-8 bytes, you can create a `String` from it with\nthe [`from_utf8`] method:\n\n```rust\n// some bytes, in a vector\nlet sparkle_heart = vec![240, 159, 146, 150];\n\n// We know these bytes are valid, so we'll use `unwrap()`.\nlet sparkle_heart = String::from_utf8(sparkle_heart).unwrap();\n\nassert_eq!(\"\ud83d\udc96\", sparkle_heart);\n```\n\n[`from_utf8`]: String::from_utf8\n\n# UTF-8\n\n`String`s are always valid UTF-8. This has a few implications, the first of\nwhich is that if you need a non-UTF-8 string, consider [`OsString`]. It is\nsimilar, but without the UTF-8 constraint. The second implication is that\nyou cannot index into a `String`:\n\n```compile_fail,E0277\nlet s = \"hello\";\n\nprintln!(\"The first letter of s is {}\", s[0]); // ERROR!!!\n```\n\n[`OsString`]: ../../std/ffi/struct.OsString.html\n\nIndexing is intended to be a constant-time operation, but UTF-8 encoding\ndoes not allow us to do this. Furthermore, it's not clear what sort of\nthing the index should return: a byte, a codepoint, or a grapheme cluster.\nThe [`bytes`] and [`chars`] methods return iterators over the first\ntwo, respectively.\n\n[`bytes`]: str::bytes\n[`chars`]: str::chars\n\n# Deref\n\n`String`s implement [`Deref`]`<Target=str>`, and so inherit all of [`str`]'s\nmethods. In addition, this means that you can pass a `String` to a\nfunction which takes a [`&str`] by using an ampersand (`&`):\n\n```rust\nfn takes_str(s: &str) { }\n\nlet s = String::from(\"Hello\");\n\ntakes_str(&s);\n```\n\nThis will create a [`&str`] from the `String` and pass it in. This\nconversion is very inexpensive, and so generally, functions will accept\n[`&str`]s as arguments unless they need a `String` for some specific\nreason.\n\nIn certain cases Rust doesn't have enough information to make this\nconversion, known as [`Deref`] coercion. In the following example a string\nslice [`&'a str`][`&str`] implements the trait `TraitExample`, and the function\n`example_func` takes anything that implements the trait. In this case Rust\nwould need to make two implicit conversions, which Rust doesn't have the\nmeans to do. For that reason, the following example will not compile.\n\n```compile_fail,E0277\ntrait TraitExample {}\n\nimpl<'a> TraitExample for &'a str {}\n\nfn example_func<A: TraitExample>(example_arg: A) {}\n\nlet example_string = String::from(\"example_string\");\nexample_func(&example_string);\n```\n\nThere are two options that would work instead. The first would be to\nchange the line `example_func(&example_string);` to\n`example_func(example_string.as_str());`, using the method [`as_str()`]\nto explicitly extract the string slice containing the string. The second\nway changes `example_func(&example_string);` to\n`example_func(&*example_string);`. In this case we are dereferencing a\n`String` to a [`str`][`&str`], then referencing the [`str`][`&str`] back to\n[`&str`]. The second way is more idiomatic, however both work to do the\nconversion explicitly rather than relying on the implicit conversion.\n\n# Representation\n\nA `String` is made up of three components: a pointer to some bytes, a\nlength, and a capacity. The pointer points to an internal buffer `String`\nuses to store its data. The length is the number of bytes currently stored\nin the buffer, and the capacity is the size of the buffer in bytes. As such,\nthe length will always be less than or equal to the capacity.\n\nThis buffer is always stored on the heap.\n\nYou can look at these with the [`as_ptr`], [`len`], and [`capacity`]\nmethods:\n\n```rust\nuse std::mem;\n\nlet story = String::from(\"Once upon a time...\");\n\n// Prevent automatically dropping the String's data\nlet mut story = mem::ManuallyDrop::new(story);\n\nlet ptr = story.as_mut_ptr();\nlet len = story.len();\nlet capacity = story.capacity();\n\n// story has nineteen bytes\nassert_eq!(19, len);\n\n// We can re-build a String out of ptr, len, and capacity. This is all\n// unsafe because we are responsible for making sure the components are\n// valid:\nlet s = unsafe { String::from_raw_parts(ptr, len, capacity) } ;\n\nassert_eq!(String::from(\"Once upon a time...\"), s);\n```\n\n[`as_ptr`]: str::as_ptr\n[`len`]: String::len\n[`capacity`]: String::capacity\n\nIf a `String` has enough capacity, adding elements to it will not\nre-allocate. For example, consider this program:\n\n```rust\nlet mut s = String::new();\n\nprintln!(\"{}\", s.capacity());\n\nfor _ in 0..5 {\n    s.push_str(\"hello\");\n    println!(\"{}\", s.capacity());\n}\n```\n\nThis will output the following:\n\n```text\n0\n5\n10\n20\n20\n40\n```\n\nAt first, we have no memory allocated at all, but as we append to the\nstring, it increases its capacity appropriately. If we instead use the\n[`with_capacity`] method to allocate the correct capacity initially:\n\n```rust\nlet mut s = String::with_capacity(25);\n\nprintln!(\"{}\", s.capacity());\n\nfor _ in 0..5 {\n    s.push_str(\"hello\");\n    println!(\"{}\", s.capacity());\n}\n```\n\n[`with_capacity`]: String::with_capacity\n\nWe end up with a different output:\n\n```text\n25\n25\n25\n25\n25\n25\n```\n\nHere, there's no need to allocate more memory inside the loop.\n\n[`str`]: prim@str\n[`&str`]: prim@str\n[`Deref`]: core::ops::Deref\n[`as_str()`]: String::as_str",
      "kind": "markdown"
    }
  },

But then it would mean that it breaks around the square brackets here:

You can append a [`char`] to

[Jcskz]

I create a new issue about prompt box #1483

[rwols]

Should be fixed for 4094. Please reopen if it's not fixed.