Vscode-java: Since July update cause java highlight break

[ericsia]

I love to use VSC Java plugin but I had so much pain working with java now.
Now the highlight is glitchy, it don't highlight properly when Semantic Highlight is turn on.
Comment don't get highlighted :( I feel sad
Also I would appreciate it if it is easier to use junit assertEqual, when I try to open a single .java file, it cannot import org.junit

Environment

• Operating System: Windows 10
• JDK version: 14
• Visual Studio Code version: 1.48.2 before update also the same
• Java extension version: -

Steps To Reproduce

1. Turn on semantic highlight
2. start debugging

Current Result

Expected Result

Additional Informations

[fbricon]

Are you saying semantic highlighting is fine when not debugging?

[ericsia]

Are you saying semantic highlighting is fine when not debugging?

after I close and open it sometime it appears fine

[0dinD]

Does the highlighting also "fix" itself after you type a character?

Not sure if it's related, but I've seen something very similar happen when applying large changes to the document, such as undo/redo or using git reset --hard. I unfortunately could not find a way to consistently reproduce the issue, but the semantic highlighting broke in the same way as in your screenshot.

@ericsia if you could find a way to consistently reproduce the issue, that would help a lot in finding a fix. Does it always happen when debugging? Also, what version of the extension are you using? The latest?

[clankill3r]

I also notice that this happens a lot since July, I'm on a mac OSX 10.12.6.
Often typing and deleting something fixes it, but it stays annoying.

[0dinD]

@clankill3r can you provide some more info about what you're doing when the issue occurs? It would make the debugging process a lot easier if we had a way to reproduce the issue. And by "since July", do you mean the 0.65.0 update or do you not know which version you were using?

Personally I've seen this issue occur earlier than 0.65.0, but I couldn't find a way to consistently reproduce it. It seemed to occur somewhat randomly when applying large changes to the document. Spamming undo/redo sometimes reproduced it, but there was not pattern to it.

[ericsia]

@clankill3r can you provide some more info about what you're doing when the issue occurs? It would make the debugging process a lot easier if we had a way to reproduce the issue. And by "since July", do you mean the 0.65.0 update or do you not know which version you were using?

Personally I've seen this issue occur earlier than 0.65.0, but I couldn't find a way to consistently reproduce it. It seemed to occur somewhat randomly when applying large changes to the document. Spamming undo/redo sometimes reproduced it, but there was not pattern to it.

most accurately I didn't know since when do I turn it on. During last 2 update if I didn't rmb wrongly, I saw VS Code prompt me a msg to enable semantic highlight. So I go and turn it on. There is where I notice the highlights broken.
It is the best to try trace back from older build

[ericsia]

Does the highlighting also "fix" itself after you type a character?

Not sure if it's related, but I've seen something very similar happen when applying large changes to the document, such as undo/redo or using git reset --hard. I unfortunately could not find a way to consistently reproduce the issue, but the semantic highlighting broke in the same way as in your screenshot.

@ericsia if you could find a way to consistently reproduce the issue, that would help a lot in finding a fix. Does it always happen when debugging? Also, what version of the extension are you using? The latest?

During I start typing code from scratch I think. When I added comment, add more code then it might occurs

[0dinD]

During last 2 update if I didn't rmb wrongly

Ok, that strengthens my belief that this issue was not introduced via any of my changes in 0.65.0, and has been present since the initial implementation.

Again, if someone could provide a way to consistently reproduce this issue (via steps), that would make the debugging process a lot less time-consuming. (Meaning a fix can be released sooner.)

@ericsia how often do you estimate that this issue appears for you? And what's the connection between this issue and debug mode? Are you saying that the issue only appears when debugging, or that it appears more often when debugging?

[ericsia]

During last 2 update if I didn't rmb wrongly

Ok, that strengthens my belief that this issue was not introduced via any of my changes in 0.65.0, and has been present since the initial implementation.

Again, if someone could provide a way to consistently reproduce this issue (via steps), that would make the debugging process a lot less time-consuming. (Meaning a fix can be released sooner.)

@ericsia how often do you estimate that this issue appears for you? And what's the connection between this issue and debug mode? Are you saying that the issue only appears when debugging, or that it appears more often when debugging?

The color highlight when get messed like that doesn't go back to normal (when stop debug) until I reopen vscode again. I am also trying hard to find how to reproduce it but right now I'm not writing any code atm.
Now when I freshly open the java code everything look normal. But maybe tomorrow or 2 days later I will need to code again I will pay extra attention

[0dinD]

While I have not come up with a reliable way to reproduce this issue, it did occur again during development, so I thought I'd share another screenshot. For me it occurred quite randomly when typing code, possibly as an effect of pasting ~10 lines of code, but I'm not sure yet whether the two were related. The bug doesn't happen very often, but it's still annoying the few times where it does.

What's interesting to note is that just like the screenshot shared in the first comment of this issue, the tokens are not completely wrong, just offset by a few characters. Weirdly enough, the Developer: Inspect Editor Tokens and Scopes command reports that the semantic tokens are in the correct positions. So the offset either only appears visually, or the Developer: Inspect Editor Tokens and Scopes command is somehow also broken and doesn't show the correct information.

The issue goes away as soon as new semantic tokens are requested, so all you have to do is type a character and the highlighting is no longer broken.

[0dinD]

This issue has proven hard to debug, but I believe I've found a possible explanation, even if I don't know the exact cause or how to fix it. It seems to occur in some rare cases, when changes are made to the document during a request for semantic tokens.

The most reliable I've been able to reproduce it is to make multiple small changes to the document in rapid succession, but the timing required to reproduce the issue is very specific. I would also guess that the issue can occur when there is a change to the document during a particularly long running request for semantic tokens. Perhaps then, the issue will occur more frequently on machines with lower performance or when working with very large Java files.

All of the information that I've gathered seems to suggest that this is a bug in VS Code, rather than this extension. But again, I'm not entirely sure, as this issue is hard to reproduce. The code below this line looks like it could be part of the cause. It seems to shift the semantic tokens around and wait for new ones, if changes are made to the document during a request for semantic tokens.

I think I've noticed this behavior when trying to reproduce the issue, as the tokens do get shifted around (like in the previous screenshot!), only to get corrected a split second later. My best guess is that the request for new semantic tokens occasionally gets ignored on this line for some reason, in which case this issue would appear since the tokens have been shifted around but not fixed by requesting new ones. Again, as soon as you type a character, new semantic tokens are requested and the issue is gone.

If someone with more time on their hands or experience with the VS Code source-code could look into this, that would be greatly appreciated. I might submit an issue over at the VS Code repo, but right now I'm not sure if it's entirely the fault of VS Code, and I don't have a very reliable way to reproduce the issue.

[JoeyRxy]

yeah, I've encountered this problem since the beginning of semantic color rendering tech.
it's a severe problem I think.

[Eskibear]

@0dinD good findings. I also observe the same behavior, but it prevents me from digging deep as I cannot stably reproduce it. The root cause might be either of the below:
a) JDT.LS calculates semantic tokens based on an out-of-date state (corresponding CompilationUnit) of current document, and vscode highlights the file using wrong tokens.
b) JDT.LS does provide the correct result, but vscode somehow wrongly highlight the document.

If you have time, one way to validate might be, setting java.trace to "verbose" and to inspect the request and response. But It won't be easy as the responses are already encoded. If we do find clues/evidence to prove it's vscode's issue, we can request them for a fix.

[0dinD]

@Eskibear thanks for the tip, I'll try to figure out more details when I can. One thing that makes me think it's more of a VS Code-issue is that the Developer: Inspect Editor Tokens and Scopes command is reporting the correct positions for the tokens, while in reality, the text color has been offset by a few characters.

[ericsia]

@Eskibear thanks for the tip, I'll try to figure out more details when I can. One thing that makes me think it's more of a VS Code-issue is that the Developer: Inspect Editor Tokens and Scopes command is reporting the correct positions for the tokens, while in reality, the text color has been offset by a few characters.

here I produced it again, after copy paste. I am copy paste from a document to another java file.
Also I used find and replace a few times already.
Not just that, if you look carefully it said the syntax that is correct as wrong, in this case the right bracket

[0dinD]

@ericsia does the highlighted problem appear at the same time as the highlighting breaks? And does it also go away when the highlighting problems "fix" themselves? If the two are related, that's an interesting find.

I spent some more time debugging the semantic highlighting issue today, and come to the following conclusions:

• The issue is definitely caused by eclipse.jdt.ls, not VS Code.
  
  
  
  • I have verified that VS Code receives tokens with incorrect positions, by decoding the tokens on the client-side and printing the text at their position.
  
  
  • The reason why the Developer: Inspect Editor Tokens and Scopes command was reporting the correct positions is because it sends a separate request for semantic tokens, which means that it will always show the correct tokens while the editor could still have wrong ones.
  
  
• This probably means that whatever model of the document eclipse.jdt.ls is using to encode the tokens does not get updated when the document has changes during a request.

@Eskibear @fbricon do you have any idea why this might be happening? Are there any good ways to check if the document has had changes during the request? If so, maybe we can recompute the token positions when a change to the document is detected during the request. I'm just not that familiar with the Eclipse JDT APIs.

[ericsia]

@ericsia does the highlighted problem appear at the same time as the highlighting breaks? And does it also go away when the highlighting problems "fix" themselves? If the two are related, that's an interesting find.

I spent some more time debugging the semantic highlighting issue today, and come to the following conclusions:

• The issue is definitely caused by eclipse.jdt.ls, not VS Code.

  • I have verified that VS Code receives tokens with incorrect positions, by decoding the tokens on the client-side and printing the text at their position.
  • The reason why the Developer: Inspect Editor Tokens and Scopes command was reporting the correct positions is because it sends a separate request for semantic tokens, which means that it will always show the correct tokens while the editor could still have wrong ones.
• This probably means that whatever model of the document eclipse.jdt.ls is using to encode the tokens does not get updated when the document has changes during a request.

@Eskibear @fbricon do you have any idea why this might be happening? Are there any good ways to check if the document has had changes during the request? If so, maybe we can recompute the token positions when a change to the document is detected during the request. I'm just not that familiar with the Eclipse JDT APIs.

Yes. When I notice the color started to change, often it tells me some syntax error. But I can still compile it. The color won't fix automatically for me, I have to close vs code and reopen to appear normal again.

[0dinD]

Finally, after another debugging session, I think I've found the real issue, and a fix for it. The strange offset for semantic tokens seems to be introduced on this line, when the encoding algorithm needs to figure out the line and column number of a semantic token. The reason for the offset is that the document used to calculate the line and column number from the offset can sometimes get out-of-sync in the middle of a semantic tokens request, due to frequent updates of the document. This means that some token positions will be calculated based on an older version of the document, which causes them to be offset by the number of characters in the document edit.

@fbricon @Eskibear I've submitted a PR to fix this issue, could you please take a look at it?

[0dinD]

After playing around a little more with my changes in eclipse/eclipse.jdt.ls#1552, I realized that there were two separate problems that can cause the tokens to become offset while editing.

1. The document used by SemanticTokensVisitor to calculate the semantic token positions can become out-of-sync with its corresponding compilation unit, causing the token position calculations to become offset. (fixed by eclipse/eclipse.jdt.ls#1552)

2. The document used on the server side can become out-of-sync with the real document on the client side. This issue is a bit less noticeable since it won't cause the tokens to become offset unless text has changed which affects the line or column number of the semantic tokens. Additionally, this issue seems to always fix itself a split second after you see it, and never "gets stuck" like the first issue. This is probably because of the fact that VS Code seems to requests new tokens after it detects changes during a request. After looking at the implementation of the TypeScript semantic tokens provider, I believe I have found a fix for this issue as well, see #1632.

When using both eclipse/eclipse.jdt.ls#1552 and #1632, I can no longer reproduce this issue, no matter how many changes I spam to the document.

@fbricon @Eskibear please have a look at both PRs.

[0dinD]

@ericsia the strange offset to warning underlines looks like it could be somewhat related, but I decided to create a new issue (#1633) for it since I do not have as much insight into the code behind that feature and how to fix it.

[ericsia]

Finally, after another debugging session, I think I've found the real issue, and a fix for it. The strange offset for semantic tokens seems to be introduced on this line, when the encoding algorithm needs to figure out the line and column number of a semantic token. The reason for the offset is that the document used to calculate the line and column number from the offset can sometimes get out-of-sync in the middle of a semantic tokens request, due to frequent updates of the document. This means that some token positions will be calculated based on an older version of the document, which causes them to be offset by the number of characters in the document edit.

@fbricon @Eskibear I've submitted a PR to fix this issue, could you please take a look at it?

omg you are amazing, you are the first one to discovered the fix. Owh that makes more sense now