Checkstyle: LineLength measures Java characters, not Unicode characters

[cushon]

LineLength checks the number of Java characters in a line (i.e. the number of 16-bit code units in the UTF-16 encoded string), not the number of Unicode characters in the line.

The line length limit in the Google Java Style Guide is supposed to be 100 Unicode characters: https://google.github.io/styleguide/javaguide.html#s4.4-column-limit, so both of the following long lines should be OK:

class T {
    String aaaaaaa = "1234567890123456789012345678901234567890" + "1234567890123456789012345" + "_";
    String aaaaaab = "1234567890123456789012345678901234567890" + "1234567890123456789012345" + "💩";
}

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE module PUBLIC "-//Puppy Crawl//DTD Check Configuration 1.3//EN" "http://www.puppycrawl.com/dtds/configuration_1_3.dtd">
<module name="Checker">
  <module name="TreeWalker">
    <module name="LineLength">
      <property name="max" value="100" />
    </module>
  </module>
</module>

 java -jar checkstyle-8.2-all.jar -c config.xml Test.java 
Starting audit...
[ERROR] /var/tmp/Test.java:3: Line is longer than 100 characters (found 101). [LineLength]
Audit done.
Checkstyle ends with 1 errors.

ATTENTION:
All users from Windows OS who would like to have this issue fixed (after release of fix) have to define charset in Checker to UTF-8, in other case calculation of length will be wrong.
Second example shows how to set charset - https://checkstyle.org/config.html#Checker

[romani]

@cushon , how IDEs(Eclipse, IDEA, NetBeans) treat such cases ?
do they count columns or symbols ?

[cushon]

IntelliJ (and reportedly Eclipse) think the length of both of those lines is 100.

[romani]

Columns are different in AST, pay attention to SEMI -> ; [2:99] vs SEMI -> ; [3:100]:

$ cat Test.java 
public class Test {
    String aaaaaaa = "1234567890123456789012345678901234567890" + "1234567890123456789012345" + "_";
    String aaaaaab = "1234567890123456789012345678901234567890" + "1234567890123456789012345" + "💩";
}
rivanov@p5510:/var/tmp$ java -jar checkstyle-8.3-SNAPSHOT-all.jar -t Test.java | grep -B 12 SEMI 
    |--VARIABLE_DEF -> VARIABLE_DEF [2:4]
    |   |--MODIFIERS -> MODIFIERS [2:4]
    |   |--TYPE -> TYPE [2:4]
    |   |   `--IDENT -> String [2:4]
    |   |--IDENT -> aaaaaaa [2:11]
    |   |--ASSIGN -> = [2:19]
    |   |   `--EXPR -> EXPR [2:94]
    |   |       `--PLUS -> + [2:94]
    |   |           |--PLUS -> + [2:64]
    |   |           |   |--STRING_LITERAL -> "1234567890123456789012345678901234567890" [2:21]
    |   |           |   `--STRING_LITERAL -> "1234567890123456789012345" [2:66]
    |   |           `--STRING_LITERAL -> "_" [2:96]
    |   `--SEMI -> ; [2:99]
    |--VARIABLE_DEF -> VARIABLE_DEF [3:4]
    |   |--MODIFIERS -> MODIFIERS [3:4]
    |   |--TYPE -> TYPE [3:4]
    |   |   `--IDENT -> String [3:4]
    |   |--IDENT -> aaaaaab [3:11]
    |   |--ASSIGN -> = [3:19]
    |   |   `--EXPR -> EXPR [3:94]
    |   |       `--PLUS -> + [3:94]
    |   |           |--PLUS -> + [3:64]
    |   |           |   |--STRING_LITERAL -> "1234567890123456789012345678901234567890" [3:21]
    |   |           |   `--STRING_LITERAL -> "1234567890123456789012345" [3:66]
    |   |           `--STRING_LITERAL -> "💩" [3:96]
    |   `--SEMI -> ; [3:100]

sublime give same column for ";" - column 100:

the same column is doing IDEA - column 100:

gedit also show ";" at the same column - 100:

[romani]

problem is in ANTLR grammar, unicode symbol is parsed as 2 characters in length, but should be 1.

[QwerTech]

+1 for this issue.

[romani]

@nmancus1 , this is also good issue to look at as introduction to ANTLR parser.

[nmancus1]

@romani Ok, so BufferedReader is reading the emoji here as two characters (97, 98):

Then, in LineLengthCheck.java:

final int realLength = CommonUtil.lengthExpandedTabs(
                line, line.length(), getTabWidth());

CommonUtil.lengthExpandedTabs() is counting the 💩 as two chars, which is what's being logged. I'm not sure how the ANTLR parser is involved here, can you point me in the right direction? It would seem to me like we would need to write a method to catch unicode characters (within CommonUtils) in a certain range of values and report them as a count of one.

Edit: Of course this would only fix Checkstyle's error reporting, not the AST itself.

[romani]

yes, it is our recent change .... https://checkstyle.org/config_sizes.html#LineLength become a Check that does not depends on java parser at all from https://checkstyle.org/releasenotes.html#Release_8.24

I am removing ANTLR label from it.
We just need to fix util method.

@nmancus1 , please fix this issue if you see a solution.

[strkkk]

Fix is merged.