Tree-sitter: Is TSPoint's column counting on bytes or chars?

[ubolonton]

Is it different between the C API and the Rust binding?

If I understand it correctly, in the Rust binding, Parser::parse_utf16_with wants the callback to count column with u16, while Parser::parse_with, Parser::set_included_ranges, Tree::edit, Tree::changed_ranges and other functions count column with u8?

Side question: Is it possible to specify input edits and included ranges with byte offsets only? Emacs's low-level APIs seem to be (char) offset-based, leaving row and column to be computed in Lisp.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label question to this issue, with a confidence of 0.92. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[maxbrunsfeld]

Good question; that should definitely be in the docs. The column counts bytes.

Side question: Is it possible to specify input edits and included ranges with byte offsets only?

Unfortunately not. Tree-sitter needs to know both in order to update nodes' row/column coordinates correctly. I would think that Emacs must have some reasonably fast way to compute the row/column for a byte offset in order for its UI to work at all. Maybe it doesn't expose it to the lisp API though. 😞

[Aerijo]

@maxbrunsfeld Was this added to the docs? The best I could find was

The TSInput structure lets you to provide your own function for reading a chunk of text at a given byte offset and row/column position.

in Using Parsers > Providing the code. This doesn't say what a column is measuring (bytes, code units, code points, grapheme, etc.), and the next sentence says

The function can return text encoded in either UTF8 or UTF16. This interface allows you to efficiently parse text that is stored in your own data structure.

Which implies the column might be code units (so 2 bytes for UTF16).

Row is similarly unclear, and making it explicit would help. In particular, which regex of \r\n?|\n or \r?\n would be correct?