Compromise: ProseMirror/CodeMirror offsets

[douglance]

How does the offset on a match relate to its position in a document?

Is it possible to get the character index of the match within an entire document?

Thanks!

[spencermountain]

hey @douglance yeah, the offset result format isn't all-too clear, but maybe this example will help:
https://runkit.com/spencermountain/5f788bcac952bf001acaae3d

start and length always refer to its position in the full document. You can avoid this, or set a 'new full doc' by using .clone() on a match, i think.

We used to have information about pre+post whitespace in the result. Maybe we should calculate offset for both the match, and the parent doc. Anyways, I agree that it's not very clear right now. I'd love some help re-thinking it, so that it's clearer.
cheers

[douglance]

Thanks for the example! I am using compromise in conjunction with ProseMirror which uses character indexes heavily.

I've been able to get the offsets of characters in both libraries; but I can't consistently get the character indexes to line up perfectly every time.

I'm not exactly sure how the two differ, so any help with understanding how the offset is calculated would be helpful. I looked through the code but it wasn't too intuitive to me.

What does the index in an offset represent?

[spencermountain]

hey cool!
Yeah, i'd love to see something like this work really well. Happy to help.

I've had a go of this using codemirror and i used a simple term.index+term.start - although I do remember there being some offset errors.

I'm happy to help you track these down, if you can find a reproducible offset issue.

yes - index is supposed to be a straight character-count in the original document. There may be issues with whitespace, or unicode, or something else entirely.

Play-around with that demo, and see if the issues you have exist there too. The code for it is very-simple.
cheers

[spencermountain]

hey @douglance - how'd you get on this? Curious how it went.

[douglance]

@spencermountain Thanks for pinging me to circle back here! 😄

I was able to use doc.match() to find parts of speech and decorate them with ProseMirror by getting offsets from the .json() call. It was actually pretty smooth sailing once I figured out how the offsets and positions were calculated.

I'm still working through how to optimize my system because I want it to run on each keystroke and for the moment it's not scaling well on larger text samples.

[spencermountain]

beautiful. Yeah, you can use nlp.tokenize() to fast-parse and cache the unchanged sentences.
That's what I would do.
cheers