slate 🚀 - fix editing with "soft keyboards" (eg. Android, IMEs)

I've done some input tests in different browsers and devices using Dan Burzo's input tester.

I think the only way we're going to arrive at proper Android support is to use a tool like this, and take extremely detailed recordings of inputing the exact same text across the different platforms. Otherwise the intricacies of the ordering of events, and when compositions do or do not start is going to be too complex to guess.

I've started with a few, we'll need more. I think we'll need:

[x] [Inserting text.](https://paper.dropbox.com/doc/Inserting-text.--AJwMZQxzP38lZQVT_J9yGceEAg-lE0z5HXg9oMSEHV72MsC4)
[x] [Inserting IME text.](https://paper.dropbox.com/doc/Inserting-IME-text.--AJzP1_C8P5~Q3W8e4sIagUwGAg-risj4IGaKhpGaXqcDks50)
[ ] Inserting a word via auto-suggest.
[ ] Inserting a word via gesture typing.
[ ] Auto-correcting a word.
[x] [Deleting characters backwards.](https://paper.dropbox.com/doc/Deleting-characters-backwards.--AJxEBLEKNA5~yqNBf0_cNagwAg-deG1bAqoqAYXMWlNDqQaG)
[ ] Deleting characters forwards.
[ ] Delete characters backwards, across word boundaries.
[ ] Selecting a word, then deleting it.
[ ] Selecting a range across blocks, then deleting it.
[ ] Moving to the end of a word.
[ ] Moving to the middle of a word.
[ ] Moving to the middle of a word, then inserting text.
[ ] Splitting a block at the end.
[ ] Splitting a block in the middle.
[ ] Joining a block from the start.

Even from just the ones I did, we can already see complex composition behaviors on Android that we're going to have to solve for:

Moving the cursor into or to the edge of a word, starts a new composition immediately (even without any input) with the word as content.
Deleting characters backward doesn't trigger any useful key events, but it does update the composition ticking down the characters in a word.
All text insertions on Android is via composition events, so in solving for its composition handling, we'll probably get desktop IME support in the process.

I think we'll need to store additional information in these tests of what the current window.getSelection() range is, because that is key to understanding how much information we can get from these composition events. Someone might need to build a custom input tester for Slate specifically to make this easier.

ianstormtaylor on 10 Aug 2018

👍6

@ianstormtaylor

You are making a lot of progress here. I’ve committed next week to working on mobile support but by a large margin, you are more qualified to solve this than I.

I think we might make more progress if you give me any menial tasks, research, etc. In order to help you. Don’t feel obligated to give me work but if you have any tasks, I will work on them full time for you starting Monday. Let me know.

thesunny on 10 Aug 2018

@thesunny thanks! One of the most helpful things would be getting more of those detailed event samplings across devices/browsers for each type of edit. That would allow anyone who works on this in the future to reference them to make sure they're thinking through the event logic correctly.

Without those I think we'd just be fumbling around guessing what logic we need to add to the After plugin to get composition events working.

ianstormtaylor on 10 Aug 2018

Okay, I will start with that Monday

thesunny on 10 Aug 2018

@ianstormtaylor Dan Burzo's input tester is exactly what I was looking for the other day to send you. Glad you found it/already had it.

@thesunny and @ianstormtaylor how/where are you laying down code for this? I don't think there's much I can contribute in the way of code in the short term, but I'm happy to test ideas on my Android device and help any other way that you two need.

Slapbox on 13 Aug 2018

I created DropBox Paper documents. I set up a testing environment and put together all the tests below

So, a few differences from @ianstormtaylor is that I tested 6 different versions of Android and they all provided different values. There are more differences within the different Android versions than across all browser/OS versions. iOS, Safari and Chrome are very similar.

[x] [Inserting text.](https://paper.dropbox.com/doc/Inserting-text--AKEykPM~oE90Hwjf3fM7RGAeAg-A7mNUCDxo6xLMMGeBQ9zf)
[x] [Inserting IME text](https://paper.dropbox.com/doc/Inserting-IME-text--AKGtukKkRPNFX8c3X~BRwPzaAg-2Zz9Pp8q0xTzDeD80qe22)
[x] [Inserting a word via auto-suggest.](https://paper.dropbox.com/doc/Insert-a-word-via-auto-suggest--AKHvEDmW6fiJvS8DGZxpBYlWAQ-ssgZ0u3tDPrt1RHHK2Owr)
[x] [Inserting a word via gesture typing](https://paper.dropbox.com/doc/Inserting-a-word-via-gesture-typing--AKEN01cbNOic4F33Bf6VDH~yAg-u5bf0xVcspArg2TmVYMMQ)
[x] [Auto-correcting a word](https://paper.dropbox.com/doc/Auto-correcting-a-word--AKHvvPQjnuF_5vnsEHhBKzL8Ag-YBwVEI2ApQeJZWCP2IQAP)
[x] [Deleting characters backwards](https://paper.dropbox.com/doc/Deleting-characters-backwards.--AKERq1WGndDZJNIykVg5sOGQAg-tglMWfjfMGFPv3l2M8IEx)
[x] [Deleting characters forwards](https://paper.dropbox.com/doc/Deleting-characters-forward--AKEqUdxyYS5UZiOtkgeDgQheAg-AttLvefxgP8THK5GzGJhH)
[x] [Delete characters backwards, across word boundaries](https://paper.dropbox.com/doc/Delete-characters-backwards-across-word-boundaries--AKGwHZz~szpd1ygohbfpVZEjAg-hGdhjd4JIE29GGYUMljaM)
[x] [Selecting a word, then deleting it.](https://paper.dropbox.com/doc/Selecting-a-word-then-deleting-it.--AKFPLqEfL6ao36kUqRM66HjhAg-6nwcmvPGlxOW0QsgsIWTX)
[x] [Selecting a range across blocks, then deleting it.](https://paper.dropbox.com/doc/Template-for-Blocks--AKFWAJ4JzNXV8RFul97q~MhOAg-TKwO4F43XO1tUPGmXQUTY) (1)
[x] [Moving to the end of a word, from inside the word](https://paper.dropbox.com/doc/Moving-to-the-end-of-a-word-from-inside-the-word--AKHBvngTnTWPaylXLYyI0HblAg-gui3frYbZwfMeEbQvax3S)
[x] [Moving to the start of a word, from inside the word](https://paper.dropbox.com/doc/Moving-to-the-start-of-a-word-from-inside-the-word--AKEgfVTfWnLG20zjVVaV~lNuAg-A1uREXL4MXtjz7CsGUlQD)
[x] [Moving to the middle of a word, from the left](https://paper.dropbox.com/doc/Moving-to-the-middle-of-a-word-from-the-left--AKHRrKrLVdxW8DMp3wfupOgZAg-o0nUZPV9QdAUtFBfxmWZc)
[x] [Moving to the middle of a word, from the left, then inserting text](https://paper.dropbox.com/doc/Moving-to-the-middle-of-a-word-from-the-left-then-inserting-text--AKE4sLwNNyzWv3AWaUKdXrFOAg-2n305Rru0Zg6NWl6Rnsjn)
[x] [Splitting a block at the end.](https://paper.dropbox.com/doc/Splitting-a-block-at-the-end.--AKEYixg7iWG7DYAdl7ibkDwMAg-OfROfC3ajPU6YKtmQ3QJy) (1)
[x] [Splitting a block in the middle.](https://paper.dropbox.com/doc/Splitting-a-block-in-the-middle--AKFVdkSzH8J4unYZQjm84fzAAg-zrvwmzORqlEbLlm9SR1sA) (1)
[x] [Joining a block from the start.](https://paper.dropbox.com/doc/Joining-a-block-from-the-start--AKGsuke0e8tVL~rzXAxm0WnmAg-GVEJ9DUFiFK9ZVtAMqSZM) (1)

(1) Forked the @danburzo app to accept HTML tags like <p></p> and can be found here

I'll keep updating this list.

thesunny on 13 Aug 2018

🎉14 ❤6 👍1

@thesunny can you list the browser versions for the Android tests? I'm guessing Chrome 68?

Also, we should agree on which field we're using in that input tester, either the Simple DOM contenteditable or the React-managed contenteditable.

I think it's reasonable to expect Slate to only have out of the box mobile support via React. This would simplify the task, I think. What do you think @ianstormtaylor ?

I believe the only events we should be relying on, if we want the widest base of support on Android, are:

compositionStart
compositionUpdate
compositionEnd
deleteContentBackward (maybe)

FYI when typing, each keystroke updates the composition, but choosing a suggestion ends the composition and creates the final word like this:

compositionData: Barnac
Suggestion: Barnacles

If you click Barnacles, the final word is constructed like this.

compositionEnd Barnac
insertText les

Barnacles

after this word, Google suggests the word, "the" - If you click this word, it is inserted like this:

insertText the (there is a space inserted before the word)

Note, the above data is from the React contentEditable, but appears to be the same in the simple contentEditable.

Slapbox on 13 Aug 2018

The browser version I believe is locked to the Android API version. I'm using whatever Chrome came with the specific Android version. I guess one thing where you can help is to look up which Chrome comes with which Android version.

I'm using the simple DOM contenteditable because that's what Ian used in his examples.

I was considering writing a web App that would store all this information in a database so I wouldn't have to screenshot everything and we'd also have the raw data as JSON somewhere. I'm not sure how long that would take though so for now I'm just manually see if I can get this all done.

I'm surprised how big the differences are between each Android version.

thesunny on 13 Aug 2018

@ianstormtaylor Just a heads up that I'm going to redo your Input Tests adding the multiple Android versions.

Edit: Completed tests that don't require blocks as the Input Tester doesn't support them.

Edit: I'm going to go back and add tests for Android API 27. It's a minor upgrade Android 8.0 - 8.1 but just to be safe.

Edit: Also missed Android API 24 (which was mistakenly API 23). Fixing with an @v1.1 to identify that it's been fixed.

Edit: Forked input-methods so that the contenteditable contained two <p> blocks with some text in it in order to complete the tests. https://thesunny.github.io/input-methods/index.html

thesunny on 13 Aug 2018

👍1

@thesunny these are absolutely amazing! Thank you!

Having them for multiple versions of Android is a great idea. And using the non-React input is best too, since we need to know what exact DOM events are fired, regardless of how React chooses to interpret them. Because we'll probably need to bypass React's event handling logic in places, until they catch up.

Another question for us to answer would be what are the usage levels of the different Android versions.

Regardless, starting by trying to get things working with the latest versions that have Chrome's that fire beforeinput seems best, since that makes things easier I think. There's a decent change that adding support for some of the older ones would be much harder.

ianstormtaylor on 14 Aug 2018

👍1

@ianstormtaylor popularity of platform can be found here but unfortunately beforeinput support starts at API27 which has only 2% market share.

If we go back to API 23, we can capture 66.4%. The biggest version in distribution is API 23 which has 23.5% distribution.

I'm just looking through some of the charts and looking for some patterns.

In Inserting Text I think we may be able to get away with:

Ignore keydown events. This is because a keydown will happen before a compositionstart so we can't know if a keydown is part of a composition until later. We can't know if we want to process based on the keydown event.
We process all keyup events that don't happen within a compositionstart and compositionend using a DOM diff. In the test, this is always a space (presumably because it doesn't require composition since it's valid in itself).
When there is a compositionstart then we ignore all the keystrokes. When a compositionend event happens, then we do a DOM diff to create a Slate insert operation.

Some analysis I did on other input methods to see if this strategy holds:

Japanese IME: It appears valid for IME in API 28 as well.
Gesture typing: appears valid for gesture typing
Moving to the middle of a word then inserting text. This appears to work but in some API versions, the text is wrapped in a composition and in others they are just simple key events with no composition. Either way, the logic still works.

It seems like the code might be written something like:

For Android only, we allow all regular insertion keypresses to all flow through naturally (not including ENTER, BACKSPACE, etc)
If we find a compositionstart event, we block all operations until a compositionend event materializes. At the time o f a compositionend, we do a diff of the DOM and create an insert op with {isNative: true}.
If we find a keyup event that is not wrapped in a composition we also do a diff and create an insert op with {isNative: true}

If we aren't in Android, we could theoretically do a diff and use the same code base for everything; however, this might be a performance issue. On the other hand, it could potentially help performance if we debounce key events and end up with a bigger diff and one insert operation at time of keyboard rest. I think we should think about this.

Of course we'd still have to handle ENTER, BACKSPACE, etc. individually but we can probably make a short list of operator keys which is probably ENTER, BACKSPACE, SPACE, and PUNCTUATION.

Usually we don't have Plugin actions from typing in Slate unless they are accompanied with one of these keys anyways. For example, a markdown plugin might use * for a bullet or #heading but all of these are punctuation. I also use o for an empty task myself but that has a space in it. So we would have to limit special Plugin handling to be around these keys but that might be acceptable and it's also a good proxy for the special handling to work in Android anyways because Android will always composition normal words so that key handling would fail in Android.

The plugins may need a different API though that exposes something like onInsertText and onNonWordKey. In this API \we can't do something like have the input onetwothree be converted on the fly to 123 as we type it but we could have one two three work because of the spaces calling an onNonWordKey. This may be a feature since onetwothree live converting could never work in Android due to the way it composes.

thesunny on 14 Aug 2018

👍3

@thesunny great points all around.

It sounds like one thing we need is a list of key presses that don't trigger compositions and instead use keydown to insert directly? It sounds like space is one of these? But we need a full list. (And as you mention, plugins that want to support Android would need to restrict their keydown usage to these keys, or we need to come up with some new abstraction later.)

Do we need to worry about isNative: true for now? Or can we re-render once the composition has finished with compositionend? Being able to punt that (even just for getting it working with later Android versions) would make it a lot easier.

Fair enough for backwards compatibility. I think whoever implements it would likely want to start with API 28, since it's the most standardized. Or even just ignore Android quirks to start and focus purely on the compositionstart/update/end trifecta. And then handle all of the specific quirks around compositions that need to be ignored.

ianstormtaylor on 14 Aug 2018

@ianstormtaylor

I'll do some more research on what keys have special cases with respect to composition like space. I assume enter is one but I'm not sure about punctuation.

That's an interesting idea about not worrying about isNative: true. So I guess the idea would be that we simply would not update the state (and hence not trigger a re-rerender) until compositionend? Feels like this could work...

In my opinion, we should focus on the compositionstart/end events. Everything I've read on this issue across all of these Editors suggests to me that it is impossible to use the events to reconstruct Editor state without doing a DOM diff with reasonable coverage of Android versions. Since we have to go there eventually, why not start with it. I've designed an algorithm for the DOM diff which I think (hope) will be quite simple to implement so it may not be that hard to get this working.

My next goal is to see if I can create a sample contenteditable React-based editor that uses compositionstart, compositionend and DOM diffing to sync a Slate like Editor State. If I can get that working, we can see how that can merge into Slate proper.

Edit: A preliminary check just on Android 28 suggests that punctuation doesn't trigger composition.

thesunny on 14 Aug 2018

👍1

@ianstormtaylor @Slapbox

OMG! I'm super excited!

I created a working prototype. It's a simple React contenteditable editor (one text node in a div) and tested this on three Android versions (including the newest and oldest API versions) and so far it works perfectly!

composition works
IME works
gesture writing works
auto correct works
suggestions work

It's generalized enough that it works on desktop with no code changes.

I created a custom editor because I don't know Slate internals well enough yet but I tried to make it as close to Slate as possible.

It uses insert_text and remove_text ops with the same semantics as Slate (but without path and marks to keep prototype simple)
They are applied to a text state (similar to Slate's editor state but simplified to one text node).

It works like this:

It reconciles on keyup unless the keyup is within compositionstart and compositionend events.
It reconciles on compositionend events
It uses text diffing to generate the operations and applies them to Editor state

I also track selection and restore it after update but Slate already does that so can be ignored.

Here's a screenshot for now...

I'll try and get this published tomorrow to get some feedback. There will probably be issues but it's promising so far!

thesunny on 15 Aug 2018

🎉12 👍10

@ianstormtaylor

Here is where you can access the Live Composition Editor

This is the GitHub Repo for Composition Editor

If you type in it from most browsers, it is diffing on every keystroke to create the insert operations and then the Editor is being rendered. After render, I set the selection so the cursor doesn't get lost.

If you use it from Android, when you start a composition via the compositionstart event which happens for most input, the Editor waits until compositionend and then does the diff and creates the operation. You can see this in the operations dump as instead of a single character insertion, you'll see an insert_text with multiple characters in it. In some cases you will also see a remove_text.

I dump the Editor state for transparency which includes all the ops or operations that have been applied.

You can reset the editor to the default text and empty the ops by clicking the Reset button on the page.

@Slapbox can you test this on your Android device?

thesunny on 15 Aug 2018

❤4 👍2

@ianstormtaylor

I think to implement this, we would have to change the Plugin event handler properties.

Here's my recommendations.

Add an onCompose synthetic event handler property on plugins.
deprecate onKeyDown and onKeyUp because it will be unpredictable in Android and could break it

We could have onCompose be the default handler for insertion of characters even for desktop where they'd always compose 1 character at a time. We could consider calling it something like onInsertText but the issue is that in some cases (like switching a word using a suggestion) we'd actually have a remove_text operation along with the insert_text operation. Maybe onChangeText? But onCompose feels okay to me even if it's a little misleading.

My suggestion is to replace the key handlers with a whitelist of specially named key handlers for all the cases we want to make sure Slate can handle. This is safer than having a catch all like keydown and assuming the user will know, for example, that letters in a composition won't fire the event or that they need to not preventDefault on those events or that they shouldn't modify Slate during a composition.

Here's handlers I'd recommend to replace onKeyDown.

onHotKeyDown which would fire whenever a key combination (i.e. a character in combination with CTRL, ALT, OPT or CMD is pressed) like CMD+B or CMD+SHIFT+2.
onSpace
onEnter
onDeleteBackward
onDeleteForward
Or possible combine the two above into onDelete with an event property that specifies the direction? Not sure but I kind of like them separate so as to avoid confusion...

I will look into punctuation but I'm not hopeful that we can rely on these being 100% handled outside compositions because of contractions like can't and isn't.

thesunny on 15 Aug 2018

👍1

@thesunny I think it might be best to separate the work of "getting Android working" from "getting the plugin system fully adapted for compositions". Because the latter is going to be more work, and require thinking lots of different API considerations through.

As for how events flow through the editor...

The <Content> component is the one that renders the contenteditable element in the DOM, and attaches events handlers when it does.
For handlers where we need the native DOM semantics, instead of React's synthetic event semantics, we have to attach them in componentDidMount to the DOM node.
The <Content> component itself is rendered in the After plugin's renderEditor hook, and it passes in the <Editor>'s handler callbacks.
The editor takes any event and runs it through the stack which cascades through all of the plugins.

That's the current flow. There's a bit of confusing indirection around how the <Content> element is rendered. But for the most part you can just look directly from <Content> to plugins and ignore it. In the future we may be able to simplify that stuff.

As for how to handle compositions in plugins. I don't think we need to deprecate onKeyDown/Up yet, because Android helpfully fires the keys with 229 codes so that they won't ever be recognized, last I checked. There may be cases where this doesn't work, but those should be researched/documented well before we make a decision.

I'd rather avoid having onSpace, onEnter, etc. all be their own handlers. As for onDeleteBackward and onDeleteForward, see the "commands" concept discussion.

ianstormtaylor on 15 Aug 2018

👍1

@ianstormtaylor Thanks for the feedback.

Just working my way through everything now.

thesunny on 15 Aug 2018

Here is my execution plan for now

[x] disable state updates in Slate from relevant events
[x] allow native updates to flow through to contenteditable div
[ ] generate diff by locating relevant slate node and comparing it to DOM node (attach initially to keyup events)
[ ] use diff to create operations to modify state and rerender
[ ] now do it with composition events
[ ] enable back all the event handlers and such that are necessary to make slate work properly like hitting enter key

Testing
For all these tests we need to make sure (a) the changes are in the Editor State (b) the selection is in the right spot

IME

[ ] go to end of a word and type hideki in Japanese
[ ] go to start of a word and type hideki in Japanese
[ ] go to middle of a word and type hideki in Japanese

Gesture Writing

[ ] cursor to end of word then use gesture writing
[ ] cursor to start of word then use gesture writing
[ ] cursor to middle of word then use gesture writing

Regular Writing
[ ] enter a full word then press space
[ ] enter a full word then press enter
[ ] enter 2 letters then select a suggestion
[ ] type "hi" at end of line then hit "enter" (broken in API28)

Suggestions
[ ] click on a word and select an alternate suggestion
[ ] click on a word, type a letter, select an alternate suggestion

Auto-correct
[ ] type cant and let it auto-correct to can't
[ ] type can and cursor away. Move cursor back to the end of can and type t and then let auto-correct fix it to can't

Physical Keyboard
[ ] type "Hello World" pretty quickly. Make sure there are no duplicate characters or incorrectly positioned selection.
[ ] hold the backspace key down

thesunny on 16 Aug 2018

👍1

@thesunny is there anything in particular I can be of help testing? I checked out what you've put together and it looks very useful. All input and output is rendered to the editor as expected.

One note, not a problem of any sort; I'm surprised by how backspacing is handled. Backspacing away the last character of a completed word fires a single remove_text event, and then another one doesn't fire until you have removed the entire word. Compositions are strange.

Slapbox on 16 Aug 2018

Thanks for working on this. Please do test with Firefox on Android too. We are looking to update Firefox's behavior (see discussion in https://github.com/w3c/uievents/issues/202) and would appreciate your feedback.

birtles on 16 Aug 2018

👍4

@birtles You're welcome. I'll test out Firefox once I get Android Chrome working.

As a quick update, I have many things working properly now although I'm not using the diffing algorithm yet and just the replace algorithm that currently exists. It's pretty much working except there are issues with selection placement, the fact that Android doesn't always have an onCompositionEnd event (which makes no sense) and also the composition events are fired before the content is actually in the DOM.

I will add the special cases we need to test for in the execution plan above.

thesunny on 16 Aug 2018

🎉4 👍4

@thesunny can you give an example of a situation where onCompositionEnd doesn't fire?

Slapbox on 17 Aug 2018

@Slapbox That's a great question.

One example is if you start typing anywhere in a contenteditable and then cursor out of the word to another word or even to another paragraph. There is no compositionend fired in this case even though the word is finished composing.

Technically, editing anywhere in the document could be considered one composition, but my algorithm (and I think most editors out there) depend on a composition happening within a single node. Otherwise we would need to reconcile the entire DOM to Editor State after each composition instead of just one node.

One issue that is slowing me down, probably the thing that is slowing me down the most, is that Android Studio (what I'm using to emulate mobile Chrome) slows down and crashes regularly. Sometimes the browser behaves differently then after a reboot it goes back to normal. I think it may also be behaving differently when I'm using the remote console/debugger with it. This is frustrating as I have 6 instances running in order to test properly. If anybody has any suggestions on making this more stable, please let me know. Maybe I just need to buy 6 Android tablets!

thesunny on 17 Aug 2018

👍1

I've been thinking about how to remedy the issue of difficulty emulating, but I've come up with nothing unfortunately. You're running 6 emulators though? It seems like _just_ 2-3 should do.

Regarding the issue with no compositionend event in those cases, we could could compare the composition on each update of the selection of the editor, and when the composition is updated to have an entirely new word, we can assume we should output the fragment from the old composition before carrying on. Does that remedy this problem and can you see any issues that introduces?

Slapbox on 17 Aug 2018

@Slapbox

Right now I'm just trying to get through 2 emulators at a time but all 6 versions work differently so I need to test on all of them eventually.

Unfortunately, the compositionend issue is complex. Updating on every compositionupdate doesn't solve the problem without introducing new ones.

Right now I'm building a shouldChangeText object that provides different logic on when the text should be changed in the editor state. I'm probably going to write a version for each API version of Android and a default version for everything else.

I was joking before but decided to buy 3 Android tablets for testing. If they seem to work well and we can upgrade/downgrade them to different API levels, I'll probably end up with 8 tablets for testing at different API levels.

thesunny on 17 Aug 2018

👍2

@thesunny, first, thanks again for all your hard work!

Which APIs are you aiming to support currently? Limiting it to 28+ seems like it should cut the number of versions you need to worry about quite a lot, no?

Slapbox on 17 Aug 2018

Hey folks — just a heads up that I'm currently in the process of going through React DOM bugs. If you have something actionable for React that we need to fix on our side please create an issue with suggestions and I'll try to help.

gaearon on 17 Aug 2018

👍5

@gaearon thank you for your offer of help.

Ian listed three React issues in his first post above which I'm sure he'd be grateful to have looked at:

React issues: https://github.com/facebook/react/issues/4079 https://github.com/facebook/react/issues/6176 https://github.com/facebook/react/issues/11231

For all of the contenteditable writers out there, I'm sure we'd all love React to fire a single event which notified the end of a composition reliably and an input method that doesn't fire in the middle of a composition.

I'm working on that logic now but it seems a pity that every contenteditable author will have to figure this out on their own.

I'm not sure how high up this is on your list (I know you have Draft.js at facebook which I have used and contributed to) as it might be too specific to put into a general React release.

@Slapbox

I'm aiming at API 23+ but I'm starting with 28 and 27 right now..

The emulators are killing me so I am buying 3 Android devices to start which I might expand to 8. API 28 adoption is non-existent so not a good target for me personally as I want to support general Android with my web app. API 23+ captures 66% of market share. I can buy a brand new Android Tablet today at Best Buy that comes with API 23 (Marshmallow) so that feels like a good minimum bar.

thesunny on 17 Aug 2018

❤1 👍1

Do you think https://github.com/facebook/react/issues/4079 and https://github.com/facebook/react/issues/6176 are closed incorrectly and we have something actionable for React itself there?

I'm not sure how high up this is on your list (I know you have Draft.js at facebook which I have used and contributed to) as it might be too specific to put into a general React release.

I think our general stance is that if it involves a lot of code and is specific to contenteditable it's more likely to end up in Draft (or somewhere else) than in React core because we have constraints on React bundle size.

That said we can take small fixes to React.

gaearon on 17 Aug 2018

👍1

@ianstormtaylor

A few questions I'm hoping you can help me with:

(1) after each render in Content the updateSelection method is called. It seems to set the selection in the browser after every render even if the selection in Editor State hasn't changed. This is problematic because render sometimes happens before I've updated the selection. My questions are (a) is this true that it will set the selection even if it hasn't changed and (b) if it is true, do you have any objection to me writing code that checks if the selection has changed and only updating it if it has. Not sure what the side effects might be. Maybe there is a reason it's this way.

(2) are there any other places other than the event handlers in after where Slate's Editor State is being updated? Related to (1) in that I am getting re-renders but I can't figure out what's causing them. I feel like I intercepted all the relevant after methods. If I don't get any unwanted renders, I may not have to worry about (1).

thesunny on 18 Aug 2018

Just a heads up to everyone that I won't be able to work on this for about a week so you may not see any (or little) activity here. I'll be back into this issue the week after.

thesunny on 18 Aug 2018

❤1 🎉1 👍1

Just some good news before I go. I have it working as a proof of concept in API28 now with a few selection issues. I removed a bunch of the existing after.js plugin code temporarily so I had less things to think about but text entry including suggestions is working. IME and autocorrect are also working but the cursor is in the wrong place which is what I'll be working on until I see you all next. :)

thesunny on 18 Aug 2018

🎉5 ❤2

@gaearon sorry about that, I thought these were issues that Ian wanted to fix. I mixed it up with the following which is what I should have posted. He mentioned in this Slate issue https://github.com/ianstormtaylor/slate/issues/2060 that he was trying to get this React issue solved to do with using native beforeinput https://github.com/facebook/react/issues/11211

thesunny on 18 Aug 2018

@thesunny feel free to make any changes to the selection updating logic. Right now it does update on every render (although I think it checks to see if it needs to, not sure), but there's no reason we have to do that specifically. It just needs to ensure that the selection doesn't get out of date.

Not sure about the other places for re-renders.

@thesunny @Slapbox @gaearon I think it's pretty reasonable for React to not include any contenteditable-related conveniences. This is the domain of Draft.js and Slate to provide these things, and if they happen to be shareable then they should be separate libraries. Trying to paper over compositionend for example is something that would be super complex (if not impossible) for React to manage, and <1% of their users will ever care about it. This definitely doesn't belong in React, and belongs in Slate instead.

@gaearon as for browser bugs, I'm not sure we have bugs specifically. But one of the big issues right now that we face with React is the discrepancy between real and synthetic events. For example...

https://github.com/facebook/react/issues/11211 - The native beforeinput is not accounted for in React right now, which requires us to attach our own listener, which behaves slightly differently. This seems like a fairly easy fix in that it just needs to update the event plugin to check for the real beforeinput event in browsers that support it (Safari, Chrome, Opera, iOS Safari, Chrome for Android). This would be awesome to get support in React core.

https://github.com/facebook/react/issues/5785 - Similarly, the onSelect handler for React has some limitations, and we're forced to attach our own selectionchange listener instead to get around them. It would be nice if React exposed the native selectionchange and selectionstart event handlers I'd think, since these are supported across most browsers going back a long ways. The only complication here might be the high frequency of them being triggered? I'm not sure what the limitations on the synthetic event pool are, but this might run into them.

https://github.com/facebook/react/issues/13104 - This is another one. If React added the beforeinput from above, we'd still need to dip into the nativeEvent right now to check the isComposing flag I think.

ianstormtaylor on 21 Aug 2018

👍2

@ianstormtaylor @Slapbox @gaearon

Hi Ian, I understand the reasoning for not including any contenteditable specific fixes; however, I'm not sure composition events shouldn't be normalized in React as they also apply to input elements.

React normalizes other events across browsers in order to make sure that writing an app using React events means your app will more or less work on all platforms.

It feels inconsistent to support composition events in React but them not being portable across browsers like the othe React events. So, in order words, my instinct is that composition events either (a) shouldn't be handled at all in React which indicates developers need to do the work to make them cross platform compatible because React isn't involved or (b) that React normalizes them so developers don't need to worry about them.

At this point, any app that uses composition events to propagate changes will fail on at least one version of Android.

One thing that perhaps should affect the decision on whether to consider fixing/augmenting/creating new synthetic events is how complicated the algorithm actually is. Figuring out how to fix this in Slate has been hard but I think there's a possibility that the solution is actually quite simple and elegant. It may only be a few lines of code and save a lot of headaches.

thesunny on 21 Aug 2018

👍3

@thesunny that's fair, if there's a simple solution for it. From the threads I've read about Android composition events though it sounded like things weren't super simple, but I don't have the context.

ianstormtaylor on 21 Aug 2018

@ianstormtaylor it’s not clear to me yet whether the solution is simple but there’s nothing I’ve seen so far that prevents it from being so.

The issue I’m seeing with compositionend is that in some implementations it either fires too soon (before the input) or doesn’t fire at all and only fires a compositionupdate.

In the first, I call set timeout but only use the callback if input event doesn’t fire In the same tick. If input does fire then I call the callback from the input event. Logic is tiny and it works.

In the second case a compositionupdate fires but I haven’t yet analyzed the logic to see when that should have been a compositionend. Hopefully it can be solved and is simple (it may require logic to do with selection position).

thesunny on 23 Aug 2018

👍2

For what it's worth, we're looking to change the order of compositionend and input in Firefox to match Chrome. (We're just waiting to clarify what value of isComposing would be most useful to Web authors for this series of events.)

birtles on 24 Aug 2018

👍1

@birtles do you know what the status is on implementing the beforeinput event from Input Events Level 2 (or even Level 1) is in Firefox? That’s one of the biggest issues.

ianstormtaylor on 24 Aug 2018

👍1

We're hoping to tackle beforeinput in Q4. From what I understand it's a pretty massive undertaking though. We should get inputType done this quarter, though.

birtles on 24 Aug 2018

👍2

@ianstormtaylor do you know why there is a a setState({isComposing: false}) in the before.js plugin? I presume it's required for a certain browser?

If I take this code out, my Android code in API28 appears to work. If it's in there, it fails because the selection is not where it's natively supposed to be (presumably because the re-render places the selection where it used to be).

Right now, I'm planning to disable it selectively (i.e. for Android only) but kind of feel icky about it.

  /**
   * On composition end.
   *
   * @param {Event} event
   * @param {Change} change
   * @param {Editor} editor
   */

  function onCompositionEnd(event, change, editor) {
    isStrictComposing = false
    editor.isStrictComposing = false
    const n = compositionCount

    // The `count` check here ensures that if another composition starts
    // before the timeout has closed out this one, we will abort unsetting the
    // `isComposing` flag, since a composition is still in affect.
    window.requestAnimationFrame(() => {
      if (compositionCount > n) return
      isComposing = false

      // HACK: we need to re-render the editor here so that it will update its
      // placeholder in case one is currently rendered. This should be handled
      // differently ideally, in a less invasive way?
      // (apply force re-render if isComposing changes)
      if (editor.state.isComposing) {
        editor.setState({ isComposing: false })
      }
    })

    debug('onCompositionEnd', { event })
  }

thesunny on 28 Aug 2018

@thesunny the comment above says why, it's to re-render the placeholder so that it will disappear if there is content in the editor I think. If you search issues there's probably a reference to compositions and placeholders that someone opened.

ianstormtaylor on 28 Aug 2018

@thesunny searching leads to: https://github.com/ianstormtaylor/slate/blob/36ed4397d880bf75126e9f8f890e999fc69c3b9a/packages/slate-react/src/plugins/after.js#L740

ianstormtaylor on 28 Aug 2018

@ianstormtaylor sorry, I'm not clear what the placeholder is.

thesunny on 28 Aug 2018

👍1

@ianstormtaylor Never mind... I see it now. I thought it was a special Slate construct that was either (a) a placeholder for the cursor in collaboration or (b) a place where in inComposition items were built. But, I see that it's just the DOM placeholder. :)

thesunny on 28 Aug 2018

👍2

Just an update, I have a working version in API28 now. Going to try API27 now...

edit: there were 2 unhandled edge cases with API28. Fixed one. Working on second (compositions that take place across multiple nodes). It's going slower as we started Y Combinator's Startup School which takes up time.

thesunny on 29 Aug 2018

🎉4 ❤2 👍1

@ianstormtaylor Can you help me? I've been stuck here for several hours but have narrowed it down to this:

This is document value from change.value:

Then this change.insertTextAtRange operation happens:

And finally we get this:

What happens is the second paragraph gets deleted (set to "") and the text "Hello there" is moved to the end of the first node. The target of the change.insertTextAtRange though is at path [1, 0] which is the second paragraph.

The place where the Hello there was inserted was where the DOM cursor was set at the beginning of the composition but I'm not sure why the cursor position at that time should have an impact on the change.

thesunny on 30 Aug 2018

👍1

I noticed at the time that Slate transitioned to points that there was some weird behavior regarding line-endings. Is there any chance it's related to a bug like the one I reported here? https://github.com/ianstormtaylor/slate/issues/2050

Slapbox on 30 Aug 2018

Thanks for chiming in @Slapbox . It helps when you dive into these issues.

I figured it out but the method behavior is not intuitive and I suggest we change it.

When you insertTextAtRange it deletes at the argument range then inserts text at the value's selection range if the current value's range is in a different node than the argument range. This doesn't make sense. It must be used to solve an edge case somewhere but it's bad behavior.

The method is named insertTextAtRange and it's not what it does.

thesunny on 30 Aug 2018

👍1

@thesunny you're right, that sounds like a bug to me. It should use the passed in range, and never the value.selection.

ianstormtaylor on 30 Aug 2018

👍1

There is an Android issue that I'm not sure we can fix.

I edited "rich" to change to "Rich" using Android's auto-suggest and it returns this in the DOM:

It obliterates the <span data-offset-key="3:1"> and only keeps the <b> tags.

Any ideas? I'm wondering if this is even fixable without some really hard work to rebuild the DOM manually.

I have to say, this is really really hard, most of all because the time to check a piece of code after writing it is about 30 seconds. Android studio is slow, requires a hard reload, and crashes often requiring restarts and reconnecting the debugger.

thesunny on 31 Aug 2018

Wow it even changes it from a <strong> tag inside to a <b> tag huh?

To add context for anyone reading, this is the before state of the DOM:

This is definitely something that's fixable, but yeah, the only solutions I see on the face of it are really ugly. Hopefully @ianstormtaylor has some better thoughts on how to manage this?

@thesunny can you describe your current testing setup? What's the oldest API you're trying to cover, and are you testing with each API version from that one up to the present? I guess, in short, how many emulators do you need to run and how many physical devices do you have?

Is the trouble with Android Studio on Mac or Windows?

I don't necessarily have anything to offer, but it seems that it would help if you had more physical devices. Does that seem likely? I don't know how Android Studio behaves with multiple devices connected though.

If you think that would help, which APIs do you need physical versions of? We can try to get some unused Android devices together to cover that need.

Slapbox on 31 Aug 2018

@Slapbox

I have a few Android devices but I couldn't get the Chrome debugger on the desktop to connect so I've been using Android Studio and restarting often.

My proof of concept React Editor (not using Slate) was set up with 6 different versions and all the basics worked. With Slate, I'm stuck on two issues with API28 so haven't moved on to earlier APIs. I'm fighting with Slate which is great but doesn't have the necessary abstractions so I'm creating new ones along with unexpected Android behavior which really makes no sense whatsoever. Android Chrome is worse than fighting early versions of IE.

My two biggest problems right now are:

Android blowing away React nodes as shown above when the composition includes the entire node
Android not calling compositionEnd when you move the selection in the middle of a composition

I had other problems which mostly boiled down to the incosistent state of the DOM and given events which I resolved by modifying the input tester which you can see here https://thesunny.github.io/input-methods/index.html which now shows both the DOM state at a given event as well as the selection position.

That compositionEnd not being called is tricky because I cannot re-render the DOM while the selection is on it because it moves the selection to the beginning of the node. So, I tried to update Slate's state on the input after compositionUpdate (which is when the DOM is correct) but then the cursor gets forced into the wrong place by Android.

My solution which I will try is to wait for the cursor to move to its new location, remembering where the last cursor position is, updating the state in the place where the cursor previously was. Hopefully this won't reset the cursor position because it's in a different node... however, I haven't tested it yet so don't know if it will work. To get this to work I have to create another abstraction which will allow Slate to update itself from an arbitrary Slate node.

I feel like this wouldn't be that hard if it weren't for the fact that I'm working in molasses. It's hard to remain focused when everything takes so long.

thesunny on 31 Aug 2018

👍3

@thesunny let me know if there's any way I can help. Tinkering in slow motion is exasperating. Are all the Chrome debugger/Android Studio issues on Mac or Windows? On Windows 7, I've found them to be somewhat buggy, but not terribly so.

Slapbox on 31 Aug 2018

It’s on the Mac. Maybe I should try windows. How is the page reload speed?

thesunny on 31 Aug 2018

Update:

I solved the Android issue where the compositionEnd won't fire if you move the selection after starting a composition. Essentially you can get a composition that spans several nodes. I solved it by keeping a Set of all the nodes that were touched during the composition. When the input after compositionEnd fires, I then go and resolve all the nodes against the Slate DOM. Seems to work.

Now I'm at the issue (final issue that I've seen) of the React nodes being removed when you choose a suggestion for a word that is wholly within a node like <b>bold</b>. In this case Android fires all the events outside of the composition so if your cursor was between o and l and you wanted to switch it to <b>Bold</b> it would do something like compositionEnd, delete, delete, insert "B" insert "o". But it would do it all simultaneously with the events firing in a weird order (keydown, input, etc. are out of order for some reason). I have to keep trying things until it works but I have a few angles I'll try.

So, I said I'd put in 1-2 weeks and this is 2 so I'll keep plugging but won't be full-time. Sorry it's been slow but Android is slow to develop with and frankly the wait events fire doesn't make much sense. This only solves API28 but my feeling is that a lot of the abstractions necessary to fix the problem are there now so the other APIs should be easier to tackle.

thesunny on 1 Sep 2018

👍11 ❤5

We're currently having issues with the long-press keyboard behavior for diacritics on macOS and I was wondering if this could be a fix for that.

Our problem is with this feature:
screen shot 2018-09-17 at 3 35 43 pm

When picking one of the options, the original letter that was typed doesn't get replaced, resulting in two letters which is not the desired behavior.

(Also, is there a simplified version of the code somewhere that works on major desktop browsers and that we could try to experiment with to see if we could help with the non-Android bugs that have been reported throughout the thread?).

quentez on 18 Sep 2018

Hi @quentez

This is a candidate to fix that problem.

These are the main concepts in understanding what problems it can solve:

It stays out of the way while the browser does its thing while composing content
At certain times, Slate will issue a command that will (a) update Slate's internal document to match the current state of the contenteditable div (b) because the state is updated, it forces React to reconcile the DOM. Assuming Slate's document is the same as the DOM, this should make no changes to the document.
The problems occur in figuring out when those moments are when we can sync the editor's document up to the DOM. For example, in the middle of a composition, if we update the internal document forcing the re-render, the composition mode gets broken. This is trickier than it first seems because compositions can sometimes span multiple HTML elements (this in my opinion is a bug) and sometimes don't sync well at the compositionend event. But I think this problem may be fairly straightforward (if time-consuming) now that the basics are in.

The problem you cite appears to fall into the category of solvable issues so long as you sync the document after the modified character is inserted and not before.

I do have a proof of concept here:

https://github.com/thesunny/composition-editor

With a demo here:

https://composition-editor.now.sh/

Can't remember how far along it was though so not sure if it would be useful for you or not.

thesunny on 20 Sep 2018

👍1

@thesunny, sorry I haven't had a chance to test your work still. Still having trouble updating my Android version to the proper API.

Slapbox on 20 Sep 2018

Mini Update:

I posted a short update in Slack which I'll summarize and expand on here.

Our project is prioritizing mobile which includes Android support. I will start again on Android within 2-8 weeks and I estimate 2-3 weeks of work which includes:

Create a dev environment with native devices. Device emulation slows me down 2-3x and crashes often.
Create an event logging tool specific to Slate. The modified Dan Burzo tool I created differs enough from Slate as to be confusing.
Create a mobile demo for easier mobile testing
Start from a clean version of the repo because there is too much testing and experimental cruft in my current version.
Do a clean rewrite of the current helper functions and modularize them.
Implement as much as I can as a plugin
Create a pseudo compositionbegin, compositionchange and compositionfinish event and block all other events in between for the Android platform. These events roughly equate to (a) we guarantee that this is the state before the composition started (b) something might have changed within the composition so note the cursor position here (c) we guarantee that the composition has ended and that the results of the composition have been commited to the DOM so it is safe to update Slate State and re-render. These roughly equate to the idea of compositionstart, compositionupdate and compositionend but abstracts away the problems and inconsistencies between them. Note: We may wish to implement this across all platforms.

thesunny on 1 Oct 2018

👍14 🎉9 ❤5

A few different resources

Here's one more that I built recently: https://input-inspector.now.sh/

It's inspired by https://github.com/danburzo/input-methods, and has a couple nice additions:

A record of all DOM mutations
Shareable URLs

Source code is here, if you're interested. Hope you find it useful! ✌️

javan on 13 Nov 2018

❤4 👍3 🎉2

I want to shed some light on Android OS versions and how those relate to Chrome and embedded WebView's in an app. I noticed some discussion here about testing and supporting particular OS versions, but I don't believe that's a great path for producing reliable input or tests.

The more relevant piece of information is the WebView or Chrome version installed on the device. I've outlined below how the Android OS versions relate to their WebView counterpart. (I've started at Android 4.4, because I don't believe it's worth anyone's time to support Android 4.3 and lower.)

Android 4.4

Starting in Android 4.4, the system WebView component is based on Chromium. https://developer.android.com/guide/webapps/migrating

WebView: shipped with and locked at version 30.0.0.0
Chrome: latest available (currently 70.x)

Android 4.4.3

WebView: shipped with and locked at version 33.0.0.0
Chrome: latest available (currently 70.x)

Android 5.x - 6.x

Starting in 5.0, the WebView component became unbundled from the OS and regular updates are provided via the Android System WebView app in the Play Store. https://play.google.com/store/apps/details?id=com.google.android.webview

WebView: latest System WebView available (currently 70.x)
Chrome: latest available (currently 70.x)

Android 7.x - 9.x

Starting in 7.0, the WebView is no longer backed by the Android System WebView app, but instead uses the installed version of Chrome on the device to render embedded WebView's. https://developer.android.com/about/versions/nougat/android-7.0#chrome--webview-together

WebView: latest Chrome available (currently 70.x)
Chrome: latest available (currently 70.x)

Support across OS versions

Given the self-updating nature of Chrome and embedded WebView's on Android 5.x+, it'd make sense to focus on those devices. A (relatively new) minimum Chrome/WebView instance could be required as a baseline based on how far back you want to support.

With all the well known composition event issues on Android, Level 2 beforeinput / input events seem like a good path forward, like @javan mentioned. These events are supported in Chrome 60+ and Android System WebView 60+ across all Android OS versions 5.0 and up.

Javan has started implementing a separate Level 2 events controller for mobile in Trix, and so far it's looking really promising. Here's the progress so far: https://github.com/basecamp/trix/compare/level-2-input. The basic gist is handle beforeinput, and implement a method for each event.inputType https://www.w3.org/TR/input-events-2/#interface-InputEvent-Attributes

You can test if Level 2 events are supported in a particular browser/device by running this in the console:

(typeof InputEvent != "undefined" && 
  "inputType" in InputEvent.prototype && 
  "data" in InputEvent.prototype)

jayohms on 27 Nov 2018

👍3 ❤2

@jayohms thank you so much for that detailed write up, I really appreciate you taking the time to share your Trix learnings! Starting with only the versions that have auto-updating behavior sounds like an extremely reasonable approach.

Question for you, does Chrome really support Input Events Level 2? Everything I've read so far stated that Chrome supports Level 1, and Safari supports Level 2. (Although my testing revealed that some of the Chrome events were cancelable, so maybe they really are at Level 2 and just haven't said so?)

ianstormtaylor on 28 Nov 2018

👍1

Yup, they're supported in Chrome: https://www.chromestatus.com/features/5656380006465536

javan on 28 Nov 2018

@jayohms - Thanks a ton for dropping in with that knowledge! That's a huge help.

@javan - Perhaps you can shed some light - The issue you linked to links out to the W3C specification for Input Events Level 1, but if you go to that specification it shows this:

That's encouraging, but it doesn't seem to explicitly state that Level 2 is supported and the Chromium issue it links to doesn't provide any further insight. Is there any additional information you know of that we can reference to confirm Level 2 support? That would be a dream.

Slapbox on 28 Nov 2018

I think they just have an old link on https://www.chromestatus.com/features/5656380006465536

https://bugs.chromium.org/p/chromium/issues/detail?id=585875 links to

http://w3c.github.io/editing/input-events.html
https://groups.google.com/a/chromium.org/forum/#!searchin/blink-dev/InputEvent/blink-dev/RrnitB0OElc/rirueVekCwAJ

I'm pretty certain Level 2 input events are here to stay in Chrome. Also, Safari supports them and Firefox is workin' on it.

javan on 28 Nov 2018

Chrome doesn't support level 2 yet. As Masayuki points out in the Firefox bug you linked to there is no support for "insertFromComposition", "deleteByComposition" or "deleteCompositionText".

Support for inputType should land in Firefox very soon (there was a lot of work to get to this point) with work on beforeinput coming after that. (There's still a number of things that need to be fixed in the spec first, however.)

birtles on 29 Nov 2018

👍1

Apologies, I think I have my terminology wrong. I was using level 2 to mean beforeinput/input events with inputType, data, etc. properties vs. the old generic input event without those properties.

What's the difference between level 1 and 2? At a glance, the specs look similar.

javan on 29 Nov 2018

Oh, I see:

According to the source code, Chrome implements Level 1.
https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/events/InputEvent.h?l=26-72&rcl=d4c3823079fe41cd1cbef1ffe2e8b24a173f5559

Since there are not definition of "insertFromComposition", "deleteByComposition" and "deleteCompositionText". So, perhaps, we should follow Level 1. I don't know how much stable of the specs, though.

Chrome doesn't implement those newer inputTypes for compositions, but it does implement "insertCompositionText". Here's a composition in Chrome (Level 1) and Safari (Level 2): https://input-inspector.now.sh/profiles/KXHldfRK9AtSj19nRR3O,q1nsTzNNdb8927yUR1vW. So, it's entirely possible to handle compositions in both browsers by responding all four composition inputTypes. I have it working in Trix: https://trix-level-2.now.sh/

javan on 29 Nov 2018

Last I checked I thought the big difference between Level 1 and Level 2 was that in Level 1 many of the events were not preventable using event.preventDefault()? So you were required to let the event happen and then try to figure out what happened. But maybe this is no longer true?

ianstormtaylor on 2 Dec 2018

@ianstormtaylor I think Level 2 encompassed quite a bit more than that. I wouldn't be surprised if some browsers implemented an ability to cancel the events without implementing the rest of the draft specification. That said, last I checked the Chrome team said they'd not be making any accommodations with regards to onBeforeInput and preventDefault. Last news I saw was about a year ago now though.

Regarding preventing events, see the first two notes in this section of the draft spec

Slapbox on 2 Dec 2018

Trix, for the most part, does not cancel events so the inability to event.preventDefault() Level 1 input events isn't an issue. Generally, our approach is:

Listen for events and translate them into editing operations on our internal document model
Render the document with every change to a detached element (we call it a shadow element)
Observe the editor element with a MutationObserver and compare it to the shadow element after each mutation
Replace the editor element's contents with the shadow element's unless they are equal

This way, we avoid touching/disrupting the editor element when contenteditable does the "right" thing, which is fairly often when typing normally. When it doesn't do what we want, we step in. We never have to "try to figure out what happened" since the internal document model is always the source of truth.

With Level {1,2} input events, step 1 becomes a lot easier. We just listen for beforeinput and translate the event's inputType, data, and getTargetRanges(). In browsers that don't support them we handle a dozen or so key and composition events.

javan on 2 Dec 2018

❤2 👍2

Firstly, sorry for missing the deadline for starting work on Slate Android support. Unfortunately, I've had more work to do on our app before I could start dedicated work on Slate Android. I will try and start around the new year. I've compiled everything I know and what my plan is and pasted them below in a kind of state of Slate Android support so you know where I am:

Sunny's State of Slate Android Support

I've tried to write this many times and failed because there are so many overlapping ideas. After many failed attempts, I think there's value in doing a brain dump even if it's imperfect. So, my apologies, if this comes out as a stream of consciousness post.

The following is a run-down on the current state of my attempts at supporting Android on Slate, what has failed or proved difficult and how I plan on implementing Android support.

I also felt the need to publish all of this because it informs the approach I'm taking at the end. Without this preamble, it would be hard to understand the approach.

1. The problem is hard to solve because of many incompatible Android versions.

Android support is hard. It is not hard like getting Chrome, or Safari or Firefox to work. It is more like it's hard like getting Chrome, Safari and Firefox to work. That is because there are at least 6 versions of Android we need to support and they all work differently. They fire different events in different order, in varying amounts and at different times. In essence, you aren't supporting one new browser, you are supporting 6. In fact, the differences between versions of Android are greater than the differences between actually different browsers in my opinion.

Also, unlike desktop browsers, Android browsers are tied to Android versions. It's easy to support the latest few versions of Chrome on desktop because you know they auto update. Mobile OSes, however, do not, so you end up having to support the last few versions in order to get any meaningful browser support.

Note that the most highest rate of usage of an Android version as of today is API 23 with 21.3% which is 4 versions away so even if you only wanted to support the most popular API version and newer, that's 5 versions. The next earlier version is API 22 with 14.4% and then after that is API 21 with 3.5% which I think is reasonable enough to cut off. The latest 6 versions encompasses 85.4% of Android API versions.

2. High number of input methods and situational edge cases

There are many different input methods that need to work to support Android propertly. International input methods, autocorrect, spellcheck correct, gesture writing and what I call word break insertion in which browser behaves differently when a word breaking character is entered.

Furthermore these scenarios may behave differently at the beginning of a word, in the middle of a word and at the end of the word as well as at the beginning of a line, end of a line and the beginning and end of a document.

Also, in some API versions a single composition can take place across multiple separate DOM nodes. That is, multiple words can be edited as part of a single composition which must be handled as a single composition or it fails. In my opinion this is a browser bug but never-the-less needs to be handled.

3. Inability to do partial support

Certain bugs appearing in Slate are inconvenient but acceptable even if annoying. For example, if a user's input fails in 10% of the input cases and that left the text a little garbled and then you go back and fix that garbled text, a case could be made that it would be acceptable. However, if 10% of the input cases left the editor in an inconsistent state in which errors compound and can never be recovered from without resetting the editor, that to me is unacceptable.

Our users get upset losing any amount of data. Imagine somebody composing a blog post and then losing their post. Android bugs tend to fall into the space of putting the editor in an inconsistent state which results in data loss and therefore, it's unfortunate but, partial support feels like a bad option unless the edge cases are truly edge (maybe 1%).

4. Don't construct Editor State from individual events

I've been advocating for this a long time, but for anybody else attempting otherwise, do not reconstruct Slate's internal state by individually reading events and trying to figure out what the user did in Android. It works in other browsers but it doesn't work in Android. Worse, it looks like it will work and in certain, very simple scenarios, it does work, but it always fails when some combination of different input methods or Android API versions are used and then you have to throw away all your Android compatibility work.

Just looking at the enormous differences from the events I logged between different input methods and Android API versions shows basically how hard it is. Honestly, I think it's actually impossible to make this work reliably because I believe there is not enough information provided in the events themselves to reconstruct in all scenarios.

5. Do Use DOM Reconciliation

Something I got working quite reliably (in one day actually) with a proof of concept editor I built was to use DOM reconciliation. Basically, you wait for a compositionStart event to happen, then you ignore everything else that happens until you get to a compositionEnd event. Then when you get one of those, you read the DOM, then you update the Editor's internal state using the DOM. If all goes well, React's DOM Reconciliation ends up being like a no op because the DOM matches the internal state. I did this successfully in my prototype also using transactions like Slate does. Then I also got this to work in almost all cases except one that I found in (I think) two versions of Android.

6. Nieve Reconciliation Fails

The version of DOM Reconciliation I built works under the idea that it is safe to ignore all the events after a compositionStart and that it is safe to read the DOM during a compositionEnd and to update the DOM immediately after a compositionEnd. Unfortunately, this is sometimes true but not always true. It is true and untrue depending on which Android version you are using and what you are doing. In some cases, it is true on an event that happens immediately after the compositionEnd but not on compositionEnd itself. Ultimately, it is unpredictable to know without testing every Android API version multiplied by every input scenario.

So one aspect of making Android work is to discover when what would be our ideal compositionStart and compositionEnd happens. We may wish to call these different events to distinguish them from the native compositionStart and compositionEnd. At any rate, as long as it follows our rules which is safe to ignore after start, safe to read at end and safe to do a React DOM refresh after then everything should work.

7. Must Support Multiple DOM Nodes in a Single Composition

Initially, I assumed that a composition must occur within a single DOM node. For example, if you started editing a word in a paragraph then clicked into another paragraph, I assumed that the first composition would end, then a second composition would begin in the second paragraph. In fact, this held true for the initial API version I tested against; however, in switching to a different API version, I found that a single composition could span multiple nodes.

So the solution must be able to know in which nodes the composition has taken place in, then reconcile all the changes across multiple nodes. I handled this by creating a JavaScript "Set" of nodes and adding the DOM node whenever a compositionUpdate event took place. Since a Set doesn't allow for duplicates, you end up with a nice clean list of DOM nodes in which you edited. I have gotten this to work in Slate.

8. Android Studio is hard to work with

There is no other way to put it than working on Android support in Android Studio is the most frustrating, slowest, and hardest development process I have ever gone through in my entire life. It executes an order of magnitude slower than an actual device, it gets slower as you work on it, then it crashes fairly soon. Sometimes you can't figure something out, and the virtual device just needed to be restarted.

In my opinion, it's more than just a matter of having more patience. When you make a code change and you have to make sure that code change didn't break the 20 other cases before it that you have already fixed, you need to be able to do those tests quickly.

Furthermore, the longer it takes to run each test, the more shortcuts you end up taking like ignoring some cases like maybe just doing a basic gesture edit and not testing at the end of a line and only in the middle. When you finally get to do the full test again, you're not sure which of the last changes broke Android support.

The slowness is compounded by the fact that these tests can't be automated short of using a robot finger (which I'm not about to build) and that these tests compound for each version of Android you need to support.

9. Use Native Devices

So use native devices where possible for the device testing. My initial two week dedicated work on Slate was done in Android Studio because I couldn't get the native device debugging to work with Chrome and I just needed to move forward. That problem has been solved now and I recommend not trying to implement Android support using Android Studio with one caveat which I will get to.

10. Regression testing

Given that (a) Android studio is hard to get running and when it is running we need 6 versions and those 6 versions can crash unreliably and (b) basically nobody has 6 Android native devices each having a different Android version that needs to be supported, then my biggest worry is regression testing. How can we possibly do regression testing when the only affordable way to test is prohibitive to set up and run and the only reliable way is expensive and would still take prohibitively long to test manually.

When somebody makes an update to Slate, there is a non-zero chance that it breaks Android support in one or more versions so there needs to be a way to test.

The only way I can see this working is to record Android events and DOM state during each event, save them in a test, replay them to Slate and see if we get what we expect out of it.

11. Recording Android Events

We need to create a web app that closely mirrors Slate's event system. Then we need to record the events into a JSON compatible format extracting the minimal amount of data for us to properly test. In addition to the events, we need to record the DOM state at the time of the event as that information is necessary to tell us if, for example, a change has been commited to the DOM at the time an event has been called. This can vary between versions of Android.

In the scenario of recording the events, this may be a good area where we can use Android Studio since we only need the recordings to execute once.

12. setTimeout and requestAnimationFrame

Another thing we need to track is setTimeout and requestAnimationFrame as a way to tell us what events get fired as a group. This can be useful in possibly simplifying code.

For example, if compositionEnd is unreliable to tell us when the edit has been commited to DOM (which it is for some verisons of Android) and in some cases the keyDown fires after it which is when the change is commited to DOM, we can't use compositionEnd. But if the keyDown is grouped with the compositionEnd, we could potentially use setTimeout or requestAnimationFrame from inside the compositionEnd with the minimum timeout and then fire the reconciliation code after that.
So in order to test this, whenever an event fires, we run a setTimeout and a requestAnimationFrame. This gets logged. All the subsequent events that get fired do not launch a new setTimeout or requestAnimationFrame. When the callback is fired, we log that. This way we know which events get fired as a continuous group without space for a setTimeout.

13. Holy Grail

I think there is a possibility that there is a holy grail like using setTimeout/requestAnimationFrame. Like maybe we can wait for a compositionEnd, fire a setTimeout and wait for the callback before reconiling and maybe it works in 90% of the edge cases in all Android versions and then we only handle the other cases; however, we can't know without running all the test cases against all the Android versions so it seems likely we need to record all the android events with setTimeout/requestAnimationFrame for grouping at a minimum.

14. Why I haven't made PRs so far

Well, to put it simply, I wouldn't accept my own PRs. Yes, it's mostly working but the code is a mess. Because of how long it takes to retest a change, I kept refactoring to a minimum because it could inadvertently stop code from working which would in turn take a long time to fix and retest.

However, you can peek into my Slate repo.

15. Separating DOM Reconciliation and making it more efficient

I have a version of DOM reconciliation which is basically the code already in Slate extracted into a separate module. It works, but it is somewhat nieve in that it replaces the entire contents of the DOM node in Slate's State even if only one character changed.

A more efficient algorithm would look for characters that match at the beginning and match at the end and only the differences in the middle would make up the actual changes. I had a version of this working in my prototype Editor. I don't think this is necessary for a version 1 of Android support but it wasn't that much work in my prototype to build.

16. Modularize Composition Handling

Right now, Slate's before plugin handles preprocessing of events which includes handling of compositions including when they start and end. This has made it difficult for to build in the different methods that Android needs to handle compositions.

We need a way to separate composition handling into it's own component which can be customized per browser and possibly per API version. The alternative is to have potentially a large number of if/else/switch statements in the before plugin itself.

Steps to Move Forward

So here's a plan for working on Slate Android Support.

[x] Separate DOM Rencociliation related code into a separate helpers library like setSelectionFromDOM, setTextFromDOMNode, setTextFromDOMSelection, setTextFromNodeKey. These methods are used to reconcile the DOM. Already did this in my existing version.
[x] Build an event logging system that can save data into a permanent format like JSON or into a database. May be ideal if we can store this in a database or something so that others can contribute use cases without having to fire up the Slate development environment. I'd like to recommend we make this part of the Slate codebase though because the events we log may be specific to a version of Slate (i.e. later versions may require us to log different events or different data from each event).
[ ] Analyze the events and see if we can derive a common pattern that might simplify development before going further. This will probably depend on the use of setTimeout or requestAnimationFrame. I'm hoping that there will be a pattern we find here that can actually eliminate most of the per version branching I've been seeing.
[ ] Create a kind of before before plugin to handle Android input. It would handle all the Android stuff before it gets to the before and after plugins. Anything non-Android related will be let through to the before and after plugin. The reason this is necessary is due to the complexity of manipulating when events need to be handled in Android. For example, we may need to handle the end of a composition, not at the compositionEnd event but at some other event that happens immediately after it. This logic just makes the before plugin messy and hard to understand for every other browser. I fear there would be so much Android specific logic that is hard to untie from the rest of the Slate logic. By placing it in a separate plugin that only handles Android logic, we keep the rest of the code clean. Also, this helps because we don't want the events in the middle of a composition to be handled through the plugin architecture. They need to be ignored (e.g. keydown, keyup, etc in a composition). That is, we don't want users handling them at all until the composition has fully ended. In Android, we can just cancel all the events that shouldn't go through the plugin system.

thesunny on 3 Dec 2018

❤9 👍9 🎉3

@thesunny amazing write-up (from what I've gotten to read so far) - Have you tried using Expo as a way to bypass the nightmares of Android Studio? Not to nitpick at all, but I wanted to let you know you missed bolding items 15 & 16.

Slapbox on 3 Dec 2018

Thanks @Slapbox and fixed.

Also, I've updated with a Steps to Move Forward.

thesunny on 4 Dec 2018

Hey @thesunny thanks for the writeup!

As for version support, can you reconcile what you said with what @jayohms mentioned above? I'm unclear about how the API xx versions reconcile with the x.x.x versions. Specifically it sounds like 7.x+ would guarantee we're working with the latest version of Chrome always.

Supporting back 6 versions sounds like a _huge_ amount of work, and it doesn't sound like it should be necessary for the first stab at Android support. It would be more useful to get it working with the latest version or two, and then see what can be done about backwards compatibility.

ianstormtaylor on 4 Dec 2018

👍1

Build an event logging system that can save data into a permanent format like JSON or into a database. May be ideal if we can store this in a database or something so that others can contribute use cases

FWIW, that's exactly what I built: https://github.com/ianstormtaylor/slate/issues/2062#issuecomment-438436799

javan on 4 Dec 2018

❤1 👍1

@ianstormtaylor

7.0+ includes Nugat and Oreo or about 50% of the ecosystem which is pretty good.

However, what may not have been clear is that this already encompasses 5 API versions which is one less than my proposal. Nougat and Oreo each had 2 API changes each. Adding one more API version (API 23) adds the biggest number of supported users. API 23 (which is Marshmallow or Android 6.0) has the largest market share out of all API versions with 21.3% of all users using it.

I think what @jayohms is talking about is the chrome version in relation to Android versions but I think, most likely, we are seeing different behaviors in the event firing and their order due to GBoard. I'm guessing this because the Samsung keyboard I believe doesn't have the GBoard issues.

I think that the approach I'm taking will work almost identically between each Android version with the differences being when a composition starts, when it ends, and which events we need to cancel. In other words, I think the big work is actually getting any 2+ version of Android to work since it has to be abstracted. At that point, we just need to configure the handling and cancelling of events.

As I mentioned in an another post, I had Slate working in 2 versions of Android with (if I remember correctly) one issue. But it was getting really hard to add the second version because I was shoehorning each version on top of the other instead of building a proper framework around it like the before before plugin which I mentioned. I think it will get deblitating going to 5+ versions! Also, I kept getting regressions which I wasn't aware of which is why I was thinking of the event capturing and replay framework (BTW, thanks @javan I was planning on talking to your more about your work in that).

thesunny on 4 Dec 2018

👍1

@thesunny the benefit to working backwards from the latest versions though is that over time all but 8.x is declining is usage, so depending on how long it takes to get it working, we can eliminate needing to serve versions like 5.x and 6.x.

But I'm still unclear on what the API changes we need to care about are... There's the Chrome version, which is sounds like as of Android 7+ will be the latest version of Chrome. So that shouldn't be an issue.

Are there other things that determine the event ordering? Is it not tied to the Chrome version that is being used?

ianstormtaylor on 4 Dec 2018

👍2

@javan

Thank you so much for building that.

There are a few things missing that I need and I wonder if you might be able to help me by adding them. I know I can fork but having some help would be great and move things along faster. I can start another issue to see if we can hash things out if you are able to. We can go deeper if you are able to, but off the top of my head:

Ability to support starting with arbitrary HTML in an HTML contenteditable
Group events that fire together until the next requestAnimationFrame
Track the HTML at the time each event fired (important so I know when I can read the DOM)
Ability to filter the JSON event properties so we only get what we need for the tests (optional)

thesunny on 4 Dec 2018

👍1

@ianstormtaylor

So, my hope initially was that we wouldn't have to handle so many versions. When I initially started recording the inputs, I was hoping that (a) API versions didn't matter and only Android versions did and (b) I'd find some patterns so that we wouldn't have to handle so many versions.

So for (a) unfortunately, each API versions is basically like handling another browser when it comes to input events. I might be missing something but I don't think we can get away from the fact that there are quite a few API versions we need to support. For example, you can buy brand new Nougat devices today which is still like 4-5 versions of support?

And for (b) there are similarities which is what I'm planning on abstracting against.

Basically, it boils down to this. (1) user starts to make an edit and then we need to lock Slate's document model from changes. (2) as the user is editing, we need to record where the edits took place in th DOM (3) we need to wait until the composition ends, reconcile Slate against the dom changes and unlock the editor .

So we need to write that no matter which version of Android and I think that's actually most of the work to handle all versions of Android. Each Android version only changes the logic of when composition starts and ends and which inputs that we need to blackhole (i.e. the events that need to be ignored lest Slate Documents gets updated which means React re-render the DOM in the middle of composition and puts editor in an incosistent state). But I think to put you at ease, I would probably release with 2 versions of Android first but I'd maybe try to keep working until we got all 6. The idea I was trying to convey is that most of the work has to be done if we are going to support at least 2 versions anyways.

I still have hope though that we can use requestAnimationFrame with an event and it might (hopefully) work for all or almost all versions of Android.

thesunny on 4 Dec 2018

👍2

@thesunny I am really need of this android support. Because of this limitation, i have gone through other libraries likes quills js, proseMirror, but i don't get the comfortable that i got with slate. any ETA for this issue to be get fixed, Since i am building a minimal editor using plain contentEditable div. once this issue is got fixed, i will thrown this code and use the slate instead.

saravanan10393 on 7 Dec 2018

It may be worth considering building an abstraction library for Android devices' input (with Slate as primary candidate user) and maintaining it as a separate package.

This may make it a little easier to test and could be proven to be useful for other devs. 2¢

dmitrizzle on 7 Dec 2018

👍1

I was playing with this today... Not that this works well by any means, but it might be helpful to others:

  componentDidMount () {
    window.addEventListener('input', this.onInput)
  }

  componentWillUnmount () {
    window.removeEventListener('input', this.onInput)
  }

  onInput = e => {
    if (isAndroid && `editor-${this.id}` === e.target.attributes.id.value) {
      e.preventDefault()
      e.stopPropagation()
      const endsWithSpace = /[ ]+$/
      const focusKey = this.state.value.selection.focus.key
      const block = this.state.value.document.getClosestBlock(focusKey)
      const lastText = block.getLastText().text
      const words = lastText ? lastText.split(' ') : ['']
      const lastWord = words[words.length - 1]

      console.log('=========================')
      console.log(e.inputType)
      console.log('key', focusKey)
      console.log('data', e.data)
      console.log('lastText', lastText.length, block.text.length)
      console.log('empty', lastText.length === 0)
      console.log('endsWithSpace:', endsWithSpace.test(lastText))
      console.log('words', words)

      switch (e.inputType) {
        case 'deleteContentBackward':
          if (e.data) { // If the soft keyboard has buffered data
            this.editor
              .deleteBackward(lastWord.length)
              .insertText(e.data)
          } else if (lastWord.length > 1 || words.length > 1) {
            this.editor.deleteBackward(1)
          } else {
            console.log('crash!')
          }
          this.editor.focus()
          break

        case 'insertCompositionText':
          if (lastWord.length > e.data.length) {
            this.editor
              .deleteBackward(lastWord.length)
              .insertText(e.data)
          } else if (lastText.length === 0 || endsWithSpace.test(lastText)) {
            this.editor
              .insertText(e.data)
          } else if (lastWord !== e.data) {
            this.editor
              .insertText(e.data.replace(lastWord, ''))
          }
          break
        default:
          console.log('unhandled: ', e.inputType, e)
      }
    }
  }
....
  render = () => {
  ...
  <Editor
          id={`editor-${this.id}`}
          key={`editor-${this.id}`}
          placeholder='Send a message...'
          ref={this.ref}
          value={this.state.value}
          onChange={this.onChange}
          renderNode={this.renderNode}
          renderMark={this.renderMark}
          plugins={this.plugins}
        />
 ...

JuanValencia on 8 Dec 2018

I think @dmitrizzle's idea is worth serious consideration.

@JuanValencia can you comment on the current state of that code and if it improves any of the behavior of Slate as it is?

Slapbox on 8 Dec 2018

I kept hacking a bit more. This is the current update:

  onInput = e => {
    if (isAndroid && `editor-${this.id}` === e.target.attributes.id.value && this.editor) {
      e.preventDefault()
      e.stopPropagation()
      const endsWithSpace = /[ ]+$/
      const focusKey = this.state.value.selection.focus.key
      const block = this.state.value.document.getClosestBlock(focusKey)
      const lastText = block && block.getLastText().text
      const words = lastText ? lastText.split(' ') : ['']
      const lastWord = words[words.length - 1]

      console.log('=========================')
      console.log(e.inputType)
      console.log('key', focusKey)
      console.log('data', e.data)
      console.log('lastText', lastText.length, block.text.length)
      console.log('empty', lastText.length === 0)
      console.log('endsWithSpace:', endsWithSpace.test(lastText))
      console.log('words', words)

      switch (e.inputType) {
        case 'deleteContentBackward':
          console.log('---- could crash')
          if (!lastWord.length > 1 && !words.length > 1) {
            // The node with key doesn't exist when you backspace here in the DOM, but does in the value
          }
          break

        case 'insertCompositionText':
          if (lastWord.length > e.data.length) {
            this.editor
              .deleteBackward(lastWord.length)
              .insertText(e.data)
          } else if (lastText.length === 0 || endsWithSpace.test(lastText)) {
            this.editor
              .insertText(e.data)
          } else if (lastWord !== e.data) {
            this.editor
              .insertText(e.data.replace(lastWord, ''))
          }
          break
        default:
          console.log('unhandled: ', e.inputType, e)
      }
    }
  }

This improves the functionality on a nexus 6 and pixel 3 a great deal. There's oddities in edge cases and I really haven't tested it thoroughly. What's more, this is external -- not internal, so there's that.

I don't see why you couldn't do this logic internally -- with several iterations, I'm sure it could catch more and more people's edge cases.

-edit- : I'm not sure how you would abstract this to a separate library. I feel like I want more control over the events not less. What's more, you have to muck with the editor's value since this is more of a compensating solution, rather than a native one like Sunny was suggesting.

JuanValencia on 8 Dec 2018

👍2

@JuanValencia Nice approach and a great way to test ideas before having to modify Slate code! I think I'll do that during prototyping.

@saravanan10393 I would say estimated time to completion will be January or February realistically.

thesunny on 9 Dec 2018

👍3

@thesunny I am really hoping to see this as well. Having even "Inserting text" and "Deleting characters backwards" work would be a huge help!

rbar2 on 17 Dec 2018

@JuanValencia I tried to play around with your code but I can't get input events to fire on the editor on desktop or mobile. They fire just fine if I input into a plain <input />, <textarea />, or <div contentEditable />

Any ideas what I might be overlooking? [email protected] & [email protected]

Slapbox on 19 Dec 2018

👍1

Just a heads up that I've almost completed an events recorder to properly understand Android in ContentEditable. It was faster to build it from scratch because of some specific requirements I had for it that weren't present in existing events recorders.

I may be asking the community to contribute recordings for various API versions. I will then use that information to build the Slate Android feature.

thesunny on 19 Dec 2018

❤5 👍4

Hey everyone, if you have time to make some event recordings from an Android device, please go to this URL with an Android browser:

https://editor-events.herokuapp.com/

The app will identify what API version you're on and also which scenarios don't have recordings for a particular API version. Please follow the instructions on each scenario carefully. Thank you!

Note: "View Recordings" doesn't work on mobile yet. You'll have to view from desktop. I wanted to deploy before I went home and didn't want to delay before the weekend. It shall be working soon enough...

thesunny on 22 Dec 2018

🎉1

Featuers:

Records React events
Records Native events
Records HTML of contentEditable when it changes
Records requestAnimationFrame and setTimeout callbacks
Records elapsed ms since start of editing for each event
Color codes events for readability
Allows you to select multiple recordings and then view them side-by-side
Identifies Android API versions
Let's you know if recording is not existing for a Scenario for your API version

If you think you can help in designing useful scenarios or editing existing ones as well as potentially curating the recordings (e.g. if a recording is bad) then let me know and I can show you how to enable admin access.

thesunny on 22 Dec 2018

@thesunny, when I visit the project on a Google Pixel 2 with Chrome, I see "Note: You are not using an Android browser"🤔. Shall I still proceed and complete recordings, or is there an issue I'm overlooking?

rbar2 on 22 Dec 2018

@rbar2

Ahh... interesting. Do you think you can share your user agent with me? I might be parsing it wrong.

It works for me from Android Studio but I haven't tested it with any real devices yet.

You can get your user agent from here:

https://www.whatsmyua.info/

Note: Recording should be fine as I store the entire User Agent for analysis but you won't be informed which of the tests require a recording for your API version.

thesunny on 22 Dec 2018

Sure thing:
Mozilla/5.0 (Linux; Android 9; Pixel 2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.99 Mobile Safari/537.36

Great, I'll do some recording for you👍

rbar2 on 22 Dec 2018

It also gives, "Note: You are not using an Android browser" for Firefox for Android (UA: Mozilla/5.0 (Android 9; Mobile; rv:65.0) Gecko/65.0 Firefox/65.0 for current beta).

(And I really hope draft doesn't end up adding UA sniffing here. UA sniffing in libraries like this makes it nearly impossible for us to update Firefox to behave more like Chrome/Safari.)

birtles on 22 Dec 2018

Thanks so much, @thesunny! I've recorded many using api-27 and the SwiftKey keyboard. I noted in some but not all the comments I was using SwiftKey rather than the default Gboard.

KrisBraun on 22 Dec 2018

@KrisBraun Oh, I better add the ability to specify the keyboard. It's Gboard that typically gives us problems. That said, the more recordings we have the better so we know what's going on in all the browsers.

@birtles FYI, this is Slate not Draft but I'll try to use as little UA sniffing as possible. I'm hoping to find some generic solutions here. That said, Firefox will probably be easier than Android. Basically all the browsers except Android/Gboard work similarly. It's only Android and it's multiple API versions that work differently that is problematic. I suggest aiming for compatibility with Chrome/Safari and my guess is Slate will either already be working or it shouldn't take too much work to do so.

@rbar2 Yeah, I just used a bad regexp. Basically I search for /^9[.]0/ after Android when I didn't account for there not being a "." after the "9". I won't be able to fix it immediately but please continue the recordings. :)

thesunny on 22 Dec 2018

👍1

@thesunny, no problem! I have completed all recordings other than the final, "hard keyboard" one :)

rbar2 on 22 Dec 2018

@thesunny this is excellent!

One issue I'm seeing is on the two recordings that span multiple lines, I can't follow the directions due to my presses getting caught on the word "Instructions," which overlaps with the editable text.

Slapbox on 24 Dec 2018

@Slapbox

Thanks, I'll fix this when I'm in the office. Not sure if I'm going to be in before the new year or not.

thesunny on 25 Dec 2018

@Slapbox Fixed.
@rbar2 Fixed.

I've started work on Android now. With the event recorder, I have a better understanding of what's going on and I'll be able to get through the issues easier. I don't have a set time frame to work on it this time. My plan is to work on it until it's done.

thesunny on 29 Dec 2018

👍4 🎉3

Thanks a lot for your work @thesunny ! We are migrating to Slate and hitting Android issues too. I have been catching up during the past week about this and I'm glad to see so much activity on this thread about it !
I have recorded some sessions on your editor-events, if you need some help I'd be more that pleased to give a hand on this one !

julienR2 on 3 Jan 2019

👍1

Just a friendly update.

Got parts of Android 8 and 9 working (3 API versions). Virtual keyboard, auto-suggest, auto-correct, gesture writing are working. Haven't checked IME yet (though I expect it will be working). Fixing a bug with splitting blocks with the cursor being in the wrong position.

thesunny on 4 Jan 2019

❤4 🎉4 👍2

I've published an early version of Android support. There are still some bugs but if you'd like to give it a try, please go for it and report back.

https://thesunny.github.io/slate/#/rich-text

I have tested with API 25, 27 and 28. 26 is probably okay. Haven't done any work on 24 or 23.

To report an issue, please mention which Android API version you are on.

If you want to know which API version you are running then visit this URL.

https://editor-events.herokuapp.com/

It may take a bit of time to spool up because it's on the free tier of Heroku. It will say something like "You are using an Android browser with this API version: API-25" in the third paragraph. The API version will be in green.

thesunny on 5 Jan 2019

❤3

Things to mention in a bug report:

[ ] Android API version
[ ] What device or whether you are using an emulator
[ ] Whether you are using gboard (the default) or some other virtual keyboard
[ ] What specific steps are required to reproduce. Please don't post if you can't reproduce.

thesunny on 5 Jan 2019

Played a bit with api 28 with SwiftKey and found none of the issues I've seen previously. 👌💯

KrisBraun on 5 Jan 2019

@thesunny this is amazing and really close to fully functional from what I can tell. Here's the bugs I did find.

Key | Value
--- | ---
API Versions | 26 & 28
Physical vs Emulator? | Physical devices
Keyboard | Gboard

Note: API 28 and 26 appear to be identical in in their behavior for all of these as best as I can tell.

Single character words

Swipe in This is a - The cursor then sets itself like this: This i|s a

Seems to be the most obvious example of a more subtle bug related to single character words in general, but most prominently in swipe. Another example of the behavior, swipe inThis was a and you'll get This w|as a The single character words can also result in text on the next line down, though I'm not sure how to reproduce that yet.

Cut All/Delete All

Cutting All and Select All/Delete lead to slightly different outcomes. Cutting All seems to truly remove every character, whereas deleting all seems to leave one whitespace character of some sort.

On the History example:

Cutting all contents and then attempting to paste back can sometimes not work because the last character of the placeholder text steals focus. (Backspace in this situation results in the same, the last character of placeholder selected)
- Select All, Copy, then Delete all contents of the undo/redo example, then Select All and finally Paste. When you paste over that whitespace character the first paragraph gets swallowed up.

Enter/Backspace

Variations of bugs related to Enter followed byBackspace and vice versa

Undo/Redo

Some issues in undo/redo behavior, but frankly dramatically less than I would have expected. I haven't identified any useful patterns here yet.

Slapbox on 5 Jan 2019

👍2

@gaearon

I was hoping you could help with this or point me in the right direction or to the right person as it's a little deeper than standard React and requires (a) an understanding of what React is doing under the hood (b) what we can expect React to do reliably in the future and (c) a potential new API to help re-stitch or re-sync DOMs with the virtual DOM. Any solution here will likely help resolve DraftJS issues as well.

Below is a before and after scenario.

In Android, we put the cursor in the word "editable". We place the cursor which is denoted by the pipe as follows "edit|able". Then we hit ENTER on the Android virtual keyboard.

It is impossible to detect that ENTER has been pressed in a reliable way because the implementers of Gboard have decided not to deliver that information in the events. There is discussion spanning years where rich text editor developers have been asking for this and the Gboard developers have said no so it's unlikely to arrive and if it does we still have old versions of Android.

So we have to let the operation continue and we end up with the DOM in the before and after state shown below:

<!-- BEFORE -->
<div data-key="1969" style="position: relative;"><span data-key="1970">
    <span data-slate-leaf="true" data-offset-key="1970:0">
      <span data-slate-content="true">This is editable </span>
    </span><span data-slate-leaf="true" data-offset-key="1970:1"><strong data-slate-mark="true"><span data-slate-content="true">rich</span></strong>
    </span><span data-slate-leaf="true" data-offset-key="1970:2"><span data-slate-content="true"> text, </span></span><span data-slate-leaf="true" data-offset-key="1970:3"><em data-slate-mark="true"><span data-slate-content="true">much</span></em>
    </span><span data-slate-leaf="true" data-offset-key="1970:4"><span data-slate-content="true"> better than a </span></span><span data-slate-leaf="true" data-offset-key="1970:5"><code data-slate-mark="true"><span data-slate-content="true">&lt;textarea&gt;</span></code>
    </span><span data-slate-leaf="true" data-offset-key="1970:6"><span data-slate-content="true">!</span></span>
    </span>
</div>

<!-- AFTER -->
<div data-key="1969" style="position: relative;">
  <span data-key="1970">
    <span data-slate-leaf="true" data-offset-key="1970:0">
      <span data-slate-content="true">This is edit</span>
    </span>
  </span>
</div>
<div data-key="1969" style="position: relative;">
  <span data-key="1970">
    <span data-slate-leaf="true" data-offset-key="1970:0">
      <span data-slate-content="true">able&nbsp;</span>
    </span><span data-slate-leaf="true" data-offset-key="1970:1"><strong data-slate-mark="true"><span data-slate-content="true">rich</span></strong>
    </span><span data-slate-leaf="true" data-offset-key="1970:2"><span data-slate-content="true"> text, </span></span><span data-slate-leaf="true" data-offset-key="1970:3"><em data-slate-mark="true"><span data-slate-content="true">much</span></em>
    </span><span data-slate-leaf="true" data-offset-key="1970:4"><span data-slate-content="true"> better than a </span></span><span data-slate-leaf="true" data-offset-key="1970:5"><code data-slate-mark="true"><span data-slate-content="true">&lt;textarea&gt;</span></code>
    </span><span data-slate-leaf="true" data-offset-key="1970:6"><span data-slate-content="true">!</span></span>
    </span>
</div>

What's happened is, from the cursor position, it's cloned all the interim <span> elements up to the enclosing <div> element. Then its moved all of the later <span> elements into the cloned <div>.

So my strategy is to restitch the DOM back into a state where React doesn't end up being confused.

My idea is to delete the second <div data-key="1969"> and move its children <span data-key="1970:1"> through <span data-key="1970:6"> back into the first <div data-key="1969">.

At this point, in the Slate editor, I'd be then making changes to Slate's state to perform the ENTER action within Slate instead of directly in the DOM. This detail isn't necessary to answering the question but provided for context.

So my question is, is this manual dom manipulation enough so that React can sync to the DOM without being confused or do I need to do more? Is there an API to force a resync under the assumption that we can't believe anything that's in the DOM (e.g. like syncing the first time from SSR). Or given your knowledge and background, should I be taking a different approach altogether?

thesunny on 5 Jan 2019

👍3

@gaearon

You asked earlier if there was anything actionable in order to help with this issue. One feature that could enable broken DOM issues related to Android (I believe there may be other cases in Android and potentially in other browsers as well) is to have a function like:

ReactDOM.stitch(element)

Which would fix the given element and all its children in relation to the virtual DOM:

Move any existing children back into the proper positions underneath it if they have moved elsewhere
Recreate any children and their event handlers if they have been removed from the DOM
Remove any children that aren't supposed to be there (i.e. do not exist in the virtual DOM)

I'm assuming that React internally keeps references to elements so that the element can be used to look up the right position in the virtual DOM; however, I'm not familiar with React internals.

thesunny on 7 Jan 2019

👍3

Hey @thesunny! Thanks for working so hard on this, you're doing an awesome job :)

I just wanted to report an issue I had seen with IMEs specifically when testing your current build. I know you're focusing mostly on Android at the moment, but thought it wouldn't help to let you know!

Cursor position incorrect when typing with IME
When typing using an IME the cursor does not move correctly with letters, rather it stays are the beginning of the typed fragment.

_Repro steps:_

Select Japanese or Chinese IME on Windows or Mac
Focus anywhere in the slate editor on your build
Type anything

_Expected outcome:_
The cursor moves along as you type

_Actual outcome:_
The cursor remains static until you complete your input with return

P.S. I have capacity to help out if you would like any extra hands on the code.

hanna-greaves on 9 Jan 2019

👍1

@sgreav I'll keep this in mind but this will likely be addressed separately.

I'm building Android support before the plugin that processes compositions in other browsers and its only added for Android. That said, I feel like in the future, we may be able to merge the composition handling that I'm building with that used by other browsers. This would likely solve the issue since the cursor issue sounds a lot like the issues we are seeing in Android.

thesunny on 10 Jan 2019

@sgreav @thesunny I got the same problem while typing with IME. It is ok when I typing in the middle of the block, and the cursor goes where I am typing. The problem apears when I type at the end of a block, while the cursor always stay where I was start typing. What's more, the editor will stop working with whitespace at the end. After that I will no longer delete any charactors.

jimshute on 10 Jan 2019

@jimshute is this on Android or another browser? If it's on Android, please let me know which API version.

Reposting myself but instructions on finding API version:

If you want to know which API version you are running then visit this URL.

https://editor-events.herokuapp.com/

It may take a bit of time to spool up because it's on the free tier of Heroku. It will say something like "You are using an Android browser with this API version: API-25" in the third paragraph. The API version will be in green.

thesunny on 10 Jan 2019

Important Update and Request for Testing

Hello everybody, this appears to be a small feature but is an important step forward. enter is working on Android API 27.

This is important because Android rearranges the DOM on enter breaking React and we can't stop the event from happening. I was afraid that this was impossible to recover from. After 3 different approaches at reassembling the DOM so that it would work with React again, the third approach is working. Other approaches reconstructed the DOM but React couldn't sync with it. I believe this same code will be able to help resolve the other Android issues.

If you can test to make sure that it is working on your API 27 (Android 8.1) devices, the latest version can be found online at:

https://thesunny.github.io/slate

Preliminary tests show that this is broken on API 26 (Android 8.0) and probably every other API version as well. I'll start working towards fixing those.

thesunny on 10 Jan 2019

🎉1 👍1

Actually, with a small fix this is working on API 26 in addition to API 27 now.

thesunny on 10 Jan 2019

😄1

Also working on API 28 and ready for testing.

thesunny on 10 Jan 2019

Enter should be working on API 24 through API 28 (Android 7.0 through to Android 9.0)

I think I will actually not support API 23 because the Editor doesn't even render even before we start trying to add Android support. API 23 appears to be missing some JavaScript features like Array.includes so it would be a fair bit of work to get it going.

thesunny on 10 Jan 2019

@thesunny What are the conditions where API 23 doesn't work for you?

Just tested it, works fine with API 23 tag, tried Chrome and Samsung browser.

adjourn on 10 Jan 2019

@adjourn Hmm... I'm using Android Studio. What device are you using specifically?

thesunny on 10 Jan 2019

@thesunny My old Samsung S5 (Android 6.0.1).

adjourn on 10 Jan 2019

@adjourn

Hmm... I am running Android 6.0.0 instead of 6.0.1 so I wonder if that's the difference.

I'm getting this error:

which I presume is referring to the Array.includes method

@Slapbox

Can you tell me which of those errors you are still seeing? Please be very specific in how I can recreate the error.

API version
Make sure to reload the editor from scratch
Where in the editor do you first touch
What specific actions do you do (type on virtual keyboard, use hardware keyboard, use gesture writing, etc.)

Many of these issues I'm having trouble recreating because they aren't specific enough.

thesunny on 11 Jan 2019

@jimshute is this on Android or another browser? If it's on Android, please let me know which API version.

Reposting myself but instructions on finding API version:

If you want to know which API version you are running then visit this URL.

https://editor-events.herokuapp.com/

It may take a bit of time to spool up because it's on the free tier of Heroku. It will say something like "You are using an Android browser with this API version: API-25" in the third paragraph. The API version will be in green.

@thesunny I got the API version on my Android 9.0，which is API-28. But I find that it is a general problem of Chrome on any plateform, including windows10, MacOSX, and Android. I've tried on Chrome/Firefox/MS Edge. When using Firefox/Edge，I can type the words on the end of a paragraph(block) even though the cursor is not in the correct position. But chrome will cause to the problem of typing and deleting disability.

jimshute on 11 Jan 2019

More bug fixes deployed to

https://thesunny.github.io/slate

API 26/27 Enter Bugs and Fixes

Note: These are all separate issues that have to be fixed separately. Please test all cases and more. It's possible that I may have broken one when I fixed another.

[x] Hit enter in the middle of a word
[x] Hit enter at the beginning of a word
[x] Hit enter at the end of a word
[x] Hit enter at the end of a word before a bold word
[x] Hit enter in the middle of a style word
[x] Hit enter at the end of a line (2019/01/10)
[x] Hit enter at start of line
[x] Hit enter two times (cursor ends up on wrong line... possible React sync breakage) (2019/01/10)
[x] Hit enter on an empty line
[ ] Hit enter at beginning of line
[ ] Hit enter at beginning of document
[ ] Later: Hit enter at end of document (after resolving end of line issue)

thesunny on 11 Jan 2019

👍1

API 26/27 Delete Bugs

[x] Hit backspace from start of a paragraph to join the paragraph before
[x] Hit backspace in an empty paragraph
[x] Hit backspace into an empty paragraph

thesunny on 11 Jan 2019

@thesunny I'm reading this thread since a couple days. I'm not even having an android but I just wanted to say; thank you! The work you do is amazing! Thanks a lot!

signalwerk on 11 Jan 2019

👍9 ❤2

@signalwerk and thumb-uppers, thanks for the encouragement! :) It means a lot.

In turn, I'm thankful to @ianstormtaylor and his organization for providing this amazing piece of open source software.

thesunny on 11 Jan 2019

👍5

If helpful, it looks like it's possible to automate keyboard interactions using monkeyrunner.

Here's a quick proof of concept: https://imgur.com/a/4OXJI2W

The scripting is pretty basic (python):

from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice

device = MonkeyRunner.waitForConnection()
device.startActivity(
    action='android.intent.action.VIEW',
    data='https://thesunny.github.io/slate/#/smoke',
)

MonkeyRunner.sleep(7)
device.touch(50, 1000, MonkeyDevice.DOWN_AND_UP)
MonkeyRunner.sleep(1)
device.drag((674, 1370), (820, 1200), 0.3)
MonkeyRunner.sleep(5)

Monkeyrunner can apparently screenshot as well, so theoretically a test suite could be built up using visual diffs for failures. I don't know much about the Android dev world and whether or not there are good ways to orchestrate that-- ideally running against VMs/cloud services, not on physical devices over USB.

alanchrt on 11 Jan 2019

@thesunny hopefully this helps with recreating the single character word related bugs. Demonstration - recorded this on API 28 - I cleared all the site data and reloaded the page before recording (which I did just moments ago, so it includes all of your newest changes.)

I won't be able to test if this still occurs on API 26 for at least a few more days, but I assume it does.

Recap of the "Single Character Words" bug:

In a sequence of swiped words, a single character word causes the caret's focus and anchor to move to the first character of the preceding word.

BTW: I meant to type, "This is a" not "if" but had already done a number of takes and this was good enough to demonstrate the issue. I just wanted to clarify that that bad English is due to my sloppy swipes and not relevant to the issue.

I used DU Recorder to make the video and then uploaded to Imgur who automatically convert the video to a GIF. Hope this helps streamline things for some other folks here.

Slapbox on 11 Jan 2019

👍1

Enter/Backspace bug:

Demonstration

Steps to recreate:

At the end of a line, press Enter.
Press Backspace to erase the newly created line.
Press Enter to add that line back in and you'll get a crash

Slapbox on 11 Jan 2019

Cut All vs Delete All behavior

This one is definitely not a show-stopper, but the odd behavior here might be helpful in understanding the nature of some other bug.

Cut All and Paste Behavior (When editor has placeholder at least)

Focus gets stuck on last character of placeholder

Delete All and Paste Behavior

Pasted paragraph is somehow swallowed by a whitespace character left behind after deleting all selected contents.

Slapbox on 11 Jan 2019

@Slapbox Thanks for the demo of the single word bug.

I'm working on all the backspace bugs right now. Found a solution requiring fixing the DOM (like the enter bug but encompassing more elements to fix) and will post when it's ready.

Thanks also for the cut demo.

thesunny on 11 Jan 2019

👍1

Backspace bugs appear to be fixed and is at https://thesunny.github.io/slate now.

Entry and Reconciliation Bugs

[x] Click in the middle of a word and select an alternate suggestion without errors
[x] Click in the middle of a word and select an alternate suggestion and make sure React Value is resolved: Note this is due to the logic when you cancel the delete (suggestion detection informs that delete is actually a suggestion) that cancels the reconcile from the compositionEnd you have to reinstate the reconcile.
[x] Fix multi-edit. Start inserting text in a block. Then click in a span after a different mark (e.g. bold). Insert more text. Click outside the editor. Both the changes should be committed to the Editor not just the last one. This is the multi-edit problem and it exists in API 27/28.

thesunny on 11 Jan 2019

👍1

Almost done for the day and taking off for the weekend.

Feels like API 26/27 is almost done. It almost never gets into a bad state now. Hitting enter at the beginning of a line and doing multiple edits in a row are the two issues. The latter should be easy (solved it before) and the former is of unknown difficulty.

Thought some of those following this issue might be interested in the backspace and enter handling notes. They will be left in the code so that others will know what's going on.

Backspace Handling

Save the state using a snapshot during a keydown event as it may end up being a delete. The DOM is in a good before state at this time.
Look at the input event to see if event.nativeEvent.inputType is equal to deleteContentBackward. If it is, then we are going to have to handle a delete but the DOM is already damaged.
If we are handling a delete then:
- stop the reconciler which was started at compositionEnd because we don't need to reconcile anything as we will be reverting dom then deleting.
- start the deleter which will revert to the snapshot then execute the delete command within Slate after a requestAnimationFrame time.
- HOWEVER! if an onBeforeInput is called before the deleter handler is executed, we now know that it wasn't a delete after all and instead it was responding to a text change from a suggestion. In this case:
- cancel the deleter
- resume the reconciler

Enter Handling

Save the state using a snapshot during a compositionEnd event as it may end up being an enter. The DOM is in a good before state at this time.
Look at the native version of the beforeInput event (two will fire a native and a React). Note: We manually forced Android to handle the native beforeInput by adding it to the content code.
If the event.nativeEvent.inputType is equal to insertParagraph or insertLineBreak then we have determined that the user pressed enter at the end of a block (and only at the end of a block).
If `enter is detected then:
- preventDefault
- set Slate's selection using the DOM
- call splitBlock
- Put some code in to make sure React's version of beforeInput doesn't fire by setting a variable. React's version will fire as it can't be cancelled from the native version even though we told it to stop.
During React's version of beforeInput, if the data property which is a string ends in a linefeed (character code 10) then we know that it was an enter anywhere other than the end of block. At this point the DOM is already damaged.
If we are handling an enter then:
- cancel the reconciler which was started from the compositionEnd event because we don't want reconciliation from the DOM to happen.
- wait until next animation frame
- revert to the last good state
- splitBlock using Slate

thesunny on 12 Jan 2019

❤3 👍3 🎉2

Update and a deploy. I think all of the enter handling and backspace bugs have been fixed in API 26/27 (Android 8.0 and 8.1). Please test.

https://thesunny.github.io/slate

The one issue that I haven't resolved yet and may not attempt to until after an initial release is holding down the backspace key or hitting backspace repeatedly quickly. I think when this is happening, a second backspace is able to manipulate the DOM before the first backspace handler has fixed the DOM and then committed the enter handling from Slate to the DOM.

Please report any bugs for API 26/27 here. I will be working on API 28 edge cases so if you have an API 28 device (Android 9), please check here once in a while if you'd like to help in the testing process.

Thank you everyone that has contributed a bug report, either here or via Slack.

thesunny on 14 Jan 2019

🎉4

Merged in the new navigation from master that came from this PR: https://github.com/ianstormtaylor/slate/pull/2450 (thanks @ericedem ).

I then added subpages support for each example so there can be more than one set of content per example. This feature is so that I could add manual tests for Slate compositions which are available now here:

https://thesunny.github.io/slate/#/composition/split-join

It's a start but the goal is to have a suite of tests that we can have users run through that will validate that compositions are working on a specific device.

I developed this before continuing work on API 28 because it was easy to miss specific cases to test. As we add more API versions and to check compatibility for other devices like iOS or desktop IME, a set of tests will have incrasing value.

thesunny on 17 Jan 2019

Fixed issues related to splitting/joining blocks and inserting text in API28 also known as Android 9.0. It may be fairly stable now.

Anybody who is able to help, please test these changes on your device or on Android studio by running it through the two sets of tests on this page:

https://thesunny.github.io/slate/#/composition

The tests are designed to trigger as many edge case bugs that I know of but I expect there are many that I haven't run into yet. Please see if you can get it to break in other ways here:

https://thesunny.github.io/slate/#/rich-text

Please note your API version when submitting an issue here. Thanks.

I will be starting work on API 25 and earlier next.

thesunny on 17 Jan 2019

Device: Google Pixel 2
API: 28

Following the instructions on "composition" - WORKS!

Playing around in "rich-text" - if I click between the "c" and "h" in "rich", then hit space, then hit delete.. it deletes the space and the "c"

rbar2 on 17 Jan 2019

👍1

@rbar2 Bug should be fixed in API28.

I've added a manual test for it. Didn't seem to fit with the other tests so I created a subpage named "special" for it. You can test it here:

https://thesunny.github.io/slate/#/composition/special

thesunny on 17 Jan 2019

❤1

@adjourn Can you verify that you can get Slate displaying (doesn't have to be working) on Android with an API 23? To clarify, this would not be the editor in my editor-events recorder but in Slate at:

https://www.slatejs.org/

thesunny on 17 Jan 2019

Both https://thesunny.github.io/slate/ and https://www.slatejs.org/ are rendering and all the regular actions that I can think of right now seem to work (splitting, cut, copy, paste, new block, etc).

Except I sometimes get Unable to find dom node <id>. This is of often because of forgetting to add props.attributes to custom component. when I press backspace in empty block, Im not sure why it happens (both editors).

adjourn on 17 Jan 2019

👍1

Fixed important but obscure bugs around periods. Added a test for this.

If you are following, please test edge cases as I'm hoping to wrap up API 26-28 support.

https://thesunny.github.io/slate/#/composition

As a note, there have been edge cases that appear to do with punctuation. It would be great to discover any more. Sometimes you need two of them. For example, It me. no. would trigger on the second . but not if there wasn't a first one.

Also, editor behaves differently if you click into a blank paragraph directly from editor start (you won't see capitalization in the keyboard in API 26/27) or if you click somewhere else first then click in the blank paragraph (you will see capitalization in the keyboard then).

So if you find a bug and can't immediately replicate it, that doesn't mean it wasn't a bug. I'd ask that you take the time to try to replicate it. If you can lock it down, there's a good chance I can fix it. It's possible I won't be returning to Android code for a while once we get a stable-ish release.

My next step will likely be to package what I have so far as a PR. Clean up the code, match Slate style guide, etc.

There are some big challenges with API 24/25 so I may do those afterwards. The major issue with them is that there is no way to tell from the event whether the user did an Enter or a Backspace. The challenge with 26-28 is that I could tell in some obscure ways through the events, but it was too late to cancel. But in 24/25 I just can't tell. So I have to inspect the DOM after the event and try to infer based on what changed in the DOM to determine that they hit Enter or Backspace.

thesunny on 18 Jan 2019

👍4 ❤1 🎉1

@thesunny I think you've fixed every issue I found except the Cut vs Delete All discrepancy and I'm in awe! I did discover one new issue though.

All reports are on API 28

Bug: Backspace after selection change

Seriousness: Critical
Likelihood user will hit issue: Likely to be experienced by some users sometimes

I'm not sure _exactly_ how to create this, but a key is focusing a different block and then coming back to where you had been typing and then backspacing. You can see towards the end of the video that when backspacing, a few character deletions before the editor crashes We becomes WeW.

Bug: Backspace on expanded selection

Seriousness: Minor
Likelihood user will hit issue: High
Backspace on expanded selections only removes the last character in the selection rather than the whole thing. The behavior is identical whether it's a single word or multiple paragraphs. Pasting over an expanded selection works properly so it's definitely related specifically to backspace.

Edit: I noticed this seems to behave differently in different scenarios. There may be more to this.

Reproductions:

Bug: Enter and Backspace in middle of words

Seriousness: Trivial
Likelihood user will hit issue: Likely to be experienced by some users sometimes

Pressing enter and then backspace while the caret is in the middle of a word will duplicate portions of the word that follow the caret.

Bug: Unfinished compositions do not suggest words when selection changes then returns

Seriousness: Minor
Likelihood user will hit issue: Likely to be experienced by some users sometimes

This is such a trivial bug it's almost more of a feature request. I'm guessing this isn't easily fixable and is probably not worth the time to fix, but for the sake of completeness.

Edit: It turns out this might be related to/might cause a slightly worse bug you can see here: https://imgur.com/a/ivWOL93

Slapbox on 18 Jan 2019

@Slapbox Thank you for the reports. Would be really helpful if you could make an attempt at reproducing from a clean example with specific steps on the "Bug: Backspace after selection change".

It does take some work to isolate but without the steps, it's difficult to diagnose.

thesunny on 18 Jan 2019

@Slapbox For the "Enter and Backspace in middle of words" I've just tested this and can't replicate. One thing that might help is (a) a video showing this and (b) the specific device you are using which I'm sure you've mentioned earlier but since I can see you editing above maybe I can just ask. :)

Also, "Unfinished compositions" I'm not sure I get what you mean. I tried to do what I think you said but wasn't experiencing the issue. Perhaps specific steps and/or video?

thesunny on 18 Jan 2019

@thesunny thanks for being so on top of this issue!

Device is a Pixel 2 on Android 9.

Regarding unfinished compositions, I assume that inserting spacebar, enter, etc is required for a composition to go from compositionUpdate to compositionEnd. It seems like failure to press one of those keys after inserting some characters causes weird behavior when you unfocus a word and then return to is (no suggestions in the suggestion bar for that word/no underline for that word indicating that it's in composition.)

Slapbox on 18 Jan 2019

@thesunny https://thesunny.github.io/slate/#/composition/special works when I follow the instructions exactly, but if I put the cursor in the middle of "middle", then hit space, then hit delete, then hit space (again😅), then hit delete (again 😅)... I end up with "The mid word" (in other words, "dle" is deleted)

rbar2 on 18 Jan 2019

👍2

Update: Fixed a few more bugs including enter/backspace. However, there is an off by 1 error but this error is an Android bug and happens in non-slate editors as well.

@Slapbox thank you for the bug testing and all the help on Slack diagnosing issues on your device.

There are a number of edge case bugs that present themselves inconsistently. In one case I was able to remove this inconsistency which happens when the cursor in a blank block sometimes ends up before and sometimes after a slate created non-breaking invisible unicode space.

In others, it appears that the logic in order to determine what the editor is doing is not able to complete its analysis before the next requestAnimationFrame and this causes unusual behavior. Furthermore, the length of the events that make up one action (e.g. some deletes or enters) comprise dozens of events.

For this reason, I will create another abstraction which I'm calling a Combo. One of the features of the combo is that whenever an event is fed to it, it resets the requestAnimationFrame and this usually gives us a bit more time to do the analysis (at least in initial testing). Since all the events fire rapidly together, by always extending the callback, I think we won't have reached the callback until we are truly done analyzing.

This should also simplify the increasingly complicated logic which is currently spread across the Android Plugin. For example, part of the enter handler is spread across many event handlers which expect things to happen in a specific way. With combos, we should be able to define them relatively simply in one place with a callback.

That said, we have some pretty good usability now and I'm pretty much done recommenting the existing code. I might put a PR out next week.

thesunny on 19 Jan 2019

🎉3 👍3

Okay, put in a pull request for Android support.

https://github.com/ianstormtaylor/slate/pull/2553

There are bugs but I don't think any will crash the browser. Some edge cases will cause the wrong text to end up in the Editor but it can be fixed without crashing. This usually seem to occur around combinations of enter/backspace sometimes in conjunction with spaces and punctuation.

More details in the PR.

thesunny on 22 Jan 2019

🎉8 🚀6 ❤6 👍1

I merged the PR into master after waiting for any more comments.

Next step is to wait for @ianstormtaylor to either release it or ask for changes.

thesunny on 28 Jan 2019

🎉9 👍7 ❤6 🚀5

This is an exciting moment! @thesunny - amazing work!

Slapbox on 28 Jan 2019

😄1

@thesunny, this is amazing! Very appreciative for this update🙌

rbar2 on 28 Jan 2019

😄1

FYI, for those following, this has been unmerged and awaiting merge approval from @ianstormtaylor from this new PR here:

https://github.com/ianstormtaylor/slate/pull/2565

A lot of discussion, questions and updates on the old PR here though:

https://github.com/ianstormtaylor/slate/pull/2553

So far resolved all the issues except for one; however, more issues are likely to come up as Ian noted that it's a big PR and will take some time to get through.

thesunny on 30 Jan 2019

❤4 👍3

@ianstormtaylor

I can start work on the Combo library which should make the Android composition code easier to follow. The existing code is a little spaghetti-ish because it handles combination events over multiple event handlers with state mixed in. The Combo would move all that into one object.

It was originally my plan to begin work on this after the current PR was handled since it would be a refactoring of this code that already works. Would it be better for you if I add the Combo to the current PR or if I wait until the current PR is worked through?

thesunny on 7 Feb 2019

Hey @thesunny, it sounds like waiting may be best. So we don’t get it too much more complex first. I’m not entirely sure how the combo concept will work.

ianstormtaylor on 7 Feb 2019

Okay, thanks. Let me know if there's anything I can do to help with the PR.

thesunny on 7 Feb 2019

Hurray! Android support was published to NPM four days ago.

thesunny on 12 Mar 2019

🎉9 ❤8 🚀7 👍6

@thesunny can you outline the next steps on this task? I know we're 95% of the way there, but you mentioned needing to address "combos" - I'm not sure what this was in reference, what it would do, or how it would work.

I know you've already done a stupid amount, but if you could briefly outline what remains to be addressed that would be a great jumping off point for completion of the remaining tasks. I know I need to file a few proper issues reports now for some minor bugs I mentioned in comments somewhere. I haven't been able to find them yet, but I'll get on that asap so we can wrap up the few remaining areas where mobile support lags behind desktop support.

Slapbox on 12 Mar 2019

@thesunny ,Are examples hosted in slate website is mobile supported now ?. So that i can check and i will migrate my code from older version to newer version, which is huge effort me since i have blendy of plugins to convert to newer version.

saravanan10393 on 14 Mar 2019

@saravanan10393 The examples on the official Slate website should have Android support on it now although I haven't checked and aren't able to right now (I'm on vacation and I don't have any Android devices on me).

@Slapbox I was waiting on getting Android support merged before I did any more work on it. As mentioned, on vacation now, but after I get back and I finish some integration work, I'll probably begin work on the Combo. I posted some source code and progress on the Combo issue https://github.com/ianstormtaylor/slate/issues/2547

thesunny on 15 Mar 2019

Sadly, just moving the cursor in code highlighting example on https://slatejs.org using Chrome on Android messes up the code.

nonsleepr on 15 Mar 2019

@nonsleepr please be more specific. Which Android version, what did you do, touch move, keyboard? Also we only have Last 2 major Android versions right now. Sounds like you might be on an earlier version

thesunny on 16 Mar 2019

I'm on Android 9, Chrome 72.
Tried again with different virtual keyboards. It looks like only Goggle's keyboard mess the code. With GBoard or Google Voice Typing when I tap in different places the code disappears. Also tried Hacker Keyboard and Grammarly. They work fine. GBoard underscores the text under cursor and suggests alternatives. I guess that reformatting is what causes the problem.

nonsleepr on 16 Mar 2019

For me it more or less works here https://www.slatejs.org/#/rich-text but if you delete all the content in this demo, you can't add any content back (editor is completely broken if empty)

glebtv on 17 Mar 2019

Hi guys, just checking in. Reading the comments but I’m on vacation right now. Will follow up after I’m back.

thesunny on 18 Mar 2019

❤3

@ianstormtaylor This is obviously a long issue now (168 comments) and we do have, shall we say, tentative support for Android.

I feel like it might be the time to close this issue and create a new one focused on fixing Android bugs which I'll use to clean up the majority of the remaining issues. After that, hopefully when most bugs are handled, we could file individual bug reports like the rest of Slate.

I'm still working on a major upgrade in the app I'm using Slate which includes updating to the latest Slate version; however, after that, my plan is to work on adding the Combo feature and fixing more bugs.

thesunny on 1 Apr 2019

For me, Slate is still unusable on both android 8 (Huawei mediapad m5) and 9 (Oneplus 5t), in Chrome. :(
Both in rich text demo and in my app.
It would be nice to have an issue with known problems or with check boxes what works and what doesn't..

Pre-populated text from rich-text demo is editable more or less, but text added from android is broken.

Text is sometimes lost when pressing enter
Backspace sometimes does not work
Placeholder disappears only when pressing enter (typing text does not remove placeholder)
Programmatically inserting content into editor (image uploads) seems to not work at all
Mark keys (bold, etc) do not work

glebtv on 1 Apr 2019

👍2

@glebtv

Thanks for the feedback. I'll be working on getting Slate working better soon (probably starting this month).

What would be helpful in the mean-time are specific steps on specific platforms, preferably with a recorded screen. Unfortunately, the issues present themselves on Android in sometimes very specific ways related to order, timing, position in text, etc. Once I can replicate it reliably though I can usually fix it.

My goal is to build a manual test suite into the examples which you can see the beginnings of in the example labelled "Composition." You'll find some issues that definitely fail if we don't have special support for them. Ideally, we want people to be able to go through the manual test suite on their own platform to ensure that we've caught all the edge cases.

thesunny on 3 Apr 2019

❤5 🚀2

Hi @thesunny, Thanks for the great work. I still have some problem on Andriod phone, please give some help. #2702

hecor on 23 Apr 2019

I'm working on fixing the remaining Android bugs now.

In order to be more efficient, please submit bugs and help with testing over in this issue:

Fix all showstopper bugs in Slate on Android (Help us by reporting and testing bugs)
https://github.com/ianstormtaylor/slate/issues/2726

thesunny on 1 May 2019

❤2 👍1

@thesunny there's a number of major bugs and some minor ones too (at least in the current repo, I know you've linked to your fork which may be better,) but only a one or two bugs I would define as real showstoppers that actually make things crash or truly unusable.

Could you clarify where you want to set the bar for this pass of bug fixes? I know the cut-offs for each classification are nebulous, but I assume your aim might be more accurately described as fixing major and showstopper bugs, the distinction in my mind being that major bugs can be worked around with great effort, and critical/showstopper bugs cannot.

I just don't want to clutter up your issue on an already complicated and hard to manage topic with things that are below the "worth fixing now" threshold, and I suspect I'll be responsible for a substantial amount of the coming bug reports.

To help make things less nebulous, I just grabbed the first set of definitions for different bug levels:

Critical / Show Stopper: An item that prevents further testing of the product or function under test can be classified as Critical Bug. No workaround is possible for such bugs. Examples of this include a missing menu option or security permission required to access a function under test.

Major / High: Defect that does not function as expected/designed or cause other functionality to fail to meet requirements can be classified as Major Bug. The workaround can be provided for such bugs. Examples of this include inaccurate calculations; the wrong field being updated, etc.

Average / Medium: The defects which do not conform to standards and conventions can be classified as Medium Bugs. Easy workarounds exists to achieve functionality objectives. Examples include matching visual and text links which lead to different end points.

Minor / Low: Minor priority is most often used for cosmetic issues that don’t inhibit the functionality or main purpose of the project, such as correction of typos in code comments or whitespace issues.

Source

Slapbox on 1 May 2019

@Slapbox That's a good question; however, I think it's okay for you to post all of them and I can make a determination what makes it in. It's probably too much to ask the bug reporters to do the analysis since some of the lower priority ones might be easy fixes so I might as well do them now.

I'll probably only not do the ones that are simultaneously (a) hard to fix and (b) not that important.

thesunny on 1 May 2019

👍3

I'm posting here instead of the Android bugs thread https://github.com/ianstormtaylor/slate/issues/2726 in order to reach the most developers.

I've come to an issue that think may be impossible to workaround. My hope is by explaining it, somebody might be able to find a solution or offer ideas that might help me get passed the issue. I'm happy to explain any more details or answer any questions.

You may wonder, since other editors have Android support, why it would be impossible with Slate? The main issue is that we cannot control when React renders to the DOM. We can push a change to our value and eventually that will flow through to the DOM, but we cannot force the render to happen synchronously.

This normally isn't a problem in Slate because we prevent an event (like an enter keypress) from happening, and then replace it with our own DOM changes. We are the only ones making changes to the DOM.

In Android, there are many events we cannot prevent. We have to wait for them to finish modifying the DOM. We then resolve the changes into Slate's value and React re-renders the DOM.

The problem happens when React re-renders the DOM at the same time as Android is modifying the DOM. This will happen when a user presses a key at the same time the re-render is happening. For example, it can happen when holding down the backspace key.

Here's a specific use case: Some events like backspace execute unintuitively. For example, if you were to delete the last letter in chicken, instead of an input:deleteContentBackward we get a input:insertCompositionText with data of chicke (without the n). This tells us Android is probably overwriting the word with a new word with the remaining characters.

While React is trying to reconcile the DOM against its virtual DOM, the user presses another key, and we end up with chickechick. I'm not sure whether React is adding chicke back in before Android's new chick insertion or visa versa.

I have found ways to reduce the likelihood of this happening and some double characters is very annoying but maybe worth the compromise. However, that's not what I'm worried about. The fact that there are conflicting DOM changes introduces a dangerous amount of complexity that could result in the editor crashing.

If Android deletes an element (which it will do when backspacing) or inserts a new element (which it will do when hitting enter), we must put the DOM back into a state before Android deleted or added the element. I've already built a library for this and it works.

If we don't do this, React tries to render into a DOM tree that does not match it's virtual DOM. This causes a crash.

However, if there are timing issues where React and Android are both trying to change the DOM at the same time, we can get into a bad state easily and, most problematic, it is not easy to replicate and to fix. I am finding that at times I can pass all my manual tests but just sometimes the editor gets into a bad state and I can't replicate it to fix it.

Eventually I found this dual edit issue but I can't think of a solution short of making changes to React itself which I have no experience with and likely would be difficult to get my changes accepted to.

thesunny on 8 May 2019

👀1 ❤1

Hey, @thesunny I'm very new here, but I'm just going to take a stab. Would the ability to check the DOM before and after ReactDOM reconciliations help? In my understanding what your saying is that things get funky when a user makes an edit between a React render and a ReactDOM reconciliation. I was thinking maybe it would be possible to compare the intended and actual DOM with the getSnapshotBeforeUpdate and componentDidUpdate methods

brynshanahan on 8 May 2019

👍2

@brynshanahan Thank you. That's an interesting idea. getSnapshotBeforeUpdate does appear to execute at an important time (i.e. immediately before the render in what appears to be a synchronous manner). I'm not clear yet on how to use this to fix the issue but it does open up some more avenues to explore and experiment with.

I wonder if we can tie our DomSnapshot to the getSnapshotBeforeUpdate so that we can fix the DOM immediately before render and whether that would help or not. I think I might need to figure out what order the DOM manipulations are happening and from where (i.e. when React's render vs Android's keypress manipulations happen). MutationObserver might help here.

thesunny on 8 May 2019

👍3

@thesunny Have you considered simply setting the contenteditable property to false in getSnapshotBeforeUpdate and setting it to true in componentDidUpdate? Does this break android?

brainkim on 8 May 2019

👍1

Yeah figuring out the edit order should be really helpful @thesunny. Also here are some bits of info I found this morning that could possibly help us.

Info on getSnapshotBeforeUpdate
Info on render/reconciliation timing. Also a really interesting article about React in depth from Dan Abromov's blog.

Consistency
Even if we want to split the reconciliation process itself into non-blocking chunks of work, we should still perform the actual host tree operations in a single synchronous swoop. This way we can ensure that the user doesn’t see a half-updated UI, and that the browser doesn’t perform unnecessary layout and style recalculation for intermediate states that the user shouldn’t see.
This is why React splits all work into the “render phase” and the “commit phase”. Render phase is when React calls your components and performs reconciliation. It is safe to interrupt and in the future will be asynchronous. Commit phase is when React touches the host tree. It is always synchronous.

And I think this diagram might help with the discussion . Arrows are where breaking user edits could be happening

I'm not entirely confident but it sounds like the _React Updates DOM and refs_ step is synchronous, and therefore maybe getSnapshotBeforeUpdate is synchronous as well. Which would cross out user edits breaking anything at point D (which is good because surely that would be the least recoverable point)

brynshanahan on 9 May 2019

👍2

@brainkim Thank you for the suggestion. That's an interesting idea but at this time I don't think I'm willing to go there. The reason is that it opens up a whole set of unspecified/unknown behaviors. For example, do we lose the selection when we switch contenteditable to false? What happens if we switch in the middle of a repeating keydown (because compositionend events happen in the middle of repeating keydown events)?

The solution might be in there somewhere but at the moment, there are so many unknowns already that I'm reluctant to try and solve the current unknowns by adding some new ones.

@brynshanahan Thank you for the additional detail. I haven't spent much time on it but I haven't wrapped my head around using getSnapshotBeforeUpdate to create a solution yet.

The big issue revolves around timing of state changes. In particular, if we have to revert, we revert to a DOM Snapshot (DOM state) at a given point in time. Then the user does something (like "backspace") and then we have to create a new Slate Value (Editor State). So now we have an old DOM State and we have a new Editor State.

Then before the Editor State gets rendered to the DOM (getSnapshotBeforeUpdate to the rescue?) we revert the DOM State. But what if before the Editor State gets rendered, the user presses backspace again. Then we are reverting to a DOM state before the second backspace has been pressed. The initial revert might save us from an unrecoverable error, but now the DOM state doesn't reflect the second backspace because its been reverted out so how do we make sure we don't lose that second backspace. I'm not entirely sure the sequence of the actions here and there may be several depending on small timing issues.

So, long story short is that I don't yet have confidence on a way to proceed that will fix the issues. I have one other idea that might fix the timing issue with respect to key repeats; however, I still lack confidence that solving that is a band aid that won't prevent small timing errors from causing React to crash.

thesunny on 10 May 2019

👍1

Note: The ideal solution is to have React not panic and die if it can't find DOM nodes. Instead, it should just rebuild the DOM. But I don't know that the React team would care to do this for the very specific use case of a "Rich Text Editor on Android".

thesunny on 10 May 2019

Hey @thesunny this may remove any need for messing with React lifecycles. https://github.com/facebook/draft-js/commit/cda13cb8ff9c896cdb9ff832d1edeaa470d3b871
Also draft just merged their android fix into master 9 hours ago, so it might be worth looking for some inspiration there.
https://github.com/facebook/draft-js/blob/master/src/component/base/DraftEditor.react.js

brynshanahan on 10 May 2019

👍2

@brynshanahan Slacked you on this but just wanted to say the find in there about ReactDOM.flushSync and friends is amazing!

Not documented but seems like that has the potential to solve the timing issues. While getSnaphotBeforeUpdate might have solved it, it seemed needlessly complex (maybe even unsurmountable) if events snuck in between setState and render and since React is aiming on async going into the future, I was worried that this was something that I could never resolve without React features which appear to have already been built. Of course, you had to be Facebook to know about them (or know React internals which I don't).

thesunny on 10 May 2019

@thesunny:

The ideal solution is to have React not panic and die if it can't find DOM nodes. Instead, it should just rebuild the DOM.

Isn't that what error boundaries are for? In our DraftJS-based editor we have accepted as a sad fact of life, that the DOM sometimes gets out of sync and causes React to panic. Our error boundary catches such errors and resets the editor to the last known state to recover.

Not ideal, but much better than getting stuck in an error state and ending up with an unusable editor.

robbertbrak on 10 May 2019

👍1

@robbertbrak Error boundaries would save the rest of the React app from crashing but it wouldn't prevent the Editor component from crashing unfortunately.

edit: Sorry, I didn't read that properly. I see what you are saying. That sounds like a useful fallback. If everything dies, then re-render using the last known data.

General Update

Lots of thanks to @brynshanahan for pointing me to the recent (as in a few days ago) update to DraftJS. There are two really important techniques that I was unaware of in React and that I haven't previously seen documented. I think these two will greatly simplify the code. They are:

The ability to flush synchronously if required using ReacDOM.flushSync and friends.
The ability to render the DOM without worrying about broken DOM state. Interestingly what they do here is they just set the key to the editor's Content area to a new value. Under the hood I'm assuming that if the key changes then the entire Element is blown away so even if there are inconsistencies, we don't have to worry about them. The negative is that you are completely rebuilding the DOM which is a definite performance degradation, especially on bigger documents; however, you get back the consistency of not crashing your Editor if the DOM is out of sync. I think that we should keep the DOMSnapshot object I built (should be much faster) but use the DraftJS method for now. Then we can introduce DOMSnapshot back slowly working out all of the edge cases and building the performance up.

Finally, it seems like ProseMirror and Draft have basically gone all in on MutationObserver over doing event analysis. DraftJS having adopted much of the ProseMirror code. The benefit, long run, seems to be that we should be able to build something on MutationObserver that can work in more than just Android which I think @ianstormtaylor would be much happier about instead of the Android only approach we've had to take when analyzing events.

Sadly, for me, this means abandoning a lot of the work I've done. Happily, these discoveries, especially with respect to React, means that Android is possible with React.

thesunny on 14 May 2019

❤10 👍5

Sadly, for me, this means abandoning a lot of the work I've done.

It always hurts to throw away hard work, but the lessons learned make the new code that much better.

I think MutationObserver is a great idea. My only real reservation about this method was the idea that they should be a solution of last resort. Seeing other editors going that route makes me think that's the best way to go for sure. Besides them providing proof that the method works, Slate will be able to poach useful related code from two other well maintained projects. In the long run that latter benefit will probably prove enormous.

Slapbox on 15 May 2019

❤2

@thesunny

Haven't looked into Android at all, I have a question: could combination of (Android) events contaminate whole document? Can't you use "blow DOM away with key" strategy in block or even smaller scope?

Even if we have to re-render whole doc.. I'd take worse performance on Android over not working any day.

adjourn on 15 May 2019

@thesunny Thank you for keep working on that issue and for the constant updates!

signalwerk on 15 May 2019

👍3 ❤1

@adjourn it feels like "blow DOM away with key" strategy should work in smaller scopes although at that granularity level, we have to worry about the edge cases so would still need the whole document version.

A few issues I can think of are

a backspace that merges into the block prior and
an enter which splits a block into the existing block and a sibling block.

thesunny on 15 May 2019

@robbertbrak Hey, I just wanted to say that the more I think about it, I really like the idea you posted about using error boundaries to recover the Editor from a React crash. Thank you so much for posting this.

thesunny on 15 May 2019

👍1

Unsure of what editors are doing for the race condition. But have we considered queueing operations while the editor is inside a composition? That way during the entire duration of a composition the editor will not apply changes, until the composition ends after which all the queued changes will be applied.

ianstormtaylor on 18 May 2019

👍1

@ianstormtaylor, @thesunny can speak to this better, but there are some edge cases this might leave behind. There might be more common issues I'm not aware of.

The edge case that occurs to me is if a user enters a word, but does not press space, enter, or enter some punctuation character, the composition event will not end. If a user goes to enter a word in the middle of an existing sentence, and then changes the editor's selection seems the most likely real world presentation of this edge case.

That solution does sound elegant to me though if we can do things that way.

Slapbox on 18 May 2019

@ianstormtaylor FYI, I haven't been doing operations until compositions have ended for a while now. Those compositions themselves work pretty smoothly and have been for some time.

The timing issue was related to backspace. When you hit backspace a second time at just the right moment, we seem to interrupt the React reconciliation and get weird issues. I think sync solves this though.

Although isolated compositions work well, I'm working on improving the boundary of compositions and non-composition events within a single tick (i.e. within a single requestAnimationFrame).

There is actually a rather larger roster of issues to deal with but I am getting closer. Every iteration I remove more edge cases and decrease complexity.

I think the error boundary to save Slate from crashing in an unrecoverable way is huge. That's already been merged and fixes previously fatal errors.

@Slapbox I think you are referring to compositions not ending and carrying over to another node. Yes, that happens sometime and requires us to keep track of all nodes that are part of the composition where previously I only tracked the last node. I might try to actually force the compositionEnd instead when the selection changes to a new node to simplify the code. I think ProseMirror does this as well.

thesunny on 19 May 2019

❤5 👍2

A short update is that I have made progress using mutations. I also bought a Google Pixel 3a which makes working on Android so much more pleasant!

Currently, the main Editor behaviors like compositions, hitting enter to splitBlock and hitting backspace to merge blocks and to delete entire elements (like bold text) all basically work. There are still issues I'm working through like restoring the DOM causing the mutations observer to disconnect and holding down backspace continuously causing timing issues.

Note: I actually had a version of the editor where it does all edits using mutations on desktop Chrome which speaks to the flexibility of using mutations. We may wish to consider using mutations for everything which could simplify code in the future. Right now on desktop we use events like keydown or splitBlock. By tying into commands (like insertText and splitBlock) rather than events for plugins we could potentially settle on a unifying codebase for regular edits (one character at a time) and composition edits (a word at a time).

WRT, the timing issues which I think are related to all the code in Slate and React that work in an async nature, I'm trying one by one to remove those cases. The good news is that while there are bugs nothing seems to crash the Editor anymore. Sometimes you get the Value out of sync with the DOM however and some weird behavior.

thesunny on 25 May 2019

❤6 👍2

Okay, I have a stable-ish version for Android 9 (edit: and 8.1) using mutations. Please test it out here here:

https://thesunny.github.io/slate/#/composition/split-join

And submit bugs here:

https://github.com/ianstormtaylor/slate/issues/2839

NOTES:

Holding down backspace does not work well (pressing it one at a time does and so do selection deletes)
Only tested on Android 9 (edit: also testing on 8.1 and works)
Passes all the manual tests but I'll likely have to add more

thesunny on 29 May 2019

🎉7

Okay, good news everyone!

I've run all my basic tests on Android 8.1 and everything works with no changes!!! This is absolutely amazing! Seriously, adding another version of Android was previously just as hard as adding the first one. The fact that it just works is boggling my mind. I actually had to go check to see if I was on the right device because the fact that it worked didn't make sense to me.

edit: Forgot to run the enter test. There was a few bugs there but all the other tests work.

thesunny on 29 May 2019

🎉9 ❤1

Android 8.1 including the enter tests are running now at https://thesunny.github.io/slate/#/composition/split-join

thesunny on 30 May 2019

@thesunny: Great to hear that using MutationObserver is working out so well! 👍 With Draft (which now uses MutationObserver for _all_ composition-based input, not just on Android) I have also seen good results.

Just one question though: when you say that "everything works" on Android 8.1 / 9, what exactly do you mean? In my experience, whether composition-based input works on Android is not just limited to the API version, but also varies depending on:

Keyboard (Gboard, Huawei Swype, Google Voice Typing, Samsung keyboard, ...)
Language (Chinese, Korean, Japanese, Arabic, ...)
Browser (Chrome, Firefox, ...)
Keyboard variant and settings (Correct words while typing yes/no, Pinyin / Handwriting, ...)

robbertbrak on 30 May 2019

To clarify I only meant that all my manual tests pass on Android Chrome on Gboard with the exception of continuous backspace. I’m sure there are more bugs which is why I’m asking for bug reports.

My tests do include gesture writing, foreign IME but not some of the others; however if they work as compositions then they theoretically should work. I did nothing extra for IME or gestures for example but they both work.

The initial goal is to get the most common use case for Android at an MVP (mostly works and no unrecoverable crashes) then build on there.

I could see a switch to using only MutationObserver being a path for Slate and I even had Desktop Chrome working on MutationObserver while developing the Android version; however that’s probably a decision ultimately for Ian.

thesunny on 30 May 2019

👍7

Made the PR for Android support using mutations here

https://github.com/ianstormtaylor/slate/pull/2853

thesunny on 5 Jun 2019

👍5

I've reported two bugs affecting Android Firefox and Chrome: https://github.com/ianstormtaylor/slate/issues/2951 and https://github.com/ianstormtaylor/slate/issues/2952. I figure they're related to this issue but that it might be good to make issues for each of them.

Macil on 30 Jul 2019

👍1

I believe that this may be fixed by https://github.com/ianstormtaylor/slate/pull/3093, which has changed a lot of the logic in Slate and slate-react especially. I'm going to close this out, but as always, feel free to open a new issue if it persists for you. Thanks for understanding.

ianstormtaylor on 28 Nov 2019

Hey, where can I find code example for this demo https://thesunny.github.io/slate/#/composition/split-join or any other example that works well on Android? This is the only demo I found that works fine on my Android device

fainir on 25 Apr 2020

@fainir likely somewhere in here: https://github.com/thesunny/slate/

Sorry I can't be more helpful with an exact location. 0.47.x doesn't work perfectly with Android, but works much better than previous versions. The biggest issues I have using Google's GBoard is holding down backspace leads to everything getting mangled.

Slapbox on 25 Apr 2020

@fainir likely somewhere in here: https://github.com/thesunny/slate/

Sorry I can't be more helpful with an exact location. 0.47.x doesn't work perfectly with Android, but works much better than previous versions. The biggest issues I have using Google's GBoard is holding down backspace leads to everything getting mangled.

Tested on my android device and and it has the same bugs as this demo https://www.slatejs.org/examples/richtext. The only demo I found that works well on my device is https://thesunny.github.io/slate/#/composition/split-join that mentioned few comments above by @thesunny. Does anyone has the code for this demo? Why isn't working on the main demo? It is a huge bug and @thesunny fixed it so I think it should be implemented inside the package and the main demo.

fainir on 25 Apr 2020

@fainir you can find out more about that situation here #3573

The code is in that repo, but may not be at the tip of the master branch. It looks like the master is on 0.50.x already.

Basically @thesunny did a ton of work to get it working on 0.47.x, but the project is still in beta and the API is often changing. 0.50.x was a total rewrite by @ianstormtaylor to address a lot of other issues, but making those changes wasn't done in coordination with @thesunny (it would have been basically impossible on a volunteer basis.) If you need Android support stick with 0.47.x. I'm not sure about a demo for that version, but the docs for it are still available alongside the latest docs.

Slapbox on 25 Apr 2020

Thank you very much. I downgraded to 0.47.9 and it works :)
I suggest that project's owners would add notice to the readme file that version 0.5.x is not supported for Android and show demo of 0.47.x with Android support because I assume that Android support is very critical for many users.
Thank you very much! great project :)

fainir on 26 Apr 2020

👍3

@Slapbox Thank you for this amazing project. Is there a roadmap to add support for android devices in 0.5.x? Thanks!

vasanthps on 11 May 2020

@vasanthps just to clarify @ianstormtaylor is the amazing person who leads the Slate project.

There's no official roadmap to add support for Android devices and it's not planned right now, hence #3573.

Slapbox on 11 May 2020

@thesunny @ianstormtaylor I am using Slate v47 and it's working perfectly on desktop and iOS devices but onChange is not triggering on Android devices. I would greatly appreciate any suggestion. Thanks.

baljeet-aulakh on 18 May 2020

Hi @baljeet-aulakh

Android uses mutations and not the onChange event. If onChange was wired to be called from a mutation, it would not behave in the same way because you could not preventDefault to stop the event.

thesunny on 18 May 2020

👍1

Hi @baljeet-aulakh

Android uses mutations and not the onChange event. If onChange was wired to be called from a mutation, it would not behave in the same way because you could not preventDefault to stop the event.

Thanks @thesunny
Is there any documentation that can help me get Slate 47 to work on Android phones?
It seems like I will have to configure slate differently for Android cuz it's working fine for desktops and iOS phones.

baljeet-aulakh on 18 May 2020

@baljeet-aulakh

Sorry, I may have been confusing your query. For clarity, the onChange should fire to update the Slate DOM but the onChange event won't work properly on the contentEditable. Honestly, it's been a while since I looked into the internals so you may have to take my thoughts with a grain of salt!

thesunny on 18 May 2020

👍2

@thesunny I'm running into the same issue with Slate 0.47.9.

Tested on a couple of different Android versions using BrowserStack (Android 10, 9, 8) and as far as I can tell the editor.value is rarely updated, and the <Editor onChange={} /> prop is only ever called when deleting characters with an expanded selection.

Kapture 2020-06-10 at 15 16 13

clauderic on 10 Jun 2020

It's been a while but this may be due to how Android decides when a composition ends which is when changes are committed.

It is possible for a composition to span multiple locations. Android is the only browser that does this.

Selection deletes always cause a change to happen I believe. But deleting/backspacing within an element won't. But backspacing across a boundary (ie. between two spans) will.

The reasons for this have to do with a balance between performance and consistency. If we reconcile too often, we can have consistency issues because we are trying to reconcile while android is still executing mutations. We have an escape hatch where when things get so far out of sync that React crashes, that we have an error boundary that just rebuilds everything from scratch. But that's jarring so we don't want to see that often. It's better than a fatal crash though.

I'm not working on that code anymore but if you want to take a shot at fixing it, a possible solution or improvement could be to identify when there is a change in selection even if it's in the middle of a composition and then reconcile the browser DOM with the Slate DOM.

If you do take a crack at it, make sure to test against all the test cases in the demo that I wrote specifically for compositions or else you may end up putting the editor in a worse state than it is now.

thesunny on 10 Jun 2020

👍1

a possible solution or improvement could be to identify when there is a change in selection even if it's in the middle of a composition and then reconcile the browser DOM with the Slate DOM.

Hey @thesunny, I'd be happy to look into that if that was the only thing that was broken, but unless I'm missing something it seems like the Slate value is never updated on Android, and therefore the onChange callback is never called.

As mentioned above, the only working scenario I have seen is with an expanded selection and pressing backspace, this will indeed trigger an onChange, otherwise nothing seems to work.

I'm trying to see if this could be a regression at some point before 0.47.9, do you remember this working properly at some point?

clauderic on 10 Jun 2020

I went as far back as this commit locally and I still was seeing the same behaviour https://github.com/ianstormtaylor/slate/commit/7d4062cde9b769586834fd86a4a73b0c998dfdd6

Maybe I'm missing something?

clauderic on 10 Jun 2020

It was working at some point so it may be a regression but it's also possibly a change in Android which may be causing the issue. I haven't looked at it since about a year ago so I don't really know what the state is. I'm pretty certain it was working at some point though as some people were relying on it. Also, a lot of the timing issues that I had to deal with had to do with how React re-renders during editing so those issues wouldn't have popped up if onChange never got called.

Also try clicking outside the editor (ie. lose focus) to see if at least that calls onChange. I believe that forces the composition end. If that's not working, it feels like it's a regression or an Android change that is causing it not to work anymore.

thesunny on 11 Jun 2020

Is there an open issue tracking this I can monitor? Latest version is not playing well with GBoard

Jtfinlay on 22 Jul 2020

@Jtfinlay it's not planned per se, but there's some hope.

See https://github.com/ianstormtaylor/slate/issues/3786 and https://github.com/ianstormtaylor/slate/issues/3573

Slapbox on 22 Jul 2020

👍1

I still see this issue with

"slate": "^0.58.4",
"slate-history": "^0.58.4",
"slate-react": "^0.58.4"

Selecting all the text and deleting it triggers a crash

jesusgp22 on 27 Aug 2020

👍1

Never before has there been such a team! Nice job @thesunny @ianstormtaylor.

I don't actually use the library but stumbled across this while trying to support Android/IoS for my own ContentEditable stuff and it's been a massive help.

ShanonJackson on 21 Nov 2020

❤1 👍1

Slate: fix editing with "soft keyboards" (eg. Android, IMEs)

Do you want to request a _feature_ or report a _bug_?

What's the current behavior?

What's the expected behavior?

Most helpful comment

All 221 comments

IME

Gesture Writing

Regular Writing

Suggestions

Auto-correct

Physical Keyboard

Android 4.4

Android 4.4.3

Android 5.x - 6.x

Android 7.x - 9.x

Support across OS versions

Single character words

Cut All/Delete All

Enter/Backspace

Undo/Redo

Recap of the "Single Character Words" bug:

Enter/Backspace bug:

Cut All vs Delete All behavior

Backspace Handling

Enter Handling

Bug: Backspace after selection change

Bug: Backspace on expanded selection

Bug: Enter and Backspace in middle of words

Bug: Unfinished compositions do not suggest words when selection changes then returns

General Update

Related issues