Slate: consider adding `schema` into `state` objects

[ianstormtaylor]

I can't find the other place this was discussed right now, but wanted to open this for discussion.

---

Right now there are two schemas: the "core" one, and the user-land one.

The core one is used internally to normalize the document after any change method are run, and never exposed to the user. It ensures that the core rules of the document model are preserved and the user never touches a document that doesn't have them in effect. (Things like block nodes can't have inlines, etc.)

Then there is another schema, that is passed into the <Editor>, and can be added to by plugins. This schema is normalized using the onBeforeChange handled of the "core" plugin.

---

A few issues from this current setup:

• The onBeforeChange handler is kind of awkward, in the if the schema was part of state it wouldn't even need to exist. And it also is leaky in that it only normalizes after the other on* handlers are run, meaning that it doesn't guarantee that a developer never sees an invalid document (according to their schema).

• Since the user-land schema isn't part of state, we can't normalize against it automatically in the change methods, because we don't have access to the editor instance there.

• If you are running changes outside the editor, and just passing them back in via setState, then you actually aren't normalizing against the user-land schema for these changes. This is unexpected, and probably is causing issues that people aren't aware of?

So those would be solved.

But it would introduce a few new problems to find solutions for:

• If it the schema was part of state, then it would be more complex for plugins to augment it. Right now you can deal with the state in server-side environments, or in places where you don't have the editor with all of its plugins. If schema needed to be inside state, then anywhere you were working with a document you'd need access to all of the plugins that the editor uses when working with that type of document? This seems very problematic.

• If the schema is part of state, and a normalization happens automatically when you create the state, how do you ensure that any changes to a state are always observable by the parent? This is required for collaborative situations, where any operations that occur must be logged.

---

I would love thoughts from others on this, either for or against, or with ideas for how to solve some of these problems.

[oyeanuj]

@ianstormtaylor A concern if we end up adding the schema to state is older documents existing with a different schema than the newer ones (and the editor's current configuration). So would that mean that the schema in the state gets re-written every time it is passed back to the editor from the DB?

Also, curious about this line below:

And it also is leaky in that it only normalizes after the other on* handlers are run, meaning that it doesn't guarantee that a user never sees an invalid document (according to their schema).

What is an example of this to better understand this case?

[ianstormtaylor]

@oyeanuj good questions:

• I think you wouldn't serialize the schema into the DB, so you'd always have the most update to date one. I think this problem exists already currently since the schema is versioned with your codebase, but your documents when unserialized might be running old versions. But this seems like a database migration problem that Slate shouldn't solve?

• For the leakiness, that should have said "that a developer never sees an invalid document" to be more clear. Basically if you have multiple chained changes in two plugins onKeyDown handlers for instance, it will only normalize against your schema after they have both run. But if the first does something "un-normal" the second will receive it.

[oyeanuj]

I think you wouldn't serialize the schema into the DB, so you'd always have the most update to date one.

Makes sense since I guess you could serialize just the document in the DB, rather than the DB.

I think this problem exists already currently since the schema is versioned with your codebase, but your documents when unserialized might be running old versions.

I was more concerned about having impact of having an outdated schema definition within the state (which doesn't happen today), but like you mentioned, one should only be serializing the document within the DB rather than the entire state.

Thanks for clarifying those points!

[Soreine]

Here's our opinion with @SamyPesse

The onBeforeChange handler is kind of awkward, in the if the schema was part of state it wouldn't even need to exist. And it also is leaky in that it only normalizes after the other on* handlers are run, meaning that it doesn't guarantee that a developer never sees an invalid document (according to their schema).

The way Slate is building and using the on* handlers seems compatible with passing down the schema and make use of it. There must be a way here to run a normalization between each handler call https://github.com/ianstormtaylor/slate/blob/master/packages/slate-react/src/components/editor.js#L132

Since the user-land schema isn't part of state, we can't normalize against it automatically in the change methods, because we don't have access to the editor instance there.

This has not been a problem for us so far (and we are using a lot of plugins). This would be a problem in case of a responsibility conflict between two plugins. But such conflicts cannot be simply resolved by normalizing after every change.

If you are running changes outside the editor, and just passing them back in via setState, then you actually aren't normalizing against the user-land schema for these changes. This is unexpected, and probably is causing issues that people aren't aware of?

We have never used this method. It is not documented, so not many people should be using it.

If it the schema was part of state, then it would be more complex for plugins to augment it. Right now you can deal with the state in server-side environments, or in places where you don't have the editor with all of its plugins. If schema needed to be inside state, then anywhere you were working with a document you'd need access to all of the plugins that the editor uses when working with that type of document? This seems very problematic.

We really need to split up plugins in two parts. A Slate plugin should expose a separate plugin and schema. One part concerning the rendering and behavior. The other concerning document validation:

plugin includes:

• on* handlers
• render method
• Rendering-only schema

schema includes:

• Normalization rules only

---

We are fine with the proposal if we correctly split concerns (see separation of behavior and validation for plugins), and only store the validation part of the schema in the state.

It would also be great to expose a schema stack factory. Something to aggregate validation schema from different sources, like what Slate.Stack does, but for schema only: const schema = SlateSchema.create([schemaDef1, schemaDef2, ...]). If we can add the Slate core schema into this, it would allow us to normalize a state properly, outside of the Editor.

[YurkaninRyan]

When I make changes outside the editor in my stores, I often end up running .normalize for my own personal schema as well as all the plugin schema that I have. It's strange to me that a change can be made that creates an invalid schema that won't be handled until the next on* function occurs

[tobiasandersen]

We have never used this method. It is not documented, so not many people should be using it.

I think what @ianstormtaylor refers to here is when you pass in a state that was created outside of the Editor. E.g. when adding/changing stuff from an external toolbar, which is something I believe people do quite often. And as @YurkaninRyan describes, Slate won't normalize that state for you. I remember being a bit confused about that some time ago.

[YurkaninRyan]

I believe the old solution was to call the editor's onChange whenever you make an update from the outside, but that caused some issues for me

[oyeanuj]

I am a little confused now.. so calling editor's onChange for external changes (say, like the toolbar) doesn't run the normalization?

Also, +1 to @Soreine's idea of separating out the schema part of the plugin from the behavior part. Another added advantage is that when in readOnly mode, all you need is the schema and not the behavior.

[Slapbox]

I'd like this. It would make handling the deprecation of onDocumentChange simpler, among other things.

[Soreine]

@oyeanuj I just noticed that since 0.22.0 there is no normalization call in componentWillReceiveProps and the constructor of Editor (see changelog). So calling onChange is not normalizing anymore.

[ianstormtaylor]

A thing to think about:

We used to have plugins define the rendering of nodes and marks via renderNode, renderMark, and renderDecorations. Potentially going back to that kind of system would be a good way to separate those pieces from the schema once again if we want.

[ianstormtaylor]

@Soreine @SamyPesse I would love to hear more about what you've been thinking about separating the "behavior/rendering" vs. "structural" parts of the schema. I think that's a good way to put it, and I agree that it's something we're probably going to need to do.

Ideally I feel like we could get to the point where "plugins" were purely the behavior/rendering. And then "schema" was purely the structural.

The tricky ones seem to be things like edit-list/table/code/etc that are both at once. They're really more like entire "features" than plugins per se. Right now it makes total sense that inside edit-list is the place to enforce the proper structure. It just going to hard to completely split them without solving that schema issue, since otherwise you're going to need to pull in all of your plugins on the server-side anyways?

I wonder if there's a way that with #1258 solved we could end up in a situation where the "feature" plugins could depend on the user setting up the schema, instead of packaging it themselves.

(This might also give users more flexibility in terms of how to handle the normalizations, since the "feature" plugins really only care to see that a specific structure is enforced, not how?)

What do you think? (Or anyone else!)

[ianstormtaylor]

Specifically, @SamyPesse and @Soreine what do you think about the idea of plugins like edit-code/list/etc not defining any structural, normalizing schema behaviors? And instead asking that the user defines the necessary schema for things to work.

For example, slate-edit-blockquote would require that the user define something like this in their own schema that they control:

{
  quote_block: {
    kinds: 'block+',
  }
}

And then it would be initialize just with the type:

EditBlock({
  type: 'quote_block',
})

And not provide any schema rules itself.

Could you see that being achievable? Otherwise I'm not sure how we can successfully split "plugins" from "schemas" in a way that doesn't force people to pull all their plugins server-side?

[SamyPesse]

@ianstormtaylor I'm not a big fan of having the user of the plugin to define its rules.

Rules for plugins like table or list are not as straightforward as we think.

I've suggested an approach in #1258 with plugin exposing some fragment of the schema, and the application can use a method to merge these fragments into a complete schema.

Otherwise I'm not sure how we can successfully split "plugins" from "schemas" in a way that doesn't force people to pull all their plugins server-side?

None of our plugins exposes rendering, they are all "schema plugins".

And maybe plugins could have two export:

import EditCodeSchema from 'slate-edit-code'; // provide schema and transforms
import EditCodeRendering from 'slate-edit-code/plugin'; // provide renderNode, etc

or

import { EditCodeSchema, EditCodeRendering } from 'slate-edit-code';

[ianstormtaylor]

@SamyPesse very fair, that makes sense.

I guess there are actually three different things: "rendering", "behaviors" and "structure". I was kinda thinking of client-side concerns as being all three, but server-side the ideal is to only have the "structure" part since the editor isn't there. But that said, I don't think there's too much to be gained from trying to force people not to combine the three into plugins, since it'll just end up restricting things.

So yeah, I'll stick with the idea of plugins bringing their own schemas.

Thanks for the thoughts! 😄 I'll respond in the others thread now too.