Xstate: Persisted machine migration across XState versions

[timoxley]

Please treat the serialized format as a part of your public API.

This is more of a plea/headsup/feature request.

I just got bitten by an XState machine format change from back in August.

• I'm persisting XState 4.6.x machines in PouchDB.
• This change, introduced in XState 4.7.0, adds an _event property to the state object: "Use SCXML.Event wherever possible"

This caused two issues upon trying to update XState:

1. Unfortunately for me, PouchDB forbids the storing of objects under a key a leading underscore, i.e. _event is an invalid top-level key, so new machines created with the latest version of XState simply wouldn't persist to the db. I don't expect XState to change format because of a restriction like this in PouchDB, however I do feel like this should not have happened without a major version bump.
2. Secondly, the new version of XState would crash with Cannot read property 'data' of undefined when trying to deserialize state saved from the old version, because this new _event property is missing: https://github.com/davidkpiano/xstate/blob/7ce4e93cda2a8cad00842725a12e9e7b731f2c24/packages/core/src/State.ts#L232

Request

Ideally:

• Breaking changes to the serialized format should be done across a major semver version bumps.
• XState should give some consideration & explicit testing to identify issues with backwards/forwards compatibility of the serialized format.
• XState could support running something akin to db "migrations" when the serialized format changes.

I think this would be appropriate, given that serialization (and deserialization?) is stated as one of the key goals:

---

My hacky workaround is to wrap all reads and writes to the db with custom serialize/deserialize to rename the keys and add the _event property by calling toSCXMLEvent on the event property. This is inconvenient and a bit haphazard because I need to wrap every location that reads from the db, but it seems to work. Included my workaround code below in case someone else experiences this issue.

import mapKeys from 'lodash/mapKeys'
import { toSCXMLEvent } from 'xstate/lib/utils'

export function serialize (state) {
  const s = JSON.parse(JSON.stringify(state)) // ensures no magic objects
  return mapKeys(s, (value, key) => { // swap keys with leading _ to leading $
    if (key[0] === '_') {
      return `$${key.slice(1)}`
    }
    return key
  })
}

export function deserialize (s) {
  const r = mapKeys(s, (value, key) => {
    if (key[0] === '$') { // swap keys with leading $ to leading _ (reverse of serialise)
      return `_${key.slice(1)}`
    }
    return key
  })
  // add missing _event if not there
  if (r.event && !r._event) {
    r._event = toSCXMLEvent(r.event)
  }
  return r
}

[Andarist]

Sorry that we have overlooked this, definitely something that we should have on our minds when releasing new minor/patch versions. @davidkpiano we could prepare different test machines that we could try to serialize from now on with each version and provide those serialized values as test inputs for rehydration, WDYT? @timoxley you seem to be interested in the stability in this regard, would you be interested in helping us out with it?

I don't expect XState to change format because of a restriction like this in PouchDB, however I do feel like this should not have happened without a major version bump.

I'm not sure if the part about major version bump for smth like that is true - this is really hard to predict, there are too many existing systems that you could integrate XState with like this and it's impractical to expect from us not to add any new properties. I believe though that we should be able to rehydrate using data that is missing newly added properties and we've not thought about this when developing the latest version.

I think this would be appropriate, given that serialization (and deserialization?) is stated as one of the key goals

It only mentions serialization/deserialization of machine definitions, not states. It's an important trait of XState though and we probably should add it to this list as an additional goal (or extend the one you have mentioned).

My hacky workaround is to wrap all reads and writes to the db with custom serialize/deserialize to rename the keys

I wouldn't call this a hacky workaround. It seems like a proper solution for the problem - given the fact that your db has such constraints to which you have to adjust this data.

[davidkpiano]

I will be adding JSON Schema soon for the:

• machine definition
• state object
• event object

The JSON-serialized machine, state, and event will be tested against this schema moving forward.

[alizbazar]

We also experienced the same problem. We were only storing value and context (and corresponding keys from history) as this seemed sufficient for fully functioning rehydration. Now getting Cannot read property 'data' of undefined if trying to upgrade to newest XState. :/

[davidkpiano]

@alizbazar Try the State.create(config) method, or:

State.from(stateValue, context);

Can you also share code in CodeSandbox? There might be a workaround.

[Andarist]

@alizbazar if you could create a repro case that would be extremely helpful

[timoxley]

@timoxley you seem to be interested in the stability in this regard, would you be interested in helping us out with it?

I would love to but I hit this issue on a part-time project. My investment in XState there is heavy but my attention to that project is also fleeting, hence me only detecting this now rather than when it occurred back in August.

[davidkpiano]

JSON Schema PR: https://github.com/davidkpiano/xstate/pull/960

[dsebastien]

I'm quite interested in first class support for state serialization/deserialization.

In my system, I find it interesting to be able to persist (also in CouchDB, as it happens) the state of some data types. Being able to reconstruct a typed state object and further on a service/machine is very appealing.

I'm still hesitating between a simple union type, disconnected from whatever I do with XState, but if I could simply persist the serialized version of the state, it would be quite cool :)

[alizbazar]

We are using XState on the backend (including history functionality) and are offloading state to a DB. When upgrading, introduction of _event property broke our build.

Is this the recommended way for handling offloading / rehydration? Or is there some other more "future proof" way?

We could use State.from() but that would not support history.

[davidkpiano]

@alizbazar Not sure how the addition of properties to a JSON object could break the build - how exactly are you storing it in the database? For example, in Postgres, if you have a column of type json / jsonb, you can just store the JSON state as-is without problems.

[timoxley]

Not sure how the addition of properties to a JSON object could break the build

@davidkpiano please see the issue description

[davidkpiano]

Ah never mind - sorry, forgot the original issue. However, IMO it's best to avoid blindly storing JSON objects as the top-level objects, because there is no guarantee that the object won't have top-level underscored properties. Storing it as:

{ state: jsonStateToPersist }

would solve it.

[timoxley]

The bigger problem is the incompatibility between serialised formats causing a crash on deserialisation.

[alizbazar]

@davidkpiano is your idea that jsonStateToPersist was acquired via JSON.stringify(state)?

Previously, we only stored state.value and state.context, but then for supporting history we also started storing state.history.value and state.history.context and state.historyValue.

If we were to store JSON.stringify(state) and then used that for rehydration, would this be backwards compatible for the future?

Just FYI here's our simplified code for serialization / deserialization:

[davidkpiano]

If we were to store JSON.stringify(state) and then used that for rehydration, would this be backwards compatible for the future?

Yes, although you can store state.toJSON() as well.