Fp-ts: Serialization/deserialization of data types

[gcanti]

This is what I get from Option now

import { fromNullable } from 'fp-ts/lib/Option'

const x = fromNullable(1)
const y = fromNullable(null)

console.log(JSON.stringify(x)) // => {"value":1}
console.log(JSON.stringify(y)) // => {}

In order to behave more like flow-static-land which doesn't change the underline representation when you call inj / prj, would be useful a toNullable function

declare function toNullable<A>(fa: HKTOption<A>): A | null

such that the pair inj / prj is equivalent to the pair fromNullable / toNullable. Also both None and Some could implement the toJSON method.

Proof of concept

export class None<A> {
  ...
  toJSON(): null {
    return null
  }
}

export class Some<A> {
  ...
  toJSON(): A {
    return this.value
  }
}

export function fromNullable<A>(a: A | null | undefined): Option<A> {
  return a == null ? none : new Some(a)
}

export function toNullable<A>(fa: HKTOption<A>): A | null {
  return (fa as Option<A>).toJSON()
}

Result

const x = fromNullable(1)
const y = fromNullable(null)

console.log(toNullable(x)) // => 1
console.log(toNullable(y)) // => null

console.log(JSON.stringify(x)) // => 1
console.log(JSON.stringify(y)) // => null

danielepolencic's comment here https://github.com/gcanti/io-ts/issues/31

Initially this is what I had.

The problem I faced is that - while the .toJSON solves the serialisation - there's no easy way to
deserialise the data structure.

As an example, I'm using a redux-like framework and messages and state are plain objects by definition. When I wrapped my Maybes into classes, I had to > have ser/deser functions all over codebase. This made the code more complicated and bloated.

For data types such as Maybe and Either, it makes sense to have the data type as a plain object. In that way there's no need to transform the data into the proper data type.

[gcanti]

@danielepolencic thanks to chime in

there's no easy way to deserialise the data structure

io-ts could be a (possible) solution, it can deserialise while validating

import * as t from 'io-ts'

import { Option, none, some } from 'fp-ts/lib/Option'

const OptionFromNumber = new t.Type<Option<number>>(
  'OptionFromNumber',
  (v, c) => t.union([t.number, t.null])
    .validate(v, c)
    .chain(n => t.success(n === null ? none : some(n)))
)

t.validate(null, OptionFromNumber)
  .fold(
    errors => console.error(errors.join('\n')),
    x => console.log(x)
  )

// => None

t.validate(2, OptionFromNumber)
  .fold(
    errors => console.error(errors.join('\n')),
    x => console.log(x)
  )

// => Some(2)

I'm using a redux-like framework and messages and state are plain objects by definition

AFAIK state can be anything while actions must be plain objects but only at the first level (the payload can contain anything), unless you use tools that make stronger assumptions. I'm not an expert on this topic though so I might be wrong.

When I wrapped my Maybes into classes, I had to have ser/deser functions all over codebase

I guess that once you enter in the Maybe context, computations may be always executed in that context (and functions are automatically lifted)

For data types such as Maybe and Either, it makes sense to have the data type as a plain object. In that way there's no need to transform the data into the proper data type.

Mhh, however you must still call inj / prj in order leverage higher kinded types, isn't calling inj / prj as awkward as calling the fromNullable / toNullable pair?

[danielepolencic]

Let me provide more context.

If the state is serialisable, it can be persisted in localstorage or saved in the database.
If the actions are serialisable, they can be forwarded to web workers or persisted in distributed queues.

As soon as you need to deserialise the data and create classes, you end up having plenty of ser/deser function pairs. Before I moved to plain object I used to have:

export interface IState {
  queue: IJob[]
}

export interface IStateSerialized {
  queue: IJobSerialized[]
}

export interface IJob {
  id: string
  order: Base.IOrderRequest
  otherOrder: Maybe.Maybe<Other.IOrder>
}

export interface IJobSerialized {
  id: string
  order: Base.IOrderRequestSerialized
  otherOrder: Other.IOrderSerialized | void | null
}

export function serialize(state: IState): IStateSerialized {
  return {
    queue: state.queue.map(job => ({
      id: job.id,
      order: job.order,
      otherOrder: Maybe.orSome(undefined, job.otherOrder)
    }))
  };
}

export function deserialize(state: IStateSerialized): IState {
  return {
    queue: state.queue.map(job => ({
      id: job.id,
      order: job.order,
      otherOrder: Maybe.of(job.otherOrder)
    }))
  };
}

While doable, this is not practical.

I guess that once you enter in the Maybe context, computations may be always executed in that context (and functions are automatically lifted)

Are you referring to the possibility of chaining calls such as Maybe.of(1).map(x => x + 1).fmap(() => Maybe.of(2))?

Mhh, however you must still call inj / prj in order leverage higher kinded types, isn't calling inj / prj as awkward as calling the fromNullable / toNullable pair?

I never call inj or prj in the code, but just in the library. The underlying data structure is a plain object/primitive. In the worse case scenario I can cast to any and save it to the database/localstorage and nothing happens. If I try to do the same with a class - the next time I retrieve it - the code crashes (cause it's a plain obj that has to be augmented).

I found inj / prj very convenient not just for data types such as Maybe and Either but also for casting obj/primitives to new types (e.g. string -> rational number) and hiding methods/properties in complex objects.

[gcanti]

If the state is serialisable, it can be persisted in localstorage or saved in the database.
If the actions are serialisable, they can be forwarded to web workers or persisted in distributed queues

Sure but using classes as internal representation in a library like fp-ts is an implementation detail, it doesn't affect your choices.

I mean, there are 2 cases

• A) domain models are plain objects
• B) domain models are not plain objects (for example they are implemented using Option, Either, etc...)

Case A

Let's say you have this simple domain model (all plain objects) and a function representing some business logic

type Person = {
  name: string,
  age: number | null
}

function businessLogic(person: Person): string | null {
  if (typeof person.age === 'number') {
    return `${person.name} ${person.age}`
  }
  return null
}

Now you want a more functional style so you want to refactor businessLogic without changing the signature. In order to use advanced functional tools you must lift the values to higher kinded types, so you must use some trick like inj / prj

const getName = (name: string) => (age: number) => `${name} ${age}`

function businessLogic(person: Person): string | null {
  return prj(option.map(getName(person.name), inj(person.age)))
}

That is any different from

function businessLogic(person: Person): string | null {
  return fromNullable(person.age) // same as inj
    .map(getName(person.name))
    .toNullable() // same as prj
}

That what I meant with

however you must still call inj / prj in order leverage higher kinded types, isn't calling inj / prj as awkward as calling the fromNullable / toNullable pair?

You still have the benefits of a serializable domain model, just a different API for Option, Either, etc...

Case B

For some reason domain models are not plain objects, for example

type Person = {
  name: string,
  age: Option<number>
}

const getName = (name: string) => (age: number) => `${name} ${age}`

function businessLogic(person: Person): Option<string> {
  return person.age.map(getName(person.name))
}

Code is even simpler though there are two problems

• (1) how can I deserialise a plain object to Person?
• (2) how can I serialise Person to a plain object?

For (1) something like io-ts can help

import * as t from 'io-ts'
import { pathReporterFailure } from 'io-ts/lib/reporters/default'

const JSONOptionNumber = new t.Type<Option<number>>(
  'JSONOptionNumber',
  (v, c) => t.union([t.number, t.null])
    .validate(v, c)
    .chain(n => t.success(n === null ? none : some(n)))
)

function deserialise(json: any): Person {
  const T = t.interface({
    name: t.string,
    age: JSONOptionNumber
  })
  return t.validate(json, T).fold(
    errors => { throw new Error(pathReporterFailure(errors).join('\n')) },
    identity
  )
}

console.log(deserialise({ name: 'Giulio', age: null })) // => { name: 'Giulio', age: None }
console.log(deserialise({ name: 'Giulio', age: 43 })) // => { name: 'Giulio', age: Some(43) }

For (2) by default you get

const a = deserialise({ name: 'Giulio', age: null })
const b = deserialise({ name: 'Giulio', age: 43 })

// wrong serialisation
console.log(JSON.stringify(a)) // => {"name":"Giulio","age":{}}
console.log(JSON.stringify(b)) // => {"name":"Giulio","age":{"value":43}}

but you can define a toJSON method for Option with a custom serialisation logic

// module augmentation
declare module '../src/Option' {
  interface None<A> {
    toJSON(): A | null
  }
  interface Some<A> {
    toJSON(): A | null
  }
}

None.prototype.toJSON = Some.prototype.toJSON = function (this: any) {
  return isNone(this) ? null : this.value
}

// OK!
console.log(JSON.stringify(a)) // => {"name":"Giulio","age":null}
console.log(JSON.stringify(b)) // => {"name":"Giulio","age":43}

[danielepolencic]

Correct me if I'm wrong, but I think what I have is a mix of A and B.

I'm referring to: https://gist.github.com/danielepolencic/71393b5b6c8c54902b1e6804a93bf81e

import * as Maybe from './maybe';

type Person = {
  name: string,
  age: Maybe.Maybe<number>
}

const person1: Person = {name: 'Daniele', age: Maybe.of(1)};
const person2: Person = {name: 'Daniele', age: Maybe.Nothing};
console.log(person1); // {name: 'Daniele', age: 1};
console.log(person2); // {name: 'Daniele', age: null};

also since I'm wrapping a primitive, I get serialisation for free:

JSON.stringify(person1); // {"name": "Daniele", "age": 1};
JSON.stringify(person2); // {"name": "Daniele", "age": null};

This is mainly because the signature for JSON.stringify is (v: any) => string.

Deserialisation is also not a problem:

const p1: Person = JSON.parse({"name": "Daniele", "age": 1});
const p2: Person = JSON.parse({"name": "Daniele", "age": null});

and I can still safely access the property:

const youAreOld = (age: number) => console.log(`You are old`);
Maybe.map(youAreOld)(p1.age); // prints: `You are old`
Maybe.map(youAreOld)(p2.age); // prints nothing

All of this happens just in the compiler. The JS code is the same. The benefit is that I get the extra type safety.

While I have inj exported, I use Maybe.of to do the lift. The value is still a primitive, but the compiler is convinced I'm dealing with a real class. No need for toJSON.

The drawback (or advantage depending on how you look at it) is that I always need to unpack a Maybe with Maybe.map or Maybe.fmap. There's no method attached to the obj.

The issue with deserialisation is that you need a deserialisation func for each interface you have. Case B is a good example of that already. It is very verbose.

[gcanti]

:) yeah, seems a kind of midway. Though if you put maybes into the domain models what you have can be considered as a B with an implicit toJSON definition for maybes.

In general I'd love to support both A and B and both static-land and fantasy-land style.

Some issues / considerations

1. JSON.parse is unsafe
2. You need a deserialisation step anyway
3. do notation

1. JSON.parse is unsafe

Basically this is an unsafe cast since JSON.parse returns any

const p1: Person = JSON.parse(`{"name": "Daniele", "age": 1}`)

this is ok as long as you trust the source but more generally you may want to validate the input

const p2: Person = JSON.parse(`{}`) // ops...

which leads to point 2.

2. You need a deserialisation step anyway

For two reasons:

• IO validation
• actual deserialisation

With "actual deserialisation" I mean the following case

// returned by some endpoint
const payload = {
  name: 'Giulio', // string
  age: 'NO_AGE'   // number | 'NO_AGE',
  birth_date: 'Fri Nov 30 1973 00:00:00 GMT+0100 (CET)' // ISO string
}

// must be transformed to

// your domain model
type User = {
  name: string,
  age: Maybe<number>,
  birth_date: Date
}

name is not a problem but age and birth_date must be trasformed.

(io-ts is born to provide both IO validation and actual deserialisation with concise syntax and minimum duplication of schemas)

The drawback (or advantage depending on how you look at it) is that I always need to unpack a Maybe with Maybe.map or Maybe.fmap. There's no method attached to the obj.

leads to point 3.

3. do notation

static-land style is great but chaining computations without do notation is painful

const double = (n: number): number => n * 2
const length = (s: string): number => s.length
const inverse = (n: number): Maybe<number> => maybe.fromPredicate((n: number) => n !== 0)(n)

const o1 = maybe.chain(
  inverse, maybe.map(
    double, maybe.map(
      length, Just('hello'))))

vs

const o2 = Just('hello')
  .map(length)
  .map(double)
  .chain(inverse)

Here is where fantasy-land style really shines: it allows for automatic dictionary selection (through prototype) and as a surrogate of do notation

The issue with deserialisation is that you need a deserialisation func for each interface you have. Case B is a good example of that already. It is very verbose.

Given that point 2. requires a validation/deserialisation step, the example contained in my previous comment can be written in a more general (and concise) way

const RuntimePerson = t.interface({
  name: t.string,
  age: createOption(t.number)
})

// this can be extracted from RuntimePerson
type Person = {
  name: string,
  age: Option<number>
}

// a :: Person
const a = unsafeValidateAndDeserialise(`{"name":"Giulio","age":null}`, RuntimePerson)
// b :: Person
const b = unsafeValidateAndDeserialise(`{"name":"Giulio","age":43}`, RuntimePerson)

console.log(a) // => { name: 'Giulio', age: None }
console.log(b) // => { name: 'Giulio', age: Some(43) }

// OK!
console.log(JSON.stringify(a)) // => {"name":"Giulio","age":null}
console.log(JSON.stringify(b)) // => {"name":"Giulio","age":43}

where

//
// library code
//

import * as t from 'io-ts'
import { pathReporterFailure } from 'io-ts/lib/reporters/default'

function createOption<T>(type: t.Type<T>): t.Type<Option<T>> {
  return new t.Type<Option<T>>(
    `Option<${type.name}>`,
    (v, c) => t.union([type, t.null])
      .validate(v, c)
      .chain(n => t.success(n === null ? none : some(n)))
  )
}

declare module '../src/Option' {
  interface None<A> {
    toJSON(): A | null
  }
  interface Some<A> {
    toJSON(): A | null
  }
}

None.prototype.toJSON = Some.prototype.toJSON = function (this: any) {
  return isNone(this) ? null : this.value
}

const JSONType = new t.Type<any>(
  `JSON`,
  (v, c) => t.string.validate(v, c).chain(s => {
    try {
      return t.success(JSON.parse(v))
    } catch (e) {
      return t.failure(v, c)
    }
  })
)

function unsafeValidateAndDeserialise<T>(value: string, type: t.Type<T>): T {
  return t.validate(value, JSONType)
    .chain(v => t.validate(v, type))
    .fold(
      errors => { throw new Error(pathReporterFailure(errors).join('\n')) },
      identity
    )
}

[danielepolencic]

The solution you are proposing is far superior compared to what I'm using at the moment. (i.e. JSON.parse/JSON.stringify).

However, it comes with a cost that I'm not prepared to pay: extra complexity. The code is parsed at run time and interfaces are generated automatically with a compiler (I guess?). So you have:

• io-ts interfaces
• typescript interfaces

This is very much a personal choice and can be largely ignored. On a side note, I use a similar validation library (Joi) but just for specific objects I don't trust (e.g. APIs).

Comments on some of the point you raised:

static-land style is great but chaining computations without do notation is painful

Tell me about it :)

However after spending some time with the code, I much prefer the compose approach. In the example your provided

const o2 = Just('hello')
  .map(length)
  .map(double)
  .chain(inverse)

this would be:

const o2 = compose(
  Maybe.chain(inverse),
  Maybe.map(double),
  Maybe.map(length)
)(Maybe.of('hello'));

Again, this fits well with the data not being augmented/decorated.

[gcanti]

However, it comes with a cost that I'm not prepared to pay: extra complexity

I think that if you do IO validation you already pay for that complexity, you have

• typescript interfaces (type Person)
• "runtime types" (Joi schemas in your case or RuntimePerson in my case)

Also there are two main problems:

• schema duplication: you must define the schemas twice, one for typescript and one for Joi
• schemas can go out of sync

AFAIK there are only 3 solutions

• (1) generate the runtime types from the typescript interfaces (something like flow-runtime or babel-plugin-tcomb)
• (2) extract the typescript interfaces from the runtime types (something like io-ts)
• (3) generate both typescript interfaces and runtime types from a third language (JSON Schema, swagger, etc..) (something like gen-io-ts <= work in progress)

(1)

In my experience in not reliable. Also it doesn't play well with TypeScript, it requires a plugin system (a la babel). Maybe in the future since there is a plugin system for TypeScript in the works.

(2)

The code is parsed at run time and interfaces are generated automatically with a compiler (I guess?).

Typescript interfaces can be statically extracted from runtime types without a generation step (io-ts's TypeOf operator):

const RuntimePerson = t.interface({
  name: t.string,
  age: createOption(t.number)
})

// static type extraction, TypeScript sees the type Person as usual
type Person = t.TypeOf<typeof RuntimePerson>
/*
Same as:
type Person = {
  name: string,
  age: Option<number>
}
*/

(3)

If you already have the API payloads expressed in a schema, for example JSON Schema, it would be great to generate both static and runtime types from that. This is the goal of gen-io-ts

Tell me about it

:)

At that time the problem was

Flow is pretty buggy at the moment and this implementation is way more solid and type safe than the other implementations (classes included) that I tried so far

maybe now the situation is different. With TypeScript I didn't have any big problem so far.

However after spending some time with the code, I much prefer the compose approach

Ah, this is interesting because I have some problem with compose and type inference, it's off topic though, I'll open another issue for that

p.s.

@danielepolencic thanks again for your feedback, it's helping me a lot to focus on some pain points I care about

[danielepolencic]

I will have a look at (2). It sounds very interesting.

However I still think it brings a lot of overhead. This is very subjective, but this is what I've been thinking so far. The current learning curve for anyone wanting to jump on my codebase right now involves learning:

• javascript (ok, this is a given)
• typescript
• monads & co

As soon as I add something like io-ts and/or babel suddenly my complexity level increases by 2 more technologies. The trade off for me is using the unsafe JSON.parse and JSON.stringify without needing any extra library. You know already how to use them since they're part of the language.

As for Joi, this is used exclusively for APIs and not object models (like in the example your provided above). While there's duplication, this is self contained in specific files that only deal with the APIs. Not ideal, but it's very easy to understand.

I'm not ready to use tcomb or io-ts. While I see the value, I can't justify the extra complexity of learning and using another tool on top of monads + typescript.

Re: compose. You're right, it's 💩 . Particularly because ts is not able to infer types for curried functions. The pragmatic approach for me is very simple: all functions are _manually_ curried only for the last argument. That covers 80% of my use cases and doesn't require any fiddling with the compiler. From there I use compose and most of the time I manually specify the types.

Being verbose isn't something I enjoy, but it's the lesser evil.

I hope this makes sense. I'm not trying to be difficult or go against you. Those are same of the choices I made while diving in the ts and fp world.

[tomByrer]

Thank you both; very educational.

[tycho01]

Hm. It's unfortunate JSON.stringify appears not to respect toString on classes.
If it did, then I suppose the serialization part could be handled by just using toString overwrites on e.g. Option like toString = () => "null" on None, toString = () => this.value on Some.
But yeah, that parsing step... good luck.