Libertem: Message schemas: form validation+generation, typing from server to client, ...

[sk1p]

Currently, message types are mostly defined in TypeScript. Messages are used as raw dict on the Python side, which means error handling is a bit... let's say, bad. Messages send from the Python API to the client are duplicated in the server code. Even on client-side, there is no further validation, which means you can enter nonsense data and only get a (rather unhelpful) message after submitting the form.

Ideally, there should be a canonical description of all messages in a suitable format. This description may include metadata for generating forms, which may not be a simple 1-to-1 mapping to messages. Note that not all messages need to have form representation, but we could validate all messages before sending them to the server.

From this description, TypeScript types should be generated; incompatible changes to any message type should be caught by the type system for the client code. We may even use mypy to check types on the Python side.

Also, validation logic for both the Python API and the forms in the GUI should be generated. Client-side validation logic could use JSON schema, if this is sufficient; this needs to be evaluated (regarding expressiveness and error messages).

Ideally, the outcome would be well-behaved, automatically generated forms just by defining a schema in a single place, which would go a long way to adding UDFs to the GUI more easily.

See also:

• Related discussion in #233 about the "open dataset" dialog
• related project: https://github.com/bcherny/json-schema-to-typescript
• related project: https://github.com/encode/typesystem (supports generating json schema); this looks very good, even though it is a young project!
• some old example code in https://github.com/LiberTEM/LiberTEM/tree/master/prototypes/schemastuff

[sk1p]

For a subset of messages (open dataset), we are now using json schema for validation. This covers most of the current cases of free user input. There is no generation of TypeScript interfaces from json schema yet, so still some duplication. Let's see how well this works and maybe we can adapt this for all messages on HTTP/websocket.

[sk1p]

Oh, and no form auto-generation yet, either - but that's not really a pain point right now.

[Iamshankhadeep]

I want to work on it as my GSOC 2020 project.

[sk1p]

More related projects: https://github.com/sinclairzx81/typebox, https://github.com/gcanti/io-ts

[sk1p]

Related blog post: https://vriad.com/blog/zod/

[Iamshankhadeep]

So which framework should we use for this issue and for validation purposes? zod or io-ts ?

[sk1p]

So which framework should we use for this issue and for validation purposes? zod or io-ts ?

... or maybe something different? The point is, we don't know, and figuring our what to use is a large part of the project itself. Important is that we reach the goals outlined in the issue description.

I'm mostly posting the links to the related project so that we have an overview for different approaches for parts of the solution, not _necessarily_ so we can directly use one of these packages.

[sk1p]

Even more links:

https://shahmeer.co/blog/end-to-api-typing.html
https://github.com/rawrmaan/restyped
https://instagram-engineering.com/types-for-python-http-apis-an-instagram-story-d3c3a207fdb7
https://docs.apistar.com/ and https://www.starlette.io/schemas/

[Iamshankhadeep]

Messages send from the Python API to the client are duplicated in the server code.

I think I did not understand it as I was going through it again. If possible can you explain it to me?

[sk1p]

Messages send from the Python API to the client are duplicated in the server code.

I think I did not understand it as I was going through it again. If possible can you explain it to me?

This means that there are structural definitions of some kind for each message that is sent from the server to the client, and they are present both on the server and the client. On the server, these definitions are often only very rough, as they are not typed, so the structure is just "whatever dict this method returns".

To understand this more deeply, pick a message type, for example FINISH_JOB. Try to read the code (both server and client) and figure out how/where a message of that type is generated on the server, sent to the client, arrives at the client, and how it is then handled on the client. Compare how the messages are created on the server with how they are then interpreted on the client. If you don't see it at the beginning, try to repeat this for a few different message types.

You can also have a look at the opposite direction, how the client calls the HTTP API and handles the structure of the data it sends as payload, and then how the server interprets this, and what assumptions it makes about the data it gets.

[sk1p]

Additional related project: https://github.com/asyncapi/asyncapi

[Iamshankhadeep]

Additional related project: https://github.com/asyncapi/asyncapi

I had come across it while I was researching, But it does not have code gen support for typescript and python. So I never mentioned about it.

[sk1p]

Related: https://github.com/sinclairzx81/typebox