C-toxcore: [RFC] Capability flags and message V2, alternative implementation

[sudden6]

As the first part, to handle backwards compatibility in a nice way, we should build on the idea of @isotoxin

Current core accepts only 1-byte length PACKET_ID_ONLINE.
My proposal is use this fact as end of capability sequence.
Old core sends 1-byte PACKET_ID_ONLINE and new core interprets it as end of sequence, so no capability info received.
New core sends number of PACKET_ID_ONLINE with capabilities with last 1-byte length PACKET_ID_ONLINE. Old core accepts only 1-byte PACKET_ID_ONLINE and works normally. New core concatenates all >1 byte PACKET_ID_ONLINE and provides complete capability info to client.

pros:

• Full compatibility between old and new cores
• Client can access to capability info just in tox_callback_friend_connection_status
• Isotoxin already uses this scheme (except concatenation)

cons:

?

For the second part, the actual encoding, I recently came across a very promising format, http://cbor.io/ or https://tools.ietf.org/html/rfc7049 (despite the .io domain it doesn't seem to be hipster technology).
In short this is like an improved and binary encoded JSON/MsgPack combination which seems to gain traction in the IETF, especially in the world of embedded systems and IoT, because of low resource usage.

The proposed format for capabilities would be a map of features, e.g.:

[
    audio : bool,
    video : bool,
    groups : bool,
    files : bool,
    client-id : tstr,
    bot : bool
    ...
]

For Message V2:

[
    msg : tstr,
    sent_hint : int
    ...
]

This would then be serialized using CBOR and split into multiple packets as stated above and get passed from toxcore directly to the client.

Pros:

• very flexible and extensible, more than a struct
• binary format -> lower encoding space needed than any text based format
• seems to be future proof, client picks only keys it's interested in, others are ignored
• there are CBOR parsers form many languages
• supports arbitrary extension, even client specific ones without breaking other clients
• we could base message V2 on this, to do it in a more flexible way this time

Cons:

• needs a solid CBOR implementation, which as any parser is not easy to get correct
• a struct is still smaller, but way less flexible

[iphydf]

I've considered CBOR before choosing msgpack before. I had (and probably still have) reasons for that choice, but those are not directly relevant now. What's relevant now however is your reasons for either introducing a second binary format next into the toktok stack or migrating everything off msgpack and onto cbor. So basically: what in cbor justifies either the additional cognitive overhead of understanding and maintaining two formats or a migration?

[tox-user]

CBOR is pretty new, right? Wouldn't it be better to use an older format that has existed for a while or does it not make any difference in terms of security?

[SkyzohKey]

Finally! I've proposed this JSON capabilities packet so long that I'm all for your proposal ! Cbor seems cool but you need to know that their is BSON too, that provides binary encoded json-like syntax.

+1 for this anyway!!

[sudden6]

@iphydf what are your reasons?

I'm not strongly opposed to using msgpack. I choose CBOR because it seems very well specified (in contrast to JSON), has a very compact encoding, a data definition language (CDDL) and seems to be the go to standard for new protocols from the IETF.

@tox-user ~5 years, I don't think it's that new.

@SkyzohKey IIRC BSON has a not as compact encoding as CBOR or msgpack

EDIT: I also think we should add a uint16 in front of the packed capabilities, exposing the following flags: audio support, video support, filetransfer support, groupchat support, extended capabilities supported, bot

[zoff99]

@sudden6 i dont fully understand the "message V2" alternative. can you give more info on this?

[sudden6]

@zoff99 basically we would transfer the message as file, like you proposed, but the content and additional fields like sent time hint and so on will be encoded using CBOR

[zoff99]

@sudden6 thats fine by me. as long as we have a uniqe hash per each message.
do you have some time to make a PoC in qTox?

[sudden6]

@zoff99 I don't think a PoC makes sense before even deciding on a serialization format.

[zoff99]

it'll go the way of groups then ...

[tox-user]

@zoff99 doesn't it have to be implemented in toxcore first to add it to clients?

[SkyzohKey]

@tox-user clients can be used to implement features by having a fork of c-toxcore to compile against. :)

[sudden6]

this will be 95% client specific implementations, so not much to change in toxcore

I'm waiting mainly for feedback from @iphydf about why to prefer msgpack over CBOR, if the reasons are good it will change the implementation, so it doesn't make sense to create a PoC

[iphydf]

We already have msgpack in several places in toktok. The question is the other way around: what in cbor justifies a migration or the maintenance of two binary formats?

[iphydf]

Also, who is going to do the migration and/or the implementation of the packing/unpacking libraries?

[sudden6]

@iphydf what is msgpack used for in toktok? Also can you state the reasons why you choose msgpack over CBOR for this?

[zoff99]

@sudden6 having a new dependency client side, that doesnt sound good.
it means now every client will implement differently with new and different bugs

[SkyzohKey]

Then implement features IN toxcore. Don't let the client dev's do the work that belongs to tox.

[sudden6]

I think clients definitely need the ability to have access to the raw data to quickly integrate new extensions without needing a new c-toxcore release every time. I can live with additionally letting c-toxcore decode the well known or standardize elements to a struct, though.

[sudden6]

So, for a revised specification I'll go with msgpack for now, since I don't want to port the rest of c-toxcore to CBOR and the availability of libraries is better.

Current ONLINE packet format:

|Length|Value|Description|
|-|-|-|
|1|0x19| PACKET_ID_ONLINE|

New ONLINE packet format:

|Name|Length|Value|Description|
|-|-|-|-|
|ID|1|0x19| PACKET_ID_ONLINE|
|Flags|2|0x0000 - 0xFFFF |Extension flags, see below|
|Size|2|0x0000 - 0xFFFF |Size of the msgpack encoded rest in bytes|
|Capabilities|[0,]| |msgpack encoded capabilities, structure to be specified|

|Flag Mask| Description|
| - | - |
|(1 << 0)| Reserved for eventually using CBOR (then this will be 1), must be 0|
|(1 << 1)| Followup packet, if this flag is set this packet doesn't contain the Size field and data starts after the flags |
|(1 << 1) - (1 << 15)| Reserved for future use, must be sent as 0|

msgpack encoded part

The top level of the Capabilities structure is a fixarray with the following elements:

| Index | Name | msgpack type | Description |
| - | - | - | - |
| 0 | version | positive fixint| Version number of this protocol, increases with feature level |
| 1 | standard | array16|fields that every client implementing a version must send |
| 2 | optional | map16 serialized to bstr|field that are defined by the toktok spec, but optional |
| 3 | custom | map16 serialized to bstr|for custom application specific extensions |

version field

This field is only written by a toxcore implementation

standard field

This field is only written by a toxcore implementation

For version 0 of the protocol, standard should be empty.

For version 1 of the protocol, standard contains the following fields (not final yet):

|Index|Name|msgpack type|Description|
|-|-|-|-|
|0|basic-feature-flags|uint32|See description below|

|Flag Mask| Name |Description|
| - | - |-|
|(1 << 0)| send-audio| 1 if the client can send audio, 0 else|
|(1 << 1)| recv-audio| 1 if the client can play audio, 0 else|
|(1 << 2)| send-video| 1 if the client can send video, 0 else|
|(1 << 3)| recv-video| 1 if the client can display video, 0 else|
|(1 << 4)| send-file| 1 if the client can send files, 0 else|
|(1 << 5)| recv-file| 1 if the client can accept file transfers, 0 else|
|(1 << 6)| group | 1 if the client supports group chats, 0 else|
|(1 << 7)| bot | 1 if the client wants to indicate it's a bot, 0 else|
|(1 << 8)|bot-info| 1 if the client can provide an answer to the !info command, 0 else (Note: most useful for bots)|
|(1 << 9)|bot-help| 1 if the client can provide an answer to the !help command, 0 else (Note: most useful for bots)|
| Rest | Must be sent as 0, for future use |

optional field

The key type for this field is fixstr and keys are to be encoded in UTF-8, with the preferred character set ofa-zA-Z0-9 as well as _ and-. There's no terminating NUL byte at the end of the string!.

The following keys and meanings are specified:

|key|msgpack type|Description|
| - | - | - |
|name|str| Name of the client|
|version|str| Version string of the client version, releases should use only numbers if applicable, for dev builds a git version hash or similar is preferred|

custom field

Each client/toxcore implementation should create their own namespace using a fixstr key with the name or shorthand name of the client, e.g. qTox, uTox, Antox, Trifa,...

API

The fields version and standard are only written by the toxcore implementation. version needs no API, because it's hardcoded and not changeable after a release of the implementation. All features supported by the implementation should have an API to enable/disable them in the application.

The implementation should provide a way to accept byte strings for the optional and custom fields. The implementation should provide a way to retrieve received byte strings for the optional and custom fields.

NOTE: Since this got way bigger than I initially thought, maybe we should go the route of messageV2 and send this structure as a filetransfer, preferably by submitting a hash of it via the ONLINE packet, to allow caching on the client side.

[sudden6]

Open for edits in https://gist.github.com/sudden6/0e7cb66749450822de8013540d780f6e