Flatbuffers: Feature RFC: Flatbuffers support for Optional types.

[CasperN]

Note: This top comment is heavily edited. It was kept up with the current state until this issue closed.

Motivation

Flatbuffers should allow users to choose to use optional types. There has been some interest in distinguishing between default values (which are not stored in the binary) and some notion of None which the user controls.

Here are some links to previous interest in this idea:

• #333
• #3777
• https://groups.google.com/forum/#!topic/flatbuffers/1hrNtBI0BQI
• https://github.com/google/flatbuffers/issues/5875#issuecomment-619737916

Currently, a user can control a field's presence in the binary by specifying "force_defaults" and checking "IsFieldPresent" which is a bit of a hack. This proposal should define proper Flexbuffers optional types, which should be a better way of doing this. Use of this feature is only advisable for new fields, since changing default values is in general backwards-incompatible.

How do we represent this in the schema file?

We will specify it like so

table Monster { mana: int = null; }

This visually implies that optional types are at odds with default values and is "consistent" since the value to the right of the equals sign is what we interpret non-presence to mean.

Change to Schema Grammar:

field_decl = ident : type [ = (scalar | null) ] metadata ;

~We can add a field tag, e.g. "optional" or "no_default", that triggers this behavior. Hopefully no one is using those tags. Maybe we can make it specifiable to flatc, an "--optional-field-keyword-tag" flag, just in case people are using it and can't stop.~

How do we represent this in the binary?

We are going with option (A).

(A) Non-Presence means None

Instead of omitting zero-like values, the generated code must store them. Non-presence for optional fields no longer means "whatever the default is," now it means None. You can interpret it as "the default value is None". This also means we cannot specify both specify a non-null default and mark the field as optional.

Pros:

• This seems more intuitive.
• It aligns with the "force_defaults" + "IsFieldPresent" hacky manual approximation of this feature.
• If Nones are more common than zero-likes then this will have smaller binaries.

Cons:

• @aardappel thinks this is harder to implement

"making presence an indicator would require we pass this special field status down to the field construction code to override the current val == default check, which means slowdown, codegen and runtime changes in all languages.. whereas my "least likely to be used default" trick requires no changes"

~(B) Some Sentinel value means None~

In this scenario, zero-like values are still not stored. Instead we choose some "sentinel" value which we interpret to be None (e.g. int can use int_min and float can use some kind of Nan).

Pros:

• @aardappel thinks this is easier to implement

"it requires the schema parser to set default values for you, and no changes anywhere else"

• If zero-likes are more common than None then this will have smaller binaries

Cons:

• Someone might want to use the sentinel value (Hyrum's law).
  
  
  
  • This can be mitigated by publishing the sentinels and letting users decide whether they need the sentinels.
  
  
• This probably won't work for fields representing raw bits.

How do we represent this in every language API?

We'll need to change the type signature of all generated code (building/reading/mutating/object/etc) around the optional type to signal its optional-ness. I think we should use the language's local standard for optional types. Suggestions:

• Python: Optional[T].
• Rust: Option<T>.
• C++17 has std::optional<T> but its not obvious what to use for earlier versions. T* would work.
• Java: Optional shows up in Java 1.8 and triggers autoboxing, so idk :/

The exact generated-API for a language should be discussed in the PR implementing this feature in that language.

Out of scope

(I'll add links if you make issues for these feature requests)

• Syntactic types
• Default values for strings and tables

TODO

| task | owner | done
|---|---|---|
| Change flatc to support schemas with optional types and cause an error if they're used in unsupported languages | @CasperN | #6026 ✅
| Implement optional type API in C++ | @vglavnyy | #6155 ✅
| Implement optional type API in Java | @paulovap | #6212 ✅
| Implement optional type API in Rust | @CasperN | #6034 ✅
| Implement optional type API in Swift | @mustiikhalil | #6038 ✅
| Implement optional type API in lobster | @aardappel | ✅
| Implement optional type API in Kotlin | @paulovap | #6115 ✅
| Implement optional type API in Python | @rw?
| Implement optional type API in Go | @rw?
| Implement optional type API in C | @mikkelfj | ✅
| Implement optional type API in C# | @dbaileychess | #6217 ✅
| Implement optional type API in Typescript/javacscript | @krojew | #6215 ✅
| Php, Dart, etc ... | ?
| Update documentation to advertise this feature | @cneo | #6270 ✅

[edits]

• added todo list
• added points from the discussion to each section.
• added out of scope section
• Decision: go with (A) and use = null syntax in schema file (cross out alternatives)
• Updated TODO list, finished parser, Rust in progress
• Change to schema grammar, link to swift PR, Note at top
• Added more languages to the TODO
• Lobster support 🦞
• Kotlin and C support
• Java, C#, TS/JS support and docs, issue closed, no longer editing.

[aardappel]

I mentioned no_default since all fields are already optional, in a way. But it's not a great name either :)

A:

• The slowdown can be avoided if a language implementation has a second set of runtime functions for each kind of scalar that does not take a default at all, and the codegen calls this function instead for fields that have this attribute set. I'd say this would have to be the implementation technique if we decide on this approach, since passing another bool to test down to the existing functions would penalize code that doesn't use this feature, which is currently all of it.
• Generally the bigger downside of this approach is that the feature won't work correctly until all languages implement it. I'd feel better about it if a PR implementing this feature would attempt to cover most languages at once.
• In C++, an int field with this attribute set would return an int * which may be nullptr, which is equivalent to the current workaround, which is to make the field of type struct Integer { i:int; }. We could then decide to make this generate std::optional when generating code explicitly for C++17 and up, though frankly that is not actually an improvement in ergonomics. The fun thing about the int * is that it can actually point into the buffer, so doesn't actually touch the int in question.

B:

• Attractiveness of this approach is that it can work reasonably even if languages don't support it explicitly. It is however not very ergonomic having to deal with these special values manually, though helper functions can make this better bool FieldPresent(int i) { return i != 0x80000000; }
• This trick is already in use by some users. @lu-wang-g

[aardappel]

@vglavnyy @rw @paulovap @mustiikhalil @mzaks @dbaileychess @dnfield @mikkelfj @krojew ..

[paulovap]

A:
On Java, Optional shows up on API 1.8, so we would have to set this API as minimum requirements. Also, the use of Optional would trigger autoboxing for scalars which is a performance penalty.

B:
I have a question here. Is the plan to set up default sentinels for each type or allowing users to define their own sentinel? Like:

table Test {
    myInt: Int32 (sentinel=-1);
}

That way the user would have control of what sentinel to use.

[mikkelfj]

I agree this is an issue. I mentioned this in the FB2 discussion as analyze SQL NULL values.

I have been thinking about this problem,

A) using is present: this doesn't work well with tables and strings etc. because these have not default value. A null value would have to be an offset to a special null object that would have to be checked on access, or the current null value would have to change interpretation. On a related note I have been considering offering a dummy object instead of NULL when the C api discovers a null because that is a safer and simpler approach. You can ask if it is a dummy object explicitly, and otherwise access it safely but getting only dummy results.

B) sentinel values are not good, and if they are, they are better done by the end user for specific use cases, i.e. FlatBuffers need not change.

C) My proposal. Add an additional bit-vector field at vtable offset 0 for table that have optional values. The bit-vector is stored inline in the table, but could also be an external offset if is believed that a) the are large enough to make this worthwhile, and b) data sharing is likely.

Now you can access data just as before, without overhead, but you can explicitly ask if the field is marked as NULL, if you care. This can be very efficient, and the builder can easily add this information as it already has vtable vector in memory that it writes out, and can add the null vector at the same time.

[mikkelfj]

The reason why B) is not good is because you cannot roundtrip SQL data transparently. This is an absolute requirement for this feature.

[CasperN]

One thing we can do for (A) and the "simultaneous implementation problem," which I agree is a huge problem, instead of implementing it in a huge PR, is to not generate code for optional fields (and warn) if the language isn't ready yet. I think this will at least make code review easier.

Perhaps we can also give users the option to use existing codegen and zero-defaults if they really want it and promise they'll always check for field presence manually... but such an option is definitely a foot-gun.

I agree with the use of nullable pointers over optional for performance reasons. Rust should prefer Option<&mut T> (if it had a mutation API) since raw pointers are frowned upon in that language and it'll compile down to a nullable pointer anyway.

[mikkelfj]

Oh, regarding C) it might be strange if a NULL field returns 4 because 4 happens to be the default value. For this reason it makes sense to support a default null value also. That is effectively option B), but without the hack since you have precise NULL information.

EDIT: this would add runtime overhead because it would need to check the null table to see if default or null default should be returned. But at least it only happens for absent values.

On Wouters suggestion to have a parallel set of accessors. That would add a lot of loud on compilers to strip out even more templates in C++ or macros generated inlines in C, and complicate code generation quite a bit in C.

[mikkelfj]

We could also require the default values only apply for non-optional fields, and always store values in optional fields, unless they are NULL, and have the bit vector. The default value then becomes the return value if NULL, and NULL can be checked separately.

[aardappel]

@paulovap the sentinal IS the default, so if you wanted to use -1, you could already write myInt:int = -1 already :)

@mikkelfj certainly do not want to add extra encoding to the binary to support this feature, I think that is too high a cost for such a simple feature, even compared to A).

@CasperN we'd have to actively error-out if someone compiles with an unsupported language, rather than ignoring the field, I think, like we currently do with other unsupported features. Doing that in multiple PRs would be ok, as long there is an effort to support most languages.

[mustiikhalil]

As @aardappel already said, most of the languages can return a null value (from the ones I’ve worked with) so why not introduce an optional flag ‘age: int (optional)’ and instead of returning a default value we just return null. Most of languages has some type of optional, if they don’t we can mark the name of the object as optional when we generate the code. This won’t require us to play with the binary data, and would only require small refactoring for each of the language generators. As as example swift already can return optional structs, tables, strings (we are just missing scaler optional)

[mikkelfj]

Well, if the cost is too high, I think we should not do it. I'd rather not have a half-baked solution on top of an already half-baked solution.

[aardappel]

@mikkelfj what is half baked about A) ? only scalars require this feature, all other types can already be null.

[CasperN]

@mikkelfj

Can you elaborate on

A) using is present: this doesn't work well with tables and strings etc. because these have not default value. A null value would have to be an offset to a special null object that would have to be checked on access, or the current null value would have to change interpretation

I'm not sure what you mean by special null object, can you give an example?

[vglavnyy]

With C++ the simplest way is to embed an optional scalar into a struct.

table Foo{
 bar : int (boxed);
}
// implicitly translated to:
table Foo{
  struct _ { bar : int = 0 }; // anonymous transparent proxy
}
//, or translated to
table Foo{
  struct bar { _ : int = 0 }; // named proxy
}

Struct might be present or not.
With C++17 it is possible to generate Optional or Expected for any pointer-like (non-scalar) fields.
In a json it can be like this:

"Foo" : {
 "bar" : [-12]
}
// or 
"Foo" : {
 "bar" : { "_" :-12 }
}

[krojew]

I'm for option A, since option B is what we have now in practice. On _how_ to implement A - well, that's another problem. As far as I know, all our supported languages (maybe except Lobster, whatever that is 😄 ) have some notion of optional data, so we're fine in terms of compatibility. Of course, this would require making changes everywhere, but I personally, am fine with such cost if we finally get proper optional data support for scalars.

[mikkelfj]

@aardappel

what is half baked about A) ? only scalars require this feature, all other types can already be null.

First, the current use of null and default values is half-baked because strings and other non-inline data do not have default values, different from scalars. We could add default values for strings.

Second, hijacking presence for NULL is half-baked because it hijacks the current use of default values for a differen purposes. Granted, it can work if you change schema semantics for optional values, but now you have yet another layer of whatever presence and null offsets means, depending on a schema flag.

I'd rather have null as a clearly stated non-negotiable data point for all imaginable data types. If not a bit vector, it could also be a special vtable offset value like 0xffff - it takes up more space, but only affects verifiers since non-supporting implementations would just see a future version of the format and choose the default value of the field, while supporting implementations would only take a performance hit if detecting the field is out of range, which it has to check anyway.

@CasperN

I'm not sure what you mean by special null object, can you give an example?

It was actually Wouter's suggestion originally when I once mentioned the lack of Null transparency for SQL data.

The idea is that to avoid interfering with the current null interpretation, you store a special object in the buffer. The uoffset (not voffset) poins to this object for NULL entries. It can be added to the buffer as an object with just an empty vtable (+/- handling of required fields for the type, but ideally just one object for all types, but NULLable shouldn't have required fields?). Any user level code can do this today, except it is hard to tell if an object is the NULL object or some other object. One could argue that you could just store and actuall NULL instead, but that changes the semantics of the current system and would prevent adding default values for strings and tables etc.

[mikkelfj]

To extract a key point from my above comment:

D) store NULL values as virtual table voffset 0xffff because it is transparent to current implementations and acts the same as a missing entry with voffset 0x0000 except it looks like a future version instead of entirely missing. in both cases a default value is returned. It is now possible to add a separate support function: IsNull() similar to IsPresent(), but it avoids overloading current semantics, and it does not introduce a new bitmap data structure as my proposal C). The benefit of C) is that you can have short vtables if all the last fields are NULL, but I think D) is probably preferable.

A larger benefit of D is that you entirely avoid changing the schema semantics with an 'optional' setting because you can just store as value as NULL by adding an API function: SetNull() in the builder. Of course you could add attributes to block, or allow NULL, but it is not required.

[mikkelfj]

Sorry, I'm not thinking straight. You cannot transparently store 0xffff as a vtable offset. Future versioning requires a field in the vtable outside the currently expected max vtable size, but you cannot store invalid offsets within the table because common reader logic will not check the validity of this offset, only verifiers will. The approach could still work if all readers learned to check this similar to the 0 check they perform now, but that is not practical. That is why the bitmap approach is better.

[CasperN]

@mikkelfj

Second, hijacking presence for NULL is half-baked because it hijacks the current use of default values for a differen purposes. Granted, it can work if you change schema semantics for optional values, but now you have yet another layer of whatever presence and null offsets means, depending on a schema flag.

I see it as as having the same semantics, just that the default value is a value outside of the type of the scalar itself, NULL.

---

I might be misinterpreting you but I think you're looking for a default vs null system for vectors, tables, and strings (in addition to scalars). The current state of the world is that scalars are default-y and the more complex objects are null-y. This proposal allows users to choose default-y or null-y semantics for scalars but you're worried that it may complicate a future feature where users can choose to make the complex stuff default-y. Therefore this proposal _should also design_ a solution so tables, strings, and vectors can be default-y.

---

On the bitmap thing,

I'm not sure of the value of introducing bits to track "default value" vs "null value". It seems to me like a premature optimization of a case where some default and null values are very common, that has a cost on every other use case, though I'm not sure what cases you have in mind.

[aardappel]

@mikkelfj

First, the current use of null and default values is half-baked because strings and other non-inline data do not have default values, different from scalars. We could add default values for strings.

I believe that is a separate issue. Not having defaults is maybe not great, but it doesn't affect the string's ability to be optional in the sense of this issue. In fact, non-inline data (and structs) are ahead of scalars in this sense.

There are "issues" with supporting default values. For strings it is reasonably easy, but some form of default table is not exactly a lightweight feature.

We're not hijacking anything, we're merely extending the existing semantics. Under proposal A), an int (optional) would have the same semantics and representation (and almost the same API) as struct Int { i:int }. It is not that new.. for me the big deal is getting all languages to support it.

And I am not at all familiar with SQL, so I really don't follow your concerns around null in general. I am mostly familiar with the nulls in C/C++/Java.. and I believe Flatbuffers follows a similar model.

[mikkelfj]

And I am not at all familiar with SQL, so I really don't follow your concerns around null in general.

NULL in SQL is controversial because the original designers (see book by C.J. Date on SQL) considered it impure in an otherwise elegant and consistent model based on algebra. However, this leads to requiring a defined 0-value value (like an empty string, or a numeric -9999, or whatever) in the domain of any given type, such as integers or strings. Pragmatics found it better to explicitly have a NULL value outside the value domain when they had lots of tables with data in which only few contained actual data. You can also select records where fields are not NULL or vice versa without having to define what a 0-value is in a given context.

In actual SQL use all of this can be condensed down to: If you have data extracted from a database that you want to process and reinsert into the database, you can expect to see NULL values, and you cannot consistently assume that you can map NULL to some other value (a 0-value) because that might overlap with a value meaning something else. So if you need to reinsert data without loss, you must keep track of the NULLs. Of course this is a mess, but a practical necessity.

Now, for this to work with FlatBuffers, you need to know exactly what is a NULL and not just what is NULL by convention wether it be a NULL string, or a default scalar value or an absent scalar that is different from a present default value.

If you add optional types to the schema, you can impose exact semantics for NULL simply by defining what NULL means for the given type and make sure that the equivalent type for SQL usage can represent all values in addition to NULL. This can actually work, but it is somewhat messy and hard to explain, but can be helped with API support.

If you do not modify the schema but just decides what constitutes a NULL, you are open to all sorts of interpretations in present use which will not work.

Optional types corresponds to SQL schema types where a field can be allowed to be NULLable, or not.

You also need to consider how the data can be represented in JSON without loosing NULL, but this should be workable.

I'd prefer a bitmap because it takes all guesswork away for any current and future types. It can be directly exported to other data formats such as Arrow that also uses bitmaps for NULLs, and it is fast to process because you can just scan the bitmap rather than special casing each kind of value that can be NULL.

Bitmaps do not interfere with current implementations because they are just an extra field in a table that you are not forced to look at if you do not care, but of course you cannot write NULLs if your API does not support it, unless you opt to write the NULL bitmap yourself, which is possible because it would just appear as a ubyte array.

So, if you do not want bitmaps, it can still be made to work if you introduce the optional attribute in the schema, but if you do not, it will be a mess.

[mzaks]

What about introducing the optional scalar types:

Byte Ubyte Bool
16 bit: Short Ushort
32 bit: Int Uint Float
64 bit: Long Ulong Double

where compiler when sees those per default creates structs:

struct Int {value:int};
struct UInt {value:int};
etc...

And languages which support actual optionals can replace this default behaviour by generating an API with optionals. This way there is an option to have a "small" non breaking change which will work for all languages and opens up gradual migration path for every language specific code generator.

PS: If by any chance people already have a Int type in their schemas, we can introduce a flag to suppress the Boxed scalar struct creation.

[CasperN]

@mikkelfj

[null] leads to requiring a defined 0-value value (like an empty string, or a numeric -9999, or whatever) in the domain of any given type, such as integers or strings. Pragmatics found it better to explicitly have a NULL value outside the value domain

This seems like options (B) and (A) respectively.

you can expect to see NULL values, and you cannot consistently assume that you can map NULL to some other value (a 0-value) because that might overlap with a value meaning something else. So if you need to reinsert data without loss, you must keep track of the NULLs ... Now, for this to work with FlatBuffers, you need to know exactly what is a NULL and not just what is NULL by convention whether it be a NULL string, or a default scalar value or an absent scalar that is different from a present default value.

I think we know exactly what is a NULL under (A) for optional scalars in a table. They are null if and only if IsPresent == False.

If you do not modify the schema but just decides what constitutes a NULL, you are open to all sorts of interpretations in present use which will not work.

I'm focusing on the words "in present use": I think you mean fields that are currently scalars cannot safely be redefined into optional scalars because we cannot distinguish between NULL and default. This is correct and intended. For existing fields, changing the optional-ness is a bad idea (and users will need to be educated about that).

Adding bits, as you suggest in option (c), will allow for retrofitting null onto existing scalars. Are you claiming retrofit-ability should be a requirement?

Fwiw, my opinion is that allowing existing scalars to convert into being optional is not worth the extra bits of option (c) and educating users that "optional (or lack thereof) is permanent" is sufficient for this feature.

You also need to consider how the data can be represented in JSON without loosing NULL, but this should be workable.

Good point. At first glance, json's null keyword/literal seems like a good fit to me.

---

@mzaks

I like your idea because it lets all languages generate a default API for optionals, though it is essentially syntactic sugar in the schema. What would happen to the Int/Uint after the language in question adds optional support? Would that be a code-breaking change?

Regardless, I think we should start with the "error-out if someone compiles [optional scalars] with an unsupported language" approach, since its a lot easier. Your idea should be revisited in the scenario where we go ahead with option (A) but haven't implemented optional scalars in all languages and really need to. However, I don't think we'll end up in that situation since a prerequisite of accepting this feature is confirming we'll actually do the work in enough languages.

---

@aardappel, what would you like to see before deciding between (a) and (b) and giving a go/no-go on this feature? What is the minimum set of languages that must support this feature for it to be worth it?

I can volunteer for the following tasks (unless someone else wants them):

• Changes to flatc (adding the tbd optional field attribute and error-ing out)
• C++ generated API changes
• Rust generated API changes

[mikkelfj]

Adding bits, as you suggest in option (c), will allow for retrofitting null onto existing scalars. Are you claiming retrofit-ability should be a requirement?

One the one hand it would be useful, on the other hand it still requires marking the table as having NULLable fields, as opposed to marking individual fields. Bitmaps also provide fast processing of data which can be rather helpful, and it takes up less space than forcing default writes on some cases.

But mostly it rubs me the wrong way that you can declare -4 to be the default value, but it really means nothing, since -4 will be stored if not NULL, and if NULL, well, it doesn't really matter what the default value is.

[mzaks]

Here is how I imagined my proposal in detail:

1. Make following changes to parser and the AST model. Make sure that parser accepts the new optional types (e.g. Int) and marks which of them were used in schema. Add a new bool property to AST model representing structs (e.g. syntaticType: bool). Generate AST models for marked optional types with syntaticType = true.
2. Start changing the code generators to not generate types for syntaticType = true, but use native types like Option<i32> in Rust.

Step one can be done by one person and will work even without step two, the "old" code generators will just generate code for the Int struct and use this type for table fields. The new generators which know about syntaticType will not generate the Int struct, but type the fields in the table appropriately (e.g. Option<i32> in Rust).

Potentially the syntaticType concept could be improved upon to support more native types for a language e.g.:

struct UUID (syntaticType) {
  bytes: [u8];
}

Where in all languages which have a native UUID type (e.g. Swift), the native type is used and for other languages a struct is generated.

[aardappel]

@mikkelfj from what you say, you'd be happy with A) also, it encodes a null value out of the domain explicitly. Adding bits wouldn't make this any different, they are redundant.

@mzaks that could work, but I wouldn't want to clutter generated code with all these definitions for these auto generated structs. Option A) is what you get if you instead access the scalar directly as if it was a struct. I'd prefer that to adding new types.

@CasperN option B) is trivial, it is merely a convenience feature on the status quo. It is just if (attr == "optional") default_val = INT_MIN. I am "warming up" to option A), but I'd like whoever person(s) commit to implementing this also commit to a "majority" of languages over not too long a time. They don't have to do it all themselves, but maybe help in tracking and coordinating with language authors (quite a few have been active on this thread already).

[CasperN]

@aardappel, I can do tracking/coordination too.

@mzaks I'm a little wary of making that part of making your SyntacticTypes feature part of optional scalars since it seems like a bit of additional work. I think it should be a separate feature request.

---

Can everyone participating vote on A vs B vs do-nothing and maybe volunteer a language to implement?

I support (A). I'll track work by editing the top comment

[aardappel]

Ok, @CasperN is in charge of this feature :)
I'll vote (A), then (B), then (do-nothing) in that order.
I can help with C++ if no-one else takes it on.. @vglavnyy @yaoshengzhe ?

[vglavnyy]

What is the plan for implementing of (A)?
In fact, all scalars in binary already are optional values. A scalar value is present or not.
We only need a special attribute:

• to enable the generation of an additional getter with MayBe return value;
• to modify implementation of setter (local force_defaults).

That’s all. These changes are compatible with already existing binaries and user programs.
The real problem is generated Object API. This attribute will change a type of a scalar inside T-generated tables.

Example:

table Monster { mana : int = -1 (maybe); }

class Monster {
  int16_t mana() const { return value or default; }
  flatbuffres::optional<int16_t>  get_mana() const;
  void add_mana(int16_t mana) {
    fbb_.AddElement<int16_t>(Monster::VT_MANA, mana); // WITHOUT default value!
  }
};
// Optional for default-constructable and copyable T.
// Can have a special implementation if T is a pointer.
template <class T > class optional {
  optional(); // none with T{};
  optional(bool none, const T& val); // only with explicit default value
  T& value(); // return ref to value or ref to default value (never throw)
  T& operator*(); // return value()
  operator bool() const;
  bool has_value() const;
  std::optional<T> get() const; // down-cast to std::optional<T>
  operator std::optional<T>() const; // implicit down-cast to std::optional<T>
};

[CasperN]

@vglavnyy,

In fact, all scalars in binary already are optional values. A scalar value is present or not.

Currently they're optional in the binary, but non-presence is interpreted as "return the default value according to the schema" so from the perspective of user's code, "all values are there". Proposal (A) allows non-presence to be interpreted to mean None (ie "Nothing" in the words of haskell's optional type, "maybe"). That is semantically at odds with "non-presence means default".

I think your table:

table Monster { mana : int = -1 (maybe); }

should be an error because I think there shouldn't be a non-null default for optional types. The following is fine.

table Monster { mana : int (maybe); }

Minor bike-shedding idea, we could specify optional types like so:

table Monster { mana : int = null; }

This visually implies that optional types are at odds with default values and is "consistent" since the value to the right of the equals sign is what we interpret non-presence to mean.

The real problem is generated Object API. This attribute will change a type of a scalar inside T-generated tables.

As discussed above, changing a field from default-y to optional is a binary-semantic breaking change. It is also intended to be a code breaking change (in all languages except maybe python) since we want to show, in code, that the type is optional. And not just in the object API, even the normal accessor should change: e.g. optional<int16_t> mana(); or int16_t* mana(); (the exact form should be debated in the C++ PR).

What is the plan for implementing of (A)?

1. First, flatc should recognize whatever syntax we choose to represent optional values and error-out when optionals are detected.

2. For each language, implement support for optional types (with whatever API is best for the native language). In that PR, remove the flatc error when generating code in that language.

3. When we have enough languages supporting this, then start changing the documentation and advertise the feature to users.

[mikkelfj]

I agree with @CasperN on why an optional type is needed. But rather than using an attribute, Casper alternatively suggests:

table Monster { mana : int = null; }

That is not bikeshedding, that is a great proposal because it removes the ugliness of having to explain why a default value is not allow for optionals, or alternatively figure out how to handle them in the context of optional. If you dislike the null keyword, it could also be none.

One potential issue is when converting existing data to a nullable type. In this case absent values gets converted from, say -1, to null. You might also want be able to return a default value when a value is null, rather than having to check for it. Essentially, you want the appearance of B by default, even if you use A. For this reason, I'm not sure it is such as good idea after all.

If we do this, I am almost sold on A without bitmaps.

Maybe we should add support for default values of strings such that one does not have to get null exceptions when reding null values by default. Here a strings default could be null, "", or "killroy wansn't here". Note that A) actually eats into the existing value domain because null is effectively part of the string domain as it is - unlike the default values of scalars, and the same for tables and unions.

As for tables I'd like a default value for those as well. Just the empty table with an empty voffset table.

Tables with required fields shouldn't be nullable. This prevents having a default table value.

In FlatCC I'm inclined to always return a value for optional types. The default value can still be null for strings and tables, but it has to be stated explicitly or the default will be an empty string or an empty table.

[CasperN]

One potential issue is when converting existing data to a nullable type...

Let's avoid the issue by saying "please don't do that." The official documentation says "You generally do not want to change default values after they're initially defined." That applies here too, doubly, because it changes the binary semantics _and also breaks code_.

Maybe we should add support for default values of strings
...
As for tables I'd like a default value for those as well

I agree, but please open a new issue for that.

[vglavnyy]

For example:

• a server uses an old schema without an optional field 'mana'.
• a client has updated schema with the mana field.

All client programs should know the default value if mana() isn't present in messages from the server.
Generate or use any addition sounded methods to get this default will be extremely terrible.

 auto opt = m.get_mana();
 // in another routine:
 opt.value_or(m.mana_default()); // here I don't want to know about monster `m` and about `mana` name.

This is isn't a big problem, I can write an ugly macro to generate my own optional value by a field name.
But I sure that optional field should have declared default value.

[CasperN]

@vglavnyy
I think you're offering an interpretation for non-null default values for optional types: That value will be available as a constant somewhere (e.g. we generate mana_default()) - though I'm not sure what would it return if the user didn't specify a default. The obvious options are null, 0, and 'not generated'.

This interpretation allows the user to track more information in the schema at the cost of a little code bloat and usage complexity. Note that, optional-defaults will be stored in the binary, unlike normal defaults, which might be surprising. What does everyone think of the complexity trade offs? I could go either way.

[mikkelfj]

Based on my previous comment, I now find it quite natural that the default value of an optional field is used as the return value for normal field access such that the field accessor function can remain within its value domain, and such that applications that do not care, do not have to care. Applications that need to check for null simply call field.IsNull, or field.IsPresent().

I have been given this some more thought: if a safe value is always returned, including empty tables, then the only unsafe access is when a union type not inspected, an offset to structs because it is not easy to provide a default struct if one isn't specified and structs could be large. Fortunately such structs only happen as union members, so it falls into the first category: check the union type before access.

Additionally, what happens if a union value is not None, but holds a value of a known type which is Null?
This is already a problem today: FlatCC does some interpretation on these border cases, and I think it is documented in the flatcc binary format document, but I don't recall the details up front.

In any case, I think Null values should always map to the union None type even though this results in a loss of type information.

[CasperN]

@mikkelfj

I now find it quite natural that the default value of an optional field is used as the return value for normal field access such that the field accessor function can remain within its value domain, and such that applications that do not care, do not have to care. Applications that need to check for null simply call field.IsNull, or field.IsPresent()

It sounds like you want optional fields to be implemented as a field-specific force-default, without changes to the generated API. While it makes existing default-y code compatible with optionals, I think that's a bad idea from a semantics standpoint. If a user provides a schema where the nullability of data is important, the type system should force them to confront that. Programmers will forget to check IsPresent, turning a compile error into a runtime logic error. Also, this system is effectively the current state of the world and is inconsistent with how nullability works with strings, vectors, unions, and structs. Those use nullable pointers.

I have been given this some more thought: if a safe value is always returned...

I think you want a "safe" default-like value to always be returned (like how it is in Flexbuffers). It is just a coincidence that optional types are sometimes "unsafe" in c because they're represented with nullable pointers that can fail to be dereferenced... but we could use optional<T>, that's safe. Imo, nullable pointers are common and therefore "safe enough" for C users, but this is a language specific implementation discussion.

The problem with always returning a default value is that there absolutely are use cases for optional types. You mentioned working with SQL, which is a great example. The schema-writer decides whether they want optionals or defaults. We must not choose for them. This is literally the most common complaint with proto3.

union type

I think that's out of scope. At risk of expanding the scope of this conversation, I think a consistent nullable-vs-default story would look like:

| type | if schema doesn't say | specifiable defaults | optional |
|---|---|---|---|
| scalars | zero like default | ✔️ users can specify any value | this issue |
| tables | optional | empty table only | ✔️
| strings | optional | users can specify a static string | ✔️
| vectors | optional | empty vector only | ✔️
| structs | optional | needs new syntax so users can specify a default | ✔️
| union | optional | needs more new syntax so users can specify a default | ✔️ |

The last 5 rows are design and implementation work is required to users to _opt into_ safe defaults and avoid nullable pointers. I'll be happy to work with you on defaults-for-all after this issue, but please let it be out of scope for now.

Aside, my flatbuffers 2 request is consistency in the "if schema doesn't say" column

[mikkelfj]

https://www.lucidchart.com/techblog/2015/08/31/the-worst-mistake-of-computer-science/

[CasperN]

@mikkelfj

https://www.lucidchart.com/techblog/2015/08/31/the-worst-mistake-of-computer-science/
The problem with NULL is that it is a non-value value, a sentinel, a special case that was lumped in with everything else.

I think this is an argument against (B) and option (A) is winning 2-0 as of this comment. The author of that article is in support of optional types that can't be implicitly downcast to the thing they contain, like Haskell's Maybe a or Rust's Option<T>. I fully agree with them, intend to use Option<T> in Rust, and suggest using std::optional<T> in C++.

Are you voting for "do nothing" then?

[mikkelfj]

No, since there is no traction for C, it must be A, because we need NULL in the real world.

But that doesn't mean that NULL should be forced in the access API for the reason of ugliness, safety, ease of use, and elegance. That is why I discussed the use of default values to present NULL when attempting to read NULL within the value domain. When that isn't sufficient, you can check for NULL. You can't check for NULL on scalars anyway, but just reading the scalars, but some languages can wrap them into an option type. (In C this happens with unions because a union is actually returned as a two field struct, but that is not reasonable for scalar values). There is also a performance aspect.

Most often you have:

if value not null then x = value else x = 42; proceed with algorithm.

You can avoid this by taking advantage of the default values.
You also often find dict.get(key, defaultvalue) in APIs which are highly useful.

[mikkelfj]

Note that this has the convenience of B) without actually hijacking the value domain. You still have exact NULL when needed, but when you don't need it for a particular computation, you can pretend to have B.

[CasperN]

@mikkelfj

Most often you have:

if value not null then x = value else x = 42; proceed with algorithm.

You can avoid this by taking advantage of the default values.

I think @vglavnyy 's interpretation of default values for optional types might be the way to go then.

table Monster { int mana = 42 (maybe); }

class Monster {
  std::optional<int> mana();  // std::nullopt for non-present values
  int mana_or_default();  // 42 for non-present values
  inline static int kManaDefault = 42;
}

What do you think of that?

Another alternative I thought of is introducing a custom type flatbuffers_optional<int,default=42>, which I'm a bit less comfortable with since not all languages can parameterize by values. It could lead to namespace pollution.

it must be A, because we need NULL in the real world.

Awesome, would you be willing to do the implementation for some language?

[aardappel]

I would agree that this new kind of optional scalar field should NOT allow a default, either by disallowing it (if its an attribute), or replacing it (by using = null, which I also quite like).

I don't see the need for there to be an accessor that still accesses these fields "blindly" by additionally providing a default. Though I am general in favor of such APIs, if you explicitly mark this field as optional, you are indicating you want to be precise about its presence. If you cared for an easy API you should be using a regular field.

And the whole point of such optional fields is maybe also that a default is not easy to find, i.e. all values are legit. Besides, it would give us 3 kinds of fields: standard, optional+default, optional+no_default, and that's frankly more complexity than its worth (remember, we still have to support this is 10+ languages).

So if you receive old data using a new schema that has such a field, then yes, the default value of that field will be null. You must check for it. I don't think this will be a problem.

@vglavnyy by default, I think the value of an optional int field should be const int *, as we return const * types for all fields that may be null currently. We can look into supporting std::optional if we're generating for C++17, but I don't think we should generate our own optional type.

[mikkelfj]

Awesome, would you be willing to do the implementation for some language?

Would someone be willing to help me out on the C language - I'm doing almost everything on my own there?

[vglavnyy]

We can look into supporting std::optional if we're generating for C++17, but I don't think we should generate our own optional type.

A flatbuffers optional isn't necessary if the default value of optional fields is null.
I just found another way to solve the default problem, I can use the native_default attribute.

[CasperN]

if you explicitly mark this field as optional, you are indicating you want to be precise about its presence. If you cared for an easy API you should be using a regular field... it would give us 3 kinds of fields: standard, optional+default, optional+no_default, and that's frankly more complexity than its worth (remember, we still have to support this is 10+ languages).

I find that pretty convincing. Let's go with option (A) and the = null syntax then.

@vglavnyy can you do C++?
@mzaks can you do Swift?
@paulovap can you do Java?
@aardappel can you do the flatc changes?

[vglavnyy]

The =null syntax will complicate the parser. The simplest way is using an explicit named attribute: optional or maybe.
If this is =null, then the parser must be changed first.
It should maps =null into a hidden attribute maybe and adds workarounds for standard scalars.
Only after the tests passed we can modify the code generators.

[mustiikhalil]

@CasperN

  std::optional<int> mana();  // std::nullopt for non-present values

This is the change that would happen for the languages, swift doesn't need anything other than the code gen changes. since it's simply going to be translated to:

var mana: Int32? { let o = ... return o == 0 : nil ? table.getValue }

Plus if we want to add the default values for these optional fields

 int mana_or_default();  // 42 for non-present values
 inline static int kManaDefault = 42;

then the code gen should handle all of this

[paulovap]

One question for @CasperN

Java: Optional shows up in Java 1.8 and triggers autoboxing, so nullable pointers might be preferred too.

How nullable pointers will prevent autoboxing?

[CasperN]

@paulovap, I don't understand Java so maybe I should just cross that out. When I wrote that, I interpreted the "box" of "autoboxing" to mean what Rust means by "box" which is heap allocation (so a pointer into the buffer would be better), but now that I think about it, the term probably means something different in Java.

[paulovap]

This change would move a primitive int on Java into an Integer object, which might incur a big performance penalty.

[mikkelfj]

The =null syntax will complicate the parser.

This is my absolute smallest concern. At least for flatcc it would be probably be easier than dig through attributes. And from what I have worked with the flatc parser, I doubt that would be a major headache, and it is not multiplied by 10 languages.

[CasperN]

Alright, we have Parser support and an example in Rust. I think we're ready to start implementing the codegen changes for our many languages. Thank you, @mustiikhalil for getting a head start for Swift in #6038.

[mikkelfj]

@CasperN can you please summarise the specs of the feature, including the language changes.?

[CasperN]

@mikkelfj, I tried to do so in the top comment, but let me know if this is clearer.

Changes:

• The schema language grammar change is only in the scalar definition:
  scalar = boolean_constant | integer_constant | float_constant | null
• Field declarations with = null are marked as optional by the parser
• This is an error at code generation time unless:
  
  
  
  • The field is in a table
  
  
  • The field's type is a scalar type
  
  
  • This feature is implemented for all the languages being generated
  
  
• Languages are expected to generate code for these fields with their local variant of optional types (instead of just the scalar type).
  
  
  
  • Non presence of optional scalar fields in a flatbuffer binary must be read into a value outside the scalar, like None/NULL/nil.
  
  
  • Serializing a type with None/NULL/nil must be represented as non-presence (vtable pointer set to 0).
  
  

[mikkelfj]

Thanks, your top comment was fine, but a lot has happened since then, unless you edited it.

As to grammar, you probably meant:

field_decl = ident : type [ = scalar | null ] metadata ;
Otherwise null also happens as an attribute, which of course it could, but we never discussed that, AFAIR.

[CasperN]

Thanks, your top comment was fine, but a lot has happened since then, unless you edited it.

Yep, I'm trying to keep the top comment up to date with the most current state

[mikkelfj]

I think it is worth allowing = null on tables and strings as well, just to indicate the semantics, and for consistency. It may be important for schema translations. For example a string may become a NOT NULL in a SQL schema like
https://www.sqlservertutorial.net/sql-server-basics/sql-server-create-schema/
It may also have significance in protocol buffer schema translations, although I haven't looked into that recently.

Sometimes I work on projects that only, or in part only, use FlatBuffers to generate JSON because FlatBuffers provides convenient encoding and decoding and schema management..

[mikkelfj]

Also, as I see you copied my schema suggestion: I made a syntax error, should have been:

field_decl = ident : type [ = ( scalar | null ) ] metadata ;

Note the parentheses and literal null.

[mikkelfj]

If tables and strings can be null, the C API could adopt the following semantics:

if string is not present and it has not been assigned a default null value, the field will return an empty string instead of a null pointer.

if a table is not present and it has not been assigned a default null value, the field will return an empty table instead of a null pointer. If the table has required fields a warning will be issued and the field will return a null pointer instead of an empty table. Empty tables can still return field default values. On a second thought, this will probably be a future feature with a command line switch, until then null is returned as before.

These semantics allow for future string default values and table default values.

As for structs, if they have a default null value, they will be converted into a union of None, and the struct, but the struct will be stored inline in the table when present and the table fields will remain the same, i.e. no special type field for the struct union.

Scalars will remain their current accessors, but also get a union accessor like structs. If the ordinary field accessor is used instead of the union accessor, a zero value is returned when absent, or a null for structs.

Note that C has a <table>_<field>(_get | _union | _type ) syntax where _union, and _type are only available for unions so far, but could be added for all fields with a null default value. The _get suffix can be omitted unless a special command line option is given to avoid name conflicts. The union returns a struct with the type and value fields.

[mustiikhalil]

@CasperN the swift implementation is completed, I added it for both tables and object-API

[aardappel]

@mikkelfj currently for tables and strings, = null is implied, since it is the only default possible. If and when we add defaults other than that for these types, we can add specifying this default. Right now, that would only be confusing, since a string field with = null would seem to imply that other fields do not have this default, which isn't the case.

[CasperN]

Bump @paulovap, @vglavnyy for Java and C++

[vglavnyy]

@CasperN, I will look this at the week.

[CasperN]

@rw can you do this for go and/or python?

[aardappel]

Support in.. Lobster ;) https://github.com/google/flatbuffers/commit/77f966f89fcf76fa67485adf941ac6e90d28029c

So Lobster has built-in support for "nillable" types, but sadly not for scalars. So the accessors for optional scalars instead return a second boolean to indicate if the value was present. Not great, but workable.

[CasperN]

@vglavnyy, how's the C++ implementation going? If you haven's started / don't have time, I can do it too.

[vglavnyy]

@CasperN I apologize for the delay (I was extremely busy). It would be great if you could do it.

I've examined C++ code generator to make PR.
I don't understand how to Implement it with raw-pointers as proposed @aardappel:

const int32_t* get_maybe_i32() const { ... }

It is impossible because a pointed value is stored as little-endian but a host can be a big-endian.
With C++ we have alternatives:

1. Generate scalar in a struct (boxing) and return a pointer to generated struct (alignment issue on re-cast).
2. Add flatbuffres::optional<T>.
3. Use the std::optional<T> or an user-defined optional (abseil and etc).

I prefer std::optional or a user-defined optional type. This is the easiest and portable way to implement it.

@mikkelfj Do you have a plan to implement support of optional is fltacc?

[aardappel]

It is impossible because a pointed value is stored as little-endian but a host can be a big-endian.

Argh, you're right, that's a shame, I liked the simplicity of this idea.

I still like the idea of a pointer, since that's how optionality is encoded in the rest of the API (for struct/table/string/vector). All of them have the nullptr value.

To keep it the same here, we'd return a pointer to a flatbuffers::Scalar<T> which has a private T and an accessor that can do the byteswap. We could provide an overloaded operator* for easy access.

I like that better than optional for consistency, but optional has the advantage that it will be more familiar to C++ users, and in C++17 we can use the native one.

[vglavnyy]

@aardappel Could you look at #6092 for discussion?

[mikkelfj]

@mikkelfj Do you have a plan to implement support of optional is fltacc?

Yes flatcc needs this feature, but it is not a top priority right now. Others are welcome to step, after discussing how to implement.

[mikkelfj]

How is this represented in the binary bfbs schema?

[mikkelfj]

What happens if a union is made optional?

[aardappel]

@mikkelfj yes, this should probably be added as a bool to Field in https://github.com/google/flatbuffers/blob/master/reflection/reflection.fbs since it's a built-in attribute. @CasperN can you add this?

The current feature is only for scalars, so does not apply to unions. Unions already have optionality built-in thru its NONE value, and do not have a default value.

[mikkelfj]

The plan for a FlatCC implementation for optionals are as follows:

All scalar table fields marked as optional will behave exactly as before, except they will get 0 as the default value when absent, and they will get a new accessor method with the _option() suffix, instead of _get(), which returns a struct field of two values <is_null, value>, where is_null is the negated is_present() method, and the value is the same as the get() method (thus 0 if is_null is true).

In this way fast and simple access is maintained for ordinary use cases while it is possible to efficiently retrieve an optional type in a struct return value.

The builder removes the force_add method from optional fields because force_add is always in effect. There is currently no support for writing an option struct: either the value is written or it is not. This is not fully aligned with how unions behave, since you can add a union value as a struct of <type, value> with the type NONE in a single add method call, but it could be added later if needed.

JSON parsing and printing have global flags to force default values to be printed or stored. These flags will be ignored for optional fields so an absent field in text results in a absent field in a buffer and vice versa with no default value processing.

FlatCC JSON does not parse or print null values. Originally I thought this was needed for optional values, I then realised that it is a separate issue and the null could be equally support for other field types. Which is a can of worms, and have a separate branch with a new JSON parser design, so I didn't want to complicate things further before that gets merged.

[mikkelfj]

OK, I have added a branch named optional to flatcc with support for optional scalars, it is ready to be merged and is only missing bfbs generation, pending reflection schema update. EDIT: I also added the nullable attribute to the bfbs schema.

https://github.com/dvidelabs/flatcc/tree/optional

There is one tricky special case: enums that do not have a 0 element because these require an initialiser among the enum elements, which of course is not possible with null. This is not a problem if you do not need the default value, but since FlatCC maintains the numeric _get() accessor along with _option() and becaue _option().value has a defined meaning even when _option().is_null is true, flatcc opts to define the default numeric value to 0 for optional enums that are missing a zero element.

I added some extra OptionalFactor enum fields to the test case:

https://github.com/dvidelabs/flatcc/blob/optional/test/optional_scalars_test/optional_scalars_test.fbs
https://github.com/dvidelabs/flatcc/blob/optional/test/optional_scalars_test/optional_scalars_test.c

[mikkelfj]

Note that the my above discussion on default values for nullable enums also apply to the binary schema export where the field needs a numerical default value, which I set to 0 for enums even when there is no 0 element, when the default is null.

[aardappel]

I'd say that if is_null is true, then value should really never be accessed. It is good to define it as always being 0 for consistency (I did the same in the Lobster implementation, and we should do it for all languages where the value of an optional is readable even when not present).

But it shouldn't affect things like enums, the fact that the enum is "out of range" is not important for a value that you're not suppose to use.

There may be cases where a user blindly wants to use value without checking is_null, and it being 0 is useful to them, but I am struggling to come up with examples.

[mikkelfj]

Now that reflection.fbs has been updated, flatcc has merged optional scalar feature to master.

[jamescourtney]

What is the intended behavior when using optional scalars as a key for a sorted vector? Has this been considered? I realize scalars are not frequently used with sorted vectors, but it did come to my mind.

[vglavnyy]

@jamescourtney I think that scalar key and optional should be mutually exclusive features.
At least until the known issues with key aren't solved (#6099, #5928).

[jamescourtney]

@vglavnyy -- that was my take on the issue as well, and it sounds reasonable enough. That behavior needs to be codified in the compiler though.

[mikkelfj]

Making keys mutually exclusive with optional could make it difficult to convert a schema to optional later, and it make it difficult to represent certain database models. Instead it could be defined that null keys sort as 0.

[mzaks]

What about just absence of value in the "map"
So say we have an array like this:

[
{
  "name": "A",
  "timeOfBirth": 123,
  "timeOfDeath": 345,
},
{
  "name": "B",
  "timeOfBirth": 120,
  "timeOfDeath": null,
},
{
  "name": "C",
  "timeOfBirth": 140,
  "timeOfDeath": 250,
}
]

When key is timeOfBirth the map / sorted array looks like this:

[
{
  "name": "B",
  "timeOfBirth": 120,
  "timeOfDeath": null,
},
{
  "name": "A",
  "timeOfBirth": 123,
  "timeOfDeath": 345,
},
{
  "name": "C",
  "timeOfBirth": 140,
  "timeOfDeath": 250,
}
]

when key is timeOfDeath it's:

[
{
  "name": "C",
  "timeOfBirth": 140,
  "timeOfDeath": 250,
},
{
  "name": "A",
  "timeOfBirth": 123,
  "timeOfDeath": 345,
}
]

[mikkelfj]

That is also an option. It would be a lot more work to implement in flatcc, and overlaps with a filtering concept that we do not currently have.

[mzaks]

It would be a lot more work to implement in flatcc, and overlaps with a filtering concept that we do not currently have.

Do I understand it correctly that the first part of the sentence is a negative and the second part a positive? 😀

[mikkelfj]

More neutral. There is the risk that we add some logic that would not fit a more general filtering work, and concretely for flatcc, if there need to be filtering complexity, it better be in a given framework.

The flatcc implementation is a LISP like nest of macros that ultimately rely on accessors doing their job. Making this work with optionals is fairly difficult. However, if a null keys is always filterered, it suffice to check for is_present() on a field. Bu then you would filter out strings thet are null as well, which is probably a good thing - maybe it crashes today?

[mzaks]

It depends how null is handled during comparison. It can be handled as smallest value, the sort doesn't have to be stable (I guess) and also there is anyways an issue if equal keys are present. OR does the API return a list/iterator of values back?

[mikkelfj]

dlatcc accessors are functions taking a table argument. For scalars flatcc returns 0 on null, for strings it returns null on null. Optional scalars also have an option accessor which is a function returning a struct with a value and a is_null field. The heavily parameterized sort logic cannot trivially switch over to selectively use a different access method for optionals. Nor can it trivially filter out data since it performs an inplace heap sort on the existing array, although it could in principle reduce the vector size of the result, leaving garbage at tail.

By treating scalar null as 0, flatcc works out of the box. But as I said, I'm not sure that strings work equally well. However, since strings are handled somewhat specially in the logic, it would be possible to test for null and treat as the smallest string, if that doesn't happen already.

[mikkelfj]

Note that flatcc sorts in-place in an existing buffer.

[CasperN]

What is the intended behavior when using optional scalars as a key for a sorted vector? Has this been considered? I realize scalars are not frequently used with sorted vectors, but it did come to my mind.

Strings are already optional and can be used as keys, so we should share the same behavior.
It looks like when annotated with (key), the string becomes a required field, hence key and optional are mutually exclusive features

[vglavnyy]

Making keys mutually exclusive with optional could make it difficult to convert a schema to optional later,

At least until the known issues with key aren't solved (#6099, #5928).

That was about C++ implementation.

By treating scalar null as 0, flatcc works out of the box.

Optional scalars don't have defaults, so there are no reasons to use zero as a default value.
In my mind, a non-presented optional key should have the lowest sorting order even less than the most negative value of that scalar type.

[jamescourtney]

Strings are already optional and can be used as keys, so we should share the same behavior.
It looks like when annotated with (key), the string becomes a required field, hence key and optional are mutually exclusive features

Agree with this. The main difference between this and strings, which are implicitly optional, is that this can be caught at compile time. In reality, this is probably a corner case inside of a corner case, but it's still worth coming to consensus on whatever behavior is expected.

The subtle difference here between null and a normal default value is that the library can choose to redundantly include a default value in the payload (which my library does for compatibility reasons to address #5833 ), whereas null has no binary expression.

In my mind, a non-presented optional key should have the lowest sorting order even less than the most negative value of that scalar type.

If we do end up allowing these to be keys, then this makes sense to me. Defaulting to 0 or otherwise would create situations where null keys were interspersed with 0 keys, and the sort wouldn't appear to be correct. It would also break binary search.

[mzaks]

AFAIU (key) feature does not work properly when there are duplicate key values, because then it is a __multimap__ and not a __map__. So it does not matter if multiple values are defaulting to the same key or null can be considered as key, the __map__ is _broken_. Or rather it returns one of the values for a key and not all of the values. And this is IMHO a bigger issue. Or rather we hit this issue with all solutions except for the one where we filter out the null values. But then we should also filter out the default values. So I think wes should address the __map__ / __multimap__ issue first.

[mikkelfj]

FWIW: flatcc uses heap-sort in-place in an existing buffer so the sort is not stable if there are duplicate key values.

[jamescourtney]

AFAIU (key) feature does not work properly when there are duplicate key values, because then it is a multimap and not a map.

Are you referring to flexbuffer maps? I don't think FlatBuffers ever specifies that keys need be unique; key is just a hint saying "sort by this" -- if the user wants to treat it as a map without duplicates they are of course free to do so.

My point is that null and 0 are logically distinct values, so coalescing null into 0 for sorting purposes means that a binary search for null may return the element with 0 as the key and vice versa. We may as well not allow optionals to be keys in this case.

[mzaks]

What I mean is following. The API which user uses to extract a value for key, returns one object or null. As explained for example in this documentation:

... you can use the ByKey accessor to access elements of the vector, e.g.: monster.TestarrayoftablesByKey("Frodo") in C#, which returns an object of the corresponding table type, or null if not found...

https://github.com/google/flatbuffers/blob/6cea45dcd3ad84c94c0062d0211736bba53188f2/docs/source/CsharpUsage.md#storing-dictionaries-in-a-flatbuffer

So if you have duplicated keys in a sorted vector eg: [5, 6, 7, 7, 7, 7, 7, 9, 11, 12, 15]. Binary search will return the first 7 it will find. And as a user you will think that there is only one value with a key 7. Which is wrong. This problem is present in FlatBuffers (key) feature and in FlexBuffers. IMHO the issue in FlatBuffers is bigger because the (key) property was not necessarily designed to be a unique value, where a key in FlexBuffers Map implies that it is a unique key. E.g. you can have the similar problem with JSON {"name": "Maxim", "age": 39, "name": Alex} Which name will be taken "Maxim" or "Alex" is AFAIK undefined behaviour.

My point is that null and 0 are logically distinct values, so coalescing null into 0 for sorting purposes means that a binary search for null may return the element with 0 as the key and vice versa. We may as well not allow optionals to be keys in this case.

@jamescourtney I agree with you on the first part, but in the grand picture IMHO it does not matter if we collapse multiple keys into one value with null because we are doing it already with default and generally in case of duplicated keys. 0 and null should be distinguished values, but if you define something as _nullable_ there will be many values behind this one key and users will get unexpected results. So I think it is important to give a user a possibility to easily iterate over all values behind one key. And if we have it, then if 0 and null are the same key, users can filter out values they do not want.

[aardappel]

Seems to me if you're trying to sort on a scalar key, you do not want that value to be optional. I'd suggest to make the combination an error in flatc rather than trying to give null a place in the sorting order.

[vglavnyy]

@CasperN
Can you look at this problem with optional?
https://github.com/google/flatbuffers/blob/96d5e359773987314c25fa5181adb77c064ac625/tests/optional_scalars.fbs#L51-L52

[CasperN]

@vglavnyy so the problem with optional enums also happens with other code generators. I didn't implement support for Rust early on and so no one else did either. I think it makes sense to support them, so I'll try and fix what I can.

[vglavnyy]

@CasperN
I've just fixed the C++ code generator: #6155. Now it should work with Optional<Enum>.

[aardappel]

@CasperN the Lobster changes are here: https://github.com/google/flatbuffers/commit/5d3cf440e50186bf6a1e841038ac887c2da06141

Once you remove optional_scalars2.fbs, please change the 2 at the top of tests/lobstertest.lobster, and delete tests/optional_scalars2_generated.lobster

[CasperN]

@paulovap, don't forget optional enum support for Kotlin! the schema is optional_scalars2.fbs

[paulovap]

@paulovap, don't forget optional enum support for Kotlin! the schema is optional_scalars2.fbs

I will take a look this week.

[mustiikhalil]

@CasperN #6160 adds enum support to swift, since swift already supported optional bools

[aardappel]

For reference, apparently Protobuf just had a 5 year debate on wether to allow optional scalars (or similar to), which they only just decided they will support: https://github.com/protocolbuffers/protobuf/issues/1606

So the fact that we went from @CasperN's suggestion to implementation in just a few months looks pretty good in comparison :)

[erwincoumans]

For reference, apparently Protobuf just had a 5 year debate

More precise: proto2 supported presence of optional scalar just fine. Then proto3 regressed by dropping the feature (requiring ugly workarounds), and some stubborn engineers didn't listen to complaining users for 5 years.

[aardappel]

@erwincoumans that's kinda similar here.. scalars in FlatBuffers always had a default value (even if not specified it was 0), and thus you couldn't reliably tell if they were "present" or not.. and I was the stubborn maintainer ignoring the compaints :) But @CasperN's design changes that on a per field basis, so now everyone is happy.

[CasperN]

Ok so what should we do to close this issue? It seems we have C++, Rust, Lobster, Kotlin (minus enums), C, and Swift. I propose we accept the remaining languages are not under active development.

Remaining tasks:

• Get Kotlin to support optional enums and drop the 2 from optional_scalars2.fbs
• I can update
  
  
  
  • the writing a schema docs.
  
  
  • the language support comparison docs
  
  

Anything else?

[krojew]

I propose we accept the remaining languages are not under active development.

I don't know if I'm understanding this correctly, but I'm quite sure other languages are still under active support.

[CasperN]

I propose we accept the remaining languages are not under active development.

I don't know if I'm understanding this correctly, but I'm quite sure other languages are still under active support.

I meant for optional scalars, specifically, not in general

[paulovap]

Remaining tasks:

• Get Kotlin to support optional enums and drop the 2 from optional_scalars2.fbs

Hey, got some time to look into that:
https://github.com/google/flatbuffers/pull/6201

[krojew]

I guess it's time to look into TS.

[aardappel]

Ok, lets get Java/JS/TS in, docs updated, any schema v2 references removed, then close this issue.. other languages can join whenever.

[adsharma]

flatc.py is an alternative flatbuffer compiler that uses flatbuffer syntax as an IDL, but not the serialization.

I just pushed support for a similar feature, but instead of using

field = null

It uses

field: type (required);

https://github.com/adsharma/flattools/commit/ba84f17a57e0037a9b27baa2d3fb9a52aa47c139

The implementation supports rust, swift and kotlin

[CasperN]

@adsharma

field: type (required);

Note that this differs from the canonical schema language where required is a special attribute that treats missing data as an error.

[adsharma]

@adsharma

field: type (required);

Note that this differs from the canonical schema language where required is a special attribute that treats missing data as an error.

@CasperN flatc.py is also using it with the same semantics. In the commit that I pushed:

https://github.com/adsharma/flattools/commit/feaa38c493e23568e919c75f8f1bb7c903a0cb85

you can not create a Person object without the required field name.

I suspect when you add flatbuffer serialization into the mix, the primitive fields and non primitive table data end up having different semantics (I may not be understanding all the nuances). This isn't a problem for flatc.py

[CasperN]

@adsharma

There are 3 ways to respond when reading a table field that's not present in the binary data

• Default: Return some default value. This is only supported for scalars, but I want to be able to set defaults for strings and vectors in #6053.
• Optional: Return some indication of non-presense, typically with the language's favorite optional type. Before this feature, this was not available for scalars.
• Required: crash and burn. This is specified with the (required) attribute, which to disagrees with your use of (required), and is documented in https://google.github.io/flatbuffers/md__schemas.html.

[leventov]

Will array fields support optional? How the grammar will look like?

[adsharma]

In the flatc.py case, there is no binary data. We use flatbuffers IDL, but
not the serialization code.

Many languages (rust, kotlin) have their own serialization frameworks. They
can choose to implement flatbuffers or another serialization scheme using
that framework. Its typically a macro/decorator.

Your explanation does make sense and helped me understand this proposal
better. Thanks.

On Tue, Oct 27, 2020, 8:54 AM Casper [e-mail] wrote:

@adsharma https://github.com/adsharma

There are 3 ways to respond when reading a table field that's not present
in the binary data

• Default: Return some default value. This is only supported for
  scalars, but I want to be able to set defaults for strings and vectors in
  #6053 https://github.com/google/flatbuffers/issues/6053.
• Optional: Return some indication of non-presense, typically with the
  language's favorite optional type. Before this feature, this was not
  available for scalars.
• Required: crash and burn. This is specified with the (required)
  attribute, which to disagrees with your use of (required), and is
  documented in https://google.github.io/flatbuffers/md__schemas.html.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/flatbuffers/issues/6014#issuecomment-717340911,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFA2AYWZDP2B2E5IR3ILTLSM3ULJANCNFSM4OLQQVUQ
.

[mikkelfj]

@adsharma
I also use the flatcc (for C) compiler as an IDL for other purposes. However, I think it is a bad idea to introduce different semantics. If you have special needs you can use attributes to specify these.

[adsharma]

To me it appears that flatcc is another implementation of flatc in a
different language. There may be implementation differences, but the
overall concept remains the same.

My claim is that "required" is used with the same semantics. Not providing
that attribute is an error (crash and burn).

Except that we do this at object construction time since no deserialization
is involved.

On Tue, Oct 27, 2020, 10:57 PM MikkelFJ [e-mail] wrote:

@adsharma https://github.com/adsharma
I also use the flatcc (for C) compiler as an IDL for other purposes.
However, I think it is a bad idea to introduce different semantics. If you
have special needs you can use attributes to specify these.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/flatbuffers/issues/6014#issuecomment-717717059,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFA2A7JSGCY43LSNDJMCH3SM6XF5ANCNFSM4OLQQVUQ
.

[mikkelfj]

flatcc generates flatbuffer code in C. It is also a schema compiler library that can be used for other purposes. I use it for other purposes in addition to processing flatbuffer files.

[CasperN]

@adsharma

Your original comment seems to suggest you're using (required) to specify optional scalars

I just pushed support for a similar feature, but instead of using
field = null
It uses
field: type (required);

[CasperN]

@leventov

Will array fields support optional? How the grammar will look like?

No, we are only focusing on table fields in this issue.

[aardappel]

@adsharma you are free to do as you please of course, but it would be nice for your users to alert them to the fact that your tool uses an incompatible dialect of FlatBuffer schemas, and how so, if you feel strongly that you want to change what things mean to flatc.

[adsharma]

@aardappel - I don't want to gratuitously do something different, but here's my thinking on the matter.

The way we're using flatc.py, we're competing with:

• SQL DDL as a way to describe data. I often advocate for flatbuffers to describe things at a conceptual level, whereas the SQL DDL tends to describe data at a physical level (although views could be used for conceptual data, but for various reasons they don't work as well as flatbuffers).

• Describe common programming language constructs in a language neutral way

If I push required for non-scalars and = null syntax for scalars in these circumstances, my users are likely to say - no thank you, I'll describe data in SQL or $favorite_type_safe_language. I must also admit that I'm sympathetic to removing null as a concept and avoid the billion dollar mistake.

In most programming languages (rust, kotlin and swift among others), we have Optional[T] that works for both scalars and non-scalars.

[aardappel]

To me, people rejecting your solution because of such a small feature would be unlikely, and I think generally that a tool adheres to (and is compatible with) a well known standard is much more important.

Also, "the billion dollar mistake" is about implicit nulls, and the new use here is explicit (requires checking to access, at least in language that support it, like C++, Rust and C#). We already had nulls before, and any implicitness is due to the language they're implemented in, not FlatBuffers.

[mikkelfj]

I'm not entirely comfortable with the way required and optional differ for different value types, so I understand @adsharmas concern. I still think schema compatibility is more important. However, if we improve FlatBuffers to more suitable as a DDL I'm all for it. That is part of my lengthy arguments in the past. But I fear that ship has sailed.

[adsharma]

@mikkelfj @aardappel - I can look into supporting the = null syntax in addition (and treat it the same as required). So we remain compatible. Does that address your concern?

[Triazic]

Voting/requesting support in C# and typescript :)

[CasperN]

Voting/requesting support in C# and typescript :)

Oh we did that! I just forgot to update the top comment (doing that now).
With the documentation updates, I'll close this issue