Bids-specification: Inheritance principle: unsetting of key-value pairs

[Lestropie]

Came across while reviewing #488, but listing separately since the discussion is not specific to the changes in that PR, and indeed isn't even specific to derivatives.

From "Common Principles", "The Inheritance Principle":

There is no notion of "unsetting" a key/value pair. For example if there is a JSON file corresponding to particular participant/run defining a key/value and there is a JSON file on the root level of the dataset that does not define this key/value it will not be "unset" for all subjects/runs.

This might require elaboration, as I would suggest that in the other direction, unsetting a key/value pair is possible.

The situation is probably best understood as a sequential loading of JSON data from the root directory up to the file currently being processed. If a key is absent in a root JSON, but present in a sub-directory JSON, then that key/value pair gets loaded into the JSON object regardless of the fact that the key was not present in the root JSON. This seems to be what the current text is describing, although it is seemingly describing the effect from the perspective of the opposite direction, i.e. the absence of that key/value in the root JSON does not preclude it from being loaded when the sub-directory JSON is read.

Where the current text may be contradicted is the case where a key/value pair is present in both JSONs, but the sub-directory JSON assigns to that key the value null. This could arguably be interpreted as "unsetting" that key-value pair.

I'm happy to have a go at rewording, but I'd prefer to make sure that there's some agreement with my thinking on this before proceeding with such.

[sappelhoff]

I always interpreted this part of the inheritance principle as such that:

• if a JSON is defined at a shallow level (closer to root) with some key some_key
• then this key some_key will be carried over to a JSON at a deeper level (further into the nested directory structure) ... if it is not present there

--> That is, the absence of a key at a deeper level does not delete (unset) values for that key defined at a shallower level.

From reading your issue, I think we agree on this.

Where the current text may be contradicted is the case where a key/value pair is present in both JSONs, but the sub-directory JSON assigns to that key the value null. This could arguably be interpreted as "unsetting" that key-value pair.

With null you mean n/a ?

Generally I think you are right.

We could possibly improve the distinction of overwriting a value with any other value at a deeper level (including the n/a value) from the notion of a key-value pair simply disappearing (which never happens and is currently referred to as "unsetting")

[Lestropie]

With null you mean n/a ?

Struggling to find a good reference, but it appears that for JSON the recommendation is to return an empty array / dictionary if that would be the type of the data if filled, or null for scalars. So null in JSON appears to provide similar functionality to None in Python, as far as I can tell.

[sappelhoff]

So null in JSON appears to provide similar functionality to None in Python, as far as I can tell.

I see, ... but I think empty arrays or these "Null" values that are not n/a should be caught be the validator. That is, we don't allow setting values to "Null" (then again, n/a is always valid)

[nicholst]

I've read and re-read the text from the spec and I agree with @Lestropie that it is particularly unclear.

For example if there is a JSON file corresponding to particular participant/run defining a key/value..

OK... so a key/value is defined deep in the hierarchy somewhere...

and there is a JSON file on the root level of the dataset that does not define this key/value it will not be "unset" for all subjects/runs.

Huh? I just can't figure out what "not unset" means. Let's be concrete, in this tree

  rawdata/
    T1w.json
    sub-01/
        anat/sub-01_T1w.nii.gz
        anat/sub-01_T1w.json
    sub-02/
        anat/sub-01_T1w.nii.gz

If (say) EchoTime is set in sub-01/sub-01_T1w.json but not in T1w.json; what does "not unset" mean for EchoTime for sub-02? I can't parse it.

(This is all independent of allowing Null vs. n/a; all I can see in the spec is that "Missing values MUST be indicated with n/a." If we make a change allowing things to be "unset" then Null would seem to be a natural choice in the JSON.)

[satra]

The situation is probably best understood as a sequential loading of JSON data from the root directory up to the file currently being processed. If a key is absent in a root JSON, but present in a sub-directory JSON, then that key/value pair gets loaded into the JSON object regardless of the fact that the key was not present in the root JSON. This seems to be what the current text is describing, although it is seemingly describing the effect from the perspective of the opposite direction, i.e. the absence of that key/value in the root JSON does not preclude it from being loaded when the sub-directory JSON is read.

completely agree with @Lestropie

The original design of the inheritance principle was meant to be simple: Allow common things to be pulled up to a higher level. This simply was meant to say that there is this sequential reading of metadata. Keep adding/replacing key-value pairs as you go into the depths of the hierarchy. There is no concept of unsetting.

(as a slight aside) having seen this in action over the years, if there was one thing i would likely remove from bids if i had a time machine is the inheritance principle. i'm at peace with replication of key:value info. and it would make parsing a bids tree significantly easier. you would know the relevant non-subject-related metadata associated with a file just from the file path + file.json

[nicholst]

So it seems we agree that the text is poorly worded. What about this fix:

There is no notion of "unsetting" a key/value pair. Once a key/value pair is set in a given level in the dataset, lower down in the hierarchy that key/value pair can only be set to a different value, possibly n/a.

However, I don't see the need for this restriction... I dimly recall there was some problem with representing null value in JSON, but that doesn't seem to be the case.

So, the much more consequential edit would be a revision of the spec:

A key/value pair can be "unset" at a given level in the dataset with a `null' value, and then lower down in the hierarchy that key/value will be regarded as having been not assigned.

[satra]

A key/value pair can be "unset" at a given level in the dataset with a `null' value, and then lower down in the hierarchy that key/value will be regarded as having been not assigned.

i am not in favor of introducing the notion of unsetting at all. nothing in the original spec suggests that this should be allowed. we currently only allow change of value and addition of keys.

yes technically one can introduce null as the value, especially since BIDS does not say anything about restrictions of values of most keys in the json. but do we wan't to go there is the question. what use cases do we support by allowing this?

[nicholst]

I'm honestly don't know of any use cases for unsetting, but I also don't understand the complications induced by allowing unsetting.

ANYWAY, I guess we hew to the conservative course and leave things as is (no unsetting).

The question then is clarifying this passage. Do we want to recommend some consistent way of flagging 'unsetting' by assigning a default value, or remain mute on the subject? I.e. option 1 or 2 below:

There is no notion of "unsetting" a key/value pair. Once a key/value pair is set in a given level in the dataset, lower down in the hierarchy that key/value pair will always have some assigned value; if needed, null or n/a can be used to indicate an undermined value.

or

There is no notion of "unsetting" a key/value pair. Once a key/value pair is set in a given level in the dataset, lower down in the hierarchy that key/value pair will always have some assigned value.

[satra]

my preference:

There is no notion of "unsetting" a key/value pair. Once a key/value pair is set in a given level in the dataset, lower down in the hierarchy that key/value pair will always have some assigned value.

[nicholst]

Easy enough to fix... #495