Bids-specification: [ENH] Allow for - (dash) (and/or +) in <label>

[yarikoptic]

I thought that we already had an issue filed for this, but may be it was just my comments in the google doc version of the BIDS specification, so filing anew. Please close it referencing original one if one already exists.

ATM <label> is as I formalized in yet to be finished PR is allowed to contain only alpha numerics. That makes it really difficult in some cases to place multiple entries (e.g. details for _acq- or abbreviate details of preprocessing in _desc- of BIDS derivatives). In ReproIn heuristic for heudiconv, where we do allow for - we just replace them with an ugly X as a separator to just stay compatible with BIDS.

I have been suggesting to extend the list of allowed characters in <label> to include at least - and possibly others (e.g., +) which otherwise should not introduce any backward incompatibility with BIDS 1.0 -- all BIDS 1.0 names would be compatible with new syntax allowing - in the <labels>.

The original concern (probably at least a year back) which precluded any action was a possible breakage of parsing BIDS filenames for BIDS-supporting software, and was suggested to be included in coming some time in the future BIDS 2.0.

N.B. I have failed to file and issue outlining plans for BIDS 2.0, so I initiated a new milestone (2.0.0) to which I am adding this new issue.

IMHO (and may be we should outline it somewhere) BIDS supporting applications should be coded adhering to Postel's principle which is driving TCP implementations to "be conservative in what you do, be liberal in what you accept from others.". And most likely, in the case of added -, they already are since they are more likely to be splitting based on _ and then on - to separate key from value instead of full rigid regular expressions limiting to the allowed characters (not so obviously) mandated by the BIDS specification. So the only catch here indeed might be splitting on - without restricting that only a single split should happen. That is where breakage could happen if - is added. The possibility of breakage in case of extending the list of allowed characters with + (or even : which I dislike to suggest due to messing up scp/ssh) unlikely to have negative ramifications.

Please share your thoughts/opinions

[effigies]

I'm personally okay with - or +, and can't really remember the arguments against them.

[oesteban]

I'm okay with +, and not very fond of - but I understand it could be added. I would not allow - to be present in the suffix entity, though.

[yarikoptic]

I would not allow - to be present in the suffix entity, though.

Great point I haven't thought to exercise @oesteban ! -- I would not allow it as well, makes parsing much more difficult etc!
The only spot I see where <label> is used not within _key-<label> is

(git)hopa:~/proj/bids/bids-specification[derivatives]git
$> git grep '_<label'
src/04-modality-specific-files/04-intracranial-electroencephalography.md:called `electrical_stimulation_<label>`, with labels chosen by the researcher and

so might need analysis/tune up (or not)

[yarikoptic]

ok, after https://github.com/bids-standard/bids-specification/pull/152 is merged in one way or another we could have a PR extending the list of allowed characters non-ambiguously in a single location ;-)

[nicholst]

Note that the request to include hyphens in key-value labels is currently in the "Suggestions for BIDS 2.0" Google doc:

Allowing hyphens in filename key values

which suggests, as I had recalled, that this was a settled issue. Including hyphens increases the complexity of parsing the filenames, was the reason I recall.

[nbeliy]

@nicholst, will '-' really increase complexity of parsing?

Each key-value parsed by '_' and everything after the first '-' is a label.

[yarikoptic]

@nicholst Oh, that is where BIDS 2.0 "extensions" are discussed -- thank you for the link! I have added the link to the 2.0.0 milestone description. I think it might be worthwhile to move them all under issues over here at some point, so elderly folks like us could easily find everything relevant in the single location (here).

I would not say that parsing complexity would increase, but indeed it might require some adjustment.

[nicholst]

@nbeliy,

@nicholst, will '-' really increase complexity of parsing?

Each key-value parsed by '_' and everything after the first '-' is a label.

Just reciting the conclusion of a long thread about the issue. Like anything, this issue can be re-raised, but I think there is the basic issue of extra clarity from having the hyphen play exactly one role in file/directory names.

[yarikoptic]

my 1c: This issue by now is probably a few years old (IIRC I have suggested it possibly even during BIDS 1.x initial development/release) . If we acted sooner, it would be no issue but even those years back I have proposed it the main argument against was "we would break the tools" of which there were much fewer. So longer we wait to act, the harder would be to adopt this change.

[oesteban]

I think at this point the procedure would be sending a draft PR with the suggested changes for a more focused discussion, is that correct @franklin-feingold @sappelhoff ?

[sappelhoff]

I think at this point the procedure would be sending a draft PR with the suggested changes for a more focused discussion, is that correct @franklin-feingold @sappelhoff ?

Yes, and on top of that I think what this discussion needs are:

1. concrete use cases where the proposed changes would significantly enhance BIDS (summaries/links to such cases)
2. a concrete PR showing the diff of the spec prior and post the proposed changes (what @oesteban said)
3. more participants to voice their opinions - especially developers

Currently this change seems to me as a minor improvement with the potential for a major confusion ... but I am very happy to be convinced otherwise :-)

[nicholst]

Use cases:

Subject or session labels come from long-established project that has
identifiers with dashes.

Related, such a project uses identifiers with underscores (but not dashes),
and the underscores need to be replaced; having dashes as an option for
replacing is helpful.

[robertoostenveld]

Is UTF8 allowed in BIDS file names, or if not, should it be? If so, there are many dashes to choose from which are not a minus (ASCII decimal 45). For example ‒ – and —. See https://www.w3schools.com/charsets/ref_utf_punctuation.asp

[oesteban]

Using alternatives that _look like_ the proposed addition doesn't help much with readability, makes it harder. Imagine a suffix like _inplane‒T2.json (I used your first UTF8 character). _inplane+T2.json breaks the suffix a bit, but is still readable. The limitation of the suffix is also important in this kind of proposal, IMHO.

I would be fine with the + symbol and I think the dash is not worth the pain. Adding the dash would feel to me as useful as allowing spaces (i.e., error-prone and likely confusing).

[yarikoptic]

I think we should explicitly forbid UTF8 and limit to ASCII to avoid such "tricks" which could potentially cause confusion.
So let's move forward with "+"?
I doubt any tool was so strict with regexp that it listed only alpha numerics. and sooner we change, less possible negative effect it would have. This issue already dragged long time (I originally suggested in the times of BIDS being a google doc), and I think it is worth to be accepted to 1.x series without awaiting 2.0 which I am not quite sure would come in any foreseeable future

[robertoostenveld]

Agreed about limiting character sets, I also don’t look forward to multiple types of dashes or emojis in filenames. But the limit should be explicitly mentioned.

Regarding the “+” I am neutral.

Verstuurd vanaf mijn iPhone

Op 2 nov. 2019 om 00:57 heeft Yaroslav Halchenko [e-mail] het volgende geschreven:


I think we should explicitly forbid UTF8 and limit to ASCII to avoid such "tricks" which could potentially cause confusion.
So let's move forward with "+"?
I doubt any tool was so strict with regexp that it listed only alpha numerics. and sooner we change, less possible negative effect it would have. This issue already dragged long time (I originally suggested in the times of BIDS being a google doc), and I think it is worth to be accepted to 1.x series without awaiting 2.0 which I am not quite sure would come in any foreseeable future

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

[yarikoptic]

ok, it seems that the path of the least resistance would be to add + as allowed character. I will look into proposing a PR here and against bids-validator for that (unless someone would beat me to it which I would really appreciate)

[yarikoptic]

oh hoh, it had been a year! Few final remarks and then I will try to find time to compose a PR.

• + implies composition of multiple things, like _proc-mc+filt -- thing was motion corrected and filtered; whenever - is more of a pure delimiter between multiple words which need to be taken together
• re _inplane‒T2. example -- "illegit" since - must not be allowed in the suffix, which it must not look like _key-value. This issue only about <label> as used in the _key-<label>, and we never use _<label> as arbitrary suffix (git grep -e '<label>\.' -e '_<label>' comes with no relevant hits)
• comparison of - to spaces is IMHO also a bit too far stretched. - does not break visual composition of the filename etc.
• arguments against - because there are UTF8 characters for longer dashes are not really pertinent. Pretty much any character could have some confusingly similar symbol in UTF8, so we just must not allow utf8 in general.

Since I and others mentioned a number of use cases, and IMHO it would be easier to strip away than to add later, I will prepare a PR which would introduce allowing both - and + and then we will see how it goes.