Docs: format and tools support

[arademaker]

Does any tool listed in https://universaldependencies.org/tools.html has the plan to support the CoNLL-U Plus format documented in https://universaldependencies.org/ext-format.html ?

[jheinecke]

It is planned for ConlluEditor, but not yet done... More urgent are problably columns > 10 (which are accepted and displayed by ConlluEditor, but cannot yet be modified

[jheinecke]

I've been looking during the last view days on how to add processing of the CoNLL-U Plus format, but before implementing it. I have a few questions:

1. the documentation says that the ID column can be absent in CoNLL-U Plus, even though it does not make much sense (and if it is absent the HEAD and DEPS columns become useless as well. But for an implementation this changes everything, so it would be easier if CoNLL-U Plus would make the ID column as a first column obligatory.
2. is the order of the standard columns (if present in CoNLL-U Plus) as in CoNLL-U or can it vary (in function of # global.columns ?
3. Would it be easier to have a format which always keeps the first 10 CoNLL-U colums and allows to define additional columns (so # global.columns becomes # additional.columns. Empty columns (filled with _) won't take so much disk space, but testing that tools process correctly a varying number of columns (with different semantics and content types (number/strings/key=value...) is additional work. It is always possible with tools like cut, gawk and join, to delete, add and reorder columns, so existing CoNLL-U processing software (including parsers etc) would only have to read 10 columns + eventual additional columns
   What is your opinion on that?

[dan-zeman]

Ad 1) At present I don't see a use case where one would want the ID column omitted. HEAD and DEPREL obviously can be omitted, and if they are not, then you probably do not want to omit ID becuase it serves to interpret the HEAD. If the ID column is present, there is no need for it to be the first column, as the column headers now must be present (although I think I would personally always want it as the first column). If the ID column is not present then I would probably generate it internally upon reading the file.

Ad 2) See above. The order can vary. From the perspective of the add-on project, the original columns may be much less important and people may actually want to shift some of them further right.

Ad 3) Yes, it would be easier. It would be less flexible and would not allow to have columns that matter first, e.g. when people read the file. But feel free to implement only partial support for a restricted subset of CoNLL-U Plus files that meet additional requirements. Maybe you can then document in the README file how people can reorder columns and add missing columns before they open the file in your tool.

[jheinecke]

I finally wrote a small programme to transform any conllu[plus] format into any other (no plausability checks, missing columns in the input required for the output will be replaced with _). ConlluEditor will accept a subset of conlluplus formats (all 10 standard columns + any additional columns). So it is possible to transform any conlluplus-file into this format and edit the file with ConlluEditor. I will push this to github in the coming days (most of it was already present, but I want to test it bit more)

[jheinecke]

I have just pushed it (ConlluEditor, version 2.4.0). Personally, I use Conllu-Plus columns in the BIO format, but for the time being, the editor does not care. entering an I:Something without a preceding B:Something won't create any warning. I do not know how many people use (and actually edit) additional columns. If there is a need, I will think about raising warnings/errors etc.

[Stormur]

Sorry if the question may be out of place, but I found that this discussion was the only one directly referring to the CoNLL-U Plus format.

From the online UD documentation I am not able to find confirmation for this: does a treebank has to _strictly_ adhere to the .conllu format, or is the .conllup one also valid? Maybe in the version which just adds new columns, instead of scrambling them?

[jnivre]

UD treebanks have to adhere to the CoNLL-U format. The CoNLL-UP format is just a proposal for other projects that want to extend the annotation for their own purposes (for example, annotate multiword expressions).

[Stormur]

OK, thanks for the (obvious) clarification! Is it foreseen to expand the admitted formats to the Plus version somewhere in the future?

[dan-zeman]

Do you mean formats admitted in the official UD releases? No, it is not foreseen.

[jnivre]

Not foreseen and not very likely, but if there is enough public demand for it, I suppose it could be considered.