Docs: Star ratings for UD treebanks

[ryszardtuora]

Hello,

I was wondering what was the methodology behind assigning each treebank some rating in stars (for example 0.824279 for Italian ISDT). Previously I've learned that the quality of treebanks varies quite a lot, and I would like to have some way of automatically filtering these bad treebanks out for the purposes of my experiment. Can I calculate these ratings myself somehow? Also, are they available somewhere for the API, I couldnt find any place in the docs, nor in the treebank files where ratings themselves would be listed.

[akoehn]

It comes from this script:
https://github.com/UniversalDependencies/tools/blob/master/evaluate_treebank.pl

My advice would be to actually look at the treebank documentations because you might care more for some parts of the quality estimation than for others.

[ryszardtuora]

Thank you!

[qipeng]

Quick follow-up question on this thread: is there maybe a bug in the rendering of stars on the UD webpage right now? For instance, for Chinese-GSD and Chinese-GSDSimp the script is reporting 3.5 (with scores around 0.7), but the website is showing no stars (with scores very close to 0).

[arademaker]

Hi @dan-zeman, I haven't noted this starts before. It would be useful to have a overview of the rules for evaluation. Although I could try to understand the Perl code, it is far from trivial. Do you have any documentation describing the start rating criteria? The Portuguese treebanks are both with zero starts??

[akoehn]

It could just be that the stars for the newest release are lower than for the dev data.  The website only shows release info.
null

[dan-zeman]

The stars on the UD page (as well as the score that you see when you place your mouse pointer over the stars) is based on the output of the script for the contents of the master branch, i.e., the last official release. The output is archived in a file called eval.log in the master branch (this file is not copied to the dev branch and it is not packed in the official release, but you can view it on Github). For example, here is the evaluation log of Chinese-GSDSimp.

There is no documentation other than the comments in the source code. Essentially, a number of quantifiable parameters is combined with weights (which are somewhat arbitrary, based on reasoning of the type "I think that feature X should be as powerful as Y, but three times as powerful as Z"). However, two things are much more influential than anything else: if the corpus does not contain the underlying text, or if it is not accepted by the validator, the score is multiplied by a coeficient that is close to zero. The idea is that invalid treebanks should always be penalized drastically, although we still want to be able to rank them among themselves. (Now that we have so many legacy treebanks (which are formally invalid), many treebanks appear without stars.)

[arademaker]

thank you, logs are very informative.

[qipeng]

@dan-zeman Thanks for the clarification!

For posterity, I found out where the discrepancy comes from: the official eval runs the validator script, which when I ran the Perl script locally was not taken into consideration.

[Stormur]

I have read the evaluate script and its ouptut, and, while most of it is clear, I don't fully understand the score regarding "Udapi bugs":

does the content follow the guidelines?

That's because it is not clear to me why something (and what exactly?) is counted wrong if, on another note, the treebank passes validation (validate.py). Is there a more detailed reference?

[dan-zeman]

The MarkBugs block in Udapi (see also here and here) was created before the official UD validator included any linguistic tests. Some Udapi tests overlap with tests that were later added to the validator, hence a valid treebank does not contain bugs of that kind.

However, Udapi also reports some instances that the validator does not catch. For some "bug" types, the validator cannot report them because it cannot be 100% sure they are errors, although it is likely that most of them are. For example, the validator now allows up to two subjects per predicate because in certain instances of non-verbal clauses in certain languages, two subjects may legitimately end up attached to the same node. However, vast majority of clauses can have at most one subject, and Udapi will report anything higher than 1 as a bug. The few false alarms will not hurt the aggregate score significantly, while it is useful that the rest is reported and can be acted upon.

[Stormur]

Great, thanks! It's a pity that I overlooked this, but now it will be a great resource for error-busting.