Docs: Guidelines for train-dev-test data splits

[dan-zeman]

We need better guidelines for data splits. We have previously issued some recommendations, most notably in connection with the CoNLL 2017 shared task, but these were too much focused on the task (and were changed on the fly).

I have created a proposal at http://universaldependencies.org/workgroups/split.html and would like to collect feedback here so that we can provide better guidance in the documentation for the future releases.

[PeterisP]

I would strongly recommend doing any split only on document boundaries, and avoiding sentence shuffling between train/test whenever possible.

The major difference is that if the test set consists of sentences that come from documents seen in training data, then there is an artificially low amount of out-of-vocabulary words, unseen proper names and entity mentions, since a rare term or name in such a test set is very likely to be repeated elsewhere in that document and so appear also in training data while a similarly rare word in new data won't be, lacking the valuable lexical information.

We've seen significant differences in system evaluation scores - if the same data is split on a sentence or paragraph level, then the scores are artificially optimistic, and only a document-level split shows a proper reflection of the expected performance on unseen data.

[amir-zeldes]

I agree and would like to add how important coherent documents are for any discourse-level tasks we might want to use the corpora for (coref, discourse parsing, information extraction...)

[dan-zeman]

Thanks for the feedback. Since there were no new comments for a while, I strenghtened a bit the urge to 1. split on document boundaries and 2. not to shuffle sentences if possible. I have moved the guidelines to the release checklist page (http://universaldependencies.org/release_checklist.html#data-split) and I am closing this issue.

[gossebouma]

I support the idea of doing splits on the document level. For the UD_Dutch-LassySmall (Wikipedia) corpus, the current split is based on random sentence shuffle. I could redo it based on document ids but then the next release would be significantly different from the current release, with material from one (current) split possibly ending up in the other (new) split. Is this advisable? (Even if we warn users that results from older releases cannot be compared to this one?)

Second, the current guidelines also suggest that the splits should reflect genre's/topics proportionally. In UD_Dutch, this is not the case, with test being exclusively from one genre. I could also redo the split for this corpus, but with the same drawback as mentioned above.

So should I plan to redo the splits for the Dutch corpora, or rather go for stability across releases?

[dan-zeman]

Since this would be a one-time, well motivated change, and the new split would be arguably better, I would vote for making an exception from the stability across releases rule. @jnivre, what do you think?

[jnivre]

It is clearly a case of two sound principles coming into conflict, and I therefore have no strong views either way. But I agree with Dan that an exception may be motivated.