Datasets: [XGLUE] Adding new dataset

[patrickvonplaten]

XGLUE is a multilingual GLUE like dataset propesed in this paper.

I'm planning on adding the dataset to the library myself in a couple of weeks.
Also tagging @JetRunner @qiweizhen in case I need some guidance

[JetRunner]

Amazing!

[patrickvonplaten]

Small poll @thomwolf @yjernite @lhoestq @JetRunner @qiweizhen .

As stated in the XGLUE paper: https://arxiv.org/pdf/2004.01401.pdf , for each of the 11 down-stream tasks training data is only available in English, whereas development and test data is available in multiple different language cf. here:

So, I'd suggest to have exactly 11 "language-independent" configs: "ner", "pos", ... and give the sample in each dataset in the config a "language" label being one of "ar", "bg", .... => To me this makes more sense than making languaga specific config, e.g. "ner-de", ...especially because training data is only available in English. Do you guys agree?

[yjernite]

In this case we should have named splits, so config ner has splits train, validation, test-en, test-ar, test-bg, etc...

This is more in the spirit of the task afaiu, and will avoid making users do the filtering step themselves when testing different models or different configurations of the same model.

[patrickvonplaten]

I see your point!

I think this would be quite feasible to do and makes sense to me as well! In the paper results are reported per language, so it seems more natural to do it this way.

Good for me @yjernite ! What do the others think? @lhoestq

[JetRunner]

I agree with Yacine on this!

[patrickvonplaten]

Okey actually not that easy to add things like test-de to datasets => this would be the first dataset to have this.
See: https://github.com/huggingface/datasets/pull/802

[lhoestq]

IMO we should have one config per language. That's what we're doing for xnli, xtreme etc.
Having split names that depend on the language seems wrong. We should try to avoid split names that are not train/val/test.
Sorry for late response on this one

[yjernite]

@lhoestq agreed on having one config per language, but we also need to be able to have different split names and people are going to want to use hyphens, so we should at the very least warn them why it's failing :) E.g. for ANLI with different stages of data (currently using underscores) or https://www.tau-nlp.org/commonsenseqa with their train-sanity or dev-sanity splits

[lhoestq]

Yes sure ! Could you open a separate issue for that ?

[stefan-it]

Really cool dataset 👍 btw. does Transformers support all 11 tasks 🤔 would be awesome to have a xglue script (like the "normal" glue one)

[patrickvonplaten]

Just to make sure this is what we want here. If we add one config per language,

this means that this dataset ends up with well over 100 different configs most of which will have the same train split. The train split is always in English. Also, I'm not sure whether it's better for the user to be honest.

I think it could be quite confusing for the user to have

train_dataset = load_dataset("xglue", "ner-de", split="train")

in English even though it's ner-de.

To be honest, I'd prefer:

train_dataset = load_dataset("xglue", "ner", split="train")
test_dataset_de = load_dataset("xglue", "ner", split="test-de")
test_dataset_fr = load_dataset("xglue", "ner", split="test-fr")

here

[lhoestq]

Oh yes right I didn't notice the train set was always in english sorry.
Moreover it seems that the way this dataset is used is to pick a pretrained multilingual model, fine-tune it on the english train set and then evaluate on each test set (one per language).
So to better fit the usual usage of this dataset, I agree that it's better to have one test split per language.

Something like your latest example patrick is fine imo :

train_dataset = load_dataset("xglue", "ner", split="train")
test_dataset_de = load_dataset("xglue", "ner", split="test.de")

I just replace test-de with test.de since - is not allowed for split names (it has to follow the \w+ regex), and usually we specify the language after a point.

[lhoestq]

Closing since XGLUE has been added in #802 , thanks patrick :)