Docs: Incorrect Lemmatization in UD_English Corpora

[bjascob]

I'm attempting to use the various UD_English corpora to test the performance of some open-source lemmatizers but I'm finding the data has a ton of errors in it. I notice this especially with ADJ types. Words like 'longer', 'higher', 'biggest',... are often left in their base form, in the lemma column. Some of the files seem better than others but I think all contain a fair number of errors.

My understanding is that these files were hand-tagged. Is there a rationale for all the lemmatization errors? (ie.. are these actually machine generated). I'm wonder if there's a way I can filter out these errors or if this is just not a good corpus for the task.

If you have any suggestions on a corpus that would be better suited for testing lemmatizer accuracy, please let me know.

[rueter]

Should the lemma of a comparative or superlative adjective form be the positive form or should it be the same as the form in the text?
Only one of the UD_English...test.conllu corpora is consistently placing positive forms of adjectives and adverbs alike in the lemma column.

LinES uses (Pos lemma):
less little, more much, longer long

GUM uses (mixed):
earlier earlier
later late

English: ParTUT uses (Cmp lemma):
higher higher

EWT uses (Cmp lemma):
earlier earlier, smaller smaller, stronger stronger

PUD uses (Cmp lemma):
better better, safer safer, earlier earlier

The Swedish treebanks are not consistent either
Talbanken uses (mixed):
vidare vidare, mindre lite, lägre låg

LinES uses (mixed):
inre inre, mer mycken, längre längre, tidigare tidigt

PUD uses (mixed):
tidigare tidigt, snarare snarare, bättre god, mer mycket

The Finnish treebanks are inconsistent:
TDT uses (almost consisten):
pidempi pitkä, parempaa hyvä, enemmän enemmän

PUD uses (almost consistent):
enemmän enemmän, suurempi suuri, afrikkalaisempaa afrikkalainen

FTB uses (mixed):
kummallisempia kummallinen, liiemmin liiemmin, vaivattomampaa vaivattomampi, parempi parempi

The German
GSD does not use Degree feature, but it uses (consistent):
besser gut, größere groß , höhere hoch

PUD uses the Degree feature, but it uses (mixed):
nützlicher nützlich, beunruhigendere beunruhigender

The Hungarians use (mixed):
komolyabb komoly, utóbbi utóbbi (analogical to ytre/inre, latter), korábban korán

Northern Sami is inconsistent:
viidáseappot viidáseappot, stuorát stuoris, eambbo eambbo, dárkilet dárkil

Estonian
EDT uses comparative forms in the lemma:
uuemal uuem, jämedamaks jämeda=m

Judging from a few languages with comparative morphology both regular and irregular, we can see that this is definitely an issue. For aligning languages with entirely analytic expression of comparison, it would be important that the positive (base) form is used in the lemma.

Observing just a few languages with comparative morphology

[dan-zeman]

I would vote for the positive form as the lemma.

For some treebanks it may be difficult to achieve if the available lemmatizer works differently. But consistent annotation where comparative form serves as lemma is still better than an inconsistent mix.

[jheinecke]

+1

[bjascob]

I was definitely expecting the positive form for English. Note this is an issue for adverbs too, not just adjectives.

Can you explain if the lemma are being machine generated or are the tags edited by a person. What is the process for making updates to the corpra?

[dan-zeman]

An approximate answer to how the lemmas were annotated is available in the metadata in the README file of each treebank, also accessible from the UD title page (if ou click on English, then on the treebank, then on "Treebank hub page"). For instance, for EWT it says:

_Lemmas | assigned by a program, with some manual corrections, but not a full manual verification_

Of course there are numerous possibilities what precisely it means, how detailed the verification was etc.

[amir-zeldes]

Discussions of lemmatization guidelines for English are surprisingly sparse online, and I can only answer for GUM, but the situation you see in that corpus is not really 'mixed', so much as POS tag dependent. Barring possible errors, it goes back to some of the CLAWS/BNC conventions. For adjectives, comparatives are lemmatized to the positive form:

http://match.grew.fr/[email protected]&custom=5cd040f6a52c0

For ADV, the comparative form itself is taken as the lemma:

http://match.grew.fr/[email protected]&custom=5cd0413228568

I originally thought this practice in the BNC was motivated by a linguistic idea that adverbs are generally derived from adjectives in English (quick > quickly), so the thinking was that morphological comparative adverbs are derivations (not inflections) from comparative adjectives (ADJ longer > ADV longer). This also makes it simpler to deal with suppletive or deponent primary comparative forms (e.g. more, less, which lemmatize to themselves).

However I did some digging now and it turns out the answer might be much more mundane - the BNC manual cited here comments on the CLAWS tag AV0 (=ADV):

adverb (general, not sub-classified as AVP or AVQ), e.g. often, well, longer, furthest. Note that adverbs, unlike adjectives, are not tagged as positive, comparative, or superlative. This is because of the relative rarity of comparative or superlative forms.

Combined with the information here about the lemmatization of the BNC based on Beale (1987), this suggests that the main reason BNC lemmatizes comparative adverbs to themselves is that the CLAWS lemmatizer did comparative adjective lemmatization based on the POS tag, but this distinction was not available for adverbs.

As for how the GUM lemmas are created: unlike XPOS tags in the corpus, which are 100% manually assigned from scratch, lemmatization is manually corrected from automatic output by the TreeTagger. This means that while some errors can be expected to slip through, it is checked and also validated against some common errors.

If we decide to change from the BNC/CLAWS practice, which is also used by TreeTagger, and want to lemmatize comparative/superlative English adverbs to the positive form, it shouldn't be difficult to achieve. GUM only has 21 unique lemmas associated with XPOS RBR/RBS in 110K tokens, and some of them would probably be left alone (e.g. more, unless you think that should be lemmatized as much).

Personally I'm open to doing it for GUM since it gives more information and would be more in line with other languages, but the price is inconsistency with non-UD English corpora (many of which are admittedly not lemmatized)

[bjascob]

As an engineer, I look at this from the simple perspective that I'm expecting the lemma to be the dictionary or simplest form of the word. To me that means the positive form.

From my coding experience, I often see ADJ and ADV get lumped together. They fulfill similar roles, have the same inflection rules (in English anyway) and sometimes the same word can be used in for either. From a practical perspective I think it's confusing to have ADV lemmatize to the comparative form when ADJ doesn't. Just my two-cents..

[martinpopel]

@bjascob: you may be interested in yet another open-source lemmatizer for English: code and data.
I guess now there are many better lemmatizers, but a decade ago this was the best I could get (it was evaluated as better than the BNC lemmatization).

[bjascob]

Thanks. Looks like they have a fairly extensive set of rules and an overriding dictionary. I'll check it out. CLiPS/pattern.en also have a fairly extensive setup. However, it's difficult to tell which is the best without a ground-truth to test against. I was hoping the UD treebanks would be useful here but looks like they probably aren't accurate enough, at least for some pos types.

[amir-zeldes]

OK, I'm convinced regarding the comparative lemmas (especially since cross-linguistically this makes more sense). I've made the changes to GUM here along with some other fixes: UniversalDependencies/UD_English-GUM@a971d8831ed44696780273ee3e886eb44c2ed674

If you notice any other lemmatization issues specific to this corpus, feel free to report them in the GUM source repo (amir-zeldes/gum)

[bjascob]

Thanks. Took a look at the revised corpora today and GUM looks much improved. I think I can use GUM and LinES for some testing, at least to get started.

[amir-zeldes]

Glad to hear it - we fix issues with the data as they are identified, so feel free to report any remaining errors you find.

[manning]

I'm also good with the decision that the lemma of comparative/superlative adjectives/adverbs should be the positive. (I presume, however, that we don't lemmatize regular adverbs to the corresponding adjective.) For UD_English-EWT, the story is that the originally used lemmatizer (Stanford CoreNLP) didn't do anything for adverbs and adjectives (only nouns, verbs, pronouns, auxiliaries), and while quite a bit of lemma correction has been done, that's been for individual mistakes not a systematic deviation. But we will aim to fix it soon. :)

I'll add a comment about this to the section in the docs on lemmata, and then close this issue.