Docs: Definition of word

[spyysalo]

Recent discussions have suggested that the UD documentation could benefit from a more detailed definition of "word". We can use this issue to discuss the existing definition and possible improvements.

( @jnivre @dan-zeman @ftyers , others? )

[spyysalo]

Currently, http://universaldependencies.org/u/overview/tokenization.html reads in part

The UD annotation is based on a lexicalist view of syntax, which means that dependency relations hold between words [...] there is no attempt at segmenting words into morphemes. [...] the basic units of annotation are syntactic words (not phonological or orthographic words)

[spyysalo]

Joakim's Coling keynote included the following related heuristic:

What is a word?

• Single part-of-speech tag
• Real syntactic relation

[spyysalo]

Greg Pringle has written at length on the topic in the context of UD Japanese at http://www.cjvlang.com/Spicks/udjapanese.html . One specifically relevant part:

Linguists later fine-tuned the definition of words with further distributional criteria:

'Positional mobility (syntagmatic mobility)': the word is free to be used at different places in the sentence. For example, John will go can be transformed into Will John go?, indicating that will, John, and go are three separate words.

'Internal stability (internal immutability)': In contrast with the positional mobility that the word enjoys, morphemes within a word are fixed in order. For example, played is a stable unit that does not permit rearrangement as ed-play.

'Uninterruptability': it is not possible to insert anything between the morphemes of a word. For instance, it is not possible to insert anything between play and -ed (e.g., play-be-ed).

[jnivre]

This is a great initiative. However, Pringle's characterisation does not quite work for syntactic words, because it seems to exclude clitics. This paper is very relevant as well:
http://coltekin.net/cagri/papers/coltekin2016turcling.pdf

[spyysalo]

Thanks! @coltekin proposes the following criteria

The term inflectional group (IG) in Turkish natural language processing literature refers to a sub-word unit. [...] the unit has been a de facto standard for representing words in Turkish NLP. [...] The current paper proposes tokenizing a surface word into multiple IGs only in case one of the following is true.
a. Parts of the word may have potentially conflicting inflectional features.
b. Parts of the word may participate in different syntactic relations.

[jnivre]

Clause a. only works for languages and words that inflect, but it can be generalized to postags, I think. Fusions are a case in point. Something like French "au" does not have conflicting inflectional features, because the preposition "à" does not inflect, but it has conflicting postags, ADP vs. DET.

[dan-zeman]

Wrt clitics, I was wondering how we would explain the difference between clitics attached to a verb (Spanish _dámelo, vámonos_) and morphemes that encode agreement of the verb with one or more arguments (and which could be interpreted as hidden pronouns when the respective argument does not appear as an overt nominal). Spanish _vamos_ itself could be said to encode _nosotros vamos_, and head-marking languages like Basque may even encode agreement with a second and third argument, but we are quite clear about UD not analyzing these as separate words ("do not annotate things that are not there"). While I agree that we should not annotate dropped subjects (or arguments in general), I would love to hear a sound rule that says why _vámonos_ is different.

One clear difference I can think of right now is exactly the possibility (in the agreement version) that the argument also appears as a separate word. However, I am not sure that a guideline based on this would work in languages that have clitic doubling.

[jnivre]

I agree that the distinction between pronominal clitics and inflectional endings that mark agreement is a bit fuzzy and that the former could develop into the latter historically. For the Spanish case, I would appeal to language-internal arguments about systematicity. Subject agreement is present in all verbs in Spanish, even with a separate overt subject, but "object agreement" is not. Therefore, when a morpheme referring to the object is glued to the verb, we avoid exceptions if we separate it from its host. How does that sound?

[dan-zeman]

On a more general note: I think that the default practice in UD is that we take orthographic words (in languages where they exist) as units. Only when we see good reasons to treat certain cases as contractions of multiple syntactic words, we split them; but if they work reasonably well as one word, we keep them together even though a parallel expression in a related language is rendered as multiple words. And there is some flexibility to these decisions, as always.

Conversely, when we have a good reason to say that one syntactic word spans several orthographic words, we connect them using the fixed relation, except for the very limited cases where we are allowed to include a space in a word form.

[dan-zeman]

@jnivre : sounds good to me, thanks. If this discussion results in a set of guidelines for UD-wordness, we should include the Spanish example with this explanation.

[spyysalo]

Stupid question: what's the rule against analyzing e.g. joined as join/VERB ed/AUX? Would it work the same if English didn't use space?

[dan-zeman]

@spyysalo : irregular verbs? If the English writing system did not use space, it would still not be clear where the auxiliary begins in _threw_.

[spyysalo]

Nice :-) How might that look as a general rule? Are there other arguments that could apply?

[manning]

There is a lot of literature in linguistics on distinguishing clitics from morphology. Among others, Arnold Zwicky has written a lot on this issue. Of course, as @jnivre notes, the results are not always crystal clear due to grammaticalization. There is an argument that some have pursued that romance "clitics" should really be treated as morphology. So I suspect that @dan-zeman's "On a more general note" is really part of the answer for us. But if you'd like to read more about how words, clitics, and morphology have been argued to be distinguishable, here's a little reading list:

• Zwicky/Pullum 1983
• Zwicky 1985
• Anderson Short encyclopedia article
• Holmstedt/Dresher Longer encyclopedia article
• Good slides from Rik van Gijn

[dan-zeman]

Awesome, thanks!

[spyysalo]

@manning: thanks, that great! I'll dig in after recovering from Coling :-)

The reason I'm asking this particular question now is the proposed UD Japanese analysis of e.g. 食べた tabeta “ate” as 食べ/VERB た/AUX (with aux dependency: http://universaldependencies.org/ja/overview/syntax.html#special-clausal-dependents). 食べた strikes me as equally obviously a single-word past form verb as the English joined, and I'd like to know what arguments in the UD framework would support keeping this as one word.

[amir-zeldes]

@spyysalo I think what you're looking for is what this paper calls "MUW" (Middle Unit Word):

http://www.lrec-conf.org/proceedings/lrec2016/pdf/122_Paper.pdf

[spyysalo]

@amir-zeldes: this work is certainly highly relevant (and discussed at length in http://www.cjvlang.com/Spicks/udjapanese.html, referenced above), but unfortunately even the longest "LUW" (Long Unit Word) in the paper splits 食べた into 食べ/VERB andた/AUX (see Figure 1).

[amir-zeldes]

You're right, sorry for the confusion! I agree, it's odd for this to be a token while other agglutinative morphemes (e.g. potential) are not.

[kanayamah]

@spyysalo, one strong reason to split 食べた (ate) into 食べ/VERB and た/AUX is other elements such as a polite marker can be inserted between these two words (食べました - 食べ/VERB まし/AUX た/AUX). Considering English perfective and French passé composé, I think it is quite natural to represent a past form with two words.

[dan-zeman]

@kanayamah, considering simple past in English as well as in good many other languages, it is also quite natural to represent a past form with one word :-) Is the range of "words" that you can insert before た wider than just the politeness morpheme? Like in English "I have just eaten", "I have not eaten", "I have already eaten" (and the inserted thing is always indisputably an independent word).

Multiple affixes can attach to a stem in morphologically rich languages, but it does not necessarily warrant wordness of these affixes. For instance, the Czech adjective _zvláštn-í_ "strange" ends with the morpheme _í_, which contributes the Case and Number features, and you can insert the comparative morpheme _ejš_ before that (_zvláštn-ějš-í_ "stranger") but it does not imply that either _ejš_ or _í_ are independent words.

[amir-zeldes]

@kanayamah I think one of the reasons some people are uncomfortable with this but have less problems with French is that the French auxiliary can stand by itself: it is white-space separated, it looks identical to either the verb 'be' or 'have' as used outside of the past tense construction, and it can be separated from the lexical verb by adverbs: 'il est déjà arrivé'. The morpheme た behaves more like an inflectional marker: its position is completely predictable and it cannot be used independently.

[spyysalo]

I agree with the points raised by @dan-zeman and @amir-zeldes. Specifically regarding the argument given by @kanayamah, I don't think the occurrence ofます between 食べ and た in 食べました is a reason to analyze た as an independent (syntactic) word rather than as a morpheme, as we can either consider 食べました as a word or た as an inflectional morpheme on ます.

[spyysalo]

@kanayamah : to clarify my concern a bit: the currently proposed UD Japanese representation implies the radical claim that Japanese has (effectively) no inflectional morphology. To the best of my knowledge, this is is a departure from both traditional and previous theoretical linguistic analyses of Japanese verb behavior.

Some of the published UD Japanese material suggests that you agree that the proposed (SUW) analysis segments into morphemes rather than syntactic words, e.g. http://www.lrec-conf.org/proceedings/lrec2016/pdf/122_Paper.pdf:

Short Unit Word (SUW): SUW is a minimal language unit that has a morphological function.

This kind of construction [...] is considered to be morphological (a word formation), rather than a syntactic relation. [regarding the derivation of e.g. かわいさ and 子どもっぽい]

However, the UD representation does not segment words into morphemes (see http://universaldependencies.org/u/overview/tokenization.html).

Are you using UD to represent the morphological structure of Japanese verbs (etc.), or does UD Japanese claim that Japanese verbs have no morphological structure?

[miyao-yusuke]

@dan-zeman @spyysalo Many other expressions (aspect, modality, passive, causative, benefactive, etc.) can appear between 食べ and た. Their composition is very systematic (no exceptions like "threw"), and some expressions can appear multiple times (e.g. double negation) and sometimes can be coordinated. I don't find any reason to prevent from segmenting them.

@spyysalo Modern syntactic theories consider these expressions as independent tokens. For example, the following book presents a comprehensive theory of Japanese syntax based on CCG, in which verb-following expressions are considered as independent tokens and given category S\S.

Bekki, Daisuke, 2010. Nihongo Bunpoo no Keesiki Riron: Katsuyoo taikee, Toogo koozoo, Imi goosee [Formal theory of Japanese grammar: The system of conjugation, syntactic structure, and semantic composition]. Japanese Frontier Series 24. Kurosio Publishers.

Anyway, I think we need a clear definition of "word" in the UD sense, which does not depend on the existence of spaces.

[spyysalo]

@miyao-yusuke : thanks for the quick response, your input is much appreciated! Some comments:

I think we need a clear definition of "word" in the UD sense, which does not depend on the existence of spaces.

Very much agreed! In the absence of more qualified volunteers, I've been trying to draft a rough proposal on this, but the more I read on attempts to define "word" cross-linguistically (e.g. Haspelmath 2011), the less confident I am...

no exceptions like "threw"

Wouldn't する → した, くる → きた, and いく → いった qualify as irregular past forms?

Modern syntactic theories consider these expressions as independent tokens. For example, [Bekki, Daisuke, 2010. Nihongo Bunpoo no Keesiki Riron: Katsuyoo taikee, Toogo koozoo, Imi goosee] presents a comprehensive theory of Japanese syntax based on CCG, in which verb-following expressions are considered as independent tokens

Thank you for the pointer! I could unfortunately not find a copy of 日本語文法の形式理論 (I take it) online, but will try to follow up on this. A couple of clarifying questions:

• Is it OK to equate "token" with (syntactic) "word" here? That is, would it be correct to say Modern syntactic theories consider expressions such as た in 食べた as independent syntactic words?

• Would it be correct to say that UD Japanese follows the approach of Bekki (2010) and rejects the view that Japanese verbs have past forms, analyzing past expression formation as a syntactic (rather than morphological) process?

Also, regarding the above statement from Tanaka et al. 2016, do you agree with

For example, the suffix さ sa changes an adjective into a noun [...] and っぽい ppoi changes a noun into an adjective [...] This kind of construction is very generative, and it is considered to be morphological (a word formation), rather than a syntactic relation.

i.e. to analyze the formation of e.g. かわいさ and 子どもっぽい as morphological rather than syntactic processes?

[miyao-yusuke]

Wouldn't する → した, くる → きた, and いく → いった qualify as irregular past forms?

Popular analysis is する is inflected to さ, し, せ, する, すれ, etc., and function words/morphemes attach to one of them; e.g. し + ない, ている, た, よう, etc. くる and いく are similar.

Is it OK to equate "token" with (syntactic) "word" here? That is, would it be correct to say Modern syntactic theories consider expressions such as た in 食べた as independent syntactic words?

They avoid defining "word". The important message here is that the distinction between word and suffix/morpheme is not very clear nor meaningful in Japanese, and considering everything as "syntactic unit" can explain various complications and interactions with syntax.

Would it be correct to say that UD Japanese follows the approach of Bekki (2010) and rejects the view that Japanese verbs have past forms, analyzing past expression formation as a syntactic (rather than morphological) process?

We are not following this specific theory, but follow the definition of SUW. However, I think we share the fundamental idea. Function words/morphemes of Japanese cannot be clearly classified into word or suffix/morpheme. Some tend to behave more like word, while others are closer to suffix, but most are somewhere in between. As far as I know, linguistics does not have a clear conclusion until now.

I'm not objecting to the inflection-based analysis. I mean both (and other possibilities in between) are acceptable and I cannot find any clear reason to reject one. We simply selected one of them (mainly due to practical reasons).

For example, the suffix さ sa changes an adjective into a noun [...] and っぽい ppoi changes a noun into an adjective [...] This kind of construction is very generative, and it is considered to be morphological (a word formation), rather than a syntactic relation.
i.e. to analyze the formation of e.g. かわいさ and 子どもっぽい as morphological rather than syntactic processes?

These specific examples might look like derivational morphemes (actually we had some discussions in the Japanese UD team about these constructions). However, they also have many characteristics of "wordness" (actually, many of the "wordness" criteria of Zwicky, Haspelmath, etc. can apply to them).

[jnivre]

I think we should form a working group on universal guidelines for word segmentation, with representation from diverse language groups. I still think we can use the paper by Cagri on principles for Turkish as a starting point, but it obviously needs to be refined and extended to cover typologically different languages. My own superficial impression of the Japanese situation is that we need a compromise solution, where some of the contested items are regarded as words and others as bound morphemes, but this does obviously not carry a lot of weight.

[spyysalo]

@miyao-yusuke :

The important message here is that the distinction between word and suffix/morpheme is not very clear nor meaningful in Japanese, and considering everything as "syntactic unit" can explain various complications and interactions with syntax.

I appreciate that, and I understand the merits of segmenting down to morphemes and representing morphological and syntactic structures in a unified fashion.

However, UD does not (currently :-)) address word formation, and I'm concerned that not defining a word unit above the morpheme level will have a negative impact on the value of UD Japanese annotations for cross-linguistic use cases.

I'm not objecting to the inflection-based analysis. I mean both (and other possibilities in between) are acceptable and I cannot find any clear reason to reject one.

I'm very happy to hear that, and I hope we can establish a good argument in support of a choice soon!

@jnivre :

I think we should form a working group on universal guidelines for word segmentation, with representation from diverse language groups. [...] My own superficial impression of the Japanese situation is that we need a compromise solution, where some of the contested items are regarded as words and others as bound morphemes

+1 on both!

[rtsarfaty]

@jnivre :

I think we should form a working group on universal guidelines for word segmentation, with representation from diverse language groups.

Great idea! happy to join the working group and represent the Semitic angle.

[mojgan-seraji]

I would also be happy to join the working group and represent the Indo-European languages with Perso-Arabic script. Word segmentation is a major challenge in the languages like Persian with regard to its special characteristics of different writing styles of "words".

[spyysalo]

I'd be happy to represent the Finnic language family :-)

[jnivre]

Following up on the discussion about UD Japanese, it seems clear that the segmentation of derivational morphology is not consistent with general UD principles, nor with practice in other UD treebanks, so I definitely think that should be changed. Similarly, when it comes to verbal morphology, all the descriptions so far point to this being a regular and systematic case of agglutination, similar to the situation in Turkish and Finnish, so it seems that an analysis using morphological features is most compelling there as well.

[spyysalo]

@jnivre : thank you for addressing this issue!

There is a lot of discussion of inflectional morphology above but less of derivational, so to illustrate, one specific problem with the proposed treatment of derivational morphology is shown in Tanaka et al. 2016 (page 4):

the suffix さ sa changes an adjective into a noun as in (5), and っぽい ppoi changes a noun into an adjective as in (6). [...] This kind of construction is very generative, and it is considered to be morphological (a word formation), rather than a syntactic relation.

mark(かわい, さ)
かわい ... さ
ADJ PART

mark(子ども, っぽい)
子ども ... っぽい
NOUN PART

As discussed above, UD does not involve morphological segmentation. The proposed representation has the obvious problem that as the derived forms (かわいさ and 子どもっぽい) do not appear as words, it is not possible to capture their derived parts of speech (NOUN and ADJ, resp.) anywhere.

The derivation is directly analogous to e.g. the derivation of "childish" or "childlike" from "child" in English. In UD, derived forms like these are words, annotated with their relevant parts of speech.

If this is not clearly stated in the documentation at the moment, we could perhaps formulate a rule along the following lines: In UD, forms created by derivational morphological processes should be represented as words tagged with their derived parts of speech. The root word and morphemes involved in the derivation are not separately represented in the UD analysis.

[miyao-yusuke]

In UD, forms created by derivational morphological processes should be represented as words tagged with their derived parts of speech. The root word and morphemes involved in the derivation are not separately represented in the UD analysis.

This statement requires the definition of "derivational morphological processes". さ and っぽい are different from -ish and -like because they can be attached to any phrase or any clause.

My intention in the previous discussions is the boundary between morphology and syntax is often unclear, and we need a clear definition.

[spyysalo]

My intention in the previous discussions is the boundary between morphology and syntax is often unclear, and we need a clear definition.

Understood. We're hoping to establish cross-linguistically acceptable UD guidelines in a working group on the topic, and it would be great to have someone representing Japanese. Would you mind participating?

[miyao-yusuke]

happy to join of course.

[jnivre]

The main point is that "changing a noun into an adjective" is not a syntactic relation, so there is no appropriate UD relation to put on an arc between these two elements. This is a clear indication that this should be treated as word formation, not syntax, and it is exactly parallel to the argument against the old "DERIV" links in the Turkish treebank.

[miyao-yusuke]

I see. Maybe the original explanation in Tanaka et al. 2016 was not very accurate. さ and っぽい do not only change a POS but they add some meaning, in a similar way to auxiliary verbs (they also change a POS in some cases).

[spyysalo]

@miyao-yusuke :

happy to join of course.

Great, thanks!

I'll put together a brief document summarizing some of the proposals so far and will get back to everyone interested in the topic next week.

[spyysalo]

さ and っぽい do not only changes a POS but they add some meaning

This does also hold for "-ish" and "-like" in English.

(Maybe of interest: Duncan A Freezing Approach to the Ish-Construction in English)

[miyao-yusuke]

Sure, but I mean they are not purely functional. Maybe they can be considered like "child-like"or "child like".

[jnivre]

But what is the syntactic relation?

[miyao-yusuke]

That's the problem, but it should be similar to "child like".

[jmnybl]

Question on (Finnish) derivation lemmas: We tag the derivational forms according to the real POS tag ("childish" is tagged as ADJ, not NOUN), but sometimes, if Finnish morphological analyzer is not able to produce lemma for the final derived form, we keep the original form in the lemma field (here for example it would be the NOUN "child"). I assume that this is not the correct approach and we plan to correct this in v2, so that the lemma would always be the derived form. Does this sound correct?

[jnivre]

@jmnybl Yes, this sounds completely correct to me.

[gcelano]

I do not know whether there is a more relevant issue opened on this, but I am wondering why rules for tokenization and word segmentation do not (routinely) allow for two ore more graphic tokens to be univerbated for the sake of syntactic analysis (e.g., "in spite of"). The guidelines explicitly advise against that (while allowing univerbation for a few exceptional cases). I can imagine that this could add some formal complexity, but I would argue that would be worth it because of a consistent formal representation of the syntactic word.

PS: I also found this link (https://universaldependencies.org/v2/word-segmentation.html), but it does not work.

[dan-zeman]

The guidelines define the relation fixed, which is used to connect parts of expressions like _in spite of_. You can think of that relation as technical means used to annotate (one particular type of) a syntactic word that consists of multiple orthographic words.

[dan-zeman]

PS: I also found this link (https://universaldependencies.org/v2/word-segmentation.html), but it does not work.

That page has been moved to https://universaldependencies.org/v2/segmentation.html. If you know where you saw the link you can fix it there. The page is part of "final reports" during the discussion before v2 guidelines were announced and I don't remember whether it is really compatible with v2 as they were approved. If there are any discrepancies, the Guidelines section on the website should rule. But this page may have some additional discussion of examples.

[gcelano]

The guidelines define the relation fixed, which is used to connect parts of expressions like _in spite of_. You can think of that relation as technical means used to annotate (one particular type of) a syntactic word that consists of multiple orthographic words.

This treatment seems to be in line with that of other "higher-order" technical dependencies, such as, for example, coordination. I have not yet reflected enough on these latter, but, in any case, I think that technical dependencies at the level of identification of the syntactic word impact formal representation consistency both crosslinguistically and intralinguistically significantly. More in general, it seems to me that we try to accommodate the orthography of a language much more than we should. Could this kind of technical dependency also be a "side effect" of the CoNLL-U format? If there were standoff annotation, this problem could be - at least formally - circumvented easily.

[akoehn]

Could this kind of technical dependency also be a "side effect" of the CoNLL-U format?

No, it is – from a technical standpoint – quite possible to have
whitespace in a token.

We have whitespace in tokens in the Hamburg
Dependency Treebank (and the WIP UD conversion) in some rare cases,
e.g. “Michael Jackson-Fans”. This is treated as a single word
because the modifier “-Fans” applies to “Michael Jackson” as a whole
and not just his last name. The canonical version would be
“Michael-Jackson-Fans” but if the author of a text used the variant
without Durchkopplung,
we can’t do anything about it. Connecting “Michael” and
“Jackson-Fans” with fixed would be incorrect IMO.

[jnivre]

Just for completeness: Another option that is sometimes relevant is the "goeswith" relation, which can be used, for example, if a word like "furthermore" is written "further more".

In general, I agree that the use of "technical" relations can be a problem, especially for users that are not familiar with UD and assumes that every relation is a true syntactic relation. It is the price we have to pay for encoding the whole (basic) syntactic structure as a simple spanning tree over words. Eliminating the "technical" relations would require a richer ontology.

[martinpopel]

I think the current solution (with the fixed relation) actually improves the consistency and usability of UD. Just that users need to be aware of the guidelines, but that is true for any guidelines and projects.

As for “Michael Jackson-Fans”: why not

flat:name(Michael, Jackson)
punct(Michael, -)
nmod(fans, Michael)

?
I mean flat (similarly to fixed) means that the whole phrase act as a single syntactic unit with no inner dependencies (as the term flat suggests), so choosing the first word as head is just a technical way how to annotate it consistently. If there was any modifier of the first word ("Michael" in our case), it would be automatic considered as a modifier of the whole fixed phrase (multi-word expression).

[sylvainkahane]

Word segmentation is part of the syntactic analysis. It is not possible to decide independently of a syntactic studies what are the words of a language. The great advantage of a purely formal tokenisation (for instance based on orthography for languages which have such a tradition) allows us to postpone this decision when we annotate and realize the syntactic analysis. If you proceed with a double-blind annotation (as it is expected for a gold-standard treebank), it could be problematic to impose a word segmentation.

@gcelano "in spite of" is certainly not a word, whatever is your definition of a word. Even its status as a MWE is debatable, because "of" is part of the subcategorization frame (Fr. régime) of "in spite".

About the treatment of MWEs in UD we made some propositions at UDW17: Multi-word annotation in syntactic treebanks - Propositions for Universal Dependencies.

[akoehn]

As for “Michael Jackson-Fans”: why not
flat:name(Michael, Jackson)
punct(Michael, -)
nmod(fans, Michael)

Then we would have to analyze "Michael-Jackson-Fans" and all other words with Durchkopplung in a similar fashion. We then would analyze "Foo-Bar" as flat(Foo, Bar), punct(Foo, -) but Foobar (which is just an alternative spelling of Foo-Bar) as a single word. One could continue that path and also analyze "Foobar" as two tokens but I think we all agree that this is not desireable.

This way, we special-case on the very rare occasion of "Michael Jackson-Fans". Otherwise we would get weird analyses in a lot of cases.

[gcelano]

If you proceed with a double-blind annotation (as it is expected for a gold-standard treebank), it could be problematic to impose a word segmentation.

In UD a word segmentation is already "imposed" after whitespace-based tokenization, in that graphic words can be regularly split.

@gcelano "in spite of" is certainly not a word, whatever is your definition of a word. Even its status as a MWE is debatable, because "of" is part of the subcategorization frame (Fr. régime) of "in spite".

I take "in spite of" as a syntactic word (on a par with "despite"). Of course, the idea is that one acknowledges that it now works as a unit (vs its more etymological analysis).

[gcelano]

No, it is – from a technical standpoint – quite possible to have
whitespace in a token.

We have whitespace in tokens in the Hamburg
Dependency Treebank (and the WIP UD conversion) in some rare cases,
e.g. “Michael Jackson-Fans”. This is treated as a single word
because the modifier “-Fans” applies to “Michael Jackson” as a whole
and not just his last name. The canonical version would be
“Michael-Jackson-Fans” but if the author of a text used the variant
without Durchkopplung,
we can’t do anything about it. Connecting “Michael” and
“Jackson-Fans” with fixed would be incorrect IMO.

@akoehn, even if you (exceptionally) allow whitespaces within a syntactic word, your example still looks to me as a workaround dictated by the limitation of the format. Of course, one could potentially take "Michael(-)Jackson-Fans" as a single concept/syntactic word, but frankly, if you compare it to other similar cases, I would here identify "Michael Jackson" and "fans" as separate syntactic words.

[akoehn]

Let me prepend that a) I'm not a specialist on word segmentation stuff and b) these decisions were made by others long before my time with the HDT.

@gcelano, the question is up to which level structure should be annotated (and I don’t think that this is a technical limitation, it is a decision made by UD). We took the (I think reasonable) approach not to go to the sub-token level. E.g., “Blumentopferde” is not subdivided (it is a single token). Similarily, the variation “Blumentopf-Erde” is also not subdivided; it means the same thing and when subdividing it, it would be logical to also split “Blumentopferde”. On the other hand, if someone wrote “Blumentopf Erde” (which is incorrect spelling), we would treat it as two tokens.

This is mainly a question of where to stop; we decided to stop rather early with subdivision. If there is consensus amongst UD people that whitespace should always trigger a new token then I am fine with it. I have no strong opinions in either direction.

[gcelano]

@akoehn, the difference between “Blumentopferde” and “Blumentopf Erde” pertains only to orthography, not to syntax. Therefore, I would expect them to be syntactically analyzed in the same way (whatever analysis one choses). If we analyze them differently, this is a case where orthography unduly influences syntactic analysis. The question is to ensure that the same syntactic phenomena get the same analyses.

[martinpopel]

everyone: If you have not done so yet, read the current Tokenization and Word Segmentation guidelines.

@gcelano: I agree with @akoehn that there is no technical limitation of UD or CoNLL-U which would prevent words with spaces. In deed, in Vietnamese UD, spaces regularly occur inside words (because they mark syllable boundaries rather than word boundaries).

In UD a word segmentation is already "imposed" after whitespace-based tokenization

The tokenization in UD is not just "whitespace-based". Punctuation is separated as well. For most treebanks the tokenization rules should be simple. In CoNLL2018, the best tokenization result for most treebanks was > 99%, but the best result for an average over all treebanks was 98.42, because there were a few languages with low tokenization results (e.g. Thai, which does not use spaces and UD has no training data for Thai).

I am not sure what is meant by the claim that in UD a word segmentation is already imposed. In UD, word segmentation is more difficult than tokenization because of multiword tokens. In CoNLL2018, the best word-segmentation result for an average over all treebanks was 98.18 (it may seem the same as 98.42 for tokenization, but it means 41% increase of errors). For languages like Arabic and Hebrew the difference in difficulty is much higher (tokenization almost perfect, but word segmentation 96.81 and 93.98, respectively).

@akoehn: So "Michael Jackson" is analyzed as two words in the Hamburg Dependency Treebank, but (the rare) "Michael Jackson-fans" as a single word with a space? As always: when you improve the consistency in one phenomenon, you decrease it in another. Let's hope your users are more interested in Durchkopplung than Michael Jackson:-).

[akoehn]

So "Michael Jackson" is analyzed as two words in the Hamburg Dependency Treebank, but (the rare) "Michael Jackson-fans" as a single word with a space?

Exactly. It is not just rare but super-rare (I would guess <= 4 tokens out of 4 million).

[gcelano]

@martinpopel, the argument is that while splitting graphic words for the sake of identifying syntactic words is allowed, merging (usually) is not (even if there are clear cases where this would be required). This has as a consequence that syntactic analysis in UD has to accommodate this via the use of technical dependencies. While technical dependencies may not be a _big_ problem intralinguistically (depending on the language), it is crosslinguistically.

On a side note: if two graphic words are to be merged, CoNLL-U could handle this rather easily if one directly follows the other; otherwise, it would be much more complex.

[amir-zeldes]

I think the argument for leaving compound modifiers inside tokens in German is not just a practical, but a theoretical one. German compound modifiers often have forms that do not correspond to any independent form of the noun (e.g. compounding truncation for Woll- from Wolle 'wool' in Wollknäuel 'ball of wool', or feminine compound linking 's' in Reinigungsfirma 'cleaning company').

This topic has been studied extensively in German morphology, and the general consensus is that these compounding stems do not constitute 'words' in German, but a kind of bound morpheme. In computational linguistics, the tradition has been very consistent in not making these into tokens. The "Micheal Jackson-Fans" example is implementing that guideline by saying 'if this is a compound modifier, it can't be a token'. I'm not necessarily sure it's worth the price of having whitespace in a token for the sake of this rare example, but I think the argument for it is actually one of theoretical consistency.

[murawaki]

Just posted a preprint on Japanese syntactic words in the context of UD Japanese: https://arxiv.org/abs/1906.09719