Docs: V2 validation

[dan-zeman]

The validation script (https://github.com/UniversalDependencies/tools/blob/master/validate.py) and its on-line output (http://universaldependencies.org/validation.html) currently fails to report certain errors that are not allowed in UD-released data. The following should be checked:

1. Presence of (just one) # sent_id = comment before every sentence.
2. Uniqueness of the sentence id (it must be treebank-wide, i.e. repository-wide unique, not just file-wide unique). This could be optionally checked by another script in tools, check_sentence_ids (https://github.com/UniversalDependencies/tools/blob/master/check_sentence_ids.pl) but the on-line test suite would have to be configured to call that script.
3. Presence and non-emptiness of (just one) # text = comment before every sentence.
4. Concatenation of all forms (multi-word tokens where applicable) from a sentence must be equal to the sentence text provided in the text comment without space characters.
5. SpaceAfter=No must be present in the MISC column whenever a (non-last) token is adjacent to the following token in text. In other words, if A is concatenation of tokens where a token either has SpaceAfter=No, or is the last token of the sentence, or we add a space after it in A, and if B is the value of text where any sequence of two or more spaces is reduced to just one space, then A and B must be equal.

[msimi]

What is the best practice for including a treebank unique sent_id (say XXX) and at the same time the sentence id (say YYY) within a specific resource (say ZZZ) contributing to the treebank?

Would something like this be considered compliant?

sent_id = ZZZ-YYY

text = …

or we could do something like:

sent_id = XXX

ZZZ id = YYY

text = …

Also should it be NoSpaceAfter=Yes or SpaceAfter=No? I found both.

— Maria

On 28 Jan 2017, at 17:55, Dan Zeman [e-mail] wrote:

The validation script (https://github.com/UniversalDependencies/tools/blob/master/validate.py https://github.com/UniversalDependencies/tools/blob/master/validate.py) and its on-line output (http://universaldependencies.org/validation.html http://universaldependencies.org/validation.html) currently fails to report certain errors that are not allowed in UD-released data:

Presence of (just one) # sent_id = comment before every sentence.
Uniqueness of the sentence id (it must be treebank-wide, i.e. repository-wide unique, not just file-wide unique). This could be optionally checked by another script in tools, check_sentence_ids (https://github.com/UniversalDependencies/tools/blob/master/check_sentence_ids.pl https://github.com/UniversalDependencies/tools/blob/master/check_sentence_ids.pl) but the on-line test suite would have to be configured to call that script.
Presence and non-emptiness of (just one) # text = comment before every sentence.
Concatenation of all forms (multi-word tokens where applicable) from a sentence must be equal to the sentence text provided in the text comment without space characters.
SpaceAfter=No must be present in the MISC column whenever a (non-last) token is adjacent to the following token in text. In other words, if A is concatenation of tokens where a token either has SpaceAfter=No, or is the last token of the sentence, or we add a space after it in A, and if B is the value of text where any sequence of two or more spaces is reduced to just one space, then A and B must be equal.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/UniversalDependencies/docs/issues/406, or mute the thread https://github.com/notifications/unsubscribe-auth/AIq8fbIG3gTG6usUFu_Y49T0vmx8nnJXks5rW3MUgaJpZM4LwjY_.

[martinpopel]

Would something like this be considered compliant?

sent_id = ZZZ-YYY

Yes, technically any sent_id matching the regex [a-zA-Z0-9_-]+ is ok.
Slash character (/) is forbidden, except for parallel treebanks as described in
https://github.com/UniversalDependencies/docs/issues/321

Personally, I would suggest to keep sent_id short (and unique within the treebank, of course).
Sent_id may be used in future for annotating coreference, discourse relations, bridging, alignment etc.
If there are legacy IDs in the original treebank conforming to the above-mentioned requirements,
they can be directly used as sent_id.
Otherwise, I would suggest to keep the orig IDs in another comment (e.g. orig_id = ZZZ-YYY).

Also should it be NoSpaceAfter=Yes or SpaceAfter=No? I found both.
SpaceAfter=No is the official way as described in
http://universaldependencies.org/format.html#untokenized-text
Please let me (or rather the treebank authors) know which treebank uses NoSpaceAfter=Yes, so this can be fixed.

Best regards,
Martin

----- Original Message -----

From: "Maria Simi" [e-mail]
To: "UniversalDependencies/docs" [e-mail]
Cc: "Subscribed" [e-mail]
Sent: Saturday, January 28, 2017 7:48:52 PM
Subject: Re: [UniversalDependencies/docs] V2 validation (#406)

What is the best practice for including a treebank unique sent_id (say XXX) and
at the same time the sentence id (say YYY) within a specific resource (say ZZZ)
contributing to the treebank?

Would something like this be considered compliant?

sent_id = ZZZ-YYY

text = …

or we could do something like:

sent_id = XXX

ZZZ id = YYY

text = …

Also should it be NoSpaceAfter=Yes or SpaceAfter=No? I found both.

— Maria

On 28 Jan 2017, at 17:55, Dan Zeman [e-mail] wrote:

The validation script
(https://github.com/UniversalDependencies/tools/blob/master/validate.py
https://github.com/UniversalDependencies/tools/blob/master/validate.py) and
its on-line output (http://universaldependencies.org/validation.html
http://universaldependencies.org/validation.html) currently fails to report
certain errors that are not allowed in UD-released data:

Presence of (just one) # sent_id = comment before every sentence.
Uniqueness of the sentence id (it must be treebank-wide, i.e. repository-wide
unique, not just file-wide unique). This could be optionally checked by another
script in tools, check_sentence_ids
(https://github.com/UniversalDependencies/tools/blob/master/check_sentence_ids.pl
https://github.com/UniversalDependencies/tools/blob/master/check_sentence_ids.pl)
but the on-line test suite would have to be configured to call that script.
Presence and non-emptiness of (just one) # text = comment before every sentence.
Concatenation of all forms (multi-word tokens where applicable) from a sentence
must be equal to the sentence text provided in the text comment without space
characters.
SpaceAfter=No must be present in the MISC column whenever a (non-last) token is
adjacent to the following token in text. In other words, if A is concatenation
of tokens where a token either has SpaceAfter=No, or is the last token of the
sentence, or we add a space after it in A, and if B is the value of text where
any sequence of two or more spaces is reduced to just one space, then A and B
must be equal.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/UniversalDependencies/docs/issues/406, or mute the thread
https://github.com/notifications/unsubscribe-auth/AIq8fbIG3gTG6usUFu_Y49T0vmx8nnJXks5rW3MUgaJpZM4LwjY_.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/UniversalDependencies/docs/issues/406#issuecomment-275866569

[dan-zeman]

@msimi

Would something like this be considered compliant?
# sent_id = ZZZ-YYY
# text = …

Yes, that is compliant.

Also should it be NoSpaceAfter=Yes or SpaceAfter=No? I found both.

SpaceAfter=No

[fginter]

Little progress report on this one:

1. sent_id now deployed (including uniqueness across multiple files)
2. text and SpaceAfter=No in the works and will be deployed later today

@martinpopel -- I'll add a check which will fail NoSpaceAfter=Yes in the validation. Good catch.

[martinpopel]

Great.
Now I see where the NoSpaceAfter=Yes comes from: @jnivre 's email. So hopefully, it is not present in any treebank (yet).

[jnivre]

Well, it was present in UD_Swedish, because of the same "slip of the brain" that caused it to appear in the email, but it is fixed now.

[dan-zeman]

@fginter : Great news you're on the case, thanks! However, the currently deployed version still seems to not recognize the ids:

.../UD_Arabic(dev)> ../tools/validate.py --lang ar ar-ud-dev.conllu
[Line                   13]: Missing the sent_id attribute.
[Line                   72]: Missing the sent_id attribute.
...
[Line                   688]: Missing the sent_id attribute.
...suppressing further errors regarding Metadata
*** FAILED *** with 786 errors
Metadata errors: 786
.../UD_Arabic(dev)> head ar-ud-dev.conllu
# sent_id = afp.20000715.0011:p1u1/ar
# text = مجموعة من الاميركيين المعارضين للحظر تزور البصرة

[fginter]

Give me an hour. This thing is now in a free-fall state as I am putting everything together and making sure everything is in sync with everything. :)

[fginter]

NoSpaceAfter=Yes will be checked in the validator now. This version will hopefully be the most paranoid version of the validator ever. :)

[spyysalo]

NoSpaceAfter=Yes will be checked in the validator now.

SpaceAfter, not, NoSpaceAfter, right?

[jnivre]

Yes, it should be SpaceAfter=No, but because of some crossed wires in my brain I wrote NoSpaceAfter=Yes in a mail to the UD list recently (and temporarily added it to UD_Swedish). Because of this, Filip is adding a check to make sure that no one is following my example (which was corrected in a later email).

[fginter]

Guys, I'm really confused with this sent_id attribute.

@martinpopel:

Yes, technically any sent_id matching the regex [a-zA-Z0-9_-]+ is ok.

@martinpopel:

Slash character (/) is forbidden, except for parallel treebanks

@dan-zeman:

sent_id = afp.20000715.0011:p1u1/ar does not validate

I mean, I can code anything you want into the validator. But please tell me how do I for example check "Slash character (/) is forbidden, except for parallel treebanks" in the validator? How do I know (in the validator and the surrounding automatic validation pipeline) what is a parallel treebank and what not?

I guess

any sent_id matching the regex [a-zA-Z0-9_-]+ is ok

is then not a formal requirement, i.e. "anything goes" is the rule? I think for the time being, I will ignore the no-slashes restriction and allow any string as the id.

[dan-zeman]

Ah, I see the problem now, sorry for the confusion. @martinpopel wanted to reserve the slash character to have a special meaning in certain tools but I am actually using these tools to generate UD data, and as a consequence the slash character appears there. Not a big deal, I can make it disappear if that's what we want to release, just please then make the error message a bit more user-friendly (not that sent_id is missing but that its value is invalid, or even better that it contains slash etc.)

In general, you obviously cannot test whether the treebank is parallel (maybe you could just have an option --slash-in-sent-id-allowed or something).

I would be inclined to allow the slash in UD data and better explain in UD docs what it means and how it is used. Then you would either not ban slashes at all, or maybe check whether there is at most one slash, and what follows is the language code (because even parallel treebanks in UD releases are currently distributed separated by language). But as I said, I can live with slashes banned entirely if that is the consensus (only I suspect that @martinpopel will then need to have the check optional so that he can use the validator for the parallel treebanks too).

[fginter]

NoSpaceAfter=Yes will be checked in the validator now.

...for non-existence that is. :smile:

[fginter]

Thanks, @dan-zeman. I will leave the slash allowed now and I can do the language code check. Since the validator gets the --lang parameter, that should, match, right? Also: will improve the error message. I would like to avoid things like --slash-in-sent-id-allowed. :smiley:

[martinpopel]

Yes. To summarize:

• with --lang=ud a slash in sent_id is allowed (otherwise the whole idea of downstream use would not make sense)
• with other lang, no slash is allowed and Dan will remove it from the Arabic data. We can weaken this rule in future.

In future we can:

• allow --lang=en,de etc. for parallel treebanks, or --lang=en,en for paraphrases etc. This would allow slash in sent_id.
• sent_id should always match ^[a-zA-Z0-9_-]+(/[a-z-]+(_[a-zA-Z0-9-]+)?)?$ (as explained in #321).

[fginter]

OK. So I will now code in that any non-space string is fine and should have a maximum of one slash, and that only in case of --lang=ud. Let's get back to the more complex stuff later. This will be deployed in 10min or so.

[fginter]

OK. So to the best of my knowledge, all of what @dan-zeman listed when opening this issue is now in and deployed. I will keep checking and debugging, but closing for the time being. Reopen this, or open new issue in case the validator fails to pass or fails to fail something. :)

[liljao]

I am getting an error message on the validator page for the Norwegian Nynorsk treebank:
"Unknown UD DEPREL: nsubj:pass".

Is this a bug? We have the same secondary types in the Norwegian Bokmål treebank and it passes validation.

[fcbr]

You need to make sure that nsubj:pass is listed in this file:

https://github.com/UniversalDependencies/tools/blob/master/data/deprel.no_nynorsk

Compare to the one for Norwegian Bokmål https://github.com/UniversalDependencies/tools/blob/master/data/deprel.no_bokmaal

[fginter]

@fcbr beats me to the reply by 10 seconds. You're fast! :smile: