Epub-specs: update xml schema reference?

[mattgarrish]

We still reference XSD 1.0 Part 2 2004 in Packages, OCF, and Media Overlays, but 1.1 was released in 2012 (after EPUB 3.0 was released).

Seems like it should have been updated in 3.0.1, unless someone has memories we chose not to for some reason? @murata2makoto this is usually your domain.

[murata2makoto]

XSD 1.1 is not supported very well. For example, .net does not support it. We should use the 2nd edition of XSD 1.0.

[laudrain]

XSD 1.1 is already in use. For instance, for ONIX3, EDItEUR started to circulate in 2017 a set of XSD 1.1 schemas to better check ONIX feeds validity. This allows for building rules for cross tags validation.
It can prove useful for EPUB as epubcheck 3.2 would rely on XSD rules without any Java development.

It's true that .net doesn't support it today, but it runs well in Saxon and Xerces.
I think we should reference XSD 1.1.

[llemeurfr]

The important evolution of XSD 1.1 is the addition of "assertions", which allow what Schematron (rule based schema) usually is used for.
Also, XSD 1.1 is based on XPath 2.0, which is much more powerful than the 1.0 version.
A more dangerous evolution is the addition of conditional type alternatives (the type of a structure is based on a condition).

XSD 1.1 is backward compatible (All schema documents conformant to version 1.0 should also conform to version 1.1).

But the support of XSD 1.1 is not so great: Saxon, Xerces, XML Spy (new versions) are ok with it, but that's all ... what about XSD implementations in relational DB (those with an XML type like SQLServer, Oracle 11G, IBM DB2), or implementations is Swift, Kotlin etc. ?

Therefore, because of its backward compatibility, I think that we CAN refer to XSD 1.1 when we specify datatypes, but we SHOULD NOT implement any 1.1 feature in any XSD schema that this WG could produce.

Cordialement,

Laurent Le Meur
EDRLab

Le 3 avr. 2018 à 09:15, Luc Audrain [e-mail] a écrit :

XSD 1.1 is already in use. For instance, for ONIX3, EDItEUR started to circulate in 2017 a set of XSD 1.1 schemas to better check ONIX feeds validity. This allows for building rules for cross tags validation.
It can prove useful for EPUB as epubcheck 3.2 would rely on XSD rules without any Java development.

It's true that .net doesn't support it today, but it runs well in Saxon and Xerces.
I think we should reference XSD 1.1.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/w3c/publ-epub-revision/issues/1024#issuecomment-378151867, or mute the thread https://github.com/notifications/unsubscribe-auth/AOQD0g4Ede03crHh2BAnEFCHR6Zp2YkXks5tkyGTgaJpZM4TDBfh.

[murata2makoto]

EPUB 2 and 3 have used RELAX NG rather than XSD. Datatypes of XSD are used via RELAX NG schemas. Thus, OCF references XSD Part 2 (datatypes) but not XSD Part 1 (structures).

epubcheck uses RELAX NG and Schematron. HTML5 validator also uses RELAX NG. Neither use XSD Part 1. If somebody would like to advocate the use of XSD rather than RELAX NG, please try. I strongly believe that XSD is unusable for narrative documents.

There are some differences between XSD 1.0 Part 2 and XSD 1.1 Part 2. Here is a summary. Unless there are strong reasons to use new features, we should stick to XSD 1.0 Part 2.

[iherman]

I do not know whether this has a relevance for EPUB right now, but the official XSD datatypes reference for RDF is XSD 1.1 (see, eg, RDF1.1 Concepts). This may become important if now or in future there is a stronger reference to RDF for EPUB metadata.

[mattgarrish]

Datatypes of XSD are used via RELAX NG schemas.

Do we even need to reference XSD in this context? The reference to XSD comes in through this repeated text:

Validation using this schema requires a processor that supports [NVDL], [RelaxNG-Schema], [ISOSchematron] and [XMLSCHEMA-2].

Could we live with a pointer to the NVDL schema and drop the explanatory prose that follows? Or can we shift this prose into the NVDL schema where it is no longer in the spec purview?

The only normative references to datatypes are to define the collection role attribute (nmtoken) and the dcterms:modified timestamp (dateTime). They're also referenced in the meta properties vocabulary to establish the expected value of each property.

I don't think it matters much whether we reference 1.0 or 1.1 for these cases, given not much changed from 1.0 and they're just establishing expected patterns.

I strongly believe that XSD is unusable for narrative documents.

They might be feasible for the package and ocf schemas. That said, if people want to translate the EPUB RNC schemas, they're free to do so. Let's not maintain multiple formats ourselves. One is enough.

[murata2makoto]

I checked ISO/IEC 29500-2 (RELAX NG). It does not require conformant RELAX NG validators to support XSD datatypes. Since our RELAX NG schemas uses XSD datatypes, we should continue to mention XSD Part 2 and need the explanatory prose you quoted.

[mattgarrish]

and need the explanatory prose you quoted

Or we drop the schemas as a part of the specification and provide them separately. HTML doesn't provide schemas anymore. SVG2 has dropped validation schemas. We'd be in good company.

Having them as a normative part of the specification has always been confusing (especially hidden behind an nvdl wrapper), as its unclear what we're even allowed to do when there are issues in them that need fixing. Finding all the schema pieces from the specification isn't exactly easy, either.

[murata2makoto]

I do not believe that dropping schemas from EPUB 3 is a good idea. Some syntactical restrictions are mechanically enforceable by such schemas. Schemas also help to avoid wordy descriptions of syntactical constraints on XML documents.

One could argue that schemas as part of epubcheck are good enough. However, I disagree, since specifications (not programs) should make restrictions clear.

Things are quite messy about HTML5. WHATWG HTML, W3C HTML5, and Validator.nu define different restrictions on permissible documents. I heard that SVG2 is not doing very well.

[mattgarrish]

I didn't say we stop supporting the schemas, though. I said we drop them as a normative component of the specifications and maintain them separately.

Locking the schemas into the revision cycle invites errors that force implementors to make their own decisions about how to solve in the interim. We see it repeatedly with epubcheck.

[murata2makoto]

I think that dropping schemas make the EPUB 3.2 specification less reliable, since requirements on documents become unclear.

Practically, I do not think that dropping schemas is a part of the 3.2 project.

[mattgarrish]

I think that dropping schemas make the EPUB 3.2 specification less reliable

That might be true if any real attention were paid to the schemas, but when was the last time consideration was given to them during a revision? How complete are they, even?

They're always left to the last minute, and the 3.1 package schema went out with errors in it.

[murata2makoto]

I did a lot of work on schemas when 3.0 was developed. But errors may remain. I agree that we should review them.

[mattgarrish]

I did a lot of work on schemas when 3.0 was developed.

I'm not questioning that or intending to disparage that work.

But errors may remain. I agree that we should review them.

This is the problem. We haven't done much review since 3.0, and they're not taken seriously enough in revisions. We don't have test documents or anything to support any changes we make. That's why I'm not a big fan of them as normative pieces of the specification.

We also should consider adding a note that you can't fully validate documents by the schemas alone. Epubcheck has many rules on top of the schemas to validate the constraints.

[murata2makoto]

We should review schemas, but they have been used as part of epubcheck. Do you have any evidence that existing schemas are harmful? Dropping schemas is certainly a very risky decision, and I strongly oppose to such a change unless you prove that existing schemas lead to more disadvantages than advantages.

[mattgarrish]

To recap:

• They've been shown to be wrong, yet they're "authoritative".
• The specification is normative and the schemas are normative, and yet there's nothing to say which is more authoritative when they don't agree.
• The schemas are not capable of validating the entirety of the constraints imposed by the specifications, yet there's nothing to tell an implementer that this is so.
• The working group rarely refers to the schemas during revisions, and only a small group often ever reviews or edits them (basically, you, me and Markus).

If the working group is not committed to their maintenance (updating them as part of every change, producing a framework for testing them, etc.), why persist them as a normative aspect. The advantage of them can't be purely theoretical.

[murata2makoto]

They've been shown to be wrong, yet they're "authoritative".

Yes, just like the prose. Both the prose and schemas have to be maintained.

The specification is normative and the schemas are normative, and yet there's nothing to say which is more authoritative when they don't agree.

This is the way it is in many specifications, which have both normative prose and some other
normative descriptions. There should be no conflicts.

The schemas are not capable of validating the entirety of the constraints imposed by the specifications, yet there's nothing to tell an implementer that this is so.

Schemas are not all mighty. But schemas are readable and also provide automated validation. Overall, they provide reasonable solutions.

The working group rarely refers to the schemas during revisions, and only a small group often ever reviews or edits them (basically, you, me and Markus).

This is true. Also, very few people carefully read every sentence in the specifications.

I would like to ask. Do you believe that the prose capture every syntactical restriction captured in schemas? I am not at all sure. Also, I find schemas much more readable than prose.

[mattgarrish]

I'm content to conclude that we disagree, as usual, and let the chairs make a call on how we deal with these going forward.

[GarthConboy]

This is huge "don't care" in my book. Though, if there was no substantive change between 3.0.1 and 3.1 on this front, I'd tend to just "stay the course." But be interested for @dauwhe to chime in.

[mattgarrish]

The problem is we have to care quite a bit about this if we want timely support for new revisions. I agree whether the schemas are in the specification or not is a bit arcane a discussion, but, for me, it ties into the epubcheck support problem we face after each revision.

This is the document I created after the 3.1 revision to try and enumerate everything that we couldn't (or didn't think to) put in the schemas:
https://docs.google.com/document/d/1kgoxSEhF_iTyixj6XaZIc0cgruBNNQknaif4fyXqKto/edit#heading=h.oykbse51y2u0

But trying to figure this out after the fact is time consuming and error prone.

We need to do a better job of logging validation issues as they arise, but given that the schemas are always going to be incomplete, at best, should we continue to try and manage them within the specification and also have to figure out when changes need to be logged with epubcheck, or should we set up maintenance of the schemas formally within epubcheck? We'd then have one place to maintain the necessary changes, and could dedicate interested individuals to fixing the non-java issues during revisions.

That's the bigger issue I'm dancing around here.

[murata2makoto]

This is the document I created after the 3.1 revision to try and enumerate everything that we couldn't (or didn't think to) put in the schemas:
https://docs.google.com/document/d/1kgoxSEhF_iTyixj6XaZIc0cgruBNNQknaif4fyXqKto/edit#heading=h.oykbse51y2u0

This list is useful. It appears that about 30% of the listed issues should be done by schemas.

Reintroduction of features (and dropping alternative mechanisms) in 3.2 will make about a half of the remaining issues unnecessary.

We need to do a better job of logging validation issues as they arise.

Certainly yes. Other committees (e.g., SC34/WG4) do a much better job than us. Whenever we introduce changes, we have to make sure that we have changed schemas.

[murata2makoto]

I spoke with oXygen, which uses schemas from our specs for guiding users. They would like to have schemas in the EPUB 3.2 specifications but suggest that schemas may be non-normative.

I think that RNG schemas should be normative (and maintained very carefully) and that Schematron schemas may well be non-normative. I understand that I implicitly commit to the maintenance of RNG schemas....

[murata2makoto]

As I reported in the teleconference of the CG, oXygen uses schemas for guided editing of every XML document (including HTML) in EPUB publications.

[murata2makoto]

Normative RELAX NG schemas in EPUB 3.0.1 have been successfully used by oXygen for guided authoring of EPUB publications. (The nxml mode of Emacs may well be used for the same purpose.)

Dropping RELAX NG schemas from EPUB 3.2 is certainly a disservice to oXygen and its users. Even if they are maintained as part of epubcheck, Publishing@W3C will no longer ensure that schemas are correct. It is more difficult for me to sell, to my sponsors, my schema work as part of epubcheck than that as part of EPUB 3.x.

The only theoretical advantage of dropping schemas from EPUB 3.2 is that the EPUB 3 community group can avoid the cost of fixing discrepancies (if any) between schemas and prose. I think that this cost is insignificant. Nobody has proven otherwise. Since we are now committed to timely maintenance by errata, in-place-editing, and even EPUB 3.n (3<= n), I strongly think that we should stick to status quo.

[llemeurfr]

I'm 100% for deciding that the RELAX NG schemas are non-normative. As @mattgarrish said, to have two different normative sources is a source of problems for developers. This is also what the people at Oxygen advocated to @murata2makoto. This is also what the IPTC NewsML WG decided long ago.

These non-normative schemas must still be well maintained (and we can thank Makoto for his work on this) and well marketed (I mean, not only hidden in epubcheck, correctly exposed on the W3C web site along with the specification, with a reference from the spec). This is also true for the Schematron schemas which act as complements of the RNG schemas for rules outside of the RNG reach.

[murata2makoto]

I prefer normative schemas much more. That said, I can live with non-normative schemas as part of EPUB 3.2.

[dauwhe]

A possible compromise:

1. We say the schemas are non-normative
2. We do copy the schemas over to EPUB 3.2 space, and link to them as before
3. As EPUBCheck makes changes to the schemas, we attempt to integrate their changes.

That does leave the question of who is willing and able to make the initial set of changes to the schemas to move from 3.1 to 3.2.

[mattgarrish]

Looking at the mix of 3.0.1 and 3.1 files you added, I'd hazard a guess that the only schema that needs updating is the packages schematron schema.

I don't recall 3.2 making any changes to the packages elements or attributes from 3.0.1, so the primary issue is to create warnings for the various elements we've deprecated instead of removing. There's also a question of how completely the schematron assertions check the package document requirements, but that's an issue that isn't new to 3.2.

@avneeshsingh, is this something we should take up now as part of the work on epubcheck?

[dauwhe]

OK to change the prose in Appendix A of packages to say the schemas are non-normative? And then make the links to the various references also non-normative?

[mattgarrish]

Might be good to note that the schemas are subject to change. Maybe something like:

NOTE

These schemas are periodically updated and corrected outside of revisions if errors or deficiencies are found. As a result, they are subject to change at any time.

[dauwhe]

References updated. Closing issue :)