Epub-specs: define exactly what backward/forward compatibility means for 3.0.2

[RachelComerford]

@llemeurfr: define exactly what backward/forward compatibility means for such an evolution. Stating that "a recent RS read an old ebook", as @therealglazou did in a mail today, is actually true but won't help defining what can be added, relaxed etc. in this evolution. In the scope of the XML structures defined by EPUB 3, we can state that optional elements and attributes may be added, contraints on their values may be relaxed, no elements can be suppressed... but the processing model of EPUB is complex and I imagine that many other rules should be expressed to guarantee backward compatibility between EPUB 3.02 and EPUB 3.01.

@murata2makoto : Backwards compatibility and forward compatibility are defined in 1.1.1 of this document. I tried to apply that definition to EPUB 3.1.
EPUB 3.1 is not backwards compatible since EPUB 3.1 RSs cannot process EPUB 3.0.1 publications. EPUB 3.1 RSs (detached from EPUB 3.0 RSs) cannot be rolled out in a way that does not break existing EPUB 3.0.1 editors.

EPUB 3.1 is not forwards compatible since EPUB 3.0.1 RSs cannot correctly process EPUB 3.1 publications. EPUB 3.1 editors cannot be deployed without breaking existing EPUB 3.0.1 RSs.

The clause "EPUB 3.1 changes retained in EPUB 3.0.2" in the original proposal is not a list of 3.0.1 problems, but is a list of retained solutions to them. From this clause, we can easily construct a list of 3.0.1 problems to be addressed. Here I provide three of them.

The specification organization of 3.0.1 is problematic, as revealed by my earlier experiment.
Since the definition of vocabularies are embedded within 3.0.1, addition of new entries is difficult.
EPUB Accessibility is detached from 3.0.1 conformance.

[murata2makoto]

I referenced [Editorial Draft] Extending and Versioning Languages: Terminology.

[llemeurfr]

In the document referenced by @murata2makoto, I found a simple definition of backward and forward compatibility:

• [Definition: Language L2 is strictly backwards compatible with Language L1 if L2 Defined Text set > (superset) L1 Defined Text Set AND every text in L1 Defined Text set is compatible with L2.]

• [Definition: Language L1 is strictly forwards compatible with Language L2 if L1 Accept Text set > (superset) Language L2 Accept Text set AND every text in L2 Accept Text set is compatible with L1.]


The CG wants the new spec to be backward compatible with EPUB 3.01. In the same text, some useful wording is proposed for expressing what the definition means:

A language change is backwards compatible if consumers (RS in our case) of the revised language can correctly process all instances of the unrevised language. Backwards compatibility means that a newer version of a consumer can be rolled out in a way that does not break existing producers (publishing solutions in our case). A producer can send a text per the unrevised version of a language to a consumer that understands the revised language and still have the text successfully processed. 

Could we settle on such definition and additional wording?

[murata2makoto]

@llemeurfr You choose mathematical definitions in "Extending and Versioning Languages: Terminology." Although I have some math background, I would rather propose to adopt informal definitions. The informal definition of "backwards compatible" is what you quoted as "useful wording".

I would like to also use "forwards compatible" as defined in this document.

A language change is forwards compatible if consumers of the unrevised language can correctly process all instances of the revised language. Forwards compatibility means that a newer version of a producer can be deployed in a way that does not break existing consumers. A producer can send a text per the revised version of a language to a consumer that understands the unrevised language and still have the text successfully processed. Of course the older consumer will not implement any new behavior, but a producer can send a text of the revised language and still have the text successfully processed.

[llemeurfr]

I see not harm using your preference as the definition, as long as the other expression (which relates one version of the language with another, not the language to its consumer) is expressed as "useful wording".

[murata2makoto]

Does 3.0.2 need some wording about backwards- and forwards compatibility? If not, I think that we are ready to close this issue.

[llemeurfr]

It is the starting point of this issue: I think that for naming the next version under discussion, we need 1/ to define backward compatibility (this issue) 2/ to define a versioning scheme using this notion of backward compatibility (#983) 3/ to apply this versioning scheme to the new version and all future versions of 3.x.y (and beyond, but this is another matter).
IMO the CG should not play with version numbers without any scheme, this is un-professional.

[murata2makoto]

So, @llemeurfr, in which document should we put the definition of compatibilities and the versioning scheme? The EPUB CG can publish a report as explained by this page. Alternatively, since they are applicable not only to EPUB 3.X but EPUB 4.X, we might want to ask the Publishing WG to publish a group note.

Here are some reports from w3c community groups.

• MusicXML Version 3.1
• Modelling Opportunity Data 1.0

[laudrain]

@llemeurfr I am not at ease with this theoritical discussion. When revising 3.1, we were more practical, and this 3.0.2/3.2 discussion IMHO should stay within this realm.
I don't say this for WP/PWP/EPUB4...

BTW, the initial goal of 3.1 included compatibility with 3.0.1 (see 3.1 Work plan at http://www.idpf.org/workplans/2015/epub/#mo).
That's why "3.1" version numbering was acceptable for me (see here issue #695 for more).

[therealglazou]

BTW, the initial goal of 3.1 included compatibility with 3.0.1

That announced "compatibility to the maximum extent possible" has always been a joke. As an implementor, it is very clear, has always been very clear, that 3.0.1 and 3.1 processors are different. I said it when we started 3.1 and the first technical bits arose.

[murata2makoto]

@laurdain What you call "theoretical" is an attempt for providing non-subjective judgement. Meanwhile, calling EPUB 3.1 "compatible with EPUB 3" is just a sales talk.

[laudrain]

@murata2makoto I don't contest the insights in this discussion, but regarding 3.1, it looks to me like a change of metric system, as we did not use this one for 3.1.

[murata2makoto]

@laudrain We need a definition of compatibilities that allows everybody to provide the same answer. In the case of EPUB 3.1, I did (and still do) believe that it is simply incompatible with EPUB 3.0 but everybody argued otherwise. I do not want to repeat the mess.

[llemeurfr]

@murata2makoto in which document should we put the definition of compatibilities and the versioning scheme?

I like the idea of a report of the CG explaining the reasons why the new EPUB 3.x(.x) is created and its relationships with 3.0.1 and 3.1. This is where the definition of backward/forward compatibility (and semantic versioning if agreed for EPUB 3.x, but this is a separate question) would make more sense IMO.
The WG may then adopt these definitions for WP and EPUB 4 (and there I'll insist on semantic versioning).

[therealglazou]

The WG may then adopt these definitions for WP and EPUB 4 (and there I'll insist on semantic versioning).

We have the choice between making backwards-compatibility normative in the scope of 3.0.2 and then a definition in a "report" can't be used since it's informative, and making it informative and then it's good to have it inside the document it refers to.

All in all, I think the definition of what means backwards-compatible in the context of the future EPUB 3.0.2 spec must be contained in that EPUB 3.0.2 spec and nowhere else. That does not block a "report" but the "report" cannot be, IMO, the reference. On the contrary, the reference is the spec, and the report should reference it.

[llemeurfr]

@therealglazou, even if the notion of backwards-compatiblilty is used in the introduction of the 3.x.x spec, i.e. an informative section?

[therealglazou]

even if the notion of backwards-compatiblilty is used in the introduction of the 3.x.x spec, i.e. an informative section?

You can use normatively only what's been defined normatively, and informative stuff can only be used informatively.

[llemeurfr]

I mean, defining backward compatibility is IMO defining guidelines for the CG to define what can / can't be done in any future EPUB 3.x.x. I suspect that the notion will not be used in the spec itself.

[murata2makoto]

@therealglazou After all, EPUB 3.0.2 in W3C whll be a report and will thus be informative.

[therealglazou]

After all, EPUB 3.0.2 in W3C whll be a report and will thus be informative.

<trollish>Oh my goodness... Sorry, I forgot this is a CG releasing IDPF-level documents while IDPF does not exist any more, and not RECs.</trollish>

[murata2makoto]

Today, I studied definitions in the Extending and Versioning Languages: Terminology carefully. I am afraid that my proposal to adopt it as a basis is not a good idea. This is because that document puts more weight on producers/consumers than documents. It does not consider maintenance of huge corpus of documents. In particular, definitions in this document cannot distinguish two important cases:

• Case 1) automatic translation from an old version to a new version is possible without changing the semantics; and
• Case 2) no translation from an old version to a new version is needed.

As far as I know, publishers are not afraid of the cost for using new software. They are very much afraid the cost of conversion of their publications. Thus, they care the differences between the above two cases very much.

[RachelComerford]

Oh my goodness... Sorry, I forgot this is a CG releasing IDPF-level documents while IDPF does not exist any more, and not RECs.

@therealglazou I appreciate your frustration and understand that you're contributing to valuable conclusion but this type of sarcasm, even if it's called out with tags, is alienating to new members who are trying to catch up on these issues and learn more. We want this to be a space that all members are comfortable contributing without fear of being mocked by members who may have more experience in this area.

[murata2makoto]

@RachelComerford I do not think that Daniel is sarcastic.

The W3C process does not allow IDPF recommended specifications (EPUB 2, 3.0, 3.0.1 and 3.1 among others) to be published as W3C recommendations, since they have not gone through the required IPR process.

The EPUB CG can only publish reports and nothing else, since it is neither a WG nor IG. Recommendations and even Group Notes are beyond the scope of the CG.

As a result, the CG can only publish 3.(0.)2 and other documents as reports. Daniel mistakenly argued that normative 3.(0.)2 cannot normatively reference a report that defines backwards compatibility. I reminded that 3.(0.)2 will also be a report, and he agreed.

In the maintenance of EPUB 2 and 3.X, our hands are very much tied. One could certainly argue that the industry will not pay any attention to reports, which are third-class documents in W3C.

My personal solution to this problem is to rely on ISO/IEC. In other words, we can first publish important documents as reports and then ask ISO/IEC JTC1/SC34/JWG7 to publish them as ISO/IEC International standards. After all, governments care ISO/IEC standards more than W3C recommendations.

[therealglazou]

@rachelcomerford first, I did really forget about it. Second, what is the expected value of a document that will be published outside of any Standards track, under the reference name of a Standard body that does not exist any more? Third, I find your reaction in front of humour - highlighting a real issue after all - suprising

[llemeurfr]

@murata2makoto, I don't see your point. The definitions of the W3C document state that EPUB 3.(0.)2 must define a superset of EPUB 3.0.1 and any valid EPUB 3.0.1 must also be a valid EPUB 3.(0.)2.
Which
a) is a good guideline for specifying the new version of the language.
b) guarantees that no translation from an old version to a new version is needed.

[llemeurfr]

@muratamakoto I agree that the main interested of doing this exercice is to get ISO validation for the new version; it is important for the Japanese industry, and this is the only standardization body which will validate this work. It may not be of sufficient value for some ex-IDPF members, which is what @therealglazou is expressing, therefore a good communication will be needed from the CG on this important "marketing" aspect.

[murata2makoto]

@llemeurfr

I don't see your point. The definitions of the W3C document state that EPUB 3.(0.)2 must define a superset of EPUB 3.0.1 and any valid EPUB 3.0.1 must also be a valid EPUB 3.(0.)2.

The definitions in the latter half of 1.1.1 in that document rely on the definition of text being compatible with a language. In my understanding, text is compatible with a language even if it does not belong to the language. More about this, see Extending and Versioning Languages: Compatibility Strategies

[murata2makoto]

@llemeurfr Yes, marketing is terribly important in standardization. I already invited SC34 officers to this discussion.

[GarthConboy]

True that.

And related to comments above, per the agreement by which the IDPF was merged into the W3C, the EPUB CG should, from a technical perspective, operate similarly to the IDPF with responsibility for EPUB 3.x and publication of any updated specs (while not W3C REC track) should have similar weight to those historically published by the IDPF. So

this is a CG releasing IDPF-level documents while IDPF does not exist any more, and not RECs

is accurate, proper, and exactly in line with expectations.

[murata2makoto]

Going back to the original point in this issue, I think that we cannot use "Extending and Versioning Languages: Terminology" as a basis. This document is my proposal.

[therealglazou]

Going back to the original point in this issue, I think that we cannot use "Extending and Versioning Languages: Terminology" as a basis. This document is my proposal.

Can I add suggestions directly there in the document or do you prefer here?

[murata2makoto]

Can I add suggestions directly there in the document or do you prefer here?

Technical comments here and editorial nits there.

[therealglazou]

Ok, so my suggestion for "strongly backwards-compatible" is as follows:

A given version of EPUB is said to be _strongly backwards-compatible_ with an older given version of EPUB if and only if any publication conformant to that older version of EPUB is also conformant to the newer one.

[murata2makoto]

I forgot to post this URL here.

https://docs.google.com/document/d/1oxFw-zXJ8PN-1QTAurhHsstpy0rD28hGKx6JKhINJm0/edit

[dauwhe]

Closing as I believe this is now stated in the changes doc.