Epub-specs: Problem with XML content requirements and external entities: make MathML and SVG1.1 invalid

[iherman]

The current text on XML conformance says that an XML-based media _MUST_ meet:

...
External identifiers MUST NOT appear in the document type declaration [XML]
...

This constraint apply regardless of whether the given Publication Resource is a Core Media Type Resource or a Foreign Resource.

However, that creates problems with SVG 1.1. The conformance rules of SVG1.1 require the presence of:

<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" SYSTEM "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">

The DTD itself, as described in the relevant appendix, is making a heavy use of external entities (via its usage of DTD Modularization). As a consequence, per the current document, _no conform SVG 1.1 file_ is valid for EPUB 3.3.

Although the upcoming SVG 2 removed this requirement, there are lots of perfectly valid SVG 1.1 out there (SVG 2 is not yet a Recommendation, b.t.w., only a CR). Most notably, Adobe Illustrator generates SVG 1.1 files with those DTD-s in them.

A similar problem may occur with MathML 3, whose official DTD relies on the same approach (although MathML is more permissive and it does not _require_ the usage of that DTD).

This problem was originally raised in https://github.com/w3c/epubcheck/issues/1114.

[iherman]

Note that, in https://github.com/w3c/epubcheck/issues/1114 two possible ways forward were already proposed:

• https://github.com/w3c/epubcheck/issues/1114#issuecomment-621117293 by @murata2makoto
• https://github.com/w3c/epubcheck/issues/1114#issuecomment-621162483 by @rdeltour

[murata2makoto]

@iherman

The conformance rules of SVG1.1 require the presence of:

No, it does not. The rules automatically add the DTD before validation.

The DTD itself, as described in the relevant appendix, is making a heavy use of external entities (via its usage of DTD Modularization). As a consequence, per the current document, no conform SVG 1.1 file is valid for EPUB 3.3.

The DTD heavily uses external parameter entities, which never occur in documents. But if memory is correct, it does not define any parsed entities.

[iherman]

@murata2makoto

@iherman

The conformance rules of SVG1.1 require the presence of:

No, it does not. The rules automatically add the DTD before validation.

Checking it again, I stand corrected:-) The _validity process_ requires the presence or the addition of the DTD. So my sentence:

per the current document, no conform SVG 1.1 file is valid for EPUB 3.3.

should rather say:

per the current documents, conform SVG 1.1. files _may not be valid_ for EPUB 3.3

But, taking into account that tools produce SVG files with the DTD, and they have the perfect right to do so, the problem is not less severe for us. (I have just ran into a different application, called draw.io, that can produce nice diagrams in SVG and adds the DTD.)

The DTD itself, as described in the relevant appendix, is making a heavy use of external entities (via its usage of DTD Modularization). As a consequence, per the current document, no conform SVG 1.1 file is valid for EPUB 3.3.

The DTD heavily uses external parameter entities, which never occur in documents. But if memory is correct, it does not define any parsed entities.

I believe you are right. But the presence of the external parameter entities in the DTD is enough to raise a red flag for EPUB.

[murata2makoto]

@iherman

But the presence of the external parameter entities in the DTD is enough to raise a red flag for EPUB.

I do not understand. Since we do not allow external DTD subsets, external parameter entities are never encountered.

I think that you would like to use the output of SVG software that always embed references to the DTD at the W3C web site. But such references have caused troubles to the W3C web site in the past.

[iherman]

I think that you would like to use the output of SVG software that always embed references to the DTD at the W3C web site.

Indeed. Such software implementations are correct in doing so, they exist, and our users may want to use them. Adobe Illustrator is probably the most prominent, and certainly very important example. I do not think we can expect, say, Adobe Illustrator users to manually edit their generated SVG files by removing that DTD reference.

I.e., this is a very practical deployment issue.

But such references have caused troubles to the W3C web site in the past.

I do not know about the past, but the troubles are not in the present. My original issue came from the fact that most of the logos used in the W3C publications (e.g., all recommendations) in the past 3-4 years (at least) use SVG files that have those DTD-s, and all the Web Browsers accept those without further ado. Converting them into valid EPUB is not possibly without manually modifying the SVG files.

I _suspect_ that this issue never came up in EPUB land so far because Reading Systems tacitly accepted such SVG files (as the core Web engines accepted them). What brought it to the fore is the much stricter newer version of epubcheck that rejects those files. (This started the whole discussion in https://github.com/w3c/epubcheck/issues/1114 after all...)

I cc @rdeltour explicitly here, to call on the epubcheck implementers to give their opinion.

[murata2makoto]

@iherman

I still do not think that this is a problem. Yes, human users or software always have to remove external DTD subsets, if any. IDPF prohibited them, since parsing documents containing external DTD subsets has caused unnecessarily network access.

[mattgarrish]

Yes, human users or software always have to remove external DTD subsets, if any.

But why are we putting this requirement on authors and not reading systems? It's a reading system optimization pushed onto authors to carry out when a developer could suppress any resolution of DTD references. Isn't that pretty standard anyway, as it's been a while since I processed XML documents but W3C used to blacklist you if you tried to continuously resolve their DTDs.

(And just for the record, I don't see that this affects MathML. MathML can only be embedded in HTML so can't have its own DTD declaration.)

[iherman]

@iherman

I still do not think that this is a problem. Yes, human users or software always have to remove external DTD subsets, if any. IDPF prohibited them, since parsing documents containing external DTD subsets has caused unnecessarily network access.

Well, this is where we disagree. I believe it _is_ a problem that one of the official content format has an unnecessary restriction compared to its official definition.

But, actually, I do not understand your note, @murata2makoto. In your comment in https://github.com/w3c/epubcheck/issues/1114#issuecomment-706527683 you seem to say that there is no such problem, that content with such DTD is o.k., and the bug is by epubcheck. In which case I believe we have to add some explanation in the spec.

[murata2makoto]

@mattgarrish

But why are we putting this requirement on authors and not reading systems? It's a reading system optimization pushed onto authors to carry out when a developer could suppress any resolution of DTD references. Isn't that pretty standard anyway, as it's been a while since I processed XML documents but W3C used to blacklist you if you tried to continuously resolve their DTDs.

I might have agreed if this proposal was made when EPUB 3.0 was being designed. But the charter for EPUB 3.3 does not allow any changes that might destroy existing reading systems. If this DOCTYPE statement occurs in an SVG document, some existing EPUB 3 reading systems might try to fetch an external DTD subset. This causes network access, which is slow or unavailable.

[murata2makoto]

@iherman

In your comment in w3c/epubcheck#1114 (comment) you seem to say that there is no such problem, that content with such DTD is o.k., and the bug is by epubcheck. In which case I believe we have to add some explanation in the spec.

My comment might not be clear. But I'm saying that the DOCTYPE statement in SVG must not be allowed by EPUB 3.3.

[mattgarrish]

If this DOCTYPE statement occurs in an SVG document, some existing EPUB 3 reading systems might try to fetch an external DTD subset.

But the only thing that stops this from happening now is the idea that only perfectly valid publications are ever loaded into reading systems. That seems unlikely given that there are often ways to side-load content.

I agree we shouldn't cause backwards incompatibilities, but It would be interesting to survey whether moving the requirement to reading systems actually changes the way any have been implemented.

[iherman]

@murata2makoto the backward compatibility issue is something we should check. You say:

If this DOCTYPE statement occurs in an SVG document, some existing EPUB 3 reading systems might try to fetch an external DTD subset. This causes network access, which is slow or unavailable.

and I am not sure such reading system actually exist. What I would think is that reading systems, so far, did not care about the DTD-s (surely all RS-s that relied on one of the web rendering engines) and this whole issue came to the fore because epubcheck reports this issue although it did not care about it before. In other words, I would expect there were a number of, technically, _invalid_ EPUB content in circulation that were never flagged as such and that become invalid now.

In other words, it may well be that the practical deployment of EPUB content as well as RS-s were actually both ignoring a constraint in the document...

[murata2makoto]

@iherman

I am not sure such reading system actually exist.

It may well exist. If my memory is correct, a key member of IDPF argued that the DOCTYPE statement for SVG must not be allowed for this reason. This happened during the development of EPUB3.

[llemeurfr]

I am not sure such reading system actually exist.

It may well exist. If my memory is correct, a key member of IDPF argued that the DOCTYPE statement for SVG must not be allowed for this reason. This happened during the development of EPUB3.

I'm confident that the overwhelming number of RS do not care about validating publication resources, like web browsers DO NOT validate web resources.

[murata2makoto]

@llemeurfr

Sorry. That is irrelevant. Surely, valid SVG documents will be correctly handled by RSs that do not validate SVG. I'm saying that invalid SVG might be handled very differently by existing RSs.

[iherman]

I have just realized today that epubcheck also refuses an SVG file (ie, an XML file) which starts with the standard

<?xml version="1.0" encoding="UTF-8"?>

Is that indeed forbidden by the standard? Or is it an epubcheck issue?

[dauwhe]

I have just realized today that epubcheck also refuses an SVG file (ie, an XML file) which starts with the standard

<?xml version="1.0" encoding="UTF-8"?>

Is that indeed forbidden by the standard? Or is it an epubcheck issue?

What's the error message? This seems absurd.

[iherman]

That it is invalid. But, if this is _not_ an EPUB restriction then I should bring it to the epubcheck issues. I just wanted to check...

[dauwhe]

That it is invalid. But, if this is _not_ an EPUB restriction then I should bring it to the epubcheck issues. I just wanted to check...

How are you using the SVG? This SVG image validates fine in an EPUB (it's used as img src)

<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 15.1.0, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->

<svg version="1.1" id="box" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
     width="108px" height="108.391px" viewBox="0 0 108 108.391" enable-background="new 0 0 108 108.391" xml:space="preserve">
...

[mattgarrish]

Ya, I suspect you have some other issue. The SVG samples validate fine with xml declarations. Are you sure there isn't a curly quote or something in your actual source?

There's definitely nothing in the spec that forbids an xml declaration, although they're generally pointless because version 1.0 and UTF-8 encoding are defaults.

[murata2makoto]

BTW, non-validating XML processors might fetch external DTD subsets (and external parameter entities) for obtaining entity declarations (and telling which whitespace is significant). But I do not know how many such processors are used.

[iherman]

Ya, I suspect you have some other issue. The SVG samples validate fine with xml declarations. Are you sure there isn't a curly quote or something in your actual source?

There's definitely nothing in the spec that forbids an xml declaration, although they're generally pointless because version 1.0 and UTF-8 encoding are defaults.

The error report of epubcheck is as follows:

FATAL(RSC-016): epub-overview-33.epub/cover_image.svg(2,6): Fatal Error while parsing file: The processing instruction target matching "[xX][mM][lL]" is not allowed.
ERROR(RSC-005): epub-overview-33.epub/cover_image.svg(-1,-1): Error while parsing file: The processing instruction target matching "[xX][mM][lL]" is not allowed.

it is a stand-along SVG file, used as a cover. If I remove the processing instruction, then things are accepted.

I still wonder whether there is a problem somewhere in our spec that triggers this.

Cc: @rdeltour @danielweck

[mattgarrish]

I still wonder whether there is a problem somewhere in our spec that triggers this.

No, you're violating XML rules. If you put an xml declaration in a file, you can't have any whitespace before it (only a BOM is allowed before). Is there an extra hard return before your declaration by any chance?

For reference, see the prolog requirements in the XML spec.

[iherman]

No, you're violating XML rules. If you put an xml declaration in a file, you can't have any whitespace before it (only a BOM is allowed before). Is there an extra hard return before your declaration by any chance?

Ouch! Touché :-) This was the problem. Sorry for the noise...

[rdeltour]

I _suspect_ that this issue never came up in EPUB land so far because Reading Systems tacitly accepted such SVG files (as the core Web engines accepted them). What brought it to the fore is the much stricter newer version of epubcheck that rejects those files. (This started the whole discussion in w3c/epubcheck#1114 after all...)

I cc @rdeltour explicitly here, to call on the epubcheck implementers to give their opinion.

I don't think this is is new in EPUBCheck (but haven't doubled checked with older versions). The conformance criteria dates back to before EPUB 3.2, and was already implemented in EPUBCheck if I remember correctly.

What EPUBCheck 4.2.x brought are schema changes, which are orthogonal to this specific issue.

[mattgarrish]

I agree with @rdeltour here. Nothing has changed in a long time on this front. External identifiers are forbidden and have been since 3.0, so it's always been an issue. There really isn't a lot of ambiguity about what that means -- you can't reference external files, which would contain the external entities that would make some uses valid.

It's not been much of an issue as most people don't use SVG, and HTML has dropped the external subset from its default doctype so the discussion isn't even relevant to that format.

But in the interests of putting this issue to rest, I mostly agree with @llemeurfr comment in the other thread that we reformulate the restriction (but adding the RS ignoring is normative), so:

• we add a requirement to ignore external subsets to the reading system specification
• we change the authoring restriction to using entities declared in an external subset because reading systems will not resolve such entities

That would allow a doctype with an external identifier, but disallow content that relies on such.

[iherman]

@mattgarrish just to be sure I understand what you propose:

• we add a requirement to ignore external subsets to the reading system specification

You mean we add an extra item in EPUB 3.3. RS saying that external subsets must simply be ignored for all XML content? (At this moment there is no counterpart for XML in that document)

• we change the authoring restriction to using entities declared in an external subset because reading systems will not resolve such entities

When you say "change" you mean removing the second item in EPUB 3.3. Core, right?

That would allow a doctype with an external identifier, but disallow content that relies on such.

That sounds o.k. to me, but that does not answer https://github.com/w3c/publ-epub-revision/issues/1338#issuecomment-706999165.

[mattgarrish]

That sounds o.k. to me, but that does not answer #1338 (comment).

I'm not following. Why does it matter if the DTD makes use of entities, and how is that problematic for authoring?

If the entities only exist for parsing the DTD, and reading systems don't validate, then is the issue just theoretical?

[mattgarrish]

Or put another way, we're banning content entities in SVG documents (or any other format) defined in the external subset. We don't care what goes on within the external DTD itself.

[murata2makoto]

But in the interests of putting this issue to rest, I mostly agree with @llemeurfr comment in the other thread that we reformulate the restriction (but adding the RS ignoring is normative), so:

I disagree.

First, will this change cause no problems to existing RSes or other programs? Unless we agree that there are no real problems, I don't think that this change should be done as part of 3.3

Second, I do not think that your proposal takes into consideration parsed entities defined in external parameter entities defined in internal DTD subsets.

[mattgarrish]

First, will this change cause no problems to existing RSes or other programs?

But this is the problem I have with the current approach. It's already a flaw in existing reading systems if they aren't accounting for external identifiers. Nothing stops authors from ignoring the requirement, especially since content can be side loaded. Trying to block authoring of bad things through validation isn't an effective approach.

I do not think that your proposal takes into consideration parsed entities defined in external parameter entities defined in internal DTD subsets.

If the reading system is required not to resolve external identifiers, isn't this blocked by default?

We need to work on the wording of what entities are blocked, of course. The above wasn't a wording proposal for the specification, only an attempt to address what is at issue.

[wareid]

I am wondering if this is a case of the specification not being grounded in the reality of reading systems. Talking to @rakutenjeff to confirm, reading systems are not actually checking external entities. I know this might not be true for all reading systems (paging @GarthConboy @llemeurfr and other RS folks) but for reading systems built with/upon browser engines, the importance is not on the referenced entities but on the well-formedness of the code in question. If the engine in question can parse SVG, it will be displayed, if it cannot, it won't.

This is particularly true if the EPUB is sideloaded or used offline, there would technically be no way for the reading system to check the entities in that situation, and then we only have two options, don't display the content, or display what we're capable of.

This, as expected, gets into testing, because I would not be surprised to find that many of our MUST/MUST NOTs are likely handled or ignored by reading systems. There's a number of "failures" in EPUBCheck that have zero effect on the rendering of an EPUB. Much like browsers gracefully handle X/HTML in its many permutations, most reading systems do as well. Reading systems tend to fall over more when they have to account for RS features that use HTML/CSS but can be done in multiple ways (think footnotes).

In short, I think I agree with Laurent, Romain, and Matt's compromise, but I did want to provide this extra context.

[mattgarrish]

Technically, we don't even have to bother banning entities in the external subset if external identifiers aren't resolved by reading systems, but it's useful to have a restriction so authors don't silently stumble on the lack of support. Unfortunately, most people will figure out what epub allows through epubcheck rather than reading specifications.

(And I agree there are numerous pointless failures that come from strict validity checking; many attribute values come to mind. It's helpful to know what is wrong with a publication, but epubcheck has inadvertently also gone from a validation tool for authors to a barrier to distribution.)

[murata2makoto]

I sent an email about this topic to an EPUB ML in Japan, which was created by Takami-san.

[Doktorchen]

SVG has no external entities, but internal within the doctype declaration can be useful sometimes.
But these are completeley within the document, no need for programs to look for resources outside the book.

But HTML5 has lots of entities defined somehow in prose now.
Programs need to have a list to resolve them similar to the few XML entities.
This might work for MathML as well.
Do authors for MathML in EPUB really rely on entities? Usually you get already an error message by the XML processor, if you preview your document in a browser before you put it into the EPUB-ZIP-archive.

If doctype declarations are handled within EPUB similar to the HTML5 approach as decoration, this should work. Only internal subsets of entities need to be regocnised for SVG, MathML and XHTML or other XML-formats.

[rdeltour]

@mattgarrish

Or put another way, we're banning content entities in SVG documents (or any other format) defined in the external subset.

We should be careful with how we deal with entity references in HTML and MathML. As mentioned by @Doktorchen, a conforming XHTML parser is supposed to dereference the usual HTML public identifiers to a DTD containing a list of the character reference names allowed in HTML. A blanket ban of all entity references may be too restrictive for HTML/MathML.

[mattgarrish]

a conforming XHTML parser is supposed to dereference the usual HTML public identifiers to a DTD containing a list of the character reference names allowed in HTML

Doesn't that only apply to the old public identifiers listed in that section? I thought named entities in XHTML were essentially a bad thing since there is no DTD anymore. Isn't that what this prose is getting at:

This specification does not define a public or system identifier, nor provide a formal DTD.

According to XML, XML processors are not guaranteed to process the external DTD subset referenced in the DOCTYPE. This means, for example, that using entity references for characters in XML documents is unsafe if they are defined in an external file (except for <, >, &, " and ').

epucheck throws errors if you use named entities, so if it's the case that they're allowed we're being too strict.

[Doktorchen]

An HTML5 parser resolves such HTML5 entities (no XML parser), presumably those from the HTML5 subset of MathML as well.
Some EPUB viewers seem to use an HTML5 parser instead of a proper XML parser, what can result in problems, because the DOM can be different (problematic for some CSS decoration or scripting).
Therefore, what you get as audience depends on the design of the EPUB viewer.

On the one hand I think, it should be no problem for authors or authoring tools to remove the doctype from documents as required for EPUB 3.0.
But if in a newer version of EPUB those become allowed as decoration, it would be a useful addition to require such a list of known entities from EPUB viewers just to avoid surprises for some authors or readers.
Because the W3C DTDs for SVG 1.0, 1.1, 1.2 have no entities, MathML can be assumed to be stable, the list of entities in HTML5 seems to be fixed, EPUB viewers will need only a small list of entities to simulate a look into the DTD to resolve entities. The EPUB version can precisely define, which DTDs and doctypes are assumed to be known, others are not allowed, because they require something like an HTTP(S) request, what is against the idea of a separated digital book.

[mattgarrish]

It seems epubcheck is technically being overzealous in reporting named entities, but the HTML specification doesn't require entity resolution even in providing the stripped-down entity declaration file. It's probably a good thing, but poses another interesting question of how far epubcheck needs to strictly validate. There is the potential for bugs if we loosen this restriction, given that it's not a requirement to resolve them even against that file, but who takes ownership for avoiding these grey areas?

We're not losing anything by continuing to not allow externally-defined entities, at any rate. There are mappings for all to numeric entities, which is what I'd expect most people are using now given the barrier epubcheck presents.

[Doktorchen]

Using UTF-8 I simply copy glyphs, which are not acessible from keyboard.
Mainly for MathML named entities are useful - who has in mind all these numbers or has text modules for special glyphs in the text editor?

Some tools generating still EPUB 2 books seem to use sometimes entities like nbsp, but this is ok for XHTML1.1, could this be another barrier so switch to EPUB3 for them? Should bei simple to replace within the source code of such programs, therefore maybe more a psychological barrier.

[mattgarrish]

So looking again at where we currently are in this discussion, I don't think we even need to say anything normative about authoring. The fixes might only entail:

• requiring that reading systems not resolve external identifiers (the HTML workaround would not be affected by a general ban, since it's not actually resolving an external identifier)
• adding a note like the HTML one that the use of named entities defined in external subsets is not advised because reading systems are not guaranteed to implement the HTML workaround and will not resolve them in other formats

[iherman]

So looking again at where we currently are in this discussion, I don't think we even need to say anything normative about authoring. The fixes might only entail:

• requiring that reading systems not resolve external identifiers (the HTML workaround would not be affected by a general ban, since it's not actually resolving an external identifier)
• adding a note like the HTML one that the use of named entities defined in external subsets is not advised because reading systems are not guaranteed to implement the HTML workaround and will not resolve them in other formats

So what this means is that named entities listed in the HTML spec are allowed, because they are listed explicitly rather than via an external identifier, right?

I think that we ought to have a note referring to that list (and making clear that we rely on the official HTML standard on that). We can add a note that even the usage of those is not really advised if it is possible to use Unicode instead (there are many, if not all, characters like that in that table), but there are obviously named entities that are very handy (one I use fairly often is   for a non-breaking space).

[mattgarrish]

So what this means is that named entities listed in the HTML spec are allowed, because they are listed explicitly rather than via an external identifier, right?

Right, they've defined the entities in a stripped down DTD and expect the user agent to resolve them against that without having to go out to an external subset. Per the note in the parsing section of the HTML spec:

"This is not strictly a violation of XML, but it does contradict the spirit of XML's requirements. This is motivated by a desire for user agents to all handle entities in an interoperable fashion without requiring any network access for handling external subsets. [XML]"

I think that we ought to have a note referring to that list (and making clear that we rely on the official HTML standard on that). We can add a note that even the usage of those is not really advised if it is possible to use Unicode instead

Agree. It should be up to the author if they want to take a chance on them. I expect all the major browsers handle the named entities even in xhtml, but it's still a risk reading systems won't (and your page would probably not render in such cases, making it at least noteworthy). It's not something we need to normatively restrict, though.

[rdeltour]

@mattgarrish

a conforming XHTML parser is supposed to dereference the usual HTML public identifiers to a DTD containing a list of the character reference names allowed in HTML

Doesn't that only apply to the old public identifiers listed in that section? I thought named entities in XHTML were essentially a bad thing since there is no DTD anymore.

Yes, that applies only if the old public identifiers are used. But if we're loosening the spec to allowing those, then this section starts to apply to EPUB 😊

epucheck throws errors if you use named entities, so if it's the case that they're allowed we're being too strict.

Yes, because as we currently disallow external identifiers, then theses entities are not defined anywhere, so we report them. But that may change if we stop disallowing said external identifiers.

It seems epubcheck is technically being overzealous in reporting named entities, but the HTML specification doesn't require entity resolution even in providing the stripped-down entity declaration file. It's probably a good thing, but poses another interesting question of how far epubcheck needs to strictly validate.

I believe EPUBCheck is correct here: if no external subset is referenced in the doctype, no external entities are
declared and no named character reference are allowed (beyond the 5 predefined in XML). This is what EPUBCheck currently checks.

@mattgarrish @iherman

So what this means is that named entities listed in the HTML spec are allowed, because they are listed explicitly rather than via an external identifier, right?

Right, they've defined the entities in a stripped down DTD and expect the user agent to resolve them against that without having to go out to an external subset.

I think there's a misunderstanding: the predefined named characters table is used by HTML parsers only. The "HTML workaround" only applies to HTML, not XHTML.
For XHTML, references to these named characters can be used only if a doctype has been declared. The spec trick consisting in making all the well-known XHTML public identifiers point to the same DTD allow user agents to use a predictable offline version of the external subset, without requiring network access.

To summarize:

• if HTML becomes allowed in EPUB
  
  
  
  • named character references are allowed as per the "Parsing HTML documents" spec rules. These are not strictly "external entity references". We'd obviously want to allow them in EPUB.
  
  
• for XHTML
  
  
  
  • when no doctype is set, or the doctype has no public identifier: no external entity reference are allowed. This is correct per XML and XHTML specs, and is what EPUBCheck currently implements.
  
  
  • if we start allowing well-known public identifiers in a doctype, then we need to allow external entity references, following the rules defined in XML and the "Parsing XML documents" section of the HTML spec.
  
  

[murata2makoto]

@rdeltour

I do not understand what you mean by "external entity references". In my understanding, they are references to entities defined by

<!ENTITY entityname [PUBLIC "public-identifier"] SYSTEM "system-identifier">

The content of this external parsed entity is the result of dereferencing the public identifier or system identifier.

So, '&lt;' is not an external entity reference.

Perhaps, you are talking about references to parsed entities in general?

[murata2makoto]

As a member of the XML WG, I feel guilty about this mess. We should have provided a mechanism for declaring entities without using DTDs. But we were trapped by compatibility requirements with SGML. Rather, we indulged in creating interesting extensions...

[Doktorchen]

The SVG 1.1 recommendation has an example for a typical use of a document internal entity definition in the section about preserveAspectRatio:
https://www.w3.org/TR/SVG11/coords.html#ExamplePreserveAspectRatio

If this entity definition would have been inside the http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd it would be an external definition - with the need to know the content of the DTD file somehow to resolve the entity, what causes problems, if the doctype does reference a DTD outside the EPUB, causing other trouble with fallback requirements, if it references a DTD within the EPUB.
Therefore currently in practice of EPUB 3 authors cannot reference any DTD file, only a doctype with an internal entity definition in the same document, the entity is used. But this is of limited use, because often one may want to reuse content in different documents having only one definition in the same EPUB.

Both for SVG and expecially poetry, lyrics, nonlinear stories written in XHTML it would be pretty useful for advanced authors to be able to define such entities in own DTDs (or similar constructs) within the EPUB archive to be able to reuse content fragments in several documents to reduce the file size.
SVG has already some constructs like the use element for such an application, unfortunately (X)HTML still has no such a functionality - because for an own DTD within an EPUB as extensions there would be a requirement to provide fallbacks for such DTD documents.

A generic mechanism already in XML to reuse content in different documents (without using DTDs) would be even better for authors instead of a specific solution in EPUB.

[mattgarrish]

I believe EPUBCheck is correct here: if no external subset is referenced in the doctype, no external entities are
declared and no named character reference are allowed (beyond the 5 predefined in XML).

Is that what the XML parsing section is saying? Every time I read it I get more confused. I read it that user agents should use the entities declared in the reference DTD for resolving external entities from this:

This specification provides the following additional information that user agents should use when retrieving an external entity: the public identifiers given in the following list all correspond to the URL given by this link.

The dependence on an external identifier actually being specified in the doctype comes after:

Furthermore, user agents should attempt to retrieve the above external entity's content when one of the above public identifiers is used, and should not attempt to retrieve any other external entity's content.

But this "should" is confusing with the first one. Does it mean the user agent should limit the entities to only those defined in the referenced DTD when a specific identifier is specified, or is it restricting the preceding statement so that entities shouldn't be resolved unless an old doctype is declared?

[iherman]

Trying to be pragmatic, I made some tests, using a chromium based browser (Edge) as the browser engine. AFAIK, a large majority of RS-s are based on that engine, i.e., its behaviour is relevant.

• Just to be sure, the following HTML file (i.e., test.html) works as expected (i.e., displays the ± character):

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" dir="ltr">
<head>
  <meta charset="utf-8"/>
  <title>Test</title>
</head>
<body>
  <ul>
    <li>&plusmn;</li>
  </ul>
</body>
</html>

The following is a syntactically correct XHTML file:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" dir="ltr">
<head>
  <meta charset="utf-8"/>
  <title>Test</title>
</head>
<body>
  <ul>
    <li>&plusmn;</li>
  </ul>
</body>
</html>

but it does _not_ work, it displays the error message: "error on line 10 at column 44: Entity 'plusmn' not defined".

I.e., a different DTD must be used. This, for example, works as expected:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" dir="ltr">
<head>
  <meta charset="utf-8"/>
  <title>Test</title>
</head>
<body>
  <ul>
    <li>&plusmn;</li>
  </ul>
</body>
</html>

(I was a bit hopeful that the chromium parser would be pragmatic and would do the mapping for the known entities like HTML does, but it does what the standard requires :-(

_However_, while chromium does accept the third version, it is actually not a valid XHTML5. The HTML5 spec only talks about the short doctype <!DOCTYPE html>; the DTD used above is XHTML 1.1., _which is an outdated specification_.

All this is to say that the proposal of @rdeltour in https://github.com/w3c/epub-specs/issues/1338#issuecomment-713388665 is correct except that, for XHTML, the only way we could get entities (like &plusmn) into the content is not only to allow the usage of DTD-s but to allow the usage of an _outdated specification_ for a content document. I do not think this is really something we should do. I.e., as far as I am concerned, the only choice in this respect is:

1. we allow HTML5 in EPUB 3.3 (alongside XHTML5) and we clearly tell authors that they _MUST_ use HTML5 if they want to use entities like &plusmn; or
2. entities like &plusmn are disallowed in EPUB 3.3 that relies on XHTML5 only (which I believe is what we have now)

I really do not know how long we can keep up _not_ allowing HTML5, to be honest...

Note that this still leaves the original issue of SVG DTD-s open.

---

(B.t.w.: when I tried to run the last version through validator.nu with the option of "load external entities", the validator timed out :-(

[mattgarrish]

when I tried to run the last version through validator.nu with the option of "load external entities", the validator timed out

Ya, I ran into the same. But the fact it appeared to just be a choice is what made me think it was a viable option to resolve with just the HTML5 doctype. Ah well...

[rdeltour]

@murata2makoto

I do not understand what you mean by "external entity references".

Sorry, I should have said more precisely "reference to a parsed entity declared in the external subset" 😊

[rdeltour]

@iherman

_However_, while chromium does accept the third version, it is actually not a valid XHTML5.

Yes, good point… which makes my last bullet (in the earlier comment) a moot point:

• if we start allowing well-known public identifiers in a doctype, then we need to allow external entity references, following the rules defined in XML and the "Parsing XML documents" section of the HTML spec.

of course if we stick to conforming XHTML5, then these legacy doctypes are not conforming, so we don't have to say anything about content references, which are not conforming either.
Of course, a RS will still have to use the DTD as per the HTML spec XML parsing rules when parsing such invalid content (as they would currently do when using a browser engine), but our EPUB spec doesn’t have to say anything about that!

[mattgarrish]

This is starting to sound like a no-win situation:

• allow authoring with external identifiers and all that entails, but surprise and annoy authors when reading systems ignore the entities or crash
• allow certain external identifiers and make reading systems have to locally resolve them, which ends up the opposite of how the web works for xhtml
• continue to ban external identifiers, and some percentage of SVG users are inconvenienced by stripping doctypes
• wait for html serialization, as @iherman says, so if you like named entities you can at least use them there (and then I guess we get into when does an SVG parsed by HTML have to conform to XML rules... :)

[iherman]

This is starting to sound like a no-win situation:
[...]

• allow certain external identifiers and make reading systems have to locally resolve them, which ends up the opposite of how the web works for xhtml

I do not think we should do this for XHTML. But allowing for the standard SVG DTD-s to be present (and ignoring them) is still a possibility, and I do not think it would affect any content. It is ugly to do that, but I still believe we should do it.

[...]

• wait for html serialization, as @iherman says, so if you like named entities you can at least use them there (and then I guess we get into when does an SVG parsed by HTML have to conform to XML rules... :)

I think this discussion has just highlighted the fact that we should give a high priority looking at the issue of accepting HTML5 as a content document.

[mattgarrish]

It is ugly to do that, but I still believe we should do it.

Which I assume becomes....

• Reading Systems must ignore external identifiers
• Authors may include the following external identifiers in doctype declarations ...

?

[iherman]

It is ugly to do that, but I still believe we should do it.

Which I assume becomes....

• Reading Systems must ignore external identifiers
• Authors may include the following external identifiers in doctype declarations ...

?

No, that is not what I meant. The only thing is that we explicitly bring in an exception clause for the standard SVG and maybe for MathML DTD-s. Otherwise we go for HTML5...

[iherman]

This issue was discussed in a meeting.

• RESOLVED: remove bullet “External identifiers MUST NOT appear in the document type declaration [XML].” Add reading system conformance saying that reading systems SHOULD NOT validate against DTDs.
  
  View the transcript

  
  Problem with XML content requirements and external entities: make MathML and SVG1.1 invalid
  
  Dave Cramer: https://github.com/w3c/epub-specs/issues/1338
  
  Dave Cramer: https://github.com/w3c/epubcheck/issues/1114
  
  Dave Cramer: External entities.
  
  Dave Cramer: https://www.w3.org/publishing/epub32/epub-spec.html#confreq-xml-extmarkupdecl
  
  Dave Cramer: Lots of XML statements in specs:
  
  External identifiers MUST NOT appear in the document type declaration [XML].
  
  Dave Cramer: This is a problem.
  
  … Breaks normal things — like inline SVG with DOCTYPE.
  
  … I think that’s bad.
  
  … Forcing folks to edit machine created content seems like a waste of effort.
  
  Brady Duga: Agree with the summary and history
  
  Dave Cramer: EPUB RS’s are not validating SVG’s with DOCTYPE of against that.
  
  … Should not expect an RS to run an validator.
  
  … Hope RS’s don’t need to be validating XML parsers.
  
  … Maybe just say “ignore DOCTYPE’s” — don’t use content that depending on processing DTD features
  
  Brady Duga: Stunned if anybody other than epubcheck is doing said validation.
  
  Wendy Reid: +1 we do this too
  
  Dave Cramer: Assuming WV’s doing to such validation.
  
  Brady Duga: What is this XML of which you speak?
  
  … RS’s likely display HTML, not XML, content.
  
  … Natively not XHTML in RS’s.
  
  Wendy Reid: +1
  
  Dave Cramer: Principle in HTML that we should try to match reality — maybe we should do the same in here?
  
  Zheng Xu (徐征): +1 to not restrict
  
  Dave Cramer: Resolve tonight restricting DTDs in XML documents that are part of an epub.
  
  Garth Conboy: I think we should try it!
  
  Wendy Reid: In line with what Dave has say it — try it for now. It will come done to testing too.
  
  … Can that bridge when we come to it.
  
  Proposed resolution: remove bullet “External identifiers MUST NOT appear in the document type declaration [XML].” Add reading system conformance saying that reading systems SHOULD NOT validate against DTDs. (Dave Cramer)
  
  Garth Conboy: +1
  
  Wendy Reid: +1
  
  Dave Cramer: RS’s have no validation requirement against DTDs that may be found.
  
  Shinya Takami (高見真也): +1
  
  Brady Duga: +1
  
  Yu-Wei Chang (Yanni): +1
  
  Ben Schroeter: +1
  
  Wendy Reid: If you use inline SVG’s out of ID (and other commercial tools) you won’t get unexpected errors.
  
  Matthew Chan: 0
  
  Wendy Reid: Found W3C listing DTDs that should be used; seems strange to prohibit.
  
  Laura Brady: +1
  
  Wendy Reid: No objection; relax restriction on external identifiers in DTDs.
  
  Resolution #2: remove bullet “External identifiers MUST NOT appear in the document type declaration [XML].” Add reading system conformance saying that reading systems SHOULD NOT validate against DTDs.
  
  

[iherman]

Transferring a comment (with permission) raised separately to the latest resolution:

This is not good enough. Non-validating XML processors may well fetch external DTD subsets. I'm quite sure that it is possible to configure XML parsers so that they fetch external DTD subsets without validation.

So, please replace "validate against DTDs " by "retrieve external DTD subsets or external parameter entities".

I would like to mention external parameter entities because it is possible to decrease external parameter entities in internal DTD subsets.

Cc @murata2makoto @wareid @shiestyle @dauwhe

[iherman]

I agree with @murata2makoto

[mattgarrish]

Agree, as validation and resolution of entities are not linked. A non-validating XML parser can still resolve external entities, which exposes it to XXE attacks.

[shiestyle]

I don’t agree with @murata2makoto.

We should request that reading system should avoid both validating and retrieving.

So I think we should add @murata2makoto ’s proposal, not replace.

[murata2makoto]

@shiestyle

Validation without fetching external DTD subsets is simply not possible. The only (highly unlikely) exception is that the internal DTD subset has all declarations copied from the external DTD subset. Moreover, validating XML processors are required to fetch external DTD subsets. The following sentence is quoted from the XML Rec.

To accomplish this, validating XML processors must read and process the entire DTD and all external parsed entities referenced in the document.

[shiestyle]

@murata2makoto

If you are right (validating includes fetching), just 'validate' is enough to say?

[murata2makoto]

@shiestyle

No. Fetching without validation is possible and implemented by some XML processors. Fetching allows the use of default values, entities, and even detection of significant/insignicant whitespace.

[shiestyle]

@murata2makoto

You said that validation without fetching is impossible and then I explained if we stop validating, we can stop fetching in our case.

Even if fetching without validation is possible technically, how bad effect will bring to EPUB processing?
After fetching or failed to fetch, by saying that we should not 'validate' in our spec, the result should be ignored and will not bring bad effect, I think.

In order to avoid discussing validating and fetching, using 'ignore external entities' may be another solution.

Anyway, we should go forward to resolve issues as soon as possible by choosing simple and easy to understand words to satisfy our objectives.

[murata2makoto]

@shiestyle

Even if fetching without validation is possible technically, how bad effect will bring to EPUB processing?

Delay and a well-known security risk. Here is the last paragraph of RFC 7303 (XML Media Types).

Note that it is also possible to construct XML documents that make
use of what XML terms "[XML-]entity references" to construct repeated
expansions of text. Recursive expansions are prohibited by [XML] and
XML processors are required to detect them. However, even non-
recursive expansions may cause problems with the finite computing
resources of computers, if they are performed many times. For
example, consider the case where XML-entity A consists of 100 copies
of XML-entity B, which in turn consists of 100 copies of XML-entity
C, and so on.

I continue to think that my proposed wording is good.

[shiestyle]

@murata2makoto

Delay and a well-known security risk. Here is the last paragraph of RFC 7303 (XML Media Types).

I understand the risks but I think it's not critical bad effect for EPUB processing because the risks will not always prevent users from reading contents.

In my understanding, the objective of this issue is to avoid that EPUBCheck or reading system will mark as ERROR when XML content has external entities.
I mean EPUBCheck and reading system may just confirm whether external entities exist or not although XML spec's 'validating' includes fetching.
So stopping only fetching is not enough and stopping 'validating: may be different meaning from @murata2makoto's recognition' is enough and then I proposed to use both.

Thank you for your comments and concerns and we will discuss and decide what expression should be in the working group.

[murata2makoto]

@shiestyle

I am going to concentrate on one issue here. Please use the terminology as defined in the XML recommendation. You appear to use "external entities" differently. Entities such as &lt; are not external entities. The original definition of external entities is here. You also use "validation" differently. Although the XML rec is not clear as it should be, validation is defined in Clause 5.1. Inventing new definitions for terms defined in other documents is not useful for anybody.

@rdeltour

By the way, "reference to a parsed entity declared in the external subset" is not correct either. How about entity declarations in internal DTD subsets, internal parameter entities, and external parameter entities? Are you talking about both internal and external parsed entities?

[Doktorchen]

Why not to say in the EPUB recommendation, if authors want to have a doctype with a DTD (with entitities beyond those from XML), the DTD has to be included completely in the EPUB archive (and the doctype has to point on this?). Obviously then EPUB has to allow DTDs as core media types.
Additionally one can recommend authors to use simply the characters instead of related entities, as far as available in UTF-8? This applies only, if the entities represent only simple characters.
For the reuse of content with own entity definitions in the DTD of course is a more useful, important application in contrast of using entities for simple characters.

[murata2makoto]

@Doktorchen

You probably mean that if an SVG content document references an external DTD subset and directly references external parameter entities, they are required to be in the EPUB document. I don't think people like this approach since SVG editors do now create such SVG documents.

While reading your comment, I spotted another problem with the proposed resolution. I will write a separate reply.

I kind of like your idea of using characters directly. We should also use character references (&#x3000, for example). But are users willing to do so when they embed MathML within XHTML5? I don't think so. I'm afraid that I do not have a good idea here.

[murata2makoto]

The proposed change is

RESOLVED: remove bullet “External identifiers MUST NOT appear in the document type declaration [XML].” Add reading system conformance saying that reading systems SHOULD NOT validate against DTDs.

This change allows an SVG document to define external parsed entities in the internal DTD subset. EPUB 3.2 does not allow such SVG documents. Is this change intentional or by chance?

Meanwhile, nothing in EPUB 3.2 prevents authors from defining internal parsed entities in internal DTD subsets. This is fine and I see no problems.

The standard SVG DTD (to be precise, the external DTD subset and external parameter entities) of SVG 1.1 does not define any internal or external parsed entities.

[Doktorchen]

Concerning SVG, I think, one has to ask the SVG WG, respecitively the authors of the SVG 1.1 second editon, why there a two contradicting (?) requirements in the recommendation.
(I was only as an invited expert member of the WG for SVG tiny 1.2., therefore I only noticed the historical ideas about these SVG 1.1.2 efforts.)

In https://www.w3.org/TR/SVG11/intro.html#NamespaceAndDTDIdentifiers
it is noted: 'While a DTD is provided in this specification, the use of DTDs for validating XML documents is known to be problematic. In particular, DTDs do not handle namespaces gracefully. It is not recommended that a DOCTYPE declaration be included in SVG documents.'

But indeed https://www.w3.org/TR/SVG11/conform.html
the implementation requirements seem to say the usage of a specific doctype is needed.

As far as I remember the discussion, there first was the intended approach, to require no doctype anymore as for SVG tiny 1.2 (because DTDs do no cover the needs of SVG anyway - one needs more detailled information to validate especially attribute values).
Presumably they simply missed to use something similar to the SVG tiny 1.2 requirements for SVG 1.1 second edition.

(Document) internal entitiy definition within the doctype of an SVG are in practical use.
I think, beyond this the usage of DTDs other than this 'decorative' doctype refering to the official W3C DTD does practically currently not appear and has limted use for SVG due to the use element to reuse content in different documents.
Unfortunately (X)HTML has no functionality like this use element to reuse content in different documents, therefore for authors it would be more an XHTML use case having an own DTD defining entities within the EPUB archive.
Maybe there is a similar use case for MathML.
Just mapping characters to entities is mainly a historical use case for times, before UTF-8 was established, one can forget/neglect in EPUB.

Concerning wording in a new EPUB version, presumably we can assume, that in these years authors (like me) any many programmers of user-agent, viewers, reading-systems are not very familiar with DTDs or related wording anymore.
Therefore it will presumable already help to explain the three different options:
a) Defining an entity within the doctype construct within the XML document, that uses the entity (XML/DTD wording: internal subset)
b) Defining an entity within an own DTD document within the same EPUB archive, referenced with doctypes in each XML document, that makes use of such an entity.
c) Pointing with a doctype to a DTD outside the EPUB archive (typically the official W3C DTDs).

b) and c) are both from the XML/DTD point of view external references with keywords 'SYSTEM' or 'PUBLIC' (external subset, right?).

For EPUB case c) is only of limited use (for authors), EPUB 2 allows this, EPUB 3.0 not.
If XML does not allow, that parers/validators keep in mind well known DTDs from usual official W3C sources, further variants of EPUB should follow the EPUB3.0 path to disallow to 'decorate' documents in EPUB archive with such doctypes pointing to resources outside the archive.

Obviously, almost no author will include copies from official W3C DTDs in an EPUB archive just to allow validation, implicating the need to direct the doctype reference to this book internal DTD document.
But if some really have a desire to do so - why not?
If they have such a strong desire, they will have enough motivation to get it right. ;o)

For a) there are examples around in tutorials, how to do this, but maybe one can add a meaningful example for both a) and b) as non normative examples within a new EPUB variant, in case, DTDs still matter somehow in practice for EPUB, if not, keep the EPUB 3.0 direction to disallow these cases b) and c).
Case b) would allow a few advanced authors to reduce the file size of their books, not more.
There is no value in having only decorative code as the 'HTML5 doctype' line in a document source code.

[mattgarrish]

And for the record, EPUBCheck is not a reading system and only enforces content authoring requirements.

Saying reading systems must not validate against DTDs has no bearing on it as it generally doesn't use doctypes or DTDs for validation (the schemas are mostly all in RelaxNG compact for 3.0 and the media type triggers the validation). We only need to be clear what authors are allowed to do while ensuring we don't expose reading systems to known attack vectors.

(I doubt reading systems actually validate content against DTDs, either, and I would hope they already all ignore external identifiers as you can't rely on content following the rules.)

[mattgarrish]

I'm wondering if one way out of this particular problem is to move this to a reading system security section.

We don't ban external identifiers in content and we don't tell reading systems how to handle DTD references, but instead explain the dangers of resolving external entities and why, if a reading system chooses to resolve entities or validate any xml format using a document-defined doctype, it should only rely on a local catalog and not retrieve any untrusted sources.

[murata2makoto]

It appears that people need more examples. I created #1354 and #1355.

[llemeurfr]

We don't ban external identifiers in content and we don't tell reading systems how to handle DTD references, but instead explain the dangers of resolving external entities and why, if a reading system chooses to resolve entities or validate any xml format using a document-defined doctype, it should only rely on a local catalog and not retrieve any untrusted sources.

We must be careful that RS developers don't develop nor are usually able to configure precisely the underlying XML parser of their XHTML resource handler (i.e. a browser engine in most cases). Therefore a reading system does not "choose" to resolve entities or not. It can only choose to validate the XML resource as a pre-processing (and again none would do that IMO) then display the resource. If the non-validating XML parser fetches external entities prior to display, the RS developers won't be able to do much about that. If there is an attack because of that, developers won't even know it can exist. If the RS is offline, external entities cannot be fetched and the content is badly rendered, users will suffer and RS developers won't be able to do much neither.

The solution must be so that EPUB authors can have as much flexibility as possible without causing harm to users, with non-validating RSes either online or offline.

[mattgarrish]

Therefore a reading system does not "choose" to resolve entities or not.

I don't think we're that far apart. That's why I'm suggesting we only note this in a security considerations section without making normative statements that potentially aren't followable. I expect browser cores already do the right thing, but it is still a security consideration.

It's like banning external identifiers in authoring. It doesn't fully solve the problem, and any vendor could continue to ban these at ingestion if they really don't want them. But not using them is something we advise as a best practice (which could be part of where core media types go).

We're arguably making a lot out of an issue that's never been questioned as problematic. I'd be surprised if any appreciable number of publishers use/define entities in a DTD subset, let alone the external, and I doubt anyone will rush to begin if we remove the restriction.

[llemeurfr]

From the start of this thread, it seems that two different notions are conflated:

1. external ids in doctype declarations, which are not dereferenced in a non-validating XML parser and therefore not harmful for RSes.
2. external ids in external entities, which cause problems for RSes even if non-validating.

Or maybe we conflate "external identifier" and "external entity". And also "document type declaration" and "document type definition".

The current contraint in the spec (_External identifiers MUST NOT appear in the document type declaration_) forbids both. A sentence like "External entities MUST NOT be defined in a document type definition" would only forbid the second notion, which would be ok (especially is we also state the RS should/must not validate content)

[llemeurfr]

I'd be surprised if any appreciable number of publishers use/define entities in a DTD subset

Just a note: I tried to include internal entities in the html doctype of a resource using Sigil and BlueGriffonEPUB : Sigil made a mess of the XHTML doc at save time, and BlueGriffon made another type of mess. So I tend to agree with you : entities in content documents must be really rare.

[murata2makoto]

@mattgarrish

I'm wondering if one way out of this particular problem is to move this to a reading system security section.

I strongly believe that requirements on data are always preferrable since data conformance is very enforceable (e.g., by epubcheck). Moving things to requirements on applications is always less satisfactory, although it is sometimes inevitable. We should first make clear what is allowed as an SVG or MathML document.

We're arguably making a lot out of an issue that's never been questioned as problematic. I'd be surprised if any appreciable number of publishers use/define entities in a DTD subset, let alone the external, and I doubt anyone will rush to begin if we remove the restriction.

Some publishers might if they have a reason. We shouldn't easily allow what is not needed. The currently proposed wording allows what is not needed.

[murata2makoto]

@llemeurfr

From the start of this thread, it seems that two different notions are conflated:

1) external ids in doctype declarations, which are not dereferenced in a non-validating XML parser and therefore not harmful for RSes.

External identifiers of external DTD subsets may well be referenced by a non-validating XML processor. This is one of my points. See Clause 5 of the XML Rec.

2) external ids in external entities, which cause problems for RSes even if non-validating.

I do not see any reasons to allow such external (parsed) entities in the case of SVG.

Or maybe we conflate "external identifier" and "external entity". And also "document type declaration" and "document type definition".

Yes, some terms are not easy to use correctly. I checked the definitions of these terms for discussions around this issue.

The current contraint in the spec (External identifiers MUST NOT appear in the document type declaration) forbids both. A sentence like "External entities MUST NOT be defined in a document type definition" would only forbid the second notion, which would be ok (especially is we also state the RS should/must not validate content)

I would like to continue to disallow 2) in EPUB 3.3. I can live with small changes to 1) if they are needed to use the output of SVG editors.

[iherman]

This thread is becoming quite confusing (for me at least), and I would love to see it converge somehow. Also, as it was mentioned several times, many of the situations in the thread are hypothetical as far as current EPUB deployments are concerned, because very few of our authors make use explicitly of remotely complex XML facilities; the only real practical issue seems to be that some tools would dump some DTDs into their output that make content unusable. With the future push to move away from XML altogether (e.g., with SVG-in-HTML and/or SVG 2 coming to the fore, or if, either in this version of EPUB or the next one, we move towards HTML5) this issue may become increasingly moot. I.e., I am not sure we have to thrive for something that is complete and exhaustive from an XML point of view…

I have seen some concrete proposals in the thread (since the WG call); can we try to see if we can keep to one of them?

1. The WG resolution:
   
   
   
   1. Remove bullet “External identifiers MUST NOT appear in the document type declaration [XML].”
   
   
   2. Add reading system conformance saying that reading systems SHOULD NOT validate against DTDs.
   
   
2. Modification of the resolution, by @murata2makoto
   
   
   
   1. Remove bullet “External identifiers MUST NOT appear in the document type declaration [XML].”
   
   
   2. Add reading system conformance saying that reading systems SHOULD NOT retrieve external DTD subsets or external parameter entities.
   
   
3. Slight modification of (2) above by @shiestyle in a separate sub-thread
   
   
   
   1. Remove bullet “External identifiers MUST NOT appear in the document type declaration [XML].”
   
   
   2. Add reading system conformance saying that reading systems SHOULD NOT validate against DTDs and SHOULD NOT retrieve external DTD subsets or external parameter entities.
   
   
4. Modification of the resolution, by @mattgarrish
   
   
   
   1. Remove bullet “External identifiers MUST NOT appear in the document type declaration [XML].”
   
   
   2. Not mention the issue in the normative part of reading system conformance
   
   
   3. Add some text to the security considerations of the spec drawing attention to the danger of using any kind of external DTD-s or external parameter entities. (Possibly referring to RFC 7303, as proposed by @murata2makoto in a separate comment.)
   
   

Did I miss/misinterpret something here? Would it be possible to see if we can converge towards one of these approaches and solve this issue?

[iherman]

Notes on my previous comment

• I think that, whatever we end up doing, we should mention the issue in the security section of the content spec, as proposed in (4)
• Personally, and regardless of which formal resolution we take, I would also add an editorial note somewhere warning authors that using any kind of DTDs and/or XML entity definition is not recommended due to the uncertainty on whether the underlying rendering engine would handle it appropriately.

[murata2makoto]

@iherman

can we try to see if we can keep to one of them?

I don't think that we are ready to do so.

Also, as it was mentioned several times, many of the situations in the thread are hypothetical as far as current EPUB deployments are concerned, because very few of our authors make use explicitly of remotely complex XML facilities

The door has been closed in the past. But the tentative resolution opens a can of worms. I don't think that we should open the can and hope no worms will cause any trouble.

I think that we should divide this issue into smaller ones and close one at a time. My four issues (#1354, #1355, #1357, and #1358) are intended to solve easier subissues.

[iherman]

I think that we should divide this issue into smaller ones and close one at a time. My four issues (#1354, #1355, #1357, and #1358) are intended to solve easier subissues.

I am not sure I agree and I am afraid that this issue may get out of hand. Those issues try to define a class of XML files that are acceptable for EPUB, looking at variations of possibilities with DTDs, entities, etc, that the XML specification offer. This is inherently a complex task. The question we have to ask ourselves is: who would really care? Is there really a user need behind these? All these possibilities around SVG would be unused by any SVG authoring tool I know, and I do not think that will change. The same would hold for SMIL or MathML, or other XML vocabularies we may care about. As I said above, with the future push to move away from XML altogether (e.g., with SVG-in-HTML and/or SVG 2 coming to the fore, or if, either in this version of EPUB or the next one, we move towards HTML5) this issue may become increasingly moot.

The door has been closed in the past. But the tentative resolution opens a can of worms. I don't think that we should open the can and hope no worms will cause any trouble.

I agree with your concern. So maybe we want to go back to the original issue an stop there. Here is a proposal:

The specification text should say

• External identifiers MUST NOT appear in the document type declaration [XML], except for the identifiers defined explicitly for W3C specifications like SVG 1.1, MathML 1.1, SMIL, or SSML [ref]; these DTD-s should be ignored.

We may want to add the XHTML1.1 DTD-s tot the lot, too, but not necessarily.

That is it. We do not really open the door, we just acknowledge the reality for our usage of W3C Recommendations.

(We should be careful about the “Perfect is the enemy of good” effect…)

[murata2makoto]

@iherman

My #1358 is a small step for achieving what you sketched at the end of your message.

[mattgarrish]

Is there really a user need behind these?

I find it doubtful. No offense, as I grew up on SGML, but we're partying like it's 1999 in this thread... :)

XSLT is the the more usual language of choice these days if publishers want to insert boilerplate, repeated strings, etc.

Your proposal seems similar to where we were somewhere way back in this thread (although I'm not sure how we disagreed on the wording and I'm not going to try and locate why). External identifiers are disallowed except for a controlled list and reading systems ignore (that has to be separately in the RS spec, though).

Works for me.

[iherman]

I have just realized that we should not forget to add the ONIX DTD (or DTDs?) to our controlled list:

<!DOCTYPE ONIXMessage SYSTEM "http://www.editeur.org/onix/2.1/02/reference/onix-international.dtd">

after all, it should be possible to add the ONIX metadata as part of the package...

[iherman]

Your proposal seems similar to where we were somewhere way back in this thread

yep, we made a big circle...

[mattgarrish]

I have just realized that we should not forget to add the ONIX DTD (or DTDs?) to our controlled list:

Maybe no harm, but not sure how critical it is. My understanding is they're pushing to move everyone to 3.0 now and 3.0 records aren't supposed to be distributed with a doctype.

I don't believe ONIX records actually end up in EPUBs all that often, either.

[iherman]

To be honest, I do not know. SOmething to be checked, possibly with Graham...

[clapierre]

ONIX files are out-of-band metadata that publishers send out separate from the EPUB. a lot of times aggregators and distributors get only the ONIX files to determine what EPUBs they want. Also publishers can send these ONIX files with updates say pricing changes etc. without updating the EPUB itself.

[iherman]

If we go down the road of listing some 'acceptable' XML DTD-s (see https://github.com/w3c/epub-specs/issues/1338#issuecomment-716489904) then adding an ONIX DTD is painless. I realize that it does not happen often, but we should not make it impossible to add an ONIX file into the EPUB container...

[murata2makoto]

I just created #1369, #1370, #1371, and #1372. Hopefully, all sub-issues are easier

[murata2makoto]

Created another issue (#1373) for ONIX.

[avneeshsingh]

+1 to have the solution that works for ONIX also. Most of times ONIX is external metadata but during ISO process of EPUB Accessibility, we observed interest in having ONIX accessibility metadata also as internal metadata.

[murata2makoto]

This article is interesting. It mentions a massive DDoS attack on the W3C web site all the time.

[iherman]

This issue was discussed in a meeting.

• RESOLVED: Merge PR #1368 to address outstanding DTD issues, and close GH issues 1369-1373
  
  View the transcript

  
  Wendy Reid: we had resolutions at the F2F, and further discussions on github
  
  … and came to a happy place
  
  Matt Garrish: https://github.com/w3c/epub-specs/pull/1368
  
  Matt Garrish: where we ended up was…
  
  … we put in an allowance for a specific set of external identifiers that we have put in an appendix
  
  … we have SVG and MathML that are allowed to be used in content docs or in separate files
  
  … and we made a restriction against external entities in the internal DTD subset
  
  … so it prevents some security issues but eases authoring
  
  … so we’ll no longer force people to remove SVG DTDs from tool-generated files
  
  … I’m hoping this is it :)
  
  Ivan Herman: tech comment
  
  … in fact, the changes are such that
  
  … makes possible something that I’m not sure we really use
  
  … I can define as part of an internal entity something that won’t go out to the network
  
  … I’m not sure if this feature is in use
  
  … formal comment
  
  … there was a formal resolution on the previous version; this PR slightly changes that
  
  … can we get a formal resolution to merge, and also close a bunch of issues which were examples of the problem?
  
  Proposed resolution: Merge PR #1368 to address outstanding DTD issues, and close GH issues 1369-1373 (Wendy Reid)
  
  Garth Conboy: +1
  
  Matt Garrish: +1
  
  Ivan Herman: +1
  
  Charles LaPierre: +1
  
  Matthew Chan: +1
  
  Wendy Reid: +1
  
  Brady Duga: +1
  
  George Kerscher: +1
  
  Laura Brady: +1
  
  Bill Kasdorf: +1
  
  Ben Schroeter: +1
  
  Resolution #1: Merge PR #1368 to address outstanding DTD issues, and close GH issues 1369-1373