Dom: Un-merge Document and HTMLDocument

[ayg]

Does anyone object? It's been years, nobody has done it, it's claimed that IE did it and then un-did it. Boris has serious reservations about web-compatibility. Is there any actual reason to merge them? Obviously, it makes sense to keep APIs on Document where reasonable, but I don't see why it needs to be wholesale.

[ayg]

@bz

[bzbarsky]

Note that my serious reservations are not based on data, but rather on lack of data. I could be convinced that this is ok, with actual data and analysis...

[foolip]

There has been some efforts to move things from HTMLDocument to Document in Blink, and it's been successful so far. https://bugs.chromium.org/p/chromium/issues/detail?id=238368

Of what remains I think that only document.all is worth worrying about. It's already on Document internally, but the risk is that some code depends on xmlDocument.all not existing.

I don't have time to spend on this _right now_, but if it'll get reverted in the spec I'd like a chance to try out the final changes in Blink, so let me know if there's a deadline.

[bzbarsky]

Right, moving particular APIs to Document might well make a lot of sense. Even moving them all might make sense. What doesn't make sense to me is doing the move in a single commit with no attempt to do per-property analysis.

[ayg]

So how about this: leave the spec as-is for now, start moving properties to Document on a per-case basis, and if we find any properties we don't want to try moving, change the spec to reintroduce HTMLDocument with those properties. Sound good to everyone?

[foolip]

Yep, that sounds good. I'm pretty optimistic about Blink being able to make HTMLDocument empty, but there could be engine-specific compat issues. Not sure how risky the final step of making HTMLDocument an alias is, we'll just have to try it.

[ayg]

Okay, so let's all try that.

[cdumez]

What's the benefit of merging HTMLDocument into Document? Do we really want various HTML-specific attributes / operations to be exposed on non-HTML documents?

This is a risky change compatibility-wise and major browser engines all had (and still have) HTMLDocument. This has been in the specification for years and yet no major browser has not implemented it yet.

I personally do not understand the benefits of this change but even if there are benefits, is it really worth the trouble / risk for all browser engines? I know I am not inclined to do this change in WebKit.

[foolip]

Missing _not_ in the last sentence?

In Blink most of what was in HTMLDocument has already been moved to Document, see https://github.com/whatwg/dom/issues/221#issuecomment-209989740.

The set of things that are in HTMLDocument in Gecko and WebKit is not aligned.

In Edge, where the IDL files aren't available, I see that Object.getOwnPropertyNames(HTMLDocument.prototype) and Object.getOwnPropertyNames(XMLDocument.prototype) both return just ["constructor"] while Object.getOwnPropertyNames(Document.prototype) returns ["constructor", "URL", "URLUnencoded", "activeElement", "alinkColor", "all", "anchors", "applets", "bgColor", "body", "characterSet", "charset", "compatMode", "cookie", "currentScript", "defaultCharset", "defaultView", "designMode", "dir", "doctype", "documentElement", "domain", "embeds", "fgColor", "forms", "head", "hidden", "images", "implementation", "inputEncoding", "lastModified", "linkColor", "links", "location", "msCSSOMElementFloatMetrics", "msCapsLockWarningOff", "onabort", "onactivate", "onbeforeactivate", "onbeforedeactivate", "onblur", "oncanplay", "oncanplaythrough", "onchange", "onclick", "oncontextmenu", "ondblclick", "ondeactivate", "ondrag", "ondragend", "ondragenter", "ondragleave", "ondragover", "ondragstart", "ondrop", "ondurationchange", "onemptied", "onended", "onerror", "onfocus", "oninput", "oninvalid", "onkeydown", "onkeypress", "onkeyup", "onload", "onloadeddata", "onloadedmetadata", "onloadstart", "onmousedown", "onmousemove", "onmouseout", "onmouseover", "onmouseup", "onmousewheel", "onmscontentzoom", "onmsgesturechange", "onmsgesturedoubletap", "onmsgestureend", "onmsgesturehold", "onmsgesturestart", "onmsgesturetap", "onmsinertiastart", "onmsmanipulationstatechanged", "onmssitemodejumplistitemremoved", "onmsthumbnailclick", "onpause", "onplay", "onplaying", "onpointerlockchange", "onpointerlockerror", "onprogress", "onratechange", "onreadystatechange", "onreset", "onscroll", "onseeked", "onseeking", "onselect", "onselectionchange", "onselectstart", "onstalled", "onstop", "onsubmit", "onsuspend", "ontimeupdate", "onvolumechange", "onwaiting", "onwebkitfullscreenchange", "onwebkitfullscreenerror", "plugins", "pointerLockElement", "readyState", "referrer", "rootElement", "scripts", "scrollingElement", "styleSheets", "title", "visibilityState", "vlinkColor", "webkitCurrentFullScreenElement", "webkitFullscreenElement", "webkitFullscreenEnabled", "webkitIsFullScreen", "xmlEncoding", "xmlStandalone", "xmlVersion", "onpointercancel", "onpointerdown", "onpointerenter", "onpointerleave", "onpointermove", "onpointerout", "onpointerover", "onpointerup", "onwheel", "adoptNode", "captureEvents", "caretRangeFromPoint", "clear", "close", "createAttribute", "createAttributeNS", "createCDATASection", "createComment", "createDocumentFragment", "createElement", "createElementNS", "createExpression", "createNSResolver", "createNodeIterator", "createProcessingInstruction", "createRange", "createTextNode", "createTreeWalker", "elementFromPoint", "evaluate", "execCommand", "execCommandShowHelp", "exitPointerLock", "focus", "getElementById", "getElementsByClassName", "getElementsByName", "getElementsByTagName", "getElementsByTagNameNS", "getSelection", "hasFocus", "importNode", "msElementsFromPoint", "msElementsFromRect", "open", "queryCommandEnabled", "queryCommandIndeterm", "queryCommandState", "queryCommandSupported", "queryCommandText", "queryCommandValue", "releaseEvents", "updateSettings", "webkitCancelFullScreen", "webkitExitFullscreen", "write", "writeln", "createEvent", "querySelector", "querySelectorAll"].

In other words, it looks like Edge has already moved everything to Document.

@cdumez, what do you think would be a reasonable thing to attempt to converge on? Based on the above, I think the most tractable path with relatively low risk is to move everything into Document, but then leave HTMLDocument an empty interface just like XMLDocument.

[cdumez]

Yes, I was missing a 'not' in my last sentence. It is now fixed.

It is interesting that Edge made did change, I was not aware.

I guess moving everything to Document while keeping an empty HTMLDocument is relatively low risk as you say. I guess I am just not clear yet on the benefits of doing this.

[foolip]

@annevk will know, but probably the original motivation was that it's possible to insert any element into any document, so having a type of the document itself doesn't achieve much. Also, I think, to reduce gratuitous differences between XHTML (XMLDocument) and HTML. (Some algorithms still have different behavior based on the document type, though.)

Whatever the history, we're now in a weird state and need to plot a course towards interop. Does at least trying to make HTMLDocument empty seem reasonable? Given the changes already made in Blink and Edge, this is very likely web compatible.

Whether to attempt the final step of aliasing I don't know. If we don't, we may need to rethink the Document constructor, perhaps instead making it such that HTMLDocument and XMLDocument are the only possible instances.

[cdumez]

I am curious to know what Firefox is planning to do. If Firefox goes in this direction as well then WebKit would likely follow. So far, Gecko and WebKit are mostly in agreement.

[foolip]

@smaug----, can you comment on Gecko plans?

[smaug----]

I agree with bz. We need to do this per property, not all at once. And while doing it, it is possible that we figure out the merge isn't really possible. We don't really know the _behavior_ of the stuff Edge has.

Does Edge still have HTMLDocument interface? What is the prototype chain of an (html) document?

I'm tiny bit worried about stuff like open/write/close for example: do they work as expect with SVG or Mathml or such?

[foolip]

I agree with bz. We need to do this per property, not all at once. And while doing it, it is possible that we figure out the merge isn't really possible. We don't really know the _behavior_ of the stuff Edge has.

We moved it in smaller parts in Blink as well. Testing what others do makes sense for each step. It seems pretty likely that it's web compatible to move everything, so the question is what you'd like to _attempt_ to do in Gecko.

Does Edge still have HTMLDocument interface? What is the prototype chain of an (html) document?

Edge still has an HTMLDocument interface, and an instance has the expected prototype chain HTMLDocument : Document : Node : EventTarget.

I'm tiny bit worried about stuff like open/write/close for example: do they work as expect with SVG or Mathml or such?

In Blink, these and writeln() all throw InvalidStateError for XML documents, as per spec:
https://html.spec.whatwg.org/#dynamic-markup-insertion

In Edge, it looks like open() and close() don't throw an exception (not sure what they do) but write() and writeln() do.

So there's still some interop work to do here...

Finally, one thing that could go badly here if we all aim to make HTMLDocument and XMLDocument aliases of Document is that one or two engines succeed, but the next to try runs into engine-specific troubles similar to that around Gecko's XMLDocument.prototype.load, leaving us in an non-interoperable state for a long time.

[cdumez]

I am working on moving some properties from HTMLDocument to Document in WebKit. I am focusing on the properties that Blink has already moved and that I think make sense to have on Document (i.e. do not throw if called on anything else than an HTML document).

Since I don't anticipate HTMLDocument going away entirely anytime soon. I don't think it makes sense to move to Document attributes / operations that work only on HTML documents (e.g. open/close/write/writeln).

[ayg]

If some engines can't move one or two properties, the easiest solution would be to just move those back to HTMLDocument in the spec. As long as we have interop, it doesn't really matter where the properties are, so take the path of least resistance.

[cdumez]

Here is the current situation:
Chrome 52: [‘alinkColor', 'all', 'bgColor', 'captureEvents', 'clear', 'fgColor', 'linkColor', 'releaseEvents', 'vlinkColor']
WebKit ToT: [‘alinkColor', 'all', 'bgColor', 'captureEvents', 'clear', 'close', 'fgColor', 'linkColor', 'open', 'releaseEvents', 'vlinkColor', 'write', 'writeln']

So the only difference between Chrome and WebKit at the moment are the following properties:
"open", "close", "write" and "writeln".

As I said before, I did not move them on purpose because it seems the expected behavior is for them to throw when calling on anything else than an HTMLDocument. Given this behavior and the fact that we still have an HTMLDocument interface in WebKit, I see little point in moving them up to Document.

My hope is that an HTMLDocument interface is re-introduced in the HTML specification for the properties that are not useful on other types of documents (and legacy HTMLDocument attributes such as alinkColor since I don't think we want to expose legacy properties or more document types).

[annevk]

If we add it back we need to be very clear which contexts end up creating it although I suppose that is the case already due to "HTML document"…

[foolip]

Some thoughts on the WebKit ToT list from https://github.com/whatwg/dom/issues/221#issuecomment-240182587

• document.all looks to me like it it'd work just fine with any kind of document. The implementation itself is already on Document in Blink and when I tried moving it my ad-hoc test worked fine.
• alinkColor, bgColor, fgColor, linkColor and vlinkColor work as long as there's a body element, and document.body is already on Document. There are also a bunch of collections like document.forms on Document.
• clear(), captureEvents() and releaseEvents() are all no-ops so it doesn't matter much.
• close, open, write and writeln would all throw exceptions and wouldn't be useful on Document. The one benefit of moving them anyway is that the exception message could say why it didn't work.

At the end of the day, I don't feel _strongly_ about any of these, any course of action that would get us to interop would be fine.

The missing piece here is what to do about the Document constructor? Per https://github.com/whatwg/dom/issues/278#issuecomment-239461359 I think we should try to retain it, but what should it return? Gecko and WebKit both return a document with contentType "application/xml", but it's really unfortunate that it isn't an XMLDocument then. @cdumez @ayg, thoughts?

[foolip]

I did a little bit of research with SELECT page,url FROM [httparchive:har.2016_08_01_chrome_requests_bodies] WHERE body CONTAINS 'instanceof XMLDocument' in BigQuery:

• htmldocument.csv (used WHERE RAND() < 1/10 to get about 10%, leaving 1254 rows)
• xmldocument.csv (76 rows)

It's immediately clear that instaceof HTMLDocument is way more common. I randomized the order and the first thing I found came from mutation-summary.js by @rafaelw:
https://github.com/rafaelw/mutation-summary/blob/421110f84178aa9e4098b38df83f727e5aea3d97/src/mutation-summary.js#L112

@rafaelw, I didn't look too closely, but I take it this code actually matters for getting the correct behavior in both XML and HTML documents?

Copies of this code seem to dominate the usage. http://cdn.clicktale.net/www/ChangeMonitor-latest.js showed up a lot but seems to include an older version of mutation-summary.

Given the volume of results, I'd be cautious about changing the truthyness of instanceof HTMLDocument without further analysis.

As for XMLDocument, after checking the most common URL patterns I couldn't find anything that seemed like it would break. So I'd give better odds to making XMLDocument an alias of Document, but that's not a good idea as long as Gecko needs XMLDocument.protoype.load.

[foolip]

https://github.com/whatwg/dom/issues/308 is also relevant.

I realized that "object HTMLDocument" and "object XMLDocument" are also relevant for compat. I find 606 instances of "object HTMLDocument" and 9243 instances of "object XMLDocument". I didn't dig deeply, but a lot of the latter seems to be MooTools:
https://github.com/mootools/mootools-core/blob/1be4d62e912e8acc25ddccd5f8059a9cba0862ed/Source/Slick/Slick.Finder.js#L22

Sigh.

[foolip]

In https://github.com/whatwg/dom/issues/221#issuecomment-210115403 I said "Not sure how risky the final step of making HTMLDocument an alias is, we'll just have to try it" upon which the issue was closed.

After https://github.com/whatwg/dom/issues/221#issuecomment-244738573 I'm a lot more cautious, and I think we should actually just revive HTMLDocument, unless there's concrete implementer interest for attempting the aliasing.

[domenic]

@foolip that comment doesn't give an argument for reviving HTMLDocument. Being an alias preserves instanceof HTMLDocument working.

[foolip]

Yes, but it'd also start "working" for things that aren't currently instanceof HTMLDocument, the whole point of the check seems to be to get at the https://dom.spec.whatwg.org/#html-document state.

It's _possible_ that it's safe because the code doesn't actually run in any non-HTML documents, but it'd require more analysis.

[domenic]

Right, my impression is that there are very few non-HTML documents out there, and people are just using it as a test that something is a document (and not, say, any other kind of node). But that is an assumption.

[foolip]

That probably happens, but the case that dominated in https://github.com/whatwg/dom/issues/221#issuecomment-244738573 was about case sensitivity.

[foolip]

I've tried to summarize everything I could find about document interfaces here:
https://gist.github.com/foolip/103963a1ae8598d2baedd296f4a1bf4c

I think it would be very good if representatives from all implementers, including from Edge, could get together and discuss which path to take here. My own take is that it depends on use of XMLDocument's load() method in Firefox. If usage is low, then the "Document and HTMLDocument" path looks good. If not, then I prefer "HTMLDocument and XMLDocument".

[smaug----]

ctor for Document is already shipping.
I would just do "Document, HTMLDocument and XMLDocument", since that requires least changes to the implementations and be done with this issue. Needs of course some spec updates.

(SVGDocument removal https://bugzilla.mozilla.org/show_bug.cgi?id=1293786 waiting for landing)

[foolip]

ctor for Document is already shipping

This was made very clear in the doc, but I've now also added that to the timeline, although I couldn't find the impl bug for Gecko, I just bisected its support to Firefox 20.

Do you agree that "When both Document and XMLDocument are concrete and represent the same thing, this is the least inconsistent state possible. It would be an odd API to avoid."?

[smaug----]

Nothing to do with consistency. The API would be a tiny bit odd yes, but in practice doesn't really matter. Yet the approach should be least likely to break anything.

[domenic]

@foolip, I would really appreciate if you would remove "HTMLDocument and XMLDocument" from the possibilities. We have heard pushback against that from two implementers, and keeping it on the table is preventing Blink from moving forward and achieving compatibility with WebKit and Gecko. All other options are interesting to discuss, and don't prevent Blink from shipping the Document constructor and achieving interop while we conduct that discussion. But leaving that particular option on the table is harmful to forward progress, in my opinion.

[cdumez]

As @smaug---- , I am in favor of the path of having HTMLDocument / XMLDocument / Document for now. I think this is the quickest path to interoperability given the current state of the engines.

[annevk]

@cdumez @smaug---- does that mean we make each of those a proper class (with HTMLDocument and XMLDocument each extending Document) and potentially give each of them their own constructor? I think that's fine. The difference between that and aliasing is really small.

[cdumez]

@annevk Yes, I meant that HTMLDocument and XMLDocument would be proper interfaces for now, inheriting from Document. However, I would also prefer that some of the properties remain on HTMLDocument We should agree on which properties would remain on HTMLDocument but Blink and WebKit are close to being aligned on this.

My personal preference would be to keep these on HTMLDocument:

• open(), close(), write(), writeln(), all, clear(), captureEvents(), releaseEvents(), bgColor, fgColor, alinkColor, linkColor, vlinkColor;

And to answer your question, yes, I think it'd be good to have a constructor on XMLDocument and HTMLDocument.

[annevk]

Keeping domain on HTMLDocument might also be good (Fx does this I'm told). We should probably go through the list once more.

[cdumez]

I would have no problem moving some properties (such as 'domain') back to HTMLDocument.

[annevk]

APIs on documents that seem fine to keep restricted to HTMLDocument objects:

• location (just use window.location)
• domain (bad feature, mutable origins)
• cookie (bad feature, sync storage)
• named getters (bad feature)
• open/close/write/writeln
• fgColor/linkColor/vlinkColor/alinkColor/bgColor
• clear/captureEvents/releaseEvents
• all

Maybe keep restricted:

• body/head
• images/embeds/plugins/links/forms/scripts/anchors/applets
• getElementsByName
• currentScript (bad feature with shadow trees and module scripts)
• designMode

Do we use HTMLDocument objects for application/xhtml+xml responses?

[foolip]

@domenic, I agree that "HTMLDocument and XMLDocument" is unlikely, as is "Just Document", but "Document and HTMLDocument" still looks plausible.

These are the spec-side steps I think make sense:

1. Un-merge HTMLDocument (now)
2. Wait for use counter data from Gecko on XMLDocument.prototype.load
3. If risk looks acceptable to Gecko, go with "Document and HTMLDocument" by making XMLDocument an alias of Document.
4. If not, go with "Document, HTMLDocument and XMLDocument"

If the Blink DOM team really wants to ship the constructor before knowing the fate of XMLDocument I will not stand in the way, but I'm not a fan because it increases the likelihood of getting stuck that silly state.

[annevk]

What does unmerge HTMLDocument mean with respect to the members of that object and Document? I agree it's a good first step, but we should probably decide on the divide, no?

[foolip]

Do we use HTMLDocument objects for application/xhtml+xml responses?

Gecko seems to be alone in doing that right now.

I think the split between HTMLDocument and Document would depend on that a bit. It'd be nice to minimize the differences between HTML and XHTML, and if application/xhtml+xml becomes XMLDocument that's a good reason to move as much as possible to Document.

[rwlbuis]

Edge and I think Safari (https://trac.webkit.org/changeset/218437/webkit) moved all properties from HTMLDocument to Document, I plan to do the same in Blink:
https://chromium-review.googlesource.com/c/chromium/src/+/695815

[Zirro]

This has caused a bit of confusion in https://github.com/whatwg/dom/pull/551. As long as this is unresolved the specification ought to contain a reference to this issue with a note mentioning that it does not necessarily reflect reality at the moment.

[annevk]

Though maybe if Chrome, Edge, and Safari have indeed shipped what the specification suggests and there's no longer a distinct HTMLDocument class this has indeed happened now (except in Firefox). I guess we still have a distinct XMLDocument class, but that's okay.

[dstorey]

Yes, Edge matches the spec here (although we haven't removed the actual HTMLDocument interface)

/// @specs "dom-level-2-html"
[Deprecated]
interface HTMLDocument : Document
{
};

[foolip]

@dstorey, the spec for this is https://html.spec.whatwg.org/multipage/window-object.html#htmldocument, which would mean that window.HTMLDocument===window.Document, ad-hoc test in https://jsbin.com/kidewim. Edge fails that way in the way one would suspect from the pasted IDL, namely that HTMLDocument is a distinct interface.

However, it's the same empty interface in Blink:
https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/html/HTMLDocument.idl

And almost empty in WebKit:
https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/html/HTMLDocument.idl

But still lots of stuff in Gecko:
https://github.com/mozilla/gecko-dev/blob/master/dom/webidl/HTMLDocument.webidl

The situation is similar for XMLDocument:
https://cs.chromium.org/chromium/src/third_party/WebKit/Source/core/dom/XMLDocument.idl
https://github.com/mozilla/gecko-dev/blob/master/dom/webidl/XMLDocument.webidl
https://trac.webkit.org/browser/webkit/trunk/Source/WebCore/dom/XMLDocument.idl

http://web-confluence.appspot.com/ is also a good way to check this, but the HTMLDocument changes seem recent so haven't reached stable yet.

[DavidBruant]

But still lots of stuff in Gecko:
https://github.com/mozilla/gecko-dev/blob/master/dom/webidl/HTMLDocument.webidl

For reference, Mozilla is working on the move like in bug 1415588

[ExE-Boss]

This is probably a duplicate of https://github.com/whatwg/html/issues/4792.

[annevk]

It needs to be tracked here as well, as it'll affect createHTMLDocument() at least, I assume.

[ExE-Boss]

createHTMLDocument() creates a Document with type "html", which causes it to become an HTMLDocument to JavaScript code.

That’s how engines have been handling it since forever.

[annevk]

That's not how that works. Internal concepts are separate from the public class. And yes, it's known that there's a mismatch with implementations there, and as part of fixing that, this needs fixing.

[ExE-Boss]

Since HTMLDocument is being defined in the HTML standard, should createHTMLDocument() be moved there as well, or should a cyclic dependency be created between the specifications?

[annevk]

Pretty sure there is one already.