Dom-testing-library: Feature request: Shadow DOM support

[apalaniuk]

Describe the feature you'd like:

Web components are an increasingly popular way of creating reusable elements. Due to the existence of shadow DOM, user-focused testing is more difficult and requires tooling focused around the issue, as in the most permissive case, element queries/observer events (eg. via MutationObservers) stop at shadow boundaries and must explicitly be projected into shadow trees. Without support from tooling, aspects of a web app written using web components are virtually untestable without using methods contrary to the goals of this project.

Suggested implementation:

• Standard element queries can be modified to be projected into successive shadow trees
• Mutation observers to support eg. waits for DOM tree changes can be attached to all shadow roots of descendants
• Solution must consider related concepts (eg. slots) insofar as they relate to testing goals and principle of least surprise according to their documented behaviour/DOM standards
• No expectation of closed shadow roots to be supported in any way, but ideally would be explicitly documented

Describe alternatives you've considered:

• As first-class support is required, alternatives involve using/writing an alternative implementation which supports shadow trees explicitly (or potentially, writing a plugin system to allow others to support it)
• As shadow tree concepts are an aspect of WHATWG/W3C DOM standards, it makes sense that they're supported first-class by tooling that supports DOM operations

I'd be happy to add this feature, and support for basic functionality (query, waits) was straightforward to implement, but I wanted to get insight on constraints/concerns in advance as I don't currently use this library.

[ashubham]

+1

It will be great to build support for WebComponent based frameworks like lit-element. If there is a adapter/plugin API, I am sure the community will jump on to build it.

[alexkrolick]

How much of this depends on JSDOM itself? This library doesn't implement the DOM part, just traversal.

[apalaniuk]

Hi @alexkrolick, it's a good question. JSDOM currently has shadow DOM support, with a few restrictions. The largest issue for testing right now is that this is almost entirely used in the context of web components, and support for custom elements is in-flight (https://github.com/jsdom/jsdom/pull/2548).

That said, the only changes required in here for shadow DOM support should be unrelated, since this would just need to support automatically traversing through shadow trees for queries/attaching MutationObservers/etc, and shadow roots can be attached to many standard elements.

Since the overall change seems small, it's probably better to just implement in a fork and get more direct feedback based on an implementation.

[kentcdodds]

A PR would be helpful. But I'm getting the impression this will be anything but a small change 😬

[ashubham]

What about when we use this library with puppeteer ? How difficult will be to have webcomponent support in that mode ?

[apalaniuk]

@kentcdodds There's definitely a lot of complexity to work out, so I'm just being (overly) optimistic 🙂 we have done similar things at my company in the past (though only supporting a subset of the required operations), but supporting it for general use cases will take some thought (complexity to get same behaviour in traversing the light DOM given there aren't any primitives for traversal/element selection when considering shadow DOM, performance, maintainability, providing unsurprising behaviour in all cases, etc.)

[Westbrook]

I've been toying with an approach to making this possible: https://github.com/adobe/spectrum-web-components/pull/420

For me, there are two issues in particular that I'm working on:
1) ES Module based support so that Karma + es-dev-server can run @testing-library/dom which has some downstream CJS dependencies, for which I'm rebuilding the project locally: https://github.com/adobe/spectrum-web-components/pull/420/files#diff-de467343944f9302cd195062f4d866b2 it also knocks off some Node vs browser differences
2) Seeing into shadow DOM appropriately (normally an anti-pattern, but acceptable, IMO, in a testing environment), for which I'm using: https://github.com/adobe/spectrum-web-components/pull/420/files#diff-33d340131f49793bce88e37a2122f57c

At the intersection between the two issues is shimming @testing-library/dom to use the non-standard querySelectorAllWithShadowDOM so that the "available" DOM is queried without altering the API of the components.

Is this looking completely crazy? Are either or both of these changes something that might look interesting in a PR here?

[Westbrook]

So, in a way, the above is also a bit of a revival of #161 as well... 😬

[kentcdodds]

Hi folks,

I just am not convinced that the complexity will be worth it. You can feel free to open a PR if you like, but if it's more than a little complex I don't think I'll want it here and instead maybe you could add a custom query and even make a library of those for the people using Shadow DOM.

[Westbrook]

Definitely the steps for full web component support could be more complex, but ensuring that DOM targetted code can actually be tested in a real DOM environment would be a great baseline for this library to support. Then, were someone to want to extend/alter it to better support the growing capabilities of the full DOM model itself they could so with little difficulty. To that end, I think the initial steps that I'd love to see are as follows:

• Don't rely on process.env: see #431
• Don't rely on CJS dependencies: This one is a little more tricky (however seems to be contained to @sheerun/mutationobserver-shim at this time) and it would be better to get some insight on what is deemed acceptable before I move forward.

The _best_ solution is to PR the library to export ESM on its own (if you were willing to support an issue/PR of that there, the voice of such a well-used and respected library would likely go a long way, so I'd be happy to open the conversation for you to jump in there), but as we've seen not everyone is excited about supporting modern APIs in their libraries. In the interim of that occuring, I'd like to pre-process @sheerun/mutationobserver-shim via an addition to the Rollup config similar to (output locations TBD, etc, etc):

    {
        input: require.resolve(
            '@sheerun/mutationobserver-shim/MutationObserver.js'
        ),
        output: {
            file: './lib/mutationobserver-shim/index.js',
            format: 'es',
        },
        plugins: [
            resolve({
                browser: true,
                preferBuiltins: false,
            }),
            commonjs(),
        ],
    },

Then it's a simple alias to slot this new output into @test-library/dom without having to affect the current contents of the JS (meaning it's plug&play when the upstream output changes!). This looks like:

            alias({
                entries: {
                    '@sheerun/mutationobserver-shim':
                        'lib/mutationobserver-shim/index.js',
                },
            }),

With these two changes, it is likely that #161 could be reopened/closed as fixed in one fell swoop and then we could have a stand-alone conversation about supporting the web component specifications. The decision as to whether that could be achieved via custom queries or should belong in @test-library/dom, in an alternate @test-library/full-dom sort of package, or be supported via an external extension of the library is an important one and should be given the focused consideration that it deserves.

Thanks in advance for your gracious insight.

[kentcdodds]

Is there any chance that you could maintain a separate package that extends the capabilities here (via custom queries or similar)? If so, I would be willing to make a few smaller modifications (like the process PR I just merged of yours [thank you]) to make that easier. What do you think?

[apalaniuk]

I haven't been able to revisit this at all, but I definitely appreciate the work others have done. I was thinking over the holidays that the relative complexity of the implementation as it trends toward "reasonable behaviour" in ways the specs don't define + possible performance implications + the relatively few users that need this in the near term means that it would likely be best as a separate package/plugin.

[Westbrook]

Thanks for your thoughts here. I wouldn't be opposed to that, to support that path requiring as few modifications as possible could you share a little more on the following point?

As per https://github.com/testing-library/dom-testing-library/issues/348#issuecomment-576824315 it seems like newer versions of JSDom will have caught up with browser specs as far as MutationObservers (Can I Use). That being so, do you foresee a future where you removed the @sheerun/mutationobserver-shim dependency from this library? I suppose that would technically be a breaking change, but it would prevent from needing to alter its output either at the repo level, where the project seems _complete_ (beyond simply not accepting issues), or at the project level, via Rollup configs as outlined above.

With that change a separate package would only need to do:

replace({
    querySelectorAll: 'querySelectorAllWithShadowDOM',
}),

and put the library behind the shim for querySelectorAllWithShadowDOM which would be a dream to maintain on my end. Thanks for helping me talk through this possibilities here!

[kentcdodds]

Hi @Westbrook,

I definitely have plans to remove that shim in the next breaking change (which is hopefully coming soon).

Sounds like once we get that done then you'll be in a pretty good place to make a simple package that supports shadow DOM which I think is the best path forward 👍

I'll be sure to ping you all in here when we've published the update. You're more than welcome to participate in discussion leading up to that major version bump. Hopefully that'll start in the next few days.

[kentcdodds]

:tada: This issue has been resolved in version 7.0.0-beta.3 :tada:

The release is available on:

• npm package (@beta dist-tag)
• GitHub release

Your semantic-release bot :package::rocket:

[kentcdodds]

:tada: This issue has been resolved in version 7.0.0 :tada:

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot :package::rocket:

[stevematney]

@Westbrook we're using dom-testing-library to test some Web Components, and would love to see our approach simplified by a package like the one you mentioned potentially creating. Do you have any plans to release something like that?

[Westbrook]

Sorry, I let this fall way way way off my to-do list, but finally got around to pushing a beta release of the wrapper I'd been working with to NPM: https://www.npmjs.com/package/testing-library__dom Hopefully it'll be useful for people!

[kentcdodds]

Thanks!

[umpox]

Might be helpful to post a different solution to querying elements in a shadow root, if anyone else is reading this:

If are able to access the shadow root (e.g. query the an element that wraps it), you can just do something like this:

    const { queryByText } = within(someNode.shadowRoot);
    queryByText('Element within a shadow root!');