Csswg-drafts: [css-transforms-2] proposal new transform-style: detached

[rcabanier]

The current transform-style property defines the values auto, flat and preserve-3d.

Currently, these 3D transform styles composite the transformed element back into the 2D page. This makes sense for classic displays such as monitors and cell phones, but on devices where the browser lives in 3D such in VR and AR devices, it is possible to render these elements in actual 3d.

We're proposing to add a new transform-style called detached. It follows the same rules as preserve-3d except it skip the step where it is recomposited into the page.

[smfr]

except it skip the step where it is recomposited into the page

So where is it composited?

[rcabanier]

So where is it composited?

It is composited in 3D space around the browser plane.

We've defined a "safe" region around the browser where detached content can be placed so the transformed content is locked to a certain area. The spec should probably call out a minimum size for this area so authors can be sure that their content will look ok across UAs.
We also limited this feature to the top frame so iframe content can't escape its box.

If the UA doesn't support detached, it will revert to preserve-3d
If detached is nested in another detached, it will revert to preserve-3d

[smfr]

Piggy-backing on transform-style seems like the wrong thing here. Isn't this more like taking an element and painting it somewhere else (like a canvas)?

[rcabanier]

Piggy-backing on transform-style seems like the wrong thing here. Isn't this more like taking an element and painting it somewhere else (like a canvas)?

No :-)
The element itself is lifted out of the page and painted on its own surface. That surface is then placed in 3D around the browser. The page is not able to get to the rendered pixels. It's exactly like preserve-3d, just without the final recompositing of the surface to the page.

The transformed element also stays interactive (hit testing, streaming video, etc) and you can apply CSS animations.

[smfr]

Well we haven't really decided what preserve-3d itself means.

How does hit-testing work in this model? Do multiple 'detached' intersect?

[rcabanier]

Well we haven't really decided what preserve-3d itself means.

Yes, that's a bit unfortunate. Was any progress made on this recently?

How does hit-testing work in this model?

When using the 6DOF controller and it intersects the plane, the cursor will move from the browser plane to the detached surface. Hit testing will then work as usual. (If there is no cursor, the user will just use the 6DOF hit position)

Do multiple 'detached' intersect?

I think so. preserve-3d also lets content intersect: https://jsfiddle.net/madehrp2/1/
If that codepen was changed to use detached instead, the 2 divs would be hovering 100 pixels above the web page on a Magic Leap device.

[dbaron]

How do detached and preserve-3d interact with each other? What happens when there's a connected tree of stuff with mix of both? Does only the one at the root of the tree matter, or do they somehow switch behaviors?

[rcabanier]

How do detached and preserve-3d interact with each other? What happens when there's a connected tree of stuff with mix of both? Does only the one at the root of the tree matter, or do they somehow switch behaviors?

Yes, we have to describe the exact behavior in the spec.
Our current thinking is that only detached at the root of the tree will trigger the behavior. If there's preserve-3d at the root or detached in the child nodes, detached will act as preserve-3d in that tree.

[progers]

How would detached interact with clipping (e.g., overflow:hidden or clip-path)? For example, if a detached box is below a clip path, would the box's pixels be clipped as if it were preserve-3d?

Similarly, how would detached interact with atomic operations such as filtering? For example, if two detached boxes have a blur filter on them, would their pixels be blurred as if preserve-3d was used?

[progers]

Would you be willing to write an explainer for this proposal?

[smfr]

Also, is there multi-vendor interest?

[rcabanier]

Would you be willing to write an explainer for this proposal?

Most definitely! I will work with my team to get that out

[rcabanier]

How would detached interact with clipping (e.g., overflow:hidden or clip-path)? For example, if a detached box is below a clip path, would the box's pixels be clipped as if it were preserve-3d?

For overflow: hidden, I suspect that the author is going to expect that the transformed element is visible. What happens today if a hidden element is transformed into view?
Are there's special ruls for clip-path? We need to do some investigation there and try it out with the current code :-)

Similarly, how would detached interact with atomic operations such as filtering? For example, if two detached boxes have a blur filter on them, would their pixels be blurred as if preserve-3d was used?

Yes, the filters would apply as usual, except the effect would be computed by the system compositor and not the browser's one.
We haven't hooked that part up yet though. Do you envision problems in that workflow?

[rcabanier]

Also, is there multi-vendor interest?

I'm unsure. I will also tell the immersive web group and see if there's interest

[dbaron]

Another alternative worth considering (which a colleague of mine at Mozilla suggested to me) is that perhaps instead of a new value, the same results could be achieved by changing the semantics of having transform-style: preserve-3d on the root element, such that it means that anything that's preserved as 3-D up to the root element would be rendered in actual 3-D.

[rcabanier]

Another alternative worth considering (which a colleague of mine at Mozilla suggested to me) is that perhaps instead of a new value, the same results could be achieved by changing the semantics of having transform-style: preserve-3d on the root element, such that it means that anything that's preserved as 3-D up to the root element would be rendered in actual 3-D.

Wouldn't that be a change in existing behavior?
If an author wants the regular preserve-3d behavior and they applied it to the root, wouldn't this break their page?
It's possible that I'm misunderstanding your proposal. Could you provide a small html snippet?

[dbaron]

I don't think transform-style: preserve-3d on the root today does anything, since the root implicitly flattens even if it has preserve-3d set, since there's no parent's 3-d scene to merge into. So I don't think people would have a good reason to be using it today -- though that doesn't mean they aren't. So then the next question would be how many are, and then what the tradeoff between extra complexity and that compatibility cost is.

[rcabanier]

I don't think transform-style: preserve-3d on the root today does anything, since the root implicitly flattens even if it has preserve-3d set, since there's no parent's 3-d scene to merge into.

Ah, yes. I understand what you mean now.
One possible drawback is that on regular browsers, the content will look flattened (ie the preserve-3d behavior is lost)

So I don't think people would have a good reason to be using it today -- though that doesn't mean they aren't. So then the next question would be how many are, and then what the tradeoff between extra complexity and that compatibility cost is.

Isn't it also a bit odd that a CSS property has a very different meaning on the root element compared to all other elements?

[mattwoodrow]

I don't think transform-style: preserve-3d on the root today does anything, since the root implicitly flattens even if it has preserve-3d set, since there's no parent's 3-d scene to merge into.

Under the gecko and blink interpretation 'preserve-3d' means that children are included into the rendering context of the element, so it should work fine on the root there.

[upsuper]

A side note, if you create an issue with wrong content, please don't just edit it to the real content, but close it and open a new one or at least post your content as a reply.

There are people read issues via emails, and this kind of issues are really unhelpful for them. Also W3C has an email-based archive for all the discussion. By doing nontrivial in place editing, you are effectively bypassing that archive, and it would look like https://lists.w3.org/Archives/Public/public-css-archive/2019Aug/0447.html

[rcabanier]

I don't think transform-style: preserve-3d on the root today does anything, since the root implicitly flattens even if it has preserve-3d set, since there's no parent's 3-d scene to merge into.

Under the gecko and blink interpretation 'preserve-3d' means that children are included into the rendering context of the element, so it should work fine on the root there.

Are you agreeing with @dbaron or are you saying that this shouldn't be an option?

[grorg]

I guess I need the explainer because I'm not sure how this is supposed to work, or whether it is useful, or why anyone would want to do it.

I assume the idea is that you're looking at a Web page in some AR/VR space, and you want the elements in that page to project out (or back) from the flat surface into the hosting space?

Do you expect the hosting space to apply effects to the projected elements? For example, will they respond to lighting, cast shadows, etc? If so, that means the hosting space needs pixel access, which we can't allow. If not, the effect will look so weird it isn't worth doing.

And what happens as 3d elements are scrolled into view (they were clipped by the viewport)? Do they all of a sudden appear in 3d space? Does half of them appear if they are partially clipped?

And why is this just applying to CSS transforms? Wouldn't a better case be having WebGL objects from the page exist in the hosting space?

My gut feeling is that we should continue to assume that CSS always projects onto a flat 2d surface. Within that surface you can emulate 3d sub-spaces, but the end result is always 2d. Just like the canvas element.

[rcabanier]

I guess I need the explainer because I'm not sure how this is supposed to work, or whether it is useful, or why anyone would want to do it.

Yes, sorry. The hurricane caused our office to close and most of my team fled to higher ground :-)

I assume the idea is that you're looking at a Web page in some AR/VR space, and you want the elements in that page to project out (or back) from the flat surface into the hosting space?

Correct. Elements will be able to "break out" of the web page.
There is a Google slide deck on declarative 3D. transform-style: detached would allow you to design such a page and have it still look good on a traditional 2D browser.

Do you expect the hosting space to apply effects to the projected elements?

Standard CSS effects such as blur? Yes (see earlier in this thread)

For example, will they respond to lighting, cast shadows, etc?

That would depend on the window manager of the UA. For Magic Leap, we only have a global light and no shadows but as the technology advances, this would certainly be possible.

If so, that means the hosting space needs pixel access, which we can't allow. If not, the effect will look so weird it isn't worth doing.

I don't know why you need lighting and shadows. The google example doesn't have them either. Don't forget that our browser is a 3D prism in the Landscape. The user can place it anywhere and walk around it. Each eye will get the correct projection of the browser plane and each detached element so it shouldn't look weird.

Of course the window manager needs pixel access but it already has pixel access to the browser plane so there's no privilege change. The important part is that the process that does layout and javascript won't have more access than before.

And what happens as 3d elements are scrolled into view (they were clipped by the viewport)? Do they all of a sudden appear in 3d space? Does half of them appear if they are partially clipped?

Yes. Objects are clipped by what we call a "prism" which is basically a clipping cube in space. The google slides have an example of that as well. (BTW our prismatic library already does this when models scroll out of view)

An open question is what should happen with overflow: hidden.

And why is this just applying to CSS transforms? Wouldn't a better case be having WebGL objects from the page exist in the hosting space?

There is a proposal from @NellWaliczek which is basically that. It's definitely something we want to consider later but right now, we want to do something that can be done easily and with regular CSS semantics (instead of difficult WebGL, lots of javascript and tricky compositing).

My gut feeling is that we should continue to assume that CSS always projects onto a flat 2d surface. Within that surface you can emulate 3d sub-spaces, but the end result is always 2d. Just like the canvas element.

3D transforms already establish a 3D scene. Why not actually make them part of our environment?

Also, we're no longer constrained by 2D surfaces :-)
AR/VR is the next frontier and we want to give the browser access to enable cool new experiences.

[rcabanier]

I started an explainer here.
It's not a full proposal but I will update it with feedback from our team and the CSS WG.

[AmeliaBR]

I agree with @dbaron's proposal, that this can logically be handled by having preserve-3d on the root mean preserve into the external 3D rendering context, if such a context exists & allows it.

transform-style: detached would allow you to design such a page and have it still look good on a traditional 2D browser.

If you want different styles in different display types, that seems like a job for a media query.

As is, I do not understand how detached mode is supposed to work in a standard display.

[css-meeting-bot]

The CSS Working Group just discussed proposal new transform-style: detached.

The full IRC log of that discussion

<dael> Topic: proposal new transform-style: detached

<dael> github: https://github.com/w3c/csswg-drafts/issues/4242

<dael> AmeliaBR: Use case is for 3d environmental displays, VR and AR. Having a way to spec css content should be displayed in 3d in that environment without getting flattened to web page rendering

<dael> AmeliaBR: Rick proposal was to add a new keyword to transform-style property that would indicate this stays in 3d separate from plane of browser rendering

<dael> AmeliaBR: Another option from dbaron is to just use existing transform-style:preserve-3d but if you say it on the body or root and rendering environment can preserve the 3d rendering gets to the overall rendering environment

<florian> q+

<dael> AmeliaBR: Question is would it be web compat or is there content that would look bad if it's not flattened?

<dael> TabAtkins: I generally support dbaron proposal absent compat data showing it's bad

<astearns> ack florian

<dael> florian: I also support it in part for a weird thing with the other way. If you have a small iframe and the page doesn't intent 3d but the iframe does leak 3d you could have intrusive ads even that the page didn't want. If you put it on the root he hosting page does what it wants

<dael> smfr: Don't think we should get too far without Rick

<dael> astearns: That's fair. Rick can ring and I'll ping him about joining a call

<dael> AmeliaBR: If you know anyone working on 3d in immersive environments get them to look at the issue

[NellWaliczek]

Please leave a comment in this issue when this topic is next on the agenda. I suspect several of us from the IWWG would like to be part of the discussion.

[egirard]

+1 to @dbaron's proposal, and to @AmeliaBR's previous summary.

The existing tags (preserve-3d, flat, and auto) could cover all of the usage patterns we need if we permit browsers to support preserve-3d at the top layer for environments that support it. Adding another transform-style seems confusing/unnecessary/superfluous.

Are there compat issues that I'm overlooking?

[astearns]

@NellWaliczek @rcabanier This is still on our agenda, and we'll take it up next week on our Oct 30 call at 9am PST.

[chrishtr]

For overflow: hidden, I suspect that the author is going to expect that the transformed element is visible. What happens today if a hidden element is transformed into view?
Are there's special ruls for clip-path? We need to do some investigation there and try it out with the current code :-)

Well we haven't really decided what preserve-3d itself means.
Yes, that's a bit unfortunate. Was any progress made on this recently?

There has been a lot of discussion of what happens with overflow:hidden and preserve-3d descendants. See for example #918 . overflow:hidden will flatten.

The plan of record arrived at at TPAC this year is to align on Firefox behavior for preserve-3d etc. We should for sure finish that work and ship it inter-operably before adding new transform styles.

Similarly, how would detached interact with atomic operations such as filtering? For example, if two detached boxes have a blur filter on them, would their pixels be blurred as if preserve-3d was used?

Yes, the filters would apply as usual, except the effect would be computed by the system compositor and not the browser's one.
We haven't hooked that part up yet though. Do you envision problems in that workflow?

Yes, I envision potential problems. Effects such as opacity, filter, and blend-mode are supposed to be atomic operations, and that would not be possible if descendant content was detached. For filter-like effects, we can in some cases apply the filter to the detached content (blur or opacity for example). Other filters that involve positioning probably won't work. Clips don't make any sense to apply to detached content.

Two options could be:
(a) Don't apply any effects or clips to transform-style: detached content
(b) Cause flattening of detached content (despite transform-style: detached) if any effects or clips exist on ancestor DOM elements

Option (a) is problematic for at least scrolling, if the scrolling transform applies to the detached content but not its clip.

Following the logic of (b) will I think end up with the same solution we arrived at at TPAC to fix these problems.

[chrishtr]

How about this proposal:

transform-style: preserve-3d on the root element causes no flattening at the root, for user agents which want to display multiple planes in 3D. Otherwise flattening will still occur.

That's it. No other changes. :)

In particular, the previous resolutions about what causes flattening and how preserve-3d is set remain. IOW:

• Any element with transform-style other than preserve-3d flatten into the ancestor element's plane
• All effects and clips cause the used value of transform-style to be flat.

If a developer wants to have a 3D plane that floats above the other content in a user agent that supports such things, they should:

• put transform-style: preserve-3d on the root element of the 3D plane and all ancestors
• Don't have flattening properties on the ancestors
• Place a 3D transform on the root element of the 3D plane

[abalasubramaniyan-ml]

Option (a) is problematic for at least scrolling, if the scrolling transform applies to the detached content but not its clip.

A possible solution would be to apply scroll clip to detached surfaces in non-root scroll areas. This can be done at a platform level by defining a boundary that corresponds to the scroll container, similar to the 'prism' concept which Rik described.

How about this proposal:

transform-style: preserve-3d on the root element causes no flattening at the root, for user agents which want to display multiple planes in 3D. Otherwise flattening will still occur.

That's it. No other changes. :)

In particular, the previous resolutions about what causes flattening and how preserve-3d is set remain. IOW:

• Any element with transform-style other than preserve-3d flatten into the ancestor element's plane
• All effects and clips cause the used value of transform-style to be flat.

If a developer wants to have a 3D plane that floats above the other content in a user agent that supports such things, they should:

• put transform-style: preserve-3d on the root element of the 3D plane and all ancestors
• Don't have flattening properties on the ancestors
• Place a 3D transform on the root element of the 3D plane

This is a very elegant solution. But the one disadvantage would be that any given website would require significant changes if certain elements are to be detached. Detaching any element would require an audit of all its ancestors to ensure they are not flattened.

I am also not clear on all the properties that could cause flattening in a node. Are they the same as the criteria for stacking contexts?

[chrishtr]

This is a very elegant solution. But the one disadvantage would be that any given website would require significant changes if certain elements are to be detached. Detaching any element would require an audit of all its ancestors to ensure they are not flattened.

Yes, it requires changes, but those changes are necessary to make the whole model well-defined. I don't think there is any way to get around that sites need to spend a bit of time thinking through how they will show 3D.

Various CSSWG members have had numerous discussions about this topic; the general consensus at this point is that to make the web well-defined in the presence of 3D, we need to require this kind of work from developers.

I am also not clear on all the properties that could cause flattening in a node. Are they the same as the criteria for stacking contexts?

All clips and visual effects cause flattening. The ED transform spec has a list of them.

[abalasubramaniyan-ml]

I think I understand now.

Under your proposal, it makes sense to prevent detaching descendants of flattening nodes.
But, assuming we had multiple descendants without flattening ancestor under a root with transform-style: preserve-3d, how do we identify which of these should be rendered on detached surfaces?

1) Each to its own surface - I cannot think of any obvious negatives, but it does not seem like a good idea.
2) Only descendants with transforms that lift it out of the page - This introduces some corner cases. Like do we still detach if it has two ancestors, one which lifts it out with translatez and another which puts it back with z=0?

I believe explicitly declaring that these nodes are detached would avoid these issues.

[chrishtr]

But, assuming we had multiple descendants without flattening ancestor under a root with transform-style: preserve-3d, how do we identify which of these should be rendered on detached surfaces?

Any 3D transform will cause a surface. Other transforms draw into the same plane as their parent.

For the same you gave, which I think is:

<html style="transform-style: preserve-3d">
  <div id=A style="transform-style: preserve-3d; transform: translateZ(10px)">
    <div id=B style="transform-style: preserve-3d; transform: translateZ(-10px)"></div>
  </div>
</div>

Then A would render 10px out, and B would render in a different surface that is (arithmetically) co-planar to the <html> element. It would not participate in the z-index style sorting within the <html> element's plane; instead z-buffering would apply. This means that B may not end up on top of <html> reliably. This issue of determining co-planarity was also discussed in the past also, and it was determined that it's computationally intractable to determine due to floating-point accuracy issues.

I don't think it is feasible to provide a way to "pop" back into an ancestor plane for descendant elements.

[AmeliaBR]

Detaching any element would require an audit of all its ancestors to ensure they are not flattened.
…
it makes sense to prevent detaching descendants of flattening nodes.

Agreed that this is a necessity.

The only way I could see this working (detaching despite flattening styles on an ancestor) is with a position: fixed behavior in addition to the 3D transform, where the layout of this element is unaffected by the surrounding layout/styles, and it instead becomes a child of the viewport/canvas within the rendering tree.

Unfortunately, the approach CSS has chosen is that for certain flattening properties, including transform, (and filter being I think the only other one), even fixed position elements get trapped by the effect on the parent element. There were lots of discussions around this at the time these effects were implemented & the decision was that following the DOM tree for these effects (so that a transform or filter affects all child elements, even fixed-position ones) was more important than maintaining the fixed position. It is too late to undo this now, although this could conceivably be a case where a new detached value could change the behavior — though I'd probably see this as a new value for position rather than for transform-style.

On the other hand, if anyone was hoping to have the 3D effect preserved while still maintaining non-fixed positioning (contextual layout and scrolling and so on), you would need to define how the 3D construct gets passed through all parent styles or effects that usually cause flattening.

assuming we had multiple descendants without flattening ancestor under a root with transform-style: preserve-3d

I'm not sure how this would be different compared to what happens currently when we have a <body> or <div> with transform-style: preserve-3d and then multiple children that have independent 3D transforms on them: both the transformed and untransformed children would collectively be rendered into the same 3D model (with the caveats that Chris mentioned about undefined rendering behavior with co-planar elements).

[abalasubramaniyan-ml]

On the other hand, if anyone was hoping to have the 3D effect preserved while still maintaining non-fixed positioning (contextual layout and scrolling and so on), you would need to define how the 3D construct gets passed through all parent styles or effects that usually cause flattening.

If any of the ancestors of an element have a flattening effect either explicitly specified (transform-style: flat) or forced due to a grouping property (filters, opacity, clip etc.), the element cannot not be detached and will be rendered to the main surface.

It is too late to undo this now, although this could conceivably be a case where a new detached value could change the behavior — though I'd probably see this as a new value for position rather than for transform-style.

I agree that transform-style is not the ideal attribute to specify detached modifier, but I don't think position is either. If I am not wrong, positioned elements are still rendered in the parent's render context and it should be possible to detach an element with any position attribute.

[abalasubramaniyan-ml]

Based on discussion here and implementation experience in Helio, we found that transform-style was not the appropriate attribute to use to implement detached behavior. We have published a revised explainer at https://github.com/magicleap/detached_explainer that accounts for this change.

The main sticking point that caused this change was defining the behavior of descendants when a element with transform-style: preserve-3d was the child of an element with transform-style: detached.

We still believe explicitly detaching elements is necessary. @chrishtr recommends implicitly detaching elements with 3d transforms. But this would be difficult to implement in cases where the 3d transform is animated making the surface pop in and out of the root surface.

As described in explainer, we think extending attribute transform to allow detaching current element is a more appropriate solution.

Please advise if a new issue is necessary or if the current issue can be renamed.

[chrishtr]

As described in explainer, we think extending attribute transform to allow detaching current element is a more appropriate solution.

I took a quick read of your new proposal. The semantics appear to be different in only one respect from what I was proposing earlier: a new detached keyword on the transform CSS property that forces the transform to be considered 3D, instead of relying on detecting a "true 3D transform" based on the matrix values.

Is that a correct reading?

[abalasubramaniyan-ml]

Yes that is correct.
The idea is to have the preserve-3d chain set up the 3d context to enable the content to be detached and the detached keyword is what actually detaches it.

[chrishtr]

Ok. I think that's fine and a good refinement of the idea I was proposing, without the problems of accidental 2D transforms.