React-native-windows: Components in rnwcpp need to show mouse states

[YuliKl]

kikisaints: I'm taking over this issue to post the draft proposal for this spec in an easily accessible place where people can provide feedback, as a gist is too private.

Mouse Event APIs

Summary

Components need the ability to handle pointer events and implement custom logic when interacting with a React Native app using any pointing device. This proposal outlines the basic props and events needed to enable those scenarios.

Motivation

There are several use-cases where end users may interact with react-native applications using a pointing device.

These include:

• Surface, iPad and Android tablet devices with a pen or mouse device connected or being used with
• react-native-windows apps that will be deployed on PCs and laptop computers
• react-native-web apps that could be used in any end point including Chromebooks, PCs and tablet devices
• Any React Native app that targets a platform that supports pointer device(s) (e.g. HoloLens)

Pointer/pointing device = generical term for any device used to control the movement of a cursor on a display screen.

Scope

This proposal deals with the minimal set of APIs needed to achieve fundamental pointing device support. The exact details of the scope of this proposal can be seen in the "Pointer events" section below. All Events listed there are musts.

Examples

Basic pointer event handling

A simple example of how to handle a hover over/pointer over an element. In this case a View component.

  <View onPointerEnter={this._onPointerEnter} onPointerLeave={this._onPointerLeave}/>

  private _onPointerEnter = (event: IPointerEvent) => {
    this.setState({ pointerOverElement: true });
  };

  private _onPointerLeave = (event: IPointerEvent) => {
    this.setState({ pointerOverElement: false });
  };

Custom pointer event handling in native components

In the following example, the app's logic takes precedence when certain keystrokes are encountered at certain event routing phases in the View before the native platform can handle them.

  <View onPointerDown={this._onPointerDown} pointerDownEvents={handledNativePointerEvents} />

  const handledNativePointerEvents: IHandledPointerEvent[] = [
     { button: 0, eventPhase : EventPhase.Bubbling },
  ];

  private _onPointerDown = (event: IPointEvent) => {
    if(event.nativeEvent.button == 0){
            //do something AFTER the native control has had a chance to handle it (eventPhase = Bubbling)
    }    
  };

Detailed design

The APIs being introduced here will follow the models created by :

• A React Native discussions and proposals issue submitted by contributor empyrical
• The corresponding W3C Web Pointer events

Pointer events

The following events will be introduced on the View component, as it covers the most common/prevalent use cases where listening for a mouse event (of any kind) would be needed.

Other individual components where they may be needed, can wrap a View around their desired element/component to capture the mouse events for the children within.

| API | Args | Returns | Description |
|:---:|:----:|:-------:|----|
| onPointerOver | IPointerEvent | void | Fires when a pointing device is moved within the hit test boundaries of a element. |
| onPointerEnter | IPointerEvent | void | Fires when a pointing device is moved into the hit test boundaries of an element, including its children.|
| onPointerDown | IPointerEvent | void | Fires when a pointing device's button (or buttons) state is non-negative. |
| onPointerMove | IPointerEvent | void | Fires when a pointer changes coordinates when within the hit test boundaries of an element.|
| onPointerUp | IPointerEvent | void | Fires when a pointing device's button (or buttons) return to negative from being non-negative. |
| onPointerLeave | IPointerEvent | void | Fires when a pointing device is moved out of the hit test boundaries of an element and all of its children. |
| onPointerOverCapture | IPointerEvent | void | Occurs when the onPointerOver event is being routed. onPointerOver is the corresponding bubbling event. |
| onPointerEnterCapture | IPointerEvent | void | Occurs when the onPointerEnter event is being routed. onPointerEnter is the corresponding bubbling event.|
| onPointerDownCapture | IPointerEvent | void | Occurs when the onPointerDown event is being routed. onPointerDown is the corresponding bubbling event. |
| onPointerMoveCapture | IPointerEvent | void | Occurs when the onPointerMove event is being routed. onPointerMove is the corresponding bubbling event.|
| onPointerUpCapture | IPointerEvent | void | Occurs when the onPointerUp event is being routed. onPointerUp is the corresponding bubbling event. |
| onPointerLeaveCapture | IPointerEvent | void | Occurs when the onPointerLeave event is being routed. onPointerLeave is the corresponding bubbling event. |

Where IPointerEvent will be a new event type added to ReactNative.NativeSyntheticEvents of type INativePointerEvent.

INativePointerEvent is a new interface and will expose the following properties:

| Property | Type | Description | Default |
|:---:|:----:|----|:--:|
| button | number | Read-only property that indicates which button was pressed on the mouse that triggered an event fire.

0 - Main button
1 - Auxiliary button
2 - Secondary button
3 - Fourth button
4 - Fifth button
5 - Sixth button, typically pen eraser | -1 |
| buttons | number | The buttons property gives the current state of the device buttons as a bitmask.

1 - Left Mouse, Touch Contact, Pen contact
4 - Middle Mouse
2 - Right Mouse, Pen barrel button
8 - X1 (back) Mouse
16 - X2 (forward) Mouse
32 - Pen eraser button | 0 |
| eventPhase | EventPhase | Current routing phase for the event. | Bubbling |

Where EventPhase is an enum to detect whether the pointer button is being tunneled/bubbled to the target component that has focus. It has the following fields:

• None : none
• Capturing : when a pointer event is being captured while tunneling its way from the root to the target component
• AtTarget : when a pointer event has reached the target component that is handling the corresponding event
• Bubbling : when a pointer event is being captured while bubbling its way to the parent(s) of the target component

Note: In the implementation of these events, the properties in NativeSyntheticEvent like target, bubbles, cancelable etc., should be hooked up. For now, we shall follow the same behaviors for these as other events in react-native today.

Declarative properties

To co-ordinate the handoffs of these pointer events between the native layer and the JS layer, we are also introducing 2 corresponding properties on the View component. Those are:

| Property | Type | Description |
|:---:|:----:|----|
| pointerOverEvents | IHandledPointerEvents[] | Specifies the pointer over events that are handled in the JS layer by the onPointerOver/onPointerOverCapture events |
| pointerOutEvents | IHandledPointerEvents[] | Specifies the pointer over events that are handled in the JS layer by the onPointerOut/onPointerOutCapture events |
| pointerDownEvents | IHandledPointerEvents[] | Specifies the button or buttons that are handled in the JS layer by the onPointerDown/onPointerDownCapture events |
| pointerMoveEvents | IHandledPointerEvents[] | Specifies pointer movement events that are handled in the JS layer by the onPointerDown/onPointerDownCapture events |
| pointerUpEvents | IHandledPointerEvents[] | Specifies the button or buttons that are handled in the JS layer by the onPointerUp/onPointerUpCapture events |

Where IHandledPointerEvents is a new type which takes the following parameters:

• a number parameter named button to declare the pointer button that is of interest to the JS layer
• a constrained string parameter named pointerState to declare the pointer state (string values can be one of the following: over, enter, move, leave) that is of interest to the JS layer
• an eventPhase paramter of type EventPhase to declare the routing phase of interest to the JS layer.

Adoption strategy

This will be a new API and not a breaking change. This is being implemented in the react-native-windows out-of-tree platform first to validate the APIs and implementations. Once vetted, we propose to add this to react-native and add documentation in the official API documentation.

How we teach this

These APIs should be presented as a continuation of React and Windows patterns. As such, it should be very familiar to existing web/React developers as well as desktop developers who can relate these APIs to concepts they already know.

Once implemented, these APIs should be documented as part of official react-native API documentation.

Open Questions

[ahimberg]

The vnext implementation today will fire onMouseEnter, onMouseLeave, onMouseMove events if they have been registered for on elements only.

[harinikmsft]

@ahimberg - are those new APIs introduced in the vnext implementation? At what layer (each component or on View/TouchableXX)? If these are new APIs, we may want to change the API names to be more about PointerXX instead of MouseXX in accordance with UWP Pointer events as well as to match the React API surface

[kmelmon]

Load balancing this over to kmelmon

[FalseLobster]

@kikisaints @jonthysell -- Some early thoughts

• Enter/Leave are the only ones in the web that do not Bubble. They're direct-only and fire if pointer enters the element or any descendant.
• In contrast, Over/Out do bubble, but they don't fire when a pointer enters a descendant.
• These behaviors make sense to me. What xaml does I find quite confusing. It bubbles, but also fires for descendants, and relies on handlers to block native bubbling behavior so that the event doesn't leak?

Someone should verify these statements I'm making, they're mostly based off my interpretations of specs & documentation.

We'll also need to reconcile the various IHandledPointerEvents props with the existing CSS-inspired pointerEvents prop. That prop is sort of what inspired my IHandled... keyboard implementation, but the prop seems overloaded. As far as I can tell, pointerEvents both dictates JS-side hit testing and native event propagation.

[chrisglein]

Hover implementation on Web was added in this patch.

[chrisglein]

I think our overall goals are:

• Windows and macOS have the same mouse APIs as offered by react-native-web
• The behavior is consistent (it was mentioned that the Messenger team has had to deal with inconsistent behavior - let's get issues on those)
• Longer term as these 3 out of tree platforms are aligned we push those APIs up to core and reconcile with iOS/Android

This issue is from quite some time ago and doesn't seem up to date with those goals. We need to get to clarity on where we are versus where want to go. What of this proposal is wanted now, what is wanted eventually, and what is no longer relevant?

[chrisglein]

One way to go about it would be like the Keyboard reconciliation doc, where the overall plan is captured there and there are fine grain issues linked from there.