Wp-calypso: (5P) Editor tracking implementation is brittle and subject to breakage

[getdave]

We have an anonymous tracking system on the Block Editor on Dotcom (see PCYsg-nrf-p2).

Currently, the implementation of this system (necessarily) relies on a few unstable APIs including:

• features marked as "experimental" in Gutenberg Core.
• Gutenberg HTML markup - which is not considered a public API for the Editor.

This makes the tracking susceptible to subtle breakages leading to data loss for our status.

For example, recently tracking of "no results" in the block inserter/picker was broken because of a regression in a Gutenberg which caused some DOM elements to go missing. This was only picked up by chance as there was no warning that things were broken.

We need to

• find a way to catch changes in Gutenberg which cause tracking regressions.
• find a way to notify stakeholders when Block tracking is failing for a given event.

I propose the following solutions:

1. Add a suite of automated e2e tests, which verify tracking events against given versions of Gutenberg.
2. Add a notification mechanic which would fire on Production if a given event tracking starts to fail. This might be some kind of dashboard or a system which fires off alerts when the number of failed alerts reaches a given threshold (do we have anything like this already available?). See paYJgx-CC-p2#comment-818 for more on this.

Justification

Whilst this may require a lot of work, the alternative is that our tracking events will continue to fail silently. This will cost us data loss and broken statistics. Given how important some of these stats are for making decisions on priorities, it feels like a sound investment of time.

cc @davemart-in

[davemart-in]

I'd say let's start with item 1 (e2e testing) and see if that remedies things. We can then layer on item 2 later on if it becomes necessary. Sound good?

[getdave]

Here is some existing prior art which runs e2e tests against the Block Editor in Calypso env.

We should be able to add another spec which asserts against all our user actions and ensures events are fired accordingly.

Also https://github.com/Automattic/wp-calypso/pull/41174 helps test against latest Gutenberg.

[obenland]

• Create a plan for how this could work
• Leverage functionality that adds blog sticker for gutenberg-edge

[getdave]

This is being worked on in https://github.com/Automattic/wp-calypso/pull/41329

[getdave]

@obenland Is there any scope for pulling this into one of our upcoming Iterations? Bear in mind that we already have a PR which is at least 50% complete.

https://github.com/Automattic/wp-calypso/pull/41329

[obenland]

Yes, putting it in 116 will give DM a chance to prioritize it

[getdave]

See paYJgx-Es-p2#comment-1213.