Revolution: Think over a mechanism for collecting statistics from users in MODX manager

[Ruslan-Aleev]

Feature request

Summary

Often discussing PR or an issue in the dialogue, the phrase "wait for feedback from users" is used.
But how do we collect this feedback? Because the sample of active users on github or in the chat is small.

Even if everyone in the chat voted unanimously, this does not mean that the majority of votes will be the same.

We need some kind of feedback mechanism and statistics collection. Heatmap, linkmap, etc. Maybe a voting form, etc.

We need collect analytics in MODX manager, some CMS use such mechanisms, indicating this when installing the CMS.

Why is it needed?

We can understand what is used by users in MODX, and what is not very much. And in general, to establish feedback.

[Mark-H]

I've got something like that in my premium extras, where a user consents to things that get shared based on pre-defined levels, ranging from only the required data (i.e. version numbers and license info) to a lot of optional data that can provide modmore with interesting insights about how extras are used, but never any personally identifiable information. Largely technical stuff, like if an extra is used in a certain configuration. Here's an example explanation of that.

I also know there's a quite a few free extras on MODX.com (e.g. FormIt) that share environment information on installation/update _without_ explicit consent nor an indication certain information will be shared (nothing in the readme/license), so that's at the very least in a gray GDPR area and something I'd most definitely want the core to do better.

So for one if we go this route, at the very least users must have control over what gets shared or not, and are explicitly asked to consent (perhaps in the setup) to certain information being shared, and it should not ever be personally identifiable.

What are the things we'd want to learn though?

[Ruslan-Aleev]

What are the things we'd want to learn though?

For example, using the MODX manager click map, we can collect statistics on what is not used by MODX users. I'm pretty sure that "HTML Import / Export" is almost never used (or rarely used), and it makes sense to remove it from the core. Less code means fewer bugs. I would start with this.

Collect statistics of countries, understand which translations are in priority.

We also need to somehow stimulate managers to give feedback (which is missing in the manager panel, which is confusing). github is cool, but the MODX manager panel is more often used by managers, it seems to me, and not by developers, while there is almost no feedback from managers.

[ant-tar]

My five cents (maybe this was the reason why Ruslan to write this feature request -if so skip this please) but there is a good illustration for this FR as for me: https://community.modx.com/t/blank-manager-with-undefined-message/3799/16

[Mark-H]

My five cents (maybe this was the reason why Ruslan to write this feature request -if so skip this please) but there is a good illustration for this FR as for me: https://community.modx.com/t/blank-manager-with-undefined-message/3799/16

I'm honestly not sure how we could automatically identify problems as described there, if it's already so difficult to identify it manually. Even sending a copy of any error or failed connector request - if it doesn't accidentally includes PII that would generate so much noise I don't think it would be useful to sift through.

For problems like that topic, where a problem happens in the browser (or at least outside of the typical server-side processes), perhaps just making sure JS errors/failed requests get logged into a dedicated error log along with browser info could work. But that's worth a separate issue from this one, as this is more about analytics than errors.

[Ruslan-Aleev]

Yes, I meant more about analytics, and feedback from users.
For example, we have implemented this PR https://github.com/modxcms/revolution/pull/15535 (showing a preview of the template) and we could ask users whether they like this feature or not, what is missing in feature, etc.
In general, there are a lot of issue / PRs that hang with the "beign-discussed" label, but there are not enough people for this discussion. Such questions could be brought up for general discussion in MODX manager, through voting form in manager panel, for example.

[Mark-H]

Perhaps we can log the namespace + action in modManagerController to something like the db registry, and send the total count for each action at most once a week to an API? That would tell us how often core:import is used as well as core:user/messages or coolextra:index without it revealing identifiable patterns or such. And the same approach could be used to log the active languages by having one language entry per manager user.

At the time those totals get sent to an API, it could also add some static information like the MODX and PHP version.

Another option for only generic environment information would be to send a ping from the setup with the installed MODX/PHP version, but that's not going to have much opportunity to share how people actually use the manager.

Regardless of approach for this, there's also a real potential people will send random or arbitrary data just to mess with the statistics so that's something to keep in mind. :D

We also need to somehow stimulate managers to give feedback (which is missing in the manager panel, which is confusing). github is cool, but the MODX manager panel is more often used by managers, it seems to me, and not by developers, while there is almost no feedback from managers.

For asking the manager users for feedback, I'd be a little cautious - some people may misinterpret that for providing feedback or asking for help on their specific sites' implementation. If there are specific manager-related questions we're looking to answer, maybe I'd suggest setting up some online survey and send that around to agencies and developers (through the MODX newsletter, blog feed, etc) asking them to both fill it in, and to also forward it to clients who may be a good fit to provide feedback on it.

[Ruslan-Aleev]

or coolextra:index

Yes, about extras is a very accurate mention. This way we can collect statistics of extras and direct our efforts to testing / improving them, for example, before the release of MODX 3, and not the entire list of modx.com extras. Suddenly it turns out that there are not so many such extras :)

For asking the manager users for feedback, I'd be a little cautious - some people may misinterpret that for providing feedback or asking for help on their specific sites' implementation.

Yes, I totally agree. We have a welcome window after installation, the same window can be done for voting.

[alroniks]

I see here two separate processes. Analytics can be collected automatically, but without identifying the users to respect privacy. And also tools that can allow us to get feedback about features.

Implement the first one now can be difficult and I would like to postpone it a bit. But if somebody will provide PoC, we can it discuss of course. But about the second, it may work in the paid with Roadmap, when we can define Epics and collect feedback via manager before we start implementing it. The easier way just posts surveys and delivers them via RSS.

So, the idea is good but my suggestion to do not it too complex from the scratch and follow Lean methodology, making small steps and improving it with every next step, tweaking destination if needed.