Cht-core: Improve how Muting is handled on device before syncs

[senseibara]

What feature do you want to improve?
The muting feature as it works right now requires Sentinel Transitions to update the relevant contact docs following a muting form submitted by the CHW. Because of very slow/unstable connectivity in the field, we notice some weird situations that arise from it. The most frequent one is that the muting form don't get synced for days/weeks, in the mean time the household or place and its descendants remain active and users sometimes continue to submit forms.
A part from the form submissions, from a UX perspective it is not ideal if the contact is not muted immediately, the users sometimes submit extra muting forms as they don't understand what's happening. Some of our UX features like UHC are affected as well.

Describe the improvement you'd like
We would want ideally a muting feature that can work fully offline and that would be immediately effective after it was triggered.
Another solution that would only prevent the form submissions after the muting, could be a improvement in configuration that would make it possible to check if the last form submitted for any of the parents of a contact or the contact itself is a muting form. we could add that check to the form properties to prevent it from being submitted in those cases ( this might be already possible and i'm just not aware of that ).

Additional context
We have noticed this issue for our partner MUSO, where CHWs continue to submit forms for a place/contact supposed to be muted, this make it confusing and difficult specially from a data analysis perspective, the reasons why those submissions are being done is unclear, is it a mistake ? Was the family not supposed to be muted in the first place ? Would the CHW had unmuted the family first before submitting the visit if the family was muted in time ? Are the chws submitting visits forms without actually visiting the family to not be penalized by the UHC scoring ?

initial ticket : https://github.com/medic/config-muso/issues/44https://github.com/medic/config-muso/issues/44

[abbyad]

Replacing https://github.com/medic/cht-core/issues/6683 in 3.13 with this issue.

[dianabarsan]

Can someone indicate who to discuss the solution / workaround with (since Bara is no longer available)?
Thanks!

[Marina-Kenf]

@dianabarsan I can help with this.

[michaelkohn]

And @eljhkrr

[dianabarsan]

Unsure of what is going on here? is this no longer ready for dev? why have I been unassigned?

[michaelkohn]

Looks like @Fatoufall unassigned, i assume that was an accident?

[Fatoufall]

hey my bad...I wanted to add @maremegaye as PO of Muso

[dianabarsan]

This is ready for AT on 6737-validation-shared-lib. (looking back, I picked a terrible name for this branch :) )

For a more technical explanation, please see: https://github.com/medic/cht-core/pull/7087#issue-631375395
Updated documentation here: https://github.com/medic/cht-docs/pull/504

There are no configuration changes that need to be made in order for client-side muting to work - having muting configured correctly in general is enough. You can opt out from running muting on the client by adding an "client: false" param to the config - check documentation for full details.
I tried to cover most situations that could come up and make up for late syncs, avoid generating conflicts. In the end, we still have two processes that execute an action at different times and in different circumstances, so the results of these actions might not always be the same. The documentation has a notes section that ennumerates some situations where we know that muting on the server will have different results than muting on the client:

• updated muting settings between client and server processing of the same report / contact
• editing the muting/unmuting reports before they are synced, but after the transition ran locally, that either change the target contact or change the validity of the report
• deleting muting/unmuting reports before they are synced
• validation rules that depend on database data (for example using the "exists" rule, which will run over different data sets, a report could be valid for the client but invalid for the server and the other way around).
• conflicts that overwrite muting_history for contacts
• delayed sync for some docs (either contacts or reports) could exacerbate the above because of the "replay" behavior (because replaying means that we run muting over some reports multiple times - if reports are edited significantly between replays, we could get different results).
• multiple clients muting/unmuting the same contacts at the same time will have no guaranteed results - conflicts are going to appear and choosing the winning rev is not deterministic.

Except for some edge cases, you should be able to:

• mute/unmute a contact on the client while having an internet connection (and Sentinel up and running) and not get conflicts on any of the docs (contact or reports) and get the same results as if muting would just run on the server. Same for unmuting.
• mute/unmute a contact on the client while not having an internet connection. Muting on the client only updates contacts (adding or removing the muted property) and does not silence/un-silence registrations. The silence part still only happens on the server. Also, in order to "cancel"/"re-enable" tasks, you configuration needs to handle muted contacts itself (just like before).
• perform multiple mute/unmute actions on the client, while not having an internet connection, and then reconnecting. After sentinel processes all the mutes/unmutes, your contacts should be in the same state as before syncing.
• forms that create and mute/unmute contacts at the same time should work (I have a couple of forms that I used for testing that I'm sharing here: https://gist.github.com/dianabarsan/1921504576f32cd9d369bba89572f573 - these forms are far from perfect or complete - and assume you configured these muting forms: https://github.com/medic/cht-core/tree/master/demo-forms/muting-forms).
• forms that create both mute and unmute reports are treated differently, and only _unmuting_ reports is processed when such a form is submitted (there's an example form in the gist).

Please be aware that

• muting on the client only works for offline users.
• it will only change contacts that are available on the client, with the "full" muting/unmuting running on the server.
• it adds some additional metadata to the contact docs (muting_history) which should be treated as read-only, it adds some additional metadata to the report docs (client_transitions) which should also be treated as read-only.
• the transition only runs on new docs - before they are saved in the database. So if you mute a contact, unmute a contact and then edit the first muting report, your contact should not become muted again. (note: some of the forms in my gists above, when edited, will create new docs every time).