Cht-core: Ability to calculate how many tasks are shown and completed by users

[garethbowen]

To improve our impact we must first measure our impact.

One of our major features is tasks and we make it very difficult to measure how many tasks are being shown to users and how many tasks are being completed. This is due to tasks being ephemeral and generated on the phone so server side analytics queries don't have access to this information. Instead the impact team create custom queries based on the configuration of each project but this is error prone and time consuming.

One solution would be to write tasks to the database and replicate to couchdb but this could have a potentially serious performance impact.

Consider various implementations and analyse the performance impacts of each before implementation.

[SCdF]

Another core issue here is that we'd also want to capture and understand missed tasks. This is tough because we don't actually have a concept of a missed task, task config just emits tasks only in certain windows, and so if you don't complete it after awhile it simply won't be re-emitted the next day. I don't even think we get to introspect those windows at this point either.

[kennsippell]

https://github.com/medic/medic/issues/5540 raises client-side memory issues which should be addressed, fixed, and measured as part of this issue.

• For some Muso Supervisors in production, our RulesEngine code retains >100MB of contact/report data (68% of total retained heap). Avoiding retention is fairly straightforward and should be prioritized (why not 3.7 :D ).
• Target emissions from Nools are also retained (0.5MB measured cost for this sample user without many targets). I made some suggestions for optimizing this at https://github.com/medic/medic/issues/5515.

[isaacholeman]

I was just reading through the FHIR documentation on Tasks. In relation to the problem @SCdF raises above, their task states diagram is interesting. It would be excellent if we were able to solve our problems in a way that aligns with FHIR. https://www.hl7.org/fhir/task.html

[MaxDiz]

@michaelkohn fyi re exploration of FHIR solution noted

[michaelkohn]

I went through a simple pregnancy visit task scenario as an exercise to think about how some of the activities in our app might relate to some of the FHIR task states. This is meant to be more of a conversation starter/thought provoker rather than making any particular recommendation.

We should also have a look at the work we did with the cleared_by field (for SMS projects) in https://github.com/medic/medic/issues/3959 as useful context. I do want to stress the importance of being able to differentiate between "clearing" (Num 6 in the table) and "completing" (Num 5 in the table).

Num | Activity | Visit task "State" | Comments
-- | -- | -- | --
1 | Register Pregnancy | Draft | Not sure if we need this or if we can just start with "Ready" once the task window starts.
2 | Start of Task Window | Ready |  
3 | Task displayed to user (ie "Emitted") | Received | Technically a task may never be displayed to a user if that user doesn't log in to the app / view the tasks tab at some point during the window. I'm not sure we need to make the distinction between Ready and Received, just a consideration.
4 | Some sort of update to the source document changes the task | Cancelled | Updating the EDD, for example. Feels safer to cancel this task and create a new one rather than mess with updating the existing task.
5 | Desired action completes the task | Completed | The desired action for this task is a Visit form, so if we receive a visit form in the window, the task has been completed.
6 | Something else clears the task during or outside of the window. | Cancelled | If a death report has been received, or the woman miscarriages, for example. These will clear the task but I don't really think we should think of them as being completed. On SMS projects we also have the concept of a "pre-window"... you can clear all reminders in a group prior to any being sent. This would happen if the CHW pre-emptively does the visit on time before being reminded to do it.
7 | Patient is muted | Cancelled | We could put them "On hold" i guess, but seems like we could just regenerate new ones if the person gets unmuted.
8 | Patient is unmuted | Ready | Generate new tasks since we "Cancelled" the old ones.
9 | End of task window passes prior to being completed or cancelled | Missed | This isn't one of the defined FHIR states but FHIR allows for this flexibility. The other option might be to mark it as "Failed".
10 | Task rules updated | Cancelled | Cancel all existing Draft/Ready/Received tasks and regenerate using new rules.

Also FWIW (thinking about docs to disk / performance topic), we could probably purge any task doc that wasn't in Draft/Ready/Received status.

[garethbowen]

Draft: Not sure if we need this or if we can just start with "Ready" once the task window starts.

If we write to disk then we will need this because tasks will be generated on doc change rather than on a schedule.

Also FWIW (thinking about docs to disk / performance topic), we could probably purge any task doc that wasn't in Draft/Ready/Received status.

This is definitely a possibility based on the project's configuration. For example, there are cases where we'll want to show targets based on task completion rates. Also I think it may be useful one day to have a filter in the UI to show recently completed tasks. The purge rule could easily be set to purge once it's in an end state and it's a month past its due date.

One additional UI feature I'd like to include at some point in the future is the ability for a user to dismiss a task. This helps to solve the problem where the UI gets cluttered with tasks that can't or won't be completed. I'm guessing we'd use the "cancelled" state but it could also be a "missed" state.

[yembrick]

@garethbowen I love the idea for a dismiss feature. I actually just got off a call with a partner who mentioned they think their biggest data quality issue now is tasks being completed, or even whole workflows canceled (like ending a pregnancy) because the person cant be found when a task is due, and its the only way the CHW knows to clear the task. I've heard anecdotes like this a number of times.

[michaelkohn]

FWIW... here's how the CHT Reference App addresses that dismissal scenario for home visit tasks.

The Pregnancy Visit form has this question:

Do you want to start this pregnancy visit? 
- Yes
- No, Miscarriage
- No, Abortion
- No, Refusing care
- No, Migrated out of area

And if refusing or migrated, we ask:

What would you like to do? 
- Clear task for this visit only. Continue to receive tasks for this pregnancy.
- Do not receive any more tasks about this pregnancy.

[MaxDiz]

Added ability for a user to dismiss a task for future roadmap consideration

[garethbowen]

I've put together a discussion document covering the different options to solve this and some other issues with pros and cons of each.

The only deliverable for 3.8.0 is to ensure that the tasks shown and completed by users are available in the postgres database, however we should take this opportunity to redevelop tasks and targets to enable improvements that are being scheduled for subsequent releases. The implementation that enables the most future features is writing tasks and targets to the medic pouch database and having these replicate to the server.

This solution could have a significant impact on performance so before starting on implementation we need to prove the performance will be acceptable. Once we have a reasonable estimation of expected performance distribute it to the team to get sign off before starting on implementation or changing strategy.

Some performance aspects to consider:

• client side: increase in number of docs (take purging into account here), decrease in memory usage, decrease in app load time, more view rebuilding
• server side: increase in changes feed length, more view rebuilding
• bandwidth: more docs going both ways

[MaxDiz]

discussion has been broadened and moved to this document

[kennsippell]

Ready for AT on tasks-v2

[kennsippell]

Here are some ideas of scenarios you might test for AT @ngaruko

• Login as a user with targets enabled and tasks disabled
• Login as a user with targets disabled and tasks enabled
• Login as a user with tasks and targets disabled
• Tasks appear in the tasks tab, but also the contact's page when you select the relevant contact or that contact's parent.
• Configuration for purging task documents #6181
• This configuration requires an updated configuration schema. Test the system without this updated schema (rules engine should be disabled). Here are the schema updates that are required for it to work: https://github.com/medic/medic-projects/pull/7014
• Read through the spec and confirm its intended functionality. List of differences below.
• Make some tasks appera using a variety of configurations. Confirm tasks appear the same in 3.7.1 vs this build.
• The muso mRDT tasks are interesting outlier cases due to their reliance on headless contacts.
• Confirm that you can see task documents replicate to CouchDB. Confirm their content is as expected.
• Maybe login as a user on two devices, resolve a task on one device, sync both devices, and confirm the tasks resolve on the other device.
• Confirm that task documents are queryable in postgres.
• There is a "freshness" thread that should recalculate tasks and targets after 2 minutes each time the webapp reloads. If you navigate to the tasks/targets tab, then it should get cancelled.
• Performance testing (my phone is broken atm, but I will be taking measures soon)
• Confirm that a task which is based on the current date-time will refresh after 7 days.
• Tasks and targets should recalculate whenever app_settings changes include: permissions to view tasks/targets, task schedules, rules code, or target configuration. Try updating the configuration via medic-conf or via fauxton and see if tasks update appropriately!

[kennsippell]

Known Issues:

• If a user has the can_view_tasks permission, creates some task documents, and then loses that permission. The user's task documents will be left as-is and will not be updated.
• #6181 - Cannot purge task or target docs

Here are some differences between the implementation and the spec.

• The schema of the task document is a little different than the proposed. Here is an actual example of how things turned out: https://github.com/medic/cht-core/blob/targets-v2/shared-libs/rules-engine/test/transform-task-emission-to-doc.spec.js#L222
• The task document schema does not include a "source" attribute on the "stateHistory" transitions as proposed. So we won't know "why" state changes happened. I think this would be a nice improvement though.
• There is no "debug-only" button to force the recalculation of tasks/targets
• There is no "recalculateTasksAtInterval" attribute in the app_settings. We always recalculate tasks/targets for each contact once per week.
• Instead of ensuring tasks/targets are freshened by a weekly background thread, we freshen tasks/targets for all contacts that haven't been refreshed within the week in the background each time the app loads.
• We did not mitigate the catastrophic failure which may result from faulty or overactive task configurations
• No "live updates" on the targets tab. For example, if the user syncs data which changes the result of their targets they will need to reload the target tab to see it.
• We said we would have one target document per user per month which is created at the end of the month. Instead we have one target document per user per month which is updated each time the user views the target tab. We should probably change this to happen once per day at max.
• There was talk of adding a can_view_targets role, which would be different from our existing can_view_analytics. For now, I'm just re-using the same role.
• Tasks are considered only if they expire within 60 days. We did not implement the "smart window" which was discussed in this channel.
• There is no telemetry for tasks/targets. I personally feel we should add this in 3.8 to better understand how the system is performing in production.
• We discussed having a custom error on the UI if a partner deploys the old configuration schema. Instead, we re-use the existing error UI and have a prescriptive error in the console. 'Rules Engine: Updates to the nools schema are required' . I think this is okay because this is an error to be seen/fixed by a tech lead, not something that is CHW-facing.

Some changes in behavior from Tasks/Targets v1:

• Tasks with multiple actions cannot target different contacts: So one task can't have an action to complete a form for contact x and another action to complete a form for contact y. All actions must target contact x.
• Targets emissions with conflicting ids were previously merged based on the order in which the requestor contact was created. It is now merged based on the requestor's reported_date.
• Multiple contacts emitting task emissions with the same id is no longer supported
• Reports with errors are now passed into nools both on startup and when those documents change
• This is no longer the case. Tasks now disappear whenever appliesIf is false, but they task document moves to state Cancelled.

once a task appears (appliesIf: true, resolvedIf: false), that task will not disappear until it is explicitly told to resolve (appliesIf: true, resolvedIf: true).

[yrimal]

@kennsippell,
The following line may need to be updated. In the task that's written now, it's not chw but patient name.

https://github.com/medic/cht-core/blob/4556aa4ae22cd19af2315c5c243ae66c07170971/shared-libs/rules-engine/test/transform-task-emission-to-doc.spec.js#L244

[kennsippell]

@yrimal That name you linked is the name of the mock contact which is owning the report in this integration test -- in this case, it is set here in the mock contact object.

I agree it is maybe a little confusing to have that contact named "chw". Is that what you're suggesting here -- that the mock contact should not be named "chw'?

[yrimal]

@kennsippell It's clear now. I thought that was template and we were writing chw's name in {emission,contact,name} but turns out it was patient named chw.

[kennsippell]

@yrimal https://github.com/medic/cht-core/pull/6041/commits/8f8c16379e48c0c1769b9a86d3b0ea89b150faf6

[kennsippell]

Results of parity testing on LG-IN, comparing 3.7.0 to tasks-v2 branch:

1. Target which are using translation arrays instead of translation keys are not getting properly translated. Fix: https://github.com/medic/cht-core/pull/6041/commits/92920b84d761cf15eb1a584de4a9bebce17ae175
2. I'm seeing a slightly different list of tasks for user aliceke on Dec 12. Of the 67 tasks showing on that day for that user, there are 4 new tasks which appear in 3.8 which do not appear in 3.7. https://github.com/medic/medic-projects/issues/7116
3. The pregnancy registration target is showing 1 for aliceke on Nov 12 on 3.7, but showing 0 on 3.8. Fix: https://github.com/medic/medic-projects/pull/7108
4. A task is showing as "high priority" on 3.7 but not on 3.8. Fix: https://github.com/medic/cht-core/pull/6041/commits/54185968549ea27716040b36e8d4fd5d81c4c698

Parity tests are passing for aliceke on another date/time, and for two datetimes for users alicekw and linetod.

That concludes parity testing on LG-IN -- two good bugs and two config issues.

[ecsalomon]

Very happy with how 3.8 is working in testing. Thanks, @kennsippell for this awesome upgrade and your support in getting up to speed on using the changes. I believe this is official Research & Learning Team sign off!

Only a couple things came up during testing:

1. It would be nice to have URL of the task be the task doc id instead of the id of the emission (see #6139)
2. It is now much easier to diagnose configuration bugs where tasks do not fail properly (https://github.com/medic/medic-projects/issues/7128)