Cht-core: Ability to generate case_id short codes

[benkags]

Is your feature request related to a problem? Please describe.
We have a really great feature that allows the generation of patient_ids for contacts. It would be even greater if we had a similar feature for reports too. With it, we would be able to generate a unique identifier sharable to a user. This is particularly useful for projects that need us to respond to the user with code that they can use to identify a report they sent to the system, especially with TextForm workflows.
One of the projects with this need is the Event-Based Surveillance (EBS) project deployed for MOH Kenya where a user reports an event and expects to get back a code for that signal/event they just reported.

Describe the solution you'd like
Ability go auto-generate shortcodes in reports.

Describe alternatives you've considered
For the EBS mentioned above, the patient_id generation feature was repurposed to meet the requirement where the signal reporting form is configured to add_person. When a report is submitted a contact is created, and it's patient_id is attached to the report which we are then able to send back to the user.

Additional context
Add any other context or screenshots about the feature request here.

[kennsippell]

@MaxDiz App Builders Backlog?

[kennsippell]

We are scaling our existing EBS systems to new counties and spinning up new EBS systems for our COVID response. Project teams feel this was the fifth most important issue to prioritise for COVID EBS scale-up response behind #5817 #6148 #6286 #6094.

[abbyad]

Fleshing this out a bit further so that we can move forward with case based workflows.

Requirements

• [ ] Must create a unique case_id for all reports that match criteria. For consistency with existing report actions, this could be configured using a trigger (eg new_case), similar to app_settings.registrations[].events[].trigger = 'add_patient'. The IDs must not collide with people or place IDs, and there is no requirement yet that the case_id be formatted differently than other short codes.
• [ ] Should show the case_id as a generated field in the report view. This would be similar to how the person ID is displayed.
• [ ] Could have the generated ID in the reports view link to a reports search. ~There is ongoing discussion as to whether we should hold off on a link until we have a case tab with summaries to link to. My impression is that a link to a reports search is beneficial, and users will welcome the improvement to a case page when that is later developed. We should not block on discussion about this, and hold off on this requirement if there is significant disagreement.~
• [ ] Could show the Case ID as the subject of the report. This would show in the LHS and the top of the RHS, and be consistent to patient reports showing the subject of the report.
  

_Edit: after confirming with @michaelkohn, the approach of linking to reports search is suitable for now. Crossed out original text for posterity._

[abbyad]

Marking priority as _High_ since upcoming deployments should use this feature rather than extending the alternate method which more knock-on effects (eg unnecessarily creates a dummy person for each case).

[garethbowen]

Related slide deck: https://docs.google.com/presentation/d/1IwJyz5BwoD5xOGb0P5S1jm30ntqbIbvsmlfxL_smJHo/edit#slide=id.g834960f923_0_13

[garethbowen]

Prototype thusfar...

Given this configuration:

"registrations": [ {
  "form": "8",
  "events": [ { "name": "on_create", "trigger": "add_case" } ]
} ],
"forms": {
  "8": {
    "meta": { "code": "8", "icon": "icon-healthcare", "translation_key": "form.signal.8.title" },
    "fields": {},
    "use_sentinel": true
  }
}

When this SMS message is received: 8

Then the case ID is generated and the sender is attached as the contact.

Then the supervisor signs off using an app form citing the case ID:

Now a user can click the case ID link and see both relevant reports.

I haven't added this optional feature: "Could show the Case ID as the subject of the report."

[michaelkohn]

OK. So in your scenario, the person submitting the original signal is a contact called "Number Two". To me it seemed a little weird that from the supervisor's view, both reports look the same on the Reports tab (other than the form label) even though they were submitted by different people. I would have expected at least the breadcrumbs to represent who submitted the report.

Also, re: not showing the Case ID... if it's a lot of work, it's better to not hold up the rest of this, but just looking at some data from a live EBS project, there are definitely many instances of the same CHW submitting a signal more than once per week (often the same signal code). I think it could get confusing to not know which case is which.

AT note: We should also make sure the supervisor workflow works on Collect.

[garethbowen]

OK. So in your scenario, the person submitting the original signal is a contact called "Number Two". To me it seemed a little weird that from the supervisor's view, both reports look the same on the Reports tab (other than the form label) even though they were submitted by different people.

You're right - that was me being lazy in my developer environment and using the same user for both roles. The title and breadcrumbs show the _submitter_ of the SMS and xform so assuming they're not both submitted by the same person they will appear different.

I'll think about how to work the case ID in to the list without blocking 3.9.0 and let you know how I get on...

[michaelkohn]

Cool thanks. I also spoke with @benkags and @joyannee this morning and they agreed that this looks like it should work. They also agreed that it would be much better to have the Case ID, especially as things scale, but that it should not be a blocker for the rest of the issue.

[garethbowen]

Ready for AT in 6291-case-id. This PR also fixes #6356 and #6149 so test those at the same time!

[joyannee]

cc @benkags

[garethbowen]

@michaelkohn The PR now includes a solution for labelling the report usefully.

JSON forms can provide a new key in the meta, eg:

"forms": {
  "8": {
    "meta": {
      "code": "8",
      "icon": "icon-healthcare",
      "translation_key": "form.signal.8.title",
      "subject_key": "signal.registration.subject"
    },
    "fields": {},
    "use_sentinel": true
  }
}

Enketo forms can also provide a subject_key, eg:

  "internalId": "signal",
  "title": "Signal signoff",
  "subject_key": "signal.list.subject",

If that property is given then instead of showing the subject or sender of the report we will translate the key passing in a summary of the report as translation parameters so you can provide a translations like...

    "signal.list.subject": "Case ID {{case_id}}",
    "signal.registration.subject": "Case registration {{case_id}}"

I've yet to write up the documentation on this but I think it'll do everything you asked for. Feel free to try out the 6291-case-id branch if you want to get a head start on AT!

[benkags]

Thank you @garethbowen for this.

[abbyad]

Looks great for case submissions! The case_id is added, and the subject is easy to set.

Some open questions:

• How do we create a task from these reports? In other words, can a task be created based a report that is not about a person/place?
• Can we see these reports on contact profiles? The contacts profiles only show reports about the contact. This makes sense in some contexts, but with case-based reporting it could be anticipated that all the reports be seen on the profile of the person that submitted them -- both Michael and I had this implicit expectation when we were testing it.
• How do we deal with self reporting? Since these reports are not currently shown on the person's profile, it is clear that this issue doesn't replace issue #6286.

[garethbowen]

How do we create a task from these reports?

Unless we've regressed it is possible to generate tasks from "orphan" reports. This will need to be confirmed.

Can we see these reports on contact profiles?

Specifically which profile should it be shown on? I assume the place for which the sender is the primary contact?

I think this feature would be easy enough to add. Is this a blocker for this issue? Can we raise it as a separate issue? Is there a workaround you can think of?

In fact, this may be solved by the work @dianabarsan is doing on #6286 ...

[dianabarsan]

My work on #6286 is to create a new transition that hydrates doc.fields.patient_id (and/or doc.fields.patient_uuid) from the sender of the report's data - for forms that are configured to do such.

So, if the case_id report is submitted by a known phone number and the new "update_patient" transition is enabled for the respective form, you would see the it on the sender's profile.

However, if the patient contact is added afterwards (after the transition has ran), then it won't.

[benkags]

I am looking forward to testing this as soon as I am able to but here are few comments.

The reason for #6286 was because without the patient_id in the report you are not able to manage schedules(and likely tasks) scheduling and clearing, especially schedule clearing because it is predicated on reports being associated with the particular patient whose schedules are to be cleared. The ability to link the case reports for purposes of message scheduling and task management is critically important here. Echoing @abbyad 's concerns, I am wondering how this is meant to work if we are relying on #6286.

Say you have a case managed by the forms 'A' and 'B', which are case reporting and case verification forms respectively. Report by form 'A'(report A by user X) generates a case and a case ID is generated, thanks to this feature. We use this case to both send an acknowledgment message to the reporting user and schedule verification reminders to user Y, the verifying user. If leveraging #6286, report A will have user X’s patient_id, while report B will have user Y’s patient_id. This means that it is not possible to clear message schedules and possibly resolve tasks for a case because these reports do not belong to the same ‘patient’.

[abbyad]

These seem like hidden requirements of this issue:

• Case reports should be shown as reports on the parent place of the person who first reported them (unless of course the place ID is provided in the report).
• Subsequent case reports for the same case ID should also be seen on that place, even though they won't have the place ID provided.
• SMS notifications on a case report can be cleared by subsequent reports with the case ID, similar to patient_reports.

Solving for these would take care of case reporting, without relying on pretending that they are self reports about a person.

[abbyad]

Also, thanks @garethbowen for confirming that can create tasks based on "orphaned" reports. @medic/quality-assurance creating tasks on these case reports should be part of the AT for this issue.

[garethbowen]

@michaelkohn @abbyad @benkags Ready for AT in 6291-case-id. This PR also fixes #6356 and #6149 so test those at the same time!

This was a reasonably large rework so there's some additional configuration required, and the reports created with the previous version of this branch probably won't work 100% - I suggest you start with a fresh case registration.

In addition to the above the latest version adds a transition called case_reports which is very similar to accept_patient_reports. The case registration still uses the registration transition to generate a case_id. Subsequent reports should cite the case_id and trigger the case_reports transition. This transition can be configured to validate that the case_id matches a known case (as well as any other validations), can silence reminders on registrations, and will assign the place_uuid to the report so that it shows up in the reports for the place that had the initial registration. The documentation PR has been updated with information about how to configure the transition.

[abbyad]

I did a walkthrough of a case-based workflow to give it a test run, and it is working really nicely! Case reports were seen on the submitter's place, and were cleared by SMS like patient reports.

We'll post more about the config as part of the feature education, but in the meantime here are some snippets that may prove useful for others exploring the feature:

Case Reports

    "accept_case_reports": [
        {
            "form": "CLEAR_COVID_SIGNAL",
            "validations": {},
            "silence_type": "signal_reminders_to_CHA",
            "silence_for": "2 days",
            "messages": [
                {
                    "event_type": "report_accepted",
                    "bool_expr": "true",
                    "message": "Case resolved",
                    "recipient": "parent"
                }
            ]
        }
    ]

Form

        "8": {
            "meta": {
                "code": "8",
                "icon": "icon-healthcare",
                "translation_key": "form.8.title",
                "subject_key": "form.8.subject"
            },
            "fields": {},
            "use_sentinel": true
        },
        "CLEAR_COVID_SIGNAL": {
            "meta": {
                "code": "CLEAR_COVID_SIGNAL",
                "icon": "icon-healthcare",
                "translation_key": "form.clear_covid_signal.title",
                "subject_key": "form.clear_covid_signal.subject"
            },
            "fields": {
                "case_id": {
                    "position": 1,
                    "type": "string",
                    "labels": {
                        "tiny": {
                            "en": "id"
                        },
                        "short": {
                            "en": "Case ID"
                        },
                        "description": {
                            "en": "Case ID"
                        }
                    },
                    "list": []
                },
            },
            "use_sentinel": true
        }

Schedule

    "schedules": [
        {
            "name": "signal_reminders_to_CHA",
            "summary": "",
            "description": "",
            "start_from": "reported_date",
            "messages": [
                {
                    "translation_key": "messages.schedule.signal.reminders.1",
                    "group": 1,
                    "offset": "8 hours",
                    "send_time": "",
                    "recipient": "parent"
                },
                {
                    "translation_key": "messages.schedule.signal.reminders.2",
                    "group": 1,
                    "offset": "16 hours",
                    "send_time": "",
                    "recipient": "grandparent"
                }
            ]
        }

Registration

    "registrations": [
        {
            "form": "8",
            "events": [
                {
                    "name": "on_create",
                    "trigger": "add_case",
                    "bool_expr": "true"
                },
                {
                    "name": "on_create",
                    "trigger": "assign_schedule",
                    "params": "signal_reminders_to_CHA",
                    "bool_expr": "true"
                }
            ],
            "validations": {},
            "messages": [
                {
                    "translation_key": "messages.covid.signal.report_accepted",
                    "event_type": "report_accepted",
                    "recipient": "reporting_unit"
              }
          ]
      },

Transitions

    "transitions": {
        "registration": true,
        "update_clinics": true,
        "accept_case_reports": true
    }

Note to @medic/quality-assurance that these snippets do not fully test the feature.

[abbyad]

In app_settings.json there is the accept_patient_reports transition which is configured in the patient_reports section. For case reports we have a case_reports transition, and the case_reports section. I like that the section name matches the transition name, but think it would be easier for app builders if it was consistent across both since these transitions are conceptually similar and behave similarly as well.

This difference is more evident when you look at the documentation:

There are a variety of other transitions that have names for historical reasons (eg update_clinics), so perhaps we'll get a chance to make them all consistent at some point, but figured I'd bring it up before this gets merged.

[abbyad]

Note for AT that case_id generation will occur even when case_reports transition is disabled. Using the following transitions will still generate the case_id and schedule seen in the screenshot below.

    "transitions": {
        "update_clinics": true,
        "registration": true,
        "case_reports": false
    }

@garethbowen, this makes me lean towards naming the case_reports transition accept_case_reports since it is (slightly) clearer that it is dealing with incoming case reports, and not everything about case reports.

[garethbowen]

Updated transition name from case_reports to accept_case_reports.

[garethbowen]

For AT make sure that tasks can be configured to be created and resolved based on case registration and signoff reports respectively. It may be that this won't work due to the reports not being patient related.

[garethbowen]

I've patched the PR to enable generating and resolving tasks - please reinstall to test the latest version. There's a related issue ( https://github.com/medic/medic-conf/issues/325 ) causing errors on task generation. To get around it use the 325-handle-unknown-contact branch of medic-conf (which you could AT at the same time if you choose!).

[newtewt]

@garethbowen can you expand on this part? I'm not real sure if all reports that have that case_id should move there or specific reports or there is a config to do that?

assigning the relevant place to the report if a registration with the given case_id exists,

[newtewt]

@garethbowen After our chat, this is what I'm seeing when sending a follow up SMS with the case_id.

I created a hierarchy like this

Health Facility 1 --> Tom's Area -->
Health Facility 1 --> Till's Area --> Bilbo's HouseHould

My original SMS case_id report was submitted using Till's phone number. I think followed up with an SMS confirmation report from Tom's phone number. The second report is being associated to Tom's area instead of where the case_id was originated on Till's area.

[garethbowen]

@newtewt Can I have a look at the second report? I'm particularly interested in if it has a place_uuid (which should be set by the accept_case_reports transition), a place_id, or a patient_id.

[newtewt]

@garethbowen I'm seeing something a little off with tasks when I submit via an eketo form.

When I view the household I see 4 tasks generated. 2 with the names of the contact and 2 blank. When viewing the tasks tab I only see 2 blank tasks.

[dianabarsan]

It looks like @newtewt has a case of a duplicate (triplicate) task problem. This is probably something different and related to task generation.

[dianabarsan]

Good news, everyone!
I managed to replicate the duplicate task issue that @newtewt ran into yesterday. It's contained within this PR's code and is a side-effect of indexing reports by place_uuid in reports_by_subject.

To replicate:

1. have a xform that you can submit at place level (@newtewt used the "ON" form from config/standard and updated the properties so it can be submitted at place level). I'll refer to this as FORM for simplicity.
2. configure registration to add_case for FORM
3. configure a task when FORM is submitted
4. create a CHW at Health Center level and login as this CHW
5. Create a Clinic and submit FORM for this clinic (calling this CLINIC)
6. Force sync, so you get the case_id and place_uuid on this new report
7. Visit tasks page and CLINIC and verify that you have 1 task generated by your report
8. submit FORM for your home place (the health center the chw belongs to)
9. Go to Tasks page (you may observe tasks that have no title, but this is not necessary)
10. Go to CLINIC contact page and see you have 2 tasks, the one from before and a new one that has no "title"

Every time you repeat steps 7 - 9, you should get additional duplicates of your original task.

The reason this is happening is:

• when we receive a change, we determine which contact this change belongs to and we mark this contact as dirty
• when we received the report for our home_place, we marked it as dirty. it doesn't need to be the same form, I just used the same form for simplicity. NB that simply editing the home_place will cause recalculation across all contacts, because we clear rules cache when anything in user lineage is edited, and will not trigger this behavior.
• when we loaded the tasks page, we only recalculate dirty contacts. and now our only dirty contact is our home_place
• because we index reports by place_uuid and registration (or accept_case_reports) populate this field with the submitter's home place, every FORM we submitted has home_place as a subject.
• when we recalculate emissions for home_place, we're now receiving all the FORM reports, but without loading their "contacts" - this means that we're not loading the existent tasks, so we don't detect we're actually creating duplicates.

[ngaruko]

Thanks @dianabarsan and @newtewt . I think we could push this back into In Progress for visibility... and someone will look at it! Right @garethbowen ?

[newtewt]

@abbyad when I submit a report that triggers schedules for parent and grandparent I have noticed there is some odd mapping. If I submit from any level from Household down it maps the parent as the district_hospital and the grand parent as the sender of the report. @dianabarsan has found the cause of this. We are not sure if this is an issue that would be an impact to these workflows. Are you able to verify?

tagging @michaelkohn and @benkags since I believe Marc might be out today.

[abbyad]

That means that you couldn't use parent or grandparent correctly, right? I imagine we'd want to map those correctly as they are commonly used in SMS workflows.

[dianabarsan]

The reason the mapping fails is because we forcefully map parent to the health-center for backwards compatibility here: https://github.com/medic/cht-core/blob/master/shared-libs/message-utils/src/index.js#L162

[abbyad]

Ah, thanks @dianabarsan. I hadn't realized that parent and grandparent weren't working in a generic way. That doesn't seem like a blocker then.

[garethbowen]

I can't find any documentation for values for recipients but looking at the code I think you want to specify something like ancestor:<type> to send it to a specific level of the hierarchy.

I think this is a separate issue that potentially affects all message sending. Please raise an issue if this isn't documented well enough, if there's some workflow that's not possible, or if it's buggy.

[garethbowen]

Ready for AT again.

[newtewt]

I think this is ready to merge. Not sure who's going to handle that @garethbowen @dianabarsan. I've re-ran through the test cases described in QA channel along with Diana's reproduction steps. Also tried with self reporting on too. It is working as expected. Can close #6149 and #6356 as well.

[garethbowen]

Merged!