Cht-core: Incorrect XLSForm meta data values in Enketo

[abbyad]

In XLSForm there are keywords for meta data, some of which are relevant and available in our app with Enketo. Of those some provide incorrect values, such as start and end which always give midnight GMT for the given day.

To Reproduce
Steps to reproduce the behavior:

1. Upload the meta form
2. Open the meta form
3. See the incorrect values as notes
4. Submit the form and see the same incorrect values being saved

Expected behavior
The start and end values should have the time included with the date.

To maintain consistency with XLSForm it would be nice to also have the user's phonenumber, username, email. That said, those fields are less relevant in our context, and are available via inputs as a workaround.

Screenshots

  "fields": {
    "start": "2019-06-27T20:00:00.000-04:00",
    "end": "2019-06-27T20:00:00.000-04:00",
    "today": "2019-06-27",
    "deviceid": "deviceid not found",
    "subscriberid": "subscriberid not found",
    "simserial": "simserial not found",
    "phonenumber": "phonenumber not found",
    "username": "username not found",
    "email": "email not found",
    "n": "",
    "meta": {
      "instanceID": "uuid:f31d64b7-19ba-43c1-8e34-657c050c81a3"
    },
    "deprecatedID": ""
  },

Environment

• Instance: gamma.dev.medicmobile.org
• Browser: Chrome
• App: webapp
• Version: 3.6 (pre-release), also tested on 2.18

Additional context
Being consistent with XLSForm keywords makes it easier for partners to build their own forms. In this case, a partner wanted to track the start and end time to see how long it takes users to complete a form. Alternatively, the once(now()) could have been used to get the start time, but that would be a bit of a hack and we don't yet support once(). Also, having ODK events and actions (https://github.com/enketo/enketo-core/issues/577) could eventually be used for this as well, but doesn't fully override the need for supporting the XLSForm meta fields.

Also, this issue highlights the need to document differences with XLSForm and ODK XForm standards. If updating the documentation is not covered in this issue we should create a new one for that.

[tookam]

@abbyad I reviewed the openrosa-xpath-evaluator

1 question / 1 remark:

1) Why is our implementation of today and now identical?
https://github.com/medic/openrosa-xpath-evaluator/blob/master/src/openrosa-xpath-extensions.js#L141

2) Related .. It does not look like the current string returned by now() is accurate (returns 2019-07-14T20:00:00.000-04:00 instead of 2019-07-15T12:45:33.917-04:00) which is what I think you noticed.

I am asking because the code has not changed much recently and given that the projects have been using now and today, I am wondering why they have not noticed the "bugs".

[abbyad]

The conversation about today vs now happened in https://github.com/medic/medic/issues/1878#issuecomment-181434710. Can you review the specs to see if anything has changed in the past three years to warrant differentiating them now?

[tookam]

@abbyad This is what the spec says:

By the way, according to the message https://github.com/medic/medic/issues/1878#issuecomment-181434710 we are supposed to be following:

today(): Return today's date
now(): Return a timestamp for this instant
– https://opendatakit.org/help/form-design/binding/

today(): return today's date
now(): return a timestamp for this instant
– https://bitbucket.org/javarosa/javarosa/wiki/xform

So it should match the spec but as you noticed, today() and now() are both returning today's timestamp at midnight. It looks like we need to distinguish between today and now and return the correct values. What do you think?

[abbyad]

It makes sense to follow the ODK XForm spec, but we may encounter timezone issues again, such as https://github.com/medic/medic/issues/1878.

If we assign time to now(), we'll have to be clear about when it` is calculated within the form. Currently it would currently be recalculated on each page turn.

[kennsippell]

Would it make sense to tackle this together with https://github.com/medic/medic/issues/5468? (also scheduled in 3.7). Maybe your fix to this issue would be impacted if the openrosa-xpath-evaluator supported Moment (as suggested by that ticket). Seems like the requirements here are evolving, but thought I'd just bring that idea up.

[abbyad]

I think we should split off the now vs today "bug" as a separate issue.

Introducing moment functions would not resolve this completely either as we would still want to be standards compliant with the existing xpath functions now, and today, and the or xlsform standards for the meta fields start, andend.

[tookam]

@abbyad

1. Could we use once(now()) to prevent multiple evaluations of now()?
2. As for the timezone issue re today(), I think we should always assume that something like 2019-07-17 is local time so that the constraint . < today() will work. I don't think that our current implementation leverages the timezone

[abbyad]

1. Could we use once(now()) to prevent multiple evaluations of now()?

Yes, if once() is implemented - I don't believe it is available yet.

1. As for the timezone issue re today(), I think we should always assume that something like 2019-07-17 is local time so that the constraint . < today() will work. I don't think that our current implementation leverages the timezone

Right, it makes sense for today() to be local date, and omit the time portion.

[vimemo]

The version that I am working provides some default settings to satisfy the odk expectations:
{
allowStringComparison: false,
includeTimeForTodayString: false,
returnCurrentTimeForToday: false
}
These settings can also be overwritten if needed to keep the expectations in our lib. Will check to make sure that we have proper tests for the timezone issue.

[garethbowen]

Ready for AT in 5768-incorrect-xlsform-metadata

[newtewt]

@wtekeu can you summarize what changed? The original ticket details talked about the start and end dates however after switching to this branch it seems they still produce the same time stamps as before.

[tookam]

@newtewt The start and end dates should be different .. this is what I get:

Can you share the link of the instance you are using? I am using the meta form posted by Marc

[newtewt]

I see it now on localhost. Must've been cached or something previously. So the start and end dates will report the same date? I'm using the same form posted above.

[tookam]

@newtewt yes end and start dates are the same because they are computed at the same time

[newtewt]

Cool thanks for clarifying @wtekeu. I'll close this out then.

[abbyad]

So the start and end dates will report the same date?

@newtewt when the report is submitted, in the resulting doc is the start date the same when you started the report (it should not change compared to what you saw on the screen)? Also is the end date updated to the time when the form was completed and submitted?

[newtewt]

@abbyad the start day and end times are recorded as what is displayed on the screen. So if I open the report at 8:54 and then submit it at 9:54 both start and end dates are recorded as 8:54 on the meta form example.

[abbyad]

I am reopening this since I would have expected the start and end datetimes to be different.

https://docs.opendatakit.org/form-question-types/#metadata

start — The time the survey was started.
end — The last time the survey was saved.

[newtewt]

@abbyad can you define when in our application this should record start and end?

For start should it be when the user clicks the action to open the form or when the form loads or something else?

For end should it be on submit or the last page or something else?

[abbyad]

I can see two options. The start and end values could be based on:
1) time in the active form
- start is when form is fully loaded
- end is when _submit_ is clicked.
1) total time to complete a form
- start is when form is initiated (eg user clicked on action or task)
- end is when user has completed the form and ready to load the next page

Based on the ODK definitions, I would interpret start and end to align with option 1. If we are interested in calculating loading and saving time we can use telemetry instead.

[abbyad]

@medic/quality-assurance just noting that for AT we also need to make sure that the meta fields are properly documented since they differ slightly from ODK and Enketo proper.

[tookam]

@abbyad Where should these be updated? (documentation)

[abbyad]

You could put it in medic-docs/config/forms.md#xforms, perhaps in a new section called _Supported XLSForm Meta Fields_. We plan to reorganize the documentation soon, and having it there will be easy to find.

[garethbowen]

Ready for AT in 5768-incorrect-xlsform-metadata

[tookam]

Doc ready for AT as well in 5768-supported-xlsform-meta-fields

[newtewt]

:+1: seeing the end time change to when submit is pressed.