Cht-core: Optimise form HTML generation

[SCdF]

Currently phones regenerate the HTML for a form every time the browser is refreshed. This slows down first time form loading by a second or two on most phones.

Instead, we should cache this generation more aggressively somehow.

Probably the most complete solution would be to actually ship the HTML to devices instead of the form xml, and then generate the HTML in either sentinel or as part of upload in medic-conf.

We have to be careful because there are quite a few things in play:

• form version
• enketo version
• our custom xslt versions
• potentially general app version (in case we change / add feature sets)
• other things?? (jquery version? other libraries that we utilise?)

FWIW I tried to do it in browser, by storing the generated HTML in a _local/formCache document. It presented many challenges, chief among ironically keeping general form requests as fast as a dumb in-memory cache:

• You need to check lots of things when you boot to make sure nothing has changed out from under you
• You can't check form version (or other things) every time a form is asked for because that is too slow: it has to just be pulling the cache from memory and looking at keys
• So you need to attach a follow to the changes feed and selectively wipe parts of the cache dependent on what comes through on that
• Understanding that this follow might not be set up in time as replication starts before you get a chance to attach so you need to make your initial boot logic (first point) as specific as this one
• And also probably understand whether or not you're admin, because writing to a remote _local makes no sense (or _does it!?!?!_), so do you then write to a local local (which admins don't have), or for admins do you just back on to the existing style of memory cache?

[SCdF]

@alxndrsn / @garethbowen interested in your thoughts on the above.

[garethbowen]

I vote for doing it server side which feels like the best solution, albeit the one requiring most change.

• We can be aggressive about rebuilding because performance won't matter much so any time the ddoc or the forms change we can regenerate and update the attachment on the form doc (if it's changed).
• It saves us about 6KB (zipped) of xsl that the client no longer needs.
• Fetching the attachments from the db might be quick enough then to drop the cache altogether and save some memory.

It could be done as part of medic-conf but the xsl used has to match the version of enketo shipped with the app so this would be difficult. Instead I think it should be done in api with a change listener. Another option is to do it as a sentinel transition but this feels out of scope for sentinel which usually only deals with reports.

[medic-bot]

Hi @alxndrsn,

Please ensure this ticket has a Priority, Status and Type label.

(See triaging old issues for more detail)

[garethbowen]

This also has the massive advantage of getting our medic/enketo-xslt fork much closer to the enketo version by removing all the cross-browser support rubbish. I think we should push this forward, because it's probably easier than solving #4386 the old way...

[garethbowen]

I can't find any native JavaScript xslt processors - all the libraries I can find require binaries to be downloaded. I don't want to start distributing binaries as it would affect our portability in unpredictable ways. I also don't want to npm install on the server because then we lose some control over the predictability of our releases.

One option is to ship medic-os with the required binaries and call them directly, but that adds additional complexity to the os which I'd really like to avoid.

My preferred option is to run the xslt in medic-conf. Anyone running medic-conf can npm install the required binaries. In order to run the correct xslt for the target server they would be bundled and distributed with the webapp as they are now and then requested by medic-conf to use in the transform. This is potentially a little slow and fiddly, and means from now on forms _must_ be uploaded using medic-conf but this is the recommended approach anyway. It will also mean migration of forms is impossible and the forms will have to be reuploaded using medic-conf on upgrade.

@SCdF @abbyad Is this getting too complicated?

[abbyad]

That does seems fiddly to handle on the medic-conf side, although understand most of the concerns with the alternatives.

don't want to npm install on the server because then we lose some control over the predictability of our releases.

I'd be curious to hear more about the downsides of this to weigh up against the medic-conf option.

Losing the ability to upload via webapp is unfortunate, but from my understanding requiring a re-push of forms on upgrade could end up leaving the instance in an unusual state. For instance, the upgrade could be done via webapp, whereas pushing the forms via terminal, so they would now have to do the upgrade from a device that has the latest full config. More importantly, I imagine the form conversion and upload can only happen once the instance is back up from the upgrade, so there would be some time between when the instance is updated yet the forms are not. What would happen to users accessing the app during this time? This seems to add complexity that may outweigh the disadvantages of the other options (including the status quo).

[garethbowen]

I'd be curious to hear more about the downsides of this to weigh up against the medic-conf option.

It puts us back to the situation gardener put us in where dependencies were downloaded at install time which meant each install could end up with different software running. So far with Horti, what we release is what gets run. With binaries we'd either have to make a release for each supported architecture, or accept losing control over the exact libraries we end up with.

I think there are solutions to the race condition you are worried about. One idea I had is that we could upload the forms using medic-conf before the upgrade and attach the original xform, as well as the form and model generated via the xsl. These would get replicated down to the phones but because they still have the old app code they would still load the xform and perform the in browser xsl transform. Then when they upgrade to the latest webapp the would switch over to using the server generated form.

Then eventually we could drop the xform from the doc altogether to save on disk space.

[SCdF]

I don't want to start distributing binaries as it would affect our portability in unpredictable ways.

Just to clarify what you mean here, you're saying that this will work (ie we will generate a legit self-contained build), but the build process will involve non-JS being downloaded that is correct for whatever Travis runs on but might be wrong once we pull it from the build server?

I have mixed feelings. On the one hand, targeting Linux (presumably the compat would be 100% inside linux, not ubuntu-only or whatever) seems like a pretty good compromise, especially since (IMO) MedicOS/Docker is going to be our only supported platform, and we get to control that heavily. Similarly if we had to pre-install libraries on our Docker images it doesn't seem like a big deal.

On the other hand, if it means I can't just use horti on MacOS even for test development then I would be annoyed, though in a selfish way. We can always make sure our docker images support dev (ie you can plug in local directories for code) if they don't already.

This is potentially a little slow and fiddly, and means from now on forms must be uploaded using medic-conf but this is the recommended approach anyway.

It is the recommended approach, but it would probably make certain types of testing really annoying, because currently (AFAIK) medic-conf is based entirely around you maintaining a legitimate production-style project structure. Not being able to just upload a hacked up test form to work through a problem seems pretty limiting.

[garethbowen]

Just to clarify what you mean here, you're saying that this will work (ie we will generate a legit self-contained build), but the build process will involve non-JS being downloaded that is correct for whatever Travis runs on but might be wrong once we pull it from the build server?

I think there are two options - either run npm install at install time and get the binary for the right architecture but lose control over what we end up running, or run npm install on travis and lock down the release but potentially have to distribute multiple releases for each supported architecture.

Well maybe shipping the binary in medic-os is the way to go. It's easy enough for devs and testers to work it out, and we only support running on medic-os in Docker in production from now on anyway. I'll look in to doing that and see how far I get. Thanks for your input!

[garethbowen]

Consider making a common lib to reuse this code in https://github.com/medic/medic-conf-test-harness/blob/master/src/form-host/wireup.js

[SCdF]

AT:

• Code in 4450-server-side-form-generation
• Creating reports via Enketo forms should work as normal for online and offline users
• Updating reports and changing them should work for online and offline users

(@garethbowen edit to change as required)

[garethbowen]

In addition to the above:

• make sure when forms are updated and replicated to the client they get the new version which will prove that API updates the form when a change is detected.
• make sure it works on an actual phone

It'd also be great to get some accurate performance comparisons on a real device between this and 3.6.0 both for first load and subsequent loads of the form. However this can wait for RT if that's easier.

[ngaruko]

LGTM
Form submissions, replications and updates work as expected. We will check for performance during RT.
There are conflicts on the branch @garethbowen

[garethbowen]

@ngaruko I've fixed those conflicts and rebased so the build is working again. Would you like to do another round of AT or are you happy as is?

[ngaruko]

Looks good! To monitor performance during RT.