Collect: Add option to send finalized forms using SMS

[yanokwa]

From our friends at https://github.com/medic/medic-collect

"Our users often do not have a reliable data connection, if at all, so this was a major reason why we hadn't used Collect previously. We added "Send via SMS" support in Collect, which also needed a fix in org.javarosa.model.xform.SMSSerializingVisitor."

This is important because some deployments have no cell or WiFi connectivity. This feature isn't straight-forward because you need to think about what you do with binary data or just very large forms. Also, there needs to be some listener on the server for this.

A good first pass at this issue should describe exactly what exists with medic-collect so we can have a discussion about would be appropriate for collect.

[max2me]

I wonder if it's also worth considering an option to send finalized forms using Email. Email clients are generally pretty good about retrying when connection is weak etc.

One of the concerns with SMS is that delivery is not guaranteed by any means so perhaps SMS option should not be used with deleting a form upon sending (otherwise data might get lost).

[yanokwa]

@max2me Email won't work because the folks who need SMS don't have any other option of sending the data. I also agree that delete after sending if SMS is enabled would be a bad idea.

@abbyad When do you guys use SMS transmissions? And what server receives them?

[alxndrsn]

@yanokwa at medic we use SMS submissions exclusively with out deployments of Collect. They are processed by medic-webapp. We'd be happy to submit a PR with our SMS-submission code as-is.

Longer-term it might make more sense to have a cleaner submission API within Collect which would make implementing different submission methods simpler (standard XML-over-HTTP vs SMS vs email vs e.g. save to local file which triggers dropbox sync. The submission method could then be selected either in config or perhaps at submission time.

[lognaturel]

Cool to know you've added SMS submission to your version of Collect! Could you please start by briefly describing how you've structured your approach and point us to the relevant code?

I totally agree with your vision of a well-structured and well-documented submission API. Let's talk about your current implementation a bit and then we can also discuss an "ideal" that would require significant server work as well.

[alxndrsn]

@lognaturel the bulk of the work's done in InstanceUploaderTask.smsOneSubmission(), which is used instead of HTTP transmission when the user clicks the _Submit by SMS_ button in the InstanceUploaderList activity. @abbyad may be able to help with more specific questions that you have!

an "ideal" that would require significant server work as well

I don't follow why a change to an internal submission API would need changes on the server - I'm not suggesting that the format or protocol of submissions to the server would need to change, just that the code for choosing submission mechanism could be more cleanly structured than in our fork. Perhaps this is already the case with the protocol selection available through Preferences.

[lognaturel]

Thanks, that's great! I will give that implementation a careful look hopefully soon and let you know what thoughts and questions come up.

Right, yes, I was thinking about the receiving end and what kind of structure of data / split of form into multiple SMSs it would be able to interpret. Now I understand what you are talking about and that would also be nice to have standardized and made easily extensible!

[abbyad]

Sorry for the silence on this issue, I hadn't noticed the original mention. So far we've had full control of the forms being deployed with our version of Collect, so we've had the benefit of being able to limit the field types and length of the forms.

Happy to help answer questions about SMS as they come up!

[lognaturel]

Thanks @abbyad @alxndrsn for the context. I dove into the code and found this commit and this one useful.

Here's my high-level understanding of the approach:

• SMS payload is saved using getFilledInFormSMS every time the form is saved.
• There is a "send as SMS" button shown in addition to the send button on the Send Finalized Forms screen. That button is shown based on usage of the medic mobile platform and existence of a gateway set in settings as described here.
• There's a fair amount of checking done to make sure the SMS is sent and delivered here.

Are there any important aspects I missed?

Am I understanding correctly that the "send via sms" button is displayed alongside a send button? Is that because sometimes enumerators want to do both?

Thanks, this is great!

[abbyad]

Your high level understanding is spot on.

Am I understanding correctly that the "send via sms" button is displayed alongside a send button? Is that because sometimes enumerators want to do both?

Yes, the _Send via SMS_ is shown alongside the usual _Send_ button. This was done so that enumerators have the option to send via web if they have wifi (or mobile data), otherwise send over SMS when they are without data. Some of our deployments have had hundreds of health workers with unknown/varying levels of connectivity, so this let them chose the option that worked best for them.

And sorry for yet another long silence on this!

[jd-alexander]

@opendatakit-bot claim

[getodk-bot]

Hello @jd-alexander, it looks like you've currently claimed 1 issue in this repository. We encourage new contributors to focus their efforts on at most 1 issue at a time, so please complete your work on your other claimed issues before trying to claim this issue again.

We look forward to your valuable contributions!

[jd-alexander]

Hey everyone! I want to start working on an implementation of this. I am following the conversation over at the ODK forum and now I have some more technical questions.

1) The fetching of the prefix and tags would that be done in Collect or would changes need to be made to Javarosa so official methods can be added to retrieve them from their respective elements?

2) I was thinking of writing a different SMSSerializingVisitor so that I can put in the logic for this new spec. Again would this go into Javarosa or Collect? The interfaces it implements are public so it could be implemented in Collect. Just checking as to what's the best option because I know that it would be good if Javarosa had the functionalit of this spec so that others could make use of it.

[lognaturel]

Pushing what can be pushed to JR seems like a good idea. If I recall correctly, SMSSerializingVisitor is pretty close to the implementation we've been discussing. Perhaps cleaning up and improving that one would be a good way to go? I suppose it depends on the extend of the changes that would be necessary. As far as I know, Medic Mobile is the only project using SMSSerializingVisitor and I don't believe they're updating the version of JR their tools use.

[jd-alexander]

thanks for the clarification @lognaturel. Appreciated.
Another question, so for now are we working with the length of a SMS being 160 characters? In the discussion over at the forum there were talks of having the admin set the length of the SMS.

[yanokwa]

Beyond that one comment on the forum, I can't find good evidence that SMS can be less than 160 characters. https://www.twilio.com/docs/glossary/what-sms-character-limit suggests that 160 is the common max. I'd assume that for now, but you know, put it as a constant somewhere.

[alxndrsn]

I've observed on (some?) CDMA networks, SMS will be transmitted in ASCII mode, which allows for 140 characters per message.

Also sometimes if the receiving number is a CDMA network then messages sent from GSM handsets with more than 140 characters per part will get truncated.

Multipart messages sent in this scenario will be a mess.

---

Probably already taken into account, but worth mentioning that you can only fit 160 characters into an SMS if they are all within the GSM 7bit character set (or extension tables, if these are supported).

[abbyad]

An SMS can have a maximum of 1120 bits, so with the GSM 7-bit character set that is 160 characters, with ASCII that is 140 characters, with UCS-2 that is 70 chars for the first part (less if multi-part).

There seem to be perfectly good methods in the Android API to divide a message into parts, and avoid assuming a constant character limit.

[jd-alexander]

good points @abbyad I think the character limit came into question because @yanokwa was considering sending a header with each message.

Every message should have a "header" with form ID/prefix, an instance ID, and the sequence number

The thing is I wouldn't know where to put this header until the messages are split i.e when I know which parts of the form will be sent in a specific message. An idea I had in mind was to use some dummy text to get a result of the length of each message from SMSManagers divideMessage and then utilize that to assist in formatting the messages once they are split up.

Another point @yanokwa If code is put in place where we ensure that a message is sent before we send the following parts is it totally necessary that we have a full on header in each message (since we have limited chars)?

Let me know your thoughts guys.

[alxndrsn]

@jd-alexander ref

Every message should have a "header" with form ID/prefix, an instance ID, and the sequence number

I've commented on the forum post, as that's where @yanokwa originally wrote that (https://forum.opendatakit.org/t/send-submissions-via-sms/12166/25?u=alxndrsn).

[getodk-bot]

Hello @jd-alexander, you have been unassigned from this issue because you have not updated this issue or any referenced pull requests for over 10 days.

You can reclaim this issue or claim any other issue by commenting @opendatakit-bot claim on that issue.

Thanks for your contributions, and hope to see you again soon!

[getodk-bot]

ERROR: This active issue has no assignee.