Spoke: Auto-refresh replies and other ToDo-screen data

[schuyler1d]

Use Cases:

• User is 'staring at the screen' of ToDos -- waiting for things to reply to.
• When the user 'clicks back', does the content refresh then?

Currently, texters often have to hard-refresh to see replies. The number of replies does not update either (#178)

Tech context:

162 improved this by increasing poll frequency, and #198 added an automatic refresh after the user returns to the ToDo list from the messaging interface. But some other things should update.

UI: Should we 'ask to notify user' (via desktop) and push a notification when there are new replies found?

Possible solution (for someone wanting to do this work):

• Assignments are too complex to cache, however when caching is available we could cache a notification when a new message came in. Then, instead of polling the whole TodosList query, we could poll whether any notifications were present, and when they are they could request stats on the whole todos page. Somewhere around here: https://github.com/MoveOnOrg/Spoke/blob/main/src/server/models/cacheable_queries/message.js#L320 on message.save we could add a notification flag if the campaignContact assignment (and user id) was cached (https://github.com/MoveOnOrg/Spoke/blob/main/src/server/models/cacheable_queries/campaign-contact.js#L43).
• A new mutation would need to be made that gets back whether there are notifications, and when there are both clears them and returns a true result.
• A todos query should also clear notifications from cache for the user.

[sandramchung]

I've looked into and tested a couple of options explained here: http://dev.apollodata.com/react/receiving-updates.html and here: http://dev.apollodata.com/react/cache-updates.html
We're currently doing polling, refetch, and normalization with dataID from object. Polling addresses updating the ToDo list while someone's staring at it (changes when admin edits assignments and when user sends text or receives replies); refetch updates the ToDo list when we close out of the texting interface and return to the ToDo list.

We could replace the first two with subscriptions in the reply/todos interface, and that would probably have the most impact on DB tax; but implementing the websocket required by subscriptions isn't a good option for us at this time.

Next I'll try monitoring the load on the DB during campaign usage. Polling can get pretty harsh, but it may be that not enough users leave the ToDo list screen open long enough for it to matter most of the time.

[sandramchung]

Last weekend I didn't see db activity at problematic levels, but it's worth checking on again as we scale up on Spoke usage.
During Monday's demo Sky and I noted that polling on the ToDo List appears to have ceased. Suspect pollInterval may be incompatible with refetch if implemented on the same query/container.

[yolfer]

+1 This would be great!

[arena]

+1

[arena]

@schuyler1d would you consider re-formatting this to match the Problem, Solution, Context of more recent issues. It would be easier to find that way. And since it's getting old, has there been any progress?

[schuyler1d]

I moved around some pieces in the ticket and re-titled it. Feel free to edit to add key words or add clarity.

[ibrand]

this addresses #466, #254, and will make #1359 no longer needed

[matteosb]

Upgrading apollo-client, react-apollo, and fixing the apollo cache key should solve this, or at least give us better tools for solving it. I'm planning on doing the upgrade soon.

[ibrand]

@larkinds you should coordinate with @matteosb on this ^ there's definitely a lot to learn here, and maybe there will be some supplementary code needed

[larkinds]

That's awesome! Will do.

[tekkamanendless]

Hi, I'm new here, and I disagree that somehow having auto-refresh of the texter dashboard is a problem. From what I've been told, apparently the relevant queries are too database-intensive for large deployments.

If that's the case, then we have two general states:

1. The database is too "large" and the queries somehow hurt the system in some way.
2. The database is not that "large" and the queries work perfectly fine.

For smaller organizations, (2) will hold just fine. I would argue that smaller organizations make up the majority of people using (or at least trying out) Spoke, and not having a reasonable auto-refresh capability on the texter dashboard is a showstopper. So it's really bad.

For larger organizations, (1) will hold, and they will have a problem. That's fine, but we should understand that massive databases that cripple the system's basic functionality are the exceptional cases. If this problem is unsolvable right now (and I disagree that that's the case), then a larger organization can go through the extra step to set auto-refresh to "disabled" or "every 1 minute" or something less aggressive than every 5 seconds.

I strongly recommend that the _default_ behavior be the _best_ behavior, and if a larger organization needs to adapt due to its size, then it can (and it also has the resources to do so).

---

We can also view the problem as a volume problem; maybe it's the sheer number of queries that's the problem, and with lots of people doing texting, having every one of them ask for the status data every 5 seconds might be too much (again, I disagree in principle, but let's say that this is the case).

Here's the fundamental problem in this case: _the texters must see the recent status in order to do their jobs_, so they will either (1) manually refresh, which is annoying and error-prone, or (2) do nothing and hurt the campaign by not being responsive. If the campaign is successful and responsive, they _must_ be refreshing often enough to do their jobs but infrequently enough to not harm the system.

In order to solve the volume problem, an "exponential backoff" approach would probably work just fine. It works generally like this: upon sending a message or detecting that a new message has arrived (let's call either of those an "event"), we can assume that something interesting is happening. As such, the next refresh will be very soon (let's say 5 seconds) because this is an active situation. After that next refresh, if nothing changed, then the action isn't happening all that fast, so maybe we double the refresh interval to 10 seconds. After that refresh, if something changed, then we go back to the beginning with an "active" situation, but if nothing changed, then we double the interval again to 20 seconds, and so on and so forth, until we hit some kind of cap, maybe 5 minutes, after which the refresh interval will remain constant, even if nothing continues to change.

This kind of strategy is used frequently in network protocols and websites trying to poll for changes but also not wanting to waste resources. For example, after submitting a request, the web site might poll for the status of that request rapidly at first so that if the request finishes quickly, it feels "real-time" to the user, but if the request is taking a long time, then the user doesn't really perceive the difference between 2 minutes and 3 minutes; it's basically a "slow" background task at that point, and they're not constantly looking to see if it's done.

In our case here, we could have 1,000 volunteers waiting around on their todos screens with nothing happening in the world (refreshing every 5 minutes) and only making 200 requests per minute on average. But without exponential backoff, those same 1,000 volunteers would be refreshing every 5 seconds, for a total of 12,000 requests per minute.

---

Anyway, that's my take:

1. The performance issues can be solved with better database design.
2. But it can't be solved that way, solve it by doing the bad things _less_ using exponential backoff.
3. And if it's a _really bad_ problem, allow people with that problem to turn the feature off.

The solutions are really easy to implement in reverse. (3) is an environment variable that large organizations can set like UI_TODO_REFRESH_INTERVAL=disabled, but it would also accept something like 5s, 1m, etc.). (2) just adds a bit of logic to the setInterval code (maybe with recursive setTimeout calls, or a monotonous setInterval that only triggers a request when it hits the appropriate ticks). (1) may or may not be time-consuming, but I don't have any docs on the schema and the performance characteristics that you're expecting from it, and my Postgres skills are only good enough to get me in trouble (the migrations code does not work with a MySQL database, so I can't troubleshoot this on my home turf).

[schuyler1d]

I agree that this is important. I think we disagree on implementation.

1. The problem with exponential backoff is it's not when there's nothing happening that the problem occurs (i.e. when there are no text messages coming in) -- the problem is precisely when there IS a lot of texting activity that the database reads can lock against writes.

I think there ARE two changes that would improve our ability to turn this on:
A. (described in the issue description) -- instead of a polling model for all the data, poll the existence of changes (i.e. notifications) -- and then only query the data when there are changes.
B. Relevant to your (1), the query for user.todos is currently structured in a 'natural' way for GraphQL, but it creates many db queries for each API call. todos just gets the list of assignments here: https://github.com/MoveOnOrg/Spoke/blob/main/src/server/api/user.js#L249-L258 and then each sub 'count' (here: https://github.com/MoveOnOrg/Spoke/blob/main/src/containers/TexterTodoList.jsx#L138-L147) is queried separately. This is the only place user.todos is called, so we can optimize the original todos call to include the aggregates in a single query. That would significantly improve the call.

I would take a PR that changes user.cacheable to user.todoRefreshable and returns true when UI_TODO_REFRESH_INTERVAL is set. I'm not ready to make it the default without A or B, but I think that's the first step.

[tekkamanendless]

Okay, I added #1727 to cover the new environment variable.

I didn't feel the need to add a new user property, since that property wouldn't change per user (it's more of a system-wide configuration). As such, I added it to the webpack config just like PHONE_NUMBER_COUNTRY. If you envision having separate configs for each user, then I'd be happy to tweak the model.

Let me know if that's kind of what you had in mind.

[tekkamanendless]

As for this:

B. Relevant to your (1), the query for user.todos is currently structured in a 'natural' way for GraphQL, but it creates many db queries for each API call. todos just gets the list of assignments here:

I tried looking for a quick and dirty way to inject a simple query that would tell me the last timestamp that anything "interesting" happened with any of the user's messages (example: select max(service_response_at) from message where user_id = 1;); however, this whole GraphQL thing is entirely impenetrable to me and I can't figure out how any part of it works. Anyway, the results of a query like that rolled into the user's todo response list would allow the todos screen to track any time that the largest service_response_at timestamp changes between calls, and when it does, speed up the refresh interval (and when it doesn't, slow it down). (Also, I'm guessing that service_response_at is what we're looking for, but there are 4 timestamps that all look promising.)

[schuyler1d]

Agreed, GraphQL is particularly a problem in this kind of context, which is why it's currently inefficient.

I think your "interesting happened" query is essentially the same thing as what I'm proposing with caching. I'm reticent to use the db though, precisely because doing SQL queries on message is exactly what locks up the DB during 'burst' moments.

[tekkamanendless]

What's your proposal for caching? I'm new to the Spoke code base, so I'm not all that familiar with your caching strategies (I know that Redis is an option, but I don't know what the system does if it's not there). But upon sending or receiving a message, it should be super easy to update a "last-message-activity:${user-id}" key with the current timestamp; that'll take all the load off the database. Where should I look for the Spoke caching guidelines? I'm happy to get this moving.

[schuyler1d]

This is mostly up-to-date in terms of caching documentation.
https://github.com/MoveOnOrg/Spoke/blob/main/src/server/models/cacheable_queries/README.md

See "Possible Solution" section at the top of this issue here:
https://github.com/MoveOnOrg/Spoke/issues/207#issue-257162021

If you want to work on this, we should probably get on the same page for details on a call. I'll email you to schedule a time.

[arena]

I see that there has been a lot of conversation on this issue. Please forgive me for not reading it all in full detail. I'm curious about 2 things:

1. is work active on this issue?
2. I don't think the "you have nothing to do" message should even exist after sending texts. Ideally (imho), users would see that there is a campaign where they sent some texts. That campaign should either have either refresh occasionally or there should be a way to refresh it (or both).

Q: Is the gist of this issue that the "you have nothing to do" screen would still exist? and if so, can we please rethink that? I think it confuses users and doesn't give them much info about what to do next.

Context: I wrote and issue about the "You have nothing to do" user experience that is blocked by this issue.

[schuyler1d]

I don't think there's active dev on this right now. @arena I agree that maybe at this stage, there should be some 'pre-discussion' around the texter's 'todos' full experience -- some things that have come up recently include seeing cross-org todos and being able to see something like 'available campaigns' (for dynamic assignment) -- I don't think these all have to be tied together -- I think there's a smaller-scope issue that focuses on this particular issue. I'm happy to work with a volunteer that wants to work on this. Otherwise, it will probably be a Q2 priority on "MoveOn"s calendar.

[arena]

Would it make sense to un-block https://github.com/MoveOnOrg/Spoke/issues/1359 ?

[schuyler1d]

I am worried that #1359 as a solution could cause too much stress on the system -- I think this issue (#207) is ultimately what we want to do, so it can be related to performance/policy (to guard against a critical mass of 'anxious' constantly refreshing texters), but I suppose the button could sometimes be disabled, but that could cause its own set of confusion.

[DinahSanders]

When there is this resting point in a texter's experience, rather than a message like "You have nothing to do" (which is often inaccurate), it would be great to be able to display a customizable message which reinforces training.

For example:

Please refresh this page to see if you have replies, and check #important_announcements in Slack for more texts to send.

🐕 Tip Terrier tip: You don’t need to handle anything in the “Past Messages” or “Skipped Messages” buckets. They are an archive, not a to-do list.

[DinahSanders]

As an organization, it would be incredibly helpful able to update that message as often as we like.

Not only might it help us reduce common errors, It would provide another micro-training point to reach people who are not reading documentation and are participating minimally in Slack.

[Frydafly]

@DinahSanders I'd encourage you to submit a feature request for your suggestion to make the "you have nothing to do" text customizable! I think it'll help that specific feature to not get lost in this larger thread. But I'm new to coordinating these issues so I welcome any input from @schuyler1d & @arena

[arena]

My long term preference would be that the “nothing to do” screen would go away entirely and you’d be back at the dashboard after you are done sending and, if there are no replies, you’d at least see that you sent some texts …maybe even see that you could request a new batch.

I don’t know if we collect metrics but I would bet money that once we stop telling people they have “nothing to do” the response rate by the original sender will go way up (especially when those original senders are first time texters).

In the meantime, I support letting admins edit the "nothing to do message" with some markdown that lets them communicate what they need from their text team or humor or whatever makes sense for the team at the time. My concern would be that texters would only see the message in some cases and less experienced admins might think that what they are writing will be seen by everyone.

[arena]

Added this simple issue about the message texters see https://github.com/MoveOnOrg/Spoke/issues/1939