Etherpad-lite: Etherpad doesn't work properly if running on k8s with more than 1 replica

[korablin]

I tried to run etherpad instance deployed on k8s (using official docker file);
I used affinity option (otherwise it just doesn't work at all, due to load balancing between 2 replicas):

  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"

but even then my team was hitting inappropriate behaviour:

Some people don’t see updates when they connect, and others have said their updates don’t show up for other people. Today, one person had two instances of etherpad running on two different computers, and one was showing their updates only whereas the other one was showing everyone else’s updates, but not theirs. This seems to be independent of browser type.

Here is what I was able to catch in logs when such case happened:

[2019-12-09 11:43:53.674] [INFO] ueberDB - Flushed 1 values
[2019-12-09 11:43:54.452] [WARN] console - Error: Can't apply USER_CHANGES, because Trying to submit changes as another author in changeset Z:9c>1|22=95=6*0|1+1$
    at handleUserChanges (/opt/etherpad-lite/src/node/handler/PadMessageHandler.js:641:13)
    at process._tickCallback (internal/process/next_tick.js:68:7)
[2019-12-09 11:43:54.478] [INFO] ueberDB - Flushed 1 values

We hit it when there is about 5-6 people typing stuff for about 10-20 minutes (about 100 lines)

EP version: 1.8.0-beta

[muxator]

Hi @korablin, there is a lot of state in the Etherpad nodejs server, and I am not aware of a way of sharing it between two servers. I think that simply running two etherpad instances and connecting them to the same DB will not work, even with session stikyness. Basically the two servers would end up having two different world views of the document, as you are experiencing.

[JohnMcLear]

Sharding for anything operational transform based in the norm and afaik we did load testing for what we defined as high usage "50k" users on a specific pad. I think we got these numbers from Google Docs but I'm not sure how they are doing their distributed computed on operational transforms now. Afaik Etherpad easily outdoes all other OT based editors tho IE box.com but I could be wrong..

So TLDR; Once you hit 4 figures active users per pad you should consider sharding, anything else will hot bottlenecks prior to OT IE DB ops.