Etherpad-lite: Stateless Etherpad

[ottomattas]

We were introducing Etherpad to be used as a part of a bigger solution when it turned out not to be stateless. We are building the solution on cloud resources and would like to really leverage the services available, granting us High availability (HA). Currently looking for workarounds in terms of our solution but the question remains - is there any initiative to make Etherpad stateless?

Looks like there are a few issues around this topic raised already. Among possibly others, I could find these:

• #3680
• https://github.com/ether/ueberDB/issues/51

[JohnMcLear]

There isn't an initiative at the moment to make Etherpad stateless, we encourage and invite you to lead on it though because it's a worthwhile en devour.

Instead of going stateless we optimized for performance on the thread, this meant our load testing numbers at simulation (see the tools we provide) are pretty impressive. This satisfies nearly all of our clients.

Our other large clients (M's of pads, 5k figures per day users) have a sharding system where pads with a specific URL are passed off to a given piece of hardware. So pads a-n to server 1, o-z to server 2 and any other random chars #!'; etc. to server 3. Each server has it's own backend too. This is not optimal though because like you said, HA is an issue... :)

The thing is, we want to keep all the Operational transform operations for a given pad on a given thread so we can optimize and keep traffic low. There is no point having 10 servers handling OTs for 5 users, it would introduce latency and fragility. So ultimately I'm of the opinion that even if you went stateless you would want to keep a given pad's contents in one instance.

Still, sounds like you know more about going stateless / HA more than me and it's been 5+ years since I looked into it so I'm all ears to know if we can help at all and I'd be super keen to see it implemented 👍

[ottomattas]

Thanks for the explanation. I completely understand the rationale behind the decision to optimise on a thread rather than make it stateless. For now, we will continue with the stateful setup as we are just building a POC which still needs organisation buy-in on a high level. After we have gained this, I am indeed very much for supporting the effort in going stateless.

Just an idea around HA and to clarify why I am seeking this in our situation. We work in a enterprise environment and this can bring a lot of complexity to any solution. HA would allow us to focus on building good solutions rather than worry about managing them later. Life is already difficult enough so every piece of stateless software helps! And not only for engineers building and running the solutions ... business side has quickly caught on and sees the value in building HA solutions. Leaving engineers' hands free of ad hoc work and firefighting makes it possible to focus on generating new value instead.

[JohnMcLear]

Awesome thanks; keep us posted and do feel free to shoot through any dislikes/quirks/bug reports and we will endeavor to action them ASPAP.

[jmaupetit]

Hi :wave:

We (OpenFUN) have developed a stateless Docker image for etherpad-lite: https://github.com/openfun/etherpad-docker

We are running this image over k8s (OKD) and it's running fine in production (see https://pad.education), but we are suffering from websocket issues while trying to scale our pods. We've drafted a redis-based abstraction layer to address this, but unfortunately, not all events seem to transit to this abstraction layer... You can check out our POC here: https://github.com/openfun/etherpad-docker/pull/8

@ottomattas I'm not sure that you are trying to run your instance in a k8s context or even in a Docker container, but we can join forces if this is the case :pray:

@JohnMcLear any advice in using socket.io-redis adapter would be of a great help. :pray:

[JohnMcLear]

Pad.education is not loading pads?

Happy to help if I can, what's your schedule look like?

[jmaupetit]

Pad.education is not loading pads?

No, it is loading pads! But we are only running one etherpad instance and we cannot scale "the kubernetes way" (_i.e._ by adding more replicas of the app and route requests with a round-robin algorithm).

At first, I thought this was only due to websockets, and thus we've tried to use a socket.io-redis adapter so that an etherpad instance is not tied to a client. But, it is not working as some events are not send to this redis adapter, and, FWIU [1], we are going to face other issues related to the application state. Am I right?

Happy to help if I can, what's your schedule look like?

For now, scaling this new service is not crucial to us (everything is running fine), but we are planing for the future. I am wondering what should be achieved to make the application stateless? Is the redis socket.io adapter a dead end?

If making etherpad stateless is not a priority or requires too much changes of the core app, then we have other alternate plans close to the sharding approach you explained earlier [2]. I am collecting feedback to choose the more efficient/pragmatic approach to scale the app.

[1] https://github.com/ether/etherpad-lite/issues/3680#issuecomment-566809617
[2] https://github.com/ether/etherpad-lite/issues/3848#issuecomment-610942249

[muxator]

@jmaupetit, right now, the most pragmatic way of horizontally scaling Etherpad is sharding on the pad names.

[jgribonvald]

I'm really interested too on a such feature, we also need HA and load-balancing.
I'm only a beginner with etherpad, but in waiting I deployed it on 2 servers and I'm distributing access depending of kind of my users, we can have around 5 000 users simultaneously, and maybe more. The 2 etherpads servers request on the same database, and for the fail-over I placed the second server as a fall-back server.

@JohnMcLear

Our other large clients (M's of pads, 5k figures per day users) have a sharding system where pads with a specific URL are passed off to a given piece of hardware. So pads a-n to server 1, o-z to server 2 and any other random chars #!'; etc. to server 3. Each server has it's own backend too. This is not optimal though because like you said, HA is an issue... :)

I would be interested to know how peoples did that, and which sharding systems they did. We can't find such feedback on large instance deployment.

[JohnMcLear]

@jgribonvald Full disclaimer here: I do this work for commercial clients so this is really the type of work I do on a daily rate. But within the spirit of Open Source and supporting our community here is a Varnish VCL you could manipulate to work.

Sharding

    vcl 4.0;

    backend static {
        .host = "staticcontent.etherpad.com";
        .port = "9001";
        .connect_timeout = 6000s;
        .first_byte_timeout = 6000s;
        .between_bytes_timeout = 6000s;
    }

    backend ep1 {
        .host = "192.168.0.1";
        .port = "9001";
        .connect_timeout = 6000s;
        .first_byte_timeout = 6000s;
        .between_bytes_timeout = 6000s;
    }

    backend ep2 {
        .host = "192.168.0.2";
        .port = "9001";
        .connect_timeout = 6000s;
        .first_byte_timeout = 6000s;
        .between_bytes_timeout = 6000s;
    }

    sub vcl_recv {
           if (req.url == "^/static/*$") {
              set req.backend_hint = static;
           } else {
        set req.backend_hint = vm1;
        }

   .. logic for socket header stickiness [ secret sauce I can't copy/pasta (see below for understanding)]

    sub vcl_recv {
           if (req.url == "^/[a-g]/*$") {
              set req.backend_hint = ep1;
           }

    sub vcl_recv {
           if (req.url == "^/[h-z]/*$") {
              set req.backend_hint = ep2;
           }

You will probably want to set a rewrite based on req.http.header which would need to include the padId for socket connectivity. The uri is always /socket.io/. You would need to extend etherpad to include padId in the headers. This is very easily done with a plugin.

Sticking

Using any Key/value system you can easily implement a pad level routing system. Basically your rewrite endpoint would have a shared (ideally quick access in memory) space of Server<>padId [1 to 1 relationship] and each routing endpoint is aware of this relationship so routes the user to the server based on the padId. This is how other large platforms work because as I have said before, you really don't want a single pads socket connectivity to be distributed. This is still sharding but it's dynamic based on criteria you stipulate (IE geographical proximity of pad creator), afaik this is what Google does.

Getting this done

The reason we don't do this as Etherpad developers is because roll out's vary (a huge amount) so we entrust our devops / sys admin friends to deploy the current popular solution to hit the scale they need. So while this question is welcome I just want to be forthcoming with the reasons it's not baked into Etherpad.

[jgribonvald]

@JohnMcLear Thanks a lot I will watch on what we can do with that !

[JohnMcLear]

Also see https://github.com/ether/etherpad-lite/issues/3335

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.