Peertube: activitypub-http-broadcast jam

[tdiglx]

On a PeerTube v2.4.0 we are experiencing a _traffic jam_ of activitypub-http-broadcast caused by likes and their distribution as it seems.

  "signatureActorId": 1,
  "body": {
    "id": "https://example.com/accounts/peertube/likes/2413",

The backlog of the distribution is currently approx 90 minutes and 6500 entries (the number is raising) log. Is there something we can do to remove the _jamming_?

[tdiglx]

Over night the jam was resolved. Still it would be nice to know if there was anything that could be done in such a situation besides waiting for the system maybe regulating itself back to normal. Thanks in advance for any input regarding that!

[Chocobozzz]

Hello,

No sorry you cannot do anything in this situation. But it's weird to have all these likes at the same time. Maybe a spam from another instance? Could you paste the id of one of these likes? Were they targeting the same video?

[tdiglx]

Sorry for the delay. It seems that the problem was not the jam, but the jam a symptom of over-load of the server (16CPU/126GB RAM). Since Sunday evening the LOAD is around 100%, on Thursday it was higher then usual but I assigned that to the jammed distribution of the AcritityPub messages. We have several 2h long videos with 10k+ views per day I assume the distribution of these videos is causing the LOAD(?)

[rigelk]

@tdiglx you bandwidth should be maxed before your CPU. What is your reverse-proxy configuration and nominal bandwidth?

[tdiglx]

A speedtest-cli run gave me an estimated upload speed of 640.46 Mbit/s.

A diff between the shipped nginx config file and the one we use, gives only differences in the server_name, log locations, and the location of the LE certificates. So it basically it the released nginx config file

[rigelk]

and I suppose you have a 1Gb/s uplink?

[tdiglx]

yes,,, we have fiber optics with 10GBit ethernet.

[rigelk]

10GBit is not 1Gbit… Anyway your link is not maxed out.

Could you check where (which process) that high load is coming from?

[tdiglx]

It's the peertube process with the lowest pid, one higher then what is saved in the peertube-latest/pid/peertube.pid file.

[Chocobozzz]

Hello,

I'm sorry but I'm not sure to understand this discussion. What is the issue with PeerTube? Do it consumes too much CPU? Too much RAM? If yes, please fill the template issue with additional information like how many jobs you have in your job queue per minute.

[tdiglx]

Yes, the situation has evolved a bit. Should I open a new issue? Where do I find the template?

[Chocobozzz]

@tdiglx yes please fill another issue with an appropriate title so it's easier to find it using the github search. Github will propose you the template when you'll fill the "bug report" issue