Invidious: 504 Gateway Time-out, The server didn't respond in time.

[ChloeMackey]

Anyone else getting that message when trying to access their subscriptions?

[Zero3K]

I can confirm that it is happening for me. I thought it was fixed.

[omarroth]

It's a problem that occurs when the load reaches a certain threshold. So it's not so much permanently fixed as the server is upgraded to handle the increased usage.

[EsmailELBoBDev2]

@omarroth i getting same error when i try open https://invidio.us/feed/subscriptions

[dimqua]

Me too.

[Discookie]

I think it's also an issue that the site doesn't have proper error pages:

• for 503/504 one that would explain that the servers are overloaded
• for 500 one that would point them to make an issue here
• etc etc.

I know some errors (like 404) do have their own pages, but I think all but the worst errors should be caught before the user sees them.

[d3d1rty]

It's a problem that occurs when the load reaches a certain threshold. So it's not so much permanently fixed as the server is upgraded to handle the increased usage.

Any idea of what the associated costs would be with upgrading the server to handle it?

[omarroth]

Right now I'm looking into optimizing rather than upgrading, but if/when I were to upgrade, it would probably be about a ~$20 increase.

[omarroth]

I just upgraded the server, and now am able to access my feed. If others find the same, then we can probably close this for now. I'm still looking into optimizing it as much as I can, but in the short-term this appears to be fixed.

[Discookie]

Looks fixed on my end, even though it still takes 20-odd seconds to load.

Also made a separate issue for proper error pages, instead of burying it here.

[omarroth]

Thanks. For folks that would like to see how Invidious queries the DB, I'd take a look here, and the table schema is here. I'd be very interested and thankful for thoughts as to how it could be improved.

[Discookie]

I don't see why the index channel_videos_ucid_idx is using btree instead of hash. It would gain quite a bit of performance, and I don't think you use < or > comparisons on it.

Another marginal gain in query speed would be to store all video ID's as 64-bit integers.

The id stores a variant of b64 characters - together these characters add up to 66 bits. But YouTube just doesn't care about the last 2 bits, so all of these open the same video:
https://www.youtube.com/watch?v=dvz9lsaIKLY
https://www.youtube.com/watch?v=dvz9lsaIKLZ
https://www.youtube.com/watch?v=dvz9lsaIKLa
https://www.youtube.com/watch?v=dvz9lsaIKLb

Channel IDs could be similarly shortened, although they are 132 bits long, and I don't know whether they ignore any bits or not.

[omarroth]

Thank you! That definitely helps a lot.

The comment about video IDs is interesting, but unfortunately the examples provided (except the first) return a Does not exist. My understanding is that YouTube uses some variant of hashids internally.

[Discookie]

Yeah, if you query them, they don't return anything, but if you open them on Youtube, they redirect to the first link. (This seems like a separate issue that needs to be fixed?)
A common example is linking the Rickroll video with the ending of XcR, XcS, etc. to trick you into clicking on an unvisited link :P

[omarroth]

Opening the links on YouTube results in a Does not exist as well. If a videoID is longer than 11 characters then YouTube will strip the extraneous ones, and Invidious does that as well. For example:

• https://www.youtube.com/watch?v=dvz9lsaIKLY
• https://www.youtube.com/watch?v=dvz9lsaIKLYXcR

Unfortunately, I'm still encountering this issue, if there's any other suggestions I would be very eager to hear them.

[omarroth]

I'm pushing out 35e63fa3f5472f847044499ce6d2c6a44220ce07 right now which should speed up subscriptions significantly, and we can hopefully close this. There's some more info on how it works in yesterday's release.

I expect the migration to take ~1.5 hours. Everything except subscriptions should still be available during this time. Thanks everyone.

EDIT: Migration should be complete. I'd recommend folks check their feed to make sure everything is working properly.

Of note is that since the feed is cached, it may take a bit to update, even if there are notifications for new videos. I'd like to fix that as well, and if feeds are now displaying properly, then I'll close this in favor of #148.

[d3d1rty]

Just checked and the issue seems to be corrected. Much faster response. I had previously adjusted my settings to only show unseen notifications, so I reset it so that it shows everything with 25 results in the feed. First attempt was successful, super fast response, so I adjusted the results to 50 and then 100 videos in the feed. I ran 3 attempts per setting (25, 50, and 100) and recorded the results below.

Response Times:

• 25 videos: 70 ms, 119 ms, 96 ms
• 50 videos: 76 ms, 166 ms, 385 ms
• 100 videos: 225 ms, 133 ms, 143 ms

[omarroth]

I'd be very interested to investigate, would you mind opening a new issue?

[omarroth]

It seems no one has encountered this for about a month, so I think I'll close this. If anyone encounters this again feel free to open a new issue, or I'll be happy to reopen.

[jeroen7s]

now i'm having this issue when checking out my subscriptions
EDIT: I do have almost 500 subscriptions though, which might contribute to the problem :/
EDIT2: its suddenly gone now, page loads in about 1.5 seconds