Invidious: After Sep 1st, the Invidio.us instance will be an interactive redirect page

[cloudrac3r]

THREAD SUMMARY from oldest to newest:

• Omar approves of the idea.
• I create the page and publish the source: https://git.sr.ht/~cadence/invidious-redirect
• The hosted page as it would look when published: https://invidious-redirect.cadence.moe/
• The instances list will be shuffled to not favour any specific instance, but it will still be sorted by daily health, so that currently online instances are at the top.

CURRENT DISCOURSE

• What should happen to DuckDuckGo bangs?
• What should happen to embedded videos on the /embed/ endpoint?

Please leave general comments and questions in another thread unless they are specifically related to this redirect page.

---

Original post:

Let's start with some concept art:

The goal of this issue is that we don't want existing invidio.us links from around the web to break after the instance shuts down.

You _could_ simply serve an HTTP redirect to invidious.snopyta.org, since perflyst is now leading, but I fear that this would overwhelm invidious.snopyta.org with new traffic and new users, possibly taking it offline or making it otherwise unusable. invidio.us has 20 times as many registered users as invidious.snopyta.org, and almost certain many more incoming links. A 20× explosion in traffic would be very, very bad for a single instance. The result would be not much better than invidio.us going offline and leaving a blank page at the destination of the link.

The value of Invidious is in its many instances. Rather than taking everyone to the same instance, I think that invidio.us should display a table of available instances, with links to the requested resource on each instance, and the user can choose where they'd like to go to. For example they might choose to go to the instance where their user account is.

The "go" links would go directly to the requested resource, for example https://invidio.us/watch?v=kJQP7kiw5Fk would provide links to https://invidious.snopyta.org/watch?v=kJQP7kiw5Fk . Same with channels and playlists.

It would also be sensible to add some footer links to appropriate places, like to the github repository and to the instance list wiki page.

I am willing to code this entire site if you approve of the idea. I don't know crystal, so I'll be doing it in node.js. It should hopefully be able to handle the request volume, since it's _almost_ just a static page (there is some templating needed to fill in the URL), though I suppose we'll never know for sure until it's live.

Rather than using a template on the server to generate links, the server could always serve the same page and JavaScript could fetch the instance list and fill in the correct URL. This would have the advantage of being a single page to put on an nginx file server, with no backing code needed, but the disadvantage of needing people to have JavaScript turned on to see the page. For this reason, I think that building a template on the server would be better.

[ajayyy]

I feel that at least the API should use a real redirect

[omarroth]

Rather than using a template on the server to generate links, the server could always serve the same page and JavaScript could fetch the instance list and fill in the correct URL. This would have the advantage of being a single page to put on an nginx file server, with no backing code needed, but the disadvantage of needing people to have JavaScript turned on to see the page.

This would be my preference. A noscript fallback could provide a hardcoded list of instances and a YouTube link.

[cloudrac3r]

Redirect website is basically done. https://git.sr.ht/~cadence/invidious-redirect

Copy the files from /build to some webserver and set it to load index.html if the path wasn't found. This way, URLs like /watch?v= will still load index.html, which is what we want.

The webserver should apply a lengthy cache-control header to all paths under /static. If the files on disk are actually changed, then the static query string in the built files will also be updated to reflect them, so they will be redownloaded exactly when necessary.

Works on mobile. Works without JavaScript to some degree.

Edit src/index.pug to add instances to the fallback list for if JavaScript is disabled. It should be pretty obvious where to put them. Then npm run build to re-create the files in /build.

There is currently no page footer. What would you like down there? @omarroth

[GauthierPLM]

@cloudrac3r would it be possible that if I call invidio.us/watch?v=..., the part after the domain (/watch?v=... is automatically appended to the URL pointed by the GO links?

Regarding the API, it would also be good if, once the API is shutdown, the replacement page returns an error code + message notifying the user of the change.

This would help transition as I users would be redirected to the page they want, and the API would clearly state why no result can be returned instead of a simple 404 or 500.

[cloudrac3r]

would it be possible that if I call invidio.us/watch?v=..., the part after the domain (/watch?v=... is automatically appended to the URL pointed by the GO links?

yes, this is what happens, assuming the webserver is configured correctly.

[billmetangmo]

@cloudrac3r good idea.
I'd just like to make a point about redirection. Instead of having a list, it might be better to recommend a different instance to the user each time. What I mean by this is that instead of proposing to the youtube user directly it would be better to replace the "another instance" by an instance randomly chosen among the existing invidious one.

I think this would add a second guardrail to avoid overloading the first instance in the list because most people will most likely always choose this one as it can considered as the default one. On the other hand, presenting the redirection to Youtube as the simplest redirection (not present in the table) may make users choose to use youtube directly as this can be considered also as default option. This is a documented behaviour called default effect: https://en.wikipedia.org/wiki/Default_effect

That's why i think it will be good to propose a random invidious instance at the same level than the youtube link.

[cloudrac3r]

You make a good point. I decided to simply shuffle the entire table (but keep better health at the top). I think if I added a link next to the youtube link, it might not have much effect, since people's eyes would be immediately drawn to the table.

[cloudrac3r]

https://invidious-redirect.cadence.moe

[CSDUMMI]

I think that is quite good. Maybe just keep the background color white, as it is with invidious.
@cloudrac3r does your single page app handle API requests?

[cloudrac3r]

location /api {
    add_header "content-type" "application/json";
    return 410 "{\"error\":\"This server no longer hosts the Invidious API.\"}\n";
}

https://git.sr.ht/~cadence/invidious-redirect/tree/master/nginx/invidious-redirect

[CSDUMMI]

And what about those DuckDuckGo Bangs?
I don't know how many people use them,
but there are three Bang (search shortcuts)
registered at DuckDuckGo that redirect to invidio.us:

!in
!invd
!invidio

All of them essentially do the same:
Let you directly search Invidious from DuckDuckGo or directly from
your Browsers Address bar, if DuckDuckGo is your Standard Search Engine.
I don't think it is in the interest of those people using these
Bangs to always first go to a page, where they then have to choose an
instance to get to their results.
Bangs, after all, are a tool to make it easier to go to your search
results faster.
Thus what should be done about them?
If many users get to invidious through those Bangs,
redirecting them to another instance is bad for the same
reason implementing a hard redirect from invidio.us to invidious.snopyta.org.
Otherwise, should there be a website setup, that redirects each request to it to a randomly choosen instance of invidious?

[cloudrac3r]

Seems like this would be a great solution, then. Instead of always going to invidio.us, the bangs will now take you to a page that lets you choose an instance.

[billmetangmo]

@cloudrac3r thanks for taking my point into account :) !

[CSDUMMI]

@cloudrac3r my problem is, that those people using Bangs do so, because don't want to go through additional sites to get to the results they needed. They don't want to search for" invidious" on DuckDuckGo and then enter their search query on the site.

Seems like this would be a great solution, then. Instead of always going to invidio.us, the bangs will now take you to a page that lets you choose an instance.

Thus I am concerned that this is not in the best interest of those using Bangs,
as they seemingly want a fast and easy access to the results they requested, without going through middleman, like DuckDuckGo's Search Results or a page of instances. What do you think?

[CSDUMMI]

Another question in a broader view on the end of the invidio.us instance:
Can you move your Invidious account registered on invidio.us easily to another instance?

[bptato]

Another question in a broader view on the end of the invidio.us instance:
Can you move your Invidious account registered on invidio.us easily to another instance?

from https://omar.yt/posts/stepping-away-from-open-source:

If you have an account on the site, you can export your data here, which will allow you to import everything to another instance or self-host.

[billmetangmo]

@CSDUMMI

Another question in a broader view on the end of the invidio.us instance:
Can you move your Invidious account registered on invidio.us easily to another instance?

For me moving invidious account just mean moving your subscriptions. So you just need to export it in official instance and import it to your new instance and that's all.

[CSDUMMI]

My solution to the problem of the DuckDuckGo Bangs:
The Bangs refer to a web service, that takes a list of instances and healths from invidio.us.
On every request it redirects to one of those instances with good health, which is chosen randomly.
Thus you would be directly getting to your results on an instance of invidious and no single healthy instances would be overly burdened.

[cloudrac3r]

So what you want is, if the previous site was duckduckgo.com, then the redirect page should automatically choose where to send you without interaction?

[CSDUMMI]

No, that would also catch people, just coming there from the Search Results of DuckDuckGo and I don't know if that is even possible.
I would create a service under a different name (invidious-auto-redirect.de?)
that fetches a list of services from invidio.us and always redirects to a random choice amongst those with high health.
The Bangs in DuckDuckGo would then be updated to this services, which would act very much like a load balancer for the known Invidious Instances.

[nathgit]

What is the recommended instance to move to in terms of privacy/security etc;?

[cloudrac3r]

@nathgit - That could be a whole separate debate. Go to a different thread.

@CSDUMMI - Good idea, but I don't think it's possible to edit a bang now that it's been created. Is it?

[B0pol]

There is an "Update an existing bang button" here.

https://duckduckgo.com/newbang

[cloudrac3r]

Great idea, except that I'm pretty sure it takes them more than 6 months to review bangs. At least it did last time I submitted one. They keep building up the backlog and then clear it out every so often.

[B0pol]

Then on your redirect site you can handle https://invidious.instance.example/search?q=something+here requests

[CSDUMMI]

And what about those embeded Videos?

<iframe id='ivplayer' width='640' height='360' src='https://invidio.us/embed/hsxukOPEdgg' style='border:none;'></iframe>

You can't ask for every one of those to please choose an instance and then we'll show you your video, or can you do that? Should you do that?

If nothing is done about this, a ton of documents will be broken.

[cloudrac3r]

I don't think that many people used the embeds. I never saw one in the wild. I _can_ make it say click here to choose an instance, and it will in fact already do that at the moment.

_Should it?_ I don't know. Looking for feedback on this one.

[CSDUMMI]

Imagine somebody has used an embed
in some post or document.
They may not use it very often,
but when they do it is important,
like in a presentation, etc.

In such a (critical) situation an unexplained 404 is
really bad.
So, I think it wouldn't be worthless at least to provide
a link.

[CSDUMMI]

Another thing we should think about
are the RSS Feeds provided by Invidious.
I use them for some of my channels the RSS Feeds Invidous provides.
Should we put into each of those either:

• A redirect to a randomly choosen good health instance
• A 404 Error with a notice about the shutdown and a list of alternate nodes (shuffled). This could be represented very differently from RSS Reader to Reader.
• An Error as RSS Feed with Links to Instances.

[CSDUMMI]

I have been making quite a few points during this day:

What should happen to the DuckDuckGo Bangs?

• The Bangs will be treated no differently to any other request, creating an extra step for anyone using them.
• The Bangs are pointed at an alternate service, that automatically assigns them a randomly selected instance with good health to redirect to

Embed Videos

Embedded Videos to Invidious can be either:

• Broken with a plain 404 Error
• List of Links to other Instances shuffled. (This is what happens currently with @cloudrac3r's proposal)
• The two above combined
• Direct redirect to a healthy instance without user interaction

RSS Feeds

Invidious provides RSS Feeds for Channels and those will no longer be available.
Instead there could be:

• Automatic redirect to another randomly chosen good health instance.
• A 404 Error with a list of links to other instances as body. We don't know how the different RSS Readers will treat this.
• A standard RSS Feed generated with a list of links, similar to what @cloudrac3r did here for HTML, just for RSS.

I think that's all.

[trymeouteh]

Why not have invidio.us be a homepage for the Invidious project and also have a link to instances.invidio.us to find an instance.

[cloudrac3r]

@trymouteh - because the purpose of this is to not break old links, which means allowing the person to easily get to the same page of content on another instance. A link to instances.invidio.us takes them to an instance, but not to the page they wanted to see. Only technical users would be able to construct a new URL to do this.

@CSDUMMI - For RSS feeds, I think just serving a 301 to YouTube's regular RSS would suffice. Yes, this does share the feed collector's IP address with YouTube, but I personally don't think that is a problem. This should technically work since I believe popular feed readers should be able to handle status code 301 just fine.

[ChxseH]

I agree with the whole "auto redirect to a healthy instance" to avoid one getting beat up, just creates an issue of your default playback settings per instance are different, requiring you to sign up multiple times if you want "your settings" to stick no matter what instance you get, not a big deal IMO.

[cloudrac3r]

requiring you to sign up multiple times if you want "your settings" to stick no matter what instance you get, not a big deal IMO.

It's a pretty big deal, and it's what I'm trying to avoid by displaying the selection, so that the person can pick a site that they either trust or are already signed up on.

Ooh, I got it! We can just use localStorage to save the order in the browser, so that it doesn't change between refreshes. And then I can add a checkbox or something that could allow people to automatically go to the selected site from then onwards.

[ChxseH]

We can just use localStorage to save the order in the browser, so that it doesn't change between refreshes. And then I can add a checkbox or something that could allow people to automatically go to the selected site from then onwards.

+1 on this idea! That's true now that I think about it, switching instances/what not would be really annoying.

[trymeouteh]

Will instances.invidio.us still be there after Sept 1st?

[scaredmushroom]

Can you integrate the country icons shown on instances.invidio.us also in the instance list?

[8chanAnon]

I'm not seeing any discussion about this important issue taking place anywhere but here. The need to be registered is a bit of a problem. Many (most?) people who would like to talk about this are not registered on GitHub.

To facilitate the discussion, I would like to offer a forum on my own board at 8kun. No registration is needed there. I have created a special thread for this topic. Please spread the word and contribute to the conversation. Thank you.

Link to board: https://8kun.top/alleycat/catalog.html
Link to thread: https://8kun.top/alleycat/res/574.html

[cloudrac3r]

actually we already have matrix #invidious:matrix.org and IRC #invidious on freenode where there is a lot of chatter

[8chanAnon]

I don't use IRC or whatever that matrix thing is. Doesn't hurt to expand the potential audience.

[Lurux]

I don't think that many people used the embeds. I never saw one in the wild. I can make it say click here to choose an instance, and it will in fact already do that at the moment.

I do use an embed invidious video on a small privacy site I'm creating here: https://lurux.github.io/foss-central/

Ooh, I got it! We can just use localStorage to save the order in the browser, so that it doesn't change between refreshes. And then I can add a checkbox or something that could allow people to automatically go to the selected site from then onwards.

I don't think saving the order is necessary (and actually don't really like the idea), but I agree an auto-select checkbox would be a great solution for this, at least for embed and bangs.

If the checkbox is set to always redirect users automatically, they won't be able to switch between instances unless they specifically know about instances.invidious.org, which should not be taken for granted.

[Kein]

How about an optional cookie (we assume user has JS disabled) and a checkbox on the redirect that will redirect user to the top healhiest instance automatically? Something like:

_☑️ Redirect automatically to the top available instance_

So if user does not care about feeds and account, he can basically set cookie and be auto-redirected therefore making whole experience of using invidio.us not much different in the end.

[cloudrac3r]

If the person has JS disabled then the site won't be able to redirect them automatically because it does not use server-side rendering at omarroth's request.

[Kein]

Fair enough. Well, then I dont much else that can be done. A lot of users already use extensions that redirect them to specific instance they want.

[Lurux]

How about an optional cookie (we assume user has JS disabled) and a checkbox on the redirect that will redirect user to the top healthiest instance automatically

Therefore using localStorage (many users periodically clear their cookies): that's another great option. I think being able to auto-select either a specific instance _or_ the top healthiest instance would be good.

[chaosmonk1]

Whatever approach is taken, I think most (only?) really important thing is to keep embeds functional so as not to ruin articles and discussions that linked to invidio.us videos while it was the flagship instance.

[cloudrac3r]

I think most important thing is to keep embeds functional

As already mentioned, iframe embeds will change from being the video to being the "select an instance" page. Upon selecting an instance, the iframe will then display the video. The "watch on youtube" link will also show youtube's standard embed in the iframe.

[elypter]

Ooh, I got it! We can just use localStorage to save the order in the browser, so that it doesn't change between refreshes. And then I can add a checkbox or something that could allow people to automatically go to the selected site from then onwards.

there should be a way for users to revert back to showing the selection page. i suggest a special link that removes the cookie and a highlighted advice to bookmark said link below the checkbox. it should also automatically redirect to the selection page if the preferred instance is offline.

it would be nice to also have links to other alternative youtube frontends. especially if invidio.us instances may become less relyable due to youtube changing it might be helpful to have other choices without having to use youtube. even sites that just use a youtube embed are better than going directly to youtube.

[elypter]

Another question in a broader view on the end of the invidio.us instance:
Can you move your Invidious account registered on invidio.us easily to another instance?

is user data stored in a hosted database? if so then one master instance could run the db and a set of slave instances just connect to that database. this way you could balance load between a set of instances but still use the same account on them. the owners of those instances must trust each other though or must be one person.

[omarroth]

Looks and works great. Thank you so much for your hard work @cloudrac3r.

[cloudrac3r]

@omarroth Looks like you've got it all set up, though I have one note: paths under /static/ should preferably be served with the header cache-control: max-age=2592000, public.

Doing this is not _necessary,_ but pages will load faster and the server will have less work if you do.

You can see an example of what this header should look like here:

[ ] » curl 'https://bibliogram.art/static/css/classic.css?statichash=x' -I
HTTP/2 200 
server: nginx/1.14.2
date: Tue, 01 Sep 2020 14:08:04 GMT
...
cache-control: max-age=2592000, public
...

I'm not entirely sure how to do this in nginx, but add_header may help. Only for paths in /static/, though!

Your etags settings are probably fine, no need to change those.

[cloudrac3r]

I'm turning off notifications here - please tag me back in if anyone needs my attention.