Invidious: Livestreams unavailable on self-hosted instances

[omarroth]

Howdy!

Got this issue with a live video just now:

> VIDEOJS: WARN: Problem encountered with the current HLS playlist. HLS request errored at URL: https://mydomain.com/videoplayback/id/yZytPU1gOBw.1/itag/92/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/gir%3Dyes%3Bitag%3D139/sgovp/gir%3Dyes%3Bitag%3D133/hls_chunk_host/r1---sn-5hne6nsz.c.youtube.com/playlist_type/DVR/gcr/nl/ei/mVRTXMTWJNuNyQW1mJHYCA/mm/30/mn/sn-5hne6nsz/ms/nxu/mv/u/pl/24/sc/yes/keepalive/yes/mt/1548964219/ip/185.94.29.100/ipbits/0/expire/1548986617/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,playlist_type,gcr,ei,mm,mn,ms,mv,pl,sc/signature/8CB4055F7743C977FFB2201BBA9E5A2122A66DC8.4EAFCA4A6C8D9E86A35D21A2819728BB64BA4E21/key/dg_yt0/playlist/index.m3u8/sq/147/goap/clen%3D31073%3Blmt%3D1548965187406893/govp/clen%3D66833%3Blmt%3D1548965187406893/dur/5.005/file/seg.ts/fvip/1 Switching to another playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist.
> video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist. video.min.js:12:1480
> VIDEOJS: WARN: Problem encountered with the current HLS playlist. Trying again since it is the final playlist.

Updated an hour ago from repo.

_Originally posted by @tmiland in https://github.com/omarroth/invidious/issues/325#issuecomment-459488515_

[omarroth]

There was some discussion about this in the matrix server, so I thought I would copy the information here so it's easier to track.

Requesting manifests from self-hosted instances returns 403. Other resources load as expected. Videos, comments, images, and other content appear to load fine.

The relevant code is here, for reference:

get "/api/manifest/hls_variant/*" do |env|
  client = make_client(YT_URL)
  manifest = client.get(env.request.path)

  if manifest.status_code != 200
    halt env, status_code: manifest.status_code
  end

  env.response.content_type = "application/x-mpegURL"
  env.response.headers.add("Access-Control-Allow-Origin", "*")

  host_url = make_host_url(Kemal.config.ssl || CONFIG.https_only, CONFIG.domain)

  manifest = manifest.body
  manifest.gsub("https://www.youtube.com", host_url)
end

@tmiland, @Perflyst, and Tobi have encountered this. As far as I know (feel free to correct me here), all three instances are behind apache, using the instructions in the wiki.

Requesting the resource from localhost returns as expected. The issue is likely that Apache is adding headers to the request that Invidious or YouTube is not handling correctly.

[tmiland]

Howdy,

I'm adding my own header in Nginx, to allow embedding in FreshRSS and Nextcloud and so forth...

Added support for CORS on Nginx now, to see if that fixes the error. (Found it after searching for application/x-mpegURL)

Just have to wait for a live stream video to show up now 😂

[tmiland]

Here is my current (fine tuned and hardened) Nginx config for Invidious:

    # Nginx Proxy pass for invidious
    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_hide_header X-Frame-Options; # Strip header from scripts
        # Add own header
        add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
        add_header Public-Key-Pins 'pin-sha256="xxxxxxxxxxxxxxxxxxxxxxxxxxxx"; pin-sha256="xxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; max-age=5184000; includeSubDomains' always;
        add_header X-Frame-Options "sameorigin;";
        add_header Content-Security-Policy "frame-ancestors 'self' nextcloud.mydomain.com rss.mydomain.com;";
        add_header X-XSS-Protection "1;mode=block";
        add_header X-Content-Type-Options "nosniff";
        add_header X-Permitted-Cross-Domain-Policies "none";
        #
        # Wide-open CORS config for nginx
        #
        if ($request_method = 'OPTIONS') {
          add_header 'Access-Control-Allow-Origin' '*';
          add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
          #
          # Custom headers and headers various browsers *should* be OK with but aren't
          #
          add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
          #
          # Tell client that this pre-flight info is valid for 20 days
          #
          add_header 'Access-Control-Max-Age' 1728000;
          add_header 'Content-Type' 'text/plain; charset=utf-8';
          add_header 'Content-Length' 0;
          return 204;
        }
        if ($request_method = 'POST') {
          add_header 'Access-Control-Allow-Origin' '*';
          add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
          add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
          add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
        }
        if ($request_method = 'GET') {
          add_header 'Access-Control-Allow-Origin' '*';
          add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
          add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
          add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
        }

        proxy_pass http://192.168.x.x:3000;
            proxy_read_timeout  90;
            proxy_redirect http://192.168.x.x:3000 https://invidious.mydomain.com;
    }

Done various tests, and it's all good. What remains is the live video, to see if it works. 👍

[Perflyst]

What remains is the live video, to see if it works.

Was this a "livestreams are working"?

[tmiland]

What remains is the live video, to see if it works.

Was this a "livestreams are working"?

No, that remains to be seen. 🙃

[tmiland]

Trying to get live video as referenced here: #https://github.com/omarroth/invidious/issues/227#issuecomment-441849018

But looks like it's not working? 🤔

Rarely get a live video in the sub. feed. Is there any other way to debug this?

[tmiland]

Okay, was able to test again now, got this error:

Specified “type” attribute of “application/x-mpegURL” is not supported. Load of media resource https://invidious.mydomain.com/api/manifest/hls_variant/expire/1549180609/signature/0B868DCC82919D99D44524A520B3FC77FD4D12BD.8E2D176757ABA6E88F5B3E3F0B1B7BCB550BA64A/key/yt6/source/yt_live_broadcast/ratebypass/yes/gcr/nl/ei/YUpWXPHZAoOX7AS__L2gDA/id/PHZVoKauNAM.0/ip/185.94.29.100/keepalive/yes/sparams/ei%2Cgcr%2Cgo%2Chfr%2Cid%2Cip%2Cipbits%2Citag%2Cmaudio%2Cplaylist_type%2Cratebypass%2Crequiressl%2Csource%2Ctx%2Ctxs%2Cexpire/maudio/1/go/1/playlist_type/DVR/itag/0/tx/23780920/requiressl/yes/txs/23780920%2C23780921/ipbits/0/dover/11/hfr/1/file/index.m3u8 failed.

Found this: https://video-dev.github.io/hls.js/stable/

And this: https://bugzilla.mozilla.org/show_bug.cgi?id=941351

Also did a search in "about:config" in firefox, and found "media.hls.enabled", it was set to false.

Set it to true, but still same problem.

Using Firefox 65 on Ubuntu 18.10.

[omarroth]

When accessing the manifest directly, does it return a 403?

[omarroth]

Is it possible that nginx is truncating the query? Trying to request something like this directly from YouTube returns 403.

[tmiland]

When accessing the manifest directly, does it return a 403?

I'm getting "Secure Connection Failed" 🤔

[tmiland]

It could be. I'll debug further regarding the header.

From nginx error log:

2019/02/03 04:06:47 [error] 28531#28531: *29225 upstream sent invalid chunked response while reading upstream, client: 192.xxx.x.x, server: invidious.mydomain.com, request: "GET /videoplayback?id=PHZVoKauNAM.0&itag=91&source=yt_live_broadcast&requiressl=yes&ratebypass=yes&live=1&cmbypass=yes&goi=160&sgoap=gir=yes;itag=139&sgovp=gir=yes;itag=160&hls_chunk_host=r1---sn-5hnekn76.c.youtube.com&gcr=nl&playlist_type=DVR&ei=fFhWXI6NIsL07ASh_qHIDQ&initcwndbps=1130&mm=30&mn=sn-5hnekn76&ms=nxu&mv=m&pl=24&sc=yes&keepalive=yes&mt=1549162952&ip=xxx.xx.xx.xxx&ipbits=0&expire=1549184220&sparams=ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,gcr,playlist_type,ei,initcwndbps,mm,mn,ms,mv,pl,sc&signature=30B7C0350C8542595DF3551777F9C4F4AAF436ED.332EF1BF0764E0B6772D0205DC3E2990F18BB263&key=dg_yt0&playlist=index.m3u8&sq=9785&goap=clen=20533;lmt=1549143252944261&govp=clen=14261;lmt=1549143252944257&dur=2.000&file=seg.ts&fvip=1 HTTP/2.0", upstream: "http://192.xxx.x.x:3000/videoplayback?id=PHZVoKauNAM.0&itag=91&source=yt_live_broadcast&requiressl=yes&ratebypass=yes&live=1&cmbypass=yes&goi=160&sgoap=gir=yes;itag=139&sgovp=gir=yes;itag=160&hls_chunk_host=r1---sn-5hnekn76.c.youtube.com&gcr=nl&playlist_type=DVR&ei=fFhWXI6NIsL07ASh_qHIDQ&initcwndbps=1130&mm=30&mn=sn-5hnekn76&ms=nxu&mv=m&pl=24&sc=yes&keepalive=yes&mt=1549162952&ip=xxx.xx.xx.xxx&ipbits=0&expire=1549184220&sparams=ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,gcr,playlist_type,ei,initcwndbps,mm,mn,ms,mv,pl,sc&signature=30B7C0350C8542595DF3551777F9C4F4AAF436ED.332EF1BF0764E0B6772D0205DC3E2990F18BB263&key=dg_yt0&playlist=index.m3u8&sq=9785&goap=clen=20533;lmt=1549143252944261&govp=clen=14261;lmt=1549143252944257&dur=2.000&file=seg.ts&fvip=1", host: "invidious.mydomain.com"

From access log:

192.xxx.x.x - - [03/Feb/2019:04:09:18 +0100] "GET /videoplayback?id=PHZVoKauNAM.0&itag=92&source=yt_live_broadcast&requiressl=yes&ratebypass=yes&live=1&cmbypass=yes&goi=160&sgoap=gir=yes;itag=139&sgovp=gir=yes;itag=133&hls_chunk_host=r1---sn-5hnekn76.c.youtube.com&gcr=nl&playlist_type=DVR&ei=fFhWXI6NIsL07ASh_qHIDQ&initcwndbps=1130&mm=30&mn=sn-5hnekn76&ms=nxu&mv=m&pl=24&sc=yes&keepalive=yes&mt=1549162952&ip=xxx.xxx.xxx.xxx&ipbits=0&expire=1549184220&sparams=ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,gcr,playlist_type,ei,initcwndbps,mm,mn,ms,mv,pl,sc&signature=44A336965BAD31F5299F76EBB3082D4C771632EB.4256D21CFCB343BC8787465D39673F86E784BE4C&key=dg_yt0&playlist=index.m3u8&sq=9860&goap=clen=20691;lmt=1549143252944784&govp=clen=46097;lmt=1549143252944786&dur=2.000&file=seg.ts&fvip=1 HTTP/2.0" 200 0 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"

[tmiland]

BAM!

Took a while, but found a fix!

Add this to the nginx conf location directive:

        proxy_http_version 1.1;
        proxy_set_header Connection "";

It's something about chunk sizes -> https://forum.nginx.org/read.php?2,247883,247906#msg-247906

Searched for upstream sent invalid chunked response while reading upstream from ngix error log.

[Perflyst]

I tried following with apache2

ProxyAddHeaders Off
RequestHeader unset X-Forwarded-Host
RequestHeader unset X-Forwarded-For
RequestHeader unset X-Forwarded-Server

but livestreams are still not working. There is nothing like proxy_set_header Connection ""; in Apache but you can remove the headers which get automatically set by ProxyPass

[tmiland]

I tried following with apache2

```

ProxyAddHeaders Off
RequestHeader unset X-Forwarded-Host
RequestHeader unset X-Forwarded-For
RequestHeader unset X-Forwarded-Server
```

but livestreams are still not working. There is nothing like proxy_set_header Connection ""; in Apache but you can remove the headers which get automatically set by ProxyPass

Tested your instance with a random live video now, and am getting a different error msg in Chromium.:

Failed to load resource: the server responded with a status of 403 (Forbidden)

And the same as my initial error:

VIDEOJS: WARN: Problem encountered with the current HLS playlist. HLS playlist request error at URL: in Firefox.

Are all folder and execute permissions set correctly?

What does it say in your logs?

Haven't used Apache for ages....

You could do Nginx with Apache though: How To Configure Nginx as a Web Server and Reverse Proxy for Apache on One Ubuntu 18.04 Server

Makes life much easier in my opinion. 🙂

Here's my current working directive:

    # Nginx Proxy pass for invidious
    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_http_version 1.1;         # Fix for live streams
        proxy_set_header Connection ""; # Fix for live streams
        # Strip header from scripts
        proxy_hide_header X-Frame-Options;
        # Add own header
        add_header Strict-Transport-Security "max-age=300; includeSubDomains";
        add_header Public-Key-Pins 'pin-sha256="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; pin-sha256="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; max-age=5184000; includeSubDomains' always;
        add_header X-Frame-Options "sameorigin;";
        # Allow embedding in Nextcloud and FreshRSS-reader.
        add_header Content-Security-Policy "frame-ancestors 'self' nextcloud.mydomain.com rss.mydomain.com;";
        # Response from browser: "Content Security Policy: Ignoring ‘x-frame-options’ because of ‘frame-ancestors’ directive."
        add_header X-XSS-Protection "1; mode=block";
        #add_header X-Content-Type-Options "nosniff";
        add_header X-Permitted-Cross-Domain-Policies "master-only";
        #
        # Strict CORS config for nginx
        #
        if ($http_origin ~* "^https?://invidious\.mydomain\.com$" ) {
            add_header Access-Control-Allow-Origin $http_origin;
                add_header Vary: User-Agent;
        }
        proxy_pass http://192.xxx.x.x:3000;
        proxy_read_timeout  90;
        proxy_redirect http://192.xxx.x.x:3000 https://invidious.mydomain.com;
    }

[r4rdsn]

I'm still getting errors when trying to get livestreams on my instance despite the configuration by tmiland.
The player shows "The media could not be loaded, either because the server or network failed or because the format is not supported".
Here's the sample of Chromium's console output from video.min.js:

GET https://tube.alfbot.xyz:3000/api/manifest/hls_variant/.../index.m3u8?local=true net::ERR_CONNECTION_REFUSED
VIDEOJS: ERROR: (CODE:4 MEDIA_ERR_SRC_NOT_SUPPORTED) The media could not be loaded, either because the server or network failed or because the format is not supported. 

[stale[bot]]

This issue has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely outdated. If you think this issue is still relevant and applicable, please let us know.